2024
Teaching and Learning International Survey
(TALIS 2024) Main
Study Recruitment and Field Test
OMB# 1850-0888 v.11
Supporting Statement Part B
National Center for Education Statistics
Institute of Education Sciences
U.S. Department of Education
Washington, D.C.
revised June 2022
revised August 2022
revised May 2023
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
The 2024 Teaching and Learning International Survey (TALIS 2024) will acquire information from classroom teachers of International Standard Classification of Education (ISCED) Level 2 programs and the principles of their schools. ISCED Level 2 is also known as lower secondary education; it begins around age 11 and usually lasts between 2 and 6 years. Middle school and junior high (grades 7 through 9) in the United States are classified as ISCED Level 2.
For the main study, the universe of schools is all schools with at least one ISCED Level 2 program in all states of the U.S. and the District of Columbia. Teachers who teach in special needs-only schools, who teach exclusively to adults, who serve as occasional or emergency teachers, or who are on long term leave are not within the scope of the study. Special education teachers working in regular public and private schools are eligible to participate.
For the main study, within schools, the school administrator and a random sample of up to 35 ISCED Level 2 teachers will be selected to participate. In schools with 35 or fewer TALIS-eligible teachers who instruct at grades 7, 8, or 9, all teachers who teach at the target grade levels will be asked to participate. In schools with greater than 35 eligible teachers instructing at grades 7, 8, or 9, 35 teachers at the target grades will be randomly selected to participate.
Selected teachers will be randomly assigned to either the core TALIS questionnaire or the Teacher Knowledge Survey (TKS) questionnaire at a ratio of 4:3 within schools. For example, for schools with 35 or more eligible teachers, 20 teachers will be assigned to receive the core questionnaire, while 15 teachers will be assigned to receive the TKS questionnaire.
For the field test, the universe of schools was limited to 10 states: Connecticut, Delaware, Iowa, Idaho, Louisiana, Michigan, Oklahoma, Rhode Island, Utah, and West Virginia. Procedures for selecting administrators and teachers within schools were the same as planned for the main study.
The Technical Standards for TALIS 2024 field test and main study established by the international governing board (IGB) include the following:
The teacher sample size must be a minimum of 200 ISCED Level 2 teachers for the field test and 4,000 ISCED Level 2 teachers for the main study, or the National Defined Target Population.
The school sample size must be a minimum of 30 schools for the field test and 200 schools for the main study, or all schools that have ISCED Level 2 teachers in the National Defined Target Population.
School response rates must be at or above 75 percent of sampled schools to enable a nation to receive the highest rating (“good”) as OECD adjudicates whether a nation’s data “are fit to provide policy relevant, robust international indicators and analysis on teachers and teaching in a timely and cost effective manner” (OECD, 2019). If a response rate is below 75 percent, an acceptable response rate can still be achieved through agreed upon use of replacement schools. Although replacement schools could be called upon as substitutes for non-responding schools, national project managers (NPMs) will be encouraged to do all they can to obtain the participation of the schools in the original sample. As the number of replacement schools increases, the sample loses its probabilistic features and becomes increasingly “purposive.” This can undermine the reliability, validity, and interpretability of the country’s results. Responding schools that yield at least 50 percent of responding teachers will be considered participating schools; schools that fail to meet that threshold will be considered as “non-participating” even though the number of responding teachers may be enough to contribute to some of the analyses. Countries that experience less than 75 percent school participation after replacement must demonstrate convincingly that their sample is not significantly biased. TALIS establishes three response rate zones—good, fair, or poor. “Good” means the country’s data will be included in the international sample. “Fair” means that the country’s data is a candidate for not being reported in international comparisons because the participation rate after replacement was less than 75 percent. However, in most cases in the past, evidence has been provided by the countries concerned that nonresponse bias was negligible. “Poor” means that the country’s data will not be included in the international comparisons. The IGB makes the final decision on whether to include the country’s data in international comparisons while considering various other factors.
The OECD target for teacher response rates is at or above 75 percent of sampled teachers. Teacher response rates below 75 percent will lower the adjudication outcome. For example, if a nation obtains a school response rate at or above 75 percent and a teacher response rate at or above 75 percent, the fitness of the data will be adjudicated by OECD as “good.” Should either of these response rates fall to between 50 percent and 75 percent, the adjudication will fall within the “fair” category.
TALIS 2024 will employ a stratified two-stage probability sampling design. This means that teachers (secondary stage units or secondary sampling units (SSU)) will be randomly selected from the list of in-scope teachers in each of the randomly selected schools (first stage units, or primary sampling units (PSU)).
The universe of interest is comprised of schools where ISCED Level 2 education is provided, the principals of these schools, and teachers at these schools who teach at least one ISCED Level 2 class. Following Indicators of Education Systems (INES) data collection definitions “the formal definition of a classroom teacher is a person whose professional activity involves the planning, organizing, and conduction of group activities whereby students’ knowledge, skills, and attitudes develop as stipulated by educational programs. In short, it is one whose main activity is teaching.” (OECD, 2004).
TALIS was designed to provide data on all ISCED Level 2 teachers in a participating country. TALIS identified policy issues that encompass the classroom, teacher, the school, and school management. No subject matter was excluded from the scope of TALIS. Thus, coverage of TALIS extends to all teachers of ISCED Level 2 and to the principals of the schools where they teach.
An ISCED Level 2 teacher is one who, as part of his or her regular duties in school, provides instruction in programs at ISCED Level 2. In the U.S., ISCED Level 2 teachers are those who provide instruction at grades 7, 8, and/or 9. Teachers who teach a mixture of programs at different levels including ISCED Level 2 programs in the target school are included in the TALIS universe. There is no minimum cut-off for how much ISCED Level 2 teaching these teachers need to be engaged in. However, the international target population of TALIS restricts the survey to those teachers who teach regular classes in ordinary schools and to the principals of those schools. Teachers teaching only adults or in special needs-only schools are not part of the international target population and are deemed “out of scope.” Special education teachers working in regular public and private schools are eligible to participate.
When schools are comprised exclusively of out-of-scope teachers, the school itself is said to be “out of scope.” Teacher aides, pedagogical support staff (e.g., guidance counselors, librarians), and health and social support staff (e.g., doctors, nurses, psychiatrists, psychologists, occupational therapists, and social workers) are not considered as teachers and thus are not part of the TALIS international target population.
For national reasons, some participating countries will choose to restrict the coverage of their national implementation of TALIS to parts of the country. For example, a province or state experiencing civil unrest or an area struck by a natural disaster could be removed from the international target population to create a national target population. Participating countries were invited to keep these exclusions to a minimum.
Ideally, all the members of the target population ought to be eligible for sampling and data collection. This is the option that TALIS chose and, therefore, the international survey population (those who can be surveyed) is identical to the international target population (those who should be surveyed).
Per OECD requirements, the following teachers will be excluded: teachers who also act as principals (no teacher data collected, but principal data collected); substitute, emergency, or occasional teachers; teachers on long-term leave; and teachers teaching exclusively to adults. Ultimately, samples of schools and teachers will be selected from the national survey population.
The population of schools for TALIS 2024 is defined as all schools containing grades 7, 8, and/or 9. As in TALIS 2018, the school sampling frame will be developed from the most up to date NCES Common Core of Data (CCD) and Private Schools Survey (PSS) datasets.
Participating countries will provide the international sampling contractor with a current and complete list of schools providing education at ISCED Level 2. This list will constitute the school sampling frame for TALIS and is expected to correspond to the survey population as defined and described on the Sampling Forms.
The international sample design for TALIS 2024 calls for a minimum of 30 schools in the field test and a minimum of 200 in the main study. Education systems are allowed to use substitute schools (selected during the sampling process) to increase the school response rate. One substitute school is allowed per sampled school in the field test and two per sampled school in the main study. To allow for reliable estimation, while allowing for some amount of non-response, the U.S. is selecting school samples slightly greater than the minimum required.
For the TALIS 2024 field test, we used the school sampling frame prepared for the 2022 National Assessment of Educational Progress (NAEP) which uses the 2019–20 CCD and PSS school data. We avoided, to the degree possible, schools sampled for the main studies of High School and Beyond (HS&B) 2022, International Computer and Information Literacy Study (ICILS) 2023, and Trends in International Mathematics and Science Study (TIMSS) 2023, all of which will also be collecting data in middle and high schools during the 2022–23 school year. The TALIS FT sample is a small convenience sample selected in just 10 states. Given the small size and design of the sample, we do not expect a substantial overlap with any of the other studies. For any TALIS sampled schools that does overlap with another study, the school will not be recruited for TALIS. This is acceptable for a FT convenience sample.
International standards do not require a formal probability sample of schools for the TALIS field test. It is sufficient that the samples of schools be inclusive of a broad range of schools from across the U.S. The U.S. selected a sample of 70 schools each with one substitute school from across 10 states,1 with 40 schools ultimately participating. While the expected school participation rate is lower than what would be desired for an operational study, it meets the international requirement of at least 30 schools for the field test. This sample size is consistent with other field tests at NCES (including TALIS 2018) and will provide an adequate teacher sample for the TALIS 2024 field test. Among the 70 schools, 60 will be public schools and 10 will be private schools. This allows for school and teacher non-response and for school level and within-school level exclusions.
While the field test sample was purposive by constraining the universe of schools to include only specific states, stratification for the U.S. field test used a similar overall scheme as the main study, except for the variables that were used as explicit strata. Explicit strata for the field test were limited to state and school control (public/private). Within each explicit stratum, we sorted schools by locality (urban/suburban/town/ rural), race/ethnicity status (enrollment above or below 15 percent Black, Hispanic, Asian, American Indian and Alaska Native, Hawaiian/Pacific Islander, or two or more races) and the number of ISCED Level 2 students. The probability of selection for each school was proportional to a measure of size (MOS), which was defined as the square root of the number of ISCED Level 2 students. The international sampling contractor selected the sample using a systematic sampling method to select from the sorted list of schools.
We implemented rigorous effort to secure the participation of all sampled schools. However, not all schools chose to participate. Consistent with TALIS 2018, as we included schools in the sample, we designated a neighboring school from the frame as a substitute. If the original school declined to participate, we contacted the substitute.
For the TALIS 2024 main study, we will use the school sampling frame prepared for NAEP 2024, which will use the 2021–22 CCD and PSS school data. The stratification will use both explicit and implicit stratifying variables. Explicit strata for the main study will be defined by school control and grade structure (schools within grades 6-8 or 7-9, schools with grades 9-12, or other schools with grade structures that include at least one ISCED 2 grade). Given the small number of private schools with a 6-8 or 7-9 grade structure, we will collapse this stratum with the private schools with the 9-12 grade structure. Within each explicit stratum, schools will be sorted by Census region, locale (urban/suburban/town/rural), percent minority students, state, and a MOS based on the number of ISCED 2 students (for a measure of school size).
The U.S. plans to select a sample of 262 schools each with two substitute schools with the expectation that 244 schools will ultimately participate. Among the 262 schools, 242 will be public schools and 20 will be private schools. This allows for school non-response and for school-level and within-school-level exclusions.
To sample ISCED Level 2 teachers within each participating school, the international procedures call for the designation of a school coordinator in each school. In addition to assistance with arranging for the study within each school, the school coordinator of each school is asked to prepare a list of all eligible teachers in the school using a standardized Teacher Sampling Form (TSF). In the U.S., eligible teachers are those who conduct at least one class (in any subject matter) of 7th, 8th, and/or 9th graders. If a teacher conducts at least one class with students in this grade range, s/he is considered an eligible teacher. Any teacher who does not teach any class at these grade levels is out of scope and will not be sampled for TALIS.
The TSF includes the following information: identification number (number generated by sampling software); name of teacher; birth year; sex; main teaching domain (main subject taught); exclusion status (i.e., excluded or not); and email address. U.S. school coordinators will submit this information to Westat on behalf of NCES. This information will be entered into the sampling software by Westat using WinW3S, a proprietary sampling software developed by the TALIS international contractor and used in other international studies such as TIMSS and the Progress in International Reading Literacy Study (PIRLS). The information collected from the TSF will remain with Westat. The names and email addresses of teachers are not associated with the TALIS database in any way; these are collected to allow for follow-up, should it become necessary. Once it is determined that no further follow-up is needed, all TSFs will be destroyed.
The international sample design calls for selecting teachers as follows: in schools with 35 or fewer eligible teachers, all eligible ISCED Level 2 teachers will be selected to participate; in schools with more than 35 eligible teachers, 35 eligible ISCED Level 2 teachers will be randomly selected to participate. As well, the international design calls for randomly assigning teachers to either core TALIS or TKS at a 4:3 ratio.
For the field test, the U.S will sample 800 teachers for the core TALIS questionnaire, plus an additional 600 teachers for the TKS questionnaire.
During the field test, the U.S. learned that more schools than expected had fewer than 35 eligible teachers. Therefore, schools provided smaller overall number of teachers for participation. Additionally, the U.S. experienced slightly lower response rates. Combined these two factors resulted in a smaller than expected yield of responding teachers. To maintain sufficient analytic power for the main study, the U.S will sample 4,880 teachers for the core TALIS questionnaire and 3,660 teachers for the TKS questionnaire.
To address any nonresponse that occurs at the school and teacher levels, we will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the international contractor calculates weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification.
For the TALIS 2018 main study, international requirements stipulated that the minimum school response rate target needed to be 75 percent for all education systems. A minimum of 50 percent of schools from the original sample of schools were required to participate for an education system’s data to be included in the international database. Education systems were allowed to use substitute schools (selected during the sampling process) to increase the response rate once the 50 percent benchmark had been reached. A school was considered participating if the principal responded to at least one item in the principal questionnaire.
TALIS 2018 also required a minimum participation rate of 75 percent of sampled teachers from schools within each education system. A teacher was classified as a participant if he or she responded to at least one item in the teacher questionnaire. Data from education systems not meeting this requirement could be excluded from international reports.
The TALIS 2018 standards also required that nonresponse bias analyses be conducted if school response rates were less than 75 percent. The National Center for Education Statistics (NCES) standards stipulate that a nonresponse bias analysis is required for any stage of data collection with a weighted unit response rate of less than 85 percent.
For the TALIS 2018 main study, using the OECD criteria for inclusion and exclusion, the U.S. achieved a 63 percent weighted response rate for schools in the original sample and an overall 78 percent weighted response rate for schools when substitutes were included in the calculation. For teachers in participating schools, the U.S. achieved a weighted response rate of 90 percent.
Our approach to maximizing school and teacher response rates in the TALIS 2024 field test includes the following:
Closely coordinate with NAEP State Coordinators (NSCs), with support from our International Studies Liaison (ISL) for recruitment activities for public schools in most states;
Use Gaining Cooperation Recruiters (GCRs) as the primary contact for private schools and some public schools;
Directly reach out to states, districts, and schools, with a focus on geographic areas and types of schools that present recruitment challenges;
Use a school coordinator (SC) who is a member of the school staff to help with coordination and who is responsible for submitting teacher lists;
Provide incentives for schools, school coordinators, and teachers (see Section A9);
Maintain contact with schools and school coordinators at set intervals throughout the year preceding the survey; and
Collect data in February-March 2023 (Field Test) and February-March 2024 (Main Study) to avoid conflicts with state testing that normally occurs in late spring/end of school year.
Given those approaches, we anticipate achieving similar response rates in TALIS 2024 to what was achieved in TALIS 2018.
Participation in the field test is an international requirement for participating in the TALIS 2024 main study. To prepare for the main study in 2024, TALIS countries conducted the field test in February of 2023, primarily to evaluate newly developed survey items but also to test the survey operations. The field test included new teacher and school questionnaire items developed to address the TALIS 2024 framework. The field test was also used to test school recruitment, data collection, and data management procedures in preparation for the main study.
Results from the field test will be used to refine the TKS instrument and to determine whether TKS is included as part of the main study. NCES will monitor quantitative metrics, such as field test response rates for the TKS, breakoff patterns, item-level missing data rates, and response patterns for TKS items. NCES will also monitor recruiting and field operations for qualitative information, such as volunteered feedback from schools, school coordinators, or TKS participants. NCES will evaluate the qualitative and quantitative information and provide feedback to the international contractor that is developing the TKS, including recommendations to modify or remove items. Further, NCES will consider whether the TKS is likely to yield high quality data without affecting the response rates for Core TALIS, which will be administered in schools at the same time. Based on this evaluation, NCES will make a final determination regarding whether to include TKS in the main study.
The TALIS main study offers the opportunity for teachers and principals to provide input into education analysis and policy development. Cross-country analysis of this data will allow countries to identify other countries facing similar challenges and to learn from other policy approaches. School principals and teachers will provide information about issues such as the professional development they have received; their teaching beliefs and practices; the review of teachers’ work and the feedback and recognition they receive about their work; and various other school leadership, management, and workplace issues.
The data from the TALIS main study will be reported via an international report that will be released in 2026. NCES intends to produce a U.S. national report based on the TALIS main study data received from the international sponsors. The U.S. national report will be released at the same time as the international report from OECD. This report will be tailored to a general audience and will report the U.S. results relative to other countries. There will not be a report on the field test results.
Many people at OECD and other organizations around the world have been involved in the design of TALIS. Overall direction for TALIS in the U.S. is provided by Mary Coleman, the TALIS Governing Board representative, and Stephen Provasnik, the International Activities Branch Chief.
Mary Coleman
National Center for Education Statistics
U.S. Department of Education
Stephen Provasnik
National Center for Education Statistics
U.S. Department of Education
References
OECD (2004), OECD Handbook for Internationally Comparative Education Statistics: Concepts, Standards, Definitions, and Classifications, OECD, Paris.
OECD (2019), TALIS 2018 Results (Volume I): Teachers and School Leaders as Lifelong Learners, TALIS, OECD Publishing, Paris, https://doi.org/10.1787/1d0bc92a-en.
1 The 10 states are Connecticut, Delaware, Iowa, Idaho, Louisiana, Michigan, Oklahoma, Rhode Island, Utah, and West Virginia. A similar model of state selection was successfully used in the recent field tests for PISA and PIRLS. This selection of states provides an adequate regional coverage of the United States, while avoiding sampling Florida, Texas, California, New York, and New Jersey: states that typically are heavily sampled due to their population characteristics.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2023-07-29 |