Part B TIMSS2023 Field Test Sampling and Recruitment

There are four target populations for the TIMSS 2023 field test: two school populations and two student populations. The two school target populations include schools offering fourth-grade instruction and those offering eighth-grade instruction in the winter/spring of 2022 in the 50 states and the District of Columbia. These schools include:

Public schools, including charter and magnet schools;
Private schools;
Bureau of Indian Education schools; and
Domestic Department of Defense schools.

Although the following schools are eligible for TIMSS 2023 and could be considered for the field test, we plan to exclude them from the field test in line with our proposal to exclude them from the main study. However, we note that changes in the number of virtual schools, the number of students attending virtual schools, and/or school classifications in light of increased remote learning during the coronavirus (COVID-19) pandemic may require adjustment of the virtual schools exclusion. Currently, our plan excludes:

Special education schools,
Virtual schools, and
Schools that include temporary housing like correctional facilities and hospitals.

There are two student target populations for the field test of TIMSS 2023: fourth- and eighth-grade students enrolled in winter/spring of 2022 in the United States (50 states and the District of Columbia) in a school in one of the two school target populations.

B.2 Statistical Methodology

Field Test Sampling Plan and Sample.

The purpose of the field test is to evaluate survey instruments and procedures, including classroom sampling procedures. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize impact on these entities while also ensuring that the field test data are collected effectively. In accordance with the TIMSS International Study Center, the field test participation goal will be 35 schools offering fourth grade and 35 schools offering eighth grade. To maximize the efficiency of data collection, only schools that report at least 45 fourth-grade or 45 eighth-grade students will be included in the sample. The field test sample is not required to be nationally representative but should include a diverse sample of schools, covering such features as public (including charter schools), private, large, small, urban, and rural schools, and schools from a variety of different states.

The 2018–2019 Common Core of Data and 2018–2019 Private School Universe Survey (or later, if available) will be used to identify the set of schools eligible for TIMSS 2023 field test, and the reported fourth- and eighth-grade enrollment counts will be used to create the student population distributions. The proposed sampling design is based on quasi-nationally representative samples of schools from six purposively selected geographically diverse metropolitan statistical areas (MSAs). Within each selected MSA, eligible schools will be stratified by school type, poverty status (high/low), and locale. Participation goals for each stratum will be determined by distributing the 35 schools for each sample approximately proportional to the student population in each stratum eligible for the 2023 TIMSS main study. A stratified random sample of 105 schools will selected and from these, 35 schools will be selected for initial recruitment. The remaining 70 schools will be available as a pool of replacement schools, and schools will be selected as needed to achieve the stratum-specific school participation goals.

Participating schools will be asked to provide a list of students and classes. Students will be selected for participation by drawing a random sample of two classes in each school using software provided by the international TIMSS study coordinator that will also be used in the main study. At grade 8, students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. Smaller classes may be combined to form ‘pseudoclasses’ for sampling. All selected students will be asked to participate in a combined TIMSS mathematics and science assessment. Only students in intact classrooms will be assessed at each grade. We estimate that on average 45 students will be selected from each school, and up to 10 percent of those students will be ineligible or excluded¹ for the field test, yielding approximately 1,400 fourth grade and 1,400 eighth grade students.

TIMSS samples are typically designed to minimize overlap with NAEP data collections happening in the same time frame. There are currently no NAEP data collections scheduled for 2022; however, if NAEP or other NCES data collections are determined to be scheduled in a similar time frame and we are provided with a list of participating schools, then steps will be taken to ensure the same schools are not selected for the TIMSS 2023 field test.

Class and student lists will be gathered from participating schools electronically using a secure electronic filing process (as explained in Part A). Electronic filing provides advantageous features such as efficiency and data quality checks. Schools will access the electronic filing system through a web site.

The main study sampling plan will be included in the next OMB package to be submitted in August 2021.

B.3 Maximizing Response Rates

Gaining cooperation from school districts and schools is paramount to the success of TIMSS 2023, and also the most significant challenge of the study. Given that classrooms are selected, student participation is not as great of a challenge. Historically, student participation rates in TIMSS have never fallen below 90 percent (see table 1). That said, it is important to U.S. TIMSS that students are engaged and try to do their best on the assessment.

Table 1. Historical TIMSS school and student participation rates

Year¹	Grade	School Participation Rate		Overall Student Participation Rate
Year¹	Grade	Before Replacement	After Replacement	Overall Student Participation Rate
2015	4	79	85	96
2015	8	78	84	94
2011	4	79	84	95
2011	8	87	87	94
2007	4	70	89	95
2007	8	68	83	93
2003	4	70	82	95
2003	8	71	78	94
1999	8	83	90	94
1995	4	86	NA	94

¹ TIMSS 2019 participation rate data have not yet been released.

Our approach to school recruitment is to:

Begin recruitment activities as early as possible;
Engage stakeholders at all levels throughout each cycle of TIMSS;
Obtain endorsements about the value of TIMSS from relevant national organizations, state and regional agencies;
Engage with the broad school community (i.e., not sampled schools) through involvement at small-scale school leader conferences and events, to provide an opportunity to test-drive our anticipated recruitment messages, build relationships with influential school-level leaders and highlight anticipated benefits of participation to build hope and excitement among schools that they may be selected into the study;
Inform Chief State Officers and state assessment directors about the sample of schools in their state;
Provide a framework alignment guide for state assessment and accountability offices, which provides information on how to compare and contrast the TIMSS framework with individual state frameworks;
Use the assistance of NAEP State Coordinators to recruit districts and schools, providing key state agency involvement in recruitment;
Send letters and informational materials to schools and districts. These letters will be customized by type of school. In addition, alternative versions are provided in Appendix A to account for school composition during recruitment, as different letters may be used for schools conducting learning virtually;
Train experienced NAEP State Coordinators about TIMSS;
Follow-up mailings with telephone calls and emails to explain the study and school involvement, including placing the TIMSS assessment date on school calendars;
Maintain continued contact until schools have built a relationship with the recruiter and fully understand TIMSS;
Use monetary school incentives of up to $800 for select schools;
Offer salient, interactive, topical webinars to participating district and school staff;
Provide school reports augmented with survey response data;

Produce and distribute a newsletter compiling relevant elementary and middle-school math and science research reports and practice guides from federal Regional Education Laboratories, Comprehensive Centers, and the What Works Clearinghouse for all levels;

Provide study updates for all levels on key study milestones and releases; and
Make in-person visits to some districts and schools, as necessary.

B.4 Purpose of the Field Test

The central goals for the field test are to evaluate new assessment items and background questions, and to ensure that classroom and student sampling procedures proposed for the main study are successful.

B.5 Individuals Consulted on Study Design

Overall direction for TIMSS is provided by Lydia Malley, National Research Coordinator, National Center for Education Statistics (NCES), U.S. Department of Education, in consultation with a number of NCES statistical staff.

The following persons are responsible for the statistical design of TIMSS:

Pierre Foy, TIMSS International Study Center, Boston College (617-552-6253); and
Peter Siegel, RTI International (919-541-6348).

RTI International is the contractor responsible for sampling, data collection, and data analysis:

Debbie Herget, Project Director, RTI International (919-485-7793);
Ben Dalton, Associate Project Director, RTI International (919-541-7228); and
David Wilson, Senior Statistician, RTI International (919-541-6990).

Analysis and reporting will be performed by:

National Center for Education Statistics, U.S. Department of Education;
TIMSS International Study Center, Boston College;
RTI International; and
AnLar, under contract to RTI International.

1 Students are considered ineligible if they are enrolled in the classroom but not in the target grade (grade 4 or grade 8) or if a student transferred from the school or class between the time that the student roster was prepared and the scheduled session at the school. Students are excluded if they meet the internationally defined criteria as having a functional disability, an intellectual disability, or if they are a non-native language speaker and unable to read or speak the language of the assessment. Students with functional or intellectual disabilities or have at least one year of English instruction or are able to overcome the language barrier would be included in the testing.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Millar, Kimrey
File Modified	0000-00-00
File Created	2021-03-29

Part B TIMSS2023 Field Test Sampling and Recruitment