Part B TIMSS 2019 Main Study Recruitment & Field Test

Trends in International mathematics and science study (TIMSS 2019)

main Study Recruitment and Field test

REQUEST FOR OMB Clearance

OMB# 1850-0695 v.10

Supporting Statement Part B

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

June 2017

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe

The respondent universe for the TIMSS 2019 field test is all students enrolled in grade 4 and grade 8 during the 2017-18 school year. The universe for the selection of schools is all types of schools in seven populous states. A sample of 40 schools will be selected for the field test, with the goal of obtaining participation from a minimum of 35 schools at each grade. Within sampled schools, students will be selected for participation by drawing a random sample of two classes. At grade 8, students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. All selected students will be asked to participate in a combined TIMSS mathematics and science assessment. Only students in intact classrooms will be assessed at each grade.

The respondent universe for the TIMSS 2019 main study is all students enrolled in grade 4 and grade 8 during the 2018-19 school year. A sample of 335 schools will be selected for the main study. Within sampled schools, students will be selected for participation by drawing a random sample of two classes. At grade 8, students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. Only students in intact classrooms will be assessed at each grade. Most selected students will be asked to participate in a combined electronically administered TIMSS mathematics and science assessment using the eTIMSS assessment platform. A sample of 1,500 students at each grade will take TIMSS on paper, in order to allow for a bridge analysis with paper-based trend lines.

B.2 Statistical Methodology

Field Test Sampling Plan and Sample

The purpose of the eTIMSS field test is to assess in the new eTIMSS platform legacy or “trend” TIMSS items as well as new items, and to ensure that classroom sampling procedures proposed for the main study are successful. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize impact on these entities while also ensuring that the field test data are collected effectively.

As required by the TIMSS international study center and the IEA, the field test sample is to consist of 40 schools and 1,400 students assessed at each grade level (4 and 8). The student sample will be obtained by selecting two classes from each school. At grades 4 and 8, mathematics and science teachers of students in selected classes will be asked to complete a teacher questionnaire. As the field test is designed only to test items, questions, and procedures, a probability sample of schools is not required. However, the sample must include a broad range of schools covering such features as public (including charter schools), private, large, small, urban, and rural schools, and schools from a variety of different states.

The school frame for the 2018 eTIMSS field test sample will be developed from the 2018 NAEP school frame. The data for public and private schools will come from the CCD and PSS, respectively, using the 2015-2016 school year data. The school sample for the field test will be a purposive sample and does not require complete coverage of schools for the target populations. The eligible schools in TIMSS include all schools with a fourth or eighth grade that are operating in selected seven states. When selecting the states, we will take into consideration the school samples already selected for NAEP 2018 and for the ICILS 2018 main study, the state assessment schedules that have conflicts during this time period, and the schools selected for the PISA 2018. This model of state selection was successfully used in the field test in previous TIMSS administrations and provided an adequate regional coverage of the United States while avoiding sampling very large states that typically are heavily sampled due to their population characteristics. The states will be spread across different regions. This selection will provide an appropriate geographic diversity for the selected schools. For efficiency, we will subset the school frame by eliminating schools that are unlikely to have two fourth- or eighth-grade classes.

The 2018 eTIMSS field test sample design is a stratified sample in which schools are selected with probability proportional to a measure of size (MOS), that is, the square root of the grade enrollment. We will use probability sampling not because it is required, but because it is an efficient way to select a sample of schools representing a variety of school characteristics.

Stratification for the field test will use a similar overall scheme as the main study. The stratification will be both explicit and implicit. Explicit strata will be defined by state and school type (public/private). Within each explicit stratum, schools will be sorted by locale (urban/suburban/town/rural), race/ethnicity status (race/ethnicity enrollment above or below 15 percent Black, Hispanic, Asian, American Indian and Alaska Native, Hawaiian/Pacific Islander, or two or more races), and the grade enrollment. Westat will select the school sample using an in-house sampling macro and a systematic sampling method to select from the sorted list of schools. The probability of selection for each school will be proportional to a MOS, defined as the square root of the grade enrollment. Using this MOS will lessen the chance of larger schools also being selected for the main study (2019) by reducing their chance of selection for the pilot.

A minimum of approximately 1,400 students per grade will be assessed across the participating schools. We will sample 40 schools per grade for the field test across the seven states with one and two replacement schools assigned per sampled public school, respectively. We will conduct the field test in the set of seven states with the aim of gaining the cooperation of at least five schools in each of the states. Assuming an overall low school response rate, we expect participation from at least 35 schools at each grade level in the eTIMSS field test.

The student sampling procedures for the eTIMSS field test will correspond as closely as feasible to what is planned for the main study, so as to try out the operational procedures for student sample selection. The sample will be selected by choosing two classes per school. Each participating school will be asked to submit an exhaustive list of students and classes (that is, one that accounts for each student in the grade exactly once). In cases for which one or more classes on the list has fewer than 10 students, smaller classes will be combined to form “pseudoclasses” for the purposes of sampling. Once the list of classes is submitted, we will use a sampling algorithm to select two classes (or pseudoclasses) with equal probability. The student sample will then consist of all students in the selected classes.

We plan to gather class and student lists from participating schools electronically using an adaptation of our secure e-filing process. E-filing was successfully used in TIMSS 2015, and provides advantageous features such as efficiency and data quality checks. Schools will access the e-filing system through the MyTIMSS web site.

Main Study Sampling Plan and Sample

The school sample design for the main study must be more rigorous than that for a field test. It must be a probability sample of schools (one sample for each of grades 4 and 8) that fully represents the entire United States. At the same time, to ensure maximum participation it must be designed so as to minimize overlap with other NCES studies involving student assessment that will be conducted around the same time.

The main study for TIMSS will take place in the spring of 2019, about two months after a very large NAEP assessment. NAEP will assess several thousand schools nationally, at grades 4 and 8. The NAEP sample will be relatively heavy in smaller states, and in a number of these states all eligible schools will be included in NAEP, especially at grade 8. Thus to be fully representative, the TIMSS sample may include some schools that will have participated in NAEP at the same grade. However, this number can be kept to minimum using the overlap control procedures outlined below.

Overlap control procedures in studies such as this, where stratified probability proportional to size samples of schools are selected, can be implemented via a procedure that applies Bayes Theorem to modify the conditional probability of selection of a given school for one study, depending upon its selection probability for a second study, and whether or not it was selected for that study. This approach was first documented in a survey sampling application by Keyfitz (1951)¹. The principles involved can be extended to more than two studies simultaneously, and a procedure for doing this is described by Chowdhury et al. (2000)².

The sample size for the TIMSS main study will be 335 schools at each of grades 4 and 8. The sampling frames of grade 4 and 8 schools will be obtained from the most current versions of NCES’s Common Core of Data (CCD) and Private School Survey (PSS) files, restricted to schools having grade 4 or 8 respectively, and eliminating schools in Puerto Rico, U.S. territories, and Department of Defense overseas schools.

The sample will be stratified according to school characteristics such as public/private, Census region, poverty status (as measured by the percentage of students in the school receiving free or reduced-price lunch in the National School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the selected sample of schools.

Schools will be selected with probability proportional to the number of estimated classes at the appropriate grade (4 or 8), with schools expected to have either one or two classes being given the same selection probability. The use of a probability proportional to sample design ensures that all students have an approximately equal chance of selection, since two classes will be selected from each school regardless of the size of the school. Note that we will modify this equal probability design in the following way. So as to increase the available sample size of students in high poverty schools, we will double the probability of selection of each school with at least 50 percent of students eligible for free or reduced-price lunch under NSLP, relative to other schools of the same size.

Student sampling will be accomplished by selecting two classes per school. Each grade 4 school will be asked to prepare a list of classes that is comprehensive and includes each grade 4 student in the school in one of the listed classes. As described above, schools will submit these class and student lists via secure E-filing. Grade 8 schools will be asked to prepare a similar list but with students listed in their mathematics classes. At either grade, any class with fewer than ten students will be combined with another class to form a ‘pseudoclass’ with at least ten students in it. We will then select two classes (regular classes or pseudoclasses) from each school, with equal probability, and all students in those classes/pseudoclasses will be included in the sample. If a school has only one or two classes, then all students in the grade will be included in the sample. At grade 8, mathematics classes are used for three reasons. First, this minimizes the burden on mathematics teachers, as only two mathematics teachers need to fill out a teacher questionnaire (but typically more than two science teachers are required to do so because the students in the two selected math classes often attend more than two different science classes). Second, it makes for sound data for conducting analyses of the extent to which classroom factors moderate the relationship of student factors to achievement (e.g., “Does having a well-qualified math teacher reduce the correlation between math achievement and parental education?”). Third, at grade 8, most students take one and only one mathematics class, and thus mathematics classes make for a foolproof partitioning of the eligible students. A subset of approximately 70 classes sampled at each grade level will take paper-based TIMSS, while the remainder will be assessed using the digitally based eTIMSS assessment platform.

Nonresponse Bias Analysis, Weighting, and Sampling Errors

It is possible that nonresponse will occur at both levels: school and student. If so, we will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the calculation of weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification (see Appendix D for more details).

B.3 Maximizing Response Rates

With the recent exception of TIMSS 2011, the most significant challenge in recruitment for TIMSS has been engaging the schools and gaining their cooperation. The circumstances that aided our success in 2011—the NAEP-TIMSS Linking Study and the involvement of NAEP State Coordinators—did not recur in 2015. However, there are important lessons to be learned from our TIMSS 2011 experience that were used in TIMSS 2015 and will be used in the TIMSS 2019 main study. Given that classrooms are selected, student participation is not as great of a challenge. Historically, student participation rates have never fallen below 90 percent (see table 1). That said, it is important to U.S. TIMSS that students are engaged and try to do their best on the assessment.

Table 1. Historical TIMSS school and student participation rates

Year	Grade	School Participation Rate		Overall Student Participation Rate
		Before Replacement	After Replacement
2015	4	79	85	96
	8	78	84	94
2011	4	79	84	95
	8	87	87	94
2007	4	70	89	95
	8	68	83	93
2003	4	70	82	95
	8	71	78	94
1999	8	83	90	94
1995	4	86	NA	94

Our approach to school recruitment is to:

• Obtain endorsements about the value of TIMSS from relevant organizations;

• Inform Chief State Officers and test directors about the sample of schools in their state.

• Use the assistance of NAEP State Coordinators to recruit districts and schools, providing key state agency involvement in recruitment;

• Send letters and informational materials to schools and districts. These letters will be customized by type of school;

• Train experienced NAEP State Coordinators about TIMSS;

• Implement strategies from NAEP’s Private School Recruiting Toolkit. This toolkit, developed for NAEP, includes well-honed techniques used to recruit a very challenging type of schools;

• Follow-up mailings with telephone calls to explain the study and schools involvement, including placing the TIMSS assessment date on school calendars;

• Maintain continued contact until schools have built a relationship with the recruiter and fully understand TIMSS; and

• Make in-person visits to some schools, as necessary.

B.4 Purpose of Pilot and Field Test and Data Uses

The central goals of the pilot was to test out items that have been transferred from paper to digital format, assess the performance of new items developed to take advantage of technology, and evaluate the mode effect of transitioning from paper to digital format. The purpose of the field test is to evaluate sampling and data collection procedures, new and legacy TIMSS items in the eTIMSS electronic platform, and student, teacher, and school questionnaires in preparation for the main study in 2019.

B.5 Individuals Consulted on Study Design

Overall direction for TIMSS is provided by Dr. Stephen Provasnik, National Research Coordinator, National Center for Education Statistics (NCES), U.S. Department of Education, in consultation with a number of NCES statistical staff.

The following persons are responsible for the statistical design of TIMSS:

• Pierre Foy. TIMSS International Study Center, Boston College (617-552-6253); and

• Sylvie LaRoche, Statistics Canada (613-863-9480).

Westat is the contractor responsible for sampling and data analysis:

• Chris Averett, Project Director, Westat (301-314-2492); and

• David Ferraro, Senior Statistician, Westat (301-251-4261).

Analysis and reporting will be performed by:

• National Center for Education Statistics, U.S. Department of Education;

• TIMSS International Study Center, Boston College; and

• American Institutes for Research, under contract to Westat.

1 Keyfitz, N. (1951). Sampling with Probabilities Proportional to Size: Adjustment for Changes in Probabilities. Journal of the American Statistical Association, 46, 105-109.

2 Chowdhury, S., Chu, A., & Kaufman, S. (2000). Minimizing overlap in NCES surveys. Proceedings of the Survey Methods Research Section, American Statistical Association, 174-179. Retrieved from http://www.amstat.org/sections/srms/ proceedings/papers/2000_025.pdf.