Part B TIMSS 2019 Field Test Recruitment & Pilot Test

Trends in International mathematics and science study (TIMSS 2019)

Field Test Recruitment and PILOT test

REQUEST FOR OMB Clearance

OMB# 1850-0695 v.8

Supporting Statement Part B

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

August 2016

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe

The respondent universe for the eTIMSS pilot is all students enrolled in grade 4 and grade 8 during the 2016-17 school year. The universe for the selection of schools is all types of schools in six populous states. A sample of 36 schools will be selected for the pilot, with the goal of obtaining participation from a minimum of 25 schools at each grade. Within sampled schools, students will be selected for participation by drawing a random sample of two classes. At grade 8, students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. All selected students will be asked to participate in a combined TIMSS mathematics and science assessment. Only students in intact classrooms will be assessed at each grade.

The respondent universe for the eTIMSS field test is all students enrolled in grade 4 and grade 8 during the 2017-18 school year. The universe for the selection of schools is all types of schools in seven populous states. A sample of 40 schools will be selected for the field test, with the goal of obtaining participation from a minimum of 35 schools at each grade. Within sampled schools, students will be selected for participation by drawing a random sample of two classes. At grade 8, students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. All selected students will be asked to participate in a combined TIMSS mathematics and science assessment. Only students in intact classrooms will be assessed at each grade.

B.2 Statistical Methodology

Pilot Sampling Plan and Sample

The purpose of the eTIMSS pilot is to test out items transferred from paper to digital format, see how new items developed especially for the digital format perform, and to study the mode effect between paper-and-pencil and digital administration methods.. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize impact on these entities while also ensuring that the pilot data are collected effectively.

The eTIMSS pilot will require assessing approximately 800 students who represent a variety of abilities and background at each grade level. These students could be sampled from 20 to 25 schools in each grade with diverse student composition. The student sample will be obtained by selecting two classes from each school. Each student will take both an eTIMSS assessment and a paper TIMSS assessment. As the pilot is designed only to test items, mode effect, and new procedures for an assessment administered on tablets, a national probability sample of schools is not required. However, the sample must include a broad range of public schools covering such features as public large and small schools, urban and rural schools, and schools from a variety of different states.

The school frame for the 2017 eTIMSS pilot sample will be developed from the 2017 NAEP school frame. The data for public and private schools will come from the CCD and PSS, respectively, with 2014-2015 school data. The school sample for the pilot will be a purposive sample and does not require complete coverage of schools for the target populations. The eligible schools in TIMSS include all schools with a fourth or eighth grade operating in six states. When selecting the states, we will take into consideration the school samples already selected for NAEP 2017, the state assessment schedules that have conflicts during this time period, and the states selected for the PISA field test in 2017. This model of state selection was successfully used in the field test for TIMSS 2015 and provided an adequate regional coverage of the United States while avoiding sampling very large states that typically are heavily sampled due to their population characteristics. The states will be spread across different regions. This selection will provide an appropriate geographic diversity for the selected schools. For efficiency, we will subset the school frame by eliminating schools that are unlikely to have two fourth- or eighth-grade classes.

The eTIMSS pilot will take place in the spring of 2017 after the NAEP 2017 assessment. NAEP 2017 is very large, involving approximately 19,000 schools at grades 4 and 8. Because the eTIMSS pilot assessment follows the NAEP assessment, we will select the eTIMSS pilot sample so as not to include any schools that are sampled for NAEP 2017. The school frame will also exclude any schools sampled for the International Computer and Information Literacy (ICILS) field test at grade 8, or other competing studies that may be occurring around the same time as TIMSS. If NCES can provide the data in time, schools that are sampled for other international studies which overlap with TIMSS will be excluded from the school frame.

The 2017 eTIMSS pilot sample design is a stratified sample in which schools are selected with probability proportional to a measure of size (MOS), that is, the square root of the grade enrollment. We will use probability sampling not because it is required, but because it is an efficient way to select a sample of schools representing a variety of school characteristics.

Stratification for the pilot will use a similar overall scheme as the main study. The stratification will be both explicit and implicit. Explicit strata will be defined by state and school type (public/private). Within each explicit stratum, schools will be sorted by locale (urban/suburban/town/rural), race/ethnicity status (race/ethnicity enrollment above or below 15 percent Black, Hispanic, Asian, American Indian and Alaska Native, Hawaiian/Pacific Islander, or two or more races), and the grade enrollment. Westat will select the school sample using an in-house sampling macro and a systematic sampling method to select from the sorted list of schools. The probability of selection for each school will be proportional to an MOS defined as the square root of the grade enrollment. Using this MOS will lessen the chance of larger schools also being selected for the field test (2018) and the main study (2019) by reducing their chance of selection for the pilot.

The eTIMSS U.S. national sample design requires a minimum of 800 students at each grade level, to be assessed in 20 to 25 participating schools. Because recruitment will be especially challenging due to the burden of assessing each student twice, we will select 36 schools across six states, six schools per state, at each grade level. These 36 schools will be large enough (minimum enrollment of 50 students) to sample two classrooms and yield 40 assessed students from each school.

A minimum of approximately 800 students per grade will be assessed across the participating public schools. We will sample 36 schools per grade for the pilot across the six states with one and two replacement schools assigned per sampled public school, respectively. We will conduct the pilot in the set of six states with the aim of gaining the cooperation of at least five schools in each of the states. Assuming an overall low school response rate, we expect participation from at least 25 schools at each grade level in the eTIMSS pilot.

The student sampling procedures for the eTIMSS pilot will correspond as closely as feasible to what is planned for the field test and the main study, so as to try out the operational procedures for student sample selection. The sample will be selected by choosing two classes per school. Each participating school will be asked to submit an exhaustive list of classes (that is, one that accounts for each student in the grade exactly once). In cases for which one or more classes on the list has fewer than 10 students, smaller classes will be combined to form “pseudoclasses” for the purposes of sampling. Once the list of classes is submitted, we will use a sampling algorithm to select two classes (or pseudoclasses) with equal probability. The student sample will then consist of all students in the selected classes.

We plan to gather class and student lists from participating schools electronically using an adaptation of our secure e-filing process. E-filing was successfully used in TIMSS 2015, and provides advantageous features such as efficiency and data quality checks. Schools will access the e-filing system through the MyTIMSS web site.

Field Test Sampling Plan and Sample

The purpose of the eTIMSS field test is to assess in the new eTIMSS platform legacy or “trend” TIMSS items as well as new items, and to ensure that classroom sampling procedures proposed for the main study are successful. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, to minimize impact on these entities while also ensuring that the field test data are collected effectively.

As required by the TIMSS international study center and the IEA, the field test sample is to consist of 40 schools and 1,400 students assessed at each grade level (4 and 8). The student sample will be obtained by selecting two classes from each school. At grades 4 and 8, mathematics and science teachers of students in selected classes will be asked to complete a teacher questionnaire. As the field test is designed only to test items, questions, and procedures, a probability sample of schools is not required. However, the sample must include a broad range of schools covering such features as public (including charter schools), private, large, small, urban, and rural schools, and schools from a variety of different states.

The school frame for the 2018 eTIMSS field test sample will be developed from the 2018 NAEP school frame. The data for public and private schools will come from the CCD and PSS, respectively, using the 2015-2016 school year data. The school sample for the field test will be a purposive sample and does not require complete coverage of schools for the target populations. The eligible schools in TIMSS include all schools with a fourth or eighth grade that are operating in selected seven states. When selecting the states, we will take into consideration the school samples already selected for NAEP 2018 and for the ICILS 2018 main study, the state assessment schedules that have conflicts during this time period, and the schools selected for the PISA 2018. This model of state selection was successfully used in the field test in previous TIMSS administrations and provided an adequate regional coverage of the United States while avoiding sampling very large states that typically are heavily sampled due to their population characteristics. The states will be spread across different regions. This selection will provide an appropriate geographic diversity for the selected schools. For efficiency, we will subset the school frame by eliminating schools that are unlikely to have two fourth- or eighth-grade classes.

The 2018 eTIMSS field test sample design is a stratified sample in which schools are selected with probability proportional to a measure of size (MOS), that is, the square root of the grade enrollment. We will use probability sampling not because it is required, but because it is an efficient way to select a sample of schools representing a variety of school characteristics.

Stratification for the field test will use a similar overall scheme as the main study. The stratification will be both explicit and implicit. Explicit strata will be defined by state and school type (public/private). Within each explicit stratum, schools will be sorted by locale (urban/suburban/town/rural), race/ethnicity status (race/ethnicity enrollment above or below 15 percent Black, Hispanic, Asian, American Indian and Alaska Native, Hawaiian/Pacific Islander, or two or more races), and the grade enrollment. Westat will select the school sample using an in-house sampling macro and a systematic sampling method to select from the sorted list of schools. The probability of selection for each school will be proportional to a MOS, defined as the square root of the grade enrollment. Using this MOS will lessen the chance of larger schools also being selected for the main study (2019) by reducing their chance of selection for the pilot.

A minimum of approximately 1,400 students per grade will be assessed across the participating schools. We will sample 40 schools per grade for the field test across the seven states with one and two replacement schools assigned per sampled public school, respectively. We will conduct the field test in the set of seven states with the aim of gaining the cooperation of at least five schools in each of the states. Assuming an overall low school response rate, we expect participation from at least 35 schools at each grade level in the eTIMSS field test.

The student sampling procedures for the eTIMSS field test will correspond as closely as feasible to what is planned for the main study, so as to try out the operational procedures for student sample selection. The sample will be selected by choosing two classes per school. Each participating school will be asked to submit an exhaustive list of students and classes (that is, one that accounts for each student in the grade exactly once). In cases for which one or more classes on the list has fewer than 10 students, smaller classes will be combined to form “pseudoclasses” for the purposes of sampling. Once the list of classes is submitted, we will use a sampling algorithm to select two classes (or pseudoclasses) with equal probability. The student sample will then consist of all students in the selected classes.

We plan to gather class and student lists from participating schools electronically using an adaptation of our secure e-filing process. E-filing was successfully used in TIMSS 2015, and provides advantageous features such as efficiency and data quality checks. Schools will access the e-filing system through the MyTIMSS web site.

B.3 Maximizing Response Rates

With the recent exception of TIMSS 2011, the most significant challenge in recruitment for TIMSS has been engaging the schools and gaining their cooperation. The circumstances that aided our success in 2011—the NAEP-TIMSS Linking Study and the involvement of NAEP State Coordinators—did not recur in 2015. However, there are important lessons to be learned from that TIMSS 2011 experience that were used in TIMSS 2015 and will be used in the eTIMSS pilot. Given that classrooms are selected, student participation is not as great of a challenge. Historically, student participation rates have never fallen below 90 percent (see table 1). That said, it is important to U.S. TIMSS that students are engaged and try hard on the assessment. The official response rates are not yet available for TIMSS 2015.

Table 1. Historical TIMSS school and student participation rates

Year	Grade	School Participation Rate		Overall Student Participation Rate
		Before Replacement	After Replacement
2011	4	79	84	95
	8	87	87	94
2007	4	70	89	95
	8	68	83	93
2003	4	70	82	95
	8	71	78	94
1999	8	83	90	94
1995	4	86	NA	94

Our approach to school recruitment is to:

• Obtain endorsements about the value of TIMSS from relevant organizations;

• Inform Chief State Officers and test directors about the sample of schools in their state.

• Use the assistance of NAEP State Coordinators to recruit districts and schools, providing key state agency involvement in recruitment;

• Send letters and informational materials to schools and districts. These letters will be customized by type of school;

• Train experienced NAEP State Coordinators about TIMSS;

• Implement strategies from NAEP’s Private School Recruiting Toolkit. This toolkit, developed for NAEP, includes well-honed techniques used to recruit a very challenging type of schools;

• Followup mailings with telephone calls to explain the study and schools involvement, including placing the TIMSS assessment date on school calendars;

• Maintain continued contact until schools have built a relationship with the recruiter and fully understand TIMSS; and

• Make in-person visits to some schools, as necessary.

B.4 Purpose of Pilot and Field Test and Data Uses

The central goals of the pilot are to test out items that have been transferred from paper to digital format, assess the performance of new items developed to take advantage of technology, and evaluate the mode effect of transitioning from paper to digital format. The purpose of the field test is to evaluate sampling and data collection procedures, new and legacy TIMSS items in the eTIMSS electronic platform, and student, teacher, and school questionnaires.

B.5 Individuals Consulted on Study Design

Overall direction for TIMSS is provided by Dr. Stephen Provasnik, National Research Coordinator, National Center for Education Statistics (NCES), U.S. Department of Education, in consultation with a number of NCES statistical staff.

The following persons are responsible for the statistical design of TIMSS:

• Pierre Foy. TIMSS International Study Center, Boston College (617-552-6253); and

• Sylvie LaRoche, Statistics Canada (613-863-9480).

Westat is the contractor responsible for sampling and data analysis:

• Chris Averett, Project Director, Westat (301-314-2492); and

• David Ferraro, Senior Statistician, Westat (301-251-4261).

Analysis and reporting will be performed by:

• National Center for Education Statistics, U.S. Department of Education;

• TIMSS International Study Center, Boston College; and

• American Institutes for Research, under contract to Westat.