TIMSS 2015 Recruitment & Field Test Part B

Trends in International mathematics and science study (TIMSS 2015) FIELD TEST AND RECRUITMENT FOR FIELD TEST AND MAIN STUDY

REQUEST FOR OMB Clearance

OMB# 1850-0695 v.3

Supporting Statement Part B

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

June 2013

Revised October 2013

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe

TIMSS. NCES will not administer the TIMSS field test to students enrolled in grades 4 and 8.

TIMSS Advanced. The respondent universe for the TIMSS Advanced field test is all students in the last year of secondary school who have taken or are taking advanced mathematics or physics courses as defined by the TIMSS 2015 Advanced Framework. For advanced mathematics, the framework includes algebra, geometry, and calculus content. In the United States, algebra and geometry are pre-requisites for calculus, so the focus will be on calculus. The TIMSS Advanced Framework includes limits, derivatives, and integrals as topics; all of which are typically included in calculus courses in the United States. Using the NCES Secondary School Course Classification System: School Codes for Exchange of Data (SCED) (http://nces.ed.gov/pubs2007/2007341.pdf ), the following courses, at a minimum, would be eligible for the advanced mathematics assessment:

02121 Calculus

02122 Multivariate Calculus

02123 Differential Calculus

02124 AP Calculus AB

02125 AP Calculus BC

02126 Particular Topics in Calculus (when it follows a survey calculus course)

02132 IB Mathematics

The physics framework includes mechanics and thermodynamics, electricity and magnetism, wave phenomena, and atomic/nuclear physics. In the United States, this content is most likely to be covered in second-year/advanced physics courses, and not in first-year survey physics courses. The following courses in the SCED, at a minimum, would be eligible for the TIMSS Advanced physics assessment:

03152 Physics—Advanced Studies

03155 AP Physics B

03156 AP Physics C (either E&M or MECH)

03157 IB Physics

Note that course eligibility will continue to be reviewed as the sampling frame is developed for U.S. participation. In particular, some students may have completed eligible coursework at a college level as a summer course, as a dual credit course, or through a distant learning program.

The advanced mathematics and physics student populations are separate, but overlapping populations. All selected students will be asked to participate in either an advanced mathematics assessment or a physics assessment, but not both.

The universe for the selection of schools is all types of eligible schools in seven populous states, where “eligibility of schools” is determined by having courses in their course catalogue (or available to enrolled students through colleges or virtual schools) that meet the content specifications for advanced mathematics and physics in the TIMSS 2015 Advanced Frameworks. Note that eligibility can only be determined after the original sample of schools is selected. To maximize the likelihood that selected schools are in fact “eligible”, we will consider using the NAEP High School Transcript Study (HSTS) schools from 2009 as the sampling frame for selecting schools. We will sample 61 schools with the goal of obtaining participation from a minimum of 35 eligible schools.

Using student transcripts and current course enrollment, two lists of eligible students (one for advanced mathematics and one for physics) will be created from which to sample students. Eligible students are those who have taken or are currently taking the relevant courses. Students who are on both lists (eligible for both advanced mathematics and physics) will be randomly assigned to one sample.

B.2 Statistical Methodology

Field Test Sampling Plan and Sample

TIMSS. NCES will not administer the TIMSS field test to students enrolled in grades 4 and 8.

TIMSS Advanced. The purpose of the TIMSS Advanced field test is to test new assessment items and background questions. Because of the nature of the TIMSS Advanced target population, the field test will also help ensure that school and student sampling procedures (including determining school and student eligibility) proposed for the main study are successful. The TIMSS Advanced field test sample is to consist of 35 high schools, and assess 1,400 students (700 for advanced mathematics and 700 for physics). The TIMSS Advanced field test is designed only to test items, questions, and procedures; so a probability sample of schools is not required.

The TIMSS Advanced field test will take place in the spring of 2014 after the NAEP 2014 assessment. NAEP 2014 will involve less than 100 schools at grade 12 (and not almost two thousand as in 2013 and 2015). The following steps describe the sampling approach for the TIMSS Advanced field test:

1. From a selection of 7 states, we will select 10 high schools in each of the four largest states, and 7 high schools in each of the three smallest states. The states will be spread across different regions. This will provide an appropriate geographic diversity for the selected schools.

2. After identifying the potentially eligible schools in each state, using the most current NCES CCD (public schools) and PSS (private schools) files, we will subset these by eliminating schools that have been included in the NAEP 2014 sample.

3. We will then select a stratified probability sample of 61 schools, ensuring representation by school type (approximately 10 percent of the schools selected will be private schools), urban/rural, and race/ethnic composition.

4. We will sample 61 schools to ensure that we have 35 eligible schools, where school eligibility is determined by offering at least one of the eligible courses for advanced mathematics and physics. We will obtain course catalogues from all 61 schools, and code the course catalogues according to the course criteria to determine eligible schools. Mathematics and science department chairs will also be asked to identify eligible courses from their catalogues. If more than 35 are eligible, we will select 35 for actual data collection.

5. For each sampled school we will identify two replacement schools, matched to the original sample in terms of state and public/private status, and matched, as closely as possible, with regard to the other characteristics used to stratify the original sample. The replacement schools will be recruited in cases where the original sample school is unable to participate.

Once the field test sample has been selected, a summary of the distribution of the characteristics of the selected schools will be prepared, showing the comparison with the national population of schools.

The student sampling procedures for the field test will correspond as closely as feasible to what is planned for the main study, so as to try out the operational procedures for student sample selection. The student samples will be obtained by conducting the following steps:

1. Collect electronic (or paper) student transcripts as well as current class enrollment of all twelfth grade students in the selected schools.

2. Determine eligible students for both advanced mathematics and physics; these are students in the target population in the school, who have taken or are currently taking eligible courses.

3. Draw random sample of 20 students for each assessment (advanced mathematics and physics) from those students eligible for each. If a student is in both populations, we will randomly assign the student to one subject.

We plan to gather class, transcript, and student lists from participating schools electronically, using an adaptation of our secure E-filing process. High schools will access the E filing system through the MyTIMSS web site.

Main Study Sampling Plan and Sample

TIMSS and TIMSS Advanced. The school sample design for the main study must be more rigorous than that for a field test. It must be a probability sample of schools (one sample for each of grades 4, 8, and 12) that fully represents the entire United States. At the same time, to ensure maximum participation it must be designed so as to minimize overlap with other NCES studies involving student assessment that will be conducted around the same time.

The main study of both TIMSS and TIMSS Advanced will take place in the spring of 2015, about two months after a very large NAEP assessment. NAEP will assess several thousand schools nationally, at grades 4, 8, and 12. The NAEP sample will be relatively heavy in smaller states, and in a number of these states all eligible schools will be included in NAEP, especially at grade 8 and 12 (for the 13 states included in grade 12 state NAEP). Thus to be fully representative, the TIMSS sample may include some schools that will have participated in NAEP at the same grade. However, this number can be kept to minimum using the overlap control procedures outlined below.

While PISA 2015 will take place in the fall of the subsequent school year to TIMSS, it will overlap with school recruiting. PISA will include a small number of grade 8 schools in the sample, but very few grade 8 students will be assessed in PISA, since very few grade 8 students meet the PISA 15 years of age requirements. Most of the PISA selected schools will include grade 9 and therefore also grade 12. Thus overlap control between the TIMSS and PISA samples will be desirable, and should not be difficult to accomplish successfully.

Overlap control procedures in studies such as this, where stratified probability proportional to size samples of schools are selected, can be implemented via a procedure that applies Bayes Theorem to modify the conditional probability of selection of a given school for one study, depending upon its selection probability for a second study, and whether or not it was selected for that study. This approach was first documented in a survey sampling application by Keyfitz (1951)¹. The principles involved can be extended to more than two studies simultaneously, and a procedure for doing this is described by Chowdhury et al. (2000)².

The sample size for the TIMSS main study will be 300 schools at each of grades 4 and 8. The sample size for the TIMSS Advanced main study will be 250 high schools (that contain grade 12). For each original sample school, two replacement schools will also be identified. The sampling frames of grade 4, 8, and 12 schools will be obtained from the most current versions of NCES’s Common Core of Data (CCD) and Private School Survey (PSS) files, restricted to schools having grade 4, 8, or 12 respectively, and eliminating schools in Puerto Rico, U.S. territories, and Department of Defense overseas schools.

The sample will be stratified according to school characteristics such as public/private, Census region, poverty status (as measured by the percentage of students in the school receiving free or reduced-price lunch in the National School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the selected sample of schools.

Determining school eligibility, student eligibility, and student sampling for TIMSS Advanced will be accomplished as described above for the field test. For TIMSS at grade 4 and 8, it will be accomplished as described below.

Schools will be selected with probability proportional to the number of estimated classes at the appropriate grade (4 or 8), with schools expected to have either one or two classes being given the same selection probability. The use of a probability proportional to sample design ensures that all students have an approximately equal chance of selection, since two classes will be selected from each school regardless of the size of the school. Note that we will modify this equal probability design in the following way. So as to increase the available sample size of students in high poverty schools, we will double the probability of selection of each school with at least 50 percent of students eligible for free or reduced-price lunch under NSLP, relative to other schools of the same size.

Student sampling will be accomplished by selecting two classes per school. Each grade 4 school will be asked to prepare a list of classes that is comprehensive, and includes each grade 4 student in the school in one of the listed classes. As described above, schools will submit these classes and student lists via secure E-filing. Grade 8 schools will be asked to prepare such a list also, but in this case the students should be organized into mathematics classes. At either grade, any class with fewer than ten students will be combined with another class to form a ‘pseudoclass’ with at least ten students in it. We will then select two classes (or pseudoclasses) from each school, with equal probability, and all students in those classes/pseudoclasses will be included in the sample. If a school has only one or two classes, then all students in the grade will be included in the sample. At grade 8, mathematics classes are used for three reasons. First, this minimizes the burden on mathematics teachers, as only two mathematics teachers need to fill out a teacher questionnaire (but typically more than two science teachers are required to do so, since the students in the two selected math classes often attend more than two different science classes). Second, it makes for sound data for conducting analyses of the extent to which classroom factors moderate the relationship of student factors to achievement (e.g., “Does having a well-qualified math teacher reduce the correlation between math achievement and parental education?”). Third, at grade 8, most students take one and only one mathematics class, and thus mathematics classes make for a foolproof partitioning of the eligible students.

Determining school eligibility for the TIMSS Advanced assessment at grade 12, as well as student eligibility and student sampling will be accomplished as described above for the field test.

Nonresponse Bias Analysis, Weighting, and Sampling Errors

It is inevitable that nonresponse will occur at both levels: school and student. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the calculation of weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification (see Appendix D for more detail).

B.3 Maximizing Response Rates

With the recent exception of TIMSS 2011, the most significant challenge in recruitment for TIMSS has been engaging the schools and gaining their cooperation. The circumstances that aided our success in 2011—the NAEP-TIMSS Linking Study and the involvement of NAEP State Coordinators—may not recur in 2015. However, there are important lessons to be learned from that experience that can be used regardless of the overall TIMSS 2015 program. Given that classrooms are selected, student participation is not as great of a challenge. Historically student participation rates have never fallen below 90 percent (see table 1). That said, it is important to U.S. TIMSS that students are engaged and try hard on the assessment, which could be an issue at grade 8 and 12.

Table 1. Historical TIMSS school and student participation rates

Year	Grade	School Participation Rate		Overall Student Participation Rate
		Before Replacement	After Replacement
2011	4	79	84	95
	8	87	87	94
2007	4	70	89	95
	8	68	83	93
2003	4	70	82	95
	8	71	78	94
1999	8	83	90	94
1995	4	86	NA	94

Our approach to school recruitment is to:

• Obtain endorsements about the value of TIMSS from relevant organizations;

• Inform Chief State Officers and test directors about the sample of schools in their state. Enclose a sample letter of endorsement they can send to schools;

• Send letters and informational materials to schools and districts. These letters will be customized by type of school;

• Train experienced school recruiters about TIMSS;

• Implement strategies from NAEP’s Private School Recruiting Toolkit. This toolkit, developed for NAEP, includes well-honed techniques used to recruit a very challenging type of schools;

• Followup mailings with telephone calls to explain the study and schools involvement, including placing the TIMSS assessment date on school calendars;

• Maintain continued contact until schools have built a relationship with the recruiter and fully understand TIMSS; and

• Make in-person visits to some schools, as necessary.

B.4 Purpose of Field Test and Data Uses

TIMSS Advanced. The main focus of a field test for TIMSS Advanced is to collect enough assessment data to perform reliable tests of the items. In addition, the field test will allow us to evaluate procedures for conducting the main study, including recruitment methods for obtaining school and student participation, and identifying and sampling eligible schools and students.

B.5 Individuals Consulted on Study Design

Overall direction for TIMSS is provided by Dr. Stephen Provasnik, National Research Coordinator, National Center for Education Statistics, U.S. Department of Education.

The following persons are responsible for the statistical design of TIMSS:

• Pierre Foy. TIMSS International Study Center, Boston College (617-552-6253); and

• Marc Joncas and Jean Dumais, Statistics Canada (613-951-0007).

Westat is the contractor responsible for sampling and data analysis:

• Chris Averett, Project Director, Westat (301-314-2492); and

• David Ferraro, Senior Statistician, Westat (301-251-4261).

Analysis and reporting will be performed by:

• TIMSS International Study Center, Boston College;

• American Institutes for Research, under contract to Westat; and

• National Center for Education Statistics, U.S. Department of Education.

1 Keyfitz, N. (1951). Sampling with Probabilities Proportional to Size: Adjustment for Changes in Probabilities. Journal of the American Statistical Association, 46, 105-109.

2 Chowdhury, S., Chu, A., & Kaufman, S. (2000). Minimizing overlap in NCES surveys. Proceedings of the Survey Methods Research Section, American Statistical Association, 174-179. Retrieved from http://www.amstat.org/sections/srms/ proceedings/papers/2000_025.pdf.