Part B TIMSS 2015 Main Study

Trends in International mathematics and science study 2015 (TIMSS 2015) main study data collection

REQUEST FOR OMB review

OMB# 1850-0695 v.5

Supporting Statement Part B

Submitted by:

National Center for Education Statistics

U.S. Department of Education

Institute of Education Sciences

Washington, DC

July 2014

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe

TIMSS

The respondent universe for the TIMSS main study is all students enrolled in grades 4 and 8 during the 2014-15 school year. National probability samples of schools containing a grade 4 or grade 8 were drawn in April 2014 and state probability samples for Florida were drawn in July 2014. All selected students will be asked to participate in a combined mathematics and science assessment.

At grades 4 and 8, a national sample of 300 schools each was selected with the goal of obtaining participation from a minimum of 255 at each grade level. The universe for the selection of schools was all types of schools containing a grade 4 or grade 8 in the 50 states and the District of Columbia, Department of Defense (DoD) domestic schools, and Bureau of Indian Education (BIE) schools. The overlap with the 2015 National Assessment of Educational Progress (NAEP) school sample will be minimized for TIMSS 2015. Up to two intact classes will be assessed in each school when possible, with an expected yield of at least 4,000 students for each grade.

A state sample of schools was drawn for Florida to produce a total state sample of 50 additional schools at each grade level, with the expectation that 99 of the 100 total Florida schools will participate. The sample design for the TIMSS Florida state samples at grades 4 and 8 will closely follow the TIMSS national design where possible. The school frame will be identical to the national frame of public schools in Florida. The school samples will be a two-stage design—a stratified systematic sample of schools with sampling probabilities proportional to size (PPS) and then classes within sampled schools. Schools will be stratified by poverty level, locale, race/ethnicity status, and estimated grade enrollment. A sample of 55 schools will be drawn at each grade to attain the target sample size of 50 eligible schools. Ideally, the objective for the Florida state samples is that they will not include the schools that were previously selected as part of the TIMSS national samples. By following the Keyfitz procedure outlined in table 2 of Chowdhury, Chu, and Kaufman (2000), the procedure allows us to minimize the overlap with the TIMSS national and NAEP public school samples. Designating substitute schools and classroom sampling will follow the national procedures.

TIMSS Advanced.

The respondent universe for the TIMSS Advanced is all students in the last year of secondary school who have taken or are taking advanced mathematics or physics courses as defined by the TIMSS 2015 Advanced Framework. For advanced mathematics, the framework includes algebra, geometry, and calculus content. In the United States, algebra and geometry are pre-requisites for calculus, so the focus will be on calculus. The TIMSS Advanced Framework includes limits, derivatives, and integrals as topics; all of which are typically included in calculus courses in the United States. Using the NCES Secondary School Course Classification System: School Codes for Exchange of Data (SCED) (http://nces.ed.gov/pubs2007/2007341.pdf ), the following courses, at a minimum, would be eligible for the advanced mathematics assessment:

02121 Calculus

02122 Multivariate Calculus

02123 Differential Calculus

02124 AP Calculus AB

02125 AP Calculus BC

02126 Particular Topics in Calculus (when it follows a survey calculus course)

02132 IB Mathematics

The physics framework includes mechanics and thermodynamics, electricity and magnetism, wave phenomena, and atomic/nuclear physics. In the United States, this content is most likely to be covered in second-year/advanced physics courses, and not in first-year survey physics courses. The following courses in the SCED, at a minimum, would be eligible for the TIMSS Advanced physics assessment:

03152 Physics—Advanced Studies

03155 AP Physics B

03156 AP Physics C (either E&M or MECH)

03157 IB Physics

Note that course eligibility will continue to be reviewed as the sampling frame is developed for U.S. participation. In particular, some students may have completed eligible coursework at a college level as a summer course, as a dual credit course, or through a distant learning program.

The advanced mathematics and physics student populations are separate, but overlapping populations. All selected students will be asked to participate in either an advanced mathematics assessment or a physics assessment, but not both.

The universe for the selection of schools is all types of eligible schools in the 50 states and the District of Columbia, DoD domestic schools, and BIE schools, where “eligibility of schools” is determined by having courses in their course catalogue (or available to enrolled students through colleges or virtual schools) that meet the content specifications for advanced mathematics and physics in the TIMSS 2015 Advanced Frameworks. Note that eligibility can only be determined after the original sample of schools is selected. To maximize the likelihood that selected schools are in fact “eligible”, we will use a file provided by the College Board that identifies schools where AP calculus and/or AP physics exams were taken in 2014. We will sample 350 schools with the goal of obtaining participation from a minimum of 300 eligible schools. The overlap with NAEP 2015 school sample will be minimized for TIMSS Advanced.

Using student transcripts and current course enrollment, two lists of eligible students (one for advanced mathematics and one for physics) will be created from which to sample students. Eligible students are those who have taken or are currently taking the relevant courses. Students who are on both lists (eligible for both advanced mathematics and physics) will be randomly assigned to one sample. The TIMSS Advanced main study sample is to consist of high schools with a target of 4,000 students in advanced mathematics and physics for a total of 8,000 students, and a goal of 3,600 assessed students per subject.

B.2 TIMSS and TIMSS Advanced Statistical Methodology

Main Study Sampling Plan and Sample

The school sample design for the main study is a probability sample of schools (one sample for each of grades 4, 8, and 12) that fully represents the entire United States. At the same time, to ensure maximum participation it must be designed so as to minimize overlap with other NCES studies involving student assessment that will be conducted around the same time.

The main study of both TIMSS and TIMSS Advanced will take place in the spring of 2015, about two months after a very large NAEP Main 2015 assessment. NAEP will assess several thousand schools nationally, at grades 4, 8, and 12. The NAEP sample will be relatively heavy in smaller states, and in a number of these states all eligible schools will be included in NAEP, especially at grade 8. Thus to be fully representative, the TIMSS sample may include some schools that will have participated in NAEP at the same grade. However, this number can be kept to minimum using the overlap control procedures outlined below.

While PISA 2015 will take place in the fall of the subsequent school year to TIMSS, it will overlap with school recruiting. PISA will include a small number of grade 8 schools in the sample, but very few grade 8 students will be assessed in PISA, since very few grade 8 students meet the PISA 15 years of age requirements. Most of the PISA selected schools will include grade 9 and therefore also grade 12. Thus overlap control between the TIMSS and PISA samples will be desirable, and should not be difficult to accomplish successfully.

Overlap control procedures in studies such as this, where stratified probability proportional to size samples of schools are selected, can be implemented via a procedure that applies Bayes Theorem to modify the conditional probability of selection of a given school for one study, depending upon its selection probability for a second study, and whether or not it was selected for that study. This approach was first documented in a survey sampling application by Keyfitz (1951)¹. The principals involved can be extended to more than two studies simultaneously, and a procedure for doing this is described by Chowdhury et al. (2000)².

The sample size for the TIMSS main study will be 300 schools at each of grades 4 and 8. The sample size for the TIMSS Advanced main study will be 350 high schools (that contain grade 12), with the aim to obtain about 300 eligible schools. For each original sample school we will identify two replacement schools, matched to the original sample in terms of state and public/private status, and matched, as closely as possible, with regard to the other characteristics used to stratify the original sample. The replacement schools will be recruited in cases where the original sample school is unable to participate. The sampling frames of grade 4, 8, and 12 schools will be obtained from the most current versions of NCES’s Common Core of Data (CCD) and Private School Survey (PSS) files, restricted to schools having grade 4, 8, or 12 respectively, and eliminating schools in Puerto Rico, U.S. territories, and Department of Defense overseas schools.

The sample will be stratified according to school characteristics such as public/private, Census region, poverty status (as measured by the percentage of students in the school receiving free or reduced-price lunch in the National School Lunch Program (NSLP)). This will ensure an appropriate representation of each type of school in the selected sample of schools.

Determining school eligibility, student eligibility, and student sampling for TIMSS Advanced will be accomplished as described above. For TIMSS at grade 4 and 8, it will be accomplished as described below.

Schools will be selected with probability proportional to the number of estimated classes at the appropriate grade (4 or 8), with schools expected to have either one or two classes being given the same selection probability. The use of a probability proportional to sample design ensures that all students have an approximately equal chance of selection, since two classes will be selected from each school regardless of the size of the school. Note that we will modify this equal probability design in the following way. So as to increase the available sample size of students in high poverty schools, we will double the probability of selection of each school with at least 50 percent of students eligible for free or reduced-price lunch under NSLP, relative to other schools of the same size.

Student sampling will be accomplished by selecting two classes per school. Each grade 4 school will be asked to prepare a list of classes that is comprehensive, and includes each grade 4 student in the school in one of the listed classes. As described above, schools will submit these classes and student lists via secure E-filing. Grade 8 schools will be asked to prepare such a list also, but in this case the students should be organized into mathematics classes. At either grade, any class with fewer than ten students will be combined with another class to form a ‘pseudoclass’ with at least ten students in it. We will then select two classes (or pseudoclasses) from each school, with equal probability, and all students in those classes/pseudoclasses will be included in the sample. If a school has only one or two classes, then all students in the grade will be included in the sample. At grade 8, mathematics classes are used for three reasons. First, this minimizes the burden on mathematics teachers, as only two mathematics teachers need to fill out a teacher questionnaire (but typically more than two science teachers are required to do so, since the students in the two selected math classes often attend more than two different science classes). Second, it makes for sound data for conducting analyses of the extent to which classroom factors moderate the relationship of student factors to achievement (e.g., “Does having a well-qualified math teacher reduce the correlation between math achievement and parental education?”). Third, at grade 8, most students take one and only one mathematics class, and thus mathematics classes make for a foolproof partitioning of the eligible students.

School eligibility for the TIMSS Advanced assessment at grade 12 will be determined by:

1. Obtaining course catalogues from school and district websites;

2. Analyzing catalogues to determine courses that appear to be eligible for either advanced mathematics or physics;

3. Contacting schools in the fall of 2014 to finalize eligibility for the 2014-2015 school year.

As we did in the TIMSS Advanced field test, lists of students who have taken or are taking the eligible courses at the school (or through the school virtually or in colleges) will be obtained for student sampling. Details of student sampling are described below:

1. Select all students eligible for both physics and math, up to a total of 40. Otherwise take a sample of size 40.

2. Assign 1/5 of the doubly-eligible students to the math assessment and the rest to physics.

3. Select all students eligible for physics only, up to a total of 20. Otherwise take a sample of size 20. This may be modified to take all up to a total of 40.

4. Let the number of math-only students in the school be M. Take a sample of size

MIN(MIN(M,MAX(10, M/4)), 40).

That is, we will sample one-quarter of the math-only students, subject to a maximum of 40 and a minimum of 10 (or all students if there are fewer than 10).

Based on data from 2009 HSTS, on average in a school with 500 twelfth-grade students we expect:

1. 17 students eligible for both math and physics. We will sample all of them, and assign 3 or 4 to math and 13 or 14 to physics.

2. 6 students eligible for physics only. We will assess all in physics.

3. 67 students eligible for math only. We will sample 16 or 17 of them.

Therefore, we will sample 39 or 40 students from such a school, with about equal numbers assessed in math and physics.

The maximum sample size in any school will be 100 students (or 120 if we permit up to 40 physics-only students). This would be in a very unusual school, that had at least 40 double specialists, at least 20 physics-only specialists (or 40), and at least 160 math-only specialists; so at least 220 specialists (or 240), when specialists are 18 percent of the total U.S. student population.

Nonresponse Bias Analysis, Weighting, and Sampling Errors

It is inevitable that nonresponse will occur at both levels: school and student. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the calculation of weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification (see Appendix D for more detail).

B.3 Maximizing Response Rates

With the recent exception of TIMSS 2011, the most significant challenge in recruitment for TIMSS has been engaging the schools and gaining their cooperation. The circumstances that aided our success in 2011—the NAEP-TIMSS Linking Study and the involvement of NAEP State Coordinators—may not recur in 2015. However, there are important lessons to be learned from that experience that can be used regardless of the overall TIMSS 2015 program. Given that classrooms are selected, student participation is not as great of a challenge. Historically student participation rates have never fallen below 90 percent (see table 1). That said, it is important to U.S. TIMSS that students are engaged and try hard on the assessment, which could be an issue at grade 8 and 12.

Table 1. Historical TIMSS school and student participation rates

Year	Grade	School Participation Rate		Overall Student Participation Rate
		Before Replacement	After Replacement
2011	4	79	84	95
	8	87	87	94
2007	4	70	89	95
	8	68	83	93
2003	4	70	82	95
	8	71	78	94
1999	8	83	90	94
1995	4	86	NA	94

Our approach to school recruitment is to:

• Obtain endorsements about the value of TIMSS from relevant organizations;

• Inform Chief State Officers and test directors about the sample of schools in their state. Enclose a sample letter of endorsement they can send to schools;

• Send letters and informational materials to schools and districts. These letters will be customized by type of school;

• Train experienced school recruiters about TIMSS;

• Involve NAEP State Coordinators in gaining and maintaining cooperation of schools selected for TIMSS and TIMSS Advanced. These Coordinators have been highly successful in gaining cooperation for NAEP.

• Implement strategies from NAEP’s Private School Recruiting Toolkit. This toolkit, developed for NAEP, includes well-honed techniques used to recruit a very challenging type of schools;

• Follow-up mailings with telephone calls to explain the study and schools involvement, including placing the TIMSS assessment date on school calendars;

• Maintain continued contact until schools have built a relationship with the recruiter and fully understand TIMSS; and

• Make in-person visits to some schools, as necessary.

Gaining cooperation of high school seniors is a challenge, so additional strategies will be used for TIMSS Advanced. We learned from NAEP assessments at grade 12 that students are responsive to the adults in the school, particularly their teachers and principal. Since most of the sampled students will be taking Advanced Placement (AP) courses, we will obtain letters of endorsement from the College Board (which overseas AP courses) for principals and teachers, asking them to encourage students to participate in the assessment. In order to illustrate the value of TIMSS and TIMSS Advanced, we will offer optional professional development web seminars throughout the school year on TIMSS and educational policy, as well as using released TIMSS questions in classroom instruction. We will also develop a short motivational video for students that talks about the value of TIMSS Advanced to understanding the U.S. international competitiveness in science, technology, engineering, and mathematics, that appeals to their own competiveness and desire to show how much they know and can do. As an additional incentive, participating students will receive a volunteer service certificate of 4 hours from the U.S. Department of Education. Assessments will be scheduled so as not to interfere with final preparations for AP exams and the exams themselves. NAEP State Coordinators and data collection staff will also share strategies for improving student participation with school staff, such as using the student video, announcements in parent meetings and student meetings, and offering pizza or other treats to students who participate.

B.4 Purpose of Field Test and Data Uses

During the TIMSS Advanced field test, we assessed approximately 1,000 12^th grade calculus and physics students in 32 schools during March and April 2014. Seventy-eight public and private schools were contacted overall. We learned that gaining cooperation was more difficult than we anticipated, primarily due to (a) late notification of selection, (b) conflicts with state assessments and Common Core assessment pilots in the same assessment window, (c) difficulty with scheduling assessments the closer the TIMSS Advanced assessment date got to the AP testing window (the first two weeks in May), and (d) schools protecting instructional time due to many weather-related school closures in the winter of 2014. To improve school participation in the main study, we are contacting schools prior to the next school year (almost one year in advance) and providing them with assessment dates so they can put the TIMSS assessment date on their school calendar. The assessment window in the main study is much longer than in the field test, so we are able to schedule the grade-12 assessments early in April, or after the AP assessment window in the first two weeks of May 2015. We are continuing to be as flexible as possible in rescheduling assessment dates to accommodate school conflicts. We also collected information about known district and state spring breaks, teacher workdays, and local assessment dates in order to facilitate trying to schedule around them.

In addition, during the field test we learned that advanced physics students are even rarer than we originally estimated, and we have increased our grade-12 school sample size to compensate for this fact. We learned that schools can provide lists of eligible students electronically, which will improve the efficiency and quality of student sampling procedures. We also were able to identify eligible courses for each school from their websites, and then confirm these courses in a short phone call with school administrators, likely reducing school burden. Lastly, more 12^th grade students than usual walked out of the field test assessments once they started, so we are enhancing our messaging to students about the value of participation and obtaining additional endorsement letters as described in section B.3. We will continue to refine all these processes in the main study.

B.5 Individuals Consulted on Study Design

Overall direction for TIMSS is provided by Dr. Stephen Provasnik, National Research Coordinator, National Center for Education Statistics, U.S. Department of Education.

The following persons are responsible for the statistical design of TIMSS:

• Pierre Foy. TIMSS International Study Center, Boston College (617-552-6253); and

• Marc Joncas and Jean Dumais, Statistics Canada (613-951-0007).

Westat is the contractor responsible for sampling and data analysis:

• Chris Averett, Project Director, Westat (301-314-2492); and

• David Ferraro, Senior Statistician, Westat (301-251-4261).

Analysis and reporting will be performed by:

• TIMSS International Study Center, Boston College;

• American Institutes for Research, under contract to Westat; and

• National Center for Education Statistics, U.S. Department of Education.

1 Keyfitz, N. (1951). Sampling with Probabilities Proportional to Size: Adjustment for Changes in Probabilities. Journal of the American Statistical Association, 46, 105-109.

2 Chowdhury, S., Chu, A., & Kaufman, S. (2000). Minimizing overlap in NCES surveys. Proceedings of the Survey Methods Research Section, American Statistical Association, 174-179. Retrieved from http://www.amstat.org/sections/srms/ proceedings/papers/2000_025.pdf.