TIMSS_PIRLS_2011_Part_B revised 4_7_2010

Trends in International mathematics and science study (TIMSS) and progress in international reading literacy study (PIRLS) 2010 and 2011 Data Collections

REQUEST FOR OMB REVIEW

Supporting Statement Part B

Prepared by:

National Center for Education Statistics

U.S. Department of Education

Washington, DC

December 20, 2009

Revised April 7, 2010

2. COLLECTION OF INFORMATION EMPLOYING STATISTICAL INFORMATION

B.1 Respondent Universe and Response Rates

The respondent universe for TIMSS and PIRLS full-scale study is all students enrolled in grade 4 (TIMSS and PIRLS) and grade 8 (TIMSS) during the 2010-2011 school year. National probability samples of these students will be drawn. All selected students will be asked to participate in a combined mathematics and science assessment (TIMSS) or a reading literacy assessment (PIRLS).

As noted in Supporting Statement Part A, NCES will pursue a combined sample for TIMSS and PIRLS at grade 4, based on response rates and feedback from the spring 2010 field test. The fielding of a combined sample in the field test indicated no adverse effect on response rates at grade 4. Furthermore, recruitment staff received positive feedback from schools based on this approach. The decision to proceed with a combined TIMSS and PIRLS grade 4 sample has been reviewed and approved by the NCES Chief Statistician.

At grade 4, a sample of 450 schools will be selected with the goal of obtaining participation from a minimum of 383 schools. The universe for the selection of schools is all types of schools containing a grade 4 in the 50 states and the District of Columbia. The TIMSS and PIRLS full-scale study will combine TIMSS and PIRLS in 350 of the schools selected at fourth grade. In these schools, up to four classes will be selected; the first two classes will be randomly assigned as a pair to either TIMSS or PIRLS, with the remaining classes assigned to the other assessment. The remaining 100 schools will be small schools with only one or two fourth-grade classes, which will be randomly assigned to TIMSS or PIRLS at the selection stage. Two fourth-grade classrooms will be selected in these smaller schools unless there is only one fourth grade class in the school. In that case, the single class will be selected. Again, only students in the selected intact classrooms will be assessed.

For grade 8, a sample of 600 schools will be selected for the TIMSS national sample, with the goal of garnering participation from a minimum of 510 schools. The universe for the selection of schools is all types of schools containing a grade 8 in the 50 states and the District of Columbia. Students will be selected by drawing a sample of two intact mathematics classrooms (in which grade 8 students are enrolled) in each sampled school. As described in Supporting Statement Part A, one of these two classes will be randomly assigned to participate in TIMSS; the other class will receive the “braided” booklets that are part of the NAEP-TIMSS Linking Study design. Based on discussions with the international organizing body, the international sampling referee and the contractor, an increase in the school sample size to 600 will counter the effect of administering TIMSS to only one class, rather than two classes. In schools that have only one mathematics classroom at grade 8, that one classroom (instead of a sample of two classrooms) will be selected to participate in TIMSS. Only students in the selected intact classrooms will be assessed for TIMSS at grade 8. Details about the portion of the NAEP-TIMSS Linking Study that will be carried out during the NAEP testing window will be provided through the OMB review and clearance process established for NAEP.

It is expected that, for schools, an overall response rate of 85-90 percent will be achieved after replacement, and that, within participating schools, between 87 and 90 percent of the students will respond. Recent field work in schools has shown that response rates of 90 percent or higher are increasingly difficult to achieve.

For the TIMSS data collection in 2007, school participation at grade 4 was 70 percent before replacement and 89 percent after replacement (see table 2). At grade 8, it was 68 percent before replacement and 83 percent after replacement. The overall student participation rate was 95 percent at grade 4 and 93 percent at grade 8. In TIMSS 2003, school participation in the United States at grade 4 was 70 percent before replacement and 82 percent after replacement. At grade 8 in 2003, school

Table 2. Recent TIMSS and PIRLS school and student participation rates

participation was 71 percent before replacement and 78 percent after replacement. The overall student participation rate was 95 and 94 percent in grades 4 and 8, respectively. This was a decline from TIMSS 1999, when the school participation rate in the United States for grade 8 was 83 percent before replacement and 90 percent after replacement. The overall student response rate within those schools was 94 percent. In 1995, the U.S. grade 4 sample achieved a school response rate of 86 percent and a student response rate of 94 percent. No replacement schools were used in grade 4 in 1995.

The school response rates for PIRLS in 2006 were 57 percent before replacement and 86 percent after replacement. The overall student participation was 96 percent. In 2001, school participation was 61 percent before replacement and 86 percent after replacement. The overall student participation rate was 96 percent.

B.2 Procedures for Collection of Information

Statistical Methodology

For TIMSS and PIRLS 2011 at grades 4 and TIMSS 2011 at grade 8, the sample design will consist of a stratified national sample of schools having the relevant grade. In addition, for the eight states participating in TIMSS grade 8 state benchmarking, as described in Part A, the sample design will consist of a stratified state sample of schools having an eighth grade. The school sampling frames will be obtained from the Common Core of Data (CCD) and Private School Universe Survey (PSS) files. The selection of schools will be with probability proportional to size, where the size measure is an estimate of the number of students enrolled in the relevant grade, obtained from the CCD and PSS files.

In both the national and state samples, two substitute schools will be identified for each original school, in accordance with the international procedures. These numbers allow for the patterns of school nonresponse that have been observed in the most recent TIMSS and PIRLS studies. Stratification variables to be used in the design for the national sample will be public/private status, region of the country, type of location, minority enrollment, and school size. Stratification variables to be used in the design for the state benchmarking sample will be public/private status, type of location, minority enrollment, and school size. In both cases, the two substitute schools associated with each original selection will be those schools that are adjacent to the original selection on the sample frame when sorted by the stratification variables. Thus the substitute schools will match the original selection as closely as possible with regard to all of the stratification variables.

The international protocol calls for the school sample to be selected by the IEA International Study Center, using a list of schools supplied by the U.S. National Research Coordinator. However, the process is to be a collaborative one, where Westat will select the TIMSS and PIRLS U.S. national and state school samples, and have the sample verified by the International Study Center. This is consistent with past practice.

For the classroom sampling, the key steps are as follows: (1) obtain a comprehensive list of all eligible classes, ensuring that each student enrolled in grade 4, or grade 8, is enrolled in exactly one listed class; (2) sort the classes in the order specified by the international procedures; (3) determine the number of classes listed, and develop a procedure to ensure that these data are available on files for weighting; (4) select a maximum of four classrooms at random, using the international procedures; (5) assign identification numbers to the students and teachers from the selected classroom and record on the appropriate international survey forms. Names are associated with unique student identification numbers in lists given to the School Coordinator to ensure that missing information can be obtained through follow-up sessions, if necessary, and so that teachers know, for attendance purposes, which students participated in the TIMSS or PIRLS assessments. It is important to note that the names of students do not leave the school and under no circumstances are the names of students or teachers included in the international or national database, nor is this information forwarded to any organization. After all data collection is complete, the School Coordinator is instructed to destroy the list of names associated with the unique IDs to ensure complete confidentiality and privacy of respondents, per NCES practice. Neither the contractor nor NCES retains these lists.

Estimation

All estimation and weighting procedures for the TIMSS and PIRLS data are prescribed by and are the responsibility of the international sponsoring agency and the TIMSS and PIRLS International Study Center. The United States, and all other participating countries, will comply with these procedures and policies by delivering the raw data, and documentation of sampling variables, to the study center for weighting and processing. (Note, however, that NCES will conduct a disclosure analysis prior to submitting the data to the international contractor so as to comply with current federal law.) The weighted data will then be returned to each country for national analyses. All data delivered to the TIMSS and PIRLS International Study Center will be devoid of any data that could lead to the identification of individuals.

There are no anticipated problems that would require specialized sampling procedures, nor will there be any use of periodic data collection cycles to reduce burden.

B.3 Maximizing Response Rates

To ensure a high response rate among teachers and principals, we will build interest in the project through individual contact and recruitment. Our estimated response rate will be at least 85 percent. Appropriate channels will be followed for securing cooperation of schools (e.g., contacting school system and obtaining permission from relevant school organizations). We have developed an invitational package to use with sample schools and also to inform their district and state level educators. As included in appendix B of the accompanying materials, the information will include a letter requesting participation, a brochure describing the projects and the types of questions that will be asked, a timeline showing the central activities for principals, school coordinators, teachers, and students, and a key information sheet providing more detail about events of the study.

B.4 Tests of Procedures

The United States participated in a field test for TIMSS and PIRLS 2011. The central goals of the field test were to evaluate the approach of assessing the same schools for both TIMSS and PIRLS in grade 4 and collecting item data to ascertain the reliability and utility of all variables to be collected and procedures to be used in the studies. As noted in Supporting Statement Part A and above, NCES determined that the fielding of a combined grade 4 sample in the field test had no adverse effect on response rates. Furthermore, recruitment staff received positive feedback from schools based on this approach. The decision to proceed with a combined TIMSS and PIRLS grade 4 sample has been reviewed and approved by the NCES Chief Statistician. Once the field test data have been analyzed and if deemed necessary, NCES may request the contractor to conduct cognitive laboratories to ascertain how students (or teachers and principals) respond to a limited number of non-cognitive items for which additional information is sought. If this is deemed necessary, and more than 9 respondents per item are needed for sufficient testing, NCES will follow standard procedures for obtaining OMB approval.

B.5 Individuals Consulted on Statistical Design

The following are responsible for the statistical design of TIMSS:

• Pierre Foy, TIMSS and PIRLS International Study Center, Boston College (617-552-6253)

• Marc Joncas, Statistics Canada (613-951-0007)

Westat will be the contractor responsible for sampling and data analysis:

• David Kastberg, Project Director, Westat (301-294-3811)

• David Ferraro, Senior Statistician, Westat (301-251-4261)

Analysis and reporting will be performed by:

• TIMSS International Study Center, Boston College

• David Kastberg, Westat, under contract with NCES

• National Center for Education Statistics, U.S. Department of Education