PISA 08 field test 09 full scale Part B

B. collections of information employing statistical methods

B.1 Respondent Universe

PISA 2009 attempts to represent students nearing the "end of their compulsory school experience". For international comparability, this is defined as students who are 15 years old, regardless of their grade in school. A range of exact birthdates is specified by the international coordinating committees based on the months in which the data will be collected. However, students must be between the ages of 15 years and three completed months and 16 years and two completed months at the beginning of the test period. The universe for the selection of schools is all types of schools in all states of the United States and the District of Columbia. Within sampled schools, students will be selected for participation by drawing a random sample among the 15-year-old students.

B.2 Statistical Methodology

The Technical Standards for PISA established by the international governing board include the following:

Standard 1.8. The student sample size must be a minimum of 4,500 assessed students, or the National Defined Target Population.

Standard 1.9. The school sample size must be a minimum of 150 schools or all schools that have students in the National Defined Target Population.

Standard 1.10. The target cluster size is typically 35 PISA eligible students, which upon agreement can be increased or reduced to a number not less than 20.

Standard 1.11. School response rates must be above 85 percent of sampled schools. If a response rate is below 85 percent then an acceptable response rate can still be achieved through agreed upon use of replacement schools. PISA establishes three response rate zones—acceptable, intermediate, and not acceptable. “Acceptable” refers to original school response rates above 85 percent and means that the country’s data will be included in all international comparisons. “Not Acceptable” refers to original response rates below 65 percent and means that the country’s data will be a candidate for not being reported in international comparisons unless considerable evidence is presented that nonresponse bias is minor. “Intermediate” refers to original school response rates of between 65 and 85 percent and means that a decision on whether or not to include the country’s data in comparisons must be made while taking into account a variety of factors, such as student response rates, quality control, etc. In addition, schools with less than 50 percent participation of students are not considered participating schools and neither that school nor those students that did participate are considered in the calculation of response rates.

Standard 1.12. The overall student response rates must be above 80 percent of sampled students.

In addition, NCES has a standard in which student response rate should be 85 percent. Thus the sampling design described below is based on that higher rate.

Overview

The design for this study will be self-weighting, stratified, will consist of two stages, and will use probability proportional to size (PPS). There will be no oversampling of schools or students. Schools will be selected in the first stage with PPS and students will be sampled in the second stage yielding overall equal probabilities of selection.

Target Populations

The PISA target population is 15-year-old students attending educational institutions located within the United States in grades 7 and higher. The plan is to implement the survey in the fall of 2009, with a field trial in the spring of 2008. The specific definition of age eligibility that will be used in the survey is “…between 15 years and 3 (completed) months to 16 years and 2 (completed) months at the beginning of the testing window.”

Sampling Frame of Schools

The population of schools for PISA 2009 is defined as all schools containing any 15-year-olds in grades 7 through 12. The school sampling frame will be developed from the NAEP school frame available in June 2008. This frame will be created from the most up-to-date NCES Common Core of Data (CCD) and Private Schools Survey (PSS). These sources were the same as those used for the school sampling frame used for previous PISA cycles.

The grade structure of the school is a key stratification variable designed to reduce sampling error, but this is especially so in PISA because data analyses have shown that achievement is highly related to grade. Other stratification variables include public/private, region of the country, location (urban/suburban/town/rural, etc.), and enrollment by race/ethnicity.

Field Trial

A formal probability sample of schools is not required for the PISA field trial. It is sufficient that the samples of schools be representative of a broad range of schools from across the United States. To achieve this end we propose a stratified convenience sample of schools having grades 9 and 10 (where most 15-year-olds are found) and having an estimated enrollment of at least 35 students per grade selected in the most populous states: California, Texas, Florida, Michigan, and New Jersey. Selecting schools in these states will meet the criterion of national representation. Within these states, schools will be stratified by State, public/private status, location (urban/center city/rural), and minority enrollment. The stratification will be implicit, where the frame is sorted by the school characteristics. The sample will be a stratified systematic sample, with sampling probabilities proportional to measures of size, where the measure of size is the estimated number of 15-year-olds in ninth or tenth grade. We propose in addition the identification within this framework of two similar replacement schools for each sampled school to ensure an adequate student yield.

The field trial requires a minimum student sample of 1,200 students, which means we should select a sample of between 40-41 schools with 2 replacement schools per original to provide for an adequate participating student sample. This allows for school and student non-response and also for school level and within-school level exclusions.

The KeyQuest sampling software provided by ACER will be used to the extent possible in selecting student samples of 42 students per school. Student lists will be formed according to international specifications and eligible students will be sampled with equal probability. Subsamples of 14 students per school will be selected to take part in the ERA, with a goal of assessing 10 students per school.

Main Study

The target number of completed assessments is 4,500 students in 150 schools. Assuming a target minimum response rate of 85 percent, we will need a sample of 150/0.85 = 176 eligible schools. Our first goal will be to achieve a pre-replacement response rate higher than achieved in earlier administrations (school response rates before replacement were 56 percent in 2000, 65 percent in 2003, and 69 percent in 2006). To achieve the target final school response rate, we will use replacement schools to complete the sample.

We will target at least 35 completed student assessments per school. Assuming a within-school response rate of 85 percent (rates were 85 percent in 2000, 82 percent in 2003, and 91 percent in 2006), the original sample size of students within schools will be 42.

Nonresponse Bias Analysis, Weighting, Sampling Errors

It is inevitable that nonresponse will occur at both levels: school and student. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES standards. After the international contractor calculates weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification.

B.3 Maximizing Response Rates

Our approach to maximizing school and student response rates includes the following:

• Use of a fall test administration, to avoid conflicts with State testing;

• Selecting and notifying schools at least a year in advance;

• Approaching schools directly, and notifying States and school districts;

• Assigning personal recruiters for specific schools;

• Incentives for schools, school coordinators, and students (see Section A9); and.

• Contact with schools and school coordinators at set intervals throughout the year preceding the assessment.

These approaches are based on recommendations from a NCES panel and experience with previous PISA administrations.

B.4 Purpose of Field Trial and Data Uses

Participation in the field trial is an international requirement for participating in the PISA 2009 main study. The main focus of the field trial is to collect enough assessment data to perform reliable tests of the items. During the field trial, procedures for conducting the main study including recruitment methods for obtaining school and student participation also will be evaluated. This information will be used to improve our recruiting strategies and materials for the main study.

B.5 Individuals Consulted on Study Design

Many people at OECD, ACER, and other organizations around the world have been involved in the design of PISA. Some of these were previously listed in section A8.

Windwalker Corporation will be the prime contractor conducting this study, with Westat and Pearson as subcontractors:

• Howard Fleischman, Project Director, (Windwalker) (703) 970-3527

• Paul Hopstock, Deputy Project Director (Windwalker) (703) 970-3522

• David Kastberg, Task Leader for Recruitment (Westat)

• David Ferraro, Task Leader for Sampling (Westat)

• Leslie Jocelyn, Field Director (Westat)

• Patricia Stearns, Director for Distribution and Scoring (Pearson)

Overall direction for PISA is provided by Holly Xie, the PISA National Project Manager of the National Center for Education Statistics, U.S. Department of Education.