Program for International Student Assessment 2022 (PISA 2022) Main Study
Supporting Statement Part B
OMB# 1850-0755 v.27
National Center for Education Statistics (NCES)
U.S. Department of Education
Institute of Education Sciences
Washington, DC
August 2019
revised February 2022
TABLE OF CONTENTS
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
B.2 Procedures for the Collection of Information 1
B.2a Statistical Methodology 1
B.3 Maximizing Response Rates 9
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
PISA 2022 assesses students nearing the “end of their compulsory school experience” and, as all prior administrations of PISA, is conducted in the United States by the National Center for Education Statistics (NCES) within the U.S. Department of Education. For international comparability, the target population is defined as students who are 15 years old, in grades 7 or higher. A range of exact birthdates is specified by the international coordinating committee based on the months in which the data will be collected. However, students must be between the ages of 15 years and 3 completed months and 16 years and 2 completed months at the beginning of the testing period. In the U.S., the universe for the selection of schools is all types of schools in the 50 states and the District of Columbia. Within sampled schools, students will be selected for participation by drawing a random sample among the 15-year-old students.
This section presents information on the PISA international standards and description for school and student sampling, recruitment, and data collection procedures for the PISA 2022 main study. Gaining schools’ and students’ cooperation in voluntary research is increasingly challenging, and employing effective strategies for gaining the cooperation of schools is central to the data collection effort. PISA 2022 main study states, districts, and schools will be recruited beginning in March 2022, and data collection will be conducted from October-November 2022.
The Technical Standards for PISA 2022 main study established by the international governing board include the following:
Standard 1.8 The student sample size for the computer-based mode is a minimum of 6,300 assessed students, and 2,100 for additional adjudicated entities, or the entire PISA Defined Target Population where the PISA Defined Target Population is below 6,300 and 2,100 respectively. The student sample size of assessed students for the paper-based mode is a minimum of 5,250. The minimum student sample size for financial literacy in the national sample is an additional 1,650 students, for a total of a minimum of 6,900 students that need to be assessed in PISA 2021 in the United States. If individual states participate in the U.S. to obtain state-level estimates, each state administering financial literacy would add approximately 550 students.
Standard 1.9 The school sample size needs to result in a minimum of 150 participating schools, and 50 participating schools for additional adjudicated entities, or all schools that have students in the PISA Defined Target Population where the number of schools with students in the PISA Defined Target Population is below 150 and 50 respectively. Countries not having at least 150 schools, but which have more students than the required minimum student sample size, can be permitted, if agreed upon, to take a smaller sample of schools while still ensuring enough sampled PISA students overall.
Standard 1.10 The minimum acceptable sample size in each school is 25 students per school (all students in the case of school with fewer than 25 eligible students enrolled).
Standard 1.11 The final weighted school response rate is at least 85 percent of sampled eligible and non-excluded schools. If a response rate is below 85 percent, then an acceptable response rate can still be achieved through agreed upon use of replacement schools.
Standard 1.12 The final weighted student response rate is at least 80 percent of all sampled students across responding schools.
Standard 1.13 The final weighted sampling unit response rate for any optional cognitive assessment is at least 80 percent of all sampled students across responding schools. In addition, NCES has a standard in which student response rate should be at least 85 percent, and the sampling design described below is based on that rate.
The design for this study will be self-weighting, stratified, consist of two stages, and will use probability proportional to size (PPS). There will be no oversampling of schools or students. Schools will be selected in the first stage with PPS and students will be sampled in the second stage yielding overall equal probabilities of selection.
The national PISA target population is 15-year-old students attending education institutions located within the U.S. in grades 7 and higher. The target population for any participating state is the same.. The specific definition of age eligibility that will be used in the survey is “…between 15 years and 3 (completed) months to 16 years and 2 (completed) months at the beginning of the testing window.”
The population of schools for PISA 2022 is defined as all schools containing any 15-year-olds in grades 7 through 12. As in previous PISA cycles, the school sampling frame for the PISA 2022 main study sample was developed from the most up to date NCES Common Core of Data (CCD) and Private Schools Survey (PSS) datasets. For the PISA 2022 main study, we used the school sampling frame prepared for the National Assessment of Educational Progress (NAEP) 2018, which uses the 2018-2019 CCD and the 2017-2018 PSS school data. The 2022 National Assessment of Educational Progress (NAEP 2022) does not include a grade 12 school sample. Therefore, overlap with the PISA 2022 school sample is not expected and the use of overlap control is not necessary. The grade structure of the school is a key stratification variable designed to reduce sampling error, but this is especially so in PISA because data analyses have shown that achievement is highly related to grade. Other stratification variables include public/private schools, region of the country, location (urban/suburban/town/rural, etc.), gender, state, enrollment of 15 year olds, and enrollment by race/ethnicity.
For the core computer-based assessment in mathematics, reading, and science, the international required minimum number of completed assessments is 6,300 students in 150 schools. An additional 1,650 assessed students are required for education systems assessing financial literacy for total of 7,950 assessed students. To achieve a larger number of students assessed than in 2018 and required for financial literacy, and to account for anticipated nonparticipation and student ineligibility, the number of students sampled within schools will again be 52 students, as it was in 2018, but with a larger school sample. Assuming the same response level as in PISA 2018, the initial target is a total sample of about 297 schools, with estimated 249 eligible, to yield about 212 participating schools (assuming a total 85% school participation rate). As allowed under the international sampling standards, to achieve the target final school response rate, we will use replacement schools to complete the sample.
Assuming a within-school student assessment rate of 80 percent (rates were 81 percent in 2000, 70 percent in 2003, 83 percent in 2006, 78 percent in 2009, 81 percent in 2012, 83 percent in 2015, and 78 percent in 2018), the original sample size of students within schools will be up to 52 students. In schools that do not have 52 PISA-eligible students, all eligible students will be sampled. Should any states participate in the 2022 assessment, each state would have a sample of 60 schools and 2,948 students to yield 2,447 assessed students (given an expected 83% assessment rate). As the main study plans for states and subnational jurisdictions are finalized, this information will be updated in the respondent burden table in the Supporting Statement Part A.
Cognitive Design.
The PISA 2022 design will follow closely that of PISA 2018. The field test contributed information to construct a multistage adaptive testing (MSAT) design for mathematics, as was done for reading in 2018. The reading MSAT will be carried forward in PISA 2022.
For the PISA 2022 main study, the United States will use an assessment design with total of 45 forms containing a combination of clusters mathematics MSAT and science, mathematics MSAT and reading MSAT, reading MSAT and science, mathematics MSAT and financial literacy, and financial literacy and reading MSAT. For the assessment, administered in a 2-hour session, each student will receive one form with a combination of clusters depending on the form.
The field test utilized form combination organized in three distinct groups in order to ensure adequate coverage of newly developed items and to examine the psychometric properties of the items. There will be MSAT in 2022 for reading as a minor domain. MSAT was successfully developed and administered for reading as a major domain in PISA 2018. As stated above, one of the goals met by the field test was collecting information in preparation of the planned introduction of MSAT for the major domain of mathematical literacy, the use of a reduced reading MSAT, and the use of previously used nonadaptive designs for the other minor and innovative domains. The field trial design included variable unit positioning within clusters and investigated the effects of variable unit positioning versus fixed positions in preparation for the main study, the hypothesis being that item parameter invariance is only supported when using intact clusters.
The field test assessment design utilized 6 trend clusters and 12 new clusters of mathematics items, 6 trend clusters of science, 2 clusters combining new and trend items for financial literacy, and for reading, a reduced version of the reading MSAT. These clusters were organized in a rotation within three groups of students. Within a school, sampled students were assigned to each of the three groups.
Group 1 received 4 trend clusters of combinations of science and mathematics, mathematics and reading, mathematics and financial literacy, or reading and financial literacy. These clusters were fixed unit order. Group 1 expected to yield 137 responses per mathematics and science item and 80 responses per reading item per participant.
The approach from 2015 and 2018 to reduce the distinction between major and minor domains has been supported by analytic approaches that utilize data from multiple cycles. Data collected during the major domain cycle provides a basis for the analyses of two subsequent minor domain cycles. In terms of the field test data analysis, Group 1 forms were directly linked to the existing data from prior cycles, The variability in the psychometric characteristics of the PISA2022 field test data collected relative to the 2018 analysis results gave a baseline for the magnitude of error expected across data collections when moving to mathematics adaptive testing. The variability in Group 1, given fixed unit order within a cluster with full construct coverage is the lower bound used to evaluate the variability of psychometric characteristics observed for Group 2 (variable unit order) and Group 3 (fixed unit order) of the new mathematics clusters. Also, the same variability was referred to for comparisons of mathematics trend clusters of Group 1 (fixed unit order) and Group 2 (variable unit order).
Group 2 received new and trend items in mathematics. These items applied variable unit ordering within clusters. The design provided variations in unit ordering within clusters and were examined relative to Group 1. Each of the 24 Group 2 forms contained a combination of one to six trend mathematics clusters and 3 of the 12 new mathematics clusters. Every trend cluster will be paired with a new cluster once and appears once in each position. The design expected to yield, per participant, 103 responses per trend item and 154 responses per new item.
Group 3 contained new mathematics clusters and was based on a fixed order of units to provide a basis of comparison to the varying unit orders in Group 2. Each form was administered to 757 students total.
The results of the field test showed no significant difference between the percentage of students that reached the items and selected the correct answers in the FUO versus the VUO forms both for reading trend and new items. The pairs for both trend items and new items were observed to be very similar in response time and no unit order effect was found. Additionally, machine-based coding was introduced, showing a 20 percent reduction of human coding burden across items, domains, and participating countries. Based on this evidence, it was determined that the order in which units were presented did not significantly impact the estimation of item and person parameters and an MSAT design could be introduced for the first time into the main study.
Financial Literacy Design. The U.S. is again participating in the optional financial literacy assessment in 2022. The 2022 assessment design is the same as the one used in PISA 2018, with an expanded student sample used to assess financial literacy in the same session that mathematics, science, and reading are assessed. Approximately 12 students were assigned financial literacy assessment items, while the remaining 40 students receiving combinations of mathematics, reading and science items. Each student sampled for financial literacy will receive two clusters of math or reading and two clusters of financial literacy. The design for financial literacy is based on a yield of 1,650 assessed students. These students will be selected separately from those sampled for the core assessment but will be administered the assessment in the same session as the students taking the core assessment. The financial literacy instrument contains 2 clusters with trend items from 2012, 2015, and 2018 as well as new interactive items.
Background Questionnaire Instruments. The questionnaires were developed to address the questionnaire framework developed for PISA 2022. The framework defines 19 modules across the school and student questionnaires comprising student background characteristics, teaching and learning practices, school governance, and non-cognitive/metacognitive constructs dealing with reading-related outcomes, attitudes, and motivational strategies. In addition, the questionnaires include items that have been included in multiple cycles of PISA, allowing the investigation of patterns and trends over time.
Global Crisis Module. The impact of the COVID-19 pandemic on schools and students is of great interest internationally. A module of pandemic-related items, called the Global Crisis Module, was developed for the field test for both the school and student questionnaires. This module has 3 distinct purposes:
capture key contextual variables to describe students’ learning experiences during times of school building closures as well as preparedness for future global crises;
focus on educational opportunities as well as students’ subjective views on distance learning; and
avoid potentially sensitive questions (e.g., about personal health issues due to COVID-19).
The school and student questionnaires, including these newly developed items, are presented in appendix C.
School questionnaire. The principal or designate from each participating school will be asked to provide information on basic demographics of the school population and more in-depth information on one or more specific issues (generally related to the content of the assessment in the major domain, which is mathematics in 2022). Basic information to be collected includes data on school location; measures of socio-economic context of the schools’ student population, including location, school resources, facilities, and community resources; school size; staffing patterns; instructional practices; and school organization. The in-depth information is designed to address a very limited selection of issues that are of particular interest and that focus primarily on the major content domain, mathematics. Global Crisis items will also be included. For the main study, it is anticipated that the school questionnaire will take approximately 45 minutes. The school questionnaire will be available to respondents online, as it was in PISA 2015 and 2018. The intent is that the school principal will respond to the questionnaire, but they may designate someone to complete the questionnaire. Principals, or their designate, access the questionnaire through a secure website with a username and password provided to them in their invitation (see Appendix A). Responding to the instrument is flexible. Respondents may break off and return to continue responding to the questionnaire by logging in again with their credentials. In addition to completing questions, respondents may review and change their responses to previously answered questions.
Student questionnaires. In the 2022 cycle, the U.S. will administer three student questionnaires that will be completed in a single student questionnaire session following the assessment session: the core student questionnaire, financial literacy questionnaire and the information and communication technology (ICT) familiarity questionnaire. The content of these instruments is described below. Students access the questionnaires through the Student Delivery System (SDS), in the same way they access the assessment, approximately 10 minutes after the assessment session has ended. Unlike the school questionnaire, the student questionnaires are only administered from the SDS, and students only access the questionnaires in the questionnaire session. That is, student access to the questionnaires can only be accomplished during the in-school session; there is no access to the student questionnaire outside of school hours.
Student core questionnaire. Participating students will be asked to provide information pertaining primarily to the major assessment domain in 2022: mathematics. Information to be collected includes demographics (e.g., age, gender, language, race, and ethnicity); socio-economic background of the student (e.g., parental education, economic background); student's education career; and access to educational resources and their use at home and at school, which have been standard questions in PISA since the earliest rounds. Domain-specific information will include instructional experiences and time spent in school, as perceived by the students, and student attitudes towards mathematics. The questions included in the Global Crisis Module have been added to the end of the core questionnaire.
The goal is for the student questionnaire to take approximately 35 minutes to complete in the field test and the main study. The PISA 2022 student questionnaire will implement a matrix sampling design. Using within-construct matrix sampling is an innovation to the questionnaire design in PISA 2022 that has not been used in previous PISA cycles.
The matrix sampling design rotates questions within constructs instead of across constructs. In the PISA 2022 within-construct matrix sampling design, every student will receive questions on all constructs but only answer a subset of all questions for each construct, thus resulting in a complete database in terms of construct-level indices. The PISA field test data for most of the scales analyzed with IRT was collected using the new within-construct matrix sampling design for questionnaires where each student answers a randomly chosen subset of five items for each construct from the pool of 8-10 items for the respective construct. The evaluation of the design confirmed that scales scores could be estimated for every student in the sample, regardless of which specific set of items the students answered. Based on this evaluation of the field test data, this design will be used only for constructs that:
can be IRT scaled across country/economy-by-language groups without substantial misfit,
meet minimal reliability criteria established in previous PISA cycles (i.e., >.6), will be reported as scaled indices (vs. reported on the item-level only), and
are not a component of the PISA ESCS index.
This approach is viable for PISA 2022 due to the limited time available for the questionnaire and the large student sample size in large-scale assessments.
Financial Literacy (FL) questionnaire. The FL questionnaire aims to examine students’ experience with money matters, such as having savings accounts, debit or prepaid cards, as well as whether they have experienced financial-related lessons in their school careers. Many of the items in the FL questionnaire were previously administered in 2012, 2015, and 2018, with a handful of new items. The FL questionnaire for students is expected to take approximately 10 minutes to complete.
Information and Communication Technology (ICT) Familiarity questionnaire. The ICT questionnaire aims to examine students’ ICT activities and domain-specific attitudes including access to and use of ICT at home and at school, students’ attitudes towards and self-confidence in using computers, self-confidence in doing ICT tasks and activities; and navigation indices extracted from log-file data (number of pages visited, number of relevant pages visited). The ICT questionnaire for students is expected to take approximately 10 minutes to complete. The U.S. successfully administered the ICT questionnaire in the 2018 cycle of PISA.
For the core computer-based assessment in mathematics, reading, and science, the international required minimum number of completed assessments is 6,300 students in 150 schools. An additional 1,650 assessed students are required for education systems assessing financial literacy for total of 7,950 assessed students. In past PISA rounds up until 2018, the U.S. typically assessed between 5,600 and 5,900 students in 165 schools and sampled 42 students per school. However, as was already done in PISA 2018 main study, in PISA 2022, in order to achieve a larger number of students assessed as required for financial literacy, and to account for anticipated nonparticipation and student ineligibility, wherever possible, we will sample 52 students per school (42 students for the core assessment + 10 students for financial literacy). Assuming the same response level as in PISA 2018, the initial target is a total sample of about 297 schools, with estimated 249 schools eligible, to yield about 212 participating schools (assuming a total participation rate among schools of 85 percent, after replacement). As allowed under the international sampling standards, to achieve the target final school response rate, we will use replacement schools to complete the sample.
The student-per-school target for the core assessment is at least 42 completed student assessments per school. Assuming a within-school student assessment response rate of 85 percent (rates were 85 percent in 2000, 82 percent in 2003, 91 percent in 2006, 86 percent in 2009, 89 percent in 2012, 89 percent in 2015, and 85 percent in 2018), the original sample size of students within schools will be 52.1 In schools that do not have 52 PISA-eligible students, all eligible students will be sampled. Should any states participate in the 2022 assessment, each state would have a sample of 54 schools and 2,808 students to yield approximately 2,330 assessed students. As the main study plans for states and subnational jurisdictions are finalized, this information will be updated in the respondent burden table in the Supporting Statement Part A.
It is inevitable that nonresponse will occur at the school and student level. We will analyze the nonrespondents and provide information about whether and how they differ from the respondents along dimensions for which we have data for the nonresponding units, as required by NCES statistical standards. After the international contractor calculates weights, sampling errors will be calculated for a selection of key indicators incorporating the full complexity of the design, that is, clustering and stratification.
Based on recruitment knowledge gained in PISA 2018 and other NCES studies, states and districts will be notified about the study before the sampled schools in their jurisdictions are contacted. School staff often wants to be assured that their school district knows about the study before they agree to participate. The planned PISA 2022 approach to state, district, and school recruitment is described in this section, and all of the respondent recruitment materials for the main study are provided in Appendix A and B.
State Recruitment. In March 2022 state education agencies (SEAs) in states that contain schools sampled for the field test will be mailed a package that includes the state letter and PISA 2022 advanced materials. We are working with the NAEP State Coordinators (NSCs) in this state recruitment effort. Some NSCs will send the state letter and personally follow-up to answer any questions; otherwise, Westat school recruiters will mail the package and follow-up with the districts and schools being recruited.
District Recruitment. Also, beginning in April 2022 shortly after the state mailing, advance packages to district superintendents will be mailed. Each package contains an introductory letter, including a list of sampled schools in the district’s jurisdiction, and the PISA 2022 advanced materials. The district mailings will come from the NSC or Westat, depending on each NSC’s preference. Shortly after the mailing, the district superintendent will be contacted by phone to inform him/her of the study, ensure they received the PISA 2022 package, and answer any questions they may have. Any issues with approaching schools in the district are also discussed at the time.
The PISA 2022 study staff responds to districts’ requirements such as research applications or meetings to provide more information about the study. If a district chooses not to participate, the recruiter documents all concerns listed by the district so that a refusal conversion strategy can be formulated. As in PISA 2018 and previous NCES studies, some NSCs talk to district staff themselves, others mail the package but do not further contact districts, while still others do not want to be involved in district recruitment at all. In cases where the NSCs do not wish to follow-up, Westat’s recruiters will work directly with the districts.
Based on prior recruitment experience on a variety of NCES studies, some districts are designated as “special handling districts.” Contacting special handling districts begins with updating district information based on researching online sources, followed by calls to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, inquiries are also made about the amount of time the district spends reviewing similar research applications.
School Recruitment. After each district of sampled public schools has been informed of the study and has confirmed the receipt of the PISA 2022 package, the school mailings will be triggered. All of the school mailings, taking place from March through August of 2022, contain an introductory letter offering each sampled school a $250 check as a thank you for participation (to be mailed with a thank you letter after the end of the data collection); the PISA brochure and timeline; and school administrator and student Frequently Asked Questions (FAQs) sheets. Shortly after the mailing, the NSCs or Westat recruiters contact the school administrator to discuss the study, gain cooperation, and assign a school staff person to serve as the PISA 2022 school coordinator who will work with the PISA staff to manage the data collection in the school. The school coordinator will act as the liaison between study staff and their school.
In cases where recruitment proves more difficult, school recruiters consult with the PISA home office to evaluate a conversion plan for each school. Typically, the types of general issues are principals who are difficult to reach, school staff who are considering participation but not providing a final decision, principals and/or staff who express a concern and need follow-up, and principals who may require additional appreciation for agreeing to participate.
School Coordinator Contact. Shortly after permission has been granted from the school administrator, Westat emails the school coordinator the MyPISA website registration information. The MyPISA website (described in more detail below) contains the e-filing instructions and templates for within-school student sampling, and other logistical information for the assessment. Each school will receive a unique ID that will support multiple school users. Each user must then provide their own contact information and set up their own unique account. This registration process has been used across all of NCES’ international studies and in NAEP.
For the field test, beginning in August 2021, school coordinators of participating schools will receive a handbook detailing the procedures for administering the PISA 2022 survey in the school, and providing timelines and instructions for submission of the list of students via MyPISA. Westat PISA staff will also call the school coordinator to discuss the PISA activities at the school, including when to begin constructing and e-filing the student list. School coordinators of participating schools in the main study will receive these materials beginning in August of 2022.
Student sampling. Beginning in August 2022, student lists will be collected to draw a student sample in each school. The student lists are designed to collect students’ names, grade, gender, and month and year of birth. Edit checks will be run during sampling to check column mappings and completeness of data to ensure that all student listing forms are constructed properly for sampling (see Appendix A).
After student sampling has been completed, student tracking forms will be generated in the Maple software system and distributed to the school coordinator via MyPISA. These forms will also be used by the PISA field staff for administering the assessment and recording student participation status.
After students are sampled and school coordinators are notified of the sampled students, each school coordinator will notify and provide consent materials to parents of sampled students using the materials provided in Appendix B. Prior to the assessment, the test administrator assigned to each school will collect a copy of the notification and consent materials the school sent to parents as proof that the school took care of parental notification and consent.
Role of the MyPISA Website. The central purpose of MyPISA is to provide a way for the schools to securely upload a list of students and to provide the tracking forms with the sampled students to the school coordinator and PISA field staff. MyPISA is also used as a source for disseminating information about PISA, such as providing copies of advanced materials and descriptions of survey activities, school actions, and instructions and templates for submitting lists. School registrants are encouraged to update their contact information and access information about the study in MyPISA.
PISA 2022 main study data collection will occur in October 2022–November 2022. Prior to data collection, Westat PISA staff will complete various pre-survey activities.
Pre-survey Activities
In August 2022, the school coordinators will receive a handbook and instructions for assembling a student list. The lists will be submitted to Westat via MyPISA, and the samples will be drawn during August and September 2022. Beginning in August 2022, school coordinators will be asked to do the following activities:
Create and submit a list of eligible students via MyPISA;
Receive and distribute to sampled study materials from the PISA Home Office;
Encourage the principal to complete their questionnaire; and
Meet with the PISA test administrator to review assessment logistics and hold a brief meeting with the sampled students to show a PISA video presentation explaining the study and the students’ role and contribution and to answer questions about PISA. Depending on the situation with the coronavirus pandemic, this meeting may be virtual facilitated by the school coordinator via Google Meet or another electronic meeting platform used by the school. In the event that Westat field staff cannot hold a student meeting, the school coordinator may hold the meeting. This has sometimes occurred in past PISA rounds due to scheduling issues and travel.
Data Collection
The school questionnaire will be available electronically, with a hard copy available upon request. The principal will be given links to their questionnaires as part of invitation to PISA and will also be emailed their personalized link to the questionnaire. The student questionnaire will be administered as part of the SDS provided to countries by the International Consortium. The SDS includes the PISA main assessment, the financial literacy assessment, and the student questionnaires. The PISA questionnaires are provided in Appendix C.
The PISA assessments are administered to students by trained PISA test administrators hired and trained by Westat. The PISA test administrators will bring all assessment equipment to the school including student laptops and peripheral equipment (power cords, additional extension cords, routers). They are responsible for set-up and breakdown of equipment and administration of the assessment to the students. All that is required from the school is an adequate space to set up the equipment and hold the assessment.
Students begin the data collection activities by entering a room containing desks and PISA laptops. Upon entering, each student is directed to their assigned computer by a PISA test administrator. On the computer is a paper log-in form (see Appendix B, p. 32) with that student’s unique log-in information. This form also has the OMB statement printed below the log-in information2. The first screen that students see is the SDS login screen (see Appendix C, p. 83), where the student enters the SDS to begin PISA student data collection activities. The PISA test administrator gives the students the verbal instruction to enter their log-in information to begin. Students complete the cognitive assessment, take a short break (about 10 minutes), and return to complete the student questionnaires which end the student data collection.
Throughout the data collection period, PISA staff and the school coordinator will monitor the return of school questionnaires and, working in conjunction, will follow-up with non-responders as needed. A ‘window-is-closing’ non-response follow-up effort will be utilized, gently reminding principals to complete their questionnaires. Near the end of data collection, the two reminders before the last are designed to establish a deadline effect and will be followed with an extension email. This campaign-style approach is designed to provide soft reminders across the data collection window, while creating a sense of urgency to respond towards the end.
School coordinator incentive checks will be distributed after Westat receives the student list and completes within-school student sampling. School incentive checks will be distributed after the assessment is completed. Incentives will be mailed to schools on a weekly basis throughout the data collection period. Student incentives will be distributed to the students at the end of the questionnaire session.
Our approach to maximizing school and student response rates in the main study includes the following:
Use of a fall test administration, to avoid major conflicts with state testing;
Selecting and notifying schools at least a year in advance;
Communicating with state and district officials early in the process and applying a more proactive approach with states by coordinating with NAEP State Coordinators to gain assistance with sampled schools;
Assigning personal recruiters for specific schools and use of personal visits to districts and/or schools early in the contact process;
Monetary incentives for schools, school coordinators, and students (see Section A.9) and below;
School report incentive for schools;
Volunteer service certificate of 4 hours from the U.S. Department of Education incentives for students;
Contact with schools and school coordinators at set intervals throughout the year preceding the assessment;
Use of an informational video about PISA 2021 to motivate student participation and full effort during the assessments; and
Use of individually tailored refusal conversion strategies to encourage participation (as detailed above in the section entitled “Respondent Recruitment”).
Our approach to gaining cooperation from respondents is multipronged and takes into consideration feedback and advice that NCES has received over the years. Beyond monetary incentives, we offer feedback to the school (in the form of the school report) and service credit for student participants (in the form of volunteer service time). In addition, we try to engage districts and schools in the study by informing them of findings and results from previous rounds in the form of short videos and focused findings offered by OECD and NCES.
PISA 2022 main study schools that meet the criteria for receiving a school report (see section A.9 of Supporting Statement Part A), will be provided school-level PISA 2022 results. While individual-level scores cannot be produced from PISA data, a school level report showing comparative data for the school can be produced when the school has a participation rate of 85 percent or better and at least 10 assessed students. The results in the school-level report will be comparative results that do not provide actual school scores, but rather indicate how the school performed compared to country averages and to other US schools with similar demographic characteristics. For PISA 2022, we are also attempting to design a second, alternate report to provide information from the contextual questionnaires for schools that do not meet the requirements for receiving the standard school report. This second “B” report will need to be reviewed and approved by the NCES chief statistician prior to implementation.
These approaches are based on recommendations from an NCES panel, experience with previous PISA administrations, as well as extensive discussions with the NCES chief statistician.
Participation in the field test is an international requirement for participating in the PISA 2022 main study. The main focus of the field test was to collect enough assessment data to perform reliable tests of the items, to evaluate newly developed assessment items and new or revised multi-stage adaptive test designs, and to test the survey operations. The field test was also used to evaluate recruitment, data collection, and data management procedures in preparation for the main study, including recruitment methods for obtaining school and student participation. The results of the field test were analyzed by OECD. In the U.S., NCES used the field test results to: (a) determine the final main study design and decide in which international options the U.S. will participate, (b) improve recruiting strategies and materials for the main study in the U.S., and (c) finalize all study procedures.
Many people at OECD, ETS, and other organizations around the world have been involved in the design of PISA. Some of the lead people are listed in section A8. Overall direction for PISA conducted in the U.S. is provided by Samantha Burg, the PISA National Study Lead, and other staff at NCES.
1 For the national main study sample, we expect to draw an initial sample of 297 schools. Taking into account closed, merged, and ineligible schools (historically, around 11% of sampled schools), as well as the historical school-level response rate, we anticipate interacting with/recruiting about 256 of these schools, of which, we estimate, 218 will participate in the PISA 2022 main study. To obtain the required number of students, we will ask to sample up to 52 students in each school. However, some of the smaller schools will not have 52 students available. We estimate: 218 schools x 52 students sampled x 0.945 (a factor used to account for variations in student population sizes across the schools) = 10,713 starting student sample size that we will work to recruit. Based on historical student assessment rates (~83%), we estimate that, in the end, we will assess about 8,892 students (10,713 x 0.83), which will assure that we meet the minimum required 7,950 assessed students.
2 Originally, we planned to include the OMB statement, including assurances that the data collection was voluntary, the OMB number and expiration date, and the paperwork burden statement, on the SDS log-in screens. We have since learned that those screens are uneditable, making this the best way to make sure that students are fully informed.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | PREFACE |
| Author | Janice Bratcher |
| File Modified | 0000-00-00 |
| File Created | 2022-03-09 |