Changes Memo

_MEMORANDUM NATIONAL CENTER FOR EDUCATION STATISTICS

Institute of Education Sciences

United States Department of Education

Date: June 23, 2021

To: Robert Sivinski, OMB

Through: Carrie Clarady, OMB Liaison, NCES

From: Samantha Burg, NCES

Re: Program for International Student Assessment 2022 (PISA 2022) Main Study Recruitment and Field Test Update (OMB# 1850-0755 v.26)

The Program for International Student Assessments (PISA) is an international assessment of 15-year-olds, which focuses on assessing students’ reading, mathematics, and science literacy. PISA was first administered in 2000 and is typically conducted every three years. The United States has participated in all of the previous cycles and planned to participate in 2021 in order to track trends and to compare the performance of U.S. students with that of students in other education systems. PISA is sponsored by the Organization for Economic Cooperation and Development (OECD). In the United States, PISA is conducted by the National Center for Education Statistics (NCES), within the U.S. Department of Education. In each administration of PISA, one of the subject areas (reading, mathematics, or science literacy) is the major domain and has the broadest content coverage, while the other two subjects are the minor domains. PISA emphasizes functional skills that students have acquired as they near the end of mandatory schooling (aged 15 years), and students’ knowledge and skills gained both in and out of school environments. The next administration of PISA will focus on mathematics literacy as the major domain. Reading and science literacy will also be assessed as minor domains, with additional assessment of financial literacy. In addition to the cognitive assessments described above, PISA 2022 will include questionnaires administered to school principals and assessed students. To prepare for the main study, PISA countries will conduct a field test in the spring of the year previous, primarily to evaluate newly developed assessment and questionnaire items but also to test the assessment operations.

The request to conduct PISA 2022 main study recruitment and field test was approved in February 2021 (OMB# 1850-0755 v.25) and included the plan to “submit a change request to OMB in May 2021 with the final main study recruitment materials and parental consent letters, details about any changes to the design and procedures for the main study, and updates to the respondent burden estimates for the main study data collection”. This change request:

1. updates the main study recruitment materials in Appendix A-2;

2. updates the burden estimate based on an adjusted main study school sample size and assumptions about the rate of ineligible and out of scope schools agreed with the international sampling contractors.

Changes to all documents are described below in more detail. Specific illustrated changes are marked in red; additions are in simple red font, while deletions are in red strikethrough. This request provides the final plans, burden and materials for the PISA 2022 main study recruitment.

Based on feedback and experience from the PISA 2022 field trial, NCES is requesting OMB approval for the changes to aspects of the PISA 2022 main study descriptions in the Supporting Statements Parts A and B and to the main study recruitment materials in Appendix A2 as detailed below.

1. Specific material changes to Part A

Section A.1: Updating the plans for OMB clearance

In order to begin recruiting schools for the main study by October 2021, we will submitted a change-request to OMB in May June 2021 with the final main study recruitment materials and parental consent letters, details about any changes to the design and procedures for the main study, and updates to the respondent burden estimates for the main study data collection.

Section A.5: Small error from previous package corrected

For example, the selection of schools to be assessed in the PISA 2022 field test (spring 2020 fall 2021) will avoid overlap with the selection of schools for NAEP and PIRLS, which will also be in the field in the spring fall of 2020 2021.

Section A.10: Updated confidentiality language with new OMB expiration date

OMB No. 1850-0755, Approval Expires xx/xx/2022 04/30/2024

Section A.12:

Page 12-13 - The estimates of burden have changed. Based on consultations with the international sampling contractors, the main study school sample increased from 288 schools to 297 schools. The expected response rate changes from 0.89 to 0.84 and the expected assessment rate changed from .83 to .80. These changes align with the experience from 2018 and are thought to be a more conservative estimate of response.

Page 13-14 - Table A-1 reflects these changes and adjusts the total burden. The total respondent burden related to recruitment decreases by 25 hours from 2,387 hours to 2,362 hours.

Table A-1. Burden estimates for PISA 2021 2022 field test and main study

	Sample	Expected response rate	Number of respondents	Number of responses	Burden per respondent (minutes)	Total burden (hours)
FIELD TRIAL—Based on core + international options
Recruitment and Pre-Assessment Activity (includes Puerto Rico)
School Administrator (US sample)	50	1.00	50	50	90	75
Special Handling Districts IRB Staff Approval (US sample)	15	1.00	15	15	120	30
Special Handling Districts IRB Panel Approval (US sample)	90	1.00	90	90	60	90
School Coordinator (US sample)	50	1.00	50	50	240	200
School Administrator
Questionnaire (US sample)	50	1.00	50	50	53	44
Parent
Student Participation Consent	3,000	1.00	3,000	3,000	3	150
Total School Burden Field Trial			3,255	3,255		589
Student
US national sample
Directions	3,000	0.83	2,490	2,490	10	415
Assessment	3,000	0.83	2,490	2,490	120	4,980
Student questionnaire (Main questionnaire)	3,000	0.83	2,490	2,490	36	1,494
Student questionnaire (ICT questionnaire)	3,000	0.83	2,490	2,490	15	623
Student questionnaire (Financial Literacy questionnaire)	3,000	0.83	2,490	2,490	15	623
Total Student Burden Field Trial			2,490	7,470		2,740
Total Burden Field Trial			5,745	10,725		3,329
MAIN STUDY —Based on core + international options
US national sample
Recruitment and Pre-Assessment Activity
School Administrator	288 297	.89 .84	256 249	256 249	90	384 374
Special Handling Districts IRB Staff Approval (US sample)	30 31	1.00	30 31	30 31	120	60 62
Special Handling Districts IRB Panel Approval (US sample)	180 187	1.00	180 187	180 187	60	180 187
School Coordinator	256 249	.85	218 212	218 212	240	872 848
School Administrator
Questionnaire	218 212	1.00	218 212	218 212	45	164 159
Parent
Student Participation Consent	10,713 10,418	1.00	10,713 10,418	10,713 10,418	3	536 521
Student
Directions	10,713 10,418	0.83 0.80	8,892 8,334	8,892 8,334	10	1,482 1,389
Assessment	10,713 10,418	0.83 0.80	8,892 8,334	8,892 8,334	120	17,784 16,668
Student questionnaire (main questionnaire)	10,713 10,418	0.83 0.80	8,892 8,334	8,892 8,334	36	5,335 5,000
Student questionnaire (FL questionnaire)	10,713 10,418	0.83 0.80	8,892 8,334	8,892 8,334	15	2,223 2,084
Student questionnaire (ICT questionnaire)	10,713 10,418	0.83 0.80	8,892 8,334	8,892 8,334	15	2,223 2,084
State samples (up to 3 states)
Recruitment and Pre-Assessment Activity
School Administrator (US states)	162	1.00	162	162	90	243
School Coordinator (US states)	162	1.00	162	162	240	648
School Administrator
Questionnaire (US states)	162	1.00	162	162	45	122
Parent
Student Participation Consent	8,424	1.00	8,424	8,424	3	421
Student
US states (includes up to 3)
Directions	8,424	0.83	6992	6992	10	1,165
Assessment	8,424	0.83	6992	6992	120	13,984
Student questionnaire (main questionnaire)	8,424	0.83	6992	6992	36	4,195
Student questionnaire (FL questionnaire)	8,424	0.83	6992	6992	15	1,748
Student questionnaire (ICT questionnaire)	8,424	0.83	6992	6992	15	1,748
Total School Recruitment Burden - Main Study			1,008 1,003	1,008 1,003		2,387 2,362
Total Burden Requested in this Submission (v.26)			6,753 6,748	11,733 11,728		5,716 5,691
NOTE: OMB Clearance Requested: Total Burden includes all burden associated with conducting the PISA 2022 Field Test and the recruitment and preassessment activities for the PISA 2022 Main Study. The estimated PISA 2022 Main Study burden is conservatively high because the PISA 2022 Main Study may include up to 3 states, however the burden is held consistent with national sample schools because of potential variability between states. The total student burden does not include time for cognitive assessment and its associated instructions, because assessment is not subject to PRA. *Special note: For the national main study sample, we expect to draw an initial sample of 297 schools. Taking into account closed, merged, and ineligible schools (historically, assuming a similar rate to 2018, around 1116% of sampled schools), as well as the historical school-level response rate, we anticipate interacting with/recruiting about 256 249 of these schools, of which, we estimate, 218 212 will participate in the PISA 2022 main study. To obtain the required number of students, we will ask to sample up to 52 students in each school. However, some of the smaller schools will not have 52 students available. We estimate: 218 212 schools x 52 students sampled x 0.945 (a factor used to account for variations in student population sizes across the schools) = 10,713 10,418 starting student sample size that we will work to recruit. Based on the 2018 student assessment rates (~83 80%), we estimate that, in the end, we will assess about 8,892 8,334 students (10,713 10,418 x 0.83 0.80), which will assure that we meet the minimum required 7,950 assessed students.

Page 14 - The hourly rates for staff and parents have changed a result of the updates to the burden estimates. The total burden time cost decreases from $122,234 to $121,746.

Section A-16

Page 16 – the schedule has a row added for the main study data collection submission.

January 2021-June 2021	Prepare data collection manuals, forms, assessment materials, and questionnaires for field test
April 2021-August 2021	Contact and gain cooperation of states, districts, and schools for field test
August – October 2021	Select student samples and collect field test data
November 2021	Deliver raw data to international sponsoring organization
January 2022	Receive Field test Report from international sponsors
October 2021–September 2022	Prepare for the main study phase/recruit schools
March 2022	Submit OMB for main study data collection
September 2022–November 2022	Collect main study data
March - April 2023	Receive final data files from international sponsors
August - December 2023	Produce General Audience Report and Survey Operations/Technical Report for the U.S.

2. Specific material changes to Part B

Section B2.a

Page 6 – Main Study Sampling: Updated the school sample size from 288 to 297 schools, the number of expected eligible schools from 256 to 249 schools, and the number of expected participating schools from 218 to 212. This change is based on revised assumptions for the number of in-scope schools and the assumed assessment rate.

3. Specific material changes to Appendix A-2

General changes: We eliminated the private school versions of the study brochure and student FAQ. These versions were very redundant to the public school versions. We cannot include information related to specific, private-school related findings for U.S. schools because the number of private schools in the samples are too small to provide valid comparative data. It is difficult to craft a version for private schools that is unique and that touches on the themes of interest to them. We did retain the private school version of the school FAQ.

State PISA 2022 Letter

Page 3 – Made a minor update in the third paragraph to refer to “school staff person” as “school staff.”

District PISA 2022 Letter

Page 4 – Separated the school incentive statement as a standalone bullet to help both readability and drawing attention to the incentive. The text now reads:

• While participation in this study is voluntary, your each school plays an important role in ensuring that the U.S. sample is representative of the knowledge and skills of all students across the country. As a thank you for your schools’ time and effort to participate in PISA: Each school will receive $250 for participating.

• As a thank you for your schools’ time and effort to participate in PISA: Each school will receive $250 for participating.

• Your Each school’s PISA school coordinator (a school staff person designated to communicate with PISA staff) will receive $200 as a thank you for their time and effort.

• Each participating student will be eligible to receive $25 and a volunteer service certificate of 4 hours from the U.S. Department of Education.

Public School PISA 2022 Letter

Page 6 – Separated the school incentive statement as a standalone bullet to help both readability and drawing attention to the incentive. The text now reads:

• While participation in this study is voluntary, your school plays an important role in ensuring that the U.S. sample is representative of the knowledge and skills of all students across the country.

• As a thank you for your time and effort to participate in PISA: Your school will receive $250 for participating.

• Your school’s PISA school coordinator (a school staff person designated to communicate with PISA staff) will receive $200 as a thank you for their time and effort.

• Each participating student will be eligible to receive $25 and a volunteer service certificate of 4 hours from the U.S. Department of Education.

• Your school may also receive a school-level report highlighting the performance of students in your school compared to their peers in other private schools, U.S. schools, and high-performing OECD countries (pending sample size and response rate requirements).

PISA 2022 Field Test Brochure

Page 8 – Made minor updates to wording in brochure for reading clarity. In the first paragragh ‘What is PISA?’, we deleted the phrase “including the United States” in the second sentence because the participation of the United States is confirmed in the next sentence.

The Program for International Student Assessment (PISA) is an international assessment of students nearing the end of compulsory education that measures how well students apply their knowledge and skills in mathematics, reading, science, and financial literacy to problems set in real-life contexts. Conducted seven times since 2000, the next round of PISA in 2022 will include students from more than 80 countries and education systems around the world, including the United States. PISA is coordinated by the Organization for Economic Cooperation and Development (OECD) and conducted in the United States by the National Center for Education Statistics (NCES) within the U.S. Department of Education.

The section “Why is PISA important and what are the benefits of participating?” is divided into two sections. These sections now read:

Why is my school’s participation in PISA important?

Through participating in PISA, your school and students contribute to important efforts to improve education in our country. You will help ensure the United States understands how well students nearing the end of compulsory education are prepared to enter the workforce or post-secondary education in comparison to their peers from more than 80 countries. Your school’s participation will also help identify U.S. students’ strengths and weaknesses in mathematics, reading, science, and financial literacy and offer insights into successful policies and practices in other countries.

Each school and student that participates in PISA plays a vital role in representing other similar schools and students. It is essential that sampled schools contribute to this important measurement of educational progress to ensure the U.S. sample reflects the knowledge and skills of all students across the country.

As a token of appreciation for participating, your school will receive $250, the school coordinator (a school staff person designated to communicate with PISA staff) will receive $200, and students who participate will each receive $25 as well as 4 hours of volunteer service time.

What are the benefits of participating in PISA?

Schools participating in PISA may receive a school-level report (pending sample size and response rate requirements) when the national results are released. The school report, provided only to your school, highlights the performance of students in your school compared to their peers in other schools like yours, U.S. schools, and high-performing countries around the world.

As a token of appreciation for participating, your school will receive $250, the school coordinator (a school staff person designated to communicate with PISA staff) will receive $200, and students who participate will each receive $25 as well as 4 hours of volunteer service time.

Page 9 – In the section “How is PISA developed?” we made minor revisions for clarity:

How is PISA developed?

PISA is developed through an international collaborative process involving input from U.S. and international experts in science, reading, mathematics, financial literacy, and educational measurement. The PISA assessment materials are thoroughly reviewed by within-country experts to make sure the materials are appropriate for each country’s students. Finally, the final Final assessment materials are endorsed by all participating countries.

PISA FAQ for school administrators – Public schools

Page 10 – Moved the section, “What does participating in PISA involve?” under the section “What are the benefits to participating?”

Adjusted the text in the section, “Why should my school and students participate?” The section reads:

The participation of selected public and private schools in the United States is vital to ensuring an accurate representation of the overall population of students nearing the end of compulsory education across the country—including U.S. public and private schools. Although the assessment is voluntary, we rely on school and student participation to ensure the results are complete and accurate. We cannot do that without the support of schools like yours.

By taking part in PISA, your school will have the unique opportunity to impact the bigger picture of education in the United States and across the world. PISA is administered worldwide in order to help participating school systems and countries understand their strengths and areas for improvement, with the ultimate goal of increasing both the quality and equity of education worldwide. As such, your school’s participation will help PISA fosters engagement among international education systems, allowing teachers, education professionals, and policy experts to compare shared experiences in the global learning community. By taking part in PISA, your school will have the unique opportunity to impact the bigger picture of education in the United States and across the world.

As a token of appreciation for participating, schools will receive $250, the school coordinator will receive $200, and students who participate will each receive $25 as well as 4 hours of volunteer service time

Page 10 – In the section, “What are the benefits to participating?” The section now reads:

Schools participating in PISA may also receive a school-level report (pending sample size and response rate requirements). The school report presents comparisons of your school’s average scores with the average scores of participating education systems including the United States. The report also provides comparisons of your school with other similar U.S. schools based on specific school-level characteristics. The school report is provided only to your school and any published reports from PISA 2022 will not identify participating schools.

Page 11 – minor edit to the text in the following sections:

How is the school questionnaire administered?

The school questionnaire is administered online from a secure website. The school questionnaire, which looks at school characteristics such as enrollment and school instructional practices, takes less than an hour to complete.

When will PISA be conducted?

PISA will be conducted between October 3 and November 25, 2022. Westat will work with schools to identify an assessment date convenient for your school in that time period.

What will happen with the collected data?

The data collected for PISA will be used to report on students’ knowledge and skills as group descriptions at the national level. All of the information provided by school staff and students may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). The U.S. Office of Management and Budget has approved this data collection under OMB# 1850-0755. Reports of the findings from PISA will not identify participating districts, schools, students, or individual staff. Individual responses will be combined with those from other participants to produce summary statistics and reports.

PISA 2022 Field Test Student FAQ

Page 12 - Under “What is PISA?”the phrase “mandatory schooling” was deleted. The sentence now reads:

The Program for International Student Assessment (PISA) is an international assessment of students nearing the end of mandatory schooling that measures how well students apply their knowledge and skills to in solving problems in mathematics, reading, mathematics, science, and financial literacy.

Under “Why should I participate in PISA?” revised the text for clarity and plainer language.

Why should I participate in PISA?

You are one of up to 52 students in your school selected to participate in PISA. Students selected for PISA have the unique opportunity to impact education in the United States and around the world. Your Pparticipation in PISA is voluntary, but by participating, you represent students like you across the country. each student who participates You will plays a vital role in ensuring the U.S. results are representative of all types of schools and education environments across the country—including public and private schools.

Plus, if you participate in PISA you will receive a certificate for 4-hours of volunteer service from the U.S. Department of Education.

• Represent students like you across the country; and

• Make an important contribution to this global effort to understand students’ ability by representing other students just like you across the United States.

Only 6,300 students will have the unique opportunity to participate in PISA across the United States. How often do you get the chance to represent your country? Participating in PISA is a national service. You can have an impact on the bigger picture of education in the United States and around the world.

Revised the first sentence in the section “What types of questions will I see on PISA?” to read:

PISA questions assess the knowledge and skills students you have learned, both in and out of school, based on situations you may encounter in real life.

Deleted “from me” in the header for the section “What is done with the information you collect from me?”

School PISA 2022 Letter – Private schools

Page 17 – Revised the last sentence of the first paragraph to read: Your school is one of about 290 300 public and private schools across the United States that has been randomly selected to take part in PISA in fall 2022.

Revised the first bulleted item to read:

• Private schools play an important role to the success of PISA. While participation in this study is voluntary, your school plays an important role in will ensureing that the U.S. sample is representative of the knowledge and skills of all students across the country.

PISA 2022 FAQ for school administrators – Private schools

Page 19 – Revised the second sentence of the first paragraph to read:

In PISA 2022, students in public and private schools across the United States will be assessed in mathematics, reading, science, financial literacy, or some combination of these subjects.

Page 19 – Moved the section, “What does participating in PISA involve?” under the section “What are the benefits to participating?”

The section “Why should my school and students participate?” was revised as follows:

Why should my school and students participate?

The participation of selected public and private schools in the United States is vital to ensuring an accurate representation of the overall population of students nearing the end of compulsory education across the country—including U.S. public and private schools. Although the assessment is voluntary, we rely on school and student participation to ensure the results are complete and accurate. We cannot do that without the support of private schools like yours.

By taking part in PISA, your school will have the unique opportunity to impact the bigger picture of education in the United States and across the world. PISA is administered worldwide in order to help participating school systems and countries understand their strengths and areas for improvement, with the ultimate goal of increasing both the quality and equity of education worldwide. As such, your school’s participation will help PISA fosters engagement among international education systems, allowing teachers, education professionals, and policy experts to compare shared experiences in the global learning community. By taking part in PISA, your school will have the unique opportunity to impact the bigger picture of education in the United States and across the world.

As a token of appreciation for participating, your school will receive $250, the school coordinator (a school staff person designated to communicate with PISA staff) will receive $200, and students who participate will each receive $25 as well as 4 hours of volunteer service time.

The section “What are the benefits of participating?” was revised to read:

Schools participating in PISA may also receive a school-level report (pending sample size and response rate requirements). The school report presents comparisons of your school’s average scores with the average scores of participating education systems including the United States. The report also provides comparisons of your school with other similar U.S. private schools. based on specific school-level characteristics . The school report is provided only to your school and any published reports from PISA 2022 will not identify participating schools.

Page 20 – minor edit to the text in the following sections:

How is the school questionnaire administered?

The school questionnaire is administered online from a secure website. The school questionnaire, which looks at school characteristics such as enrollment and school instructional practices, takes less than an hour to complete.

When will PISA be conducted?

PISA will be conducted between October 3 and November 25, 2022. Westat will work with schools to identify an assessment date convenient for your school in that time period.

What will happen with the collected data?

The data collected for PISA will be used to report on students’ knowledge and skills as group descriptions at the national level. All of the information provided by school staff and students may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). The U.S. Office of Management and Budget has approved this data collection under OMB# 1850-0755. Reports of the findings from PISA will not identify participating districts, schools, students, or individual staff. Individual responses will be combined with those from other participants to produce summary statistics and reports.

Student video

Pages 104-122: The current PISA 2022 field test video script is provided with screen shots. A link to the video is also provided on page 104. This video will be updated for the main study by removing mentions of the field test and adjusting any student counts to align with main study numbers. The updates will be completed in fall 2021 and included in an updated version with the main study data collection request in early 2022.