MEMORANDUM NATIONAL CENTER FOR EDUCATION STATISTICS
Institute of Education Sciences
United States Department of Education
Date: December 23, 2019
To: Robert Sivinski, OMB
Through: Kashka Kubzdela, NCES
From: Patrick Gonzales, NCES
Re: Program for International Student Assessment 2021 (PISA 2021) Main Study Recruitment and Field Test Change Request (OMB# 1850-0755 v.24)
The Program for International Student Assessments (PISA) is an international assessment of 15-year-olds which focuses on assessing students’ reading, mathematics, and science literacy. PISA was first administered in 2000 and is conducted every three years. The United States has participated in all of the previous cycles and is participating in 2021 in order to track trends and to compare the performance of U.S. students with that of students in other education systems. PISA 2021 is sponsored by the Organization for Economic Cooperation and Development (OECD). In the United States, PISA is conducted by the National Center for Education Statistics (NCES), within the U.S. Department of Education. In each administration of PISA, one of the subject areas (reading, mathematics, or science literacy) is the major domain and has the broadest content coverage, while the other two subjects are the minor domains. PISA emphasizes functional skills that students have acquired as they near the end of mandatory schooling (aged 15 years), and students’ knowledge and skills gained both in and out of school environments. PISA 2021 will focus on mathematics literacy as the major domain. Reading and science literacy will also be assessed as minor domains, with additional assessment of financial literacy. In addition to the cognitive assessments described above, PISA 2021 will include questionnaires administered to school principals and assessed students. To prepare for the main study in 2021, PISA countries will conduct a field test in the spring of 2020, primarily to evaluate newly developed assessment and questionnaire items but also to test the assessment operations. The PISA 2021 field test will be conducted in April-May 2020. The PISA 2021 main study recruitment will begin in October 2020 and data collection will be conducted October-November 2021. The request to conduct PISA 2021 main study recruitment and field test was approved in December 2019 (OMB# 1850-0755 v.23). This change request: (1) makes small edits and replaces outdated screenshots of the MyPISA project website with screenshots of the MyPISA system that is now ready for the Field Test data collection, scheduled to begin in Spring 2020; (2) replaces the 2018 PISA Main Study video with the 2021 PISA Field Test Video; and (3) updates the package to remove all references to this change request.
This request does not introduce changes to respondent burden or the cost to the federal government.
A detailed summary of the changes in this submission is provided below.
1. Nonsubstantive text edits and new screenshots for the MyPISA online information collection system to reflect changes made for the PISA 2021 Field Test.
Appendix A-1
Throughout the document: the addition of contact information to all screens and transcripts of screens. That text reads as follows:
If you need help accessing this website, please contact the PISA help desk by phone between 8 a.m. and 8 p.m. Eastern Standard Time at 1-888-638-2597 or email [email protected].
pp. 16, 20, 65, 69 – new screenshot that accurately reflects the birthdates for student participants for the PISA 2021 Field Test (change from “July 1, 2002 and June 30, 2003” to “January 1, 2004 and December 31, 2004”)
p. 72 & 73 – screenshot and text updated with new dates for OMB expiration
OMB
No. 1850-0755, Approval Expires xx/xx
12/31/2022
p. 74 – screenshot and explanatory text updated to align with previously provided text on p. A-1-75
This screen is the Welcome page for the MyPISA.us site. It provides
overview information about the survey and navigation to specific
areas of the site. For the field test,
references to the teacher list and the teacher FAQ will be removed.
The text of the page as it will appear for MyPISA 2021 is
shown on the following page.
pp. 75-76 – minor edits to the wording of the website
PISA is an
international assessment that measures student learning in science,
mathematics, science
reading, and for
the upcoming 2021 assessment,
financial literacy. The
assessment occurs every 3 years (2000, 2003, 2006, 2009, 2012, 2015
and 2018), and provides information about the knowledge and skills of
U.S. students in comparison with students in more than 80
countries and education systems.
PISA is intended to provide a measure of students' overall preparedness for the future, not just their academic achievement. Knowing how U.S. students perform on PISA provides us with valuable information on how our education system compares with education systems of the other countries with whom we collaborate and compete in the world economy.
PISA is sponsored by the Organization for
Economic Cooperation and Development (OECD), an intergovernmental
organization of industrialized countries. In the United States, PISA
is managed
and
conducted by the National Center
for Education Statistics (NCES)
in), within
the U.S. Department of Education,
and administered by Westat, a firm located in Rockville, MD.
For more specific information on the upcoming
PISA 2021 field
test
Field Test
data collection, please see the FAQs and
study brochure with timeline linked below.
p. 77 – new screenshot, easier to read
p. 80 – new screenshot that accurately reflects the birthdates for student participants for the PISA 2021 Field Test (change from “July 1, 2002 and June 30, 2003” to “January 1, 2004 and December 31, 2004”), along with updated explanatory text
This screen provides the E-file instructions and template and is
where the school coordinator will securely upload the completed
student list. The birth date range will
be changed to “born in 2004”.
p. 81 – reorganized text
How to Submit Your Student List
Student E-File Submission - Schools are able to submit student lists for sampling electronically through this website. Download the Excel Student E-File Template (below) and import the list of eligible students into the template for submission. The template contains all the necessary fields for sampling.
Student E-File Template
PISA has also developed other resource documents to use including step-by step instructions, quick checks for preparing and submitting lists, as well as an annotated e-file template. These documents provide information and answers to many common questions about the e–filing process.
E-filing Instructions
Student E-File Annotated Template
Student E-file Quick Checks
E-file you school’s student list
It is
preferred that you include column headers as the first row in your
E-File (see the template above for example). However, E-Files without
column headers will be accepted. If you cannot submit your student
data with this information in an Excel file, please call or email the
PISA help desk at [email protected] or
call 1-888-638-2597. These
documents, linked to below, provide information and answers to many
common questions about the e–filing process.
PISA
Student e-filing template
Student
E-file - Annotated Template
p. 82 – new screenshot, better quality image.
p. 85 – new screenshot and explanatory text, removing reference to the teacher survey
This page lists the activities of the PISA assessment team on
assessment day. References to
completion of the teacher questionnaire will be removed.
The text of the page as it will appear for MyPISA 2021 is shown
below.
p. 86 – new screenshot with links to PISA 2021 FT documents
This screen provides information about post-assessment activities
such as scoring of assessment data and reporting plans. If
the PISA 2018 results are available the links will be updated for
PISA 2018. The text of
the page as it will appear for MyPISA 2021 is shown on the following
page.
pp. 88-89 – new screenshot showing links to PISA 2021 Field Test Documents and deleting reference to Teacher forms, which are not used in the PISA 2021 Field Test
Download your school's Questionnaire Logon
Forms and Student and Teacher
Tracking Forms
Form here: Download
SC1
Download your school’s Session Attendance Forms here: Download SC2
Important
documents and letter templates for the
PISA
|
p. 90 – new “Important Websites” screenshot with links to updated PISA results and presentations.
Presentation for Students - PISA 2021 Field Test: Click this link to play the PISA 2021 Presentation for Students.
p. 92 – new screenshot with contact information. New text is seen below
Contact Us
PISA Test Administrator
No TA designated for your account.
[This field is filled once Test Administrators are hired and assigned to work areas]
Contact by Phone
Contact by Email
PISA Help Desk
Appendix A-2 – Although this document will be more thoroughly updated in a change request in May 2020 to reflect the full recruitment strategy for the Main Study, we did update this document with some new screenshots to better reflect the changes made for the Field Test.
pp. 23, 27, 68, 72 – new screenshot that accurately reflects the birthdates for student participants for the PISA 2021 Field Test (change from “July 1, 2002 and June 30, 2003” to “January 1, 2004 and December 31, 2004”)
p. 75 & 76 – screenshot and text updated with new dates for OMB expiration
OMB
No. 1850-0755, Approval Expires xx/xx
12/31/2022
p. 89, 91 – new screenshot with links to PISA 2021 FT documents
This screen provides information about post-assessment activities
such as scoring of assessment data and reporting plans. If
the PISA 2018 results are available the links will be updated for
PISA 2018. The text of
the page as it will appear for MyPISA 2021 is shown on the following
page.
p. 93 – new “Important Websites” screenshot with links to updated PISA results and presentations.
Presentation for Students - PISA 2021 Field Test: Click this link to play the PISA 2021 Presentation for Students.
p. 94 – new screenshot with contact information. New text is seen below
Contact Us
PISA Test Administrator
No TA designated for your account.
[This field is filled once Test Administrators are hired and assigned to work areas]
Contact by Phone
Contact by Email
PISA Help Desk
2. New Video for the PISA 2021 Field Test: changes begin on p. 93 of Appendix A-1. The full script, with screenshots, can be seen in Appendix A-1. Below is the full script of the video, with changes highlighted.
PISA 2021 Field Test Student Presentation Video
This presentation video is designed to
both inform students of the details and importance of the study and
to motivate them to participate and do their best. The presentation
video is planned to be approximately 12
9 minutes in length or less. The video provides the following
information:
an overview of what PISA is and its goals internationally;
describes what students need to do
demonstrations of how to work with PISA computer-based items; and
where to find more information about PISA.
For
reference, we have provided the PISA 2018 video and script in this
package. By December 2019, we will submit a change request to OMB to
augment this page with a link to the video and its script. The final
video will be captioned.
The
link to the PISA 2018 video is
https://www.dropbox.com/s/s96grwjmwmnczg8/PISA%202018%20With%20Audio%20Mix%204.19.18.mp4?dl=0
The link to the PISA 2021 field test video is: https://www.dropbox.com/s/x25d29nrsy602q5/PISA%202019%20FINE%20CUT.mp4?dl=0
The final video will be 508 compliant and will include subtitles.
PISA 2018
2021 Field Test Student Video Script
Welcome, and
thank you for attending this presentation on PISA, the Program for
International Student Assessment. Congratulations!
You are among a select few students from across the nation
that will join students from nearly
more than 80 countries and education systems around the world
participating in PISA. This presentation will provide some background
information and what you can expect on assessment day. To get started
let’s look at a brief overview of what
is PISA.
[OECD video segment:]
What is PISA? PISA is an acronym that stands for the Program for International Student Assessment. It's the brainchild of the OECD - and what's the OECD? It's another acronym that stands for the Organization for Economic Cooperation and Development. The OECD brings together 34 countries with the aim of developing better policies for better lives.
In the late-1990s, countries that are members of the OECD came up with the idea to measure whether 15 year-olds around the world are well-prepared to participate in society. We chose 15 year-olds, rather than 12 or 17 year olds, because most 15 year olds are about to complete their compulsory education.
Experts in the field of education from around the world work together to create a two-hour test that focuses on core subjects, like reading, mathematics, and science. Participating countries decided to administer this test every three years and to rotate the main focus of the test among the three core subjects.
All very well, but testing students is nothing new, so what's so special about PISA?
PISA surveys are designed to find out whether students can use what they have learned in school and apply that knowledge to real-life situations and problems. PISA is less interested in knowing whether the students can repeat - like parrots - what they have been taught in class. Rather, the survey is designed to find out whether, for example, students can use the reading skills they have learned at school to make sense of the information they find in a book, a newspaper, on a government form, or in an instruction manual.
But the point of PISA is not to tell each individual student how well he or she has mastered a set of skills. Instead, PISA results are analyzed and extrapolated to the national level.
Picture one student sitting at a desk in a classroom somewhere taking the PISA test. Now, zoom out as though you're on the space shuttle and you can see the entire country in which that student is sitting. That's what PISA does with its test results.
Adding on to what you just heard….
PISA is the world’s largest international education study with more than 500,000 students participating in each cycle.
It measures how well 15-year-old students apply what they’ve learned in and out of school to real-world problems in reading, science, mathematics, and financial literacy.
PISA provides information about how well students in the United States are performing in comparison to students in countries around the world.
So, which countries participate in the largest international education study in the world?
To understand how far-reaching PISA is, let’s take a look at this map.
You can think of PISA as an Olympics of the mind. Countries and education systems from every continent, except Antarctica, participate in PISA – in total of more than 80 countries and education systems participate in PISA.
You can
think of PISA as an Olympics of the mind.
The National Center for Education Statistics website provides
a list of each of the countries and education systems that are in
PISA this cycle and provides a historical look at participation.
You’ll notice that many countries participate in PISA every
cycle, while some countries participate in certain cycles but not
others. Countries like Argentina, Australia, Brazil, Canada, Denmark,
Finland, France, Germany, Ireland, Italy, Israel, Japan, Korea,
Malaysia, Mexico, the Netherlands, Norway, Spain, Sweden, the United
Kingdom and United States take part in PISA most every cycle. In
total, nearly 80 countries and education systems will participate in
PISA 2018 participating, and
provides a look at historical participation in PISA.
In the
United States, certain states participate as individual education
systems. North Carolina and Massachusetts participated in PISA 2015
as separate education systems.
Who participates in PISA within the
United States? PISA samples schools across the country to participate
take part in the assessment. Not every school in every state
is asked to participate, since PISA relies on a sample of students
and schools to represent all the students and schools like them
across the country.
In the United States, approximately
8 only about 3,000
students were scientifically sampled to
participate in selected for the
PISA 2018
2021 field test. The PISA 2021 field test, conducted in the spring
of 2020, is a small scale trial run of the PISA 2021 assessment.
On assessment day, you will be asked
questions in two or three subjects –
either -- math,
reading, science, math,
or financial literacy. The questions will be based on
real-life situations. Remember, PISA is about applying the knowledge
you have learned in and out of school, and not just memorizing facts.
In a few moments we will go over some
example show you a sample PISA
items.
item.
PISA is completely computer-based. Assessment questions will be multiple-choice and short-answer. There are no essays or extended writing responses.
After the main assessment, you will be
asked to complete questions about your background
attitudes and experiences in school, experience with financial
matters and your familiarity with information and communication
technology. This helps policymakers learn more about students’
experiences and attitudes and how they relate to students’
level of knowledge and skills.
Because no one student takes the entire assessment, unlike many tests you take, you will not get your own score on PISA.
PISA takes place at your school. You
will be excused from class to
during the PISA assessment. Now
let’s take a look at some example items from previous PISA
tests and what you can expect to see on assessment day.
Video of
PISA 2015 released items & assessment demo
On
assessment day, you will receive a logon form, your school and name
will be preprinted on the form along with your student ID and
password for accessing your individual assessment.
Enter
your student ID and password in the appropriate places to begin the
assessment, when instructed.
Let’s
Now, let’s take a look at an example interactive
science item from PISA
2015, Running in Hot Weather.
previous PISA tests, and what you can expect to see on assessment
day.
PISA items are presented in a layout in which the question, or description of the problem, is presented on the left side of the screen.
[highlight the left panel where question text is presented.]
and the information you will use to answer the question is provided on the right side of the screen.
[highlight the right panel where simulation is presented.]
Many of the science
PISA questions are designed to simulate experiments, letting
you manipulate different variables such
as temperature, water intake, and humidity,
or conditions to produce data that will help you answer
the question. determine, for
example, if a runner will experience dehydration or heat stroke.
[[Zoom in
and focus on question, defocus and darken rest of screen]]
In most
items, like this item, the question is presented on the left and the
simulation controls are located on the right. After reading the
prompt and the question, you determine that the effect of drinking
water during a run needs to be examined to figure out how it may
impact the runners risk of dehydration and heat stroke.
Drinking
water [[Circle drinking water]] is the variable that will need to be
manipulated.
Notice
the green star on the lower left of the screen [[point arrow]]. In
many simulations you must select rows of data from tables to support
your answer and receive full credit.
To answer
this question, “what would be the effect of drinking water
during the run on the runner’s risk of dehydration,” you
must first read the prompt to learn what conditions the runner is
facing. Set the other variables, ‘air temperature’ and
‘humidity’ to the conditions stated in the prompt –
35 degrees Celsius and 60 percent humidity.
[[Box in
Air humidity and air temp on right side, then set to 60 and 35
respectively]].
Next,
click run to get the results.
Next,
change the drinking water to ‘no’ and run again.
Now you
have all the data you need, in the table below, to determine the
effect of drinking water on the risk of a runner experiencing
dehydration or heatstroke.
[select
two rows of data] After you’ve run the simulation, select the
two rows of data that support your answer. You’ll notice that
water loss for a runner, when not drinking water, in the second row
is greater than the 2% threshold indicated above, and that by
drinking water, water loss is zero, as indicated in the first row.
Looking
at the risk of heat stroke threshold [[lighten and focus on
heatstroke in upper right]] …
…and
the body temperature data in the table, you can see that the risk of
heatstroke remains regardless of water intake.
Therefore,
the answer to this question is drinking water would reduce the risk
of dehydration but not heatstroke. Select the answer on the left….
…
and click the right arrow at the top to move onto the next question.
To answer questions, you will select an answer from a list of possible answers or enter a short a response.
At the top of the screen, there are other tools available to you to answer PISA questions, including:
[highlight the top bar of the screen.]
An indicator that shows your current progress in the assessment,
[highlight the green and white bars next to the PISA 2021]
A timer that shows you how much time you have left to work,
[highlight the circle]
Navigation buttons that take you to the next question or back to the previous question.
Let’s
take a look at some websites that provide additional PISA items from
previous cycles.
The
National Center for Education Statistics, NCES, which is responsible
for PISA in the United States, maintains a website with most of the
released items from PISA to date:
[[go to
http://nces.ed.gov/surveys/pisa/ start video at landing page and
clicking on PISA Released Assessment Items]]
Here
you will find paper-versions of items
[highlight the arrows]
And, for some items, if you see a question mark near the arrow buttons, there is additional information available to you in a Help Screen.
[highlight the question mark]
[Transition to the NCES PISA study site. No voice. See next screen.]
[Highlight the NCES PISA web address.]
The International studies website at NCES has questions that are similar to the ones you will take during the PISA test.
[Website address shrinks back and study page is clear]
There are a few computer items that can be shared publically. Assessment items that are scheduled to be used again cannot be shared publically because they are confidential, and doing so would jeopardize the results.
On the NCES released items page, you can click on the word version or PDF version of released items.
Let’s
click on the reading PDF to get an idea of what PISA reading items
look like [[open pdf of reading items]].
The PDF
is designed to be accessible and can be navigated through by clicking
on the bookmarks. Sticking with the running theme, let’s take a
look at ‘runners’ [[click on runners after opening PDF].
While
this is not a computer-based item, it is organized in the same way.
First
there is a passage of text, or stimulus. This one being about world
footballers’ shoes, …
[scroll
through questions]
…and
then 4 questions related to the text passage.
In the
computer based assessment, you will be able to flip pages and
navigate mock webpages. The format of the questions range from
multiple choice to short answer. Again, there are no essay questions.
[[Go to
questionnaires]]
In
addition to the assessment items, NCES website also contains general
information about PISA, and questionnaires from previous cycles.
Let’s
take a look at the student questionnaire that was administered in
2015.
The 2018
version of the questionnaire is very similar.
The
student questionnaire is designed to collect information about you,
and your experiences in school.
Another PISA resource with released
assessment items and recent PISA results is the OCED
OECD website.
[From the
NCES website, go to OECD international website
https://www.oecd.org/pisa/test/]
The OECD website for PISA includes interactive maps,
…released items,
…and other PISA information and videos.
Your
participation You play a
crucial role in PISA is critical
for the success of the
study. You PISA because you
are the most important part!
Each student who
participates takes
part represents thousands of students across the nation.
Your
participation can help improve the U.S. education system for future
students. By participating in this
assessment,
AND you
are also performing a national service. And
wait, there’s more…..
As a thank-you for participating:
You will receive a Certificate of Volunteer Service awarding you 4 hours of service time.
And
you You can also
mention your participation in PISA on future resumes and
college applications as an important assessment to which you
contributed.
There is a lot of information about PISA. It is most easily found on the PISA website at the National Center for Education Statistics.
Thank you for your time and participation in PISA. Your PISA School Coordinator or Test Administrator will now share with you the details of when PISA will be happening in your school.
3. Updates the package to remove all references to this change request and bring administrative text up-to-date.
Part A
(p.
3) Field test and main study recruitment materials, including letters
to state and district officials and school principals, and text for a
PISA field test brochure, summary of activities, and “Frequently
Asked Questions” are provided in Appendix A. Parental consent
letters and related materials for the field test are provided in
Appendix B (main study materials will be based on these, but will
reflect the main study design and components to be administered). The
final U.S.-adapted versions of the international field test
questionnaires are provided in Appendix C. By
December 2019, we will submit a change request to provide updated
materials, including new screenshots of the reprogrammed MyPISA
website and an updated video script for PISA 2021. (See Appendices
A-1 and A-2 for more details.)
Appendix A-1
(p. 2) Throughout
this document are screenshots that show the web content of MyPISA.us
for schools and school coordinators from the main study in PISA
20182021
field test, along with text for the
PISA 2021 field
test. The last section of this document,
“MyPISA project website for School Coordinators and
schools,” documents the contents of the website most
extensively, but MyPISA screenshots also appear throughout the
sections, “Instructions for School
Coordinators to E-file student lists,” and “PISA
2021 Field Test School Coordinator Handbook”. The
systems for the PISA 2021 main study are not yet active, but will not
change other than references to the current round of PISA and updates
to the specifics of the main study, such as the appropriate birth
year of students within the sample, the number of students to be
sampled, and removal of teacher information pertaining to teacher
participation in the questionnaire.
MyPISA.us is described in Part B of the main study package.
Updated
screenshots and, if necessary, any updated text will be provided in a
change request by December 2019, including an updated video script
for PISA 2021.
The systems for the PISA 2021 Field Test are active, but will need to change slightly to update references to the PISA Main Study, such as the appropriate birth year of students within the sample and the number of students to be sampled. These changes will be made to Appendix A-2 Main Study Recruitment Materials in change requests by May 2020.
(p. 71) The
following screenshots have been updated
for the PISA 2021 field test and show
the web content of MyPISA.us for schools and school coordinators from
the main study in PISA 2018 with updated
with text following
for each screen shown
below the screen. The systems
for the PISA 2021 field test are not yet active, but will not change
other than references to the current round of PISA and updates to the
specifics of the field test, such as number of students to be sampled
and removal of teacher information pertaining to teacher
participation in the questionnaire.
MyPISA.us is described in Part B of the main study package.
Updated
screenshots and, if necessary, any updated text will be provided in a
change request by December 2019. The text displayed below each
screenshot in this section is the text as it will appear on MyPISA.us
for PISA 2021.
Appendix A-2
(p. 2) Throughout
this document are screenshots that show the web content of MyPISA.us
for schools and school coordinators from the main
study field test in
PISA 2020
18,
along with text for the main study for
PISA 2021. The last section of this
document, “MyPISA project website for School
Coordinators and schools,” documents the contents of the
website most extensively, but MyPISA screenshots also appear
throughout the sections, “Instructions
for School Coordinators to E-file student lists,” and “PISA
2021 Main Study School Coordinator Handbook”. MyPISA.us
is described in Part B of the main study package. The systems for the
PISA 2021 main study
field test are active, but will not
need to change slightly to update other
than references
to the PISA main study and updates to the specifics of the main
study, such as the appropriate birth year of students within the
sample and the number of students to be sampled.
and removal of teacher information pertaining to teacher
participation in the questionnaire.
Updated
screenshots and, if necessary, any updated text will be provided in a
change request by December 2019.
Updated screenshots and, if necessary, any updated text will be
provided in a change request by May 2020. That change request will
also add the Main Study version of the 2021 PISA Field Test
recruitment video, as seen beginning on p. 93 of Appendix A-1, and
may include some updated versions of other materials currently found
in Appendix A-1 but not included here.
(p. 74) Updated
screenshots and, if necessary, any updated text will be provided in a
change request by December 2019
May 2020. The text displayed
below each screenshot in this section is the text as it will appear
on MyPISA.us for PISA 2021.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | ERSMCGILL |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |