Part A PIRLS 2021 MS Recruitment & Field Test

The Progress in International Reading Literacy Study (PIRLS) is an international assessment of fourth-grade students’ achievement in reading. PIRLS reports on four benchmarks in reading achievement at grade 4 (Advanced, High, Medium, and Low) and on a variety of issues related to the education context for the students in the sample, including instructional practices, school resources, curriculum implementation, and learning supports outside of school. Since its inception in 2001, PIRLS has continued to assess students every 5 years (2001, 2006, 2011, and 2016), with the next PIRLS assessment, PIRLS 2021, being the fifth iteration of the study. Participation in this study by the United States at regular intervals provides data on student achievement and on current and past education policies and a comparison of U.S. education policies and student performance with those of the U.S. international counterparts. In PIRLS 2016, 58 education systems participated. The U.S. will participate in PIRLS 2021 to continue to monitor the progress of its students compared to that of other nations and to provide data on factors that may influence student achievement.

PIRLS is coordinated by the International Association for the Evaluation of Educational Achievement (IEA), an international collective of research organizations and government agencies that create the assessment framework, the assessment instrument, and background questionnaires. The IEA decides and agrees upon a common set of standards and procedures for collecting and reporting PIRLS data, and defines the studies’ timeline, all of which must be followed by all participating countries. As a result, PIRLS is able to provide a reliable and comparable measure of student skills in participating countries. In the U.S., the National Center for Education Statistics (NCES) conducts this study, with support from U.S. Department of Education contractors, and works with the IEA to ensure proper implementation of the study and adoption of practices in adherence to the IEA’s standards. Participation in PIRLS allows NCES to meet its mandate of acquiring and disseminating data on educational activities and student achievement in the U.S. compared with foreign nations [The Educational Sciences Reform Act of 2002 (ESRA 2002) 20 U.S.C. §9543].

Compared to PIRLS 2016, PIRLS 2021 differs in that it offers a redesigned digital format, referred to as digitalPIRLS to distinguish it from the traditional paper-and-pencil format, called paperPIRLS, which will still be available to countries. As the U.S. will be administering the digital version of PIRLS, in this submission all discussion of PIRLS 2021 refers to digitalPIRLS unless otherwise specified. The other difference to note is that in an effort to reduce the burden on schools and students, the assessment will be administered for one day only (rather than over two days as was done in 2016 when ePIRLS was administered for the first time independently). Given the transition to a digitally based assessment (DBA) for PIRLS, a bridge study between the two PIRLS formats will be conducted by the IEA during the main study. This design will require that approximately 1,500 students will be administered the paperPIRLS booklets, which will increase the sample size required during the main study.

In preparation for the PIRLS 2021 main study, all countries are asked to implement a field test in 2020 in order to evaluate new assessment items and background questions, to ensure practices that promote low exclusion rates, and to ensure that classroom and student sampling procedures proposed for the main study are successful. In selecting a school sample for this purpose, it is important to minimize the burden on schools, districts, and states, while also ensuring that the field test data are collected effectively. PIRLS staff will work to help respondents understand the study’s value relative to the burden imposed, and to ensure a high level of school participation.

Data collection for the field test in the U.S. will occur from March 1 through April 15, 2020 and will involve a sample of 45 public schools and about 1,650 students (selecting two classes from each school). The U.S. PIRLS 2021 main study will be conducted from March through June 2021 and will involve a nationally-representative sample of 290 schools and about 9,280 students.

The submission describing the overarching plan for all phases of the data collection, including the 2021 main study, and requesting approval for all activities, materials, and response burden related to the field test recruitment was approved in April 2019 with a change request in September 2019 (OMB# 1850-0645 v.11-12)¹. This request is to conduct all aspects of the field test and recruitment for the main study and, due to overlap in timing, carries over the approved respondent burden, procedures, and materials related to the PIRLS 2021 Field Test recruitment. The supporting statements Parts A and B describe all aspects of the study; Appendices A1 and A2 provide the field test and main study communication materials, respectively; Appendix B provides the non-response bias analysis plan; Appendix C1 lists changes from the PIRLS 2016 to the PIRLS 2021 U.S. instruments; and Appendix C2 provides the approved by IEA international versions of PIRLS 2021 background questionnaires with the U.S. adaptations proposed by NCES, which are subject to IEA approval. In the case the final, U.S. adapted versions of the PIRLS 2021 field test questionnaires will differ from those provided in this submission, the final versions will be submitted to OMB for approval as a change request in December 2019. A request for all aspects of the PIRLS 2021 main study, including data collection, will be submitted to OMB for approval, with an accompanying public comment period, in in March 2020. We expect the main study materials and procedures to be very similar to those used in the field test.

Because PIRLS is a collaborative effort among many parties, the U.S. must adhere to the international schedule set forth by the IEA, including the availability of draft and final questionnaires. In order to meet the international data collection schedule for the spring 2020 field test, recruitment activities are scheduled to begin in October 2019. Recruitment for the main study will begin in March of 2020 to align with recruitment for other NCES studies [e.g., the National Assessment of Education Progress (NAEP)], and for schools to put the assessment on their calendars.

A. Justification

A.1 Importance of Information

Benchmarking of U.S. student achievement against other countries continues to be of high interest to education policymakers and informs policy discussions of economic competitiveness and workforce and post-secondary preparedness. PIRLS provides a unique opportunity to compare U.S. students’ reading knowledge and skills at fourth grade with that of their peers in countries around the world. The continuation of U.S. participation allows for the study of past and current education policies that have shaped reading achievement over the past 20 years. Furthermore, participating countries are not only able to obtain information about students' knowledge and abilities, but also about the cultural environments, teaching practices, curriculum goals, and institutional arrangements that are associated with student achievement.

PIRLS complements what we learn from national assessments such as the National Assessment of Educational Progress (NAEP) by identifying the strengths and weaknesses of student reading achievement relative to participating countries around the world. It provides valuable benchmarking information about educational policies enacted in other countries and policies that could be applied to U.S. educational practices.

Based on earlier PIRLS data releases, it is likely that the results of these studies will draw great attention in the U.S. and elsewhere. It is therefore expected that PIRLS will contribute to ongoing national and international debates and efforts to improve reading learning and achievement.

A.2 Purposes and Uses of Data

PIRLS assesses reading knowledge and skills at grade 4. PIRLS is designed to align broadly with curricula in the participating countries. The results, therefore, suggest the degree to which students have learned concepts and skills likely to have been taught in school. PIRLS also collects background information on students, teachers, schools, curricula, and official education policies to allow cross-national comparison of educational contexts that may be related to student achievement.

Data compiled and collected from PIRLS 2021 allows for evidence-based decisions to be made for educational improvement. Each successive participation in PIRLS provides trend information about student achievement in reading relative to other countries, as well as indicators that show how this achievement relates to demographic and curricular, school, teacher, and student factors that provide the educational context for achievement. These high-quality, internationally comparative trend data are key in informing education policy discussions.

Through the participation in PIRLS and other international assessment programs, NCES is able to provide comparative indicators on student performance and school practices across countries in order to benchmark U.S. student performance, and to suggest hypotheses about the relationship between student performance and factors that may influence performance as well as areas in which students have strengths or weaknesses. The international studies identify differences among countries over time in instructional practices, school policies, and opportunity-to-learn that can inform discussions about how to improve students’ ability to read.

This collection of data is consistent with the NCES mandate, which specifies that "The purpose of the Center [NCES] shall be to collect and analyze and disseminate statistics and other information related to education in the U.S. and in other nations." [§406 of the General Education Provisions Act, as amended (20 U.S.C. 1221e-1)]. ESRA 2002 (20 U.S.C., §9543) also mandates that NCES shall collect, report, analyze, and disseminate statistical data related to education in the U.S. and in other nations, including acquiring and disseminating data on educational activities and student achievement in the U.S. compared with foreign nations.

PIRLS 2021 Components

The reading assessment is organized around a content dimension that specifies the subject matter to be assessed and a cognitive dimension that specifies the thinking processes to be assessed. PIRLS assesses two purposes for reading that fourth-grade students typically engage in: reading for literary experience and reading to acquire and use information. PIRLS also assesses four broad processes of comprehension predominantly used by fourth-grade readers: focusing on and retrieving explicitly stated information; making straightforward inferences; interpreting and integrating ideas and information; and evaluating and critiquing content and textual elements. The PIRLS 2021 framework is similar to 2016, but has been slightly updated to provide more specificity for item writers, and to better reflect current curricula in participating countries. There were no revisions to the content domains or cognitive domains, nor were there changes to the target percentages for either domain.

Assessment Instruments

To minimize burden and ensure broad subject-matter coverage, PIRLS will use a matrix sampling approach where the reading items are organized into sets for administration. For the bridge study to be conducted during the main study, the paper-and-pencil reading test will also use a matrix sampling approach with the items organized in a set of test booklets, with each student receiving only one booklet. This design is consistent with past practice.

Questionnaires

The background questionnaires for PIRLS 2021 were developed to address the issues outlined in the PIRLS context questionnaire framework. The U.S. is adapting the questions to fit the U.S. education context, as appropriate, including adding a few questions, such as about the race/ethnicity of students.

School Questionnaire. A representative from each participating school will be asked to provide information on reading resources, teacher availability and retention, principal leadership, school emphasis on academic success, school climate, and parental involvement in school activities. These questionnaires will be offered online, with a paper-and-pencil backup.

Teacher Questionnaire. Reading teachers of students in selected classes will be asked to complete a teacher questionnaire, which will include questions about teacher preparation and experience, reading instruction, instructional resources and technology, instructional time, instructional engagement, and classroom assessment. These questionnaires will be offered online, with a paper-and-pencil backup.

Student Questionnaire. Student information will be collected about home resources, motivation, self-concept, self-efficacy, and student characteristics such as gender and race/ethnicity. It should be administered to all students who have received parental permission to participate in PIRLS. Student questionnaire for digitalPIRLS will be administered via the student assessment player after the achievement assessment. For the PIRLS 2021 main study, students in the paperPIRLS bridge study will complete the student questionnaire as part of the paper booklet after the achievement assessment.

A.3 Improved Information Technology (Reduction of Burden)

Each participating nation is expected to adhere to the internationally prescribed design. In the U.S., the school and teacher questionnaires will be made available to school administrators and teachers online as the main mode of administration, with a paper-and-pencil backup to facilitate user preference for participation. The online questionnaires will be provided on the secure Westat data server. A communication website will be used for PIRLS 2021 during the field test and main study to provide a simple, single source of information to engage and maintain high levels of school involvement. This portal will be used throughout the assessment cycle to inform schools of their tasks and to provide them with easy access to information tailored for their anticipated needs. We plan to gather class and student lists from participating schools electronically using a secure electronic filing process. Electronic filing is an electronic system for submitting lists of student information, including student background information in school records. The electronic filing system provides advantageous features, such as efficiency and data quality checks.

The process of collecting teacher and student-teacher linkage information has been streamlined to improve user flexibility and efficiency. Teacher information is now collected through the MyPIRLS website on the “Submit Class List” page during the e-filing process. On that page, school coordinators are asked to enter a complete and current list of all of their school’s fourth-grade classes, including the student roster for each listed class as well as the name and email address of the associated reading teacher. Excel templates of the student list are posted on the “Submit Student List” page for the school coordinators to provide student information and link teachers to the students of each class listed on the “Submit Class List” page. This data collection procedure eliminates the need for the previously used Student-Teacher Linkage Form (STLF). Furthermore, information about the associated teachers is securely stored in Westat’s e-file system, which is connected to the database for the MyPIRLS website and is updated with teacher questionnaire participation status on a regular basis, making it easy and efficient for test administrators to track questionnaire status, and eliminating the need for the Teacher Tracking Form (TTF). This approach centralizes the information to be shared with the school coordinators in one secured online location through the MyPIRLS website, rather than having multiple forms the school coordinators have to verify and confirm at different stages of the data collection phase.

A.4 Efforts to Identify Duplication

In the U.S., reading achievement is systematically assessed at (1) the Federal level, where trend data have been collected on a fairly regular basis since 1971 through the National Assessment of Educational Progress (NAEP); (2) the state level, where data are routinely collected as part of state testing programs, though they vary across the states in terms of the frequency of testing, age/grades tested, and types of cognitive items administered; and (3) the district level, where data are collected through the use of commercially or locally developed standardized tests as well as tests developed in conjunction with the instructional programs used in schools. PIRLS 2021 does not duplicate these assessments.

PIRLS 2021 is part of a program of international cooperative studies of educational achievement supported and funded, in part, by the U.S. Department of Education. These studies represent U.S. participation in international studies involving 55 to over 70 countries each. As part of international cooperative studies, the U.S. must collect the same information at the same time as the other nations for purposes of making both valid international comparisons with other countries and with the previous PIRLS data. While some studies in the U.S. collect similar, though not identical, kinds of information (e.g., NAEP), the data from those studies cannot be substituted for the information collected in PIRLS in that they do not allow for comparisons outside the U.S. Furthermore, the data collected through PIRLS is based on a unique framework that is not shared by any other state, national, or international data collection effort. In order to participate in these international studies, the U.S. must agree to administer the same core instruments that are administered in other countries. Because the items measuring reading achievement have been developed with intensive international coordination, any changes to the instruments require international coordination and approval.

A.5 Minimizing Burden for Small Entities

The school samples for PIRLS contain small-, medium- and large-size schools, including private schools, selected based on probability proportionate to their size. All school sizes are needed to ensure an appropriate representation of each type of school in the selected sample of schools. The burden will be minimized wherever possible. In addition, staff will conduct all test administrations and will assist with parental notification, sampling, and other tasks as much as possible within each school. The assessment will be administered to intact classes to minimize disruption to school schedules.

A.6 Frequency of Data Collection

The field test data collection is scheduled for March 1 through April 15, 2020, and the main study data collection for March through June 2021. This schedule is prescribed by the international collective for PIRLS, and adherence to this schedule is necessary to establish consistency in survey operations among participating countries as well as to maintain trend lines.

A.7 Special Circumstances

None of the special circumstances identified in the Instructions for Supporting Statement apply to the PIRLS study.

A.8 Consultations outside NCES

Consultations outside NCES have been extensive and will continue throughout the life of the project. The IEA studies are developed as a cooperative enterprise involving all participating countries. An international panel of reading and measurement experts provide substantive and technical guidance for the study and National Research Coordinators participate in extensive discussions concerning the projects, usually with advice from the national subject matter and testing experts. In addition, the IEA convened a panel of reading experts from around the world to develop cognitive items.

The majority of the consultations (outside NCES) have involved the TIMSS & PIRLS International Study Center at Boston College in the U.S. Key to these ongoing consultations are: Dirk Hastedt (Executive Director of the IEA), Michael Martin, Ina V.S. Mullis, and Pierre Foy, all of whom have extensive experience in developing and operating international education surveys (especially related to PIRLS).

A.9 Payments or Gifts to Respondents

In order to achieve acceptable school response rates, schools have historically been offered incentives to participate and as a thank you for the time they invest in and the space they make available for the international assessments. High response rates are required by both IEA and NCES, and are difficult to achieve in school-based studies. The U.S. has historically had difficulties in achieving sufficient participation levels. As in PIRLS 2016, in PIRLS 2021 schools will be offered $200 for participation. Furthermore, to address challenges that may be encountered with securing school cooperation, we will utilize a second-tier incentive during the main study that will allow us to offer up to $800 to schools that are historically very difficult to recruit. This second-tier incentive will be offered only to:

Private schools in the original sample or substitute private schools (e.g., in ICILS 2018, only 40% of original private schools participated, with the overall rate of about 57% after substitutes were added);
Public schools that are selected to participate in NAEP 2021 and also selected for PIRLS 2021 (typically these schools participate only in NAEP because it is required, and refuse participation in PIRLS; there are about 40 such schools nationwide);
All substitute schools (typically, refusals by original schools occur in the mid to late fall prior to the assessment in the spring, at which point substitute schools are recruited; recruitment of substitutes is typically very challenging (less than 25% participate) because recruitment is late, after the school year has begun and school calendars are set; also, some states refuse to recruit schools this late in the school year); and
Refusing original schools if it is necessary to meet international participation requirements.

A similar second-tier recruitment strategy has been used in other international studies conducted by NCES. Most recently, in TIMSS 2019, we offered the second-tier incentive to the aforementioned groups. We were able to successfully recruit over 85% of eligible schools, which was the target recruitment percentage. We expect the extra incentive strategy to be successful in the PIRLS main study as well.

The school staff serving as School Coordinators will receive $100 for their time and effort in coordinating the PIRLS assessment. The School Coordinator serves a critical role in data collection, functioning as the central school contact, and facilitating arrangements for the assessments. They are asked to file class and student listing forms; arrange the date, time, and space for the assessment; and disseminate information to parents and students.

Consistent with prior administrations of PIRLS, as a token of appreciation for their participation, the students will receive a small gift valued at approximately $4. In PIRLS 2016, each participating student received a small sports wristwatch, a pencil imprinted with “USA”, and a certificate with their name thanking them for participating in and representing the United States in PIRLS 2016. Similarly priced items will be distributed to participating students for the PIRLS 2021 data collection. Some schools also offer recognition parties with pizza or other treats for students who participate; however, these are not reimbursed by NCES or Westat.

Teachers will be offered a $20 Amazon gift card for completing the PIRLS teacher questionnaire. Historically, participation is high among school administrators without offering incentives; therefore, no incentive will be offered for completion of the school administrator questionnaire.

A.10 Assurance of Confidentiality

Data security and confidentiality protection procedures have been put in place for PIRLS to ensure that Westat, the PIRLS contractor for the U.S. and its subcontractors, comply with all privacy requirements, including:

The statement of work of PIRLS contract;
Privacy Act of 1974 (5 U.S.C. §552a);
Family Educational and Privacy Act (FERPA) of 1974 (20 U.S.C. §1232(g));
Privacy Act Regulations (34 CFR Part 5b);
Computer Security Act of 1987;
U.S.A. Patriot Act of 2001 (P.L. 107-56);
Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9573);
Confidential Information Protect and Statistical Efficiency Act of 2002;
E-Government Act of 2002, Title V, Subtitle A;
Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151);
The U.S. Department of Education General Handbook for Information Technology Security General Support Systems and Major Applications Inventory Procedures (March 2005);
The U.S. Department of Education Incident Handling Procedures (February 2009);
The U.S. Department of Education, ACS Directive OM: 5-101, Contractor Employee Personnel Security Screenings;
NCES Statistical Standards; and
All new legislation that impacts the data collected through the inter-agency agreement for this study.

Furthermore, Westat will comply with the Department’s IT security policy requirements as set forth in the Handbook for Information Assurance Security Policy and related procedures and guidance, as well as IT security requirements in the Federal Information Security Management Act (FISMA), Federal Information Processing Standards (FIPS) publications, Office of Management and Budget (OMB) Circulars, and the National Institute of Standards and Technology (NIST) standards and guidance. All data products and publications will also adhere to the revised NCES Statistical Standards, as described at the website: http://nces.ed.gov/statprog/2012/.

The laws pertaining to the collection and use of personally identifiable information are clearly communicated in correspondence with states, districts, schools, teachers, students, and parents. Letters and information materials will be sent to parents and school administrators describing the study, its voluntary nature, and the extent to which respondents and their responses will be kept confidential (see copies in appendix A).

Letters and supporting materials will read:

The National Center for Education Statistics (NCES) is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543), and to collect students’ education records from educational agencies or institutions for the purpose of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3)(iii) and 99.35). All of the information [you provide/ provided by school staff and students] may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). In the United States, PIRLS is conducted by NCES, part of the U.S. Department of Education, and the data are being collected by Westat. The U.S. Office of Management and Budget has approved the data collection under OMB # 1850-0645.

The following statement will appear on the login page for PIRLS and the front cover of the printed questionnaires (the phrase “search existing data resources, gather the data needed” will not be included on the student questionnaire):

The National Center for Education Statistics (NCES) is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9543), and to collect students’ education records from educational agencies or institutions for the purpose of evaluating federally supported education programs under the Family Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3)(iii) and 99.35). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless such collection displays a valid OMB control number. The valid OMB control number for this voluntary information collection is 1850-0645. The time required to complete this information collection is estimated to average [XX] minutes per [respondent type], including the time to review instructions [, search existing data resources, gather the data needed,] and complete and review the information collection. If you have any comments or concerns regarding the accuracy of the time estimate(s), suggestions for improving the form, or questions about the status of your individual submission of this form, write directly to: Progress in International Reading Literacy Study (PIRLS), National Center for Education Statistics, Potomac Center Plaza, 550 12^th Street, SW, 4^th floor, Washington, DC 20202.

OMB No. 1850-0645, Approval Expires 04/30/2022.

The PIRLS 2021 confidentiality plan includes signing confidentiality agreements and notarized nondisclosure affidavits by all contractor and subcontractor personnel and field workers who will have access to individual identifiers. Also included in the plan is personnel training regarding the meaning of confidentiality, particularly as it relates to handling requests for information and providing assurance to respondents about the protection of their responses; controlled and protected access to computer files under the control of a single database manager; built-in safeguards concerning status monitoring and receipt control systems; and a secured and operator-manned in-house computing facility. Data files, accompanying the software, and documentation will be delivered to NCES at the end of the project. Neither names nor addresses will be included in any data file. In digitalPIRLS, students log into the automated assessment using non-identifying ID’s, and thus, the resulting data are collected and stored with only the non-identifying PIRLS assigned ID.

NCES understands the legal and ethical need to protect the privacy of the PIRLS respondents and has extensive experience in developing data files for release that meet the government’s requirements to protect individually identifiable data from disclosure. Westat will conduct a thorough disclosure analysis of the PIRLS 2021 data when preparing the data files for use by researchers, in compliance with ESRA 2002 (20 U.S.C. §9573). Schools with high disclosure risk will be identified and, to ensure that individuals may not be identified from the data files, a variety of masking strategies will be used, including swapping data and omitting key identification variables (i.e., school name and address) from both the public- and restricted-use files (though the restricted-use file will include an NCES school ID that can be linked to other NCES databases to identify a school); omitting key identification variables such as state or ZIP Code from the public-use file; and collapsing or developing categories for continuous variables to retain information for analytic purposes while preserving confidentiality in public-use files.

A.11 Sensitive Questions

The questionnaires do not have items considered to be of a sensitive nature.

A.12 Estimates of Burden

This package shows the estimated burden to respondents for all PIRLS 2021 activities, and requests approval for the burden to respondents for field test recruitment (demarcated in black font in Table A.1). For the field test, in order to assess the minimum required 1,400 students, we will sample 45 schools and 1,650 students (sampled from two randomly selected classes per school). For the main study, in order to assess the minimum required 5,000 students from 150 schools for the digitalPIRLS plus 1,500 students from 50 schools for the bridge study paperPIRLS, we will sample 290 schools and about 9,280 students. The time required for students to respond to the assessment (cognitive items) portion of the study and associated directions are shown in gray italicized font and are not included in the totals because they are not subject to the PRA. The burden estimates also include burden for: (1) contacting states, districts, schools, and parents in order to recruit for the PIRLS field test and main study, including: (a) sending recruitment letters to districts and schools selected for each study, (b) contacting and seeking research approvals from special handling districts, where applicable, and (c) notifying parents of sampled students about their participation in either study; and (2) collecting data and reviewing lists submitted by school coordinators. Burden estimates for the field test data collection and all aspects of the main study data are not requested at this time and are provided in Table A.1 for information purposes.

The district and school contact letters for the field test and main study are assumed to impose a small burden on all contacted parties, both those that refuse and those that agree to participate in the PIRLS studies. Thus the burden hours for this activity are based on the total number of districts and schools contacted rather than the total number agreeing to participate.

The special handling districts are those known to require completion of a research application before they will allow schools under their jurisdiction to participate in a study. Based on an initial assessment of previous PIRLS data collections, we estimate that there may be up to 10 special handling districts in the sample. Contacting special handling districts begins with updating district information based on what can be gleaned from online sources. Calls are then placed to verify the information about where to send the completed required research application forms, and, if necessary, to collect contact information for this process. During the call, an inquiry is also made about the amount of time the districts spend reviewing similar research applications. The estimated number of such districts represents those with particularly detailed application forms and lengthy processes for approval. This operation should begin in the spring of the year preceding the start of the data collection to allow sufficient time for special handling districts’ review processes. We will continue to work with these districts until we receive a final response from each district (approval or denial of request) up until March 1, 2020 for the field test and March 1, 2021 for the main study.

The total burden estimate for the main study and field test recruitment is based on 10 minutes for districts to read materials and respond, 20 minutes for schools to read materials and respond, 120 minutes for district IRB staff review, 60 minutes for district IRB panel approval, and 10 minutes for parents to read and review recruitment materials and notification forms. The total burden estimate for the main study and field test data collection is based on 40 minutes for each questionnaire (school administrator, teacher, and student) plus 240 for all school coordinator activities. The total student session is estimated to take up to 2.5 hours, including 10 minutes for instructions, two 40-minute assessment sessions, two 5-minute breaks following each assessment, the 40-minute questionnaire, and up to 10 minutes for students to be seated and refocused after breaks.

Based on the estimated hourly rates for principals/administrators, school coordinators, teachers, and parents of $46.85, $29.25, $29.25, and $24.34, respectively^², and the federal minimum wage of $7.25 as the hourly rate for students, and based on the estimated total of 3,692 burden hours for PIRLS field test recruitment, the associated total estimated respondent burden time cost is $80,798.

Table A.1. Burden estimates for PIRLS 2021 Field Test and Main Study.

Activity	Sample size	Expected response rate	Number of respondents	Number of responses	Per respondent (minutes)	Total burden (hours)
Field Test Recruitment
Contacting Districts	45	1	45	45	10	8
Contacting Schools	45	1	45	45	20	15
District IRB Staff Study Approval	5	1	5	5	120	10
District IRB Panel Study Approval	30	1	30	30	60	30
Parental notification	1,650	1	1,650	1,650	10	275
Total Field Test Recruitment			1,775	1,775		338
Field Test Data Collection
Student
Assessment digitalPIRLS	1,650	0.85	1,403	1,403	80	1,871
Directions	1,650	0.85	1,403	1,403	10	234
Student Questionnaire	1,650	0.85	1,403	1,403	40	936
School Staff
School Administrator	45	0.95	43	43	40	29
Teacher (2 per school)	90	0.95	86	86	40	57
School Coordinator	45	1	45	45	240	180
Total Field Test Data Collection			1,577	2,980		1,436
Total Field Test			3,352	4,755		1,774
Main Study Recruitment
Contacting Districts	290	1.00	290	290	10	49
School Recruitment (Original Schools)	290	0.70	203	203	20	68
School Recruitment (Replacement Schools)	80	0.50	40	40	20	14
District IRB Staff Study Approval	30	1.00	30	30	120	60
District IRB Panel Study Approval	180	1.00	180	180	60	180
Parental notification	11,600	0.80	9,280	9,280	10	1,547
Total Main Study Recruitment			10,023	10,023		1,918
Main Study Data Collection
Student
Assessment digitalPIRLS	7,280	0.85	6,188	6,188	80	8,251
Bridge study paperPIRLS assessment	2,000	0.85	1,700	1,700	80	2,267
Directions	9,280	0.85	7,888	7,888	10	1,315
Student Questionnaire	9,280	0.85	7,888	7,888	40	5,259
School Staff
School Administrator	243	0.95	231	231	40	154
Teacher (2 per school)	486	0.95	462	462	40	308
School Coordinator	243	1.00	243	243	240	972
Total Main Study Data Collection			8,824	16,712		8,008
Total Main Study			18,847	26,735		9,926
Total requested in this submission			13,375	14,778		3,692

Note: Burden in black font represents the activities for which burden approval is sought in this submission. The burden for the activities shown in gray font will be requested in future submissions. Burden for student assessments is shown italicized font and are not included in the burden total because cognitive assessments are not subject to PRA. The totals in the “Number of respondents” column do not include duplicate counts of respondents.

A.13 Total Annual Cost Burden

No cost to respondents is anticipated beyond the estimated burden cost described in Section A.12.

A.14 Annualized Cost to Federal Government

The cost to the Federal Government for conducting the PIRLS 2021 field test is estimated to be $1,832,621 over a 2-year period. All direct and indirect costs for this study are valued at an estimated $6,765,955 over five years, from March 2019 to March 2024.

Table A-2. Estimated costs for PIRLS 2021 field test and main study

Components with breakdown	Estimated costs
FIELD TEST (2020)
NCES salaries and expenses	101,400
Recruitment	410,146
Preparations (e.g., adapting instruments, sampling)	244,480
Data collection, scoring, and coding	1,076,595
Field Test Total (current package components)	$1,832,621
MAIN STUDY (2021)
NCES salaries and expenses	202,800
Recruitment (current package component)	469,443
Preparations (e.g., adapting instruments, sampling)	293,974
Data collection, scoring, and coding	3,206,494
Reporting and dissemination	760,623
Main Study Total	$4,933,334
Grand total for PIRLS 2021	$6,765,955
Total Request in this Submission (current package components)	$2,302,064

A.15 Program Changes or Adjustments

The apparent increase in burden time is due to the fact that this submission is for all aspects of the PIRLS 2021 field test plus recruitment for the main study, while the last approved burden was for the PIRLS 2021 field test recruitment only.

With regards to content, PIRLS 2021 differs from PIRLS 2016 in that PIRLS 2021 will be administered as a DBA (digitalPIRLS). Additionally, for the main study, some students will be administered digitalPIRLS and some students will be administered the booklet form of paperPIRLS for the purpose of the bridge study. Furthermore, PIRLS 2021 will be administered to students on one day, rather than on two as was done in 2016.

A.16 Plans for Tabulation and Publication

The PIRLS field test is designed to provide a statistical review of the performance of items on the cognitive assessment and questionnaires in preparation for the main study data collection.

Based on the data collected in the main study, the TIMSS & PIRLS International Study Center will prepare a report to be released in December 2022. As has been customary, NCES plans to also release a report at the same time as the international report is released, interpreting the results for the U.S. audience. NCES reports on initial data releases are generally limited to simple bivariate statistics. There are currently no plans to conduct complex statistical analyses of PIRLS 2021 dataset. An example of the past PIRLS report can be found at https://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2018017. In late 2022, the International Study Center will also prepare a technical report, describing the design and development of the assessment as well as the scaling procedures, weighting procedures, missing value imputation, and analyses. After the release of the international data, NCES plans to release the national data and an accompanying User’s Guide for the study.

Electronic versions of each publication are made available on the NCES website. Schedules for tabulation and publication of PIRLS 2021 results in the U.S. are dependent upon receiving data files from the international sponsoring organization. With this in mind, the expected data collection dates and a tentative reporting schedule are as follows:

Dates	Activity
August 2019—January 2020	Prepare data collection manuals, forms, assessment materials, questionnaires
October 2019—February 2020	Contact and gain cooperation of states, districts, and schools for field test
February 2020—March 2020	Sample classes and collect student lists
March 1, 2020—April 15, 2020	Collect field test data
May 15, 2020	Deliver raw data to international sponsoring organization
July 2020—August 2020	Review field test results
March 2020—February 2021	Prepare for the main study/recruit schools
March 2021—June 2021	Collect main study data
June 2021—September 2021	Deliver perturbed data to international sponsoring organization
February 2023	PIRLS 2021 International Database distributed by international sponsoring organization
December 2021—December 2023	Produce report

A.17 Display OMB Expiration Date

The OMB expiration date will be displayed on all data collection materials.

A.18 Exceptions to Certification Statement

No exceptions to the certifications are requested.

1 The materials that will be used in the 2021 main study will be based on the field test materials. Additionally, the v.11-12 submissions were designed to adequately justify the need for and overall practical utility of the full study and to present the overarching plan for all of the phases of the data collection, providing as much detail about the measures to be used as was available at the time of those submission. As part of the v.11 submission, NCES published a notice in the Federal Register allowing first a 60- and then a 30-day public comment period. For subsequent submissions covering the details of the field test data collection and all aspects of the main study, including this request, NCES will publish a notice in the Federal Register allowing an additional 30-day public comment period.

2 The average hourly earnings of principals/education administrators in the May 2017 National Occupational and Employment Wage Estimates sponsored by the Bureau of Labor Statistics (BLS) is $46.85, for school coordinators and teachers is $29.25, and of parents is $24.34. Where mean hourly wage was not provided, it was computed from the mean annual wage assuming 2,080 hours per year. The student wage is based on the federal minimum wage. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ data type: Occupation codes: Education Administrators, Elementary and Secondary Schools (11-9032), Education School Teachers, Except Special Education (25-2021), and all employees (00-0000); accessed on December 7, 2018.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Calvin Choi
File Modified	0000-00-00
File Created	2021-01-15

Part A PIRLS 2021 MS Recruitment & Field Test

Part A PIRLS 2021 MS Recruitment & Field Test.docx

Progress in International Reading Literacy Study (PIRLS 2021) Main Study Recruitment and Field Test

OMB: 1850-0645

TABLE OF CONTENTS