Part B NAEP 2021

The possible universe of student respondents for NAEP 2021 is estimated to be 8.5 million at grades 4 and 8, attending the approximately 125,000 public and private elementary and secondary schools in 50 states and the District of Columbia, and including Bureau of Indian Education and Department of Defense Education Activity (DoDEA) Schools. Note that territories, including Puerto Rico, are not included in the national samples.

Respondents are selected according to student sampling procedures with these possible exclusions:

The student is identified as an English language learner (ELL), but is prevented from participation in NAEP, even with accommodations allowed in NAEP.
The student is identified as having a disability (SD) which prevents participation in NAEP, even with accommodations as allowed in NAEP, and has an Individualized Education Plan (IEP) or equivalent classification, such as a Section 504 plan.

Additional information regarding the classification of students is provided in Section B.2.b.

B.1.a. Sampling Procedures

To assess a representative sample of students, the process begins by identifying a sample of schools with student populations that reflect the varying demographics of a specific jurisdiction, be it the nation, a state, or a district. Within each selected school, students are chosen at random to participate and each has the same chance of being chosen, regardless of socio-economic status, disability, status as an English language learner, or any other factors. Selecting schools that are representative helps ensure that the student sample is representative.

The following are characteristic features of NAEP sample designs:

for state-level assessments, approximately equal sample sizes (2,000–3,000 assessed students) from each participating state’s^¹ public schools;
for district-level assessments, sample sizes of approximately 1,000–2,000 from each participating district’s public schools;
sample sizes of approximately 6,000–20,000 for national-only operational subjects, depending on the size of the item pool;^²
samples sizes of approximately 3,000–12,000 for pilot assessments, depending on the size of the item pool;^³ and
in each school, some students to be assessed in each subject (students may take one or two subjects).

Additional information about the sampling procedures used in NAEP can be found in the technical documentation at http://nces.ed.gov/nationsreportcard/tdw/sample_design/. Note, while the latest documentation for main NAEP that has been published (as of the drafting of this document) is from 2013, the procedures have essentially remained the same. A summary of the sampling procedures is included below. Additional details (taken from the main NAEP 2013 procedures on the technical documentation website) can be found in Appendix G (NAEP 2013 Sample Design).

As in the past, NAEP samples are based on multistage designs. For the national samples, a two- or three-stage design is used. If a three-stage design is used, the first stage is the selection of primary sampling units (PSUs), which are individual counties or groups of contiguous counties. The next stage is the selection of schools (within PSUs, when a three-stage design is used) and the final stage is the selection of students within schools. The national samples have sufficient schools and students to yield results for public schools, private schools, each of the four Census Regions of the country, as well as gender, race, degree of urbanization of school location, and participation in the National School Lunch Program (NSLP).

The following steps are used to select a sample of public schools and students in a year when NAEP reports state-level results. Private schools are not included in a state-level sample, which focuses solely on public schools.

Generate a sampling frame.
For sampling frames, NAEP uses the most current versions of the NCES Common Core of Data (CCD; public schools) and Private School Universe Survey (PSS; private schools) files. In addition, to address the fact that the CCD file does not necessarily include the most recent changes to schools by the time of the assessment, NAEP also conducts a survey of NAEP State Coordinators to check for additional new schools in a sample of public school districts.
Classify schools into groups.
Using the list, schools are classified into groups, first by type of location and then by the race/ethnicity classification within those locations. This step takes into account the distribution of schools and students across rural, suburban, and urban areas in each state, and the diversity of the student population at each school.
Within each group, order schools by a measure related to student achievement.
Within each group, schools are sorted by student achievement to ensure that schools with varying levels of student achievement are represented in the NAEP sample. This is done using school-level results on state achievement tests. In a few cases where recent achievement data are not available, schools are sorted by the median household income for the area where the school is located.
Assign a measure of size to all schools.
All schools on the list are assigned a measure of size. A school’s measure of size is based on the size of its enrollment in relation to the size of the state’s student population at the selected grade-level. Larger schools have a larger measure of size as they represent a larger proportion of the state’s student population. This step ensures that students from schools of different sizes are appropriately represented in the sample.
Select the school sample.
After schools are assigned a measure of size and grouped on an ordered list based on the characteristics that are referred to in previous steps, the sample is selected using stratified systematic sampling with probability proportional to the measure of size using a sampling interval. This procedure ensures that each school has the required selection probability. By proceeding systematically throughout the entire list, schools of different sizes and varying demographics are selected, and a representative sample of students will be chosen for the assessment. Additional details regarding the selection of the school sample is included in the technical documentation (https://nces.ed.gov/nationsreportcard/tdw/sample_design/2013/sample_design_for_the_2013_state_assessment.aspx).
Confirm school eligibility.
The list of schools selected to participate is sent to each state to verify that the school is eligible for participation. Some factors that would make a school ineligible include schools that have closed or if the grade span has changed so that a grade level or age assessed by NAEP is no longer in the school. Eligibility counts are included in the technical documentation (https://nces.ed.gov/nationsreportcard/tdw/sample_design/2013/eligible_schools_sampled_for_the_2013_state_assessment.aspx). Information on response rates can be found in Section B.3.b.
Select students to participate in NAEP.
School principals are notified that their schools have been chosen to participate in NAEP. Within each sampled school, a systematic sample of students is selected with equal probability from a complete list of students at the grade or age to be assessed.

NAEP alternates between national-level administration years (i.e., even years) and state-level administration years that include one or more assessments that support national, state-by-state, and certain urban districts reporting (i.e., odd years). For assessments where results are reported at the national, state, and urban district (TUDA) levels, a single sample of public school students is selected and used for reporting at each level. That is, a student who is sampled from a school located in a TUDA district contributes to the estimates at each of the district, state, and national levels. For assessments where results are reported at the national level, but not for states and districts, schools are sampled from across the United States, without any oversampling of particular states or districts.

The process for private school selection is similar to the public school selection process but depends on the U.S. Department of Education’s private education system databases to create the initial list of all known private schools. Private schools are sampled to be representative of private schools nationwide. The results for private schools are not included in state-level results which are solely focused on public schools.

NAEP yearly sample design plans are not available until the spring of the year preceding the assessments. For this clearance submission, we have included the 2019 sample design memorandum (see Appendix C) which detail the specific sampling procedures used for the 2019 assessments. The 2021 sampling memo will be added in an amendment by April 2020.

Additional information about the sampling procedures used in NAEP can be found in the technical documentation at http://nces.ed.gov/nationsreportcard/tdw/sample_design/.

B.1.b. Weighting Procedures

Since each selected school that participates in the assessment effort and each student assessed constitutes only a portion of the full population of interest, weights are applied to both schools and students. The weights permit valid inferences to be drawn from the student samples about the respective populations from which they were drawn and, most importantly, ensure that the results of the assessments are fully representative of the target populations.

Additional information about the weighting procedures used in NAEP can be found in the technical documentation at http://nces.ed.gov/nationsreportcard/tdw/weighting/. Note, while the latest documentation that has been published (as of the drafting of this document) is from 2013, the procedures have essentially remained the same. A summary of the sampling procedures is included below. Additional details (taken from the main NAEP 2013 procedures on the technical documentation website) can be found in Appendix G (NAEP 2013 Sample Design).

The final weights assigned to each student as a result of the estimation procedures are the product of the following steps (which are described in additional detail below):

assignment of a “base” weight, the reciprocal of the overall initial probability of selection;
adjustment of the school base weights to reduce extreme variability, arising from special circumstance;
adjustments for school and student nonresponse;
adjustment (if needed) to reflect assignment to a specified assessment subject; and
adjustment of the student weights in state samples so that estimates for key student-level characteristics were in agreement across assessments in different subjects.

School base weights are assigned separately by grade or age and, as noted, are the reciprocal of the school’s probability of selection for that grade or age level.

Each sampled student receives a student base weight, whether or not the student participated in the assessment process. The base weight reflects the number of students that the sampled student represents in the population of interest. The sum of the student base weights for a given subgroup provides an estimate of the total number of students in that subgroup.

Since nonresponse is unavoidable in any survey of a human population, a weighting adjustment is introduced to compensate for the loss of sample data and to improve the precision of the assessment estimates. Nonresponse adjustments are applied at both the school and the student levels; the weights of responding schools are adjusted to reflect the nonresponding schools, and the weights of responding students, in turn, receive an adjustment to account for nonresponding students. School nonresponse adjustment cells are formed in part by geography (state or TUDA for state samples and census division for national samples), urbanicity, and race/ethnicity. Student nonresponse adjustment cells are formed in part by SD/ELL status, school nonresponse cell, age, gender, and race/ethnicity.

The complexity of the sample selection process as well as the variations in school enrollment can result in extremely large weights for both schools and students. Since unusually large weights are likely to produce large sampling variances for statistics of interest, and especially so when the large weights are associated with sample cases reflective of rare or atypical characteristics, such weights usually undergo an adjustment procedure that “trims” or reduces extreme weights. Again, the motivation is to improve the precision of the survey estimates. The student weight trimming procedure uses a multiple median rule to detect excessively large student weights.

Weighted estimates of population totals for student-level subgroups for a given grade or age will vary across subjects even though the student samples for each subject generally come from the same schools. These differences are the result of sampling error associated with the random assignment of subjects to students through a process known as spiraling. For state assessments, in particular, any difference in demographic estimates between subjects, no matter how small, may raise concerns about data quality. To remove these random differences and potential data quality concerns, a new step was added to the NAEP weighting procedure starting in 2009. This step adjusts the student weights in such a way that the weighted sums of population totals for specific subgroups are the same across all subjects. It was implemented using a raking procedure and applied only to state-level assessments.

Estimates of the sampling variance of statistics derived through the assessment effort are developed through a replication method known as “jackknife.” This process of replication involves the repeated selection of portions of the sample (replicates). A separate set of weights is produced for each replicate, using the same weighting procedures as for the full sample. The replicate weights, in turn, are used to produce estimates for each replicate (replicate estimates). The variability among the calculated replicate estimates is then used to obtain the variance of the full-sample estimate.

Additional information about the weighting procedures used in NAEP can be found in the technical documentation at http://nces.ed.gov/nationsreportcard/tdw/weighting/.

B.2. Procedures for Collection of Information

B.2.a Recruitment of Schools

Note: Appendices D1 and D2 contain sample communication materials from 2019. The final versions of the 2021 communication and recruitment materials and their Spanish translations will be submitted with future amendments once the final versions are available. The targeted submission dates are no later than April 2020, June 2020, and October 2020. (See pp. 5-6 of Part A for more detail on the amendment schedule.)

Once the sample of schools is selected for the 2021 NAEP administration the NAEP State Coordinator and NAEP field staff typically follow a standard set of procedures for securing the participation of public and nonpublic schools. The process includes:

sending initial contact letters to chief state school and testing officers (for a sample from 2019, see Appendix D2-15 for the letter and Appendices D2-13 and D2-14 for the included information);
sending a notice to the district superintendents of which and how many schools were selected for NAEP from their district (for a sample from 2019 see Appendix D2-3 for the letter and Appendix D2-1 for the included information);
sending a notice of each school’s selection for NAEP to the principal or other administrative official, along with an assessment information packet containing introductory information and materials (for a sample from 2019, see Appendix D2-4 for the letter and Appendix D2-2 for the included information);
sending a notice with each school’s NAEP assessment date to the principal or other administrative official, along with additional assessment information (for a sample from 2019, see Appendix D2-7 for the letter and Appendices D2-9 [public schools] and D2-8 [private schools] for the included information);
sending a letter to each school’s principal with instructions for assigning a school coordinator (for a sample from 2019, see Appendix D2-5); and
sending information to each school coordinator regarding his/her role (for a sample from 2019, see Appendix D2-6 for the letter and Appendices D1-2 [public schools] and D1-3 [private schools] for the brochure describing the role).

B.2.b School Coordinator Responsibilities

The school coordinators are responsible for preparing for the NAEP assessment in the school using the MyNAEP system, which is an online secure site that provides participating schools with a convenient way to prepare for the upcoming assessment. MyNAEP serves as the primary resource and action center throughout the assessment process. The secure MyNAEP system is used for all special studies. The site also offers school coordinators an electronic way to prepare for the assessment at their own pace. The NAEP field representative will schedule an initial call in December to pre-review the major areas of the MyNAEP system with the school coordinator. The content of the 2019 MyNAEP system is provided in Appendix I; the 2021 version of MyNAEP will be provided in future amendments targeted for submission in April 2020, June 2020, and October 2020. (See pp. 5-6 of Part A for more detail on the amendment schedule.)

The MyNAEP menu is a virtual checklist of all activities that school coordinators will need to complete throughout the school year. The following describes the different sections and activities that need to be completed, and the purpose and timeframe for each.

Tasks: Register for the MyNAEP website and provide school contact information and school characteristics, including student enrollment for the selected grade, charter school status, and important dates.
Purpose: Gain access to the secure MyNAEP website as the designated school coordinator and ensure that NAEP has the most up-to-date information about the school.
Timeline: August and October 2020.

Submit Student List/Sample

Tasks: NAEP collects a list of all students in the selected grade for each school. The school submits an Excel file with all students and their demographic data (see Appendix H for a sample from 2019). Note, as described in Section A.12, the school coordinator is only responsible for this task if the state coordinator has not previously submitted the student list for sampling. As such, only a portion of the school coordinators are responsible for this task.
Purpose: Draw a representative sample of students from the school to participate in the NAEP assessments. Ensure all students have an opportunity to be sampled.
Timeline: October and November 2020.

Review and Verify List of Students Selected for NAEP

Tasks: Review demographic data to make sure they are correct and add any missing demographic data. School coordinators will be asked to review and verify student information and also to indicate whether students were displaced from a natural disaster.
Purpose: Demographic data are used for reporting results of student groups in The Nation’s Report Card.
Timeline: December 2020 and January 2021.

Complete SD/ELL Student Information

Tasks: Determine how students participate in NAEP (i.e., without accommodations, with accommodations, or do not test). Provide the Individuals with Disabilities Education Act (IDEA) disability status, English proficiency, primary language, grade- or age-level performance, and accommodations, using the state-specific NAEP inclusion policies (see Appendices D1-5 and D1-6 for samples of the NAEP 2019 SD and ELL inclusion policies, which are customized by the NAEP State Coordinators).
Purpose: Make sure students have appropriate supports to access the NAEP assessment.
Timeline: December 2020 and January 2021.

Notify Parents

Tasks: Download and customize the parent notification letter (for NAEP samples from 2019, see Appendix D2-12 [public schools], and D2-11 [private schools]), upload the customized letter to the system, and certify the date parents were notified. A translation notice is available to accompany the parent notification letter in instances where parents do not speak English or Spanish (for sample from NAEP 2019, see Appendix D1-7-ML).

Purpose: Ensure that parents/legal guardians are notified of their student’s selection to participate in NAEP, which is a requirement of the Reauthorized Elementary and Secondary Education Act (ESEA).⁴
Timeline: December 2020 and January 2021.

Manage Questionnaires

Tasks: For the main NAEP administration only, identify respondents for school and teacher questionnaires, send respondents links to online questionnaires, and monitor completion of questionnaires. Distribute information about NAEP to teachers (see Appendix D1-4 for a sample from 2019)
Purpose: Results are used to provide contextual data from schools and teachers in The Nation’s Report Card.
Timeline: December 2020 and January 2021.

Update Student List

Tasks: Identify any newly enrolled students since the original list of students was provided in the fall. Upload a current list of students via Excel or review original list and add newly enrolled students.
Purpose: Ensures all students have an opportunity to be sampled so NAEP can assess a representative sample of students.
Timeline: January 2021.

Plan for Assessment Day and Encourage Participation

Tasks: Determine assessment session times and locations, share cell phone policy to ensure security of NAEP items, and make a plan to encourage student participation.
Purpose: Ensure that the school is prepared for a successful administration of NAEP.
Timeline: December 2020 and January 2021.

Support Assessment Day Activities

Tasks: Print resources to notify students and teachers.
Purpose: Ensure students arrive at assessment location prepared and on time.
Timeline: One week prior to assessment date.

Before the assessment, the NAEP field representative will hold a Pre-assessment Review Call with the school coordinator to review the status of the completion of the tasks in the MyNAEP system, answer any questions, and review assessment day procedures.

As part of the ongoing quality control of the assessment process, schools will be asked an additional follow-up survey. Survey questions solicit feedback on pre-assessment, assessment, and procedural processes. The post-assessment follow-up survey from 2019 is included in Appendix E. The 2021 version of the survey will include fewer survey questions in an effort to streamline the data collection process.

The final school coordinator responsibility occurs at the end of the school year during which he or she securely destroys any documents with student identifying information.

B.2.c Administration Procedures

Trained NAEP field staff will set up and administer the assessment and provide all necessary equipment and assessment materials to the school, including tablets with an attached keyboard, stylus, earbuds, and, for some subjects, mouse for the digitally based assessments (DBA). Internet access is not required for DBA. NAEP field staff will pack up the equipment and leave the space as they found it.

The traditional NAEP design assesses each student in 60-minutes for one cognitive subject. In 2021, NAEP will begin to transition to a design in which students will take additional set(s) of cognitive items, from another subject area. This design will allow more information to be collected from each individual student, thereby reducing the number of overall students (and, thus, schools) that are required. The two different designs will be administered in different schools. Given that 2021 is the initial transition year, this design will only be implemented in reading and mathematics; U.S. history and civics will be assessed via the traditional design of 60-mintues of cognitive items in one subject. In schools in which students are assessed for 60 minutes in one subject are typically held in sessions of approximately 25 students, with two sessions conducted sequentially during the school day, although additional concurrent sessions may be required. In schools in which students are assessed for 90 minutes in two subjects are held in sessions of approximately 40 students, typically with one session per school, but possibly multiple sessions held concurrently or sequentially in the schools.

The field staff use scripts and carefully timed sections to administer the assessment. In addition, many of the instructions are provided on the tablets from the assessment system for DBA assessments.

B.3. Methods to Secure Cooperation, Maximize Response Rates, and Deal with Nonresponse

Schools within each state will be selected and the chief state school officer and the NAEP State Coordinator will be asked to solicit their cooperation. Since states and school districts receiving Title I funds are required to participate in the main NAEP reading and mathematics assessments (grades 4 and 8) under the National Assessment of Educational Progress Authorization Act, NAEP response rates have improved for these assessments. An area that has typically had lower response rates in NAEP is private schools. As such, NCES has created specialized materials targeted at this audience:

Videos and additional information on the NAEP website for schools, students, parents, and teachers (see http://nces.ed.gov/nationsreportcard/about/schools.aspx).
Additional brochure and resources targeting private schools, includes NAEP in Your Private School (see Appendix D2-8 for a sample from 2019), and a webpage dedicated just to private schools (http://nces.ed.gov/nationsreportcard/about/nonpublicschools.aspx).

B.3.a. Methods to Maximize Response Rate

There are four main areas that can be focused on in order to maximize completion rates: (1) early distribution of information and materials; (2) effective communication with school personnel; (3) efforts to encourage student participation; and (4) efforts made by field staff to avoid refusals and to convert initial refusals to cooperating schools.

Early Distribution of Information and Materials

Over the years, feedback from schools and states had indicated that notification of a school’s selection in the NAEP sample earlier rather than later is beneficial to the school for planning purposes and improves school response rate. As such, NAEP generally notifies schools of selection in May of the year prior to the assessment. In addition, to facilitate the school coordinators’ completion of the tasks associated with the administration, the MyNAEP system is available to the school coordinators approximately 6–7 weeks before the administration window begins.

Effective Communication with School Staff

The participation of schools can be increased by effectively communicating information about NAEP, including what NAEP measures, the various assessment components, why it is important that schools, students, and teachers participate, and the role of the school staff. Effective communication materials from the State Coordinator and the field staff (as described in Section B.2.a) will help maximize the participation of schools. In addition, an intuitive and easy-to-use MyNAEP system (as described in Section B.2.b) will help ensure that the school coordinator’s experience is positive.

In addition, NCES may thank school staff and the principal for their participation in NAEP (see Appendix D2-16 for a sample from 2019).

Encouraging Student Participation

Previous feedback from school administrators has shown that students respond more positively to the assessment when they know the assessment has the support of the school administration. Therefore, the field staff will encourage the school coordinator to make efforts to encourage students to do their best, including having the principal introduce the assessment. In addition, field staff will suggest to the school coordinator that grade 8 schools may want to issue community service credits for participating.

Avoiding Refusals and Converting Initial School Refusals

Field staff will be trained in methods to maximize school participation, which will include being flexible in the assessment scheduling, following up with the school coordinators, and scheduling in-person preparation meetings, at the school coordinator’s request.

B.3.b. Statistical Approaches to Nonresponse

Not all of the students in the sample will respond. Some will be unavailable during the sample time period because of absenteeism or other reasons. If a student decides not to participate, the action will be recorded, but no steps will be taken to obtain participation. The NAEP response rates follow AAPOR (American Association for Public Opinion Research) guidelines. Response rates, in percentages, from the 2017 NAEP DBA assessment are shown below. Response rates from 2017 remained similar to those from 2015. Previous years’ response rates can be found in the technical documentation (see for example, https://nces.ed.gov/nationsreportcard/tdw/sample_design/2012/2012_samp_econ_resp_school.aspx).

	Grade 4	Grade 8
Student Response Rates	94	92
School Response Rates
Public Schools	100	100
Private Schools	61	60

Note: The public school response rate for grades 4 and 8 were rounded to 100, but were actually slightly lower (i.e., 99.65 and 99.51 respectively).

We are working to increase engagement of private school organization leaders in recruitment efforts and requesting customized endorsement letters from these organizations (Appendix D2-10 for a sample from 2019). We have also expanded outreach efforts to schools to promote the use of NAEP data tools to highlight the value of NAEP data to private schools. Furthermore, efforts are underway to develop a customized dashboard for private schools on The Nation’s Report Card site.

NCES and the National Assessment Governing Board have established participation rate standards that states and jurisdictions are required to meet in order to have their results published. Beginning in 2003, if a state’s school response rate is below 85 percent, the results will not be published by NAEP, regardless of the response rate after substitution (see https://nces.ed.gov/nationsreportcard/about/participrates.aspx and https://www.nagb.org/content/nagb/assets/documents/policies/samplingpolicy1.pdf).

B.4. Pilot Testing and Data Uses

Pilot testing of cognitive and non-cognitive items is carried out in all subject areas. The purpose of pilot testing is to obtain information regarding clarity, difficulty levels, timing, and feasibility of items and conditions. In addition to ensuring that items measure what is intended, the data collected from pilot tests serve as the basis for selecting the most effective items and data collection procedures for the subsequent operational assessments. Pilot testing is a cost-effective means for revising and selecting items prior to an operational data collection because the items are administered to a small, nationally representative sample of students and data are gathered about performance that crosses the spectrum of student achievement. Items that do not work well can be dropped or modified before the operational administration.

Prior to pilot testing, many new items are pre-tested with small groups of sample participants (cleared under the NCES pretesting generic clearance agreement; OMB #1850-0803). All non-cognitive items undergo one-on-one cognitive interviews, which is useful for identifying questionnaire and procedural problems before larger scale pilot testing is undertaken. Select cognitive items also undergo pre-pilot testing, such as item tryouts or cognitive interviews, in order to test out new item types or formats, or challenging content. In addition, usability testing is conducted on new technologies and technology-based platforms and instruments.

B.5. Consultants on NAEP Design

ETS, Fulcrum, Westat, and NCES staff have collaborated on aspects of the design. The primary persons responsible from NCES are: Peggy Carr, Patricia Etienne, Holly Spurlock, and William Tirre; from ETS: Jay Campbell and Amy Dresher; and from Westat: Keith Rust and Greg Binzer. In addition, the NAEP Design and Analysis Committee, the NAEP Validity Studies Panel, and the NAEP Quality Assurance Technical Panel members (see Appendices A-1 through A-3) have also contributed to NAEP designs on an on-going basis.

1 Participating states vary depending on the subject and grade assessed, but may include the 50 states, the District of Columbia, the Department of Defense Education Activity, and (for mathematics assessments only) Puerto Rico.

2 NAEP IRT scaling requires a minimum sample size of 1,500-2,000 students per item in order to estimate stable item parameters. Therefore, national assessments with larger item pools have larger samples.

3 NAEP IRT scaling is conducted for most pilot assessments, requiring a minimum of 1,500-2,000 students per item in order to estimate stable item parameters. Therefore, pilot assessments with larger item pools have larger samples.

4 Please note that parents/legal guardians are required to receive notification of student participation but NAEP does not require explicit parental consent (by law, parents/guardians of students selected to participate in NAEP must be notified in writing of their child’s selection prior to the administration of the assessment).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	System Clearance Part B - revisions with track changes
Author	joconnell
File Modified	0000-00-00
File Created	2021-01-14