Change Memo

September 14, 2023

MEMORANDUM

To: Bev M. Pratt, OMB

From: Pat Etienne, NCES

Through: Carrie Clarady, NCES

Re: National Assessment of Educational Progress (NAEP) 2024 Materials Update #3
(OMB# 1850-0928 v.31)

The National Assessment of Educational Progress (NAEP), conducted by the National Center for Education Statistics (NCES), is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, and civics. The National Assessment of Educational Progress Authorization Act (Public Law 107-279 Title III, section 303) requires the assessment to collect data on specified student groups and characteristics, including information organized by race/ethnicity, gender, socio-economic status, disability, and limited English proficiency. It requires fair and accurate presentation of achievement data and permits the collection of background, noncognitive, or descriptive information that is related to academic achievement and aids in fair reporting of results. The intent of the law is to provide representative sample data on student achievement for the nation, the states, and subpopulations of students and to monitor progress over time. The nature of NAEP is that burden alternates from a relatively low burden in national-level administration years to a substantial burden increase in state-level administration years when the sample has to allow for estimates for individual states and some of the large urban districts.

Amendment #2 to the NAEP 2024 clearance package received OMB approval in August 2023 (OMB#1850-0928 v.30). Between Amendment #2 and Amendment #3, NCES made the decision to no longer use school staff to proctor accommodation sessions as previously included in early versions of Amendment #2. These changes are reflected in Amendment #3, and the communication materials, burden hours, and costs to the Federal Government to remove these activities as a result. The increased cost to the Federal Government is due to the need to hire additional Field Staff to conduct these separate sessions, resulting in an additional $3,700,000. Additionally, the Field Trial will include teacher and school survey questionnaires, resulting in an increase in 80 burden hours between Amendment #2 (470,264 hours) and Amendment #3 (470,344 hours). This revision provides minor updates Part A to detail the removal of the staff proctored accommodation sessions in the burden table and addition of teacher and school questionnaires for the Field Trial to the burden table, updates to communication materials placeholders in Part B as well as adding a reference to Best Practices materials (Section B.3), updated and added final communication materials to Appendix D, new Assessment Management System (AMS) screenshots in Appendix I, and minor update to two items in Appendix J1 removing subitem text, and revised eNAEP and NAEPq login screenshots and paper booklet covers in J1, J2, J3, and J-S.

The following table and pages below provide a summary of the changes that were made to the documents included in this submission as compared to the documents submitted in the 2024 NAEP Amendment #2 Package. As noted above, the costs to the Federal Government increased by $3,700,000 in this Amendment #3 submission.

Summary of All Changes

Document	Changes
Part A	Updates to scope: 1) Removal of any remaining references to staff proctored accommodation sessions. 2) Addition of teacher and school questionnaires for the Field Trial, adding 80 burden hours between Amendment #2 and Amendment #3. Minor text edits to update references from Amendment #2 to Amendment #3. Updates to cost to Federal Government; since school staff will not be proctoring separate accommodation sessions, Field Staff will need to be hired to proctor these separate sessions, resulting in an additional $3,700,000. Updates to a footnote under A.8 Consultations Outside the Agency to clarify that some of the 2024 NAEP work may fall outside the current contract schedule for the NAEP Alliance contractors.
Part B	Updates to section B.2.a communication material references. Updates to B.3 adding the Best Practices material references.
Appendix D	Appendices D-14a to D-14d, D-41 to D-43, D-50, D-51a to D-51l, D-13-S-PR to D-15-S-PR are new in this submission.
Appendix I	Updated English and Spanish (PR) AMS log in screenshots showing the PRA language. Appendices I1, I3, I4, I6-I14, I16 (I16a-e), I17 (I17a-e; was Appendix E), I18, I19
Appendix J1	eNAEP login screen updated Pilot Core grade 4 and 8 in J1-32 (p. 161) and J1-33 (p. 163) item VR586105 (#5) - "with others" removed from subitem d. Appendix J1-6 Operational Grade 8 (Core, Mathematics) (p. 24) – revision to item - subitem e has “Google Pixel” rather than “HTC One” added as an example in parenthesis. See items outlined in orange. Appendix J1-34 Pilot Grade 4 (Mathematics)-added VR760467. See items outlined in green. Appendix J1-35 Pilot Grade 8 (Mathematics)-added VR760467. See items outlined in green. Appendix J1-36 Pilot Grade 4 (Reading) (p. 174) – added VR590984 (#13) and VR591075 (#14), added VR591015 (#21) and VR591010 (#22), added VH 811176 (#25). See items outlined in orange. Appendix J1-37 Pilot Grade 8 (Reading)- added VR590984 (#18) and VR591075 (#19), added VR591015 (#21) and VR591010 (#22), added VR590986 (#20), revised VR 591021 (#31) to remove “texts” from sub items a and b, revised VR 591033 (#26) to remove “Kindle Fire” from subitem c, added VH 811176, added VH811222, added VH853176 (#1), added VR775375. See items outlined in orange and green.
Appendix J2	NAEPq login screen and paper booklet cover updated. Appendix J2-19 Pilot Grade 4 (Reading) (p. 167) – revised VH858404 (#3) to add “using print and digital texts.” See item outlined in orange. Appendix J2-20 Pilot Grade 8 (Reading) (p. 175) – revised VH812038 (#10) subitem c. See item outlined in orange.
Appendix J3	NAEPq login screen and paper booklet cover updated. Appendix J3-22 Operational Grade 8 NIES (p. 110) – revised VR 616342 (#18) tense of subitems a, b, c. See item outlined in orange.
Appendix J-S	eNAEP login screen, NAEPq login screen, and paper booklet cover updated.

Changes from 2024 NAEP Part A Amendment #2 (v.30) to Amendment #3 (v.31) are shown below:

A.1.a. Purpose of Submission

The full 2024 NAEP Administration OMB is Clearance package (OMB# 1850-0928 v.28; cleared April 2023) will have a newincluded a clearance request, with both a 60-day and 30-day consecutive public comment period notice published in the Federal Register, as well as. Tthree Amendments to this the Clearance package (OMB# 1850-0928 v.29-31) are planned to be submitted in the coming months to update materials, like the Assessment Management System materials, survey questionnaires, communication materials, etc. This submission reflects the third and final Amendment for the 2024 NAEP Administration.

NAEP 2024 Amendment Schedule Table
Amendment #2 (this document)	Part A: Updates to the 2024 NAEP administration Part B: Updates to the 2024 NAEP administration Appendix A: External Advisory Committees Appendix B: 2018 Weighting Procedures Appendix C: 2024 Sampling Memo Draft Appendix D (formerly D1 and D2): Updated communication materials (English, Spanish, and HSTS) Appendix G: 2018 Sampling Design Final Appendix I (formerly I and I2): New AMS System – Initial Access Components; HSTS AMS screens. Note: Appendix H is no longer included as a separate document since the AMS now includes the eFiling process in Appendix I. As a result, it is removed from the list of documents. Appendix J1-J3 and J-S: All 2024 SQs, including NIES (formerly in Appendix K)
Amendment #3 (August 2023This document)	Part A: Updates to the 2024 NAEP administration (if needed) Part B: Updates to the 2024 NAEP administration (if needed) Appendix C: 2024 Sampling Memo Final Appendix D: Final NAEP communication materials Appendix E: Assessment Feedback Forms Appendix I: New AMS System – Final Access Components Appendices J1-J-S: Minor wording edit to 2 Pilot Core items’ subitem in grades 4 and 8 in J-1; updated eNAEP login screens, NAEPq login screens and paper booklet covers in J-1, J-2, J-3, and J-S

A.8. Consultations Outside the Agency

The NAEP assessments are conducted by an alliance of organizations under contract with the U.S. Department of Education.⁸

A.12. Estimation of Respondent Reporting Burden (2024)

• Field Trial—The assessment and procedures are the same as those for the operational pilot assessment. In descriptions of the Field Trial as described in previous packages, they did not include, with one exception: the teacher and school principal questionnaires. That has been amended with Amendment #3 and teacher and school principal questionnaires are now included in the burden estimates in Exhibit 1 below are not administered in the field trial.

• School Staff Proctoring Accommodation Sessions—NOTE: In previous versions of this package NAEP shared plans for the 2024 administration to utilize school staff to proctor accommodation sessions for students on assessment day. After further stakeholder feedback and program evaluation, the program has decided not to use school staff to proctor these sessions; rather, NAEP will continue to use Field Staff and follow the traditional model for accommodations.

EXHIBIT 1⁷

Estimated Burden for NAEP 2024 Assessments

(Note: all explanatory notes and footnotes are displayed following the table)

		4th Grade					8th Grade							12th Grade
	Subjects	OP and Pilot	Puerto Rico OP and Pilot	NIES4	Field Trial			OP and Pilot	Science	Puerto Rico OP and Pilot	NIES4	Field Trial		OP	HSTS5	Field Trial	Total
Students	# of Students	253,200	5,000	8,000	260 324			250,200	22,000	5,000	6,500	340 420		52,000	0	105 50	588,194 588,105
	Avg. min. per response	30	30	20	30			30	30	30	20	30		30	0	30	N/A
	Burden (in hours)	126,600	2,500	2,667	130 162			125,100	11,000	2,500	2,167	170 210		26,000	0	53 25	298,931 298,887
Teachers	# of Teachers	19,821	750	2,000	24 0			25,408	4,064	1,000	1,500	30 0		0	0	0	51,043097
	Avg. minutes per response	30	30	20	30			20 and 10 minutes for each additional subject3	20	20	20	20 and 10 minutes for each additional subject 0		0	0	0	N/A
	Burden (in hours)	9,911	375	667	12 0			10,587	1,355	333	500	13 0		0	0	0	23,728753
School Questionnaire (school principal)	# of Schools	6,607	250	2,200	8 0			6,352	1,016	250	2,000	10 0		1,545	0	3 0	16,020041
	Avg. minutes per response	30	30	30	30			30	30	30	30	30		30	0	30	N/A
	Burden (in hours)	3,304	125	1,100	4 0			3,176	508	125	1,000	5 0		773	0	2 0	10,11122
Pre-assessment Technology Activities6	# of Technical Staff	19,821	750	N/A	24			19,056	3,048	750	N/A	30		4,635	0	9	48,123
	Burden (in hours)	7,158	271	N/A	9			6,881	1,101	271	N/A	11		1,674	0	3	17,378
Pre-assessment, sample submission, & assessment feedback (school coordinator)	# of School Coord.	6,607	250	N/A	8			6,352	1,016	250	N/A	10		1,545	1,545	3	17,586
	Burden (in hours)1	33,387	1,263	N/A	40			32,099	5,134	1,263	N/A	51		7,807	4,255	15	85,314
SD/EL (school personnel)	# of Schools	6,607	250	N/A	8			6,352	1,016	250	N/A	10		1,545	0	3	16,041
	# of SD/EL Students2	68,364	1,350	N/A	70 87			55,044	4,840	1,100	N/A	75 92		8,320	0	17 8	139,205 139,180
	Avg. minutes per response	15	15	N/A	15			15	15	15	N/A	15		15	0	15	135
	Burden (in hours)	17,091	338	N/A	18 22			13,761	1,210	275	N/A	19 23		2,080	0	4 2	34,802796
Total Burden (in hours)		197,451	4,601 4,872	4,434	224 213			184,723 191,604	19,207 20,308	4,496 4,767	3,667	284 269		36,660 38,334	4,255	77 42	470,264 470,250

Total number of respondents: 737,007 736,993

Total number of responses: 860,171 860,132

EXHIBIT 1

Notes for 2024 table in Exhibit 1

7. School Staff Proctoring Accommodation Sessions—NOTE: In previous versions of this package NAEP shared plans for the 2024 administration to utilize school staff to proctor accommodation sessions for students on assessment day. After further stakeholder feedback and program evaluation, the program has decided not to use school staff to proctor these sessions; rather, NAEP will continue to use Field Staff and follow the traditional model for Amendment 2 was amended after 30-day public posting and prior to OMB approval.

EXHIBIT 2

Total Annual Estimated Burden Time Cost for NAEP 2024 Assessments

Data Collection Year	Number of Respondents	Number of Responses	Total Burden (in hours)
2024	737,007 736,993	860,171 860,132	470,264 470,250

The estimated respondent burden across all these activities translates into an estimated total burden time cost 470,264250 hours¹, broken out by respondent group in the table below.

	Students		Teachers and School Staff		Principals		Total
	Hours	Cost	Hours	Cost	Hours	Cost	Hours	Cost
2024	298,931 298,887	$2,167,250 $2,166,931	161,222161,241	$5,257,449 $5,258,069	10,111 10,122	$498,978 $499,521	470,264 470,250	$7,923,677 $7,924,520

A.14. Estimates of Cost to the Federal Government

The total cost to the federal government for the administrations of the 2024 NAEP data collections (contract costs and NCES salaries and expenses) is estimated to be $149,999,426 153,699,426. The 2024 assessment cost estimate is shown in the table below.

NCES salaries and expenses	$1,400,300
Contract costs	$148,599,126 152,299,126
Printing, packaging, and distribution, and scoring	$16,732,431
Item Development	$20,602,000
Sampling, recruiting and training, data collection, and weighting	$88,641,881 92,341,881
Recruitment and State Support	$1,000,000
Design, analysis and reporting	$8,373,314
Securing and transferring DBA assessment data	$120,000
DBA system development	$13,129,500

A.15. Reasons for Changes in Burden and Budget (from last Clearance submittal)

As noted in Exhibit 1, NAEP is no longer utilizing school staff to proctor separate accommodation sessions. As such, additional Field Staff will need to be hired in order to conduct these separate sessions. The resulting change in the budget reflected in this Amendment #3 submission is an additional $3,700,000. Additionally, the Field Trial will include teacher and school survey questionnaires, resulting in a small decrease in burden hours between Amendment #2 (470,264 hours) and Amendment #3 (470,250 hours). Since the submission of Amendment #1, there have been some additional activities that have decreased burden and increased budget for this current Amendment #2. Changes to scope include the following:

1) Addition of Reading Router Pilot for grades 4 and 8, increasing costs;

2) Addition of School and District Technology Coordinator roles and SBE survey completion, increasing burden hours;

3) Addition of protocols for the health and safety of field staff, increasing costs;

4) Reduction in SQ burden time for students, teachers, and schools since COVID-19 learning recovery items are no longer adding additional time to the SQs; rather, other items were dropped to accommodate these items, reducing burden hours;

5) Addition of Field Trial for grades 4, 8 and 12, increasing burden hours and costs.

The combination of the new activities, while changing the scope of the administration between Amendments, equates to a 49,341 burden-hour decrease in 2024 Amendment #2 compared to Amendment #1 (470,264 burden hours in Amendment #2 compared to 519,605 burden hours in Amendment #1). The costs of added scope activities (#1, #2, #3, and #5 in the above listing), however, equate to an increase in $4,059,127 dollars in Amendment #2 compared to Amendment #1 ($149,999,426 in costs in Amendment #2 compared to $145,940,299 in costs in Amendment #1).No changes to burden or budget were needed for this amendment submission.

Changes from 2024 NAEP Part B Amendment #2 (v.30) to Amendment #3 (v.31) are shown below:

B.2.a. Recruitment of Schools

• sending a letter to each school’s principal with instructions for assigning a school coordinator (see letter Appendix D-6); along with additional information (for the included information see Appendix D-10, D-11, D-14b and D-35, D-46, D-47 and D-48). Current placeholders for D-46 through D-48 will be updated in Amendment #3;

• sending a letter to each school’s coordinator with instructions (see letter Appendix D-6); along with additional information (for the included information see Appendix D-10, D-11, D-14b and D-35, D-46, D-47 and D-48). Current placeholders for D-46 through D-48 will be updated in Amendment #3.

• Note: Below outlines the flow of technical logistics communications to be included in Amendment #3.

B.3. Methods to Secure Cooperation, Maximize Response Rates, and Deal with Nonresponse

• The Best Practices provides resources and strategies to increase twelfth-grade student motivation and participation (see Appendix D-51).

Changes to Appendix J-1 from Amendment #2 (v.30) to Amendment #3 (v.31) are shown below:

Appendix J1-6 Operational Grade 8 (Core, Mathematics) – revision to item - subitem e has “Google Pixel” rather than “HTC One” added as an example in parenthesis.

Appendix J1-34 Pilot Grade 4 (Mathematics)-added VR760467. See items outlined in green.

Appendix J1-35 Pilot Grade 8 (Mathematics)-added VR760467. See items outlined in green.

Appendix J1-36 Pilot Grade 4 (Reading) – added VR590984 and VR591075, added VR591015 and VR591010, added VH 811176.

Appendix J1-37 Pilot Grade 8 (Reading)- added VR590984 and VR591075, added VR591015 and VR591010, added VR590986, revised VR 591021 to remove “texts” from sub items a and b, revised VR 591033 to remove “Kindle Fire” from subitem c, added VH 811176, added VH811222, added VH853176, added VR775375.

Changes to Appendix J-2 from Amendment #2 (v.30) to Amendment #3 (v.31) are shown below:

Appendix J2-19 Pilot Grade 4 (Reading) – revised VH858404 to add “using print and digital texts.”

Appendix J2-20 Pilot Grade 8 (Reading) – revised VH812038 subitem c.

Changes to Appendix J-3 from Amendment #2 (v.30) to Amendment #3 (v.31) are shown below:

Appendix J3-22 Operational Grade 8 NIES – revised VR 616342 tense of subitems a, b, c.

8 The current contract expires on June 30, 2024. As such, some of the 2024 administration activities will occur after this contract expires. Specifically, the reporting and analysis efforts for 2024 NAEP will fall outside of the current contract for ETS-DAR.

1 The average hourly earnings of teachers and principals derived from May 2021 Bureau of Labor Statistics (BLS) Occupation Employment Statistics is $32.61 for teachers and school staff and $49.35 for principals. If mean hourly wage was not provided, it was computed assuming 2,080 hours per year. The exception is the student wage, which is based on the federal minimum wage of $7.25 an hour. Source: BLS Occupation Employment Statistics, http://data.bls.gov/oes/ datatype: Occupation codes: Elementary school teachers (25-2021); Middle school teachers (25-2022); High school teachers (25-2031); Principals (11-9032); last modified date May 2021.