Revisions Memo

MEMORANDUM OMB # 1850-0803 v. 211

DATE: September 26, 2017

TO: Kashka Kubzdela

National Center for Education Statistics

FROM: Linda Hamilton

National Center for Education Statistics

SUBJECT: National Assessment of Educational Progress (NAEP) Technology and Engineering Literacy (TEL) and eNAEP Pretesting Study - Revised [Revision to a previously approved package (1850-0803 v.199)]

The NAEP Technology and Engineering Literacy (TEL) and eNAEP Pretesting Study was approved on June 21, 2017 (OMB #1850-0803 v. 199). This study is designed to test the TEL and eNAEP test delivery systems with students in advance of the main NAEP administration in order to identify system issues early in the software development process. The rationale for this study is based on lessons learned and issues encountered by students in the field during the 1) 2017 assessments and 2) the 2014 TEL administration during normal testing. It is believed that students use and interact with the systems differently than adult QC testers. Therefore, including students as part of the pretesting and QC process should allow for issues to be identified and addressed prior to the operational use of the system. Results from this study will be used to inform future decisions about the NAEP administration model.

Since OMB approval of this study, we have identified a need to conduct an additional round of pretesting with students to ensure that all TEL and eNAEP content is covered sufficiently. The additional round of pretesting will cover grades 4 and 8 mathematics survey questionnaires and the student debriefing questions that will be included in the 2018 Assessment Delivery Study (OMB #1850-0803 v.203).

To reflect this change and the adjustments necessary to the study procedures and materials, we revised the following parts of Volume I and Volume 2 (the original, approved text is shown in red font, and the revised text in blue font) followed by an explanation of each revision (the revisions require 555 additional respondents/responses and 244 additional burden hours):

Revisions to Volume 1

1. Cover page:

Updated version number and changed “June 2017” to “revised September 2017”

2. Section 2: Background and Study Rationale

Original text:

Up to 10 pretesting events will take place over the course of the TEL and eNAEP development, refinement, and quality control with event sessions lasting approximately 100 minutes each.

Revised Text:

Up to 11 pretesting events will take place over the course of the TEL and eNAEP development, refinement, and quality control with event sessions lasting approximately 100 minutes each.

Explanation of revision:

One additional round of eNAEP pretesting is needed to cover grades 4 and 8 mathematics survey questionnaires and the student debriefing questions that will be included in the 2018 Assessment Delivery Study.

3. Section 3: Recruitment and Data Collection:

Original Text:

For each round of eNAEP testing (up to seven rounds), EurekaFacts will recruit a maximum of 102 students (42 students in grade 4, 30 students in grade 8, and 30 students in grade 12). Each session will be held at the EurekaFacts facility in Rockville, Maryland, most likely on a Saturday.

Revised Text:

For each round of eNAEP testing (up to eight rounds), EurekaFacts will recruit a maximum of 102 students (42 students in grade 4, 30 students in grade 8, and 30 students in grade 12). Each session will be held at the EurekaFacts facility in Rockville, Maryland, most likely on a Saturday.

Explanation of revision:

One additional round of eNAEP pretesting is needed to cover grades 4 and 8 mathematics survey questionnaires and the student debriefing questions that will be included in the 2018 Assessment Delivery Study.

Original Text:

⁵ The final items will consist of those selected for NAEP 2018 administration (currently under OMB review: OMB #1850-0928 v. 5). The questionnaire components in Volume II are a subset of the questionnaires provided in the 1850-0928 v.5 submittal.

Revised Text:

⁵ The final items will consist of those selected for NAEP 2018 administration (OMB #1850-0928 v. 5-8) and the NAEP Assessment Delivery Study (OMB #1850-0803 v.203). The questionnaire components in Volume II are a subset of the questionnaires provided in these submittals.

Explanation of revision:

The footnote was revised to account for the inclusion of the NAEP Assessment Delivery Study (OMB #1850-0803 v.203) submittal which includes the student debriefing questions that have been added to Volume 2 of this package, and to account for the fact that all NAEP 2018 materials have by now been approved by OMB.

4. Section 9: Estimate of Hourly Burden:

The burden table has been updated to reflect one additional round of eNAEP pretesting:

Original Table of Total Hourly Burden

Table 1. Estimate of Hourly Burden

Respondent	Number of respondents	Number of responses	Hours per respondent	Total hours
Parent or Legal Guardian for Student Recruitment
Initial contact	3,572	3,572	0.05	179
Follow-up via phone	1,786*	1,786	0.15	268
Consent & confirmation	893*	893	0.15	134
Recruitment Totals	3,572	6,251	-	581
Participation (Pretesting)
Students in TEL Events	90	90	1.67	150
Students in eNAEP Events	714	714	1.67	1,192
Participation Totals	804	804	-	1,342
Total	4,376	7,055	-	1,923

* Subset of initial contact group

Note: numbers have been rounded and therefore may affect totals

Revised Table of Total Hourly Burden

Table 1. Estimate of Hourly Burden

Respondent	Number of respondents	Number of responses	Hours per respondent	Total hours
Parent or Legal Guardian for Student Recruitment
Initial contact	4,025	4,025	0.05	201
Follow-up via phone	2,013*	2,013	0.15	302
Consent & confirmation	1,006*	1,006	0.15	151
Recruitment Totals	4,025	7,044	-	654
Participation (Pretesting)
Students in TEL Events	90	90	1.67	150
Students in eNAEP Events	816	816	1.67	1,363
Participation Totals	906	906	-	1,513
Total	4,931	7,950	-	2,167

* Subset of initial contact group

Note: numbers have been rounded and therefore may affect totals

Explanation of revision:

These account for the additional burden of adding a round of eNAEP pretesting (to achieve 102 students participating in the additional eNAEP pretesting round, requiring initial contact with 453 more parents).

Original Table of Estimate of Costs

The total cost of the study is $523,656 as detailed in Table 2.

Table 2: Estimate of Costs

Activity	Provider	Estimated Cost
Recruiting students and providing facilities for the study	EurekaFacts	$384,268
Administering the study	Westat	$139,388
Total		$523,656

Revised Table of Estimate of Costs

The total cost of the study is $598,464 as detailed in Table 2.

Table 2: Estimate of Costs

Activity	Provider	Estimated Cost
Recruiting students and providing facilities for the study	EurekaFacts	$439,160
Administering the study	Westat	$159,304
Total		$598,464

Explanation of revision:

The data collection costs increased by $74,808 to account for the additional costs of recruitment and administration of the additional round of eNAEP pretesting.

Revisions to Volume 2

1. Cover page:

Updated version number and changed “June 2017” to “revised September 2017”

2. Page 4:

Updated the table of survey questionnaires to add three additional questionnaires:

• II-p: Grade 4 Mathematics

• II-q: Grade 8 Mathematics

• II-r: Assessment Delivery Study Debriefing Items

Explanation of revision:

The table was updated to reflect the addition of three survey questionnaires that will be pretested as part of the additional round of eNAEP pretesting

3. Pages 176-214

Added three components that will be included in the additional round of pretesting to end of Volume 2:

• II-p: Grade 4 Mathematics

• II-q: Grade 8 Mathematics

• II-r: Assessment Delivery Study Debriefing Items

Explanation of revision:

These three survey questionnaires will be pretested as part of the additional round of eNAEP pretesting.

Appendices

No changes were made to the Appendices

Page 6