Justification

National Center for Education Statistics

National Assessment of Educational Progress

Volume I

Supporting Statement

National Assessment of Educational Progress (NAEP)

2019 eNAEP Pretesting

OMB# 1850-0803 v.233

June 2018

Volume I Table of Contents

1) Submittal-Related Information 3

2) Background And Study Rationale 3

3) Recruitment And Data Collection 4

4) Consultations Outside The Agency 7

5) Justification For Sensitive Questions 7

6) Paying Respondents 7

7) Assurance Of Confidentiality 8

8) Estimate Of Hourly Burden 9

9) Cost To Federal Government 10

10) Project Schedule 10

Attachments

Volume II – Questionnaires

Appendix A – Communication Materials for Simulated Classroom Pretesting

Appendix B – Communication Materials for Field Trial Pretesting

1. Submittal-Related Information

This material is being submitted under the generic National Center for Education Statistics (NCES) clearance agreement (OMB# 1850-0803), which provides for NCES to conduct various procedures (such as pilot tests, cognitive interviews, and usability studies) to test new methodologies, question types, or delivery methods to improve survey and assessment instruments and procedures.

2. Background and Study Rationale

The National Assessment of Educational Progress (NAEP) is a federally authorized survey, by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622), of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is conducted by NCES, which is part of the Institute of Education Sciences, within the U.S. Department of Education. NAEP’s primary purpose is to assess student achievement in the different subject areas and collect survey questionnaire (i.e., non-cognitive) data to provide context for the reporting and interpretation of assessment results.

Over the last few years, NAEP has been transitioning to digitally based assessments (DBA) that are administered on tablets using a test-delivery system developed for NAEP, known as eNAEP. The eNAEP system was successfully used in the 2015 and 2016 pilot assessments.¹ The first operational use of the eNAEP system was in conjunction with the 2017 NAEP assessments. Enhancements have been made after each administration of eNAEP to address issues identified in the field, to make the system more user-friendly, and to allow for the assessment of additional content and scenario-based task item types.

This request is to conduct a real-world test of the eNAEP system with students, allowing the system to be tested in the manner that will be used in the national study to help identify system issues at this stage of the software development process. The rationale for this study is based on lessons learned and issues encountered by students in the field during the 2018 assessments. It is believed that students use and interact with the system differently than adult quality control (QC) testers. Therefore, including students as part of the pretesting and QC process should allow for issues to be identified and addressed prior to the operational use of the system.

Two series of eNAEP pretesting will be conducted with students from grades 4, 8, and 12.² The first series of pretesting, referred to in this submission as Simulated Classroom Pretesting, will be held in a simulated classroom, while the second series of pretesting, referred to in this submission as the Field Trial Pretesting, will be held in real schools, in a live classroom environment. eNAEP is developed in phases referred to as “builds.”

During the Simulated Classroom Pretesting, a different round of pretesting will be conducted to test each build as it becomes available. As part of this series, up to 5 rounds of pretesting will take place over the course of eNAEP development, refinement, and quality control, with event sessions lasting approximately 100 minutes each. Up to 75 fourth-, 75 eight-, and 25 twelfth-grade students will participate in each round (see Table 1 for item content assignment matrix by grade).

Table 1. Simulated Classroom Pretesting NAEP 2019 Content Assignment Matrix

Subject	Grade 4	Grade 8	Grade 12
Mathematics Operational	X	X	X
Reading Operational	X	x	X
Science Operational	x	x	X
Puerto Rico Spanish	x	x
Mathematics Pilot	x	x
Reading Pilot	x	x

During the Field Trial Pretesting, up to 2 pretesting events will take place at the end of eNAEP development with event sessions lasting approximately 120 minutes each. For the Field Trial Pretesting portion of the study, 25 schools (ten grade 4 schools; ten grade 8 schools; and five grade 12 schools) will be recruited by NAEP State Coordinators. This portion of the study is designed to test the updated platform and administrative procedures to identify any logistical challenges and address them prior to the 2019 NAEP administration. The procedures to be tested include:

• Conducting pre-assessment activities such as contacting the school to confirm logistics;

• Arriving on the day of the assessment with all necessary equipment;

• Setting up, administering, and breaking down the assessment following standard NAEP procedures; and

• Using the E-File and MyNAEP systems (see Appendix B) as would be done in an operational assessment.

The Field Trial Pretesting will be conducted in schools by NAEP field administration staff, replicating the actual testing conditions to the fullest extent possible. It will cover the cognitive blocks and questionnaires of all NAEP 2019 digitally based subjects delivered on tablets to up to 1,250 fourth-, 1,250 eighth-, and 750 twelfth-grade students (see Table 2 for the Field Trial content assignment matrix). Teacher and school survey questionnaires will not be included as part of this study.

Table 2. Filed Trials Pretesting NAEP 2019 Content Assignment Matrix

Subject	Grade 4	Grade 8	Grade 12
Mathematics Operational	X	X	X
Reading Operational	X	x	X
Science Operational	x	x	X
Mathematics Pilot	x	x
Reading Pilot	x	x

Results from this study will not be publicly released, and will be used to identify issues and generate solutions or workarounds in advance of the main NAEP 2019 administration.

3. Recruitment and Data Collection

Recruitment and Sample Characteristics for the Simulated Classroom Pretesting

For each round of pretesting, an NCES subcontractor for NAEP, EurekaFacts, will recruit a maximum of 175 students (75 grade 4 students; 75 grade 8 students; and 25 grade 12 students) per round, for up to 5 rounds of pretesting conducted with a maximum of 875 student participants. National assessment content will be tested in grade 12 (NAEP 2019 operational), and National, Puerto Rico Spanish, and State assessment content will be tested in grades 4 and 8 (NAEP 2019 operational and pilot). The location for each event will either be at the EurekaFacts facility in Rockville, Maryland or another facility in the DC area that EurekaFacts obtains.

EurekaFacts will recruit participants for the pretesting study from the District of Columbia, Maryland, Virginia, and West Virginia. Although the sample will include a mix of student characteristics, the results will not explicitly measure differences by those characteristics. Students will be recruited to obtain the following criteria:

• A mix of race/ethnicity (Black, Asian, White, Hispanic, etc.);

• A mix of socioeconomic background;

• A mix of urban/suburban/rural areas;

• A mix of students requiring accommodations; and

• A mix of Spanish-speaking students.

While EurekaFacts will use various outreach methods (see Appendices A1-A8) to recruit students to participate, the bulk of the recruitment will be conducted by telephone and will be based on acquisition of targeted mailing lists containing residential addresses and landline telephone listings. EurekaFacts will also use a participant recruitment strategy that integrates multiple outreach methods and resources such as newspaper and internet ads, community organizations (e.g., Boys and Girls Clubs, Parent-Teacher Associations), and mass media recruitment (e.g., postings on the EurekaFacts website).

Interested students 18-years of age or over (see Appendix A-9) and parents of students under 18-years of age (see Appendix A-6) will be screened to ensure that the recruited students meet the criteria for participation in the study (i.e., that the students are from the targeted demographic groups outlined above). When recruiting participants, EurekaFacts staff will speak to the parent/legal guardian of the interested minor before starting the screening process. During this communication, the parent/legal guardian will be informed about the objectives, purpose, and participation requirements of the data collection effort as well as the activities that it entails. After confirming that a participant is qualified, willing, and available to participate in this study, he or she will receive a confirmation email/letter and phone call. Written, informed parental consent and consent of student participants age 18 or over (see Appendix A11 and A12) will be obtained for all respondents who are interested in participating in the study. Shortly after each of the eNAEP builds is released, a pretesting event with students will be held in a simulated or actual classroom.

Each Simulated Classroom Pretesting session will last approximately 100 minutes and will be structured as follows:

• Each student will be asked to take the assessment under standard NAEP assessment conditions (approximately 90 minutes). Westat is a NAEP contractor and will administer the session using standard procedures. Students will take the full assessment, including the tutorial, cognitive items and tasks, and the survey questionnaires.³

• A group debrief (up to 10 minutes) will be conducted to solicit feedback from the students (see Volume II for the debriefing script).

As part of the assessment administration in all event sessions, students will take a set of survey questionnaires. The maximum time for the survey questionnaire component is 15 minutes (included in the 100-minute time estimated for each session). Students will take a “core” section regarding general student and contextual information and a subject-specific section. Volume II includes the library of possible student survey items to be administered.⁴ Not all of the items presented in Volume II will be administered in this eNAEP pretesting study. The number of items selected for each student will be appropriate to the time allocated. As the items for the 2019 NAEP administration are finalized throughout the development process, the final subset will be included in the eNAEP system for pretesting. As such, the earlier builds may include different items selected from the library in Volume II than the final build.

Recruitment and Sample Characteristics for the Field Trial Pretesting

States will be asked to participate in the study on a voluntary basis. NAEP State Coordinators (“Coordinators”) in volunteer states will recruit schools in their state that are not part of the main NAEP 2019 sample (see Appendix B2 for a sample letter from a Coordinator to a school principal). Coordinators will leverage relationships within the state, including the Principal and Teacher Panels, to contact schools and identify those willing to participate in the study. The NAEP State Coordinator will forward the contact information for participating schools to Westat.

A total of 25 schools will participate in the study. While the study will seek participation from schools with various demographic characteristics, including a mix of urban/suburban/rural locations and students with a mix of race/ethnicity and socioeconomic backgrounds, detailed sampling requirements will not be targeted because of the small number of schools that will be asked to participate in the study. Additionally, schools that are within a relatively close proximity to each other within a state will be selected to ensure that one field administration staff member is able to conduct the study for all schools in the state. Private schools will not be recruited for this study.

Students will be selected to participate in the study via standard NAEP procedures, using the e-filing system for sampling (see Appendix B6 for a step-by-step overview of the tasks completed as part of the e-filing process). Either the sampled State Education Agency (SEA) or school will submit a list of all potential student participants, from which a random sample will be drawn and each school will be notified of the list of selected student participants. A summary of the targeted student sample is provided below:

Grade	Assessment Content	Number of Schools	Number of Subject Matters	Number of Sessions	Students Per Session	Average Number of Students Per School	Total Students Per Grade
4	State	5	2	2	25	100	500
4	National	5	3	2	25	150	750
8	State	5	2	2	25	100	500
8	National	5	3	2	25	150	750
12	National	5	3	2	25	150	750
Total	-	25	13	10	-	-	3,250

After the sample is drawn and schools are notified of selected students, the subsequent tasks to prepare for the assessment are completed by the school coordinator in the MyNAEP system including:

• Registering and Providing School Information

• Submitting Student List/Sample

• Reviewing and Verifying the List of Students Selected for NAEP

• Completing SD/ELL Student Information

• Notifying Parents

• Updating Student List

• Planning for Assessment Day and Encouraging Participation

• Supporting Assessment Day Activities

See Appendix B7 for the full content of the MyNAEP system that will guide school coordinators through the preassessment tasks they will need to complete.

Each Field Trial Pretesting session will last approximately 120 minutes and will be structured as follows:

• During each session, each student will be asked to take the assessment under standard NAEP assessment conditions (approximately 110 minutes including transition time and instructions). Westat will administer the session using standard procedures. Students will take the full assessment, including the tutorial, cognitive items and tasks, and the survey questionnaires.⁵

• A group debrief (up to 10 minutes) will be conducted to solicit feedback from the students (see Volume II for the debriefing script).

Data Collection Process

Normal data collection will be enabled by the eNAEP systems, and any errors generated will be collected automatically by the system. Note that student responses will not be scored. In addition to the eNAEP systems recording information, administrators and observers from NCES, Westat, Fulcrum, ETS, and/or EurekaFacts will monitor the assessments and record notes detailing any issues encountered by the students, as well as what the students were doing at the time each issue occurred. In addition, observers may ask individual students for clarification of the actions he or she took prior to an issue or error occurring. For example, observers may ask questions such as, “What is the error?”; “What was the last thing you saw before the error?”; “What were you expecting to happen?”; or “What did you do right before the error happened?” Understanding and documenting what caused the system error is necessary in order to have enough information for staff to replicate the error and develop a fix for it.

The Simulated Classroom Pretesting sessions will be audio and/or video recorded to capture information regarding any student actions that resulted in system errors or issues. The Field Trial Pretesting sessions will not be recorded.

4. Consultations outside the agency

Westat is the Sampling and Data Collection contractor for NAEP. Westat will provide the tablets for the students’ use and carry out the pretesting study.

Fulcrum is the NAEP contractor responsible for the development and ongoing support of NAEP DBAs for NCES, including the system to be used for the eNAEP pretesting study. Fulcrum will be onsite to assist Westat in the administration of the study.

ETS serves as the Planning and Coordination; Item Development; and Design, Analysis, and Reporting contractor for NAEP. ETS staff may assist in administering and/or observing some sessions.

EurekaFacts is located in Rockville, Maryland. It is an established for-profit research and consulting firm, offering facilities, tools, and staff to collect and analyze both qualitative and quantitative data. EurekaFacts is working as a subcontractor for ETS to recruit participants and provide the facilities to be used for the study. In addition, EurekaFacts staff may assist in administering and/or observing some sessions.

5. Justification for Sensitive Questions

Throughout the item and debriefing question development processes, effort has been made to avoid asking for information that might be considered sensitive or offensive.

6. Paying Respondents

To encourage participation in the Simulated Classroom Pretesting events and thank students for their time and effort, a $25 gift card from a major credit card company will be offered to each participating student. If a parent or legal guardian brings their student to and from the testing site, he or she will also receive a $25 gift card to thank him/her for the time and effort in transporting their child.

The schools that participate in the Field Trial Pretesting events, will each receive a $200 gift card to an office supply store (e.g., Staples or Office Depot) to encourage their participation and to thank them for their time and effort. The study will take place during regular school hours, and thus there will not be any monetary incentive for the student participants, although students will get to keep the NAEP earbuds they will use to participate in eNAEP.

7. Assurance of Confidentiality

The study will not retain any personally identifiable information after the end of data collection. Prior to the start of the study, prospective participants will be notified that their participation is voluntary and that the information they provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

The eNAEP systems used in this study are the draft and/or final versions of the operational NAEP assessment systems. In addition to the Cybersecurity Enhancement Act of 2015 (6 U.S.C. §151), the confidentiality of the operational NAEP assessment is governed by the Confidential Information Protection and Statistical Efficiency Act of 2002 (CIPSEA; see OMB# 1850-0928). As such, the confidentiality pledge associated with CIPSEA⁶ appears in the systems. Confidentiality of the pretesting activities described in this submission is governed by the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C. §9573) as well as 6 U.S.C. §151. Students will be notified of the correct confidentiality language on the login card provided to access the assessment system. The correct confidentiality language will also be listed on the student participation consent forms. School coordinators will be notified of the correct confidentiality language on the letter providing directions to access the MyNAEP system.

Before students can participate in the study, written consent will be obtained from students 18 years of age or older or from the parents or legal guardians of students less than 18 years of age. Participants will be assigned a unique student identifier (ID), which will be created solely for data file management and used to keep all participant materials together. The participant ID will not be linked to the participant name in any way or form. The consent forms, which include the participant name, will be separated from the participant interview files, secured for the duration of the study, and will be destroyed after the final report is released. Pretesting activities may be recorded using audio or screen capture technology. The only identification included on the files will be the participant ID. The recorded files will be secured for the duration of the study and will be destroyed after the final report is completed.

8. Estimate of Hourly Burden

The estimated burden for recruitment assumes attrition throughout the process.⁷

Estimated Burden for the Simulated Classroom Pretesting

In all events, each student will participate in one session for a total of 100 minutes. Table 3 details the estimated burden.

Table 3. Estimate of Hourly Burden for the Simulated Classroom Pretesting

Respondent	Number of Respondents	Number of Responses	Hours per Respondent	Total Hours
Parent or Legal Guardian for Student Recruitment
Initial contact	3,803	3,803	0.05	190
Follow-up via phone	1,902*	1,902	0.15	285
Consent and confirmation	951*	951	0.15	143
Recruitment Total	3,803	6,656	-	618
Student Participation
Students in Spanish events	250	250	1.67	418
Students in eNAEP events	625	625	1.67	1,044
Participation Total	875	875	-	1,462
Simulated Classroom Pretesting Total	4,678	7,531	-	2,080

* Subset of initial contact group

Note: numbers have been rounded and therefore may affect totals

Estimated Burden for the Field Trial Classroom Pretesting

The school principal burden is estimated at 20 minutes for initial contact communications. The school coordinator burden is estimated at four hours and 30 minutes for school personnel to complete the coordinator activities in MyNAEP, including looking up information to enter into the system. Additionally, school coordinators may incur up to 100 minutes to enter information on each Student with Disabilities (SD) and English Language Learner (ELL) that is sampled for the study. Furthermore, if e-filing is completed at the school level (estimated to be completed by 7 schools), the school coordinator will incur an estimated 120 more minutes. Parents/legal guardians of participating students will receive a letter explaining the study (Appendix B5), for which the parent/legal guardian’s burden is estimated at three minutes. An additional burden of 15 minutes is estimated for a small portion of parents/legal guardians (up to 2%) who may write to refuse approval for their child or may research information related to the study. Approximately 3,250 students from 25 schools will participate in the study. Student burden is calculated based on 15 minutes for setup and reviewing the tutorial, 15 minutes to respond to the survey questionnaire, and up to 10 minutes for a group debrief, for a total study session time of 120 minutes. ⁸ Table 4 details the estimated burden.

Table 4. Estimate of Hourly Burden for the Field Trial Pretesting

Respondent	Task	Number of Respondents*	Number of Responses	Hours per Respondent	Total Burden (in hours)
School principal	Initial contact	25	25	0.33	8
School coordinator	Scheduling and logistics	25	25	4.5	113
	e-filing		7	2	14
	SD/ELL Information		25	1.7	43
Parents/Legal guardians	Initial notification	3,250	3,250	0.05	163
Parents/Legal guardians*	Refusals or additional research	65	65	0.25	16
Students	NAEP 2019 Field Trial Pretesting (all subjects)	3,250	3,250	0.67	2,178
Total		6,550	6,647	-	2,535

* These parents are a subset of those who were initially notified.

Note: numbers have been rounded and therefore may affect totals

Table 5 details the total estimated burden for the Simulated Classroom Pretesting and the Field Trial Pretesting sessions.

Table 5. Estimate of Total Hourly Burden

Respondent	Number of Respondents	Number of Responses	Total Hours
Simulated Classroom Pretesting Sub Total	4,678	7,531	2,080
Field Trial Pretesting Sub Total	6,550	6,647	2,535
Total Burden	11,228	14,178	4,615

Note: numbers have been rounded and therefore may affect totals

9. Cost to federal government

The total cost of the study is $1,069,529 as detailed in Table 6.

Table 6. Estimate of Costs

Activity	Provider	Estimated Cost
Recruiting students and providing facilities for the study	EurekaFacts	$715,057
Administering the study	Westat	$354,472
Total		$1,069,529

10. Project Schedule

The schedule for this study, including all activities, is provided in Table 7.

Table 7. Project Schedule

Activity	Dates
Recruitment, pretesting, data collection, analysis and final report	July–December 2018

1 More information about NAEP DBA can be found at http://nces.ed.gov/nationsreportcard/dba/default.aspx.

2 For sessions occurring during the summer, students entering or who have completed grades 4, 8, and 12 will be recruited.

3 Draft content may be used in the earlier builds.

4 The final items will consist of those selected for NAEP 2019 administration (OMB #1850-0928 v.10). The questionnaire components in Volume II are a subset of the questionnaires provided in these submittals.

5 Draft content may be used in the earlier builds.

6 National Center for Education Statistics (NCES) is authorized to conduct NAEP by the National Assessment of Educational Progress Authorization Act (20 U.S.C. §9622) and to collect students’ education records from education agencies or institutions for the purposes of evaluating federally supported education programs under the Family 9 Educational Rights and Privacy Act (FERPA, 34 CFR §§ 99.31(a)(3)(iii) and 99.35). The information provided will be used for statistical purposes only. In accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347 and other applicable Federal laws, responses will be kept confidential and will not be disclosed in identifiable form to anyone other than employees or agents. By law, every NCES employee as well as every NCES agent, such as contractors and NAEP coordinators, has taken an oath and is subject to a jail term of up to 5 years, a fine of $250,000, or both if he or she willfully discloses ANY identifiable information about students. Electronic submission of student information will be monitored for viruses, malware, and other threats by Federal employees and contractors in accordance with the Cybersecurity Enhancement Act of 2015. The collected information will be combined across respondents to produce statistical reports.

7 Assumptions for approximate attrition rates are 50 percent from initial contact to follow-up, 50 percent from follow-up to confirmation, and 90 percent from confirmation to participation.

8 Similar to main NAEP assessments, the cognitive item portions of the study, totaling about 80 minutes per student, are not included in the burden totals because they are not subject to the Paperwork Reduction Act (PRA).