Revised Memorandum--Vol I Wave 2

Note: The core questions for grade 8 and the charter school questions were submitted and approved (OMB# 1850-0790 v.37, July 2013) as part of Wave 1 but are included here because they are a component of TEL.

1. Explanation for This Submittal

This document contains supplemental information pertaining to the 2014-2016 National Assessment of Educational Progress (NAEP) system clearance proposal. NAEP is administered by the National Center for Education Statistics (NCES) in the Institute for Education Sciences of the U.S. Department of Education. The National Assessment Governing Board (henceforth referred to as the Governing Board) sets policy for NAEP and determines the content framework for each assessment. The system clearance package was approved in March 2013 (OMB #1850-0790 v.36). The terms of clearance for OMB approvals state that NCES will publish a notice in the Federal Register allowing a 30-day public comment period on the details of each collection concurrent with submission to OMB.

The 2014 submittal is divided into two waves to meet scheduling and question development requirements. The first wave contained the core,¹ social studies (civics, geography, U.S. history), and science (paper-and-pencil) descriptions, burden, and questionnaires. Wave 2 contains the technology and engineering literacy (TEL) descriptions, burden, and materials, as well as information regarding school coordinator activities, including the collection of information on students with disabilities (SD) and English language learners (ELL), and the assessment feedback materials.

This Wave 2 submittal contains burden information and survey questions (also referred to as noncognitive questions) for the components of the NAEP 2014 assessments listed in Table 1.

Table 1: List of Components in Wave 2

Component	Part of Volume II
Student Surveys: TEL - Grade 8	Part 1
School Surveys: School Characteristics and Policies (SCP), TEL, Charter School- Grade 8	Part 2
Assessment Feedback Surveys: School Coordinator Debriefing Interview, Post-Assessment Follow-up Survey, Teacher/School Online Completion Feedback	Part 3

Note: There is no teacher questionnaire for TEL.

2. Overview of Wave 2 NAEP 2014 Assessments

Much of the following broad overview of the 2014 NAEP assessments was included as part of the 2014–2016 system clearance submittal. The Governing Board determines NAEP policy and the assessment schedule, and future Governing Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, the overall methodology and assessment process remains consistent. The NAEP assessments contain two kinds of questions: cognitive, or assessment, questions that measure student knowledge of an academic subject, and survey, or background, questions that gather information on demographic as well as classroom instructional procedures. The Governing Board is responsible for selecting and approving all of NAEP’s survey and cognitive questions.

The NAEP assessments are currently supported by an alliance of organizations under contract with NCES, within the U.S. Department of Education. The main NAEP assessment reports current achievement levels and trends in student achievement at grades 4, 8, and 12 for the nation and, for certain assessments, states and select urban districts. These assessments follow subject-area frameworks developed by the Governing Board and use the latest advances in assessment methodology. The subject-area frameworks evolve to match instructional practices.

3. Sampling Information

The 2014 Sampling Design Memo² (see appendix F) provides specific sampling information regarding the 2014 NAEP assessments. It includes information on the following:

Assessment types
Primary Sampling Units Selection
Stratification and Oversampling
Overlap Control of Samples
Substitute Samples
Student Sampling Information
Weighting Requirements

4. Information Pertaining to the Materials in This Submittal

Student Questionnaires

The 2014 NAEP Technology and Engineering Literacy (TEL) Framework broadly defines technology and engineering literacy as the capacity to use, understand, and evaluate technology, and to understand technological principles and strategies needed to develop solutions and achieve goals. In 2014, the first-ever national TEL assessment will be conducted at grade 8. This assessment will be administered on computers and include interactive scenario-based tasks, as well as discrete items. The TEL survey questions were previously pretested in cognitive interviews (2011), tryouts (2012)³, and a pilot assessment (2013). Reviews of the TEL pilot results, as well as documentation regarding final decisions for inclusion in 2014, are provided in appendix E. Committees that helped define the cognitive and survey questions pertaining to the 2014 subjects in Wave 2 are listed in appendix A. Core questions for grade 8 were submitted and approved as part of Wave 1 but are included here because they are a component of TEL.

Teacher Questionnaires

Teachers of students participating in the TEL assessments will not be administered teacher questionnaires.

School Questionnaires

School questionnaires are comprised of multiple sections, which include:

School Characteristics and Policies (SCP; also referred to as the school core) – One version of SCP questions will be given at each grade. The SCP questions are comprised of trend or previously piloted questions. SCP questions were submitted and approved as part of Wave 1 but are included here because they are a component of the TEL school questionnaire.

TEL – The operational school questions are previously piloted questions.

Charter School – Charter school questions are comprised of trend questions that were administered in previous assessments. Note that the charter school section is only completed if the school is a charter school. Charter school questions were submitted and approved as part of Wave 1 but are included here because they are a component of the TEL school questionnaire.

All 2014 school questionnaires will be completed online.

School Personnel Assessment Responsibilities

E-filing – Demographic data are collected for potential students who may participate in NAEP. This sample information is gathered at the state, district, or school level.

Pre-Assessment Activities – Prior to the assessment, school personnel must prepare for the assessment. Part of the preparation includes logistical preparation for the upcoming assessment, information gathering, communication of teacher and student information, and review of NAEP procedures.

NAEP is moving in the direction of paperless administrations. In previous assessment years, in preparation for conducting assessments, NAEP field staff conducted an in-person pre-assessment visit with each school. The in-person pre-assessment visit (PAV) involved the following procedures:

Sending a package of hardcopy materials to the schools in early January
Schools preparing materials for an in-person visit by NAEP field staff
NAEP field staff spending at least an hour at each school conducting the pre-assessment visit
Follow-up phone calls and emails to obtain all required information before the assessment day

In 2014, a new electronic pre-assessment visit (e-PAV) system is being developed so that school coordinators will provide requested administration information online, including logistical information, updates of student and teacher information, and the completion of inclusion and accommodation information. The benefits of adopting this approach include enhanced security of assessment items and student and school information, efficiencies in data collection, reduced burden on school staff, improved quality control, and reduced costs. The electronic process will provide the following enhancements:

Materials will be provided to the school via a secure electronic application in early December
Schools will provide information in a secure electronic application at their own pace
NAEP field staff will review school entries and conduct a confirmation phone call with schools in January

The e-PAV system is being developed in consultation with the state coordinators, so that the system considers the needs of the states and the schools. In addition, the system has undergone a usability review, the recommendations of which are included in appendix D. These recommendations have been incorporated into subsequent development of the e-PAV system.

Multiple sources of support will be available to schools as part of the electronic process, including:

dedicated Help Desk, accessible by email, phone, or live chat;
NAEP field staff, accessible by phone or email; and
NAEP state coordinators, accessible by phone or email.

In some instances, NAEP field staff will visit schools that are having trouble accessing the e-PAV system and help the schools complete the electronic process in-person.

In order to evaluate the success of the e-PAV and inform refinements to the e-PAV process and procedures for 2015, NAEP will be conducting a number of evaluation activities, including

debriefing with NAEP state coordinators, field staff, and school coordinators;
analyzing observable data from the electronic system (such as time spent on each activity);
monitoring in-person visits made by field staff for frequency, issues that prompted the visits, and feedback from both school and field staff; and
reviewing Help Desk issues.

A sample brochure communication from the NAEP state coordinators to the participating schools describing the pre-assessment and e-filing activities is included as appendix C and will be finalized prior to the 2014 assessment year. It outlines what the school coordinator will be responsible for.

SD and ELL information – The SD and ELL inclusion information is provided by the teachers or school personnel most knowledgeable about the school’s SD and ELL students. This information will then be entered into the e-PAV system by the school coordinator rather than completed via hardcopy forms. With the e-PAV system, inclusion guidelines can now be customized on a state-by-state basis. The SD and ELL information provided can be formatted with appropriate skip patterns and consistency checks, allowing field staff to monitor exclusions in real time. The e-PAV system is currently under development but the types of SD and ELL information gathered will be the same as was previously obtained on the SD and ELL worksheets. Examples of the 2013 SD and ELL worksheets and instructions, along with samples of state specific summary guidelines, are included for reference purposes in appendix G.

Assessment Feedback Activities

School Coordinator Debriefing Interview – After each assessment, the field staff will meet with the school coordinator for a debriefing interview. The purpose of this interview is to obtain feedback on

how well the assessment went in that school,
usefulness of NAEP materials (publications, letters, etc.),
school staff feedback,
preparation activities,
strategies utilized for increasing participation, and

any issues that were noted.

Post-assessment Follow-up Survey – As part of the on-going quality control of the assessment process, 25% of the schools will be randomly selected for an additional follow-up survey. Survey questions solicit feedback on pre-assessment, assessment, and procedural processes. A sample survey, which will be updated for 2014, is included in Volume II, Part 3.

Teacher/School Online Completion Feedback – As discussed earlier, in 2014, all teacher and school questionnaires will be completed online. An evaluation question will be added to the end of the questionnaires, seeking feedback regarding the online completion process.

5. 2014 Wave 2 Burden Information

The Wave 2 burden includes TEL student and school survey burden, school coordinator burden for the electronic pre-assessment, electronic filing processes, debriefing interview, and post-assessment follow-up survey, and school personnel burden for completion of SD and ELL information. Note that while Wave 2 does also include a question to teachers and school administrators regarding feedback on the online completion process, the addition of one question does not significantly impact the already approved burden and, therefore, there is no additional burden associated with the administration of this question. The total burden for the Wave 2 materials is shown in Table 2.

Table 2: Total Burden for Wave 2

Respondent Category	Number of Respondents	Total Burden Hours
Student	20,000	10,000
School	865	433
School Coordinator (for pre-assessment activities, e-filing and assessment feedback)	1,398	3,745
School Personnel (for SD & ELL data completion)	1,398	1,434
Total	23,661	15,612

Further breakdown of the survey per-respondent and overall burden times are included in Tables 3 and 4, respectively, and include the following categories:

Students – For the grade 8 TEL assessment, students are presented two 30-minute cognitive sections followed by a 15-minute survey questionnaire section, which includes core and TEL-specific items.

Additional student burden time is included for receiving instructions, reading directions, and taking a tutorial for using the system. For computer-based assessments, this additional burden is estimated at 15 minutes, thus, the total burden for students is 30 minutes⁴ for computer-based assessments.

Appendix B includes a sample parental notification letter regarding NAEP. The letter will be adapted for each grade/subject combination and the school principal may edit it. However, the information regarding confidentiality and the appropriate law reference will remain unchanged.

Principals/Administrators – The school administrators in the sampled TEL schools will be asked to complete a school questionnaire. The school questionnaire burden is estimated to be 30 minutes.

Table 3: Wave 2 Questionnaires – Survey Per Respondent Burden Time

Respondent Category	Time per Respondent
Grade 8
Student Core + TEL	15 minutes
Student Directions and Set-up	15 minutes
School Questionnaire	30 minutes

School Personnel: Pre-Assessment, Electronic Filing Activities, and Assessment Feedback Activities

Pre-Assessment Activities – As described in section 4, school personnel perform multiple activities in preparation for the assessment, including information gathering, communication of teacher and student information, and review of NAEP procedures. These pre-assessment activities are estimated to require two hours of school personnel time.

E-filing –If the survey sample information needed for e-filing is obtained at the school or district level, school or district personnel will incur some burden. This e-filing burden is estimated at one hour per school. For the most recent national-only NAEP assessment, approximately 52 percent of schools and districts participated in the e-filing process. Thus, 52 percent of total schools were used for e-filing burden calculation purposes.

Assessment Feedback Activities – As described in section 4, school personnel will participate in a School Coordinator Debriefing Interview immediately following the assessment. The estimated burden for this interview is 7 minutes. In addition, 25% of the school personnel will be contacted for a more in-depth Post-assessment Follow-up Survey. The burden for this survey is estimated at 10 minutes per school.

School Personnel: SD and ELL – SD and ELL burden is the average number of hours school personnel spend on completing the SD/ELL information for students identified as SD and/or ELL (estimated at 10 minutes per student). The overall SD/ELL burden is a factor of the number of students identified as SD/ELL. Based on most recent data, 22 percent of grade 4 students, 17 percent of grade 8, and 13 percent of grade 12 students were identified as SD/ELL.

All 2014 school personnel activities (pre-assessment activities, e-filing, feedback activities, and SD and ELL information collection) are included as part of this Wave 2 submittal. Therefore, the burden for school personnel who complete them is included with Wave 2 for all 2014 assessments (i.e., those included in Wave 1 and those included in Wave 2).

Table 4: Estimated Burden for NAEP 2014 Activities Described in This Submittal (Wave 2)

6. How, by Whom, and for What Purpose the Data Will Be Used

The purpose of NAEP is to collect and report assessment data on student achievement in the subject areas assessed for use in monitoring educational progress. In addition to reporting overall results of student performance and achievement, NAEP also reports student performance results for various subgroups of students and on various educational factors. The Governing Board sets guidance for what is asked in the questions. NCES is responsible for developing the questions and for selecting the final set of questions. The questions are designed to (a) provide the information for disaggregating data according to categories specified in the legislation,⁵ (b) provide contextual information that is subject specific (e.g., reading, mathematics) and has an impact and known relationship to student achievement, and (c) provide policy-relevant information specified by the Governing Board.

In the original request to OMB for system clearance, NCES requested approval of the data-gathering instruments to be used in the 2014–2016 national- and state-level NAEP assessments. An expanded description of the purposes of the data is provided in the system clearance submittal.

7. Estimates of Costs to the Federal Government

Table 5 provides estimates for the Wave 2 portion of the 2014 administrations.

Table 5: Administration Cost Estimates (Wave 2 subjects)

Activity	Provider	Estimated Costs
Web support and maintenance related to the TEL assessments	Fulcrum IT Services	$983K
Field administration, including administration of assessments and data collection from the field	Westat (the Data Collection contractor and the NAEP Support and Service Center [NSSC] contractor)	$864K
Totals		$1.847 million

Appendix A: Committee Lists

NAEP Background Variable Committee

Name Affiliation

Patricia Alexander University of Maryland

Arthur Applebee State University of New York at Albany

Claudia Buchman Ohio State University

Lizanne Destefano University of Illinois

Robert Hauser University of Wisconsin-Madison

Kathleen Heid Penn State University

Henry Levin Columbia University

Peter Levine Tufts University

Linda Levstik University of Kentucky

Samuel Lucas University of California-Berkeley

Senta Raizen WestEd,

Washington, DC

Andrew Sum Northeastern University

William Walstad University of Nebraska-Lincoln

NAEP Technology and Engineering Literacy (TEL) Committee

Name Affiliation

Keith Barton Indiana University

John Behrens Pearson e-LEADS Center,

Mishawaka, IN

Brooke Bourdelat-Parks BSCS (Biological Sciences Curriculum Study),

Colorado Springs, CO

Barbara Bratzel Shady Hill School

Cambridge, MA

Lewis Chappelear James Monroe High School,

North Hills, CA

Britte Haugan Cheng SRI International,

Menlo Park, CA

Meredith Davis North Carolina State University

Chris Dede Harvard Graduate School of Education

Richard Duran University Of California, Santa Barbara

Maurice Frazier Oscar Smith High School,

Chesapeake, VA

Camilla Gagliolo Arlington Public Schools,

Arlington, VA

Christopher Hoadley New York University

Eric Klopfer Massachusetts Institute of Technology

Beth McGrath Stevens Institute of Technology,

Hoboken, NJ

Greg Pearson National Academy of Engineering,

Washington, DC

John Poggio University of Kansas

Erin Reilly Annenberg School for Communication & Journalism,
Los Angeles, CA

Troy Sadler Missouri University Science Education Center,

Columbia, MO

Kimberly Scott Arizona State University

Teh-Yuan Wan New York State Education Department,

Albany, NY

Appendix B: Sample Parent/Guardian Notification Letter

NAEP (NAEP Assessment Year) PARENT/GUARDIAN NOTIFICATION LETTER

(School Letterhead)

(Insert Date Here)

Dear Parent or Guardian:

We are pleased to notify you that (school name) has been selected to represent schools across the nation by participating in the National Assessment of Educational Progress (NAEP). First administered in 1969, NAEP is the largest nationally representative assessment of what students know and can do in various subject areas. It is administered by the National Center for Education Statistics, within the U.S. Department of Education. NAEP is different from state assessments because it provides a common measure of student achievement across the country. The results of NAEP are released as The Nation's Report Card, which provides information about student achievement to educators, parents, policymakers, and the public.

In our school, the NAEP assessment will be given on (date) in (subject). Your child (may be/has been) selected to take the assessment. In addition to subject area questions, students will be asked some questions about themselves and their educational experience. These questions provide contextual information for the assessment, as well as information that may be related to students’ learning. If you would like to view sample subject area and student questions, please visit http://nationsreportcard.gov/parents.asp.

It will take about (assessment time) minutes for most students to participate in the assessment. The results are completely confidential (in accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347), and the information provided will be used for statistical purposes only. Your child’s grades will not be affected. Your child may be excused from participation for any reason, is not required to complete the assessment, and may skip any question. While NAEP is voluntary, we depend on student participation to provide an accurate measure of student achievement that will inform improvements in education. Your child will represent many other students, so participation is very important. However, if you do not want your child to participate, please notify me in writing by (date).

There is no need to study in preparation for NAEP. We do, however, ask parents to encourage their children to do their best and get plenty of rest the night before the assessment. A brochure that explains what participation in NAEP means for you and your child is available at http://nces.ed.gov/nationsreportcard/pdf/parents/2012469.pdf. Should you have questions, please contact me at (telephone number) or at (e-mail address).

We are excited that our school will be participating in NAEP and pleased that your child (may be/has been) selected. We know that (school name)'s students will help us show what our nation’s students know and can do.

Sincerely,

School Principal

1 NCES collects student question data that are required by law (20 U.S.C. § 9622; i.e., race, ethnicity, and socioeconomic status) and provide a context for reporting student performance. These questions are referred to as core questions.

2 The 2014 Sampling Memo (dated 3-25-2013) contains information related to the grades 4 and 12 social studies assessments, which were subsequently dropped.

3 Reports for the TEL cognitive interviews and tryouts were approved in July 2012 as part of clearance for the pilot assessment (OMB #1850-0790 v.35).

4 The time for the cognitive sections is not part of the burden calculation.

5 Education Sciences Reform Act of 2002 (ESRA), National Assessment of Educational Progress (20 USC § 9622).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	NATIONAL ASSESSMENT OF
Author	joconnell
File Modified	0000-00-00
File Created	2021-01-30