Volume 1 NAEP Main 2015 Wave 1

This document contains supplemental information pertaining to the 2014-2016 National Assessment of Educational Progress (NAEP) system clearance proposal. NAEP is administered by the National Center for Education Statistics (NCES) within the Institute for Education Sciences of the U.S. Department of Education. The National Assessment Governing Board (henceforth referred to as the Governing Board) sets policy for NAEP and determines the content framework for each assessment. The system clearance package was submitted in November 2012 and approved in March 2013 (OMB #1850-0790 v.36). The terms of clearance for OMB approvals state that NCES will publish a notice in the Federal Register allowing a 30-day public comment period on the details of each collection concurrent with submission to OMB.

The 2015 submittal is divided into two waves to meet scheduling and question development requirements. The first wave contains the core,¹ reading, and mathematics descriptions, burden, and survey questionnaires (student, teacher, and school). A subset of grade 4 and 8 mathematics students will receive the KaSA (Knowledge and Skills Appropriate)² mathematics assessments. These students will receive the mathematics survey questions plus three additional survey questions, which are also included in Wave 1. This Wave 1 submittal contains burden information and survey questions (referred to as background or noncognitive questions) for the components of the NAEP 2015 assessments listed in table 1.

Wave 2 will contain the descriptions, burden, and questionnaires as well as information regarding school coordinator activities, including the collection of information on students with disabilities (SD) and English language learners (ELL). The Wave 2 submittal will include: science (paper and pencil); special study mathematics, reading, and science technology-based assessments; science pilot for ICT (Interactive computer tasks) and HOTs (Hands-on tasks) assessments; and special studies (including the National Indian Education Study (NIES) and KaSA study in Puerto Rico) to be administered with the 2015 main NAEP administrations.

Table 1: List of Components in Wave 1

	Component	Part
Student	Grade 4 — Core, Reading, Mathematics, KaSA	1a
	Grade 8 — Core, Reading, Mathematics, KaSA	1b
	Grade 12 — Core, Reading, Mathematics	1c
Teacher	Grade 4 — Background, Education, Training (BET); Classroom Organization and Instruction (COI) - Reading & Mathematics	2a
Teacher	Grade 8 — Background, Education, Training (BET) – Reading; Classroom Organization and Instruction (COI) - Reading Background, Education, Training (BET) – Mathematics; Classroom Organization and Instruction (COI) - Mathematics	2b
School	Grade 4 — School Characteristics and Policies (SCP), Reading, Mathematics, Charter School	3a
	Grade 8 — School Characteristics and Policies (SCP), Reading, Mathematics, Charter School	3b
	Grade 12 — School Characteristics and Policies (SCP), Reading, Mathematics, Charter School	3c

2. Overview of Wave 1 NAEP 2015 Assessments

Much of the following broad overview of the 2015 NAEP assessments was included as part of the 2014–2016 system clearance submittal. The Governing Board determines NAEP policy and assessment schedule, and future Governing Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, the overall methodology and assessment process remains consistent. The NAEP assessments contain two kinds of questions: cognitive (assessment questions that measure student knowledge of an academic subject) and survey (background questions that gather demographic information, as well as classroom instructional procedures). The Governing Board is responsible for approving all of NAEP’s survey questions, as well as the cognitive items.

The main NAEP assessment reports current achievement levels and trends in student achievement at grades 4, 8, and 12 for the nation and, for certain assessments, states, and select urban districts. These assessments follow subject-area frameworks developed by the Governing Board and use the latest advances in assessment methodology. The subject-area frameworks evolve to match instructional practices.

3. Sampling Information

The 2015 Sampling Design Memo³ (appendix D) provides specific sampling information regarding the 2015 NAEP assessments. It includes information on:

Assessment types
Primary Sampling Units Selection
Stratification and Oversampling
Overlap Control of Samples
Substitute Samples
Student Sampling Information
Weighting Requirements

4. Information Pertaining to the 2015 Questionnaires in This Submittal

Student Questionnaires

Core — the first survey question block for all students contains core questions, many of which are related to demographic information.

Mathematics — operational mathematics subject-specific questions are trend questions and some previously piloted new items, at grades 4 and 8, and trend questions at grade 12.

Reading — operational reading subject-specific questions are trend questions and some previously piloted new items, at grades 4 and 8, and trend questions at grade 12.

KaSA (mathematics) — the grade 4 and 8 KaSA survey questions are the same as the mathematics survey questions plus three additional KaSA-related questions. These additional questions were given in KaSA administrations in 2011 and 2013.

Membership of the committees that helped define the cognitive and survey questions pertaining to the 2015 subjects in Wave 1 is provided in this document, in appendix A. All of the assessments covered by Wave 1 will be administered by paper-and-pencil.

Teacher Questionnaires

Teacher questionnaires are comprised of multiple sections, which include:

Background, Education, and Training (BET; also referred to as the teacher core) — the operational BET questions are trend or previously piloted questions.

Classroom Organization and Instruction (COI) — the operational COI questions in the mathematics and reading subjects are trend or previously piloted questions.

The plan is for the teacher questionnaires to be completed online in 2015. If a grade 8 teacher teaches more than one subject, the online questionnaire system will administer the appropriate BET and COI questions to the teacher.

School Questionnaires

School questionnaires are comprised of multiple sections, which include:

School Characteristics and Policies (SCP; also referred to as the school core) – One version of SCP questions will be given at each grade. The SCP questions are comprised of trend or previously piloted questions.

Reading – The operational school questions in reading are trend or previously piloted questions.

Mathematics – The operational school questions in mathematics are trend or previously piloted questions.

Charter School – Charter school questions are comprised of trend questions that were administered in previous assessments. The charter school section is only completed if the school is a charter school.

The plan is for the 2015 school questionnaires to be completed online. Table 2 depicts the composition of the 2015 school questionnaires. The shaded components will be included in Wave 2.

Table 2: School Survey Questionnaire Structures - 2015 NAEP Assessments

Shaded sections are not part of the Wave 1 materials and will be included in Wave 2.

SCP = School Characteristics and Policies; NIES = National Indian Education Study
¹ This section is only completed by Charter schools.
²This section is only administered to schools with students participating in the NIES special study.

Appendix C lists the new questions contained in the various questionnaires (student, teacher, and school).

5. Burden Information for 2015 Wave 1

The nature of NAEP is that samples of students alternate from a relatively low volume in national-level only administration years to a substantial volume increase in state-level administration years. These state-level administration years include one or more assessments that support the reporting of results by state and certain urban districts. In 2015, NAEP will conduct national, states, and select urban district assessments, so the estimated burden is much higher than in 2014 NAEP. The total combined burden for the 2015 Wave 1 materials is provided in table 3.

Table 3: Total Burden for Wave 1

Respondent Category	Number of Respondents	Total Burden Hours
Students	601,000	250,417
Teachers	71,366	30,136
Schools	10,785	5,392
Total	683,151	285,945

Note: School coordinator activities, including pre-assessment activities, along with related burden, will be included in the Wave 2 submittal.

The per respondent and overall burden times(see tables 4 and 5) include the following categories:

Students - Students in fourth, eighth, and twelfth grades complete assessment booklets that commonly contain two 25-minute cognitive blocks; these are followed by two background question sections that require a total of 15 minutes to complete.

The first background question block contains core questions, many of which are related to demographic information. The second background block contains subject-specific questions. Additional student burden is included for time to read directions and distribute test booklets. For paper-and-pencil assessments, this additional burden is estimated at 10 minutes, thus the total burden for students is 25 minutes⁴ for the paper-and-pencil assessments.

Appendix B includes a sample parental notification letter regarding NAEP. The letter will be adapted for each grade/subject combination and the school principal may edit it. However, the information regarding confidentiality and the appropriate law reference will remain unchanged.

Teachers – In 2015, teachers will be asked to complete questions about their teaching background, education, training, and classroom organization and instruction. At fourth-grade, the teacher burden is estimated at 30 minutes because most teachers teach multiple subjects and, therefore, will complete the questionnaire for all subject areas. The eighth-grade teacher burden is estimated to be 20 minutes for most teachers (who only teach one subject area), and 30 minutes for a small percent (10%) who may teach more than one subject. There is no teacher questionnaire for the teachers of students taking the twelfth-grade assessments.

Schools – The school administrators in the sampled schools will be asked to complete a school questionnaire. The school questionnaire burden is estimated to be 30 minutes at all three grades.

Table 4: Wave 1 Questionnaires - Per Respondent Burden Time

Grade	Respondent Category	Time per Respondent
Grade 4	Student Core + Reading	15 minutes
	Student Core + Mathematics	15 minutes
	Student Core + Mathematics (KaSA)	15 minutes
	Student Directions and Set-up	10 minutes
	Teacher Questionnaire	30 minutes
	School Questionnaire	30 minutes
Grade 8	Student Core + Reading	15 minutes
	Student Core + Mathematics	15 minutes
	Student Core + Mathematics (KaSA)	15 minutes
	Student Directions and Set-up	10 minutes
	Teacher Questionnaire (single subject)	20 minutes
	Teacher Questionnaire (more than one subject)	30 minutes
	School Questionnaire	30 minutes
Grade 12	Student Core + Reading	15 minutes
	Student Core + Mathematics	15 minutes
	Student Directions and Set-up	10 minutes
	School Questionnaire	30 minutes

Table 5: Estimated Burden for NAEP 2015 Questionnaires Contained in Wave 1.

Subjects	# of Students	Avg. time per respondent (minutes)	Student Burden (hours)	# of Teachers	Avg. time per respondent (minutes)	Teacher Burden (hours)	# of Schools	Avg. time per respondent (minutes)	School Burden (hours)
4th Grade
Reading, Mathematics, Mathematics (KaSA)	284,000	25	118,333	34,387	30	17,194	5,731	30	2,866
8th Grade
Reading, Mathematics, Mathematics (KaSA)	284,000	25	118,333	36,979	20 or 30¹	12,943	4,622	30	2,311
12th Grade
Reading, Mathematics	33,000	25	13,750	-	-	-	431	30	216
Total	601,000	N/A	250,417	71,366	N/A	30,136	10,785	N/A	5,392
Total number of respondents: 683,151					Total burden hours: 285,945

Note: Due to rounding, some totals may differ slightly from sum of subtotals.

¹ It is estimated that 90% of the grade 8 teachers teach one subject and, thus, have 20 minutes of burden, while about 10% of the grade 8 teachers teach more than one subject and, thus, have 30 minutes of burden.

6. How, by Whom, and for What Purpose the Data Will Be Used

The purpose of NAEP is to collect and report assessment data on student achievement in the subject areas assessed for use in monitoring educational progress. In addition to reporting overall results of student performance and achievement, NAEP also reports student performance results for various subgroups of students and on various educational factors. The Governing Board sets guidance for what is asked in the questions. NCES is responsible for developing the questions and for selecting the final set of questions. The questions are designed to (a) provide the information for disaggregating data according to categories specified in the legislation,⁵ (b) provide contextual information that is subject specific (e.g., reading, mathematics) and has an impact and known relationship to student achievement, and (c) provide policy-relevant information specified by the Governing Board. An expanded description of the purposes of the data is provided in the system clearance submittal.

7. Estimates of Costs to the Federal Government

Table 6 provides estimates for the Wave 1 portion of the 2015 administrations.

Table 6: Administration Cost Estimates (Wave 1 subjects)

Activity	Provider	Estimated Cost
Printing, packaging, and distribution phases of the administration including: prepare and package the assessment and all auxiliary materials distribute assessment booklets and materials to the test administrators for each school	Pearson (the Materials, Distribution, Processing and Scoring [MDPS] contractor)	$11.6 million
Field administration including: administration of assessments and data collection from the field	Westat (the Data Collection contractor and the NAEP Support and Service Center [NSSC] contractor)	$22.7 million
Technology support and maintenance including web operations maintenance costs related to the support of the technology-based assessments	Fulcrum IT Services	$ 289.5 thousand
Total		$ 34,589,500

Appendix A: Committee Lists

NAEP Survey Questionnaires Standing Committee

Name Affiliation

Gerunda Hughes Howard University

David Kaplan University of Wisconsin-Madison

Henry Levin Teachers College, Columbia University

Stanley Presser University of Maryland

Leslie Rutkowski Indiana University Bloomington

Rob Santos Urban Institute, Washington, DC

Norbert Schwarz University of Michigan

Jonathon Stout Lock Haven University

Roger Tourangeau Westat, Rockville, MD

Ting Yan University of Michigan Institute for Social Research

NAEP Mathematics Committee

Name Affiliation

Lourdes Latorre Alvarado Escuela Segunda Unidad Botijas #1,Orocovis, PR

Jennifer Alvarez Sultana Elementary School, Ontario CA

Daniel Chazan University of Maryland

Herb Clemens Ohio State University

Carl Cowen Indiana University-Purdue University

Julie Guthrie Texas Education Agency, Austin, TX

Kathleen Heid Penn State University

Mark Howell Gonzaga College High School, Washington, DC

Russ Keglovits Nevada Department of Education, Carson City, NV

Carolyn Maher Rutgers University

Michele Mailhot Maine Department of Education

Brian Nelson Curtis Corner Middle School, Wakefield, RI

Matthew Owens Spring Valley High School, Columbia, SC

Carole Phillip Alice Deal Middle School, Washington, DC

Ann Trescott St. Michael’s School, Poway, CA

NAEP Reading Committee

Name Affiliation

Marilyn Adams Brown University

Peter Afflerbach University of Maryland

Patricia Alexander University of Maryland

Margretta Browne Richard Montgomery High School, Silver Spring, MD

Julie Coiro The University of Rhode Island

Valerie Harrison Claflin University, Orangeburg, SC

Karen Malone Fort Wingate High School, Fort Wingate, NM

Pamela Mason Harvard Graduate School of Education

Margaret McKeown University of Pittsburgh

P. David Pearson The University of California, Berkeley

Monica Verra-Tirado Florida Department of Education

Karen Wixson University of North Carolina

Victoria Young Texas Education Agency, Austin, TX

Zynia Zepeda Gadsden Elementary, Gadsen, AZ

Appendix B: Sample Parent/Guardian Notification Letter

NAEP (NAEP Assessment Year) PARENT/GUARDIAN NOTIFICATION LETTER

(School Letterhead)

(Insert Date Here)

Dear Parent or Guardian:

We are pleased to notify you that (school name) has been selected to represent schools across the nation by participating in the National Assessment of Educational Progress (NAEP). First administered in 1969, NAEP is the largest nationally representative assessment of what students know and can do in various subject areas. It is administered by the National Center for Education Statistics, within the U.S. Department of Education. NAEP is different from state assessments because it provides a common measure of student achievement across the country. The results of NAEP are released as The Nation's Report Card, which provides information about student achievement to educators, parents, policymakers, and the public.

In our school, the NAEP assessment will be given on (date) in (subject). Your child (may be/has been) selected to take the assessment. In addition to subject area questions, students will be asked some questions about themselves and their educational experience. These questions provide contextual information for the assessment, as well as information that may be related to students’ learning. If you would like to view sample subject area and student questions, please visit http://nationsreportcard.gov/parents.asp.

It will take about (assessment time) minutes for most students to participate in the assessment. The results are completely confidential (in accordance with the Confidential Information Protection provisions of Title V, Subtitle A, Public Law 107-347), and the information provided will be used for statistical purposes only. Your child’s grades will not be affected. Your child may be excused from participation for any reason, is not required to complete the assessment, and may skip any question. While NAEP is voluntary, we depend on student participation to provide an accurate measure of student achievement that will inform improvements in education. Your child will represent many other students, so participation is very important. However, if you do not want your child to participate, please notify me in writing by (date).

There is no need to study in preparation for NAEP. We do, however, ask parents to encourage their children to do their best and get plenty of rest the night before the assessment. A brochure that explains what participation in NAEP means for you and your child is available at http://nces.ed.gov/nationsreportcard/pdf/parents/2012469.pdf. Should you have questions, please contact me at (telephone number) or at (e-mail address).

We are excited that our school will be participating in NAEP and pleased that your child (may be/has been) selected. We know that (school name)'s students will help us show what our nation’s students know and can do.

Sincerely,

School Principal

Appendix C: List of New/Revised Items (Mathematics and Reading)⁶

List of New or Revised Mathematics Items

Questionnaire	Grade	New Operational Items
Math Student	4	9
Math Student	8	12
Math Teacher	4	2, 13c
Math Teacher	8	2, 7, 13c
Math School	4	5, 6a-d, 7, 8a-c, 9i
Math School	8	5, 6a-d, 7, 8a-c, 12i

Note: Revised items are considered “new” items because they have not been administered in previous operational questionnaires. There are no new questions for Grade 12 mathematics.

List of New or Revised Reading Items

Questionnaire	Grade	New Operational Items
Reading Student	4	1, 2, 17
Reading Student	8	1c, 5, 7f, 8a-c, 9d, 10e
Reading Teacher	4	2, 7g, 10, 14a-h
Reading Teacher	8	2, 6e-g, 7f, 10, 14a-h
Reading School	4	1c, 1e, 1g, 2, 3a-b, 4, 5a-b, 6i, 8a-8f, 9a-f
Reading School	8	1c, 1e, 1g, 2, 3a-b, 4, 5a-b, 6i, 8a-8f, 9a-f

Note: Revised items are considered “new” items because they have not been administered in previous operational questionnaires. There are no new questions for Grade 12 reading.

1 NCES collects student question data, referred to as core questions, that are required by law (20 U.S.C. § 9622; i.e., race, ethnicity, and socioeconomic status) and provide a context for reporting student performance.

2 KaSA study enables more accurate measurement of student abilities in the lower region of the math scale. Wave 1 includes the versions to be administered in the U.S. mainland while Wave 2 will include the versions to be administered in Puerto Rico.

3 The draft 2015 Sampling Memo (dated 2-26-2014) contains the most recent sampling information at the time of this submittal.

4 The 25-minute burden calculation is comprised of 15 minutes for survey questions and 10 minutes for set up and directions. The time for the cognitive sections is not part of the burden calculation.

5 Education Sciences Reform Act of 2002 (ESRA), National Assessment of Educational Progress (20 USC § 9622).

6 New/revised items have been pretested in cog labs prior to operational use. Cog lab reports for these reading and mathematics items were previously submitted to OMB.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	NATIONAL ASSESSMENT OF
Author	joconnell
File Modified	0000-00-00
File Created	2021-01-30

Volume 1 NAEP Main 2015 Wave 1