NAEP Vol. 1

NATIONAL ASSESSMENT OF

EDUCATIONAL PROGRESS

VOLUME I

SUPPORTING STATEMENT

FOR

WAVE 3 OF 2007 SUBMITTAL

(PART OF 2005-2007 SYSTEM CLEARANCE PROPOSAL

OMB# 1850-0790)

Enhanced Background Questionnaire (Student)

Geo-coding Survey (School, State Personnel)

September 21, 2006

Explanation and Burden Information for This Submittal (Wave 3 for 2007)

This document contains supplemental information pertaining to the 2005-2007 NAEP System Clearance proposal (submitted on December 15, 2003). OMB approval for the System Clearance submittal was received on 4/27/04 (OMB # 1850-0790). The terms of clearance for the OMB approval state that each subsequent submittal activity under the System Clearance is to be submitted to OMB.

This submittal contains burden information and the actual background and survey questions for the following components of the 2007 assessments:

• Enhanced Background Questionnaires (related to socioeconomic status) (Student)

• Geo-coding Component and Survey (School and state personnel)

These specific instruments are the third group (Wave 3) of instruments submitted for approval for usage in 2007. Two prior (Waves 1 and 2) submittals included the Reading/Writing and Mathematics operational questions, the Reading and Mathematics pilot questions, the National Indian Education Study (NIES) questions, and the Students with Disabilities/English Language Learners (SD-ELL) questions.

The burden estimates for this portion of the overall 2007 assessment are:

Students

EBQ 18,000 students at 5 minutes each = 1,500 hours

School & State Personnel

Respondents for the geo-coding component and survey pilot test may be personnel at the NAEP-sampled-school, district, or state level depending on the location of student records within a state. The maximum number of participants will be 320. This number assumes one participant representing each school sampled for the geo-coding component and survey pilot test and that no schools refuse to participate.

Users

Respondent Sample 320

Estimated Burden (Hours) per Respondent 3.67

Total Burden (Hours) 1174.4

Total hours of burden for users include the time it takes to create a student address file, read the geo-coding software user’s manual (included in Volume II), attend a training (optional), geo-code student addresses using the geo-coding software, and complete the one-time only user’s survey (included in Volume II), and return materials. The estimated time for each of these steps is based on information gathered from a small-scale pilot conducted with nine users. The estimated time allotments are presented below:

Task	Time (Hours)
1. Gathering the addresses	.5
2. Reading the user’s manual	.5
3. Training (optional)	1
4. Geo-coding	1.5
5. Survey	.167

These numbers also assume a maximum number of about 180 addresses per school requiring geo-coding. Given that the total burden hours for the maximum number of participants are estimated to be 1174.4 hours, student address geo-coding will take 1.2 minutes per address.

Overall burden across all groups of respondents is presented below. This table includes the burden for both the EBQ and the geo-coding process.

Estimated Respondent Burden for 2007 Assessments

Contained in this Submittal (Wave 3)

Year	Subjects/ Questionnaire	# of Students	Student Burden (in hours)	# of Teachers	Teacher Burden (in hours)	# of Schools/State Staff	School Burden (in hours)	SD/ELL (# of school personnel	SD/ELL Burden (in hours)
2007
	EBQ	18,000	1,500	N/A	N/A	N/A	N/A	N/A	N/A
	Geo-coding (Total)	N/A	N/A	N/A	N/A	320	1174.4	N/A	N/A
	Total	18,000	1,500	N/A	N/A	320	1174.4	N/A	N/A

Overview of NAEP 2005-2007 Assessments

The following broad overview of the 2005-2007 NAEP assessments was submitted as part of the initial systems clearance submittal. As previously noted, the National Assessment Governing Board (NAGB) determines NAEP policy and assessment schedule, and future Board decisions may result in changes to some aspects of an assessment (e.g., which subjects are assessed in which years). However, overall methodology and assessment process will remain constant. In the 2007 assessment year, questionnaires will be administered to students at grades 4, 8, and 12; to teachers at grades 4 and 8; and to school administrators at grades 4, 8, and 12.

The 2007 assessments consist of:

• National and state operational assessments in reading and mathematics at grades 4 and 8, a national and state assessment in writing at grade 8, and a national only assessment in writing at grade 12. Operational mathematics assessments, at grades 4 and 8, will also be administered in Puerto Rico.

• pilot assessments for reading at grades 4, 8, and 12; for mathematics at grades 4 and 8 (including Spanish versions for the Puerto Rican administration).

• a pilot of enhanced background questionnaire (EBQ) administered to subsets of students at grades 4, and 8.

• a pilot of geo-coding software and procedures in a subset of schools.

• an expanded sample of American Indian/Alaskan Native (AI/AN) students selected to take the mathematics and reading assessment at grades 4 and 8. In addition, specific questions about the inclusion of native languages and cultural perspectives in their curriculum will be included in a separate NIES survey.

How, by Whom, and for What Purpose the Data Will be Used

In the original request for system clearance, NCES asked for approval of the instruments to be used to gather data from the 2005-2007 national and state assessments. This submittal applies to the third wave of instruments — the EBQ and geo-coding survey — that will be submitted for the 2007 assessments.

Given that the purpose of NAEP is to gather data on the achievement of students in the subject areas assessed for use in monitoring education progress, and because of the program's increasing visibility, it is incumbent on the program to develop the most reliable and valid instruments possible. To do so, NAEP employs four strategies:

1. Small-scale pilot testing of new materials and test administration techniques;

2. Pilot testing items to determine which items best measure the constructs under consideration;

3. Field testing of operational assessments to accommodate the mandated six-month reporting; and

4. Full-scale operations.

Questionnaire development follows the same pattern as that of cognitive item development, although we tend to field test fewer items with less duplication and use the resulting data to refine the questions. Guidance for what is asked is set by the National Assessment Governing Board. NCES develops the questionnaires, which NAGB then approves for submission to OMB in a two-stage process. NAGB approves the questionnaires prior to pilot testing, and then again after pilot testing using the data to select the final set of questions. The questions are designed to provide the information for disaggregating data according to categories specified in the legislation, to provide contextual information that is subject specific (e.g. reading, mathematics) and has an impact and known relationship to achievement, and to provide policy relevant information specified by NAGB.

Description of the 2007 Socioeconomic Status Indicator Study

The National Center for Educational Statistics (NCES) of the U.S. Department of Education is planning a pilot test for 2007 of a new method for measuring socioeconomic status. Collecting more valid and reliable information on socioeconomic status will greatly enhance the interpretation and reporting of socioeconomic status results and, in turn, NAEP's ability to inform education policy. The 2007 pilot test will advance the development of a new measure of socioeconomic status for NAEP by allowing the program to 1) assess the validity of new socioeconomic status background items, which will be collected from students in the EBQ, 2) assess the validity and feasibility of using Census data as a measure of students’ socioeconomic status , 3) assess the process for incorporating Census data into the existing NAEP framework, and 4) develop a valid composite measure of socioeconomic status that may incorporate some combination of student and Census information.

Subsamples of approximately 6,000 students per grade will be administered the enhanced background questionnaire (EBQ) containing the set of six (grade 4) or seven (grades 8) new socioeconomic status background questions. Students at grades 4 responding to the EBQ will be selected in schools located in the 34 counties where the American Community Survey¹ (ACS) will have gathered significant amounts of demographic information as of May 2007. Students at grade 8 responding to the EBQ will be students enrolled in NAEP schools who are either in one of these 34 early ACS counties or involved in testing under the NCES ECLS-K program. The connection between the EBQ and ECLS-K is important for validating the potential new measure of socioeconomic status because ECLS-K provides both parent questionnaire responses regarding socioeconomic status and address information from all participants. Address information from the participants can be linked to Census community identifiers which can then be linked to Census data. (See description below of geo-coding procedures being pilot tested in 2007.) The parental reports on socioeconomic status will be compared to the students’ responses to the EBQ.

As indicated above, one of the goals of this project is to assess whether Census data could be used to create a valid and unobtrusive measure of socioeconomic status for NAEP. Linking Census data to NAEP examinees and item responses involves the development of a set of new procedures and new geo-coding software. Geo-coding is the process of linking an individual address to a Census geographic region (i.e., tract, block group, ZCTA) which can then be linked to aggregate data for that region collected by the Census Bureau. Since this would be a new process for NAEP, part of the 2007 pilot test will be to test these procedures and collect information from participants, who will be school, district or state personnel, on their experiences to refine and streamline the process in the future.

Linking student achievement data from NAEP to Census data would involve a modification of the standard e-filing system² that NAEP currently employs. People at the state, district, and school levels who currently do e-filing will use new geo-coding software to link a student’s address to a Census block group which will then be linked to associated Census data. In order to accomplish this, new geo-coding software has been developed that allows a user to input a set of addresses, associate the addresses with geo-codes, and return the file with only the student ID and the community identifier included.

In 2007, this file will be sent to the appropriate NAEP contractor who will link the student ID to NAEP booklet IDs. The student ID’s will be stripped from the file. This file, which now contains NAEP booklet IDs and Census block group identifiers, will then be sent to the U.S. Bureau of the Census, where the appropriate socioeconomic status census data will be appended to the file. The Census Bureau will then suppress the block group identifiers, leaving a file with only the NAEP booklet ID and the Census data. No block group identifiers or student identifying information will appear in the final NAEP data files.

For the 2007 pilot test of geo-coding, data provided by the Census will only be available to NCES and NAEP contractors to examine the feasibility of creating more reliable and valid socioeconomic status indicators. The Census data will not be used for reporting NAEP results and will not be included on public-access NAEP data files.

Information for the 2007 Socioeconomic Status Indicator Study Materials Contained In This Submittal

3a. Pilot Enhanced Background Questionnaire (EBQ)

The Enhanced Background Questionnaire (EBQ) is a separate questionnaire that will be administered to a subsample of students immediately following the administration of the NAEP cognitive assessment (which also include a student demographic questions block and a subject-specific background questions block). A grade 4 EBQ and a Grades 8 EBQ will be administered. These questionnaires are being piloted in 2007 for possible inclusion in future NAEP assessments.

3b. Geo-Coding Survey

The geo-coding survey is a survey that will be administered to the school, district or state volunteers who use the new geo-coding software as part of the 2007 pilot. The survey is intended to evaluate the usability of the software and to assess the feasible of incorporating the geo-coding procedures and software into routine NAEP operations. The geo-coding survey will only be administered once during the 2007 pilot testing of the software and procedures to inform refinements to the process.

3c. Geo-Coding User’s Manual

The Geo-coding User’s Manual is also included in Volume II of this OMB submission. The manual explains how to use the geo-coding software.

1 In 2010, the ACS will provide on an annual basis the information that is currently collected every ten years by the Census long-form, so in the future this will be the best source of demographic information about the neighborhoods where NAEP examinees reside.

2 “e-filing” refers to schools or states submitting electronic student enrollment files to the NAEP sampling contractor. These files are used to select the student samples and to pre-assign students’ NAEP identification numbers and booklets.

8