NAEP Technical Documentation on the Web Survey 2012

This material is being submitted under the generic U.S. Department of Education clearance agreement (OMB # 1880-0542). This generic clearance provides for the Department to carry out collections of routine customer feedback.

The National Assessment of Educational Progress (NAEP) is a federally authorized survey of student achievement at grades 4, 8, and 12 in various subject areas, such as mathematics, reading, writing, science, U.S. history, civics, geography, economics, and the arts. NAEP is administered by the National Center for Education Statistics (NCES), part of the Institute for Education Sciences, in the U.S. Department of Education.

NCES proposes to conduct a brief online survey of the NAEP data user community to gather feedback on the current Technical Documentation on the Web (TDW) portion of the NAEP website and evaluate the recommendations for future improvements.

Study Purpose, Design, Content, and Data Collection Process

The TDW is a section of the main NAEP website that is written for researchers in the fields of educational measurement and testing who are interested in the technical procedures and methods of NAEP¹. The published web pages contain a wealth of technical information and data tables that relate to NAEP operational assessments. Information contained in the TDW is organized around the following topics pertaining to the creation and distribution of NAEP materials: Instruments, Sample Design, Data Collection, Processing Assessment Materials, Scoring, Database, Weighting, and Analysis and Scaling. The site is structured in such a way that users are taken to a main page where they can then drill down on each topic to obtain detailed information specific to a particular assessment.

Given that the site was designed in 2000 and web design has since progressed, it is now appropriate to consider whether a different structure would be more amenable to our users. A common way to determine appropriate structure on a website involves surveying a targeted audience to ascertain how the site might be best arranged to meet their needs. This survey will gather and analyze audience feedback on the TDW.

The survey has been designed with selected-response and open-ended questions that will probe how people use the current TDW content and whether the current format is helpful for people locating and using that content. Data collection will specifically focus on overall user satisfaction with the website and its content, breadth and depth of use, and general relevance and usefulness of specific search elements. The survey will also provide an opportunity for users to suggest ways in which the website might be improved so that they can easily obtain the technical information they need to support their work.

Because the TDW is a public site and does not include registered users, the targeted sample for the survey will comprise the members from the major technical groups that are associated with NAEP. The major technical groups are as follows:

NAEP Design and Analysis Committee (DAC) members – The DAC focuses primarily on methodologies and statistical quality for NAEP, and provides technical advice.
NAEP Validity Study (NVS) Panel members – The NVS, a broadly representative group of the NAEP research community, is an independent advisory group to NCES that performs validity studies and advises on validity issues.
American Educational Research Association (AERA) NAEP special interest group members – The AERA NAEP special interest group was established to facilitate secondary analysis of NAEP’s ongoing surveys regarding U.S. students’ knowledge, attitudes and experiences in diverse learning areas.

NAEP State Coordinators – NAEP State Coordinators serve as the liaison between the State Education Agency and NAEP. In performing this function, State Coordinators analyze and report NAEP data, and coordinate the use of NAEP results for policy and program planning.
Researchers with secondary research licenses – NAEP data are used by researchers and educators who have diverse interests and varying levels of analytical experience. Licenses are granted to researchers to perform secondary analysis on micro-level data.
NAEP Technical Documentation working group – The working group is comprised of the contractors who produce, and frequently use, the NAEP TDW.

The combined number of members/individuals from the above listed audiences is approximately 200. In order to maximize population coverage, the survey will be distributed to all members of these targeted user groups. Based on typical response rates², we anticipate a response rate of 40%, or 80 responses. This should yield a sufficiently robust sample to support the planned analyses.

The survey (shown in Appendix B) will reside on an independent website that will not require the participants to provide either their name or a password, thus assuring the confidentiality of their responses. An e-mail (shown in Appendix A) containing a link to the web-based survey will be sent by the Educational Testing Service (ETS; see section 3 for more information) to the targeted sample identified above. The survey period will be designated as two to three weeks in the fall of 2012. In order to ensure that the final response rate is at least 40%, a reminder e-mail will be sent to all original recipients within the first week of the response window.

Survey responses will be collected and tabulated by Fulcrum IT (see section 3 for more information) and analyzed and reported by ETS. Analyses will include frequencies and descriptive statistics for quantitative questions as well as qualitative analyses of answers to open-ended questions. Based on the survey indications of user practices and preferences, ETS will then draft specific recommendations for how the TDW should be restructured to meet the explicit needs of the user community. The results of the survey will be used internally by NCES to evaluate the satisfaction with the current TDW and the appropriateness and feasibility of the recommended improvements.

Consultations outside the Agency

Educational Testing Service (ETS) serves as the Design, Analysis, and Reporting contractor on the NAEP project. ETS staff have been involved in the development of the survey and will be responsible for the analysis and reporting of the survey results.

Fulcrum IT is the NAEP contractor responsible for the development and ongoing support of NAEP websites and computer-based assessments. Fulcrum IT will host the survey website and gather the survey results.

Confidentiality

Potential respondents will be notified that their participation is voluntary and that their answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002 (20 U.S.C. §9573)]. No individual respondent identifying information will be collected or included in the survey database.

The following text will precede the actual survey questions as part of the web-based survey:

Your participation is voluntary and your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law [Education Sciences Reform Act of 2002, 20 U.S.C §9573].

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary survey is 1880-0542. The time required to complete this survey is estimated to average 10 minutes, including the time to review instructions, gather the information needed, and complete and review the survey. If you have any comments concerning the accuracy of the time estimate or suggestions for improving this survey, please write to: U.S. Department of Education, Washington, DC 20202–4537. If you have any comments or concerns regarding the status of your individual submission of this survey, write directly to: NAEP/2012 TDW Survey, National Center for Education Statistics, 400 Maryland Avenue, SW, Washington, DC 20202.

Estimated Burden Hours

The initial and follow-up e-mails will be distributed to a maximum of 200 potential respondents. The estimated burden to read both e-mails is four minutes per person. Of the 200 potential respondents, we estimate that approximately 40% will respond and that the survey can be completed in 10 minutes or less. Table 1 provides the burden information.

Table 1. Estimate of TDW Survey Burden

Respondent/Activity	Hours per respondent	Number of respondents	Total hours

Read initial and follow-up notification e-mails	.067	200	14
Respond to survey	.167	80*	14
Total		200	28

*Subset of the previous line

Cost to Federal Government

The total estimated cost to the federal government for conducting this survey is $27,862 for the survey design and development, implementation, data analysis, and reporting.

No incentives or stipend will be provided to the survey respondents.

Schedule

The following table provides the schedule of milestones and deliverables:

Table 2. Schedule

Activity	Date
Develop survey and identify target respondents	August 2012
Submit survey to OMB for clearance	September 2012
Distribute survey and collate responses	September – November 2012
Summarize responses and develop recommendations	October – December 2012
Provide report to NCES	January 2013

Appendix A – Initial Notification E-mail

Given your experience with NAEP, on behalf of the National Center for Education Statistics (NCES), Educational Testing Service (ETS) is seeking your help in revising an important NAEP resource. As you may know, the NAEP Technical Documentation on the Web (TDW) section of the main NAEP website is written for researchers in the fields of educational measurement and testing who are interested in the technical procedures and methods of NAEP. To ensure that the TDW is of most benefit to our audience, ETS is reaching out to those who may potentially use the site. If you use the TDW, we ask that you please take approximately 10 minutes of your time to complete this brief voluntary survey at [LINK].

Please note that your name and contact information will not be used for any other purpose and will not be maintained in the data file. The survey will be active through [DATE].

Your contributions will greatly aid our efforts to ensure the website is helpful to your research needs.

If you have any questions about the survey, please contact me. Thank you very much for your cooperation.

Robert Finnegan

Educational Testing Service

Director of NAEP Reporting

[email protected]

Appendix B – TDW Survey Content

The Technical Documentation on the Web (TDW) section of the main National Assessment of Educational Progress (NAEP) website is written for researchers in the fields of educational measurement and testing who are interested in the technical procedures and methods of NAEP. To ensure that the TDW is of most benefit to you, please take a few moments to complete this brief survey. Your contributions will greatly aid our efforts to ensure the website is helpful to your research needs.

According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this voluntary survey is 1880-0542. The time required to complete this survey is estimated to average 10 minutes including the time to review instructions, gather the information needed, and complete and review the survey. If you have any comments concerning the accuracy of the time estimate or suggestions for improving this survey, please write to: U.S. Department of Education, Washington, DC 20202–4537. If you have any comments or concerns regarding the status of your individual submission of this survey, write directly to: NAEP/2012 TDW Survey, National Center for Education Statistics, 400 Maryland Avenue, SW, Washington, DC 20202.

How many times in the past year have you visited the TDW?

 None

 1-5

 6-10

 11-20

 More than 20

What information were you looking for in the TDW? Please select all sections that apply.

 Instruments
 Sample Design
 Data Collection
 Processing Assessment Materials
 Scoring
 Database
 Weighting
 Analysis and Scaling

 Other: ______________________

Were you able to find information within the TDW successfully?

 Always

 Usually

 Sometimes

 Rarely

 Never

How satisfied were you with the following aspects of the TDW?

How often do you use the Table of Contents to find information in the TDW?

 Often

 Sometimes

 Never

[Based on the response to the previous question, one of the following questions would be presented:]

[If Often or Sometimes:] How is the Table of Contents helpful?

[If Never:] Why do you not use the Table of Contents?

How often do you use the “Search Technical Documentation” feature in the TDW?

 Often

 Sometimes

 Never

[Based on the response to the previous question, one of the following questions would be presented:]

[If Often or Sometimes:] How is the Search feature helpful?

[If Never:] Why do you not use the Search feature?

Would a list of “top searches by other users” be helpful?

 Yes

 No

Have you had any problems printing from the TDW?

 No problems

 Some problems

 I do not print from the TDW.

[The following question would be presented only if the previous answer is Some Problems:]

Please describe the problems you have encountered when printing from the TDW.

Please provide any recommendations you have to improve the TDW.

Please provide any other comments you have about the TDW.

Appendix C – Follow-up Notification E-mail

We recently sent you an e-mail requesting your completion of a survey for the NAEP Technical Documentation on the Web (TDW) section of the main NAEP website. If you have completed the survey, thank you for your feedback. If you have not yet completed the survey, please see the instructions below. Your contributions will greatly aid our efforts to ensure the website is helpful to your research needs.

Given your experience with NAEP, on behalf of the National Center for Education Statistics (NCES), Educational Testing Service (ETS) is seeking your help in revising an important NAEP resource. As you may know, the TDW section of the main NAEP website is written for researchers in the fields of educational measurement and testing who are interested in the technical procedures and methods of NAEP. To ensure that the TDW is of most benefit to our audience, ETS is reaching out to those who may potentially use the site. If you use the TDW, we ask that you please take approximately 10 minutes of your time to complete this brief voluntary survey at [LINK].

Please note that your name and contact information will not be used for any other purpose and will not be maintained in the data file. The survey will be active through [DATE].

If you have any questions about the survey, please contact me. Thank you very much for your cooperation.

Robert Finnegan

Educational Testing Service

Director of NAEP Reporting

[email protected]

1 The TDW website can be found at: http://nces.ed.gov/nationsreportcard/tdw/.

2 The average response rate for e-mailed surveys is 40% (http://www.utexas.edu/academic/ctl/assessment/iar/teaching/gather/method/survey-Response.php).

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Background Cog Lab OMB Submission V.1
Subject	NAEP BQ
Author	Donnell Butler
File Modified	0000-00-00
File Created	2021-01-31