NTPS 2015-16 Pilot Test Instruction Experiment Change Request Memo

U NITED STATES DEPARTMENT OF EDUCATION

National Center for Education Statistics

December 5, 2014

MEMORANDUM

To: Shelly Martinez (OMB)

From: Andy Zukerberg, NCES

Through: Kashka Kubzdela, NCES

Re: National Teacher and Principal Surveys (NTPS) 2015-16 Pilot Test Instruction Placement Experiment Change Request (OMB# 1850-0803 v.120)

The National Teacher and Principal Survey (NTPS) Pilot Test was approved in July 2014 (1850-0803 v.105), including two interrelated studies: the main pilot test and an instruction placement experiment. The Pilot test has been underway since October 2014 and the instruction experiment is scheduled to begin in January 2015 and end in April 2015.

The original clearance package described the purpose and method of the instruction placement study:

The purpose of the Internet Questionnaire Design Experiment is to identify the optimal placement, length, wording, and formatting of item wording and instructions in an internet survey. NTPS has many long and technical questions, some of which have a full page of explanation accompanying them. Recent cognitive interviews suggest that respondents are ignoring the instructions leading to measurement error. This experiment will attempt to identify strategies for improving compliance with instructions. This test will be run separately from the other experiments. The elements that will be experimentally varied are: placement of the instructions (before or after the question), format of the instructions (paragraph or list) and the presence of a preface introducing the item. The control for this experiment is a treatment with no instructions. This will allow NCES to compare each of the treatment groups to a no instructions group. In addition to including a subset of planned NTPS teacher questionnaire items, the instruction experiment questionnaire contains several items that have been designed to elicit a specific effect. For example, an item has been added that asks teachers to report the number of test and quizzes they give. Instructions are provided that greatly narrow the definition of test and quizzes, so if teachers read the instruction we should see a significant reduction in reporting compared to the control group which did not have the modifying instruction. These modified items will be the key items for analysis of the experiment.

As noted in the revisions to the NTPS Pilot Test 2014-15 Experiment and Respondent Materials (OMB# 1850-0803 v.115) approved in October 2014, the Census Bureau conducted cognitive testing on the items that were proposed for the instruction experiment. These interviews led to revision and in some cases replacement of items that were proposed for the instruction experiment. Attachment A contains the final set of items to be included in the experiment. Attachment B provides a crosswalk of the items from the original clearance (1850-0803 v.105; July 2014) to the final set provided in this submittal’s Attachment A. Additionally, we have received valuable feedback on the respondent contact materials being used in the main pilot study. Our original plan was to use the same contact materials in the main pilot study and the instruction study, however based on the feedback we are proposing significant revisions to the contact materials for the instruction study (see attachment C for the revised contact materials). This request is to use the revised items and contact materials in the instruction placement experiment.

1990 K Street, N.W., Washington, DC 20006-5650

Our mission is to ensure equal access to education and to promote educational excellence throughout the Nation.