Response to OMB Passback

MEMORANDUM June 11, 2015

Date: June 11, 2015

To: Shelly Martinez, OMB

From: Chelsea Owens, NCES

Through: Kashka Kubzdela, NCES

Re: NTPS 2015-16 Full-Scale Response to OMB Passback (OMB# 1850-0598 v.11)

Passback of June 11^th, 2015:

1. The discussion about tailoring nonresponse follow up strategies in SS B3 seems to be primarily or exclusively about schools. The NSCG, cited, is a survey of individuals and not a clustered multi-stage sample as NTPS is. What adaptive design methods is NCES proposing for the NTPS principals and teachers specifically? While the plans do not need to be as elaborate as they were for B&B, BPS and HSLS, we would expect several paragraphs of detail here.

Reply: The 2015 NTPS adaptive design approach will not target principals, as individuals, or teachers. A priority flag, based on the response propensity scores developed by the Census Center for Adaptive Design (CAD), is assigned at the school level. During data collection, the priority flag will be used to move high priority schools to field operations earlier in the process along with schools that have not been assigned a coordinator. Schools in the high priority group generally do not respond until later in the data collection process and ultimately require field intervention. By moving those schools to field contact after the first two mailouts, we will reduce costs by eliminating the third and fourth mailings, which have low probabilities of impacting response, and raise the probability of response by providing the field staff more time to secure the completed questionnaires. Since the principal questionnaires are pursued at the school level and not at the individual level, intervening with the principals falls into this operation. Throughout data collection the cases assigned to field will be reviewed by NTPS staff on a daily basis. The priority flags will be used to direct the Field Representatives on the order to pursue cases. Regarding the references to NSCG -- The response propensity model developed for NTPS was based on the response propensity model developed for NSCG.

Efforts will be focused on obtaining cooperation and improving response rates at the school level for a number of reasons. Past administrations of the Schools and Staffing Survey (SASS) have shown that if cooperation is obtained at the school level, teachers and principals are highly likely to respond. Additionally, evaluation of the schools’ response propensities using the model developed by CAD showed that the nonresponse in past administrations of SASS was driven primarily at the school level. Results showed that schools in special districts are the primary driving force behind low response propensity. Special districts are those that require additional applications or documentation to collect data in their schools. Nearly 80% of the schools with high propensity for non-response reside in these special districts. For this reason, resources will be reallocated to focus heavily on obtaining approvals from these special district schools in order to boost response rates for this group.

Additionally, past administrations of SASS demonstrated that response rates tend to be very low for urban schools. The 2015 NTPS includes a flier that will be sent specifically to urban schools (as determined from frame data on the CCD). This flier will be roughly the size of one half of a standard sheet of paper and is customized to include interesting statistics about city schools and why their participation is important (see Attachment B).

Finally, the NTPS teacher-level response rates are calculated by multiplying response at the school level to the Teacher Listing Form (TLF) by response at the teacher level. In the past this has meant that if the school did not cooperate by not completing the TLF, teachers from that school could not be sampled, which ultimately lowered the teacher response rate. As detailed in this submission, in the 2015 NTPS, TLFs received from sample schools will be supplemented with vendor-purchased teacher lists and a clerical look-up operation utilizing school websites. These methods were tested in the Pilot and showed high levels of comparability to lists obtained directly from schools. This operation will help to improve the overall teacher response rate by allowing sampling teachers from schools that have not submitted a TLF.

2. What is the nonresponse bias analysis plans for the NTPS, specifically of teachers and principals?

Please see the attached “NTPS 2015-16 Full-Scale Nonresponse Bias Analysis Overview.docx”.

3. Where is the discussion of the internet test in Part B, including the specific hypothesis and power analysis?

Reply: The NTPS Pilot Study showed that response rates for the Internet treatment group (for schools and principals) were lower than for the paper treatment group. Anecdotal information from the Pilot test suggested that internet response rates could potentially be increased by adjusting elements of our instruments and contact materials. We have changed the login procedure for the Internet instrument and revised the contact materials from those used in the Pilot. We are proposing to conduct a small-scale experiment of 1,000 schools/principals in the 2015-16 NTPS who will be asked to complete the instrument online rather than on paper. Additionally, because we will be in a full-scale collection, these 1,000 schools will have the option to designate a school coordinator. We hypothesize that the changes to the login procedure as well as the designation of a school coordinator will increase the response rates for the Internet treatment groups to be comparable with the paper respondents. The sample size of 1,000 was developed to detect differences of 5%. Based on the resources that we have for this collection and the information that we learned in the pilot, we believe that a 5% detectable difference with a 95% alpha is appropriate. The sample of 1,000 schools will not be included in the released data file and will instead be used for analysis to inform design decisions for the 2017-18 NTPS. [this description has been added to Part B, section 4.]

4. What specific results from the pilot test caused NCES to propose to reinstate the school coordinator role in the full scale data collection? What is the anticipated difference that the coordinators are hypothesized to make over the pilot design?

Reply: The pilot study was explicitly designed to test the impact of a paper vs. Internet instrument platform and to test the impact of asking for email addresses on the TLFs vs. not asking for this information. Consequently, we deliberately eliminated the school coordinators from the Pilot in order to keep the test treatments robust and clean. In the field, we cannot control whether a school designates a coordinator, and we believe that the uncertainty of that factor would have confounded the results of the experiment. Instead, the schools principals were contacted as if they were the school coordinator including sending all of the packages addressed to the school principal at the school address. We have evaluated the impact of using a school coordinator in past rounds of SASS and have found that having a school coordinator operation greatly increases response rates. We have always anticipated including school coordinators in the 2015-16 Full-Scale NTPS.

2