Response Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Evaluation

July 18, 2012

TO: Sharon Mar, Shelley Martinez

FROM: Marsha Silverberg, Yumiko Sekino

RE: Proposed changes to NLTS2012 baseline data collection – responses to comments

We appreciate this opportunity to discuss the remaining issues you raised regarding changes we propose to the NLTS2012 baseline data collection, including the proposed incentives experiment. Below we both provide an initial response to your questions and request some clarification before submitting a revised change package.

1. $5 pre-paid cash incentive-  We see the NHES reference on p. 5’s footnote, but would appreciate if IES could provide more information on how that example is relevant to the NLTS study.  Additionally, a few sentences on the advantages of pre-paid cash methods would be helpful.

• Several commonly cited studies have documented that small cash incentives delivered prior to survey administration attempts builds trust and rapport with respondents and results in larger response rates compared to no incentive (Singer, Groves, and Corning (1999); Church (1993)). For the 2007 Health Tracking Household Survey (HTHS), Mathematica tested the effectiveness of a $5 cash prepayment in an advance mailing for a telephone survey.  All respondents, both those receiving and those not receiving the $5 prepayment, were offered a post payment of $20 for completing the survey. Households that received the cash prepay had a response rate that was two percent higher than those without the prepayment.  However, the prepay households also took 1.5 fewer calls to complete and were more likely to have others in their household complete interviews.  Like, NLTS-2012, multiple eligible household members were interviewed.¹ For these reasons, we believe that the prepayment would be a cost effective approach for NLTS 2012.

• While there have not been many published tests of alternative prepayment amounts, the NHES experiments are most convincing and, we believe, applicable to NLTS 2012. NHES tested $5 vs. $2 prepaid cash incentives in the context of first-time contact with respondents while NLTS 2012 wants to use it to convert soft refusals and families who have failed to respond to our multiple letters and calls. However, much like with the NHES field test, our objective is to use the prepaid cash incentive to get the parent’s attention and buy-in to complete the interview. We already know that the offer or a “post-completion” incentive, alone, has not been successful

• We have examined the pros and cons of prepaid cash vs. other methods. Compared to other forms of prepayment such as debit cards, prepaid cash incentives provide parents with something that is immediately tangible (i.e., immediate reinforcement) and involves no logistical hurdles. In addition, our contractor has investigated and determined that a prepaid cash method is less costly to administer than a prepaid debit card. There would be an additional $3 fee for each debit card, which increases the costs per respondent by an additional 50%.

We will elaborate further on the above points in the revised memo.

2. Schedule-  Can IES clarify what happens after mid-August?  How is IES envisioning implementation after they arrive at the best treatment?

We envision implementing the “best” incentive treatment in a sequential manner:

• First, we would make this offer to all remaining hard-to-reach and soft-refusal respondents in our initial sample (about 4000 parents) who did not participate in the incentives experiment. Making this incentive change in August is strategic, allowing us to get parents’ attention before the school year begins.

• Second, we would use the new incentive structure with cases that enter the status of hard to reach or soft refusal going forward in early fall. We anticipate some of the other new strategies being used (especially the change to the consent process requested under #6 below) will affect the proportion of the sample ultimately reaching soft refusal or hard to reach status. Nevertheless, we currently estimate that approximately 3000 cases will move into these categories (approximately evenly distributed across the sample already released and the new sample to be released as described under #5 below).

3. Letterhead- Can IES please provide the advance letters on Dept. of ED letterhead when submitting non-substantive change.

• When submitting the change, we will submit revised letters on ED letterhead.

4. $10 child gift card-  Can IES clarify whether or not this is a new incentive or was this already approved previously?

• The $10 card for the individual student was previously approved. It is not a new incentive.

5. Non-response bias plan-  Table 1 shows “additional changes under consideration.”  Just for clarification, our preference is to have these items figured out after mid-August so OMB is approving one package versus several.  These 2 items, which we would expect a non-response bias plan to be included, should be sent along with the new incentive plan treatment.

• We understand that OMB would prefer to approve one package rather than two. But the time pressures and number of changes we think are necessary may warrant two. We would like to expedite approval for several procedural changes we want to make now and then submit a separate package for any change to incentives resulting from the experiment.

• While we are ready to carry out the experiment as soon as OMB gives us the green light, realistically it will be almost a month before we can report back formally and we don’t believe we can wait to introduce the other changes. In addition to the plans we have discussed with OMB for converting refusals to mitigate non-response bias, we are also releasing new sample. We would like to implement some of the procedural changes with the new sample right away.

• The changes for which we would like approval at this time include:

1. Run the incentives experiment (random sub-sample of initial refusals and hard-to-contact)

2. Field interviewing in communities with a density of sample members who we have been unable to reach after many attempts (four communities from initial sample and a likely four additional communities for the new sample}

3. Contacting schools for updated parent locating information (initial and new sample)

4. Using ED letterhead for all new mailings (initial and new sample)

5. Revised consent text (see below) (initial and new sample)

• In mid-August, we propose to submit a second package that would include: if warranted, a more detailed proposal to change incentives (based on the results of the experiment), a plan to implement the new incentives (including which groups will be targeted), and the use of a shortened version of the parent interview instrument if we decide to pursue this.

Question: Could you please clarify what you mean by a “non-response bias plan” and what this plan would include? As we indicated in our previous memo, we have very little information with which to do non-responses bias analysis at this point since we have not yet collected baseline data.

6. Changes to the consent process – Our contractor has received IRB approval to modify the consent language/process and we would like to implement these changes immediately.

The changes include:

• Securing parental consent/youth assent for the study interviews at the beginning of the baseline interview and moving consent/assent for other study components that will come later to the end of the baseline interview.

• Deferring language regarding the possible future request for student’s social security number until the follow-up interview, which is when we would make that request.

Attached are examples of the before and after language, which we would also include in the revised change package (see Attachments A and BAppendix L revised parent and student protocols with changes to the consent language).

Clarifying next steps:

• Is it OK to submit two change packages?

• What documents will need to be submitted with the OMB83C change form?

1 Barton, Tom, Karen CyBulski, and Barbara Lepidus Carlson. “Prepays, Promises, and Postpays: Additional Evidence on What Helps Response Rates.” Presentation at the American Association for Public Opinion Research, New Orleans, May 18, 2008