ED Response to OMB Passback

Phase II of the National Longitudinal Transition Study 2012 Phase II (NLTS 2012)

Response to OMB comments for #1850-0882

1.       “How will NCEE define success? Is any increase in response a success? Any statistically significant increase? Any increase that offsets the additional cost of the incentive?”

Based on the NLTS 2012 sample characteristics, their prior consent refusal, and our experience with sample members in similar age groups, we consider a 10 percent conversion rate an indicator of success. Each of the 60 sample members whose consent for transcript collection we would obtain in the pilot (or 170 sample members if applied to the full 1,700 students who previously refused) represents many more students in our nationally representative sample. We therefore believe the benefits of a 10 percent response outweigh the $600 (pilot only) or $1,700 (full scale) cost of the incentive. To the extent that we find greater or lesser success in converting refusals among some key sample groups (e.g., those with particular disabilities), we can use the information from the pilot to target efforts during the full scale collection to mitigate differential non response and potential for bias. A spillover benefit from the pilot is that the results may help us better tailor procedures for collecting the last school attended by students who transferred during the NLTS 2012 study period.

Defining success for the pilot is challenging due to two factors. First, the sample members targeted are expected to be difficult to obtain responses from – those who explicitly refused consent for transcript collection when surveyed during NLTS 2012 Phase I. They are also primarily students with disabilities and their families, who we suspect may respond at rates different from those of other students and families with whom we have had the most experience. Second, there isn’t a direct comparison or precedent activity to use for predicting a response rate. Response rates for activities with similar structures, such as panel maintenance activities used to obtain address updates for longitudinal study members, provide some insight for the NLTS pilot.  Panel maintenance activities can range from 5 – 30 percent response rates depending on the sample characteristics and incentives offered.  Past experience has achieved the greatest response rates, approximately 30 percent, when using a $10 offered incentive, though we expect a lower response due to the characteristics of the sample noted above.

2.       “Past research has shown that $2 - $5 prepaid incentives are often as effective, if not more, than larger amounts. What led NCEE to choose $10 as the incentive level, especially in light of the low burden associated with the collection?”

While it is true that prepaid incentives have been shown to be effective when compared to larger promised incentives, we decided against prepaid incentives due to our experience using $10 promised incentives on other studies and our expectation of a relatively low response rate. Given that we may observe a response rate of only 10 percent, using a $10 promised incentive would result in lower overall costs than if a $5 prepaid incentive was sent to all sample members. Therefore the total expenditure is anticipated to be lower without the prepaid approach.  Furthermore, ED has experience performing experiments for longitudinal study panel maintenance/address update activities, and found the $10 promised incentive to be effective.  In the Education Longitudinal Study of 2002 (ELS:2002) third follow-up field test, we conducted an incentive experiment on the panel maintenance activity. Half the sample was offered a $10 incentive and no incentive was offered to the other half of the sample. Overall, the $10 treatment group had a higher participation rate (25 percent) than the control group (20 percent, t=1.90, p<.05). This experiment was the basis for offering a $10 incentive for the panel maintenance activity on the full-scale third follow-up for ELS:2002 as well as the High School Longitudinal Study of 2009 and the 2008-12 Baccalaureate and Beyond Longitudinal Study.