Incentive Memo

M EMORANDUM

P.O. Box 2393

Princeton, NJ 08543-2393

Telephone (609) 799-3535

Fax (609) 799-0005

www.mathematica-mpr.com

TO: Yumiko Sekino and Marsha Silverberg

FROM: John Burghardt, Anne Ciemnecki, and Holly Matulewicz DATE: 7/10/12

NLTS-125

SUBJECT: Request for Approval of NLTS 2012 Baseline Incentive Experiment

and Field Locating in Selected Districts

This memo requests several changes to the plan for implementation of baseline parent data collection in the National Longitudinal Transition Study (NLTS) 2012, Phase I (see Table 1). To date, we have completed 5,500 of 12,000 interviews expected and believe we need to implement significantly different strategies if the data collection is to be successful. In particular, we would like the option of increasing respondent payments and initiating field locating efforts in a limited number of study sites on a pilot basis.

1. Overview of NLTS 2012

NLTS 2012 is a longitudinal study focused on the educational experiences of youth with disabilities who were between the ages of 13 and 21 in December 2011 and their transition from school. The main objectives of the study are to describe the background, secondary school experiences, transition, postsecondary experiences, and outcomes of youth who currently have an individualized education plan (IEP) (and therefore receive special education services under the Individuals with Disabilities Education Act (IDEA)). The study will compare this group with two other groups who are part of the NLTS 2012 sample: (1) youth who have no identified disability, and (2) youth who do not have an IEP but who have a condition that qualifies them for accommodation under Section 504 of the Vocational Rehabilitation Act of 1973. Ultimately, we will also compare the NLTS 2012 IEP sample to similar cohorts of youth with an IEP who were studied in the past.

Key features of the baseline data collection include:

• Student sample has been selected from approximately 400 participating LEAs (out of a total of 572 LEAs selected).

• Parent consent and baseline survey are completed by phone, with digital recording of oral consent at the same time basic data about the student and household are obtained.

• OMB has approved an incentive of $20 for each respondent who completes the parent interview.

• Subsequent to the parent consent and interview—either in the same phone call or another one—we obtain student assent and conduct a baseline interview with the student.

• There are web surveys for the student’s principal, Math or Language Arts teacher, and, if applicable, Special Education teacher, that all follow after obtaining parent consent.

2. Current Status of Parent Interviewing and Targeting Recommendations

Our requests, in particular to increase respondent payment and initiate field locating, focus on the parent baseline interview. The parent provides consent for the student to participate, and is thus the foundation for access to all other key sources of data for the study. Given the difficulty getting through to many parents and the substantial number of parents declining to be interviewed, we are concerned both about our ability to acquire sufficient sample to support analysis for key subgroups and about potential nonresponse bias

The number of parent interviews is well below our target of 12,000. We have released approximately 18,000 cases for interviewing and 5,500 cases have been completed. Achieving our target samples would result in an unweighted response rate at the student level of approximately 66 percent.

Table 2 shows target samples for different groups of key interest for the study, number of sample released for parent interviews, and the number and percent of parent interviews completed by July 1, 2012.¹ It reveals that:

• Completion rates are higher for students with an IEP (31.2 percent) than for students with no IEP (26.7 percent)

• There are some differences across the 12 disability categories. Within the IEP group, completion rates range from nearly 33-38 percent (students with autism, visual impairment, and intellectual disability) to 27-28 percent (students with specific learning disability, speech or language impairment, emotional disturbance)

The differences between some key subgroups fall in the 5-10 percentage point range. This puts them at a level that raises concerns about the possibility of differential non-response bias that may affect estimates of differences between groups. However, a greater concern is that between two-thirds and three-fourths of each group (and the sample as a whole) are currently non-responders after a fielding period of 4 and one-half months. Unless we can bring overall response rates to a level closer to two thirds, our ability to generalize from our sample to all youth nationwide will be subject to question.

The large fraction of our sample that so far has not responded has lead us to examine information from our CATI system on the status of cases that have not completed the parent interview. This will allow us to tailor strategies for securing a completed interview to the circumstances and barriers of the case. Table 3 summarizes interview status data for all students with an IEP, all students with no IEP, and the sample as a whole.² We plan to focus our efforts to improve response rates for the following three groups:

• Could not reach (“hard to reach’). These are cases for whom we are able to confirm that we have a good phone number, and with whom we have made numerous phone attempts without being able to reach the parent to explain the study and ascertain willingness to participate (total = 3,075 or 17 percent of the sample)

• Soft refusals. These are cases that have said they did not want to participate or hung up during the introduction. In many cases, we are not certain we have reached a sample member’s parent (total = 1,980 or 11 percent of sample).

• In locating/not located. These are cases for which we do not have a phone number (or address that has allowed us to obtain the parent’s phone number). Of these cases locators have searched but exhausted all avenues and assigned a final status for 645; search efforts are ongoing for 3,078. (3,723, total, or 21 percent of sample.)

3. Proposed Experiment with Additional Incentives

Currently, we offer $20 to parents who complete the parent interview and mail a check along with a thank you note and a copy of what they agreed to during the digitally recorded oral consent process when the interview is completed. We are now requesting approval to conduct an experiment that includes a small prepaid incentive, increases the incentive for respondents, and transmits information about the study using a delivery mode more likely to gain the attention of the sample member.

Target Groups. The experiment will focus on two groups: parents who are soft refusals and parents for whom we have one or more valid, working phone numbers but whom we have not been able to reach (“hard to reach” group who are “avoiders”). Soft refusals and avoiders fall into several categories all of which are good candidates for an incentive experiment. Some confuse Mathematica with telemarketers, fundraisers, and political pollsters. Others have forgotten or never saw the initial letter describing the study. Thus, some cases identified as soft refusals or avoiders hang up or do not answer the phone without knowing the purpose of our call. Currently, approximately 2,000 cases are soft refusals and 3,000 cases have reached the maximum number of calls. In order to avoid confounding the effects of field locating with effects of the changed incentives, members of the target population for the incentive experiments who are in districts in which field locating is planned will not be part of the incentive experiment. This will reduce the number of soft refusals from which we will sample for the incentive experiment from 2,000 to 1,750 and the number of hard to reach cases from 3,000 to 2,550.

Experimental Structure. A subsample of the two targeted nonresponse groups (soft refusal cases and hard to reach cases) will be randomly assigned to one of three treatments. Accordingly 300 cases (150 soft refusal cases and 150 hard to reach cases) will be assigned to each of three treatment cells (900 total). The three groups will receive the following incentives and notifications about the study:

• Treatment 1: Prepaid $5.00 cash incentive mailed USPS first class in standard-sized envelopes (4-1/2” x 10 3/8”) with ED return address. Receives a fresh letter on ED letterhead that promises a $15 post check. The total respondent payment will be the same as already approved. [Cost per individual³ = $20 incentive + $0.45 mail. Total cost = $6,135.]

• Treatment 2: Prepaid $5.00 cash incentive mailed USPS first class in standard-sized envelopes (4-1/2” x 10 3/8”) with ED return address. Receives a fresh letter on ED letterhead that promises a $30 post check. [Cost per individual = $35 incentive + $0.45 mail. Total cost = $10,635.]

• Treatment 3: Prepaid $5.00 cash incentive mailed USPS priority mail flat envelope⁴ (12½” x 9½”). Receives a fresh letter on ED letterhead that promises a $15 post check. The total respondent payment will be the same as already approved. [Cost per individual = $20 incentive + $5 priority mail. Total cost = $7500.]

All three treatments include the same $5 prepayment⁵. Comparisons of treatments 1 and 2 examine the effects of increasing the total post-completion incentive offered by $15. Comparison of treatments 1 and 3 examine the effects of a different delivery packaging, which has been shown in NCES experiments to be promising⁶. Ideally the experiment would include a fourth treatment arm to examine the combined effects of increased incentives and a different delivery package, but we did not want to dilute the experiment. Attachments A and BAppendix N1 and N2 contain the revised letters.

Reporting. Two measures will be used to estimate the effects of the experiment: 1) the percentage of cases in group who complete the survey, and 2) the percentage of cases that either completed the survey or have a call back appointment.

TABLE SHELL A

SHELL FOR REPORTING PERCENTAGE COMPLETING SINCE INITIATION OF EXPERIMENT

Target Group		T1	T2	T3
Soft Refusals
standard error
N		150	150	150
Hard to Reach
standard error
N		150	150	150
Total
standard error
N		300	300	300

We will report the two primary measures described above and their standard errors for each subgroup and treatment group in the format of Table Shell A.

The estimated minimum detectable difference (MDD) depends on the number of observations per cell, the assumed percentage completing in the cell, and the specification of the null hypothesis being tested. We assumed values of the proportion completing an interview in the absence of intervention of 0.10, 0.2, and 0.33 and use a one-tailed hypothesis test at a 95 percent confidence level to examine the MDDs for the following comparisons:

• Pairwise comparisons of each treatment group (effect of differences in treatments across subgroups, 300 vs 300)

• Pairwise comparisons of each treatment group for each subgroup (150 vs 150)

For the comparison of each treatment across subgroups (300 vs 300), the range of MDD is .061 to .096. For the comparison of each treatment by subgroup (150 vs. 150) the range is .086 to .135. Details are provided in Table 5. We will also look descriptively at response/completion rates of the experiment’s treatment groups compared with rates we were obtaining for these target groups prior to the experiment, to provide some sense of whether any of the possible new incentive strategies appear better than what we were already implementing.

Schedule for Incentive Experiment. We will prepare for selecting and tagging the samples selected for the four groups, and setting up systems to handle the three treatment conditions. If OMB approves the experiment by July 13th, we expect to report results by mid August, which is approximately a three-week period from the start of the experiment to when we would report back to OMB with results.⁷

4. Field Presence in Priority Areas

We plan to send field experienced ED cleared interviewers to four priority areas to address two objectives: (a) school visits to update locating information for parents and students, and (b) home visits to administer consent form and connect parent to our survey operations center. The four areas and 14 districts were selected because they have large samples and some combination of low completion rates, high proportions of hard to reach cases, and high proportions of students eligible for free or reduced price meals.⁸

1. New Jersey/Philadelphia (Newark, Atlantic City, Philadelphia), with approximately 250 difficult to reach cases;

2. Chicago area/Indianapolis with 200 difficult to reach cases;

3. Six central Florida districts (with a combined total of 700 difficult to reach cases; and

4. San Diego and nearby districts in southern California with 125 difficult to reach cases.

Should this effort be successful, we will add San Antonio and Dallas Texas, with approximately 150 difficult to reach cases. Sample in the field sites will be excluded from the incentive experiments.

Letters to the parents will be sent in advance to any field visits that will be conducted (see Attachments C and DAppendix O1 and O2 for sample letters).

We plan to send two types of field interviewers into these areas. One type of interviewer will have experience inside schools and will visit schools to update locating information for parents and students. Schools often have information that is more up-to-date than the district information we originally sought.  When on site, she will prompt principals and teachers who may still be at the schools to respond to their surveys.  Any new student contact information will be passed to our Survey Management System (SMS) so the information is available for telephone interviewers.  Should we find a new address, we will attempt contact by telephone before attempting an in-person visit.

For those without new contact information, a second set of interviewers who are adept at home visits will attempt to make face-to-face contacts. These interviewers will be prepared to respond to parent questions or concerns about the study.  In areas with Hispanic populations we will identify bilingual (English-Spanish) interviewers.⁹  The field interviewer will administer the consent form in person rather than digitally recording the consent.  To avoid mode effects, she will call the survey operations center so the interview can be conducted by telephone. 

Schedule Recruitment and hiring of field interviewers and modifications to data management systems are in progress. The field effort will be initiated in Newark with training of interviewers 5 days after OMB and IRB approval are received. Training and initiation of field work will occur in the other locations as staff are identified, hired, and receive security clearance. We will have approximately one field interviewer per 50 cases, so that field work can be completed within 4 to 6 weeks. Assuming mid-July approval, we expect to be able to evaluate the effort beginning in mid-August, and to complete field work and report on it by the end of August.

5. Other Planned Changes Requiring OMB Approval

As noted in Table 1, the incentive experiment and field effort for which approval are requested in this memo are elements of a broader set of strategies being considered for implementation. Additional ones that we plan to implement include contacting schools to obtain updated contact information in selected districts. This is an important element of our strategy to reduce the number of cases for whom we do not have locating information. We hope the use of ED letter head to send new letters to pending cases will be salient for some of our hard to reach sample.

6. Additional Changes under Consideration for Future OMB Approval.

Table 1 includes two changes under consideration that will require OMB and IRB approval, if IES decides to implement them. These two potential changes are creation of an abbreviated questionnaire which can be self-administered (Short Version of Parent Interview), and revisions to the study consent form (Shorten Consent). Because shortening the parent interview requires careful consideration of what to give up and would likely be used near the end of the fielding period, we did not want to hold up OMB consideration of the proposed incentive experiment and field locating effort.

TABLE 1

SUMMARY OF PROPOSED CHANGES TO NLTS BASELINE DATA COLLECTION

Strategy	Description	Target Sample	Schedule
Incentive Experiment	Randomly vary method of mailing & postpaid incentive amount, in context of $5 prepaid incentive	(1) Hard to reach cases; (2) soft refusals. Excludes four areas to be included in the field interviewing	Initiate 3-week experiment five days after OMB and IRB approval; report to OMB five weeks after approval
Field Interviewing at Selected Sites	ED security cleared, experienced field interviewers visit sample members to explain study, conduct consent, and facilitate CATI interview	All cases not complete or hard refusals in 4 low response areas: NJ/Philadelphia; Chicago area, San Diego area; and central FL districts (to be excluded from incentive experiment)	Initiate 6 week pilot five days after OMB and IRB approval
School Updates to Contact Unlocatable Sample Members	Professional staff contact schools	Schools in selected districts amenable to contact; sample members not included in either incentive experiment nor field interviewing because contact information is not currently available	Mid June –end of August, with hiatus in July
New Mailing with ED Letterhead	Send letters on ED letterhead, rather than NLTS letterhead	All cases not complete or hard refusals ; sample overlaps with the incentive experiment and will be used to follow contact information updates from schools	Initiate upon receipt of printed letterhead from GPO; continue as necessary
Additional Changes Under Consideration For Future OMB Approval
Shorten Consent	Review partial completes and consent wording	To be determined	To be determined
Short Version of Parent Interview	15 minute self-administered version of parent questionnaire	Reluctant sample willing to complete an abbreviated, self-administered version of survey; target sample to be determined after reviewing effectiveness of other strategies	To be determined

TABLE 2

NLTS SAMPLE AND PARENT INTERVIEW COMPLETES ON JULY 1, 2012

	Target Sample	Released for Interview	Number of Completes	Percent Completed
Autism	1,000	1,509	569	37.7%
Deaf-Blindness	100	68	19	27.9%
Emotional Disturbance	1,200	1,712	484	28.3%
Hearing Impaired (Including Deaf)	520	982	309	31.5%
Intellectual Disability	1,200	1,806	585	32.4%
Multiple Disabilities	900	1,312	433	33.0%
Orthopedic Impairment	450	748	243	32.5%
Specific Learning Disability	1,600	2,136	584	27.3%
Speech or Language Impairment	1,000	1,430	400	28.0%
Traumatic Brain Injury	230	471	147	31.2%
Visual Impairment	200	427	142	33.3%
Other Health Impairment	1,200	1,783	575	32.2%
Total IEP	9,600	14,384	4,490	31.2%
Section 504 Plan, No IEP	600	1,073	299	27.9%
No Section 504 Plan No IEP	1,800	2,671	701	26.2%
Total With No IEP	2,400	3,744	1,000	26.7%
Total Sample	12,000	18,128	5,490	30.3%

TABLE 3

PERCENTAGE IN KEY PARENT INTERVIEW STATUSES, ALL IEP AND Non IEP STUDENTS

	All Students with IEP			All Students with No IEP			Total Sample
	Percentage	Number		Percentage	Number		Percentage	Number
Final Status
Completes	31.2%	4,490		26.7%	1,000		30.3%	5,490
Final Refusal	8.3%	1,200		10.5%	393		8.8%	1,593
Other Final	0.8%	120		1.2%	46		0.9%	166
Total Final Status	40.4%	5,810		38.4%	1,439		40.0%	7,249
Interim Status
Could Not Reach	17.1%	2,459		16.5%	616		17.0%	3,075
Interim Refusal	10.2%	1,463		13.8%	517		10.9%	1,980
In Locating/Not Located	21.0%	3,014		18.9%	709		20.5%	3,723
Other Interim Status	11.4%	1,638		12.4%	463		11.6%	2,101
Total Interim Status	59.6%	8,574		61.6%	2,305		60.0%	10,879
Total Sample		14,384			3,744			18,128

TABLE 4

PERCENTAGE IN KEY PARENT INTERVIEW STATUSES, BY IDEA SUBGROUP

	Autism	Deaf-Blindness	Emotional Disturbance	Hearing Impaired (Incl. Deaf)	Intellectual Disability	Multiple Disabilities	Orthopedic Impairment	Specific Learning Disability	Speech or Language Impairment
Final Status
Completes	37.7%	27.9%	28.3%	31.5%	32.4%	33.0%	32.5%	27.3%	28.0%
Final Refusal	9.2%	5.9%	8.6%	6.5%	6.5%	9.6%	7.5%	8.0%	9.4%
Other Final	0.8%	2.9%	0.4%	1.5%	0.8%	1.1%	1.3%	0.5%	1.3%
Total Final Status	47.7%	36.8%	37.3%	39.5%	39.7%	43.7%	41.3%	35.8%	38.6%
Interim Status
Could Not Reach	15.6%	19.1%	16.6%	15.2%	16.3%	18.0%	17.0%	19.7%	18.3%
Interim Refusal	11.0%	4.4%	9.5%	11.1%	7.3%	9.0%	9.5%	10.0%	12.7%
In Locating/Not Located	14.0%	23.5%	25.8%	20.2%	27.5%	19.7%	18.3%	22.8%	19.2%
Other Interim Status	11.6%	16.2%	10.9%	14.1%	9.2%	9.7%	13.9%	11.8%	11.2%
Total Interim Status	52.3%	63.2%	62.7%	60.5%	60.3%	56.3%	58.7%	64.2%	61.4%
Total Sample	1,509	68	1,712	982	1,806	1,312	748	2,136	1,430

	Traumatic Brain Injury	Visual Impairment	Other Health Impairment	Total IEP	Section 504 Plan, No IEP	No Section 504 Plan No IEP	Total With No IEP	Total Sample
Final Status
Completes	31.2%	33.3%	32.2%	31.2%	27.9%	26.2%	26.7%	30.3%
Final Refusal	9.6%	7.3%	9.4%	8.3%	12.6%	9.7%	10.5%	8.8%
Other Final	1.3%	1.2%	0.3%	0.8%	0.4%	1.6%	1.2%	0.9%
Total Final Status	42.0%	41.7%	42.0%	40.4%	40.8%	37.5%	38.4%	40.0%
Interim Status
Could Not Reach	13.6%	17.1%	16.9%	17.1%	16.9%	16.3%	16.5%	17.0%
Interim Refusal	12.5%	9.8%	11.6%	10.2%	14.1%	13.7%	13.8%	10.9%
In Locating/Not Located	20.8%	17.1%	18.1%	21.0%	16.6%	19.9%	18.9%	20.5%
Other Interim Status	11.0%	14.3%	11.4%	11.4%	11.6%	12.7%	12.4%	11.6%
Total Interim Status	58.0%	58.3%	58.0%	59.6%	59.2%	62.5%	61.6%	60.0%
Total Sample	471	427	1,783	14,384	1,073	2,671	3,744	18,128

TABLE 5

MINIMUM DETECTABLE BETWEEN GROUP DIFFERENCES FOR INCENTIVE EXPERIMENT

Comparison	Sample Size	MDD for p=.1	MDD for p=.2	MDD for p=.3
Pairwise Comparisons of Treatments	300 vs. 300	0.061	0.081	0.096
Pairwise Comparisons of Treatments, by Subgroup	150 vs. 150	0.086	0.115	0.135

ATTACHMENTS

1 We have also conducted a limited analysis of response across other key study subgroups. Differences in response rates for several other subgroups are fairly small including groups defined by district size category, student grade, length of time in interviewing status and percentage of district students eligible to receive free or reduced price school lunch (proxy for family income of district students). For example, we created district level measures of the percentage of students in the district eligible for free or reduced price school meals, and examined survey status of districts by tercile. As of mid-June, average percentage of the sample that had completed a parent baseline was similar across terciles (low, 30 percent; medium, 27 percent; high, 28 percent).

2 Table 4 in the Attachment provides this data by IDEA category subgroup and by Section 504 status for students with no IEP.

3 ED letterhead and envelope printing costs are approximately $2305 for 30,000 copies of each, which is an insignificant cost per mailing ($0.04 per letterhead or envelope). Thus, these costs are not included in the total costs.

4 US Priority Mail has been selected as the alternate delivery option for this experiment because FedEx does not deliver to PO boxes, which is the delivery address we have for many families.

5 Results from NHES 2011 Field Test demonstrated that $5 prepaid cash outperformed $2 prepaid cash by 4.4 percentage points.

6 NHES 2011 Field Test demonstrated higher responses to mailings sent using distinctive packaging (i.e., distinctive envelopes as compared with standard craft envelopes).

7 Letters will be sent July 20th; two-week period for observing response starts July 25th and ends August 8th; report to OMB by August 15th

8 The criterion of high proportion eligible for free and reduced price meals is included because we observed difference in the reasons for nonresponse across districts with high, medium, and low percentages of students receiving free or reduced price meals. Refusal rates tended to be are higher in districts with low percentages free and reduced price and rates of no location data tended to be lower. However, in districts with high percentages free and reduced price rates of no location information are higher and rates of refusal lower. Therefore we wanted to be sure field interviewing effort was concentrated in areas likely to have more cases in the group likely to require locating.

9 We will not attempt interviews in other languages. In total we have 157 parents who speak neither English nor Spanish.  They speak a variety of different languages and no language is prevalent enough to merit a translation or use an interpreter unless we can identify an interpretation service that is suitably DOE cleared.

An Affirmative Action/Equal Opportunity Employer