Appendix G: Incentive Study

To: Lauren Supplee, Head Start CARES Project Officer      

From: Head Start CARES Project Team     

Date: March 29, 2012     

Subject: Head Start CARES Incentive Study

This memo describes the results of a study built into the data collection process of the Head Start CARES project by OMB request. The study assesses whether the payment amount provided to teachers for completion of the teacher self survey affects teacher response rates.

Background

Teacher self surveys were collected in the Head Start CARES study in order to collect information about teachers’ demographics, psychosocial variables, and background characteristics. Teacher self surveys were collected twice in the CARES project: 1) at baseline, during the spring prior to the school year that the enhancements were implemented, and 2) at follow-up during the spring of the implementation year. At baseline, classroom observations were also being collected, and observers dropped the teacher surveys off to the teachers when they came to the classroom for the observation, and teachers were asked to mail the surveys back to the survey firm. Incentives were mailed to teachers after surveys were received. 

At follow-up, the project was also collecting child assessments, and assessors dropped off the surveys during their first day of assessments and collected them from the teacher on their last day of assessments. Any teachers who did not return the surveys to the assessor were asked to mail them to the survey firm, and incentives were again mailed to teachers after surveys were received. 

The CARES project was approved by OMB with a $15 incentive for the completion of the teacher self survey for a subset of sites (OMB Control Number: 0970-0364). This amount was consistent with MDRC’s and the survey firm’s prior experience interviewing similar populations. For a second subset of sites, OMB required including a planned variation study. This memo describes the results of that variation study.

Research Question

The study addresses the following research question: Does the amount of payment provided to teachers for completion of the teacher self survey affect response rates? Specifically, we assess whether a $5 difference in incentive payment amount affects response rates for the teacher self survey.

Description of the Study

The study includes 12 grantees in eight regional hubs. We randomly assigned each hub to one of the following two groups:

• Group A: Teachers receive a $15 payment if they complete the teacher self survey at baseline and a $15 payment if they complete the survey at follow-up.

• Group B: Teachers receive a $10 payment if they complete the teacher self survey at baseline and a $20 payment if they complete the survey at follow-up.

Teachers in both groups are eligible to receive the same total incentive amount over the two data collection periods ($30). Four hubs (containing six grantees) were randomly assigned to Group A, and four hubs (containing six grantees) were randomly assigned to Group B. Random assignment results are shown in Table 1.

To assess whether a $5 difference in incentive payments makes a difference for response rates, we compare grantee-level response rates of Groups A and B at baseline ($10 vs. $15) and the response rates of the two groups at follow-up ($15 vs. $20).

Results

Tables 2 and 3 show the response rates in each grantee at baseline and follow-up, respectively. There was only a difference between the two groups at baseline. At baseline, Group A, who received $15, had an overall response rate of 100%, and Group B, who received $10, had an overall response rate of 93%. At follow-up, both the $15 group and the $20 group had response rates of 100%.

Conclusion

The seven percentage point difference at baseline suggests that, at an incentive level of $10, a $5 increase may make a difference for improving response rates. However, because we randomly assigned eight hubs, there is not sufficient statistical power to perform a t-test to formally test whether this seven percentage point difference is statistically significant. Our results do not show evidence that a $5 increase in payment amount at the $15 level improves response rates.

When interpreting the results of these comparisons and considering incentive levels for other projects, it is useful to keep in mind two factors specific to the CARES project that may largely explain the very high response rates overall:

• The project’s data collection process allowed for multiple contacts with the teachers in the study, both in person and over the phone. In fact, the higher response rates at follow-up compared to baseline may be in part due to the fact that assessors came to the classrooms at follow-up to pick up the surveys in person, rather than having the teachers send them back by mail as was done at baseline.

• The project’s survey firm had a strong commitment to achieving high response rates and had the flexibility to invest additional resources in pursuing non-respondents when needed.

Table 1. Random Assignment Results
Hub #	Hub	Name of Grantee/Delegate Agency	Location	Spring Baseline Incentive	Spring Follow up Incentive	Total
5	Winona, MS	Central Mississippi, Inc.	Winona, MS	$10	$20	$30
6	San Diego, CA	Episcopal Community Services	Chula Vista, CA	$10	$20	$30
7	East Texas	Region 7 Education Service Center	Kilgore, TX	$10	$20	$30
		Tyler ISD	Tyler, TX	$10	$20	$30
8	Chicago, IL	Chicago Youth Centers	Chicago, IL	$15	$15	$30
9	Central Ohio	Child Development Council of Franklin County	Columbus, OH	$15	$15	$30
		LEADS Head Start	Newark, OH	$15	$15	$30
		WSOS Community Action Commission, Inc.	Fremont, OH*	$15	$15	$30
10	Bay Area, CA	Santa Clara County Office of Education	San Jose, CA	$10	$20	$30
		Berkeley-Albany YMCA Head Start	Berkeley, CA	$10	$20	$30
11	Los Angeles, CA	Pacific Asian Consortium In Employment	Los Angeles, CA	$15	$15	$30
12	Denver, CO	Rocky Mountain SER Head Start	Denver, CO	$15	$15	$30

Table 2. Baseline Teacher Self-Survey Response Rates
		Total Teachers	Total Completed	% Complete
		Group A: $15 incentive
Central Ohio
Child Dev. Council of Franklin County (Columbus)		16	16	100%
LEADS Head Start (Newark)		8	8	100%
WSOS Comm. Action Commission, Inc. (Fremont)		14	14	100%
Los Angeles, CA
Pacific Asian Consortium in Employment (LA)		40	40	100%
Denver, CO
Rocky Mountain SER Head Start (Denver)		12	12	100%
Chicago, IL
Chicago Youth Centers (CYC)		17	17	100%
Total		107	107	100%
		Group B: $10 incentive
Winona, MS
Central Mississippi, Inc. (Winona)		38	36	95%
San Diego, CA
Episcopal Community Services (Chula Vista)		23	23	100%
East Texas
Region 7 Edu. Service Center (Kilgore)		31	26	84%
Tyler ISD (Tyler)		11	11	100%
Bay Area, CA
Santa Clara County Office of Education (San Jose)		10	9	90%
Berkeley-Albany YMCA Head Start (Berkeley)		8	8	100%
Total		121	113	93%
Table 3. Follow-Up Teacher Self-Survey Response Rates
	Total Teachers		Total Completed	% Complete
	Group A: $15 incentive
Central Ohio
Child Dev. Council of Franklin County (Columbus)	15		15	100%
LEADS Head Start (Newark)	8		8	100%
WSOS Comm. Action Commission, Inc. (Fremont)	14		14	100%
Los Angeles, CA
Pacific Asian Consortium in Employment (LA)	39		39	100%
Denver, CO
Rocky Mountain SER Head Start (Denver)	11		11	100%
Chicago, IL
Chicago Youth Centers (CYC)	16		16	100%
Total	103		103	100%
	Group B: $20 incentive
Winona, MS
Central Mississippi, Inc. (Winona)	38		38	100%
San Diego, CA
Episcopal Community Services (Chula Vista)	21		21	100%
East Texas
Region 7 Edu. Service Center (Kilgore)	30		30	100%
Tyler ISD (Tyler)	8		8	100%
Bay Area, CA
Santa Clara County Office of Education (San Jose)	10		10	100%
Berkeley-Albany YMCA Head Start (Berkeley)	8		8	100%
Total	115		115	100%