FACES2014 Parent Experiment Results

T O: OMB

FROM: ACF/OPRE

DATE: 5/1/2015

RE: FACES 2014 Parent Survey Experiment results

In the fall of 2014, Mathematica conducted an experiment to examine the effects of offering parents a web option on parent survey response rates and costs. We were especially interested in (1) whether it is cost-effective to use a web survey as compared to a telephone survey with Head Start parents and (2) whether parents’ choice of a web survey is a function of how this option is introduced to them. This memo describes the design of the experiment, and summarizes our data collection procedures for each experiment group, the findings, and our recommendations for spring 2015 parent survey data collection. The initial data collection package (approved July 7, 2014) described the purpose and plans for this experiment. In the spring update package (approved February 20, 2015), findings from the experiment were referenced to help explain the low interim response rates for the parent survey.

A. Background

Administering surveys to Head Start parents in a cost efficient manner is an important part of the FACES study design. The FACES 2014 parent survey collects important information about parent’s and children’s activities, experiences with Head Start and community agencies, health care, and parents’ feelings and attitudes about themselves. While prior rounds of FACES have collected this information through interviewers with parents, either in person or by telephone, the reduced length of the parent survey in FACES 2014 made possible the option of using the lower-cost web survey.

Little is known about the feasibility of using a web survey with a low-income population such as Head Start parents and whether the way in which this option is introduced will affect parents’ decision to respond to the survey online and overall response rates. A Mathematica study conducted in 2012 by Mooney et al. found that initially limiting the response mode to web only significantly increased web completes and had no negative impact on the overall response rate. Additionally, in 2011, the U.S. Census Bureau found that significantly more households accessed and responded to a survey online when offered only a web survey at the beginning of data collection.

B. Random Assignment and Data Collection Procedures for Experiment Groups

The experiment was to be conducted in all 60 programs participating in the FACES Classroom + Child Outcomes Core Study fall 2014 data collection, and the parents of all 2,462 sampled, eligible and consented study children in those programs. Prior to data collection, the parents were randomly assigned in equal numbers to one of two groups, within program sampling strata:

1. Web with telephone follow-up after three weeks (Web First) – Parents in this group were sent study materials that only referenced the web option for completing the survey. However, if parents in this group did not complete the survey in the three weeks following the invitation to participate, we called them to attempt to complete the survey and sent follow-up materials notifying them that they could call Mathematica’s phone center to complete their interview.

2. Choice of web or telephone (Choice) – Parents in this group were sent study materials indicating from the start that they had the option of completing the survey online or by telephone. In the first three weeks after a case’s release, we used a passive telephone effort in which we completed surveys only with parents who called Mathematica’s phone center, allowing us to determine the parents’ choice of a web or phone survey. After three weeks, we began efforts to reach parents by phone to complete the survey using the same procedures that we used for the Web First group.

Parents received a base amount of $15 for completing the survey with bonus amounts of $5 each for completing it online or completing it within three weeks of being invited to do so.

In week 11 of the fall data collection period, parent response rates were lower than expected. With only 4 weeks left in the data collection period to achieve our target response rate, we decided to suspend the experiment for cases less than three weeks past their release date. This affected about 350 parent cases. All findings reported below were based on analyses of 2,112 cases were data collection efforts had commenced prior to the suspension of the experiment.

C. Findings

Table 1 shows the distribution of parent cases across the two experiment groups. Initially, parents were assigned about equally to the Web First and Choice groups (50.3 percent and 49.7 percent, respectively). However, once the 350 parent cases were removed, we have slightly more parents in the Web First group than in the Choice group in our analytic sample.

Table 1. Random assignment to experiment group

Parent experiment group	Frequency	Percent
Web First	1092	51.7%
Choice	1020	48.3%
Total	2112	100.0%

On the consent form, we asked parents to indicate whether they “have access to a smart phone, laptop, computer or other device that gives you access to the Internet?” Table 2 shows that the majority of Head Start parents reported having access to the Internet. The percentages of parents with and without internet access was roughly equal for the two experimental groups.

Table 2. Parents’ reported access to the Internet by experiment group and overall

	Web first group	Choice group	Overall
Yes	928 (85.0%)	883 (86.6%)	1,811 (85.8%)
No	142 (13.0%)	112 (11.0%)	254 (12.0%)
Not reported	22 (2.0%)	25 (2.4%)	47 (2.2%)
Total	1,092	1,020	2,112

Overall, 78.2 percent of parents completed the survey with 48.2 percent responding online and 51.8 percent by phone (Table 3). Completion rates for the two experimental groups were very close as were the percentage of completed surveys that were done online and by phone. Limiting parents to a web survey option only (Web First) during the first three weeks of data collection did not produce a higher percentage of web completes than CATI completes for this group. In fact, although a higher number of the total web completes came from the Web First group, the Choice group had a slightly higher percentage of web completes than the Web First group (49.2 percent and 47.2 percent, respectively).

Table 3. Parent survey completes by parent experiment group

Parent experiment group	CATI completes	Web completes	Total completed cases
Web First	449 (52.8%)	401 (47.2%)	850 (77.8%)
Choice	407 (50.7%)	395 (49.2%)	802 (78.6%)
Total	856 (51.8%)	796 (48.2%)	1652 (78.2%)

As expected, most of the cases completed online were completed by parents who reported having internet access. About 53 percent of the parents who reported having internet access completed the survey online as compared with only 7 percent of the parents who reported no internet access.

About 53 percent of the completed parent surveys were completed within the first three weeks of being invited to participate in the survey, and 71 percent of these were completed online. Web completes during the first three weeks were higher than telephone completes for both experimental groups. Overall, more than 70 percent of the surveys completed online were completed by parents within the first two weeks of being invited to participate (Table 4).

Table 4. Parent survey Web completes by week after release

Final status	Weeks after release
	1	2	3	4+	Total
Web completes	439 (55.2%)	124 (15.6%)	50 (6.48%)	183 (23.0%)	796 (100.0%)

D. Cost Analysis

Our parent survey budget expected 40 percent of parents would complete the survey by web and 60 percent by phone. In the end, about 48 percent of parents completed the survey online and 52 percent by phone. We estimate that the 846 web completes saved approximately $19,500 in telephone interviewer costs.¹

E. Implications of Findings for Parent Survey Procedures

Based on the findings from the parent experiment, we propose the following procedures for the spring 2015 and future waves of FACES parent survey data collection:

1. Offer all parents the choice of online or telephone completion: The Web First group did not complete surveys on the web at a higher rate than parents in the Choice group and the two groups reported similar rates of access to the Internet. Therefore, all parents will be given the choice of completing the survey online or via telephone from the start.

2. Retain existing incentive structure: Although the experiment did not manipulate the incentive payments to parents, we believe that the incentive for early completion worked to motivate parents to complete the survey early, eliminating the need for active calling to these respondents and additional follow-up efforts. Therefore, the bonus incentives of $5 for early response and $5 for web completion will be retained.

3. Initiate active calling two weeks after initial invitation to complete the survey: Because most early completers responded to the survey within the first week or two of being invited to do so, we will reduce the delay in active calling from three weeks to two weeks. This will allow more time to pursue cases that are released later in the field period and do not respond using the web option.

References

Mooney, Geraldine, Cheryl De Saw, Xiaojing Lin, Andrew Hurwitz, and Flora Lan. “Influencing Mode Choice in a Mixed-Mode Survey.” Presented at the American Association for Public Opinion Research Annual Conference, Orlando, FL, May 2012.

Tancreto, J., Zelenak, M. F., Davis, M., Ruiter, M., & Matthews, B. (2012). “2011 American Community Survey Internet Tests: Results from First Test in April 2011” U.S. Census Bureau: Decennial Statistical Studies Division. 2012 American Community Survey Research and Evaluation Report Memorandum Series. #ACS12-RER-13-R1

1 The actual savings will be higher. This figure does not include other related costs such as supervision and computer and phone charges.