NonSub Change Memo - ECE ICHQ PostPay Experiment - May 2021

DATE: May 6, 2021

TO: Jordan Cohen, Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Meryl Barofsky, Office of Planning, Research, and Evaluation, Administration for Children and Families

RE: Request for an increase in respondent token of appreciation amounts for Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study (OMB: 0970-0499)

Background

• Type of Request: Non-substantive change to token of appreciation structure to a survey in a multipart study with an experiment to compare the type of token of appreciation, speed of response, the response rate, and the data collection costs between the two groups.
  
  

• Study Features Salient to Request:

The goal of the Implementation and Cost of High Quality Early Care and Education (ECE-ICHQ) study is to create a technically sound, feasible, and useful instrument that will provide consistent and systematic measures of the implementation and costs of quality to help fill the knowledge gap about the cost of providing and improving quality in early care and education (ECE). Mathematica conducted work for the ECE-ICHQ project from September 29, 2014 through September 30, 2019. During that time, the ECE-ICHQ study team developed a conceptual framework; conducted a review of the literature; consulted with a technical expert panel; collected and summarized findings from two phases of data collection; and developed instruments and measures of implementation, costs, and time-use. In October 2019, Mathematica began preparing for a field test of the new measures and launched the field test for three weeks in March 2020 until it needed to cease due to the COVID-19 pandemic. Beginning in March 2021, the team will restart the field test to (1) validate key center measures of costs and implementation and further improve the psychometric properties of the implementation measures and, (2) test preliminary associations between implementation, cost, and quality. The field test will also help us learn how our instruments and measures of implementation and cost are able to capture the service changes that are occurring across the child care industry due to the COVID-19 pandemic. Recruitment for the field test began in March 2021 and data collection began in April.

During the field test, the study team will return to early care and education centers that participated in Phase 2 as well as reach out to additional early care and education centers who have not previously participated in the study. Each center will be asked to participate in the latest round of data collection that includes: (1) center management participating in phone interviews to gather information about center implementation activities and completing a cost work book, and (2) eligible center staff completing a 15-minute survey to learn how they spend their work time (a time-use survey). Though we expect staff turn-over within centers since Phase 2, we do expect some overlap in respondents. The time-use survey is the focus of this request.

Progress to Date:

We began recruitment in March 2021. We recruited 13 (out of 22) Phase 2 centers to participate in this data collection. We will now recruit up to 67 additional centers (that have not previously participated in the study) for a target total of 80 participating centers in the field test. From prior phases, we estimate each center will have, on average, 16 center staff eligible to complete the time-use survey, resulting in a sample size of about 1,280 for the survey.

• Previous Terms of Clearance:

The following tokens of appreciation for the ECE-ICHQ time-use survey were previously approved:

• PHASE 1: a $10 gift card, 15 minutes to complete, no field staff on site

• PHASE 2: a $10 gift card, 15 minutes to complete, field staff on site

• FIELD TEST: a $10 gift card (POSTPONED; focus of this change request), 15 minutes to complete

• Time Sensitivity: In March 2021, we began recruiting centers that had participated in a prior round of data collection of this study. In May, we are beginning to recruit new centers to be part of the field test. Based on prior rounds of this study, we have a staged approach that has proven successful to completing data collection and compressing the time in which we need to engage each center with the full data collection process. Using this approach, we conduct the implementation interview with the center director first and then release the time use survey to staff while we work with the center on the cost reporting. This process typically has us release the time use surveys about two weeks after a center is recruited into the study. Delays in administering the survey could stretch out the data collection process and impact our ability to get centers to complete each component of the data collection.

Request Overview

ACF requests a non-substantive modification to the ECE-ICHQ time-use survey token of appreciation structure from a $10 token of appreciation to $20. We would like to conduct an experiment in which staff at half of participating field test centers are provided a $10 gift card with the initial survey invitation and an additional $10 after completing the survey. Staff in the other half of participating field test centers will receive a $20 gift card after completing the survey. We would then compare the speed of response, the response rate, and the data collection costs between the two groups. (We will provide a $10 gift card after completing the survey to the staff in the 13 centers that had also participated in Phase 2 and in which we need to move quickly in launching the time use surveys. These centers will be excluded from the experiment due to timing constraints in preparing and delivering initial survey invitation packets.)

The goal of this change is to ensure we have a large enough sample size for analysis and address lower-than-expected response rates and observable non-response bias in the data we collect. The study attempts to collect data from all teaching staff at a center to capture the variation within centers among staff with similar roles. A 90% response rate is our threshold for being able to do this.

In Phase 1 of data collection our response rate was 48%. In Phase 2 of data collection our response rate was 89%. We were able to accomplish this improvement by having field staff visit the centers and distribute gift cards in person immediately after center staff completed their surveys. For this data collection, we expect response rates on the time-use survey to be lower than the 89% response rate we obtained in Phase 2 due to two reasons:

1. We will not be able to send field staff to centers to follow up with staff; and

2. Staff will need to access and complete a web-based version of the survey, which will require additional time and resources compared to a paper copy that is distributed and collected by field staff on-site.

The current data collection will occur with 80 centers, with a total estimated sample size of 1,280 potential respondents. However, time-use varies significantly by role and some roles, like center administrators, have as few as 160 potential respondents. With such a small sample size, it is imperative that we achieve a strong response rate to give us enough data to conduct meaningful analyses by position.

Respondent	Target n	n Secured	Response Rate
Phase 1 (no field staff)	212	101	48%
Phase 2 (field staff visit centers to invite center staff to complete time-use survey, and distribute gift cards upon completion)	483	430	89%

Mitigation to Date

Since Phase 1 of the study, as part of regular survey administration protocol, ACF has actively monitored survey response rates in the multi-Phase ECE-ICHQ study. In Phase 2, the contract team took the following steps to address emerging concerns with lower than expected response rates and potential non-response bias:

• Sending field staff to centers to invite center staff to complete the time-use survey in-person;

• Giving center staff that complete the time-use survey during our visit the $10 gift card upon completion, rather than mailing it to them afterwards

Plans for Future Mitigation

• The study team plans to adjust the schedule of email reminders that will be sent to non-respondents. We will incorporate additional reminders and decrease the length of time between each reminder. We will add a reminder call to the center director to encourage staff responses.

• The study team proposes to increase the token of appreciation amount from $10 to $20 by adding a pre-paid $10 gift card to potential respondents in half of participating field test centers and increasing the post-pay gift card amount to $20 in the other half of participating field test centers (described in the next section).

• For the current ECE-ICHQ data collection, ACF proposes distributing electronic gift cards to respondents immediately after a respondent completes the survey on the web. This approach will attempt to mimic what the study team did in Phase 2—having field staff distribute gift cards immediately after a respondent handed in their completed survey.

• We believe the combination of this approach and a modified token of appreciation structure can mitigate our concern of not having a large enough sample size for analyses and having a lower response rate. The time-use data allows the research team to allocate center labor costs to various “functions” based on time spent on various activities. It is important to achieve a high response rate because our findings from Phase 2 indicate substantial variation in time use among staff in the same center. Responses from a large proportion of respondents will support estimation of time use for staff members in different job categories within a center (such as assistant teachers and lead teachers), which will increase the precision of labor cost allocations. Conversely, a lower response rate within a center is likely to introduce bias into the time-use estimates due to limited coverage of staff in different job categories.

Proposed Intervention for OIRA Approval

For the ECE-ICHQ field test, we propose adding a token of appreciation experiment in which the field test centers (about 67 in total) will be randomly assigned to one of two groups prior to recruitment:

1. Group A: $10 prepay gift card distributed to potential respondents in sealed envelopes by center directors prior to survey completion and $10 post pay electronic gift card emailed immediately after completing the web survey.

2. Group B: $20 post pay electronic gift card emailed immediately after completing the web survey.

The total (pre- and post-) token of appreciation for respondents completing the survey will increase from $10 to $20. The 13 Phase 2 sites that have already been recruited will just receive the $10 post-completion electronic gift card. This experimental design will allow us to measure impact on response, cost, and speed of response. This experiment will also be beneficial to the field since there have been few survey experiments conducted on high response surveys. We know our messaging works and staff are engaged because of the 89% phase 2 response rate, but this experiment allows us to see if we can maintain the high response without being physically on-site.

We expect the increased token of appreciation amount will be necessary to achieve a response rate that supports the development of measures related to center cost allocations across center functions. Because there is substantial variation in time use among staff members and across job categories within the same center, high response rates within each center are necessary to support precise allocations of labor costs to different center functions.

Expected Benefits and Proposed Assessment

We expect that our proposed token of appreciation design change will simulate the response benefit we saw in Phase 2. The new pre-token of appreciation is similar in intent to the in-person visits from Phase 2 and the electronic post-interview token of appreciation is similar to handing out gift cards on-site immediately following completion. The total increase in the gift card value will also help offset the increased effort required of staff to access and complete a web-based survey and is expected to result in more center staff completing the time-use survey and getting the response rate for the survey close to levels seen in Phase 2 of data collection.

Pre-paid tokens of appreciation have been shown to be more effective than post-paid tokens of appreciation across a variety of studies.¹² This has also been shown in studies with very similar population and sponsor to the current study. For example, one study³ shows the effective use of a $10 prepaid token of appreciation in a government sponsored survey of teachers.

What makes our proposed use of token of appreciations in this survey unusual is that we had a design that achieved a high response (89%), but we cannot use that design due to the COVID-19 pandemic. If we could do in-person site visits we would not be proposing a change to the token of appreciation. For the time use survey, the prepaid token of appreciation is a mechanism to replicate the success of face-to-face contact with respondents. The prepaid token of appreciation provides an opportunity to have center directors have contact with staff about the survey and encourages staff to open the invitation envelope. We are surveying a population that is inclined to help about a topic that is important to them. The invitation with a token of appreciation is designed to get them to prioritize the request so it is not forgotten rather than to convince them the survey important.

Pre-paid token of appreciations are particularly well suited for high-response small sample studies like this one because the cost of incentivizing people who do not ultimately respond is low. Randomizing field test centers into the two experimental groups will allow us to see if the combination of both the prepay and post pay gift cards helps us to achieve our response targets. We do not want to drop the post-paid gift card because it can help convince the slow responders we reach by email.

Collecting responses from as many staff members as possible within each center will support precise allocations of a center’s labor costs to functions and the development of cost measures that correctly reflect the allocation of center resources. Accurately estimating these measures is a fundamental goal of the study.

The study team will analyze differences in response rates, at the center-level and by respondent type (such as lead and assistant teachers), between the two experimental groups. They will also examine differences in the timing of survey completion between respondents in centers in each of the two experimental groups and assess the costs associated with each approach. The study team will prepare a memorandum that presents the results.

Revised Materials

We have revised the supporting statements to reflect the changes to the token of appreciation and proposed token of appreciation experiment. We have also included outreach and survey reminder materials for use with respondents in the time use survey pre- and post- gift card centers. Outreach materials for the post-survey gift card only centers were most recently approved with nonsubstantive changes on 4/28/21.

1 Singer, Eleanor and Ye, Conge, 2013. The Use and Effects of Incentives in Surveys. The ANNALS of the American Academy of Political and Social Science. Volume 645, Issue 1, January 2013, Pages 112-141

2 Andrew Mercer, Andrew Caporaso, David Cantor, Reanne Townsend, How Much Gets You How Much? Monetary Incentives and Response Rates in Household Surveys, Public Opinion Quarterly, Volume 79, Issue 1, Spring 2015, Pages 105–129

3 Robbins, M. W., Grimm, G., Stecher, B., and Opfer, V. D. (2018). A Comparison of Strategies for Recruiting Teachers into Survey Panels. Sage Open. Volume 8, Issue 3, pages 1-12.