NonSub Change Attachment - Incentive Change Justification

DATE: DATE, 2021

TO: Jordan Cohen, Office of Information and Regulatory Affairs, Office of Management and Budget

FROM: Meryl Barofsky, Office of Planning, Research, and Evaluation, Administration for Children and Families

RE: Request for an increase in respondent incentive amounts for Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study (OMB: 0970-0499)

Background

• Type of Request: Non-Substantive Change to Incentives Structure to a Survey in a Multipart Study
  
  

• Study Features Salient to Request:

The goal of the Implementation and Cost of High Quality Early Care and Education (ECE-ICHQ) study is to create a technically sound, feasible, and useful instrument that will provide consistent and systematic measures of the implementation and costs of quality to help fill the knowledge gap about the cost of providing and improving quality in early care and education (ECE). Mathematica conducted work for the ECE-ICHQ project from September 29, 2014 through September 30, 2019. During that time, the ECE-ICHQ study team developed a conceptual framework; conducted a review of the literature; consulted with a technical expert panel; collected and summarized findings from two phases of data collection; and developed instruments and measures of implementation, costs, and time-use. In October 2019, Mathematica began preparing for a field test of the new measures and launched the field test for three weeks in March 2020 until it needed to cease due to the COVID-19 pandemic. Beginning in March 2021, the study will conduct a feasibility study under the current approval to learn how our instruments and measures of implementation and cost are able to capture the service changes that are occurring across the child care industry due to the COVID-19 pandemic.

During the feasibility study, the study team will return to early care and education centers that participated in phase 2 of the multi-case study and ask them to participate in the latest round of data collection that includes: (1) center management participating in phone interviews to gather information about center implementation activities and completing a cost work book, and (2) eligible center staff completing a 15-minute survey to learn how they spend their work time (a time-use survey). Though we expect staff turn-over within centers since phase 2, we do expect some overlap in respondents. The time-use survey is the focus of this request.

Progress to Date:

We have not yet begun data collection for the feasibility study. We expect to begin recruitment in March 2021. We will recruit nine phase 2 centers to participate. From prior phases, we estimate each center will have, on average, 16 center staff eligible to complete the time-use survey, resulting in a sample size of about 144 for the survey.

• Previous Terms of Clearance:

The following tokens of appreciation for the ECE-ICHQ time-use survey were approved by OMB under Control Number 0970-0499:

• PHASE 1: a $10 gift card, 15 minutes to complete, no field staff on site

• PHASE 2: a $10 gift card, 15 minutes to complete, field staff on site

• FIELD TEST: a $10 gift card (POSTPONED; Focus of this change request), 15 minutes to complete

• Time Sensitivity: We plan to begin data collection in March 2021. Many child care centers close for the summer, so we have a 3-month window to collect time-use surveys from staff in the recruited centers. Additionally, based on the findings of the feasibility study, we will determine when we can launch the full field test. Any delays in beginning data collection would make our data collection window shorter and could impact our ability to get completed surveys from center staff. This would also delay the full field test. Additionally, we need time prior to our launch to purchase gift cards.

Request Overview

ACF requests a non-substantive modification to the ECE-ICHQ time-use survey incentive structure from a $10 gift card to a $20 gift card for each respondent. The goal of this change is to ensure we have a large enough sample size for analysis and to address lower-than-expected response rates and observable non-response bias in the data we collect. In phase 2 of data collection, our response rate on the time-use survey was 89%. We were able to accomplish this by having field staff visit the centers, distribute paper surveys, and distribute gift cards in person immediately after center staff completed their surveys. In phase 1 of data collection our response rate was much lower, 48% when we did not send field staff to centers to distribute paper surveys and distribute gift cards immediately after center staff completed their survey. For phase 3, we expect response rates on the time-use survey to be lower than the 89% response rate we obtained in phase 2 due to two reasons: (1) we will not be able to send field staff to centers to follow up with staff; and (2) staff will need to access and complete a web-based version of the survey, which they cannot complete during regular work hours because they do not have access to computers or phones at this time.

The study attempts to collect data from all teaching staff at a center to understand the variation within centers among staff with similar roles. A 90% response rate is our threshold for being able to do this. Furthermore, our phase 3 data collection will occur with nine centers that participated in phase 2, with an estimated number of 144 center staff that would make up our universe of time-use survey respondents. With such a small sample size, it is imperative that we are able to achieve strong response rates to allow us to have sufficient data to conduct meaningful analyses.

The remainder of this memo highlights ACF’s efforts to date to ensure high response rates and address potential non-response bias; describes proposed changes for phase 3 of data collection; and explains the hypothesized benefits of the proposed approach to increase the time-use survey incentive from a $10 gift card to a $20 gift card for each respondent.

Mitigation to Date

Since phase 1 of the study, as part of regular survey administration protocol, ACF has actively monitored survey response rates in the multi-phase ECE-ICHQ study. In phase 2, the contract team took the following steps to address emerging concerns with lower than expected response rates and potential non-response bias:

• Sending field staff to centers to invite center staff to complete the time-use survey in-person and providing paper and web copies of the surveys;

• Giving center staff that complete the time-use survey during our visit the $10 gift card upon completion, rather than mailing it to them afterwards

Respondent	Target N	N Secured	Response Rate
Phase 1 (no field staff)	212	101	48%
Phase 2 (field staff visit centers to invite center staff to complete time-use survey, and distribute gift cards upon completion)	483	430	89%

Plans for Future Mitigation

• For the feasibility study of the ECE-ICHQ study, ACF proposes distributing electronic gift cards to respondents immediately after a respondent completes the survey on the web. This approach will attempt to mimic what the study team did in phase 2—having field staff distribute gift cards immediately after a respondent handed in their completed survey.

• The study team also plans to adjust the schedule of email reminders that will be sent to non-respondents. We will incorporate additional reminders and decrease the length of time between each reminder. We will add a reminder call to the center director to encourage staff responses.

• We believe the combination of this approach and a modified incentive structure can mitigate our concern of not having a large enough sample size for analyses and having a lower response rate. The time-use data is essential to allow the research team to allocate center labor costs to various “functions” based on time spent on various activities. It is important to achieve a high response rate because our findings from Phase 2 indicate substantial variation in time use among staff even in the same center. Responses from a large proportion of respondents will support estimation of time use for staff members in different job categories within a center (such as assistant teachers and lead teachers), which will increase the precision of labor cost allocations. Conversely, a lower response rate within a center is likely to introduce bias into the time-use estimates due to limited coverage of staff in different job categories.

Proposed Intervention for OIRA Approval

For the feasibility study of the ECE-ICHQ study, we propose increasing the incentive amount of the 15 minute time-use survey from $10 to $20. We expect the increased incentive amount will be necessary to achieve a response rate that supports the development of measures related to center cost allocations across center functions. Because there is substantial variation in time use among staff members and across job categories within the same center, high response rates within each center are necessary to support precise allocations of labor costs to different center functions.

Expected Benefits and Proposed Assessment

We expect that our proposed incentive increase will help offset the increased effort required of staff to access and complete a web-based survey and result in more center staff completing the time-use survey and getting the response rate for the survey close to levels seen in phase 2 of data collection. Collecting responses from as many staff members as possible within each center will support precise allocations of a center’s labor costs to functions and the development of cost measures that correctly reflect the allocation of center resources. Accurately estimating these measures is a fundamental goal of the study.