NonSub Change Memo - ECE ICHQ TUS Pretest Updates

To: Jordan Cohen

Office of Information and Regulatory Affairs (OIRA)

Office of Management and Budget (OMB)

From: Ivelisse Martinez-Beck

Office of Planning, Research and Evaluation (OPRE)

Administration for Children and Families (ACF)

Date: September 29, 2021

Subject: NonSubstantive Change Request – Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study (OMB: 0970-0499)

This memo requests approval of nonsubstantive changes to the approved information collection, Assessing the Implementation and Cost of High Quality Early Care and Education: Comparative Multi-Case Study (OMB: 0970-0499).

Background

The goals of the Assessing the Implementation and Cost of High Quality Early Care and Education field test are to (1) refine the implementation measures to further improve their psychometric properties, (2) refine the cost measures to identify costs associated with each center function and across ages of children, and (3) test the associations between implementation, cost, and quality measures.

In light of the COVID-19 pandemic, the field test data collection scheduled for 2020 could not proceed as planned. We received approval for a nonsubstantive change request in March 2021 to conduct data collection in centers that had previously participated in the study. In April 2021, we received approval to engage additional centers in the field test (up to the approved 80 centers under this control number—some that previously participated in earlier phases and some new centers). Combined, these two requests were in line with the plans for the original field test. In September 2021, we received approval of the addition of a teaching staff survey (the SEQUAL).

After data collection began with the time-use survey in spring 2021, we discovered issues with the quality of data being submitted. As a result, we submitted and had approved the following changes:

• Administer a revised version of the time-use survey with the entire field test sample.

• Administer the revised time-use survey and the SEQUAL at the same time to all eligible center staff, and as a result:

  • combine the tokens of appreciation, and revise the approved experiments examining the timing of the token of appreciation.

We are now requesting approval of non-substantive changes to the time use survey based on a pretest of the revised survey with seven respondents. Data collection is scheduled to begin October 11, 2021 and the final survey will need to be programmed prior to fielding the instrument. As a result, we request a response as soon as possible.

Overview of Requested Changes

We pretested the revised time-use survey with 7 staff, including teachers, directors, and assistant directors, in a range of early care and education centers (such as Head Start centers and community-based centers that do not receive Head Start funding). We embedded pretest questions in the web-based instrument to gather immediate feedback and also conducted debrief calls with the respondents. We aggregated our feedback notes from all respondents. We learned that the corrections we made to the presentation of the time use grids and programming (such as presenting running sums in hours and having more soft checks on numbers) supported ease of completion in the web-based instrument, both on computers and mobile devices. These changes greatly improved respondent experience with the survey both in terms of understanding the questions and navigating through the survey efficiently.

We found trends in the experiences that respondents had in understanding certain survey questions or instructions that point to improvements we could make. We have revised the survey to improve its clarity in these areas and to further support collection of high quality, consistent, and comprehensive data across respondents. We have included the updated time use survey with this request.

The changes include:

Revision and item number	Reason for revision based on pre-test
Added A3 and A3a to ask about unpaid hours and changed the sum in N4a to total paid and unpaid hours combined. We revised N4a instructions to reflect these changes. Revised wording in A2 and throughout the survey to focus on paid hours, regardless of whether the hours were scheduled. We want respondents to focus on the distinction between paid and unpaid hours, rather than schedules.	Pre-test respondents were confused about what to do with their extra (not scheduled and unpaid) hours in A2 and N4a, and several expressed frustration with the requirements of the questions because they could not distinguish hours in activities between paid and unpaid hours. Five respondents said they worked more than their scheduled and paid hours, and two broke through the soft checks to report unpaid hours despite instructions. Collecting information on unpaid hours will help us capture essential information on what might be happening in those unpaid hours. In addition, the revisions will remove respondent frustration, improving their ability and willingness to give the attention needed to answer relevant questions accurately and proceed with the rest of the survey.
Added some activity descriptions to categories in N4a and N4b.	Several respondents felt that key activities were missing from the activities grids and did not know where to put the relevant hours.
Added a header to the N4 series grids to clarify the distinction between N4 (weekly) and N5 (periodic) activities. The header says: “Please do not include time you spend on activities that do not typically occur every week. We will ask about activities that occur periodically, including those that take place once a month or as needed throughout the year, later.”	Several respondents thought that questions were repeated (between N4 and N5) and did not report N5 hours or reported N5 periodic activity hours in N4.
Added N4x to ask about activities that are done during unpaid hours	While respondents could not distinguish specific hours of activities between paid and unpaid work hours, we would like to explore whether respondents can report about the types of activities they tend to do during unpaid hours, generally.
Revised N4d to clarify the comparison of current hours to pre-pandemic hours by specifying March 2020 as the period before the COVID pandemic	Several respondents misunderstood the N4d instructions and compared time periods within the pandemic, or early periods of the pandemic to pre-pandemic periods.
Placed at the top of each N4 series grid the instruction: “If you did more than one task at a time, please think about which one was your primary task”. It was previously in the N4 series introduction and the hyperlinked pop-up guidance for other grids in the series.	Several respondents indicated they had trouble separating hours among activity types when they felt like they were often doing multiple things at once.
Added text, “Your best guess is fine” to the N5 series questions to help respondents understand estimates (not precise figures) are acceptable.	Several respondents indicated they didn’t feel confident in their answers to N5 questions, or weren’t sure if they were doing it right and did not fill them in as a result. We want to provide assurance that estimates are acceptable to encourage respondents to complete the questions.
Edited for plain language	We improved readability and access to questions and instructions throughout the survey.