Alternative Supporting Statement for Information Collections Designed for
Research, Public Health Surveillance, and Program Evaluation Purposes
Variations in Implementation of Quality Interventions
Formative Data Collections for ACF Research
0970 - 0356
Supporting Statement
Part B
JUNE 2022
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers:
Ivelisse Martinez-Beck
Amy Madigan
Part B
B1. Objectives
Study Objectives
The information collection proposed as part of the Variations in Implementation of Quality Interventions: Examining the Quality-Child Outcomes Relationship in Child Care and Early Education Project (VIQI Project), consisting of up to 130 one-time interviews with center administrators and up to 20 interviews with state and local ECE stakeholders, will help contextualize the implementation and impact findings of the VIQI study. The information collection has the following objectives:
To understand the experiences, successes and challenges faced by centers who participated in the VIQI project and how those experiences compared to prior years, and
To understand the state and local context (e.g., economic and employment conditions, the COVID-19 pandemic, and changes to ECE standards or requirements) during the 2021-2022 school year that may have affected participating ECE centers.
Generalizability of Results
This study is intended to present descriptive information about the ECE context during the VIQI study. This information collection is not intended to promote statistical generalization within or beyond the target population.
Appropriateness of Study Design and Methods for Planned Uses
Semi-structured interviews are proposed because they enable in-depth description and understanding of the contextual information needed to help inform and contextualize the study’s findings. They allow all interviewers to have the same structured set of questions while also allowing the interviewer to follow-up and probe on individual interviewee’s responses as needed.
We will not use the data to make statistical inferences or generalize findings beyond the study sample. The information will only be used to describe the centers, localities, and states in the VIQI sample. We will include the limitation that the sample is not representative in any written products associated with the study.
As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.
B2. Methods and Design
Target Population
The target population is center administrators at centers participating in the VIQI project and state and local ECE stakeholders from the localities or states in which the VIQI project is being conducted.
Sampling
The study team will leverage its relationships with stakeholders in each of the VIQI localities and at VIQI centers to recruit participants for these interviews. The team will use the email templates (see Appendix A) and talking points (see introductions to Instruments 1 and 2) in reaching out to their contacts.
One center administrator from each of the centers participating in the VIQI study will be asked to participate in an interview. This will be the administrator who was previously selected to participate in the VIQI project (e.g. to take the administrator survey under OMB #0970-0508).
Key state and/or local-level ECE stakeholders who have been engaged in the VIQI project (e.g., as part of our initial landscaping discussions in each locality) will be asked to participate in an interview to help us understand aspects of the local context (e.g., ECE systems-level changes, the COVID-19 pandemic context, local employment and economic conditions) that may influence implementation and business-as-usual ECE. These will include administrators who have oversight of child care and Head Start programs in that locality. We will aim to identify up to two locality or state-level informants in each of the ten VIQI localities.
B3. Design of Data Collection Instruments
Development of Data Collection Instruments
The interview protocols were developed by the project team to address the research questions. We first identified key information needed for a fuller understanding of the context under which the VIQI impact evaluation unfolded, particularly related to the effects of COVID-19 on center functioning, staffing, and instruction as well as on the locality or state. The protocols have been designed so that the interviewer can tailor each interview to the intervention group the center administrator was in and to the particular role different state/local ECE stakeholders may play.
In order to avoid measurement error (e.g., leading, or unclear questions), we revised the interview protocol based on feedback from the project team, including both contractor and ACF staff.
B4. Collection of Data and Quality Control
Members of the research team will contact potential participants and facilitate the semi-structured one-on-one discussions in person, on the phone, or via video conferencing platform (e.g., Zoom). We expect that recruitment will be an iterative process. Members of the project team will contact center administrators and state and local ECE stakeholders individually by e-mail, explaining the purpose of the interview, describing the nature of the discussion, and inviting their participation (see Appendix A for an e-mail template/talking points). If the administrator agrees to participate, a member of the project team will schedule a call at a time that is convenient to the participant. If an administrator does not respond, a member of the project team may reach out by phone.
If an invited individual is unable to participate, the research team will ask whether another representative of the individual’s organization might be appropriate and available for a discussion and will invite suggested administrators as needed. For state and local ECE stakeholders, if no alternative is given, the research team will contact an alternative informant that we expect can discuss the local context in that locality or state.
A member of the research team will facilitate each interview according to the Center Administrator Protocol (Instrument 1) or the State and Local ECE Stakeholder Protocol (Instrument 2) depending on the participant’s role, tailoring the conversation according to participants’ backgrounds and expertise. Each interview will last no more than 60 minutes. The interviews will be recorded, with participants permission, and transcribed. A member of the research team will also take detailed written notes on each conversation.
B5. Response Rates and Potential Nonresponse Bias
Response Rates
The interviews are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion. Response rates will not be calculated or reported.
NonResponse
As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. We will qualitatively assess non-responses to monitor gaps in the research conditions, ECE setting type, and localities represented in our data and to tailor our recruitment efforts to improve the data collected and reduce burden on respondents.
B6. Production of Estimates and Projections
The data will not be used to generate population estimates, either for internal use or dissemination.
B7. Data Handling and Analysis
Data Handling
Electronic notes and audio recordings taken during interviews will be stored in a secure, password-protected location, according to the project’s data security plan. Audio recordings will be deleted after transcription.
Data Analysis
Interview transcripts will be analyzed by research staff to identify themes. The qualitative data analysis will follow a deductive coding process. Codes will be developed to capture information about the ECE landscape and workforce during the VIQI project. The coding team will be trained on the scheme prior to coding and, during the coding period, will meet regularly to review assumptions (ensure inter-rater reliability); address analytic challenges; discuss, compare, and cross-check themes; identify major themes that were not reflected in the deductive coding scheme; and, reach shared understandings that are valid and reliable. These analyses will be used to provide contextual information for the impact and implementation findings.
Data Use
Reports and/or briefs will be published to summarize the findings from the Impact Evaluation and Process Study. Themes and quotes from the Center Administrator and State and Local ECE Stakeholder interviews may be included in these dissemination products to contextualize implementation and impact findings. Any limitations to the data, including generalizability, will be included in publicly shared information. No personally identifying information, like names, will be shared.
B8. Contact Persons
JoAnn Hsueh, MDRC, [email protected]
Michelle Maier, MDRC, [email protected]
Marie-Andree Somers, MDRC, [email protected]
Electra Small, MDRC, [email protected]
Margaret Burchinal, University of Virginia, [email protected]
Jean Lennon, RTI International, [email protected]
Jennifer Kenney, RTI International, [email protected]
Attachments
Instrument 1: Center administrator interview protocol
Instrument 2: State and local ECE stakeholders interview protocol
Appendix A: Interview outreach
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
Author | Michelle Maier |
File Modified | 0000-00-00 |
File Created | 2023-10-30 |