SSB - Formative GenIC - ECE RISE

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Child Care Research and Evaluation Capacity Building Center Application for Receiving Individualized Support for Evaluation Capacity Building (ECE-RISE)

Formative Data Collections for ACF Research

0970 - 0356

Supporting Statement

Part B

October 2021

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Alysia Blandon

Part B

B1. Objectives

Study Objectives

The Child Care Research and Evaluation Capacity Building Center’s goal is to build the capacity of CCDF Lead Agencies to do and use research to inform decisions. This information collection request focuses on identifying a specific subset of CCDF Lead Agencies that need additional support. Information gathered on the application and self-assessment will allow us to assess CCDF Lead Agencies’ research and evaluation capacities. The information collected will contribute to OPRE’s understanding of State, Local and Tribal CCDF agencies’ strengths and challenges related to doing, partnering to get, and using research and evaluation.

This is a project funded by ACF to both build the capacity of the CCDF Lead Agencies themselves but also to inform future research and evaluation capacity building efforts for ACF-funded programs.

Information gathered will be used to identify Lead Agencies that may benefit from intensive research and evaluation capacity-building supports. We will use the information collected through the application form, self-assessment, and interview to select participants for ECE-RISE (Receiving Individualized Support for Evaluation Capacity Building). ECE-RISE will provide selected agencies with research and evaluation support as they carry out their own projects. We will use three criteria: 1) How well an agency’s research and evaluation needs match the types of support ECE-RISE offers; 2) Whether an agency’s questions can be studied and answered within an 18-month period; 3) Whether the agency team and leadership can invest sufficient time and resources to work with the ECE-RISE team.

ACF has contracted with Mathematica to complete this work. Mathematica staff will provide research supports to two CCDF Lead Agencies selected through the application process for up to 18 months.

Generalizability of Results

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information. Data collection associated with this study is intended to present internally-valid description of CCDF lead agencies and their capacities; it is not to promote statistical generalization to other sites or service populations.

Although not generalizable, ACF will learn valuable lessons to inform future research and evaluation capacity building efforts for ACF-funded programs.

Appropriateness of Study Design and Methods for Planned Uses

The information collected will be used to identify Lead Agencies that may benefit from intensive research and evaluation capacity-building supports. As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information. The data collected are not intended to be representative, and findings may not necessarily apply to all CCDF Lead Agencies. The application form was designed to identify an agency’s interest in this work. The self-assessment is designed to assess the research and evaluation capacities of applicant agencies and the interviews are designed to capture additional detail about the question of interest identified by agencies in the application form, areas of need highlighted in responses to the self-assessment, and the potential and feasibility of conducting a research project during the ECE-RISE period. All data collected will be used to make informed decisions about agencies in order to offer them tailored supports for research and evaluation. This study does not include an impact evaluation and will not be used to assess participants’ outcomes.

B2. Methods and Design

Target Population

Although all CCDF Lead Agencies will be eligible to apply to ECE-RISE, this opportunity is intended to benefit CCDF Lead Agencies that have little or no experience with research and evaluation but want to start to build sustained internal research capacity. There is no expectation that CCDF Lead Agencies already have a research or evaluation project planned in order to participate in ECE-RISE.

Respondent Recruitment and Site Selection

We will conduct both purposive and broad recruitment. We will use a purposive approach to recruit applicants and to identify up four agencies to participate in interviews. We will reach out to staff from the ACF Office of Child Care (OCC) Central and Regional Offices, and National Technical Assistance (TA) centers (such as the Child Care State Capacity Building Center, the National Center on Subsidy Innovation and Accountability, and the Tribal Early Childhood Development Center) to help identify Lead Agencies that might benefit from intensive support. We will also use information from the needs assessment survey to identify CCDF Lead Agencies that might be good candidates for this opportunity based on their Needs Assessment survey responses.¹ Once we have created a list of potential agency applicants, we will conduct targeted outreach to ensure that CCDF Lead Agencies that may benefit the most from ECE-RISE are aware of the opportunity. Additionally, we will also conduct broad outreach to all lead agencies through email listservs and messages from OCC, OPRE, TA Centers and other trusted sources.

We will use information from the application form and self-assessment to identify applicant Agencies with research and evaluation capacity that is neither (1) too high, and not in need of intensive support, or (2) too low and lacking resources, staff knowledge and skills, and agency supports and infrastructure to be able to make progress toward capacity building in an 18-month period. The self-assessment is identical to the previously fielded Needs Assessment Survey and will only be completed by agencies that did not already complete the survey for the needs assessment study.

Using information from the application and self-assessment, we will select four agencies for interviews based on areas of strength and weakness in capacities, and potential for conducting a research project (with support) and benefiting from the individualized support provided.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

ECE-RISE capacity building activities are intended to reach CCDF Lead Agencies that may have little experience with federal application processes. We developed a three-stage approach to the application and selection process focusing on collecting key information at each stage, minimizing burden as much as possible, and engaging rather than deterring applicants.

Application form (Instrument 1). The application form was designed to identify an agency’s question of interest, with the understanding and expectation that ECE-RISE staff will support them in their research and evaluation efforts. The application form is written in plain language and is designed to collect key information about agency staff and interests through short open-ended responses.

Self-assessment (Instrument 2). To assess the research and evaluation capacities of applicant agencies, we chose to use an instrument previously used for the Child Care Research and Evaluation Capacity Building Center Needs Assessment Study. Survey questions focus on gaining an understanding of research and evaluation capacities of the agencies. The instrument follows best practices in eliciting accurate self-reports, including selecting or designing measures that ask for concrete examples of past performance to support opinions about past research activities and providing benchmarks and other comparative methods when assessing technical skills. The survey was refined with input from former State and Tribal agency administrators.

Interview guide (Instrument 3). The interview protocol is designed to capture additional detail about the question of interest identified by agencies in the application form, areas of need highlighted in responses to the self-assessment, and the potential and feasibility of conducting a research project during the ECE-RISE period.

B4. Collection of Data and Quality Control

Mathematica staff will conduct all outreach and data collection activities related to the application form, self-assessment, and interview.

Recruitment of applicants

We will establish an external project website to house project information, including contact information for the ECE-RISE team, and application materials that will be useful to CCDF lead agencies. To advertise the opportunity to all CCDF Lead Agencies, we will send email notifications (Appendix B1) through the OCC Director and/or through the Child Care Research and Evaluation Capacity Building Center email distribution list. We will send the announcements to staff identified as the CCDF Lead Administrator(s). The email notifications will describe the opportunity and encourage Agency staff to consider applying. We will send an email notification once after the application materials have been posted and then again halfway through the application period. The project information sheet (Appendix A) will be attached to the notification emails. The project information sheet will describe the goals of the intensive capacity building support and the benefits of participation and provide an overview of the application and selection process. The project information sheet will include links to access the application materials and register for an information session.

We will also work to advertise the opportunity through traditional means for federal grant and technical assistance initiatives, including posting announcements on ACF and National T/TA Center websites, distributing announcements through their listservs (Appendix B2), and making verbal announcements at in-person or virtual meetings (Appendix B3).

We will also conduct targeted outreach to ensure that CCDF Lead Agencies that may benefit the most from ECE-RISE are aware of the opportunity. We will reach out to OCC Central Office staff to help us identify Lead Agencies that might benefit from intensive support and also connect us with National TA centers that are most knowledgeable about CCDF Lead Agency capacities and needs around research and evaluation. Based on input from OCC, we will reach out to the relevant TA centers (such as the Child Care State Capacity Building Center, the National Center on Subsidy Innovation and Accountability, and the Tribal Early Childhood Development Center) and to OCC Regional staff for further help in identifying and reaching out to potential participants.

Once we have created a list of Agencies for targeted outreach, we will ask TA staff or OCC Regional staff to share information about the opportunity to potential participants. We will provide an email template to support this outreach (Appendix B4). We expect to begin targeted outreach after application materials have been posted.

Data collection

Application form. The application form (Instrument 1) will be posted on the project website and sent as an attachment in email notifications sent by OCC (Appendix B1). Applicants will be asked to upload their completed applications to a project-specific Box site.

Self-assessment. We will send an invitation email (Appendix B4) to the designated point of contact at each agency as identified through the application form. The email will contain a password protected link to the web-based assessment (Instrument 2).

Interview. We will conduct 60-90 minute semi-structured interviews via web conference with key staff from each of the four Lead Agencies selected based on applications and self-assessments. We will reach out to applicants by email to schedule the interview (Appendix B5). Using the interview guide (Instrument 3) we will discuss in more detail the following topics:

• Feasibility of proposed research project

• Confirming areas of need

• Partners and strengths that can help address areas of need

• Agency goals for success and potential to make progress on research project within 18 month period

• Openness to capacity building process

• Potential for the Agency to sustain (or expand) support for research and evaluation beyond the ECE-RISE support period

The same two Mathematica researchers who score the application forms will conduct all interviews to ensure consistency; one person will lead each interview and the other will take notes and ask clarification questions as needed. With approval from the respondent, a member of the interview team will record responses to each question into an interview template to prepare the material for review and scoring. A senior researcher not involved with the interviews will review the interview write-ups for quality and completeness. We will conduct limited individualized follow-up by email for clarification purposes, if needed, to ensure we have complete information across the four Agencies.

B5. Response Rates and Potential Nonresponse Bias

Response Rate

The information collected through the application, self-assessment, and interview are not designed to produce statistically generalizable findings and participation is wholly at the respondent’s discretion.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

The data will not be used to generate population estimates, either for internal use or dissemination.

B7. Data Handling and Analysis

Data Handling

Two Mathematica researchers will independently score each application form received using a scoring rubric. We will conduct limited individualized follow-up by email for clarification purposes, if needed, to ensure we have complete information across the four Agencies. To ensure that scoring is complete and consistently implemented, we will use a standard scoring template.

The self-assessment will be collected via Confirmit which allows automatic logic checks to be built into the design of surveys, including mandatory response fields, prespecified valid value ranges and prespecified patterns of logical and illogical responses across items. We will use these checks to reduce item missingness, require that responses fall within possible ranges and request confirmation from respondents when reported values are possible but implausible.

Researchers responsible for scoring applications and interviews will access each agencies’ materials on the project’s restricted-access folder that is only accessible to Mathematica staff who have a business need-to-know, as restricted by identity-based policies and access control lists.

Data Analysis

Mathematica researchers will score applications using a scoring rubric. They will compare their scores and discuss areas of discrepancies to reach an agreement in the application scoring. They will provide their scoring sheets along with a written explanation of any items or areas that required reconciliation for review by a senior researcher. The scoring for the application is broken into four sections: (1) information about the potential research project, (2) agency and staff partners, (3) commitment and support, and (4) goals and strengths.

Responses to the self-assessment will also be coded using a scoring rubric created for ECE-RISE. The rubric codes different assessment items associated with aspects of research and evaluation capacity as: “Area of significant need,” “Area of need,” “On track but could improve,” and “Thriving.” Once the items are coded, we will assign ratings of “high” or “low” for the following 10 categories of research and evaluation capacity: policy and program knowledge, evaluation and technical ability, organizational resources and supports for research, organizational learning, and dissemination that signal capacity. We developed a rating guide to characterize the applicant’s capacity in each of the 10 categories. Specifically, for each category, we developed thresholds for scores to assign a rating of “low” or “high” capacity by considering the number and content of all the items. The scoring guide will help ensure that ratings are consistent across agencies.

We will combine information from the application form and self-assessment to examine both the number and nature of low and high capacity categories for each agency. We will assess each agency’s specific strengths and weaknesses in light of their responses and select agencies that are the best fit for ECE-RISE based on their capacities, potential research projects, and team structures that could most benefit from the intensive supports over 18 months.

We will use each Lead Agency’s interview responses to write a narrative assessment of (1) demonstration of level of interest and commitment to the ECE-RISE opportunity and (2) intensity or type of support that may be needed, as signaled by the capacity constraints the agency faces and their ability to carry out the research project. Exhibit 5 presents guiding factors to consider by interview section. Each researcher will assign a rating from 1 to 4, highest to lowest to designate “strongly agree,” “agree,” “disagree,” “strongly disagree,” with each of the guiding factors. They will also prepare a short narrative for each of the six interview sections to justify their rating, drawing on excerpts from the interview transcripts to demonstrate points. They will use their section ratings and narratives to assign an overall rating from 1 to 4 (from highest to lowest) for each Agency with a summary of their rationale.

Data Use

The data collected will be used to select agencies that might benefit from more intensive support. Based on our assessment of interest and types of supports needed, we will work with ACF to select two Lead Agencies to participate in ECE-RISE.

B8. Contact Person(s)

Alysia Blandon, Office of Planning, Research, and Evaluation, [email protected]

Pia Caronongan, Mathematica, [email protected]

Gretchen Kirby, Mathematica, [email protected]

Attachments

Instrument 1: Application form

Instrument 2: Self-assessment

Instrument 3: Interview Guide

Appendix A: Project information sheet

Appendix B: ECE-RISE outreach materials

1 The Needs Assessment Survey was fielded in April to August 2021 and information collection during that time was approved under the same umbrella generic, Formative Data Collections for ACF Research (0970-0356)

7