Supporting Statement B - Chafee SOTA Generic Request_7.18.23

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Chafee Strengthening Outcomes for Transition to Adulthood Project Overarching Generic

OMB Information Collection Request

New Umbrella Generic

Supporting Statement

Part B

July 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Kelly Jedd McKenzie and Harmanpreet Bhatti

Part B

B1. Objectives

Study Objectives

The information collected under this umbrella generic clearance is intended to inform whether innovative learning methods can be utilized to evaluate interventions and services for youth transitioning out of foster care through the Chafee Strengthening Outcomes for Transition to Adulthood (Chafee SOTA) project. The data collections under this umbrella generic clearance consist of a series of mixed methods studies to evaluate promising practices and improve the feasibility and rigor of evaluations that test the effectiveness of program services or components. The purpose of these data collection efforts is to inform ACF programming by building the evidence about what works to improve outcomes for the target population, and to identify innovative learning methods that address common evaluation challenges.

Generalizability of Results

The population of interest is youth in or transitioning out of foster care, and information collected in this project is intended to inform our understanding of how to best serve that population. The information collected under this umbrella generic is intended to produce estimates of the impact of program services or components in chosen sites, not to promote statistical generalization to other sites or service populations.

Appropriateness of Study Design and Methods for Planned Uses

As noted in Supporting Statement A, this information is not intended to be used as the principal basis for public policy decisions and is not expected to meet the threshold of influential or highly influential scientific information.

B2. Methods and Design

Target Population

Target sites for this series of studies consist of regional, state, and local programs providing services to youth and young adults with experience in foster care. Potential sites have been identified through a process previously approved by OMB under ACF’s overarching generic, Formative Data Collections for ACF Research (0970–0356)¹, which included (1) a call for nominations; (2) follow-up telephone interviews with program leadership; and (3) onsite evaluability assessments, including program observations, focus groups, and interviews with program staff and participants. Based on these information activities, the research team and ACF will select programs to invite to participate in the evaluation activities covered under this generic. Criteria for selection include organizational capacity to support an evaluation and consistency with proposed research methods.

The target respondents in these generic information collections (GenICs) include but are not limited to:

• Program directors and program staff from programs serving youth with foster care experience, as well as directors and staff from their partner agencies.

• Current, past, or potential participants in programs serving youth and young adults with foster care experience (i.e., including potential participants who are included in comparison groups)

Sampling and Site Selection

Individual GenICs will provide ACF with information to inform ACF decision-making and program support. In studies covered under this overarching generic, obtaining probability-based samples to reach the desired subpopulations of interest would be cost-prohibitive and not needed for achieving study goals. As such, ACF does not, at this time, anticipate undertaking a statistically sophisticated strategy for respondent selection. A description of the plans for selecting respondents will be provided to OMB as part of each individual GenIC request. Purposeful, targeted sampling through specific programs and other non-probability sampling designs will be used to develop a pool of potential respondents. The limitations associated with purposive or any sampling method will be described in any GenIC submission, and will be clearly stated in any publications produced for this project.

B3. Design of Data Collection Instruments

Development of Data Collection Instruments

Data collection activities may include:

• Semi-structured interviews (in-person, telephone, or web-based)

• Focus groups

• Questionnaires/Surveys

• Administrative data

• Document analysis

For the collection of primary data through semi-structured interviews, focus groups, and questionnaires/surveys, we will draft data collection instruments and consent forms that will be used by evaluation staff and/or program staff. Example instruments are provided in the attached Example Instruments A-F.

We will use cognitive testing and pre-testing to review and strengthen instruments before data collection begins. Informed consent will be obtained before any primary data collection begins. Individuals will be identified for data collection through protocols tailored to each program included in the evaluation. Strategies may include invitations to current, former, or potential program participants via email and consent-to-contact processes through program enrollment (i.e., program participants will be invited by program staff to share their contact information with the study team, who will then invite them to participate in the study).

Program staff will provide deidentified administrative data from internal program records, which may be used in combination with administrative records from state and local agencies in focal and comparison sites. Documentary data may include program websites, training materials, pamphlets, or similar materials that describe program activities, theories of change, or frameworks.

B4. Collection of Data and Quality Control

For primary data collected, we will implement a process for routine monitoring of the data collection process and the quality of the data. We anticipate establishing processes for electronic data collection that ensure data are collected and computerized in a standardized fashion and for secure transfer procedures that will facilitate safe and prompt submission of primary data collected on site. We will establish a schedule for routine submission of the data to ensure that there are no lags in the collection and submission of data that could result in incomplete or inaccurate information about the program or its participants. We will implement a process for immediate review of the data for quality, completeness, and integrity. As part of our quality review process, we will meet with program staff on a regular basis to review the data, discuss any issues, and problem-solve challenges or barriers staff may be facing in the timely and accurate collection of the data. If any primary data collected are of insufficient quality to meet the needs of the study, we will implement corrective actions through trainings and technical assistance or, if needed, through amendments to the data collection procedures to address quality issues. We will also consult with the program staff to ensure we understand any program decisions, such as when and how exits from the program are recorded, that may impact our analysis or interpretation of the data.

B5. Response Rates and Potential Nonresponse Bias

Response Rates

Expected response rates will vary for individual information collection requests. Information about expected response rates will be provided with each GenIC request.

In general, callbacks will be used to maximize response rates for interviews; reminder phone calls, letters, emails and/or second questionnaires are some methods that will be used to maximize response rates in mail and web-based surveys. Reminder phone calls, letters, and/or emails to participants are some methods that will be used to encourage them to keep their appointments. Each GenIC request will provide specific information about methods to maximize response rates and deal with nonresponse.

For data collection from selected sites, the research team will work closely with administrators and staff to develop recruitment strategies for participants and program staff for focus groups and interviews, particularly to make sure we gather a group that reflects a mix of experiences. To further increase the likelihood of participation, we will also offer tokens of appreciation to participants for focus groups, interviews, and/or surveys, as discussed in Supporting Statement Part A.

NonResponse

As participants will not be randomly sampled and findings are not intended to be representative, non-response bias will not be calculated. Respondent demographics will be documented and reported in written materials associated with the data collection.

B6. Production of Estimates and Projections

Quasi-experimental impact estimates, primarily produced through linear and logistic regression, will be made using methods such as predictive analytics, single-case designs, propensity score analysis, nonequivalent group designs, and regression discontinuity. When appropriate, subgroup analyses may be conducted.

B7. Data Handling and Analysis

Data Handling

We will use several methods to protect the integrity of collected data while we are preparing and analyzing them. Data collected through interviews and focus groups will be transcribed and coded by multiple individuals, and interrater reliability will be established. When possible, quantitative analyses that require linking multiple datasets will be merged with two or more identifiers to ensure an accurate match of records. We will develop all appropriate codebooks and data logs documenting decisions made. Finally, we will run basic quality assurance checks on all data. For example, we will conduct initial analyses that include basic quality assurance checks. These will involve assessing for consistency across variables that should be consistent, examining outlier values, addressing any needed modeling assumptions, and identifying incomplete or missing data.

Data Analysis

The research team will develop a mixed methods analysis plan for each site. The quantitative analytic approach will depend on the study design selected, in addition to the type of data collected. Our plan also will include analysis of qualitative data collected through interviews, focus groups, and program documents. The analysis plan will describe how different types of qualitative data will be prepared for analysis. We will use NVivo, a qualitative software program, to organize the data and thematically code interview transcripts and document abstractions. If appropriate, similar sets of codes will be used across multiple programs to allow for cross-site analyses. The analytic plan will detail our approach to establishing inter-coder reliability, including meetings to review the codes and a process whereby pairs of researchers code the same data and check for inter-rater reliability. We will aim for a reliability of 95% agreement in codes. When appropriate, the plan will include information about integrative analyses that pull across the different qualitative and quantitative data sources to provide a broader understanding of the program and inform cross-site learnings.

Data Use

Under this umbrella generic, information collected is meant to inform ACF activities and may be incorporated into documents or presentations that are made public such as through conference presentations, websites, or social media.

The following are some examples of ways in which we may share information resulting from these data collections: technical assistance plans, webinars, presentations, infographics, issue briefs/reports, project specific reports, or other documents relevant to the field, such as federal leadership and staff, grantees, local implementing agencies, researchers, and/or T/TA providers. We may also request information for the sole purpose of publication in cases where we are working to create a single source for users (clients, programs, researchers) to find information about resources such as services in their area, TA materials, different types of programs or systems available, or research using ACF data.

In sharing findings, we will describe the study methods and limitations regarding generalizability and as a basis for policy. Any planned uses, including for publication or sharing of information from this IC will be described and submitted for approval in each individual GenIC.

B8. Contact Persons

Debra J. Rog, Ph.D.

Westat

Address: 1600 Research Blvd, Rockville, MD, 20850

Phone: 301-279-4594

Email: [email protected]

Kathryn A. Henderson, Ph.D.

Westat

Address: 1600 Research Blvd, Rockville, MD, 20850

Phone: 301-610-4849

Email: [email protected]

John D. Fluke, Ph.D.

Kempe Center for the Prevention and Treatment of Child Abuse and Neglect

Address: 13123 East 16th Avenue, B390, Aurora, CO, 80045

Phone: 303-864-5219

Email: [email protected]

Attachments

Instrument A: Sample Administrator Interview

Instrument B: Sample Staff Interview

Instrument C: Sample Youth Focus Group

Instrument D: Sample Youth Survey

Instrument E: Sample Administrative Data Extraction

Instrument F: Sample Program Document Request

Appendix 1: Sample Interview Consent Form

Appendix 2: Sample Focus Group Consent Form

1 GenIC approved August 7, 2022: The John H. Chafee Foster Care Program for Successful Transition to Adulthood Strengthening Outcomes for Transition to Adulthood (Chafee SOTA) Project

7