Supporting Statement A - Chafee SOTA Generic Request_7.18.23

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Chafee Strengthening Outcomes for Transition to Adulthood Project Overarching Generic

OMB Information Collection Request

New Umbrella Generic

Supporting Statement

Part A

July 2023

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers: Kelly Jedd McKenzie and Harmanpreet Bhatti

Part A

Executive Summary

• Type of Request: This Information Collection Request is for a new overarching generic. We are requesting 3 years of approval.

• Description of Request:

The information collected under this generic clearance is intended to inform whether innovative learning methods can be utilized to evaluate interventions and services for youth transitioning out of foster care through the Chafee Strengthening Outcomes for Transition to Adulthood (Chafee SOTA) project. The Administration for Children and Families’ Office of Planning, Research, and Evaluation (OPRE) anticipates conducting multiple data collections using very similar methods. This work is rapid and iterative in nature. For these reasons, OPRE is seeking approval for an overarching generic clearance to conduct this research. Under this generic clearance, programs of interest include those that have potential to inform our understanding of what works to promote positive outcomes for youth with foster care experience. We do not intend for this information to be used as the principal basis for public policy decisions.

A1. Necessity for Collection

The Administration for Children and Families’ (ACF) Office of Planning, Research, and Evaluation (OPRE) seeks Office of Management and Budget (OMB) approval for a new 3-year overarching generic clearance to submit individual information collection (GenIC) requests for evaluations of programs serving youth transitioning out of foster care as part of the Chafee SOTA Project. Potential data collection efforts will request similar information using similar methods, which could include conducting interviews, focus groups, and surveys with program directors (e.g., from programs serving youth with foster care experience and from their partner agencies) and current, past, or potential participants in programs serving youth with foster care experience (e.g., including potential participants who are included in comparison groups), as well as extracting administrative or other program data. The purpose of these efforts is to inform ACF programming by building evidence about what works to improve outcomes for the target population, and to identify innovative learning methods that address common evaluation challenges.

Study Background

The John H. Chafee Foster Care Program for Successful Transition to Adulthood (Chafee program) funds state and tribal programs that help youth with foster care experience¹ to successfully transition to adulthood. When the Chafee program was created following the passage of the Foster Care Independence Act (FCIA) of 1999 (Public Law 106-169), the legislation required that a small percentage of funding be set aside for rigorous evaluations of independent living programs that are “innovative or of potential national significance.” In keeping with this directive, OPRE is conducting the Chafee SOTA project, which aims to utilize innovative methods for testing promising practices in programs serving youth transitioning out of foster care, and to improve the feasibility and rigor of evaluations that test the effectiveness of program services or components. The Chafee SOTA project builds on prior OPRE evaluations, which observed that for many programs, traditional, large-scale impact evaluations were not feasible due to issues such as program size, lack of appropriate comparison groups, or implementation challenges. The Chafee SOTA project will conduct evaluations of the effectiveness of program services and components in improving outcomes for youth and young adults transitioning out of foster care. To address the common evaluation challenges previously identified, these evaluations will utilize innovative methods tailored to each program, including rapid cycle learning techniques that require an iterative approach. An important aspect of research with this population and a guiding principle of the Chafee SOTA project is that evaluations need to be designed in consultation and partnership with young adults with lived experience.

The iterative and rapid nature of the methods to be utilized, as well as the necessity of ongoing, authentic engagement of experts with lived experience, poses a challenge to complying with the timeline for seeking full approval of each individual information collection activity subject to the Paperwork Reduction Act (PRA). Since OPRE knows the types of methods that will be used and types of information that will be requested but does not yet know what will be needed to tailor the information to each site, OPRE is seeking approval for an overarching generic clearance to conduct this work. For each GenIC, instruments will be tailored to the specific intervention and the specific site; once a set of instruments for a particular site is developed, and prior to use in the field, OPRE will submit a supporting statement Part A and B and the specific instruments to be used to OMB for approval.

A2. Purpose

Purpose and Use

The goal of the GenICs under this umbrella generic is to conduct mixed methods studies using innovative learning methods, which may include methods such as rapid cycle evaluation, continuous quality improvement, or core components analysis, to evaluate promising practices and improve the feasibility and rigor of evaluations that test the effectiveness of program services or components.

Intended use of the resulting data is to identify practices and program components that have the potential to improve the delivery and/or quality of services administered by human service programs and agencies in the areas of child welfare and independent living services for youth and young adults with foster care experience. ACF may use this information to advise states, territories, and tribes who administer Chafee funding on ways to be most impactful for youth in and transitioning out of foster care. This information will also provide valuable information to the field, including researchers, program evaluators, and administrators, on what program services or components show evidence of effectiveness and will highlight opportunities for future evaluation efforts.

The project is designed to identify and utilize innovative evaluation methods to test the effectiveness of program services or components in improving outcomes for youth and young adults with foster care experience as they transition to adulthood. The project seeks to address common evaluation challenges by using research methods that incorporate rapid cycle techniques and are adaptable to the specific needs of individual sites. The interventions and programs that we test in this study will address problems that have broad relevance for youth-serving child welfare programs.

These data collection activities are critical to designing evaluations that can test services or components of youth serving programs, informing ACF’s understanding of how to maximize programs’ impact. These activities will allow the team to gather structured in-depth information to understand the program process from both the administrative and participant perspectives, as well as associated outcomes for youth and young adults.

While it is our intention for the findings from each site to provide information that could be useful in the design and operation of programs that provide similar services to similar populations, the specific findings from these interventions will only be suggestive and preliminary, based on this research. The limitations of such findings will be made clear in any related communications. The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker and is not expected to meet the threshold of influential or highly influential scientific information.

Research Questions or Tests

For the purposes of evaluating program services or components, we will conduct interviews, focus groups, and surveys with administrators, staff, and participants. Programs of interest serve youth and young adults with foster care experience in areas such as housing, employment, education, and life skills coaching/mentoring. Research questions will center around how and which program services or components are being delivered to youth and young adults, and whether they are associated with improved outcomes for participants. Due to common evaluation challenges in programs serving this population (e.g. small sample sizes, lack of appropriate comparison groups), an important component of this work is identifying evaluation methods that can help overcome these challenges. Therefore, in addition to testing program effectiveness, activities submitted in the GenIC requests will also explore the utility and appropriateness of different methodologies.

Please see Instruments A-F for sample interview, focus group, and survey questions. Once sites are selected and study designs are tailored for each program, we will submit individual GenIC requests with additional detail about the site, the final tailored instruments, and the program-specific study methodology.

Exhibit 1: Sample Research Questions and Instrument Matrix

Sample Research Questions	Administrator interviews/focus groups	Staff interviews/focus groups	Youth interviews/focus groups	Youth surveys	Staff Survey	Administrative/program data
Which youth are targeted for the intervention? How are they identified and recruited?	X	X			X
What services do youth receive through the intervention?	X	X	X	X		X
To what extent is the intervention implemented with fidelity?	X	X	X	X	X
What are the patterns of participation in the intervention (e.g., dosage, attrition)?	X	X			X	X
To what extent do intervention activities influence the following types of short-term program outcomes? Education (e.g., education planning, credits earned, persistence, college applications, financial aid applications) Employment (e.g., career readiness, career exploration, career planning, job applications) Housing (e.g., rental applications) Self-sufficiency (e.g., financial literacy; established budgets and savings accounts) Mental health (e.g., psychoeducation, coping skills, self-regulation) Relational (e.g., interpersonal/parenting skills, attitudes, intentions, beliefs) Justice System (e.g., legal literacy, positive peers and role models identified, behavioral attitudes and intentions)	X	X	X	X	X	X
To what extent do intervention activities influence the following types of long-term program outcomes? Education (e.g., enrollment, graduation, GPA, degrees earned, certificates earned) Employment (e.g., jobs attained, tenure) Housing (e.g., stability) Self-sufficiency (e.g., income, self-reliance) Benefit attainment (e.g., SSI/DI, TANF, housing vouchers) Mental health (e.g., socioemotional learning, symptoms and diagnoses, suicidality, use of therapeutic services) Relational (e.g., social support, child welfare involvement, community involvement) Justice System (e.g., arrests, citations, convictions)	X	X	X	X	X	X
What contextual factors influence implementation of the intervention (e.g., challenges, barriers, organizational culture, external systems)?	X	X	X	X	X
What are the participant perspectives on their response to the intervention?			X	X
How effective are the innovative methods employed in the evaluation at addressing common evaluation challenges and adding to the evidence base for programs serving youth and young adults transitioning out of foster care?	X	X	X	X	X	X

Study Design

Under this clearance, we may use a variety of approaches and study designs. The exact data collection methods and samples for each GenIC will depend on the program-specific needs for each individual program and evaluation component. The project seeks to utilize innovative methodological techniques, which will be strengthened by the collection of data using well-established methodologies including:

• Semi-structured interviews (in-person, telephone or web-based)

• Focus groups

• Questionnaires/Surveys

• Administrative data

• Direct observation

• Document analysis

The populations to be studied include regional, state, and local child welfare, independent living, and youth-serving program directors, staff, and current, former, or potential participants.

Study designs may include: rapid cycle evaluation, theory of change refinement, core components analysis, continuous quality improvement, implementation and fidelity assessment, predictive analytics, single case design, propensity score analysis, regression discontinuity design, synthetic controls analyses, non-equivalent group designs, and interrupted time series design.

Exhibit 2 below lists some sample data collection activities for illustrative purposes.

Exhibit 2: Sample Data Collection Activities

Sample Data Collection Activity	Sample Instruments	Sample Respondent, Content, Purpose of Collection	Mode and Potential Duration
Semi-structured Interviews	Instruments A and B	Respondents: Program directors, data administrators Content: Qualitative program data Purpose: Implementation evaluation	Mode: Telephone, web-based, in-person Duration: 90 minutes
Focus groups	Instrument C	Respondents: Staff, youth Content: Qualitative program data Purpose: Implementation evaluation	Mode: web-based, in-person Duration: 90 minutes
Questionnaires/ Surveys	Instrument D	Respondents: Youth Content: Demographic and program data, youth outcomes Purpose: Outcome evaluation	Mode: Telephone, web-based, in-person Duration: 30 minutes
Administrative Data	Instrument E	Respondents: Data administrators Content: Demographic and program data, youth outcomes Purpose: Outcome evaluation	Mode: CSV file transfer Duration: 4 hours per extraction
Document Analysis	Instrument F	Respondents: Program directors Content: Program resources and activities Purpose: Implementation and outcome evaluation	Mode: existing format (e.g., PDF, PPT, Word documents) Duration: 60 minutes

GenICs submitted under this control number will consist of the following criteria:

• A full Supporting Statement A and Supporting Statement B will accompany each of the GenICs submitted under this generic clearance. These will include:

  • A discussion of the respondents. Program directors, staff, and current, former, or potential participants and administrators and staff from partner agencies are the intended subjects of our research during this IC.

• Information about the context of each specific IC. Researchers will speak with and conduct information collections with specific populations in a particular geographic location/setting/agency.

• An overview of the planned collections, methods, and program evaluations associated with each specific IC. This will include:

  • A description of the planned qualitative data collection including submission of the specific instruments for review. Anticipated instruments include focus group/interview protocols and surveys specific to each informant group (program directors, staff, and participants).

  • A description of the qualitative analyses planned. Audio recordings and notes from interviews/focus groups will be analyzed for patterns and themes.

  • A description of the quantitative analyses planned.

  • A description of the administrative data that agencies and programs are already collecting and that the project will utilize. It is important to note that collecting administrative data imposes minimal burden on respondents or record keepers, as we ask sites to provide data as it currently exists.

• Information about planned communication about the findings. We expect to communicate study outcomes to state and national stakeholders in a position to consider and implement site-specific improvements to ACF programs.

• Final proposed instruments will accompany each of the GenICs submitted under this generic clearance.

• Any supplementary materials (advance letters, emails, etc.) will accompany each of the GenICs submitted under this generic clearance, as appropriate.

Other Data Sources and Uses of Information

The project obtained prior OMB approval under ACF’s overarching generic, Formative Data Collections for ACF Research (0970–0356)² to identify and evaluate readiness of potential sites for inclusion in subsequent evaluation. Activities covered under that approval included a call for program nominations, interviews with program staff, and evaluability assessments. The project team is currently completing evaluability assessments which will guide site selection.

A3. Use of Information Technology to Reduce Burden

Planning site visits and data collection will be done collaboratively with each of the sites. We will use conference calls and emails to the extent possible to minimize burden.

Interviews or focus groups will be conducted either in-person or virtually depending on what works best for the site. To minimize the burden, we will hold semi-structured group discussions (focus groups), rather than individual conversations, whenever possible. For example, one group discussion may be held with multiple front-line workers at the same or similar levels, such as case managers, youth advocates, coaches, or education and employment specialists. A separate group discussion may be held with supervisors of front-line staff. A third discussion group may include staff at the management or administrative level, such as program directors. If there is a single staff member in a particular level, however, an individual discussion will be held. Staff at each of these levels often have different perspectives and different experiences. Group discussions will allow us to reduce the length of time spent with the site while still obtaining valuable feedback on the program implementation and outcomes from staff with a range of experiences. Surveys will be administered via the web, mobile devices, or in-person via paper and pencil.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The study team has done a careful review to determine what information is already available from existing studies and program documents to inform this work and what will likely need to be collected for the first time. Wherever possible, the team will utilize existing administrative data, however in many cases, there will not be sufficient information that would inform how program services or components are serving youth and young adults. This data collection is intended to yield new and useful information about programs serving youth and young adults with foster care experiences. Potential interviews and focus groups will support a deeper exploration of patterns seen in survey and/or administrative data or review of documents.

A5. Impact on Small Businesses

Staff at smaller sites may be part of this data collection effort if they serve youth and young adults with foster care experience. If we need to conduct interviews with individuals in small sites, we will schedule interviews at times that are convenient to minimize disruption of daily activities.

A6. Consequences of Less Frequent Collection

Rigorous evaluation of innovative initiatives is crucial to building evidence of what works and how best to allocate scarce government resources. These data collection undertakings represent an important opportunity for ACF to both learn about activities associated with child welfare and the Chafee program, and to design evaluations that increase our knowledge of how to improve service delivery and uptake. Not collecting information from the three categories of respondents (program directors, staff, and youth) would limit the government’s ability to learn about and test program services and components that lead to improved outcomes for youth and young adults transitioning from foster care.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on May 15, 2023 (88 FR 30978) and provided a sixty-day period for public comment. During the notice and comment period, we did not receive any comments.

Consultation with Experts Outside of the Study

We have consulted and may continue to consult with relevant stakeholders and experts on the study design and data collection instruments. When needed, specific consultants will be identified in each GenIC.

A9. Tokens of Appreciation

One of the guiding principles of the project is the authentic incorporation of youth voice in evaluation activities. Youth advisors on our evaluation team have strongly emphasized the importance of providing tokens of appreciation to demonstrate that the expertise and contributions of youth with lived experience are valued. To be responsive to this feedback and to ensure we are able to recruit enough youth of varied background, we propose to offer youth a token of appreciation for their participation in the data collection activities in the form of cash or gift cards ranging from $15 to $40 per round of data collection. Participation in data collection activities that require less time from youth, such as brief web-based surveys, will be acknowledged with smaller denominations whereas activities that require more time (e.g., driving to a specific location to participate in an in-person 90-minute focus group) will be acknowledged with larger denominations. These tokens of appreciation are intended to offset the incidental costs that may result from travel, additional cell-phone data or phone minutes, or childcare costs associated with participation in focus groups, interviews, and surveys. For any collection over 90 minutes or where significant travel is required (e.g., travel to central location for focus groups), participants may be offered an additional amount to account for incidental expenses.

Tokens of appreciation will not be used as a substitute for other best-practice strategies designed to increase participation, such as explanatory advance letters, endorsements by people or organizations important to the population being surveyed, and assurances of privacy.

We will include a written justification in the specific GenIC requests for any planned tokens of appreciation. We will secure Institutional Review Boards (IRB) approval for the use and monetary value of the tokens of appreciation prior to fielding any surveys, interviews, or focus groups. Additional information will be provided in each individual GenIC.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information

Data collection involves five sources of data: (1) program directors; (2) data administrators; (3) program staff; (4) youth; and (5) administrative data. Personally Identifiable Information (PII) in the form of names, organizations, and positions may be collected from program directors, data administrators, and program staff. PII collected from youth directly through surveys and through administrative data may include name, Social Security Number, date of birth, and address so that client-level data can be linked accurately across different sources and rounds of data collection. Procedures for protecting privacy of information include limiting the number of individuals who have access to identifying information, using locked files to store hardcopy forms, assigning unique IDs to each participant to ensure anonymity, and implementing guidelines pertaining to data reporting and dissemination. For all data collection activities, respondents will be informed that their participation is voluntary, that they have the right to discontinue participation at any time without impacting any services they receive, and of the risks and benefits of participation. Informed consent will be obtained from all respondents participating in interviews, focus groups, and surveys.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

Information collected will be kept private. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private. As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

At least some of the information collected under these GenICs will likely be retrieved by an individual’s personal identifier in a way that triggers the Privacy Act of 1974, as amended (5 U.S.C. 552a). The system of records notice (SORN) for this collection will be OPRE Research and Evaluation Project Records, 09-80-0361. Each individual will be provided with information that complies with 552a(e)(3) prior to requesting information that will be placed into that system of records. This means respondents will receive information about the authority, the purposes for use, the routine uses, that the request is voluntary, and any effects of not providing the requested information.

Due to the potentially sensitive nature of this research (see A.11 for more information), an individual GenIC may specify that the evaluation will obtain a Certificate of Confidentiality. If applicable, the study team will apply for this Certificate and include mention of this in the GenIC request materials. The Certificate of Confidentiality will help to assure participants that their information will be kept private to the fullest extent permitted.

Data Security and Monitoring

As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ PII. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. 

As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard.  The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with the most current National Institute of Standards and Technology (NIST) requirements and other applicable Federal and Departmental regulations. In addition, the Contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents that contain sensitive or PII that ensures secure storage and limits on access.  

A11. Sensitive Information ³

Most of the questions that will be included in these activities will not be of a sensitive nature. However, it is possible that some potentially sensitive questions may be included under this clearance. For example, some programs deliver services that address topics that are sensitive in nature, such as mental health or experiences of trauma. For proposed collections that include questions of a sensitive nature, we will provide a full explanation when submitting an individual GenIC request.

A12. Burden

Explanation of Burden Estimates

We will include up to five programs in the evaluation. The total burden hours requested across all sites under this umbrella generic information collection is 4,500 hours. The burden table below is illustrative, based on our best estimate of the types of instruments that will be used for each site. While we will not exceed the total burden cap for this generic, we may use more or less burden within each instrument type. Exhibit 4 below details the estimated burden hours.

Explanation of Cost Estimates

Estimated costs per respondent were estimated using data from the Bureau of Labor Statistics, Occupational Employment Statistics (May 2022)⁴. Exhibit 3 details median hourly wage for respondent types. Exhibit 4 details estimated total annuals costs per information collection.

Exhibit 3: Median Hourly Wages for Respondents

Respondent	Occupation	SOC Code	Median hourly wage
Administrators	Social and Community Services Manager	11–9151	$35.69
Program staff	Social and Community Services Specialist, All Others	21-1099	$22.74
Youth	Federal minimum wage	--	$7.25

Exhibit 4: Estimated Burden and Costs to Respondents

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Median Hourly Wage Rate	Total Respondent Cost
Administrator Interviews	40	4	1.00	160	$35.69	$5,692.80
Staff Discussions and Focus Groups	80	4	1.50	480	$22.74	$10,934.40
Youth Discussions and Focus Groups	160	4	1.50	960	$7.25	$6,960.00
Youth Surveys	1,800	3	0.50	2,700	$7.25	$19,575.00
Administrative Data Extraction	10	4	4.00	160	$35.69	$5,692.80
Document Delivery	10	4	1.00	40	$35.69	$1,423.20
Total	2,100			4,500		$50,278.20

A13. Costs

There are neither capital nor startup costs, nor are there any operations or maintenance costs. There are no additional total annual cost burdens to respondents or record-keepers beyond the labor cost of burden-hours described in item 12 above.

A14. Estimated Annualized Costs to the Federal Government

Exhibit 5. Estimated Cost to Federal Government

Cost Category	Estimated Costs
Field Work	$403,106.00
Analysis	$470,150.00
Publications/Dissemination	$292,773.00
Total costs over the request period	$1,166,029.00
Annual costs	$291,507.25

A15. Reasons for changes in burden

This is a request for a new umbrella generic.

A16. Timeline

We anticipate beginning data collection in early 2024 (as soon as we receive OMB clearance). Data collection activities will continue through December 2025. January through September 2026 will be dedicated to analysis, report writing, and dissemination.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Instrument A: Sample Administrator Interview

Instrument B: Sample Staff Interview

Instrument C: Sample Youth Focus Group

Instrument D: Sample Youth Survey

Instrument E: Sample Administrative Data Extraction

Instrument F: Sample Program Document Request

Appendix 1: Sample Interview Consent Form

Appendix 2: Sample Focus Group Consent Form

1 Chafee programs generally serve both youth (14 – 17 years of age) and young adults (18 – 26 years of age). For the purposes of this submission, unless otherwise stated, we will use the term “youth” to refer to the full population of individuals served under Chafee (those 14 – 26 years of age).

2 GenIC approved August 7, 2022: The John H. Chafee Foster Care Program for Successful Transition to Adulthood Strengthening Outcomes for Transition to Adulthood (Chafee SOTA) Project

3 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

4 https://www.bls.gov/oes/current/oes_stru.htm

13