SYSIL YARH-3 Supporting Statement A_Non Sub CLEAN_20220114

Alternative Supporting Statement for Information Collections Designed for

Research, Public Health Surveillance, and Program Evaluation Purposes

Supporting Youth to be Successful in Life (SYSIL) Study

OMB Information Collection Request

New Collection

Supporting Statement

Part A

January 2022

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officer: Mary Mueggenborg

Part A

Executive Summary

• Type of Request: This Information Collection Request (ICR) is to make revisions to the previously approved Youth Survey (Instrument 1) and to increase the number of program director check-ins. We are requesting three years of approval.

• Description of Request: ACF is requesting approval for the following revisions to the planned data collection:

• Add two rounds of 30-minute program director check-in calls with 4 sites, who are not participating in the full implementation study data collection. The research team will ask program directors at these sites the same questions during the check-in calls as at sites participating in the full implementation study. This will enable the research team to describe the services offered to all youth participating in the study, in particular to understand how the presence or absence of additional federal funding affected the services available during the study.

• Drop seven items from the youth survey (Instrument 1) about youth’s placement history. The research team will obtain the information from administrative data sources rather than from youth completing the survey.

• Drop a duplicate item in the approved survey.

A1. Necessity for Collection

Preventing homelessness among young people involved in the child welfare system remains an urgent issue for child welfare policymakers and practitioners. Housing stability is essential for achieving self-sufficiency and promotes health and well-being, particularly during the transition to adulthood. Unstable housing can initiate a negative cycle of poor health, limited employment, and continued housing instability (Dion et al. 2014). A combination of disadvantages places youth with a history of foster care, especially those aging out of care, at greater risk of experiencing homelessness than their peers.

The current body of evidence does not provide sufficient guidance to child welfare practitioners and other stakeholders about programs to support transitions for older youth and young adults in foster care. For example, evidence of effectiveness exists for some programs that promote positive youth development, but these programs generally have not been tested with youth who are aging out of foster care (Courtney et al. 2017). Moreover, not enough is known about the best ways to implement programs for youth transitioning out of foster care (Courtney et al. 2017).

The Supporting Youth to be Successful in Life (SYSIL) study will build evidence on how to end homelessness among youth and young adults with experience in the child welfare system by continuing work with an organization (Colorado Department of Human Services) that was part of Phases I and II of the Youth At-Risk of Homelessness (YARH) project (OMB Control Number: 0970-0445). The Administration for Children and Families (ACF) has contracted with Mathematica to conduct a summative evaluation of Colorado’s Pathways to Success comprehensive service model (Pathways). Pathways is an intensive, coach-like case management model for youth at risk of homelessness with child welfare involvement. The summative evaluation will assess the impact of Pathways on key outcomes related to housing stability and a successful transition to adulthood for youth and young adults with experience in the child welfare system. The summative evaluation includes an impact study and an implementation study.

There are no legal or administrative requirements that necessitate this collection. ACF is undertaking the collection at the discretion of the agency.

A2. Purpose

Purpose and Use

The purpose of SYSIL is to determine whether and how the Pathways program affects the targeted outcomes for youth and young adults at risk of homelessness and to provide a rich understanding of what is required of organizations to implement Pathways. The data collected through the surveys and will provide evidence on targeted outcomes while data collected through the interviews and focus groups will provide more detailed information about what is required for successful program implementation and service delivery. ACF will use the data collected through SYSIL to provide important information to the field around the effectiveness and implementation of Pathways, which could inform decisions by other states on their Chafee services.

The information collected is meant to contribute to the body of knowledge on ACF programs. It is not intended to be used as the principal basis for a decision by a federal decision-maker, and is not expected to meet the threshold of influential or highly influential scientific information.

Dissemination Efforts

Findings from the impact and implementation studies will be of interest to federal and state agencies providing child welfare and homelessness services, program providers, and youth and young adults. We anticipate traditional reports will be supplemented by other dissemination efforts – such as briefs, infographics, videos, and podcasts – to ensure the findings are shared broadly. We will work with stakeholders and experts to identify appropriate products and strategies to reach the full range of intended audiences. We hope findings will be of interest to both those who participate in the evaluation, live or work in Colorado, and more generally who work with youth and young adults.

Research Questions

Impact Study

The impact study is designed to answer two types of research questions: (1) impact research questions about the magnitude of the effect that Pathways has on participant outcomes and (2) exploratory research questions that link features of implementation to participant outcomes.

Answers to two impact research questions will provide evidence of the effect of Pathways for the full study population and for key subgroups:

1. What is the impact of Pathways on key outcomes, including but not limited to housing, educational attainment, employment, permanency, and well-being?

1. What are the impacts after the first six months of Pathways (about halfway into the average length of participation in Pathways)?

2. What are the impacts immediately following participation in Pathways (12 months after entry)?

3. What are the impacts 12 months after the end of participation in Pathways (24 months after entry)?

1. Is Pathways particularly effective for key subgroups of the target population? Specifically, how do findings differ for the following:

1. Youth approaching age 17.5 who are able to decide to remain in foster care or leave foster care

2. Youth with varying foster care backgrounds (for example, age at entry, time in care, second-generation child welfare status, permanency status)

3. Youth by gender identity

4. Youth who have mental health or substance abuse challenges (potentially stemming from trauma)

5. Youth by race and ethnicity

6. Youth by sexual orientation

7. Youth by level of connectedness at program entry

8. Youth by the experience of the site implementing Pathways (for example, new implementers or seasoned implementers)

The answer to one exploratory research question will provide information on the relationship between implementation and outcomes:

3. Do features of the Pathways implementation influence youth outcomes?

1. Does level of adherence to the intended model (for example, dosage and duration of services) have a strong relationship with youth outcomes?

2. Do particular components of the model (for example, receiving flexible funds, the frequency or duration of contact with a Pathways Navigator) have a strong relationship with youth outcomes?

Implementation Study

The implementation study will explore research questions (RQs) related to five topics that will provide information necessary for interpreting the impacts of Pathways and understanding its implementation in different settings:

1. Topic: Differences between the Pathways service model and comparison services. The question on this topic addresses the ways in which the Pathways service model differs from services offered to youth and young adults in the comparison condition. The research question is:

   1. How are services under the Pathways service model distinct from services available in the comparison condition?

2. Topic: Barriers to and facilitators of Pathways implementation. Questions on this topic address how Pathways leadership and staff put components of the Pathways service model into operation and the factors that contributed to or inhibited implementation. Research questions are:

1. What did the site do to support initial service delivery (that is, start-up activities) in Pathways sites?

2. To what extent did Pathways sites use continuous quality improvement (CQI) to support implementation? How did CQI support implementation?

3. What factors (facilitators and barriers) contributed to or hindered initial and ongoing service delivery in Pathways sites?

3. Topic: Fidelity to the Pathways service model. Questions on this topic address the extent to which sites delivered core services in the comprehensive service model as intended and factors that might have contributed to or hindered fidelity. Research questions are:

1. To what extent did the Pathways sites deliver Pathways services with fidelity?

2. To what extent did levels of fidelity vary across Pathways sites?

3. What factors (facilitators and barriers) contributed to or hindered achieving and sustaining fidelity?

4. Topic: Service, resource, and policy environment surrounding Pathways implementation. Questions on this topic address the services generally available to youth and young adults in the evaluation sample (both treatment and comparison groups) and the child welfare policies that may affect youth and young adults in the evaluation sample. Research questions are:

1. Which services and resources are available to youth and young adults in the Pathways and comparison groups in the sites where this summative evaluation occurs?

2. What does site leadership perceive to be key child welfare policies and regulations that affect youth and young adults in the Pathways and comparison groups in the locations where this summative evaluation occurs?

5. Topic: Youth and young adults’ responsiveness to the Pathways service model. Questions on this topic address youth and young adults’ acceptance of Pathways services and their perceptions of the services. Research questions are:

1. What were the characteristics of the population served by Pathways?

2. What services were delivered to youth and young adults in Pathways?

3. What strategies did Navigators use to promote and maintain youth and young adult engagement?

4. How did engagement vary among youth and young adults participating in the Pathways service model?

5. What were youth and young adults’ perceptions of Pathways services? How did they describe their experience in Pathways?

Study Design

Impact Study

The impact study approach consists of a cluster QED that will use survey data as the primary data source for key outcomes of interest and be supplemented by administrative data elements.¹ Thirty-seven counties within Colorado will participate in the impact study.² Data will be collected from youth in participating counties at baseline, and at 6-, 12- and 24-months after study enrollment. By comparing youth in treatment sites to youth in comparison sites, the study will be able to show the effectiveness of Pathways on a large number of outcomes collected in the survey and available in the administrative data. 

Given the available data from the baseline survey and from the administrative data systems, we are confident that we can demonstrate the equivalence of the treatment and comparison groups on a large number of potential confounders. Demonstrating equivalence at baseline helps the study to produce strong, internally valid impact estimates. The demonstration of baseline equivalence helps address the limitation of all QEDs that it is impossible to know whether the youth are well-matched on unmeasured characteristics.  A second potential limitation is that the study may not be well powered to detect program impacts on several outcomes of interest, unless the observed impacts are quite large. We will therefore conduct an additional impact study design that uses the administrative data sources to more fully supplement the main impact study. Specifically, we will use a larger pool of potential sample members to expand the comparison group and conduct an analysis that obviates the need to do a clustering correction. This change will address the chief limitation of the main study design: statistical power. We will acknowledge these limitations in all publications that describe the findings of this study.

Implementation Study

The implementation study will adapt the Consolidated Framework of Implementation Research (CFIR) to guide clear conceptualization and systematic assessment of the range of contextual factors that facilitate or hinder implementation of the Pathways service model.

We will conduct two rounds of site visits to 12 sites (6 sites delivering Pathways and 6 sites delivering comparison services) for the implementation study. During these site visits, we will speak with a range of stakeholders involved in Pathways services and comparison services. We will also conduct focus groups with youth and young adults who receive services in both the Pathways and comparison sites. The site visits will occur at about 4 to 6 months after enrollment begins and at 18 to 20 months after enrollment begins. We will collect and analyze program administrative data to assess patterns of service delivery and describe the extent to which Pathways services are delivered with fidelity.

We will also conduct two additional “check-ins” by telephone with program directors from all 16 sites to ask about current service delivery. We anticipate the first check-in will occur between the first and second site visit. The second check-in will occur after the second site visit.

Table A.2 provides an overview of the data collection instruments.

Table A.2. Information Collections

Data Collection Activity	Instrument(s)	Respondent, Content, Purpose of Collection	Mode and Duration
Baseline Data Collection	Youth Survey*	Respondents: Youth with consent to participate Content: Demographics, attitudes about and outlook toward the future, experiences with the child welfare system, education and employment history and goals, involvement with the criminal or juvenile justice system, access to available system resources, connections with adults and peers, and parenting Purpose: To describe the target population and allow for a demonstration of baseline equivalence	Mode: Web and phone Duration: 30 minutes
Follow-up data collection (6-months, 12-months, and 24-months)	Youth Survey	Respondents: Youth with consent to participate Content: Demographics, attitudes about and outlook toward the future, experiences with the child welfare system, education and employment history and goals, involvement with the criminal or juvenile justice system, access to available system resources, connections with adults and peers, and parenting Purpose (6-months): To examine short-term impacts across outcome domains of interest Purpose (12-months): To examine interim impacts across outcome domains of interest Purpose (24-months): To examine long-term impacts across outcome domains of interest	Mode (6-months): Web and phone Mode (12- and 24-months): Web, with phone and field non-response follow-up Duration: 30 minutes
Interviews	Interview Guide for Pathways Sites (Treatment Sites)	Respondents: Key stakeholders and program staff in treatment sites Content: Youth recruitment and enrollment; program service components; partnerships that support the program; service, resource, and policy context; implementation experience; program resources; and continuous quality improvement Purpose: To understand the contextual information that may influence implementation and fidelity and provide context for findings from the impact study	Mode: In-person or virtual+ Duration: 1.5 hours
Program Director Check-ins	Subset of questions from the interview guides for Pathways (Treatment) Sites**	Respondents: Program Directors from all Pathways sites (treatment sites) Content: Details around the services provided to youth, including any updates or changes since the site visit Purpose: To understand the breadth of services provided to youth in Pathways sites	Mode: Phone Duration: 30 minutes
Interviews	Interview Guide for Comparison Sites	Respondents: Key stakeholders and program staff in comparison sites Content: Youth recruitment and enrollment; program service components; partnerships that support the program; service, resource, and policy context; implementation experience; program resources; and continuous quality improvement Purpose: To understand the contextual information that may influence implementation and fidelity and provide context for findings from the impact study	Mode: In-person or virtual+ Duration: 1.5 hours
Program Director Check-ins	Subset of questions from the interview guides for Comparison Sites**	Respondents: Program Directors from all comparison sites Content: Details around the services provided to youth, including any updates or changes since the site visit Purpose: To understand the breadth of services provided to youth in comparison sites	Mode: Phone Duration: 30 minutes
Focus groups	Focus Group Discussion guide for Pathways Youth (Treatment Youth)	Respondents: Youth in treatment sites with consent to participate in the focus group Content: Experiences with program services and staff, including services the youth received, how they accessed the services, if they found the services helpful and why or why not, and suggestions on how to improve them Purpose: To understand youth’s experiences with services	Mode: In-person or virtual++ Duration: 1.5 hours
Focus groups	Focus Group Discussion Guide for Comparison Youth	Respondents: Youth in comparison sites with consent to participate in the focus group Content: Experiences with program services and staff, including services the youth received, how they accessed the services, if they found the services helpful and why or why not, and suggestions on how to improve them Purpose: To understand youth’s experiences with services	Mode: In-person or virtual++ Duration: 1.5 hours

^*The youth survey is the same instrument administered at four different points in time: (1) baseline, (2) 6-month follow-up, (3) 12-month follow-up, and (4) 24-month follow-up.

** The program director check-ins do not require a unique instrument; they will be conducted using a subset of items from the interview protocol.

⁺ Mode of administration for the interviews and focus groups will be determined based on existing travel restrictions and public health guidelines concerning COVID-19 at the time of data collection.

⁺⁺ During a webinar for YARH-3 stakeholders, we received feedback on youth recruitment and engagement activities for evaluation activities. Stakeholders suggested that virtual focus groups are more effective than in-person focus groups for engaging youth participants. We will have the flexibility to conduct in-person focus groups, if necessary.

Other Data Sources and Uses of Information

The study will also use administrative data from the child welfare system, homelessness management information system, and program providers. Administrative data from the Linked Information Network of Colorado (LINC) will provide additional outcomes on child welfare, public assistance, and employment, among others. We will use the administrative data from LINC as an alternative outcome data source for estimating impacts on the outcomes. We expect that some outcomes measured in the administrative data will not be captured in the survey data (in particular, long-term outcomes). LINC data will also be used to validate the subset of constructs measured in both the survey and administrative data (for example, by comparing youth self-reports on recent child welfare status with administrative data on recent status). We will also collect and analyze program administrative data from the Pathways Management Information System (PMIS)³ to assess patterns of service delivery and describe the extent to which Pathways services are delivered with fidelity. Administrative data from the LINC and PMIS will be used in their existing format. Use of these data will not impose any new information collection or record-keeping requirements on respondents.⁴

A3. Use of Information Technology to Reduce Burden

The SYSIL data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden. The youth surveys will be web-based surveys. Trained staff will provide participants with smartphones or tablet computers, along with a unique URL to access the survey from the device. Youth can also complete the survey on their own device.

Web-based surveys are an attractive option for surveys for adolescents and young adults, and in particular for surveys that ask sensitive questions and present various pathways based on responses to those questions. Web-based surveys can decrease respondent burden and improve data quality. Unlike paper instruments in which respondents must determine the question routes themselves, the web-based application will include built-in skips and route respondents to the next appropriate question based on their answers. The web-based program automatically skips them out of any questions that are not relevant to them, thus reducing the burden on respondents having to navigate through various paths. In addition, data checks can be programmed into the survey to eliminate responses that are out of range or that conflict.

Mathematica will conduct semi-structured interviews with program administrators, supervisors, and direct service staff and focus groups with participating youth and young adults. Mathematica will also conduct brief check-ins with program directors. The information to be collected during interviews, focus groups, and check-ins is not conducive to the use of information technology, such as computerized interviewing. In-person administration, or virtual administration via a secure web platform such as WebEx offers the best opportunity to tailor the data collection to staff and youth participants with minimal burden on respondents. These recordings, with participant approval, can assist in minimizing burden as verbatim transcripts will be made, decreasing the need for the interviewer to ask the respondent to repeat themselves to ensure the notes are accurate.

A4. Use of Existing Data: Efforts to reduce duplication, minimize burden, and increase utility and government efficiency

The information collection requirements for the SYSIL study have been carefully reviewed to avoid duplication and to maximize opportunities to use existing data, including administrative data. By using the administrative data from the PMIS, we will avoid requesting this information through a separate data collection instrument specifically for the SYSIL study, thus reducing the potential burden on case managers. Additionally, as noted earlier, survey questions that request information that can be answered by data items available through administrative data sources have been removed from the survey, thus reducing burden on youth and minimizing duplication of information.

A5. Impact on Small Businesses

No small businesses will be involved with this information collection.

A6. Consequences of Less Frequent Collection

Data collected as part of this study are essential to conducting a rigorous evaluation of the Pathways program. Without outcome data collected through the impact study, we could not estimate the short- term effects of the program (using the 6-month follow-up survey), or determine whether those effects are sustained in the long-term or translate to the expected outcomes (using the 12- and 24-month follow-up surveys). Data collected through the implementation study will be essential for understanding the results of the impact study and assessing the implementation of Pathways.

A7. Now subsumed under 2(b) above and 10 (below)

A8. Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on December 30, 2020, Volume # 85, Number # 250, page # 86589, and provided a sixty-day period for public comment. During the notice and comment period, no substantive comments were received.

Consultation with Experts Outside of the Study

The following experts in their respective fields were consulted on the data collection instruments for which clearance is requested:

• Tym Belseth, Research Coordinator with Texas Institute for Child and Family Wellbeing, University of Texas at Austin

• Mark Courtney, Samuel Deutsch Professor in the School of Social Service Administration, University of Chicago

• Lanae Davis, Senior Research Associate, Center for Policy Research

• Jan DeCoursey, Senior Research Scientist and Program Area Co-Director, Child Welfare, Child Trends

• Amy Dworsky, Research Fellow, Chapin Hall, University of Chicago

• Jennifer Haight, Supervisory Children and Family Program Specialist, Children’s Bureau

• Chereese Philips, Child Welfare Program Specialist, Children’s Bureau

• Kaylene Quinones, LMSW, BraveLife Coordinator

• Cassandra Simmel, Associate Professor and Director, MSW Certificate in Promoting Child and Adolescent Well-Being, Rutgers University

• Nancy Thoennes, Associate Director, Center for Policy Research

A9. Tokens of Appreciation

We propose offering tokens of appreciation to youth for (1) completion of each 30-minute survey, and (2) participation in 1.5-hour focus groups. Additionally, youth will receive an item such as a dry bag, water bottle, document portfolio, or other item of similar value when they enroll in the study. ⁵. In our discussions with experts in the field, program staff, and stakeholders, several stressed the importance of providing tokens of appreciation to youth that reflect the value of their input, as no one can replace them for this data collection. The data collection requires input from the small, specialized population of youth and young adults with experience in the child welfare system. The surveys contain introspective and potentially sensitive questions that only youth with their experiences can speak to. The tokens of appreciation reflect the value of the specific experiences these youth provide in responding to the surveys. The following proposed amounts for the surveys were determined based on consultation with experts in the field, stakeholders, and program staff:

• $40 gift card for baseline survey

• $45 gift card for the 6-month follow-up survey

• $50 gift card for the 12-month follow-up survey

• $65 gift card for the 24-month follow-up survey.

Participating youth will be asked to participate in multiple waves of data collection. The proposed amounts for the tokens of appreciation serve to develop an on-going relationship with youth to encourage their continued participation for each survey, which is essential for assessing program impacts on long-term outcomes. We value youth’s participation in the data collection and the proposed tokens of appreciation reflect the value of their participation in data collection activities and contributions to the data collection that only they can provide. Estimates of program impacts may be biased if respondents differ from non-respondents and those differences are correlated with group assignment. The risk of biased impact estimates increases with lower overall responses or larger differences in survey response rates between research groups (What Works Clearinghouse 2013). Concern about the potential for low overall response rates are particularly relevant to this study because the target population is youth and young adults with experience with the child welfare system who are at risk of homelessness. A number of factors could complicate tracking participants over time including:

• System fatigue

• Unstable housing

• Fewer permanent connections with others

• Less use of leases, public utility accounts, cell phone contracts, credit reports, memberships in professional associations, licenses for specialized jobs, and appearance in publications such as newspapers and blogs

• Use of an alias to get utility accounts because of poor credit and prior payment issues

• Use of pay-as-you-go phones

Youth and young adults may be in the custody of the state (foster care) at time of early data collection efforts. If they are not in the custody of the state, we anticipate they will be low-income given their age, likely education level, and employment opportunities. We propose increasing the value of the token of appreciation as time passes to account for the decreased contact between the study and youth once they are no longer receiving services and the increased value of money to help with bills once living on their own. For surveys that are completed in-person, data collectors will distribute gift cards to youth immediately after completing the survey. For surveys that are not completed in-person, gift cards will be distributed to youth as quickly as possible (either via mail or electronically), using youth’s available contact information.

Additionally, we expect that as participating youth get older and age out of the foster care system, they will be more transient and harder to locate for data collection. We have budgeted for field staff to locate and complete surveys with these hard to reach youth, however we anticipate that offering proposed tokens of appreciation will create cost savings for the field tracking efforts. We expect that the proposed tokens will encourage youth to complete the surveys, resulting in a smaller proportion of youth who will need to be located by field staff.

The proposed tokens of appreciation are designed to boost overall response rates for this low-income, hard-to-reach population as well as to minimize differential response rates between the treatment and comparison youth. Participants assigned to the comparison group may be less motivated to participate than those assigned to the treatment group because they are not receiving the intervention and may not feel that the surveys are relevant to them. Additionally, youth in the comparison group may not have developed the strong connections with their Chafee workers that treatment youth may have developed with their Navigators and therefore may be less likely to feel connected to the study and participate in the data collection. Pejtersen (2020) showed incentives can increase the response rates in surveys of youth, and higher incentive amounts can have some beneficial impacts on response rates (Brown and Calderwood, 2014). Oh et al (2021) found larger incentives may encourage youth to complete study activities.

We also propose offering an item such as a mug, tote bag, or other item of similar value to the Chafee workers as a way to engage them and obtain buy-in to the study and facilitate data collection. The proposed token shows our appreciation for the additional work related to the study, mainly screening and obtaining consent/assent for participation. Additionally, the token will help a feeling of being part of a team.

We also propose offering youth who participate in the 1.5-hour focus groups a $40 gift card. Focus group data are not intended to be representative in a statistical sense, in that they will not be used to make statements about the prevalence of experiences youth and young adults with experience in the child welfare system. However, it is important to secure participants with a range of background characteristics to capture a variety of possible experiences. All participants are expected to be low income youth and young adults. Without offsetting the direct costs incurred by respondents for attending the focus groups, such as child care, additional use of data plans or minutes or phones, or transportation if focus groups are able to be held in-person safely, the research team increases the risk that only individuals able to overcome financial barriers to attend will participate in the study. The proposed $40 gift card will help offset these incidental expenses that may otherwise prevent their participation.

A10. Privacy: Procedures to protect privacy of information, while maximizing data sharing

Personally Identifiable Information (PII)PII will be collected on consent forms and through the youth surveys. Each youth will be assigned a unique study ID for the duration of the study. This ID will be linked to the user-specific URLs that youth will use to access the web-based surveys. The unique ID will be used to link survey responses by a single respondent across surveys. PII will be stored in secure files, separate from survey and other individual-level data.

Table A.3 below lists the PII that will be collected and its use.

Table A.3. Personally Identifiable Information (PII) To Be Collected and Its Use

PII	Intended Use
First and last name	Contact information collected through the consent forms and surveys will be used to contact respondents about completing the follow-up surveys.
Address (street, city, state, and zip code)
Phone numbers (cell, home)
Email address
First and last name for alternate contacts	Youth participants will be asked to provide contact information for up to three people who will know their location should they move during the study. This information will be used only to help locate respondents for follow-up data collection. We will not reveal anything about the respondents or the focus of the study when reaching out to alternate contacts.
Address (street, city, state, and zip code) for alternate contacts
Phone numbers (cell, home) for alternate contacts
Email address for alternate contacts
Date of birth	Date of birth will be used to verify respondent at follow-up rounds of data collection.
Race	These variables will be used to assess baseline equivalence and examine whether Pathways is particularly effective for key subgroups of the target population. They will include at a minimum a baseline measure of the outcome and demographic characteristics, because these variables are likely to be strongly predictive of the outcomes of interest.
Ethnicity
Employment status
Estimated monthly income
History of arrests/convictions
State child welfare identification (ID) number	We will request the youth’s state child welfare ID number from the Chafee workers when they enroll youth. We will use this ID to help communicate with Chafee workers about youth throughout the study.

PII will not be kept in the same location as any data collected. Access to respondents’ contact information is restricted to those working on the SYSIL evaluation. Any files containing PII are stored on Mathematica’s network in a secure project folder whose access is limited to select project team members. Only the principal investigator, project director, and key study staff have access to this folder. Furthermore, approved study team members can only access this folder after going through multiple layers of security. A secure FTP site (Box.com) will be used to transfer administrative data, which will contain as limited PII as possible.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

Assurances of Privacy

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law (see Appendix A). As specified in the contract, the Contractor will comply with all Federal and Departmental regulations for private information.

Due to the sensitive nature of this research (see A.11 for more information), the evaluation will obtain a Certificate of Confidentiality. The study team will apply for this Certificate and will provide it to OMB once it is received. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.

The project team will seek Institutional Review Board (IRB) approval from the Health Media Lab IRB.

Data Security and Monitoring

As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Safety and Monitoring Plan that assesses all protections of respondents’ PII. The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements. 

As specified in the evaluator’s contract, the contractor shall use Federal Information Processing Standard-compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Information Processing Standard.  The contractor shall ensure that this standard is incorporated into the contractor’s property management or control system and establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information. Any data stored electronically will be secured in accordance with National Institute of Standards and Technology (NIST) requirements and other applicable federal and departmental regulations. In addition, the contractor must submit a plan for minimizing to the extent possible the inclusion of sensitive information on paper records and for the protection of any paper records, field notes, or other documents with sensitive information or PII that ensures secure storage and limits on access.  

Interviews and focus groups will be recorded only with permission from participants. Before the discussions begin, participants will be informed that we would like to record the discussion and will be asked to give their permission. Discussions will not be recorded if the participants do not give their permission. Participants will be asked to keep each other’s information private (see Appendix A).

Recordings from the interviews and focus groups will be deleted as soon as they have been transcribed. We will wait to begin recording the discussion until after everyone has introduced themselves. The transcribed notes will not include any names. All recording will be stored on Mathematica’s secure network and destroyed per the contract requirements.

The survey data will be archived, likely with the National Data Archive on Child Abuse and Neglect (NDACAN) (hhs.gov).

A11. Sensitive Information ⁶

This study includes questions on sensitive topics. It is necessary to include these questions in order to describe the population and determine baseline equivalence between the treatment and comparison groups, to measure the impacts of Pathways on the targeted outcomes (Research Question 1), and to determine whether Pathways is particularly effective for key subgroups of the population (Research Question 2).

Table A.4 includes the sensitive topics found in the youth survey, along with the justification for their inclusion.

The items have been carefully selected based on experience and consultation with outside experts. We have considered whether the benefits and utility of the measures outweigh concerns about the heightened sensitivity among sample members and program staff to specific issues. We have obtained feedback from youth during the pre-test on the proposed measures and asked specifically about questions that they may consider to be too sensitive. Items that pre-test respondents raised as being too sensitive were removed from the current version of the survey.

Table A.4. Summary of Sensitive Questions To Be Included in the Youth Survey and Their Justification

Topic	Justification
Race	Will use to assess baseline equivalence between groups and identify subgroups for subgroup analyses.
Ethnicity	Will use to assess baseline equivalence between groups and identify subgroups for subgroup analyses.
Gender identity	Will use to assess baseline equivalence between groups and identify subgroups for subgroup analyses.
Sexual orientation	Will use to assess baseline equivalence between groups and identify subgroups for subgroup analyses.
Ever experienced homelessness	Will use for baseline equivalence.
Risk behaviors (including unprotected sex, drug and alcohol use, domestic violence)	Will use to assess baseline equivalence between groups; identify subgroups for subgroup analyses and assess social-emotional well-being, a key outcome domain.
Income	Related to employment and self-sufficiency, which may affect housing stability, both key outcome domains for the study.
Criminal justice history	This is a key outcome domain for the study. Involvement in the criminal justice system makes it harder to find employment and to secure stable housing, both of which are key outcomes of the study.
Child welfare placement history	Will use for baseline equivalence and to assess impacts for key subgroups (for example, looking at effects based on age at entry, time in care, and so on). This is also a key outcome domain for the study.
Economic hardship	Includes items such as missing meals or needing to borrow money from friends. These outcomes reflect a lack of self-sufficiency and may affect housing stability, a key outcome of the study.
Government services received	Will use to assess ability to access system resources, which is a key outcome domain for the study.

Respondents may feel uncomfortable answering some of the questions on the surveys that address sensitive topics. To minimize this risk, youth will be told they can skip any questions they do not want to answer. We will train all staff who administer the surveys how to follow the protocols on dealing with sensitive questions and situations. Youth are able to end their participation in the study at any time without affecting the services they receive.

Prior to collecting baseline data, Mathematica will seek active consent from a parent, legal guardian, or legally authorized representative for those participants younger than age 18. Youth age 18 or older will be asked to consent for themselves. The consent form will state that answers will be kept private to the extent permitted by law and not seen by anyone outside of the study team, that participation is voluntary, and that they may refuse to participate at any time without penalty (Appendix A). They will be informed that, to the extent permitted by law, individual identifying information will not be released or published; rather, data collection will be published only in summary form with no identifying information at the individual level. In addition, our protocol during the self-administration of the web instrument will provide reassurance that we take the issue of privacy seriously. It will be made clear to respondents that identifying information will be kept separate from questionnaires.

The project team will seek IRB approval from the Health Media Lab IRB.

A12. Burden

Explanation of Burden and Cost Estimates

Table A.5 provides the estimated annual burden and cost calculations for the data collection instruments included in this ICR. The total annual estimated burden is 467 hours. The total annual cost to all respondents is $4,593.24.

Youth Survey: Based on previous experience with similar instruments and the pre-test, the youth survey is estimated to take 30 minutes (0.5 hours) to complete. The cost to respondents to the youth survey is estimated by assuming that 50 percent of youth will be age 18 or older at baseline,⁷ and then assigning a value to their time of $12.32 per hour, the 2021 Colorado minimum wage. The estimate of the proportion of youth who will be age 18 or older is based on the average age at intake for youth in Chafee services in comparison sites (18.6) and treatment sites (18.8).

• Baseline. We expect to survey about 700 youth at baseline. The total burden over three years is estimated to be 350 hours (700 * .5). The annual burden for this data collection is estimated to be 117 hours (350/3). The annual cost to respondents for youth age 18 and older is estimated to be $720.72 (117*0.5*$12.32).

• Follow-Up 1 (6 Months). We expect to survey approximately 630 youth at the 6-month follow-up. The total burden over three years is estimated to be 315 hours (630 * 0.5) and the annual burden for this data collection is estimated to be 105 hours (315/3). The annual cost to respondents for youth 18 and older is estimated to be $711.48 (105*0.55*$12.32).

• Follow-Up 2 (12 Months). We expect to survey approximately 595 youth at the 12-month follow-up. The total burden over three years is estimated to be 298 hours (595 * 0.5). The annual burden for this data collection is estimated to be 99 hours (298/3). The annual cost to respondents for youth age 18 and older is estimated to be $731.81 (99*0.6*$12.32).

• Follow-Up 3 (24 Months). We expect to survey approximately 490 youth at the 24-month follow-up⁸. The total burden over three years is estimated to be 245 hours (490* 0.5). The annual burden for this data collection is estimated to be 82 hours (245/3). The annual cost to respondents for youth age 18 and older is estimated to be $707.17 (82*0.7*$12.32).

Interview Guide for Pathways Sites (Treatment Sites) and Comparison Sites. We expect to interview approximately 30 program leaders and staff who deliver Pathways services and approximately 30 program leaders and staff who deliver comparison site services. The interview guide is estimated to take one and a half hours to complete. The total burden over three years is estimated to be 90 hours (60 x 1.5) for treatment and comparison sites. The annual burden for this data collection is estimated to be 30 hours (90/3) for treatment and comparison sites. The annual cost to respondents is estimated to be $1,247.70 (30*$41.59) for treatment and comparison sites.

The hourly wage estimate program leaders and staff is based on the mean hourly wage rate ($41.59) for social and community service managers (State Occupational Employment and Wage Estimates for Colorado, Bureau of Labor Statistics, Department of Labor, May 2019).

Program Director Check-ins for Pathways sites (Treatment Sites)⁹ and Comparison Sites. Mathematica will conduct two “check-ins” over the phone with program directors to ask about current service delivery in all Pathways sites and all comparison sites. A check-in will use five questions from the interview protocol related to the services being provided. Each check-in is estimated to be 30 minutes. We anticipate the first check-in will occur between the first and second site visit; the second will occur after the second site visit. The total burden over 3 years is estimated to be 16 hours ((16 x .5) *2) for treatment sites and for comparison sites. The annual burden for this data collection is estimated to be 5.34 hours (16/3) for treatment sites and for comparison sites. The annual cost to respondents is estimated to be $249.54 (6*$41.59) for treatment sites and for comparison sites.

Focus Group Discussion Guide for Pathways Youth (Treatment Youth) and Comparison Youth. We expect to interview about 50 youth who receive services at Pathways sites and about 50 youth who receive services at comparison sites. Based on previous experience with similar interviews, the focus group is estimated to take one and a half hours to complete. The total burden over three years is estimated to be 150 hours (100 x 1.5) for treatment and comparison youth. The annual burden for this data collection is estimated to be 50 hours (150/3) for treatment and comparison youth. The annual cost to respondents for youth age 18 and older is estimated to be $308 (50*0.5*$12.32) for treatment and comparison youth.

Table A.5 Estimated Annualized Cost to Respondents

Instrument	No. of Respondents (total over request period)	No. of Responses per Respondent (total over request period)	Avg. Burden per Response (in hours)	Total Burden (in hours)	Annual Burden (in hours)	Average Hourly Wage Rate	Total Annual Respondent Cost
SYSIL Youth Survey – Baseline	700	1	0.5	350	117	$12.32	$720.72
SYSIL Youth Survey – Follow-Up Survey 1 (6 months)	630	1	0.5	315	105	$12.32	$711.48
SYSIL Youth Survey – Follow-Up Survey 2 (12 months)	595	1	0.5	298	99	$12.32	$731.81
SYSIL Youth Survey – Follow-Up Survey 3 (24 months)10	490	1	0.5	245	82	$12.32	$707.17
Interview Guide for Pathways Sites (Treatment Sites)	30	1	1.5	45	15	$41.59	$623.85
Program Director Check-ins for Pathways Sites (Treatment Sites)*	10	2	.5	10	4	$41.59	$166.36
Interview Guide for Comparison Sites	30	1	1.5	45	15	$41.59	$623.85
Program Director Check-ins for Comparison Sites*	6	2	.5	6	2	$41.59	$83.18
Focus Group Discussion Guide for Pathways Youth (Treatment Youth)	50	1	1.5	75	25	$12.32	$154.00
Focus Group Discussion Guide for Comparison Youth	50	1	1.5	75	25	$12.32	$154.00
Total				1,464	489		$4,676.42

* The program director check-ins do not require a unique instrument; they will be conducted using a subset of items from the interview protocol.

A13. Costs

There are no additional costs than what is outlined in A12 and A14.

A14. Estimated Annualized Costs to the Federal Government

The total cost to the federal government for the data collection activities under this ICR is estimated at $7,373,853. Annualized costs to the federal government are estimated at $2,457,951 for the proposed data collection. These estimates of costs are derived from Mathematica’s budgeted estimates and include labor rates and, direct and indirect costs and are displayed below in Table A.6.

Table A.6. Estimated Annualized Costs

Cost Category	Estimated Costs
Instrument Development and OMB Clearance	$802,283
Field Work	$5,913,439
Analysis	$605,647
Publications/Dissemination	$52,484
Total costs over the request period	$7,373,853
Annual costs	$2,457,951

A15. Reasons for changes in burden

The number of respondents for the 30-minute check-in calls increases from 12 to 16, reflecting the inclusion of the 4 Pathways (Treatment) hubs that will not participate in the full implementation study data collection activities (that is no visits).

A16. Timeline

Table A.7 presents the estimated timeline for data collection, analysis, and reporting for the impact and implementation study. The survey data will be archived, likely with the NDACAN (hhs.gov).

Table A.7. Plan and Time Schedule for Information Collection, Tabulation, and Publication

Project Activity	Time Period
Impact Study
Data collection (baseline youth surveys)	30 months, following OMB approval
Data collection (follow-up youth surveys)	48 months, beginning 6 months after baseline+
Data analysis	36 months, 3–6 months after data collection begins
Draft final report	About 6 months after completion of data analysis
Revised final report	About 9 months after completion of data analysis
Implementation Study
Data collection (first round of focus groups and interviews)	2–3 months, 4–6 months after enrollment begins
Data collection (first round of Program Director check-ins)	1 month, about 12 months after enrollment begins
Data collection (second round of focus groups and interviews)	2–3 months, 18–20 months after enrollment begins
Data collection (second round of Program Director check-ins)	1 month, about 24 months after enrollment begins
Data analysis	6–12 months after data collection begins
Draft final report	About 6 months after completion of data analysis
Revised final report	About 9 months after completion of data analysis

⁺ This time period represents the data collection period for all rounds of follow-up data collection: 6 month follow-up survey (to begin 6 months after baseline and continue for 24 months); 12-month follow-up survey (to begin 12 months after baseline and continue for 24 months); and the 24 month follow-up survey (to begin 24 months after baseline and continue for 24 months). Data collection for the 24-month follow-up is likely to extend beyond 3 years. A request to continue data collection will be submitted prior to the expiration date of OMB approval.

A17. Exceptions

No exceptions are necessary for this information collection.

Attachments

Instrument 1: SYSIL Youth Survey (Baseline and Follow-Ups 1-3)

Instrument 2: Interview Guide

2a: Interview Guide (Treatment Sites)

2b: Interview Guide (Comparison Sites)

Instrument 3: Program Director Check-ins

3a: Program Director Check-ins (Treatment Sites)

3b: Program Director Check-ins (Comparison Sites)

Instrument 4: Focus Group Guide

4a: Focus Group Guide (Treatment Youth)

4b: Focus Group Guide (Comparison Youth)

Appendix A: Consent and Assent Forms

Appendix B: List of Surveys Referenced

Appendix C: Emails and Text for Outreach to Youth

References

Brown, M., L. Calderwood. “Can Encouraging Respondents to Contact Interviewers to Make Appointments Reduce Fieldwork Effort? Evidence from a Randomized Experiment in the UK.” Journal of Survey Statistics and Methodology 2, 484-497 (2014).

Dion, R., A. Dworsky, J. Kauff, and R. Kleinman. “Housing for Youth Aging Out of Foster Care.” Washington, DC: Mathematica Policy Research, 2014.

Oh, A.Y., A. Caporaso, T. Davis, L. A. Dwyer, L. C. Nebling, B. Liu, and E. Hennessy. “Adolescents’ Participation in an Accelerometer Data Collection Component of a National Survey”. Field Methods (2021).

Pejtersen, J.H. “The effect of monetary incentive on survey response for vulnerable children and youths: A randomized controlled trial.” PLoS ONE 15(5), 2020.

U.S. Interagency Council on Homelessness (USICH). "Framework for Ending Youth Homelessness." Washington, DC: USICH, February 2013.

What Works Clearinghouse. “Assessing Attrition Bias.” Available at: https://ies.ed.gov/ncee/wwc/Docs/ReferenceResources/wwc_attrition_v2.1.pdf. 2013.

1 The youth surveys include all possible questions. The study team will remove any questions that can be addressed through the use of administrative data

2 A “site” is a service unit that may include one or more counties. The 37 counties form 16 sites (10 intervention and 6 comparison) due to coordination of services across small adjacent counties.

3 The Performance Management Information System (PMIS) is an online case management information system developed by the Center for Policy Research (CPR) for Pathways to Success.

4 If the survey questions and data provided through administrative data are identical or similar enough to provide the necessary information, the project team will remove the survey question(s) from the survey.

5 A dry bag is a flexible, waterproof bag with a roll-top closure. For SYSIL, the study contact information will be printed on the dry bag to help us familiarize youth with the study.

6 Examples of sensitive topics include (but not limited to): social security number; sex behavior and attitudes; illegal, anti-social, self-incriminating and demeaning behavior; critical appraisals of other individuals with whom respondents have close relationships, e.g., family, pupil-teacher, employee-supervisor; mental and psychological problems potentially embarrassing to respondents; religion and indicators of religion; community activities which indicate political affiliation and attitudes; legally recognized privileged and analogous relationships, such as those of lawyers, physicians and ministers; records describing how an individual exercises rights guaranteed by the First Amendment; receipt of economic assistance from the government (e.g., unemployment or WIC or SNAP); immigration/citizenship status.

7 For follow-up data collection, we estimate a 5 percent increase in the estimated number of youth age 18 and older for every six months. The estimated percentage of youth age 18 and older for follow-up data collection are: 55 percent for 6 months, 60 percent for 12 months, and 70 percent for 24 months.

8 Data collection for the 24-month follow-up is likely to extend beyond 3 years. A request to continue data collection will be submitted prior to the expiration date of OMB approval.

9 The program director check-ins do not require a unique instrument; they will be conducted using a subset of items from the interview protocol.

10 Data collection for the 24-month follow-up is likely to extend beyond 3 years. A request to continue data collection will be submitted prior to the expiration date of OMB approval.

19