SSA_TLPEval_Extension_20180712_CLEAN

The Family and Youth Services Bureau (FYSB) and the Office of Planning, Research, Evaluation (OPRE) in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) are requesting an additional three years of approval for the currently approved information collection for the Evaluation of the Transitional Living Program (TLP) (OMB No. 0970-0383). The purpose of the approval is to continue data collection for the pre-post Youth Outcomes Study.

The Evaluation of the Transitional Living Program is composed of three project components, two of which have been completed: 1) a Pilot Study to test the feasibility of using a random assignment study design with the TLP and runaway and homeless youth; 2) interviews with TLP staff as part of a qualitative study to gain insight into program implementation (Implementation Interviews); and 3) a Youth Outcomes Study that uses a pretest/posttest study design and data collected through youth surveys to measure changes in youth outcomes associated with participating in the TLP.

Data collection for the first two project components (Pilot Study and program staff interviews) is complete. Data collection activities for the Youth Outcomes Study are expected to continue beyond the current OMB expiration date of July 31, 2018. As such, ACF is requesting a three- year approval to complete the planned data collection activities, which include enrolling youth in the study, surveying youth participants at enrollment (baseline), and again 6 months and 12 months after enrollment using previously approved data collection instruments.

Study Background

The purpose of the Evaluation of the Transitional Living Program is to provide ACF with information about how youth who participate in FYSB-funded TLPs fare over the short and longer term in the areas of stable housing, positive social connections, social and emotional well-being, and education or employment. The evaluation will also provide information on receipt of services (e.g., housing assistance, employment, education, substance use treatment, mental health care, physical health care, and life skills building, among others) prior to and following enrollment in the TLP.

OMB approved the initial information collection request for activities related to the Evaluation of the Transitional Living Program in July 2015 (OMB No. 0970–0383). ACF subsequently requested and received approval to revise the study design from a randomized controlled trail to a pre-post outcomes study with fewer data collection time points (approved May 2018).

Legal or Administrative Requirements that Necessitate the Collection

The Runaway and Homeless Youth Act (RHYA), as amended by Public Law 106-71 (42 U.S.C. 5701 et seq.), provides for the Transitional Living Program (TLP), a residential program designed to prepare older homeless youth ages 16-21 for a healthy and self-sufficient adulthood. The following amendment was included in the 2003 “Runaway, Homeless Youth and Missing Children’s Assistance Act” (Public Law 108-96), which reauthorized the RHYA. This amendment directed the Secretary of Health and Human Services to evaluate RHYA part B programs to report on long-term housing outcomes for youth after exiting the program.

SEC. 119. STUDY OF HOUSING SERVCIES AND STRATEGIES

The Secretary of Health and Human Services shall conduct a study of programs funded under part B of the Runaway and Homeless Youth Act (42 W.S.C. 5714-1 et seq.) to report on long-term housing outcomes for youth after exiting the program. The study of any such program should provide information on housing services available to youth upon exiting the program, including assistance in locating and retaining permanent housing and referrals to other residential programs. In addition, the study should identify housing models and placement strategies that prevent future episodes of homelessness.

The RHYA was again reauthorized in 2008 under the Reconnecting Homeless Youth Act of 2008 (Public Law 110-378).

A2. Purpose of Survey and Data Collection Procedures

Overview of Purpose and Approach

ACF’s request for an additional three years of approval is specific to the Youth Outcomes Study, which is designed to assess whether participation in the TLP is associated with improvements in the developmental domains targeted by the intervention—namely, stable housing, positive social connections, social and emotional well-being, and employment or education. To accomplish this, the research team is using a pretest-posttest design and longitudinal survey data collection.

The surveys focus on youth outcomes that can only be measured through surveys of youth. The outcome domains are:

Stable housing (e.g., residential mobility, homelessness)
Positive social connections (e.g., supportive relationships with adults, delinquent peers)
Social and emotional well-being (e.g., depressive symptoms, traumatic stress, risky behavior, history of abuse)
Employment or education (e.g., employment status and effort, educational status and effort), and
Other measures related to self-sufficiency and well-being (e.g., child bearing, money management).

The surveys also collect data from youth on their demographic characteristics, the types and amounts of services they receive after enrolling in the TLP, and contact information for the purposes of locating and contacting youth for follow-up data collections.

Research Questions

The survey data will be used to address the following research questions about the relationships between TLP participation and outcomes important to the long-term self-sufficiency and well-being of homeless youth:

What is the relationship between TLP participation and housing stability?
1. For example, are youth who participate in the TLP more stably housed 6 months and/or 12 months after program enrollment relative to baseline?

What is the relationship between TLP participation and positive social connections?
1. For example, do youth who participate in the TLP have more supportive relationships with adults 6 months and/or 12 months after program enrolment relative to baseline?

What is the relationship between TLP participation and social and emotional well-being?
1. For example, do youth who participate in the TLP have improved social and emotional well-being (e.g., fewer symptoms of depression, higher levels of self-efficacy) 6 months and/or 12 months after program enrollment relative to baseline?

What is the relationship between TLP participation and employment or education?
1. For example, do youth who participate in the TLP have higher levels of employment or education 6 months and/or 12 months after program enrollment relative to baseline?

In addition, the evaluation will address service receipt by answering the question:

What types of services are related to TLP participation?
1. What types of services do youth who participate in the TLP receive in the 6 months and/or 12 months after program enrollment?

Study Design

The Youth Outcomes Study will use a pretest-posttest design to measure the relationship between TLP participation and youth outcomes. The relationship will be estimated as the change in outcomes after 6 months (the difference between outcomes measured at enrollment and 6 months later) and after 12 month (the difference between outcomes measured at enrollment and 12 months later). In a pretest-posttest design, the pre-condition (prior to TLP participation) can be thought of as being like a control group and the post-condition (6 or 12 months after TLP participation) can be thought of as being like the treatment group.

The research team is in the process of identifying and recruiting 50 study sites from among the organizations awarded TLP grants in 2017 to participate in the Youth Outcomes Study. Once study sites are recruited, data collection will involve 600 runaway and homeless youth who enroll in the TLP at one of the 50 study sites and agree to take part in the study. At the time of program enrollment, youth will be invited to participate in the outcomes study and undergo informed consent (or assent with parental consent in the case of minors). (See Attachments A and B for youth and parent study consent forms.) Youth who agree to participate will be asked to complete three surveys: at baseline (upon program enrollment and study consent), 6 months after program enrollment, and 12 months after program enrollment. (See Attachments C and D for the survey instruments.)

The mode of data collection is consistent across all sites and all data collection time points. For privacy and data security reasons, all surveys will be self-administered by youth and collected via a secure web-based platform accessible from any Internet-enabled computer (desktop, laptop, tablet, or other such device). The survey will be available in English and Spanish and will allow youth to select their preferred language.

Universe of Data Collection Efforts

The proposed collection for the Youth Outcomes Study is being carried out through a set of surveys administered to youth, ages 16-21 at baseline, who are homeless and enroll in the TLP at one of 50 study sites across the U.S. The previously approved survey instruments are presented in Exhibit A.2.1.

Exhibit A.2.1 Overview of Data Collection from Youth

	Instrument	Timing	Purpose	Estimated Length
1	Young Adult Baseline Survey	At the time of enrollment	Measure background characteristics, initial status on outcomes, gather contact information	0.62 hours
2	Young Adult 6-Month Outcomes Survey	6 months after baseline	Measure full set of outcomes	0.61 hours
3	Young Adult 12-Month Follow-up Survey	12 months after baseline	Measure full set of outcomes	0.61 hours

The first collection will occur at baseline, at the time of enrollment into the program and study (prior to delivery of the intervention). The main purpose of the baseline survey is to measure participants’ initial status with respect to the outcome variables. The baseline survey queries participants’ experiences across varied aspects of their lives, including housing and homelessness, social connections, social and emotional well-being, education or employment, and other areas relevant to self-sufficiency (e.g., money management). To provide a basis for understanding changes in service receipt associated with participation in the TLP, the baseline survey also measures current or prior receipt of TLP or TLP-like services (e.g., housing, employment, education, substance use treatment, mental health care, physical health care, life skill building, etc.). In order to obtain information necessary to describe the sample, the baseline survey asks about demographic characteristics (as race/ethnicity, primary language). The same main outcome and service receipt measures are collected on the 6- and 12-month follow-up surveys and are necessary to assess the program’s effects on youth outcomes.

A crosswalk showing the measures collected at the baseline, 6-month, and 12-month collections, their corresponding survey item numbers, and the sources of the measures is presented in Attachment E. A listing of the items included in each instrument and their sources is presented in Attachment F.

A3. Improved Information Technology to Reduce Burden

ACF is not proposing any changes to the previously approved data collection instruments (listed in Section A1) or to the previously approved data collection procedures as part of this approval request. The previously approved data collection procedures reflect sensitivity to issues of efficiency, accuracy, and respondent burden. The information being requested through surveys is limited to that for which the youth are the best or only information sources. For all youth surveys (baseline, 6-month follow-up, and 12-month follow-up) secure web-based technology will be used to reduce burden, improve accuracy of responses, and ensure data security. The survey will be hosted on a secure, encrypted, passcode protected digital platform, which will capture and store data in real time. Each response to a question (as it is entered) is sent immediately to a central and secure database and no information is stored on local computers. Research has demonstrated that surveys self-administered online are characterized by higher levels of self-disclosure, an increased willingness to answer sensitive questions, and a reduction in socially desirable responses.¹ The baseline survey has been translated into Spanish and the follow-up surveys will be translated into Spanish, so that respondents can choose the language in which they take it.

A4. Efforts to Identify Duplication

This study is a unique, one-time event studying a population that has not previously been surveyed nationally or as intensively and systematically as this research envisions. ACF collects in-service information on non-identified youth through the Homeless Management Information System (HMIS). Information from HMIS can provide background and context but not in sufficient detail or individual specificity to produce high quality research. Moreover, HMIS reports on youth’s needs and services received while participating in the TLP; however, youth’s ongoing housing status after exiting the program is not available in HMIS, only their immediate destination at exit.

A5. Involvement of Small Organizations

Most TLP grantees are small entities, operated by community-based organizations. The plan for collecting data from youth is designed to minimize burden on such sites by utilizing a web-based survey platform and providing training and technical support from the research team that will enable staff to help youth register and log onto the survey portal.

For the follow-up surveys, the contractor, Abt Associates, will provide training, support, ongoing monitoring, and other assistance to help grantee staff in their outreach efforts and bolster the follow up collection effort. In addition, the contractor will also provide support and technical assistance to youth respondents who would like assistance or have difficulty accessing or completing the survey on their own. For example, the contractor staff will help youth troubleshoot connectivity issues, provide passwords for web completion, and as needed conduct phone-based survey completions. Grantees will be asked to help contact youth who have not been heard from when their expected follow-up surveys are due.

Findings from the survey will in no way impact the funding or management of the grantees, although improvements in program design may ultimately occur once the effects of the program are determined.

A6. Consequences of Less Frequent Data Collection

The data being collected are essential to conducting an evaluation of the TLP as the reauthorization language (Public Law 108-96) requires. The data are necessary for determining whether short-term (6 month) or longer-term (12 month) changes in youth outcomes are associated with TLP participation. Without this information, ACF cannot answer the primary research questions or understand the extent to which TLP may (or may not) promote positive short- and longer-term outcomes for the runaway and homeless youth served.

A7. Special Circumstances

There are no special circumstances for the proposed data collection effort.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on May 16, 2018, Volume 83, Number 95, page 22688, and provided a sixty-day period for public comment. A copy of this notice is attached as Attachment G. During the notice and comment period, we did not receive any public comments.

Consultation with Experts Outside of the Study

Most of the items in the survey are from previously approved data collection instruments. Researchers have and will continue to consult with experts in their respective disciplines from ACF and Abt Associates in developing the survey instruments, study design, and protocols for data collection.

A9. Incentives for Respondents

ACF is not requesting any change to the previously approved incentives provided to respondents. As previously approved, each youth will receive an electronic gift card to Amazon.com upon submission of his/her survey. The amount of the incentive is $30 each for the baseline survey and the 6-month survey. The 12-month survey incentive is $40, slightly more to emphasize its importance to the study. The Youth Outcomes Study is a longitudinal study that has the added complexity of recruiting, locating, and following up with a very hard-to-reach population of minority and underrepresented youth that is particularly distrustful of others and that does not easily disclose information of a sensitive nature. This incentive model is designed to increase the survey response rates, with specific attention to youth that will be dispersing widely into the general population as they gain independence, which is the purpose of TLP. We believe that these gift card amounts are necessary to encourage participation in the study and completion of the follow-up surveys.

Gift cards are provided to study participants as an incentive to participate in the study. These gift cards are important for this evaluation because the study participants are unaccompanied runaway and homeless youth who are overwhelmingly disconnected, highly mobile, and hard to reach, making them particularly difficult to track and survey over extended time periods. In addition, the surveys include highly sensitive questions. The challenge for the study is increased by the desire to measure longer-term outcomes beyond program exit (e.g., at the 12-month collection). Research has shown that incentives are effective at increasing response rates for populations similar to participants in TLP (i.e., lower education levels, minority, and underrepresented populations).^²,Research also suggests that providing an incentive for earlier surveys may contribute to higher response rates for subsequent surveys.^³

Without the modest incentives OMB has approved, the evaluation risks having unacceptably low response rates for the 6- and 12-month follow-up surveys which would lead to insufficient statistical power (due to a small analytic sample size) and nonresponse bias (when survey respondents differ in meaningful ways from non-respondents. Response rates for undereducated, minority, and high-risk youth, which make up the majority of the homeless youth population served by TLPs, have been shown to increase with the use of the types of incentives we propose.⁴

A10. Privacy of Respondents

Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law.

Data collection will only occur if informed consent (or assent with parental consent in the case of minors) is provided by youth themselves. Copies of the youth and parent consent forms for the outcomes study are provided as Attachments A and B. In order to ensure the privacy and safety of the runaway and homeless youth participating in the study, the contractor has obtained a waiver of parental permission from its Institutional Review Board (IRB). All identifying information that will be collected with each youth’s informed consent and all records will be protected with security systems to guarantee privacy and confidentiality. All data will be securely protected under IRB-certified ethical research methods. Each survey will contain an assurance of this, which will also convey that answers will be kept private, that youths’ participation is voluntary, and that they may refuse to participate at any time.

Due to the sensitive nature of this research (see A.11 for more information), the evaluation will obtain a Certificate of Confidentiality. The study team is applying for this Certificate and will provide it to OMB once it is received. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.

Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

As specified in the contract, the Contractor shall protect respondent privacy to the extent permitted by law and will comply with all Federal and Departmental regulations for private information. The Contractor has developed a Data Security Plan that assesses all protections of respondents’ personally identifiable information.

As specified in the evaluator’s contract, the Contractor shall use Federal Information Processing Standard compliant encryption (Security Requirements for Cryptographic Module, as amended) to protect all instances of sensitive information during storage and transmission. The Contractor shall securely generate and manage encryption keys to prevent unauthorized decryption of information, in accordance with the Federal Processing Standard. The Contractor shall: ensure that this standard is incorporated into the Contractor’s property management/control system; establish a procedure to account for all laptop computers, desktop computers, and other mobile devices and portable media that store or process sensitive information.

In addition, the Contractor shall have a plan for the protection of any paper records, field notes, or other documents that contain sensitive or personally identifiable information.

The Contractor shall ensure that all of its employees, subcontractors (at all tiers), and employees of each subcontractor, who perform work under this contract/subcontract, are trained on data privacy issues and comply with the above requirements.

A11. Sensitive Questions

Some questions in the youth surveys ask for information of a sensitive nature—for example, substance use, sexual behavior, delinquent activity, involvement with law enforcement, health and mental health symptoms (e.g., depressive symptoms, PTSD symptoms), and participation in certain categories of social services (e.g., addiction/substance use treatment, mental health services). These topics are very personal but also essential to understanding a youth’s overall situation, and they are collected because the program under evaluation (TLP) is designed specifically to address and reduce risk and improve well-being related to these very factors.

The less sensitive measures such as housing circumstances and attachment to education and employment are valued outcomes of TLP, but risk reduction, healthy choices, and pro-social adaptation are important as well. Youth will be assured of privacy and told they do not have to answer a particular question if they do not wish to. Copies of the youth and parent consent forms for the Youth Outcomes Study are provided as Attachments A and B.

A12. Estimation of Information Collection Burden

Previously Approved Information Collections

Total Burden Previously Approved

A total burden of 1,104 hours for the Youth Outcomes Study component and 140 hours for the Implementation Interview component of the TLP Evaluation was previously approved (OMB No. 0970-0383).

Exhibit A.12.1 Total burden table updated to reflect current status of data collection and extension approval request

Instrument	Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total Burden Hours
Implementation Interviews
Program Overview Survey: Executive Director Interview Guide (1 Executive Director respondent per grantee)	14	1	1	14
Program Overview Survey: Program Staff Interview Guide (4 Program Staff respondents per grantee)	56	1	2	112
Youth Development Survey Interview Guide (1 Executive Director and 1 Program Staff respondent per grantee)	28	1	0.50	14
Young Adult Surveys
Young Adult Baseline Survey	600	1	0.62	372
Young Adult 6-Month Follow Up Survey	600	1	0.61	366
Young Adult 12-Month Follow Up Survey	600	1	0.61	366
Total				1,244

Newly Requested Information Collections

All data collection for the implementation interviews is now complete. To date, none of the approved burden hours for the Youth Outcomes Study have been used. As shown in Exhibit A.12.2, future collections for the Youth Outcomes Study will involve a total of 1,800 respondents. Since we will collect data over a three year period, this has been annualized. The total burden over three years is expected to be 1,104 hours of burden, or 368 hours annualized. These burden estimates are based on pretests of the survey instruments, as well as experience with similar data collection efforts for other studies.

Exhibit A.12.2. Simplified Burden Table: Current Burden Status

Instrument	Total Number of Respondents	Annual Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total/Annual Burden Hours	Average Hourly Wage	Total Annual Cost
Young Adult Baseline Survey	600	200	1	0.62	124	$0	$0
Young Adult 6-Month Follow Up Survey	600	200	1	0.61	122	$0	$0
Young Adult 12-Month Follow Up Survey	600	200	1	0.61	122	$0	$0
Total					368		$0

Total Annual Cost

There are no annual costs to respondents. The youth surveys will be completed by unpaid youth.

A13. Cost Burden to Respondents or Record Keepers

There are no additional costs to respondents.

A14. Estimate of Cost to the Federal Government

The annual cost of the research activities covered under this request will be $227,603.75. The total cost of the research activities covered under this request will be $682,811.25. This includes costs for recruiting and training sites, and ongoing data collection, support, and monitoring.

A15. Change in Burden

The current request does not increase the overall burden previously approved, it just extends the time for data collection.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

When data collection is completed, the Contractor will analyze the data and provide ACF with a written report detailing the findings. Detailed data tabulations will be part of the report and available for study. No personal information will be included in the report.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the expiration date for OMB approval.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

1 Gnambs, T. & Kaspar, K. (2015). Disclosure of sensitive behaviors across self-administered survey modes: a meta-analysis. Behavior Research Methods, 47, 1237-1259.

2 Berlin, Martha, Leyla Mohadjer, Joseph Waksberg, Andrew Kolstad, Irwin Kirsch, D. Rock, and Kentaro Yamamoto. 1992. An experiment in monetary incentives. In JSM proceedings, 393–98. Alexandria, VA: American Statistical Association.

3 Singer, Eleanor, John Van Hoewyk, and Mary P. Maher. 1998. Does the payment of incentives create expectation effects? Public Opinion Quarterly 62:152–64.

4 Berlin, Martha, Leyla Mohadjer, Joseph Waksberg, Andrew Kolstad, Irwin Kirsch, D. Rock, and Kentaro Yamamoto. 1992. An experiment in monetary incentives. In JSM proceedings, 393–98. Alexandria, VA: American Statistical Association

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	OPRE OMB Clearance Manual
Author	DHHS
File Modified	0000-00-00
File Created	2021-01-20

SSA_TLPEval_Extension_20180712_CLEAN