SSA_TLPEval_NonSubChange_20180503_clean

The Family and Youth Services Bureau in the Administration for Children and Families (ACF) at the U.S. Department of Health and Human Services (HHS) seeks approval for a non-substantive change to the to the currently approved information collection for the Evaluation of the Transitional Living Program (TLP) (OMB No. 0970-0383) in order to revise the study design from a randomized controlled trial to a pre-post outcomes study with fewer data collection time points.

This nonsubstantive change is limited to the data collection activities that were planned for the random assignment study involving surveys of youth enrolled in the TLP grantee sites. The proposed nonsubstantive change is necessary to obtain outcomes on youth when it was determined through a pilot study designed to examine the feasibility of the random assignment study with this particular program and population, would not be feasible to reach the target sample size of 1250 youth within the time frame for study completion.

Study Background

The purpose of the TLP Evaluation is to provide ACF with information about how youth who participate in the TLP fare over the short and longer term in the areas of stable housing, positive social connections, social and emotional well-being, and education or employment. The evaluation will also provide information on services available to youth during and after exiting the program, including assistance in locating and retaining permanent housing and referrals to other residential programs, and services for employment, education, health, and mental health.

The collection involves three components:

A pilot study to test the feasibility of using a random assignment study design with the TLP and runaway and homeless youth;
Interviews with TLP staff as part of a qualitative study to gain insight into program implementation; and
Youth surveys as part of a quantitative study to understand program effects on youth outcomes.

The first two data collection activities began in October 2016 and are complete.

After nine months of collection for the pilot study, we reviewed the study enrollment rate (which was 12.8 percent of the target sample for youth surveys across the treatment and control groups) and the 3-month follow up rate (20 percent across the treatment and control groups) and determined it would not feasible to reach the target sample size of 1,250 youth within the time frame for study completion.

Based on these results, we propose to the following changes to the youth survey component of the TLP Evaluation:

1) Drop the control group and conduct a pre-post outcomes study with youth who enroll in the TLP;

2) Decrease the sample size to roughly half the number of participants originally proposed (i.e., to 600 youth); and

3) Focus resources on fewer follow-up data collection points.

Instead of six follow up points as initially approved, we propose three: baseline, 6 months after enrollment, and 12 months after enrollment. We will implement the same follow up survey (i.e., the Young Adult Follow-Up Survey) at the 6-month and 12-month collections. We do not propose any changes to the content of the surveys from what was previously approved by OMB.

Legal or Administrative Requirements that Necessitate the Collection

The Runaway and Homeless Youth Act (RHYA), as amended by Public Law 106-71 (42 U.S.C. 5701 et seq.), provides for the Transitional Living Program (TLP), a residential program designed to prepare older homeless youth ages 16-21 for a healthy and self-sufficient adulthood. The following amendment was included in the 2003 “Runaway, Homeless Youth and Missing Children’s Assistance Act” (P.L. 108-96), which reauthorized the “Runaway and Homeless Youth Act”. This amendment directed the Secretary of Health and Human Services to evaluate RHYA part B programs to report on long-term housing outcomes for youth after exiting the program.

SEC. 119. STUDY OF HOUSING SERVCIES AND STRATEGIES

The Secretary of Health and Human Services shall conduct a study of programs funded under part B of the Runaway and Homeless Youth Act (42 W.S.C. 5714-1 et seq.) to report on long-term housing outcomes for youth after exiting the program. The study of any such program should provide information on housing services available to youth upon exiting the program, including assistance in locating and retaining permanent housing and referrals to other residential programs. In addition, the study should identify housing models and placement strategies that prevent future episodes of homelessness.

The RHYA was again reauthorized in 2008 under the Reconnecting Homeless Youth Act of 2008 (P.L. 110-378).

A2. Purpose of Survey and Data Collection Procedures

The proposed collection for the youth outcomes study is being carried out through a set of surveys administered to run away and homeless youth who enroll in the TLP. The surveys are designed to measure youth’s baseline, shorter-term (6-month), and longer-term (12-month) outcomes and the types and amounts of services they receive.

If this request is approved, the study will collect data from youth on their demographic characteristics, receipt of TLP or “TLP-like” services, housing, employment, education, social connections (e.g., social relationships, civic engagement), psychosocial well-being (e.g., depressive symptoms, traumatic stress, risky behavior, history of abuse), and other measures related to self-sufficiency and well-being (exposure to violence, financial competence). The study will also collect identifying information, including contact information, for the purposes of tracking youth throughout the evaluation and locating them for follow-up. As shown in Exhibit A.2.1 below, data will be collected from youth at three points in time.

Exhibit A.2.1 Overview of Data Collection from Youth

	Instrument	Timing	Purpose	Estimated Length
1	Young Adult Baseline Survey	At the time of enrollment	Measure background characteristics, initial status on outcomes, gather contact information	0.62 hours
2	Young Adult 6-Month Outcomes Survey	6 months after baseline	Measure full set of outcomes	0.61 hours
3	Young Adult 12-Month Follow-up Survey	12 months after baseline	Measure full set of outcomes	0.61 hours

The first collection will occur at baseline, at the time of enrollment into the program and study (prior to delivery of the intervention). The main purpose of the baseline survey is to measure participants’ initial status with respect to the outcome variables. As shown in Attachment C, the baseline survey queries participants’ experiences across varied aspects of their lives, including housing and homelessness, social connections (e.g., supportive relationships with adults, supportive relationships with peers, peer delinquency, civic engagement, civic attitudes); psychosocial well-being, (e.g., self-esteem, self-efficacy, depressive symptoms, traumatic stress, delinquency, criminality, substance use, risky sexual behavior); education or training, employment, and other relevant measures (e.g., money management, health care needs and use, history of abuse and neglect, personal goals and progress toward them). In order to obtain information necessary to describe the sample, the baseline survey asks about such demographic and background characteristics as race, ethnicity, language, citizenship, and gender/sexual identity. Many of these characteristics will be used as statistical controls in the analysis. To provide a basis for understanding changes in service receipt associated with participation in the TLP, the baseline survey also measures current or prior receipt of TLP or TLP-like services.

The baseline survey is important in several ways for the analysis. First, it provides the baseline measures against which changes in outcomes will be measured—that is, the pretest measures for the outcomes analysis. Second, baseline measures may also be used to define subgroups for analyzing program effects. Third, they may be used to adjust estimates of program effects for the baseline characteristics of non-respondents to the follow-up survey (e.g., non-response weights). Fourth, time invariant traits (e.g., race-ethnicity) and other background characteristics (e.g., age and gender) collected at baseline will be included in our statistical models as covariates in order to improve the precision of estimates of program effects. As shown in Attachment D, at 6 and 12 months after baseline, youth will complete a similarly comprehensive follow-up survey. The follow up surveys are nearly identical to the baseline survey, using the same measures to obtain data on participants’ experiences in each of the outcome categories. There are minor adjustments appropriate to a follow-up survey—for example, a change in the reference period to query experiences since baseline and removal of such demographic items as race and ethnicity that do not change over time. The data collected on the 6- and 12-month follow-up surveys are necessary to assess the program’s effects youth outcomes. The 6- and 12-month follow-up surveys collect information about the receipt of TLP or TLP-like services—namely the nature and amount of housing, employment, education, alcohol or drug treatment , mental health treatment and counseling, mentoring, and coaching, physical health care, life and interpersonal skill building, legal, family reunification, case management, and auxiliary services (e.g., assistance with food, transportation, etc.). By collecting this information, the study will be able to estimate TLP’s effect on these service areas. These service-receipt data may help explain any observed program effects or lack thereof.

The data collection instruments themselves are provided in the following attachments: Attachment C is the baseline survey; Attachment D is the follow-up survey (administered at both 6 months and 12 months).

A3. Improved Information Technology to Reduce Burden

The data collection plan reflects sensitivity to issues of efficiency, accuracy, and respondent burden. Where feasible, information will be gathered from existing data sources; the information being requested through surveys is limited to that for which the youth are the best or only information sources. For all youth surveys (baseline, 6-month follow-up, and 12-month follow-up) secure web-based technology will be used to reduce burden, improve accuracy of responses, and ensure data security. The survey will be hosted on a secure, encrypted, passcode protected digital platform, which will capture and store data in real time. Each response to a question (as it is entered) is sent immediately to a central and secure database and no information is stored on local computers. Research has demonstrated that surveys self-administered online are characterized by higher levels of self-disclosure, an increased willingness to answer sensitive questions and a reduction in socially desirable responses. The survey instruments will be translated into Spanish, so that respondents can choose the language in which they take it. There are no plans to back-translate the survey from Spanish back into English. A reputable professional translator who has produced accurate and reliable translations for other large scale national studies will be used.

A4. Efforts to Identify Duplication

This study is a unique, one-time event studying a population that has not previously been surveyed nationally or as intensively and systematically as this research envisions. ACF collects in-service information on non-identified youth through the Homeless Management Information System (HMIS). Information from HMIS can provide background and context but not in sufficient detail or individual specificity to produce high quality research. Moreover, HMIS reports on youth’s needs and services received while participating in the TLP; however, youth’s ongoing housing status after exiting the program is not available in HMIS, only their immediate destination at exit.

A5. Involvement of Small Organizations

Most TLP grantees are small entities, operated by community-based organizations. The plan for collecting data from youth is designed to minimize burden on such sites by utilizing a web-based survey platform and providing training and technical support from the evaluation contractor team that will enable staff to help youth register and log onto the survey portal.

For the follow-up surveys, the contractor, Abt Associates will provide training, support, ongoing monitoring, and other assistance to help grantee staff in their outreach efforts and bolster the follow up collection effort. In addition, the contractor will also provide support and technical assistance to youth respondents who would like assistance or have difficulty accessing or completing the survey on their own. For example, the contactor staff will help youth troubleshoot connectivity issues, provide passwords for web completion, and as needed conduct phone-based survey completions. Grantees will be asked to help contact youth who have not been heard from when their expected follow-up surveys are due.

Findings from the survey will in no way impact the funding or management of the grantees, although improvements in program design may ultimately occur once the effects of the program are determined.

A6. Consequences of Less Frequent Data Collection

The data being collected are essential to conducting a rigorous evaluation of the TLP as the reauthorization language requires. The data are necessary for determining whether the intervention had short-term (6 month) or longer-term (12 month) effects on program participants. Furthermore, without additional study, funding decisions about TLPs will continue to be based on insufficient information on program effectiveness.

A7. Special Circumstances

There are no special circumstances for the proposed data collection efforts.

A8. Federal Register Notice and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), ACF published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on July 29, 2014, Volume 79, Number 145, page 44040, and provided a sixty-day period for public comment. During the notice and comment period, we did not receive any public comments.

OPRE and FYSB have worked to partner with other agencies in the review of the research design as well as the collection tools. Staff and leadership from OPRE have weighed in. Additionally, FYSB project officers and technical assistance staff have provided feedback regarding the project and have performed programmatic assessments of the potential grantee organizations to ensure that they have the capacity to support a study of this nature.

A9. Incentives for Respondents

The population targeted for the evaluation presents a challenge for the study that is increased by the desire to measure longer-term program effects beyond program exit. By design, the program in this evaluation targets youth who are disconnected and highly mobile and hard to reach. To achieve an acceptable response rate for the 6- and 12-month follow-up surveys, it is important to take steps to attach youth firmly to the study at the outset and to maintain that attachment over time. These steps are also essential to prevent attrition, which could lead to insufficient statistical power (due to a small analytic sample size) and nonresponse bias (when survey respondents differ in meaningful ways from nonrespondents). To this end, OMB previously approved modest incentives for survey completion for each participant at each survey point.

As previously approved, each youth will receive an electronic gift card upon submission of each survey. The incentive schedule is presented below in Exhibit A.9.1. For the baseline and follow-up surveys, incentives are in the form of electronic gift cards to Amazon.com.

The research team considered a variety of different incentive approaches, including prepaid VISA gift cards. The research team selected the Amazon.com gift card as the best option because these cards do not expire, can be canceled without a fee, can be replaced with a new code that is sent electronically to the participant, and are controlled and monitored by the study team.

By contrast, the study team opted against the use of VISA gift cards for several reasons:

VISA gift cards come with an assortment of additional fees—activation fees, shipping fees, cancellation fees (e.g., if lost or stolen), and inactivity fees (e.g., if the cards remain idle for too long), which add costs to the overall study and decreases the value of the card to participants.
Unlike Amazon.com gift cards, VISA gift cards have expiration dates that can vary depending on which bank issues the gift card.
To use VISA gift cards, a prepaid VISA gift card must be physically handed to each youth upon completing the baseline survey, which requires the study team to mail hundreds of prepaid gift cards to the TLP grantees that will participate in the study. Grantee staff would be responsible for distributing the gift cards to youth. This approach is highly undesirable because: (a) the study team loses control over the distribution of the gift cards, and thus it increases the opportunities to misuse the gift cards and reduces the team’s ability to monitor the incentive process; and (b) lost or stolen gift cards must be replaced with new cards, which requires a physical mailing address to receive them.

Exhibit A.9.1 Schedule of Compensation

Survey	Incentive
Young Adult Baseline Survey	$30
Young Adult 6-Month Follow-up Survey	$30
Young Adult 12-Month Follow-up Survey	$40

The amount of the incentive is $30 each for the baseline and 6-month surveys. The 12-month incentive is $40, slightly more to emphasize its importance to the study. Homeless youth are an especially transient population, making them particularly difficult to survey over many months. Thus, we believe that these gift card amounts are necessary to encourage participation in the study and completion of the follow-up surveys.

The electronic gift cards will be emailed or text messaged to study participants, to an email address or cell phone number they designate. If a youth does not have access to email or cell phone, the study team will ship the gift card codes to the address of their choice. Youth will receive their gift card after they have submitted their survey. For a suspended, partially completed survey, youth will be shipped their gift card after their individual window for completion of the said survey has closed

This compensation model is designed to increase the survey response rates, particularly from youth dispersing widely into the general population as they gain the independence, which is the purpose of TLP.

In addition to providing incentives to bolster follow up survey response rates and reduce study attrition, we anticipate the need for both passive and active participant tracking approaches. We will rely on multiple tracking mechanisms to ensure the highest response rate possible, and the type of mechanism utilized will vary depending on the status of youth.

A10. Privacy of Respondents

Protections for privacy are embedded in the study design. Data collection will only occur if informed consent (or assent with parental consent in the case of minors) is provided by youth themselves. Information collected will be kept private to the extent permitted by law. Respondents will be informed of all planned uses of data, that their participation is voluntary, and that their information will be kept private to the extent permitted by law. Copies of the youth and parent consent forms for the outcomes study are provided as Attachments A and B. In order to ensure the privacy and safety of the runaway and homeless youth participating in the study, the contractor will obtain a waiver of parental permission from its Institutional Review Board (IRB). Federal regulations permit the IRB to approve research without parent permission “if the IRB determines that a research protocol is designed for conditions or for a subject population for which permission is not a reasonable requirement to protect the subjects, provided an appropriate mechanism for protecting the children who will participate as subjects in the research is substituted and provided further that the waiver is not inconsistent with federal, state or local law”.

All identifying information that will be collected with each youth’s informed consent and all records will be protected with security systems to guarantee privacy and confidentiality. Due to the sensitive nature of this research, the evaluation will obtain a Certificate of Confidentiality from the National Institutes of Health (NIH). The study team currently has a Certificate for the pilot study’s random assignment design and will update it with NIH so as to extend the protections to cover individuals participating in the youth outcomes study. The Certificate of Confidentiality helps to assure participants that their information will be kept private to the fullest extent permitted by law.

All data will be securely protected under IRB-certified ethical research methods. Each survey will contain an assurance of this, which will also convey that answers will be kept private, that youths’ participation is voluntary, and that they may refuse to participate at any time. Information will not be maintained in a paper or electronic system from which data are actually or directly retrieved by an individuals’ personal identifier.

A11. Sensitive Questions

Some questions in the youth surveys ask for information of a sensitive nature—for example, substance use, sexual behavior, delinquent activity, involvement with law enforcement, health and mental health symptoms (e.g., depressive symptoms, PTSD symptoms), and participation in certain categories of social services (e.g., addiction/substance use treatment, mental health services). These topics are very personal but also essential to understanding a youth’s overall situation, and they are collected because the program under evaluation (TLP) is designed specifically to address and reduce risk and improve well-being related to these very factors.

The less sensitive measures such as housing circumstances and attachment to education and employment are valued outcomes of TLP, but risk reduction, healthy choices, and pro-social adaptation are important as well. Youth will be assured of privacy and told they do not have to answer a particular question if they do not wish to.

A12. Estimation of Information Collection Burden

A total burden of 3,125 hours was previously approved (OMB No. 0970-0383). As part of the pilot study and site visit interviews with TLP staff, information was collected from 291 respondents for 249 hours of burden. The burden remaining from previously approved information collections is 2,876 hours. As shown in Exhibit A.12.1, only a portion of this is needed as a result of the change in study design. Exhibits A.12.1 and A.12.2 show the current propose burden for the updated information collection. These burden estimates are based on pretests of the survey instruments, as well as experience with similar data collection efforts for other studies.

Exhibit A.12.1 Burden table updated to reflect current status of data collection and nonsubstantive change Instrument	Total/Annual Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total Burden Hours
Site Visit Interviews
Program Overview Survey: Executive Director Interview Guide (1 Executive Director respondent per grantee)	Data collection complete
Program Overview Survey: Program Staff Interview Guide (4 Program Staff respondents per grantee)	Data collection complete
Youth Development Survey Interview Guide (1 Executive Director and 1 Program Staff respondent per grantee)	Data collection complete
Young Adult Surveys
Young Adult Baseline Survey	600	1	0.62	372
Young Adult 3-Month Follow Up Survey	No longer planned
Young Adult 6-Month Follow Up Survey	600	1	0.61	366
Young Adult 9-Month Tracking Survey	No longer planned
Young Adult 12-Month Follow Up Survey	600	1	0.61	366
Young Adult 15-Month Tracking Survey	No longer planned
Young Adult 18-Month Follow Up Survey	No longer planned
Estimated Total Burden Hours				1,104

Exhibit A.12.2. Simplified Burden Table: Current Burden Status with Nonsubstantive Change Request Approved

Instrument	Total/Annual Number of Respondents	Number of Responses per Respondent	Average Burden Hours per Response	Total/Annual Burden Hours
Young Adult Baseline Survey	600	1	0.62	372
Young Adult 6-Month Follow Up Survey	600	1	0.61	366
Young Adult 12-Month Follow Up Survey	600	1	0.61	366
Total				1,104

Total Annual Cost

No annual cost to respondents. The youth surveys will be completed by unpaid youth.

A13. Cost Burden to Respondents or Record Keepers

There are no annualized capital/startup or ongoing operation and maintenance costs associated with collecting this information. There are no direct respondent costs associated with this data collection other than opportunity costs of respondents’ time required to complete the surveys. The evaluation does not place any capital equipment, start-up, or record maintenance requirements on respondents.

A14. Estimate of Cost to the Federal Government

The overall one-time cost of the research, including survey design, information collection, analysis, and reporting will be $2,643,465. This includes: developing the study foundation ($833,333); data collection ($1,507,489), analysis ($155,227), and reports/briefing ($147,416). Because the contract takes place over six years, the estimated annualized is $440,577.

Annual costs of Federal employee time is estimated to be: $12,700 Executive Management Oversight (10% FTE), and $39,962 Project COR and Task Lead (40% FTE).

A15. Change in Burden

The requested change results in a net reduction in burden of 1,772 hours.

A16. Plan and Time Schedule for Information Collection, Tabulation and Publication

When data collection is completed, the Contractor will analyze the data and provide ACF with a written report detailing the findings. Detailed data tabulations will be part of the report and available for study. No personal information will be included in the report.

[NOTE: The description of the analytic methods previously presented here under the subheading “Plans for Tabulation” has been moved to SSB, Section B1. “Respondent Universe and Sampling Methods” and is presented under the heading “Analytic Methods.”]

Pending OMB approval, participant enrollment into the study and collection of the baseline survey data will occur over a 24-month period. Follow-up data collections are projected to occur over a 32-month period. Analysis and reporting would occur after baseline data collection is completed and continue until after follow up data collections are complete.

A17. Reasons Not to Display OMB Expiration Date

All instruments will display the OMB number and the expiration date.

A18. Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

File Type	application/msword
File Title	OPRE OMB Clearance Manual
Author	DHHS
Last Modified By	SYSTEM
File Modified	2018-05-04
File Created	2018-05-04