PART B Supporting Statement TLP 2015 7 20 15clean

PART B Supporting Statement TLP 2015 7 20 15clean.docx

Evaluation of the Transition Living Program

OMB: 0970-0383

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 0970-0383 can be found here:

Document [docx]

Download: docx | pdf

B. Statistical Methods (used for collection of information employing statistical methods)

Respondent Universe and Sampling Methods

The sampling frame includes all TLPs funded in FY2013, and 14 TLPs will be selected to participate in the study. The selection process is purposive. A pool of 28 candidate TLPs were initially identified from among FYSB’s TLP grantees for recruitment into the study based on two primary criteria:

Contrast, which measures the level of expected service contrast between the TLP and its corresponding Continuum of Care (CoC),
Expected entries, which estimates the TLP service volume over an 18-month period

In addition, Maternity Group Homes are excluded from consideration because these programs serve a special subpopulation and offer a unique set of services to address their needs. New TLPs with little prior program experience will also be excluded.

The contractor was provided a complete list of TLP grantees currently funded, as well as information about their service volume. The contrast metric represents the proportion of a CoC’s projected, 18-month service volume in any type of transitional housing program that is provided by a TLP. The metric is used to identify TLPs that will provide the greatest level of contrast to other services available in the community. The expected entries metric is an estimate of the TLP service volume and is a gauge of the grantee’s size. To identify an initial set of 14 grantees, TLPs were sorted by contrast, and the top quartile (31 TLPs) was given priority status as “high contrast TLPs”. The high contrast TLPs were then rank-ordered according to their projected service volume to identify TLPs with a high service volume. Also, FY2013 TLP grantees within a 1-hour drive (or in the same CoC) of one of the high-contrast and high-service-volume TLPs will also be considered to minimize the potential for cross-over (i.e., control group members who receive the TLP services).

The contractor is currently in the process of screening the 28 candidate TLPs to identify those to include in the evaluation. This involves evaluating grantees based on three primary criteria:

Size of the agency,
Oversubscription (i.e., excess demand for services or the size of the waiting list or turn away rate), and
Control environment (i.e., likelihood of crossover).

Achieving the requisite sample size for both treatment and control groups requires that the study target relatively large TLPs with adequate excess demand for services. Size is important because large TLPs evince high youth entry/exit rates and thus contribute significantly to the ability to recruit enough youth in the study to produce reliable outcome estimates. We estimate needing a sample of at least 1,250 to adequately power the impact analysis, which means that on average TLPs participating in the study would need to enroll 89 youth in the study. Most TLPs serve a relatively small number of youth. In FY2010, the average number of youth served in a TLP was 35, ranging from 10 to 187 youth during the year.

Related to the size of the agency is the program’s level of subscription—i.e., whether the agency has a waiting list from which youth can be enrolled into the study when a bed becomes available. The level of subscription is critical, because those that have the capacity to serve more youth than enroll in the program (i.e., under-subscribed programs) risk serving fewer youth than would have been the case without the random assignment study, because a portion of their applicants will be assigned to the control group. By contrast, those programs that are over-subscribed (that have more applicants than they can serve at any given time) are better candidates for this study, because the high demand-to-service-slot ratio facilitates random assignment. For example, assume that an agency with no waiting list has an available bed. An eligible youth approaches the agency, agrees to participate in the study, and is subsequently assigned randomly to the control group. The available bed will remain vacant until another eligible youth approaches the agency for services and is potentially assigned randomly to the treatment group. An empty bed is not ideal from a service provider perspective and could lead to frustration on the part of TLP staff. When an agency has a waiting list, the assignment of an applicant to the control group provides another eligible youth in the queue with the opportunity for enrollment into the study, possibly as a treatment group member. The likelihood that a service slot will remain empty diminishes as the size of the waiting list increases. For these reasons, over-subscription is an important inclusion criterion.

Finally, the control environment is important because impacts reflect the differences between the treatment and control conditions. Thus, the availability of other TLPs in the broader community must be considered. The aim of the selection process is to achieve a “high contrast” between the treatment and control conditions, meaning that the TLP services offered to the treatment group are distinct from those otherwise available to them in the community. A high treatment-control contrast implies a low likelihood of “cross-over,” which occurs when control group members enroll in a TLP or receive a combination of services similar to what they would have received had they been assigned to the treatment group. For these reason, the selection process will prioritize TLPs in environments that lack other TLPs or TLP-like programs. When multiple TLPs exist within a city or metropolitan area, we would opt to include all of them in the study so as to reduce the likelihood that control group members would enroll in an alternate TLP. Ideally, control environments would also be similar across sites, as this will assist in interpreting the results, but this is a lesser consideration than the strength of the treatment-control contrast

Across the 14 ultimately grantees included in the study, the intent is to randomly assign a total sample of 1,250 youth. Thus, the average agency will enroll 89 youth in the study over an estimated 18-month period, assigning 59 of them to the TLP treatment and the remaining 30 assigned to the control group. This sample will allow us to detect impacts of TLP on binary outcomes (e.g., stable housing) of between 5 and 10 percentage points. In Exhibit B.1.1, we present calculated Minimum Detectable Effects (MDEs) for this design. The exhibit shows MDEs of 0.20 standard deviations for continuous outcomes (e.g., delinquency score at18 months) and 5 - 10 percentage points for binary outcomes. These estimates assume that two-thirds of 1,250 enrolled youth are randomly assigned to the treatment group and one-third is assigned to a control group. The estimated MDEs also assume a response rate of 70 percent for the follow-up survey and a regression R-square of 0.04. This low R-square was selected to provide a conservative MDE estimate. Impact estimates with a higher R-square value would yield smaller MDEs. It is based on effects observed on homelessness in a study of housing vouchers on welfare families. For analyses involving the entire study sample,

Exhibit B.1.1inimum Detectable Effects when 33% of Youth are Assigned to the Control Gr

Number of Youth Enrolled in Study:	N = 1250
Continuous Outcomes ^a	0.20 (standard deviations)
Binary Outcomes ²
Control Mean of 10% (or 90%)	6 percentage points
Control Mean of 30% (or 70%)	9 percentage points
Control Mean of 50%	10 percentage points

Procedures for the Collection of Information

The evaluation will collect information on youth baseline characteristics and behaviors from approximately 1250 youth across 14 grantees. The research approach uses a series of web-based surveys to collect data from youth. A secure, encrypted, passcode protected website will serve as the portal for data collection and will allow research staff to monitor survey completion rates. The website will permit youth survey respondents to log-in using a unique username and password and complete their respective surveys online.

The baseline survey will be taken at TLP facilities. Prior to the baseline survey, trained TLP staff will obtain youth consent and administer the baseline survey, which involves seating each sample member at a computer (in the designated private space) and assisting them in registering and logging into the web portal in order to complete the survey. English and Spanish versions of the survey will be available, so the respondent can choose their preferred language. The respondent is then left to complete the survey in private. Once the sample member has completed the survey, the last screen will inform youth that they have completed the survey and ask them to confirm the method though which they would like to receive their incentive (an electronic gift card or gift card code, sent by email or text or mail if neither of those options is possible). The youth will leave the computer, real-time verification of completion will be recorded in the survey database, and the youth will receive their incentive.

For the follow-up surveys administration will be individual. The contracted research team will invite all youth enrolled in the study will be invited to complete the 3, 12, and 18 month follow up surveys as well as the 6, 9, and 15 month tracking surveys using agree-upon communication strategy (email, texting, etc.) as well as assistance from program staff, when possible, to remind them about the follow-up survey and provide instructions on how to access the Web survey and a PIN/password to enable access. Repeated reminders will be sent by electronic media until the survey has been accessed and completed.

The evaluation will also collect information from grantees during site visits. The procedures for this data collection involve in-person interviews with management-level and frontline staff at each grantee. Interviews will be conducted by trained members of the research team, using the Program Overview Survey – Executive Director Interview Guide, the Program Overview Survey – Program Staff Interview Guide, and the Youth Development Survey Interview Guide. A total of five staff members per grantee will be interviewed: One management level and four frontline staff.

Methods to Maximize Response Rates and Deal with Non Responses

Collecting data from homeless youth will be the greatest challenge of this study because many are expected to be transitory and lack fixed addresses. To obtain adequate response rates, we will implement a robust data monitoring and tracking process. (For information about projected response rate and statistical power, see “Sample Size and Statistical Power” below.) The study team will employ several outreach tactics to obtain the highest response rates possible from study participants for each of the surveys. The primary mode of outreach will be email or cell phone text message. (Note that upon enrollment into the study, youth will have an opportunity to refuse text messaging from the study team if this approach forces youth to incur additional costs. Some cell phone data plans have unlimited text messaging, while others have an additional charge.)

• Survey Invitations and Reminders: Abt SRBI (via their call center) will send email or text message invitations for the 3-, 12-, and 18-month surveys. The invitations will include a link to the study’s web portal where youth will login using their unique username and password (set up during enrollment). Up to two email/text reminders will be sent to non-responders before moving them to phone contact. Those who do not complete one of those surveys within 48 hours of the email/text invitation will receive up to five telephone calls. The telephone calls will prompt youth to go to the study web portal and complete the survey. Staff will use secondary and tertiary contact information obtained on the baseline and tracking surveys to contact individuals who may know of the youth's whereabouts or have updated contact information for them. If the second telephone reminder does not result in a completed interview, Abt SRBI will attempt to complete the survey by phone with each youth.

• Tracking Surveys: Abt SRBI (via their call center) will send email and cell-phone text messages to youth in-between the survey deployments (i.e., at months 6 and 9) to invite youth to confirm or update their contact information. The invitations will include a link to the study’s web portal where youth will login using their unique username and password (set up during enrollment). Up to 2 reminder emails and 2 text reminders will be sent to non-responders over the course of 4 weeks after the initial invitation is sent. Note that invitations to complete the 6 and 9 month tracking surveys will not be followed by phone calls to encourage completion or obtain data.

• Private Messages via Social Media: Abt or Abt SRBI may send private messages that only the youth can see through social networking sites such as Twitter and Facebook (if the youth opt to provide these points of contact). Private messages are not posted publically on the external-facing portion of these websites. Such messages would be sent out at the 6- and 9 -month tracking milestones to update youth’s contact information with a link to the study’s web portal and a reminder of the incentive payment. Youth can update contact information by logging into the web-based survey portal using their username and password. Similar messages would also be sent to youth at the 3-, 12-, and 18-month survey milestones with a link to the study’s web portal and a reminder of the incentive payment. Because Facebook routinely changes its privacy policies and the software that enforces them, we plan to have a research team member regularly monitor Facebook’s and Twitter’s privacy and security policies and upgrades to protect against the unintended exposure of participant information to others. This is a requirement of the study’s Institutional Review Board.

• Monitoring: Monitor data collection and produce bi-weekly reports on the status (e.g., response rates) of each participating TLP agency.

• Incentives: Distribute youth incentives as surveys are completed.

Recent research suggests that homeless and runaway youth are connected to the Internet¹ —that is, youth access the internet frequently and use social networking sites. Thus, social networking presents new opportunities to maintain contact with study participants and remind them to complete the study surveys. Social networking opportunities are a low-cost approach to tracking youth. Under this approach, we will create a Twitter and Facebook profile using nondescript/neutral profile names for the study and then “follow” (on Twitter) or “friend” (on Facebook) study participants. We will activate privacy and confidentiality settings within these social networking sites to ensure that study participants cannot see the profiles of other study participants, thereby protecting the confidentiality of each participant. We will also be required to develop additional protocols that govern how researchers use Twitter and Facebook, how to limit exposure to other information posted on each participant’s social webpages (e.g., Facebook’s “wall”), and how and when communications with youth are permissible. We have had preliminary discussions with members of Abt’s IRB to gauge the viability of this approach, and the response has been encouraging. We will work closely with the IRB to develop these social media protocols and plan to submit them in a modification to the IRB package at a later date.

Test of Procedures or Methods to be Undertaken

Early versions of the youth surveys was subjected to a pretest involving fewer than 10 individuals. This pretested occurred during visits to three agencies, during which time youth in three TLPs, the same group to be measured, were asked to take and review the surveys. Since that time, the surveys have been significantly modified to include additional outcome measures and a revised design strategy that includes random assignment and measurement of service dosage. The modified surveys were pretested with three junior members of the contractor staff for the purposes of timing survey administration under differing response scenarios. The surveys rely heavily on questions that have been validated and used in many other national studies, especially the questions associated with the study’s key outcome domains—e.g., homelessness, psychosocial wellbeing, and employment and education. As such, the study team and HHS are confident that the survey questions are worded properly and thus there are no plans to cognitively test the surveys with a sample of youth. However, HHS and the study team will monitor survey completion rates throughout the study to assess whether study participants are completing the survey. If survey completion rates are low, the study team will engage grantees to understand if study participants are having difficulties with the survey questions.

Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Administration of the data collection will be overseen by Abt Associates (statistical and research contractor) and its subcontractor, Abt SRBI. The same contractor will analyze data with support from evaluation colleagues at Wayne State University. Members of this research team include:

Alvaro Cortes

Abt Associates