Transitional Living Program Evaluation
OMB Information Collection Request
0970 - 0383
Supporting Statement
Part B
July 2018
Submitted By:
Office of Planning, Research, and Evaluation
Administration for Children and Families
U.S. Department of Health and Human Services
4th Floor, Mary E. Switzer Building
330 C Street, SW
Washington, D.C. 20201
Project Officers:
Caryn Blitz, Office of Planning, Research, and Evaluation
Angie Webley, Family Youth Services Bureau
Table of Contents
B1. Respondent Universe and Sampling Methods 3
B2. Procedures for Collection of Information 3
B3. Methods to Maximize Response Rates and Deal with Nonresponse 5
The sampling frame includes all 108 TLPs funded in September 2017. From among these, 50 TLPs will be selected to participate in the study. The selection process is purposive and based on the following criteria: 1) number of expected entries, which estimates the TLP service volume over a 12-month period; 2) TLP grant type (Maternity Group Home programs will be excluded from the study because they serve only pregnant and parenting youth and offer a unique set of services to address this subpopulation’s special needs); and 3) length of time providing TLP services (new TLPs with little prior program experience will be excluded from the study).
ACF has provided the research team with a complete list of the 2017 TLP grantees along with information about their service volume. To identify an initial set of 50 grantees, the research team has rank-ordered TLPs according to their projected service volume to identify those with a high service volume. The contractor is currently in the process of screening the candidate TLPs to identify those to include in the Youth Outcomes Study.
Across the 50 grantees ultimately included in the study, the intent is to achieve a total sample size of 600 youth.1 Thus, the average agency will enroll 12 youth in the study over an estimated 24-month period. At the participant level, study eligibility requirements mirror program eligibility and enrollment requirements. Thus, the study will include youth who are eligible to enter the TLP and enroll in the program. Study participation is voluntary, and youth who enroll in the TLP will elect to participate in the study (or not) through an informed consent process.
The evaluation will collect information on youth baseline characteristics and behaviors from approximately 600 youth across 50 grantees. The research approach uses a series of web-based surveys to collect data from youth. A secure, encrypted, passcode protected website will serve as the portal for data collection and will allow research staff to monitor survey completion rates. The website will permit youth survey respondents to log in using a unique username and password and complete their respective surveys online.
Before data collection begins, trained TLP staff will obtain youth consent and, as needed for minors, assent with parental consent (Attachments A and B). Then they will administer the baseline survey at the TLP, which will involve seating each youth respondent at a computer (in the designated private space) and assisting them in registering and logging into the web portal in order to complete the survey. English and Spanish versions of the survey will be available so that respondents can choose their preferred language. The youth respondent will be left to complete the survey in private. The final screen of the survey will inform youth they have completed the survey and ask them to confirm the method by which they would like to receive their incentive (an electronic gift card or gift card code, sent by email or text, or shipped if neither of those options is possible). The youth will then exit the survey, real-time verification of survey completion will be automatically recorded in the survey database, and afterward the incentive will be processed and delivered to the youth.
The follow-up surveys will be self-administered. The contracted research team will invite all youth enrolled in the study to complete the 6- and 12-month follow-up surveys using the communication strategy previously indicated by the youth (email, texting, etc.). Youth respondents will also receive support from program staff, as needed, to remind them about the follow-up survey, provide instructions and assistance with accessing the survey, assistance resetting their password, or use of a computer with Internet access. Repeated reminders will be sent electronically until the survey has been accessed and completed, with telephone and in-person outreach from TLP staff and outreach to youth and, if permitted by youth, to their secondary or tertiary contacts (see Attachment H).
Analytic Methods
The goal of the outcome analysis is to estimate the change in youth outcomes associated with participating in the TLP. Changes in the outcomes of interest for each youth i after participating in a TLP will be estimated at 6 months after enrollment and at 12 months after enrollment. As shown in Equation 1, the change at 6 months (ΔE6moi) is calculated by subtracting the score on a given outcome (e.g., risky behavior) for youth i prior to TLP participation (Ybi) from the score for that outcome 6 months after enrollment (Y6moi). As shown in Equation 2, the change at 12 months (ΔE12moi) will be calculated using the same method—that is, by subtracting the score on a given outcome for youth i prior to TLP participation (Ybi) from the score for that outcome 12 months after enrollment (Y12moi).
(1) ΔE6moi = Y6moi - Ybi
(2) ΔE12moi = Y12moi - Ybi
The statistical model to estimate the average baseline to follow-up change at 6 or 12 months after program enrollment on an outcome Y (e.g., risky behavior) is presented in equation (3):
(3) ΔYi = β̂0 + ϵi
where:
ΔYi is the posttest outcome of interest for youth i measured at 6 months or 12 months;
β0 is the average baseline to follow-up change 6 or 12 months after program enrollment; and
ϵi is a random error term measured with mean of 0 and variance σ2.
To determine if TLP participation has a statistically significant association with changes in youth outcomes, we will conduct a t-test for each outcome measure. If the estimate of β0 is statistically significant at the 5-percent level using a two-tailed test, we will conclude that we have found scientific evidence that TLP participation is related to change in the outcome. Otherwise, we will conclude that there is no scientific evidence that TLP participation is related to a change in this outcome.
For continuous or categorical outcomes, we plan to estimate the model above using ordinary least squares (OLS), which assumes that the outcome data have a normal distribution (i.e., form a bell-shaped curve) with homoscedasticity (e.g., a common variance). For binary (dichotomous) outcomes, models will be estimated using logistic regression and we will report the marginal effect on the probability of observing the binary outcome.
We have no reason a priori to expect homoscedasticity, because some TLPs could have higher variability in youth outcomes than other TLPs (Angrist & Pischke, 2008). To address the potential of heteroscedasticity and account for variation in continuous and categorical youth outcomes across TLPs, we will include site-level indicator variables (“fixed effects”) as covariates in our linear models, and we will compute robust standard errors (i.e., Huber-Eicker-White robust standard errors; Huber, 1967; Greene, 2003; White 1980, 1984).
Degree of Accuracy Required
Across the ultimately 50 grantees included in the study, the intent is to achieve a total sample size of 600 youth. This sample will allow us to detect effects of TLP on binary outcomes (e.g., stable housing) of between 5 and 10 percentage points. In Exhibit B.1.1, we present calculated Minimum Detectable Effects (MDEs) for this design at 6 months and at 12 months following program enrollment. At 6 months, the exhibit shows MDEs of 0.167 standard deviations for continuous outcomes (e.g., delinquency score after 6 months) and 5 to 8 percentage points for binary outcomes. At 12 months, the exhibit shows MDEs of 0.205 standard deviations for continuous outcomes and 6 to 10 percentage points for binary outcomes. The estimated MDEs assume a response rate of 75 percent for the 6-month follow-up survey and 50 percent for the 12-month follow-up survey. Both the 6-month and 12-month follow-up surveys assume a pre-post correlation of 0.2. This low pretest-posttest correlation was selected to provide a conservative MDE estimate. Outcome estimates with a higher R-square value would yield smaller MDEs.
Exhibit B.1.1: Calculated Minimum Detectable Effects (MDEs)
|
6 months after enrollment |
12 months after enrollment |
Number of Youth in Analytic Sample: |
N = 450 |
N = 300 |
Continuous Outcomes a |
0.167 (standard deviations) |
0.205 |
Binary Outcomes 2 |
|
|
Control Mean of 10% (or 90%) |
5.0 percentage points |
6.2 |
Control Mean of 30% (or 70%) |
7.7 percentage points |
9.4 |
Control Mean of 50% |
8.4 percentage points |
10.3 |
Expected Response Rates
We anticipate a 75 percent response rate at the 6-month follow up and a 50 percent response rate at the 12-month follow up. This rate is based on the mobility of the target population, our experiences locating and contacting youth for follow-up data collection during the TLP random assignment pilot study, and conversations with TLP grantees about their success rate in reaching youth who have exited the program. We expect the proposed incentive to assist with reaching these anticipated response rates and would expect a lower response rate in the absence of the proposed incentive.
Dealing with Nonresponse
Missing data are of concern in any study, particularly one with a longitudinal design such as this. The presence of non-response to the follow-up survey can threaten the generalizability of results to the entire enrolled sample if the subsequent outcomes of survey non-respondents differ from respondents. Without the modest incentives OMB has previously approved, the evaluation risks having unacceptably low response rates for the 6- and 12-month follow-up surveys which would lead to insufficient statistical power (due to a small analytic sample size) and nonresponse bias (i.e., when survey respondents differ in meaningful ways from non-respondents. Response rates for undereducated, minority, and high-risk youth, which make up the majority of the homeless youth population served by TLPs, have been shown to increase with the use of the types of incentives in the model we propose.2 This incentives model is designed to increase the survey response rates, with specific attention to youth that will be dispersing widely into the general population as they gain independence, which is the purpose of TLP. Research suggests that providing an incentive for earlier surveys may contribute to higher response rates for subsequent surveys.3
The research team will take steps to understand the quantity and quality of any non-response and to describe and address the threat that it may pose to the validity of findings from the evaluation. Using data from the baseline survey, research team members will test for differences between respondents and non-respondents on key demographic and outcome variables. Any statistically significant differences will be documented in the findings report. Should non-response bias be evident, the research team will prepare a set of weights that adjusts for non-response in the 6-month follow-up survey and a separate set of weights that adjusts for non-response in the 12-month follow-up survey. The weights will be used in all regression models.
Maximizing Response Rates
Collecting longitudinal data from runaway and homeless youth is a challenge because many are transitory and lack fixed addresses. To obtain adequate response rates, we will implement a robust data monitoring and tracking process. The study team will employ several outreach tactics to obtain the highest response rates possible from study participants for each of the surveys. The primary mode of outreach will be email or cell phone text message. (Note that upon enrollment into the study, youth will have an opportunity to refuse text messaging from the study team if this approach forces youth to incur additional costs. Some cell phone data plans have unlimited text messaging, while others have an additional charge.)
In addition, the study team will send email or text message invitations for the 6- and 12-month surveys (see Attachment H). The invitations will include a link to the study’s web portal where youth will login using their unique username and password (set up during enrollment). Up to two email/text reminders will be sent to non-responders before moving them to outreach by TLP staff. For those who do not complete the surveys within 48 hours of the second email/text invitation, specially trained TLP staff will reach out to the youth via telephone or in person for the completion of the survey using a study-provided laptop/tablet with an internet connection (Attachment H). Alternatively, TLP staff may offer to help troubleshoot any issues the youth may have connecting to the survey (e.g., password reset), logging into the study web portal, or completing the survey. Staff will use secondary and tertiary contact information obtained from the most recent completed survey to contact individuals who may know of the youth's whereabouts or have updated contact information for them.
The study team will actively monitor data collection and produce bi-weekly reports on the status (e.g., response rates) of each participating TLP agency.
ACF is not proposing any substantive changes to the previously approved data collection instruments or procedures as part of this request. ACF has not conducted any additional testing of the data collection procedures or methods since receiving initial OMB approval.
Consultations on the statistical methods used in this study have been undertaken to ensure the technical soundness of the research. Administration of the data collection will be overseen by Abt Associates (statistical and research contractor). The same contractor will analyze data. Members of this research team include:
Dr. Alvaro Cortes
Abt Associates
6130 Executive Boulevard
Rockville, MD 20852
(301) 634-1857
Dr. Jill Khadduri
Abt Associates
6130 Executive Boulevard
Rockville, MD 20852
(301) 634-1745
Mr. Cristofer Price
Abt Associates
6130 Executive Boulevard
Rockville, MD 20852
(301) 634-1854
Ms. Alisa Santucci
Abt Associates
6130 Executive Boulevard
Rockville, MD 20852
(301) 634-1854
Dr. Jessica Thornton Walker
Abt Associates
6130 Executive Boulevard
Rockville, MD 20852
(301) 347-5622
Angrist, J.D., & Pischke J.S. (2008), Mostly harmless econometrics: An empiricist's companion. Princeton, NJ: Princeton University Press.
Greene, W. H. (2003). Econometric analysis, 5th edition. Upper Saddle River, NJ: Prentice-Hall Inc.
Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability. Berkeley, CA: University of California Press, 1, 221–233.
White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817–830.
White, H. (1984). Asymptotic Theory for Econometricians. Orlando, FL: Academic Press.
1 Most TLPs serve a relatively small number of youth. Among the 108 TLPs funded in September 2017, the average number of youth served annually was about 10, ranging from 1 to 54 youth annually.
2 Berlin, Martha, Leyla Mohadjer, Joseph Waksberg, Andrew Kolstad, Irwin Kirsch, D. Rock, and Kentaro Yamamoto. 1992. An experiment in monetary incentives. In JSM proceedings, 393–98. Alexandria, VA: American Statistical Association.
3 Singer, Eleanor, John Van Hoewyk, and Mary P. Maher. 1998. Does the payment of incentives create expectation effects? Public Opinion Quarterly 62:152–64.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | OPRE OMB Clearance Manual |
| Author | DHHS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-20 |