SSB_TLPEval_Nonsubstantive Change_clean_07132020_for OMB

Transitional Living Program Evaluation

OMB Information Collection Request

0970 - 0383

Supporting Statement

Part B

Updated July 2020

Submitted By:

Office of Planning, Research, and Evaluation

Administration for Children and Families

U.S. Department of Health and Human Services

4^th Floor, Mary E. Switzer Building

330 C Street, SW

Washington, D.C. 20201

Project Officers:

Caryn Blitz, Office of Planning, Research, and Evaluation

Christopher Holloway, Family Youth Services Bureau

Table of Contents

Introduction 3

B1. Respondent Universe and Sampling Methods 3

B2. Procedures for Collection of Information 3

B3. Methods to Maximize Response Rates and Deal with Nonresponse 11

B4. Tests of Procedures or Methods to be Undertaken 11

B5. Individual(s) Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 12

References 13

Introduction

B1. Respondent Universe and Sampling Methods

The sampling frame includes all TLPs funded in September 2017 and September 2018. From among these, 30 TLPs were recruited to participate in the study. The selection process was based on the following criteria: 1) number of expected entries, which estimates the TLP service volume over a 12-month period; 2) program structure (excluded TLPs that operate in collaboration with sub-grantees or across several locations); 3) length of time providing TLP services (new TLPs with little prior program experience were excluded from the study) and 4) program model (excluded TLPs that did not provide youth with fully or nearly-fully subsidized transitional housing).

ACF provided the research team with a complete list of the 2017 and 2018 TLP grantees along with historical information about their service volume. To identify an initial set of grantees for recruitment, the research team rank-ordered TLPs according to their projected service volume to identify those with a high service volume. The contractor screened the candidate TLPs to identify those to include in the Youth Outcomes Study.

Across the 30 grantees ultimately included in the study, the intent is to achieve a total sample size of up to 400 youth.¹ Thus, the average agency will enroll 10-15 youth in the study over an estimated 4-month period. To be eligible for the study, youth may be entering the TLP (prospective enrollment) or may have already entered within the past 12 months and have at least 1 month of program participation (retrospective enrollment). Otherwise, study eligibility requirements mirror program eligibility and enrollment requirements. Study participation is voluntary, and youth enrolled in the TLP will elect to participate in the study (or not) through an informed consent process.

B2. Procedures for Collection of Information

The study will collect information on youth characteristics and experiences from approximately 400 youth across 30 grantees. The research approach uses a Background Information Form (BIF) and a Youth Information Form (YIF) (Attachments C and D) to collect data directly from youth (via the BIF) and from TLP programs (via the YIF). Both the BIF and the YIF will be administered via a secure, encrypted, passcode protected website that is built to comply with FISMA moderate standards. This website will serve as the study’s data collection portal (“study portal”), and it will allow research staff to monitor study enrollments and BIF and YIF completion rates.

Before data collection begins, trained TLP staff will obtain youth consent and, as needed for minors, assent with parental permission (Attachments A and B). They will then administer the BIF, which will involve seating each youth respondent at a computer (in a designated private space) and assisting them in completing the BIF via the web portal. The youth respondent can ask for assistance from the TLP staff to complete the BIF if they wish. Once the youth completes and submits the BIF, they will be automatically signed out of the study portal for added security. BIF data will be collected prospectively as participants enroll in the study.

The YIF draws on existing program data collected by TLP grantees. TLP staff will be asked to transfer this program data to the evaluation team via the same secure web-based study portal used to enroll youth into the study and collect the BIF. YIF data may be entered into the study portal either retrospectively or prospectively, depending on the timing of a youth's enrollment into the TLP and the study (see B1).

Analytic Methods

The study analyses are aligned with five overarching research questions:

1. What are the prevalence and distribution of housing stability, education, and employment outcomes among TLP youth before, during, and after program participation (as examined using means and frequencies)?

2. What are the trajectories of employment and earnings among TLP youth before, during, and after program participation (as examined using latent growth curve analysis)?

3. What is the extent to which youth sustain employment (overall and with the same employer) and enrollment in post-secondary education (as examined using survival analysis)?

4. What are the differences in primary study outcomes for key subgroups of youth based on demographic and program characteristics (as examined using cross-tabs and graphical display of trends)?

5. In what ways are TLP youth's housing, education, and employment experiences affected by the COVID-19 pandemic?

The analyses will rely on data collected in the BIF and YIF as well as on administrative data obtained from the National Student Clearinghouse (NSC) and the National Directory of New Hires (NDNH). The data we will collect features repeated retrospective observations of employment and earnings that look back over a period of up to 11 quarters (33 months) and repeated observations of education that look back over a period of up to 48 months. Our analyses are designed to capitalize on the data’s longitudinal nature, as described below.

In Exhibit B.2.1, we present the main outcomes examined, their sources, the analyses in which they will be used and the research question with which they are aligned. In addition, we will use demographic measures (youth age, gender, race/ethnicity) and whether the youth was referred to the TLP from a Basic Center Program, obtained at study enrollment through the BIF for cross-tabulations in descriptive analyses and as covariates in survival and latent growth curve analyses.

Exhibit B.2.1. Description of primary study outcomes, data sources, and analyses

Outcome	Source	Number of times assessed†	Descriptive analysis	Growth curve analysis	Survival analysis
Employment: Formally employed	NDNH administrative data	Up to 11 quarters (33 months) of data collected retrospectively via up to 4 data pulls from NDNH over the study period	RQ1, RQ4, RQ5	RQ2, RQ4, RQ5
Employment: Maintained formal employment	NDNH administrative data	Up to 11 quarters (33 months) of data collected retrospectively via up to 4 data pulls from NDNH over the study period	RQ1, RQ4, RQ5		RQ3, RQ4, RQ5
Earnings: Average earnings from formal employment	NDNH administrative data	Up to 11 quarters (33 months) of data collected retrospectively via up to 4 data pulls from NDNH over the study period	RQ1, RQ4, RQ5	RQ2, RQ4, RQ5
Education: Enrolled in post-secondary education	NSC administrative data	Up to 48 months of data collected retrospectively via up to 2 data pulls from NSC over the study period	RQ1, RQ4, RQ5	RQ2, RQ4, RQ5
Education: Maintained post-secondary enrollment	NSC administrative data	Up to 48 months of data collected retrospectively via up to 2 data pulls from NSC over the study period	RQ1, RQ4, RQ5		RQ3, RQ4, RQ5
Housing: Homeless or unstable housing status	YIF	Program entry, program exit, and up to 5 encounters post-exit (for a total of up to 7 encounters) collected retrospectively or prospectively from TLPs over the study period	RQ1, RQ4, RQ5

Notes: NDNH = National Directory of New Hires, NSC = National Student Clearinghouse, RQ = Research question, YIF = Youth Information Form.

^† Follow-up administrative data collection will occur over a period of up to 8 months for the YIF and 10 months for NDNH and NSC data. The “number of times assessed” differs from the length of time the contractor will be collecting this data due to the retrospective nature of this administrative data collection. NDNH data will be requested once per quarter up to 4 times. Each data request provides employment and wage data for the prior eight quarters. NSC data will be requested dating back to August 2017.

In the first set of analyses addressing research questions one, four and five, we will use descriptive statistics to report overall rates of homelessness or unstable housing arrangements, education, employment, and earnings at program entry, program exit, and 3 months after exit. To address our fourth research question (about subgroup differences in outcomes), these results will be cross-tabulated by youth age (under 18 versus 18 and over), gender, race/ethnicity, and housing status prior to program entry for descriptive purposes.

In the second set of analyses, addressing research questions two, four and five, we will describe trajectories in average formal employment rates, earnings from formal employment, and post-secondary enrollment rates for the study sample of TLP enrollees over the full study observation period using administrative data. We will describe trajectories from repeated measures of these outcomes using latent growth curve (LGC) analysis. We will explore what shape of trajectory (functional form of change) best fits the data and conduct statistical tests of whether the shape or rate of change differs in the periods prior to, during, and after participation in a TLP.

The formal matrix expression of a latent curve model without predictors (unconditional model) is

y = Λη + ε

where:

• y is a T x 1 vector of repeated measures for each individual i and T is the number of time points;

• Λ is a T x m matrix of factor loadings (that specify the functional form of time);

• η is a m x 1 vector of m latent factors (representing the growth parameter estimates); and

• ε is a T x 1 vector of residuals (difference between individual’s predicted and actual score).

To identify the best-fitting trajectory shape, we will use a base model specifying simple linear growth in each time period and then test alternative models specifying more complex trajectory shapes to see whether they fit the data better than the simpler base model. Our base model will be a piecewise linear model with separate linear growth parameters for observations before (pre), during, and after (post) program participation, assuming a constant rate of change in each of these three time periods, but allowing this rate to differ in each period. We will then test whether assuming change is quadratic (where the rate of change varies, depicted by a curved line) in each period improves model fit. Maximum likelihood estimation is used to identify the parameter estimates that best fit the data.

After determining the best-fitting trajectory shape within each time period, we will test whether assuming that the rate of change between each time period is the same results in worse model fit. For example, if there is little difference in the sample’s earnings trajectory during and after program participation, assuming that a single line fits best would not result in significantly worse model fit. However, if employment rates sharply decline after program exit, assuming the trajectory is the same in both time periods would result in model fit statistics indicating worse fit.

We will use an individually varying specification of time, where youth contribute all observations they have during each time period to estimating the trajectory for each time period. That is, a youth who remains in a TLP for two quarters and then has four quarters of earnings data after program exit would contribute two quarters of earnings data for estimating the trajectory of youth during program participation and four quarters of earnings data for estimating the trajectory of youth after program exit. Under this specification of time, results would indicate the average change in the outcome per quarter prior to, during, and after program participation. Graphical depictions of these growth curves will be constructed based on the average program length of stay and average length of time youth were observed prior to and after program participation.

We will use statistical fit indices to assess how well each model fits the data in terms of both overall fit to the data and relative fit between competing models. The Root-mean Square Error of Approximation (RMSEA) will be used to assess both overall and relative model fit based on point estimates and confidence intervals. RMSEA values of less than .05 indicate a very good overall fit, .05 to less than .10 moderate fit, and greater than .10 a poor fit, based on simulation studies (Browne & Cudeck, 1993; Steiger, 1989). RMSEA also can be used to assess whether an alternative model fits better or more poorly than a base model by examining whether the point estimate of the base model is contained within the 95% confidence interval of the alternative model. For example, if RMSEA is .05 for a model where change during program participation was linear, and RMSEA is .02 in an alternative model where change during program participation is quadratic with a 95% confidence interval of .01 to .03, we would conclude an assumption of quadratic change during program participation fits the data better than linear change.

With a sample size of 400, we would be able to detect a difference between perfect overall model fit (RMSEA = 0) and close model fit (RMSEA = .05) 97.7% of the time for our base model. We would need a minimum sample size of 245 to be able to detect a difference in fit of this size 80% of the time (the conventional standard for adequate statistical power). We also expect to have high power to detect differences in relative model fit for our key tests of whether the rate of change across two time periods is significantly different. We would be able to detect a difference in RMSEA of .01 or more (e.g., RMSEA = 0.04 vs. 0.05) 96% of the time between our base model where the rate of change differs in each time period and one where the rate of change is assumed to be the same in two time periods. We would need a minimum sample size of 228 to be able to detect a difference of this size 80% of the time.

We will identify the best fitting model for describing the trajectory of youths’ outcomes, accounting for their experiences before, during, and after program participation, and evidence of whether youth trajectories during program participation appear to differ from those prior to program entry or after program exit.

As part of our fourth research question, after fitting latent growth models without covariates, we will run and graphically depict exploratory analyses with dummy variable covariates for youth age (18 and over), gender (female), minority race/ethnicity status (minority), and referral from a basic center program (BCP) to examine whether growth trends appear to differ by these characteristics. For example, a positive coefficient for a dummy variable for female youth in predicting intercept term would indicate females entered TLPs at a higher employment rate than males and a positive value in predicting the slope during TLP participation would indicate female employment rates rose faster than male employment rates during TLP participation.

We anticipate minimal missing data because we are collecting outcomes data from administrative sources (i.e., national administrative data clearinghouses for education and employment and program administrative records (required by funders) for housing outcomes). We will use direct maximum likelihood estimation (also known as full information maximum likelihood estimation) to address any missing data. This method assumes that missing data are unrelated to the outcome after accounting for any covariates included in the model. For example, it may be that child age is predictive of missing data on employment, but after accounting for children’s age, there are no unobserved variables that are predictive of whether an observation is missing or not.

We will also run supplementary analyses examining whether average employment rates, earnings, and enrollment rates prior to the program and after the program differ from those observed during the program. We will use a mean differences model specified as follows:

Where

• is the observed outcome measure for youth j at time i;

• is a dummy variable set to 1 when time i is prior to program entry and to 0 when time i is on or after program entry;

• is a dummy variable set to 1 when time i is on or after program exit and to 0 when time i is before program exit;

• is the intercept term (representing the mean in the period during the program for the omitted person j;

• is the mean difference in the outcome for all observations prior to program entry (indicated by the dummy variable = 1) compared to the mean of observations during program participation (where = 0 and = 0);

• is the mean difference in the outcome for all observations after program exit (indicated by the dummy variable = 1) compared to observations during program participation (where = 0 and = 0);

• is a series of dummy variables for each youth up to the j – 1 youth;

• is an error term for person j at time i.

It is possible that observations that are closer together in time are more strongly correlated than observations that are farther apart (i.e., autocorrelation), so we will compute robust standard error estimates (using Huber-White sandwich estimators), which will ensure correct estimation of standard errors even if auto-correlation or other sources of heteroscedasticity are present (Huber, 1967; White, 1980, 1984).

The focus of this analysis is to provide a supplementary analysis to the growth curve analysis, not on testing the statistical significance of mean differences. Nonetheless, based on a sample size of 400 and a critical value of .05, we would expect to be able to detect an effect size between .09 and .18 for the difference between pre-TLP and during TLP outcomes with 80% power, depending on the correlation between the pre and during period (range based on estimated correlations of .80 and .20, respectively, with similar power for during versus post-TLP). This translates to a minimum detectible difference of approximately $450 to $900 in quarterly earnings, 4 to 8 percentage points for educational enrollment, and 3.5 to 7 percentage points for employment based on program exit findings from the Youth Villages Evaluation (Valentine, Skemer, & Courtney, 2015).

In the third set of analyses, addressing research questions three, four and five, we will use survival analyses to describe the extent to which youth who are employed or enrolled in post-secondary education continuously maintain their employment or enrollment status. Survival analyses properly account for youth who experienced an event (i.e., loss of initial employment after program enrollment) in an earlier time period no longer being at risk for experiencing this outcome. It also accounts for youth not having yet experienced the event by the end of the study observation period still being at risk for experiencing it in the future (termed right censoring). Results will be presented both numerically in terms of the expected number of events per unit of time (the event hazard) and graphically as survival curves.

With educational and employment outcomes being measured in quarters and semesters, respectively (as this is how data is reported back from NSC and NDNH), it is likely that there will be a high proportion of events being recorded as occurring at the same time (termed ties). Standard Cox regression models assume time is continuously measured, rather than being measured in quarters or semesters. When a survival analysis finds that outcomes frequently occur at the same time, statistical adjustments need to be made to account for ties.

Exact methods for ties assume there is a true, but unknown ordering of events recorded as occurring at the same time. This assumption fits well for employment outcomes, where there is an exact ordering of the time of a job loss and ties are largely a result of continuous employment time being aggregated into quarters in NDNH administrative data. Discrete methods assume events are actually occurring at the same time. This assumption fits well for enrollment outcomes, where the outcome of interest is enrollment status for a whole semester as a discrete unit of time, even if non-enrollment or drop-out may occur at varying times.

For both employment and post-secondary enrollment outcomes, we will use a discrete time logistic regression model where each unit of time is an observation. For post-secondary enrollment outcomes, a logit link function will be included to provide estimates of the discrete time proportional odds of maintaining enrollment in each semester. This model takes the form of

where

• P_it is the conditional probability that youth i is no longer enrolled at time t given that disenrollment has not already occurred to that youth as of time t.

• α_t is a series of constants, one for each time point.

• is a covariate value of covariate k for youth i at time t.

• is the coefficient estimate for the k^th covariate.

For employment outcomes, a complementary log-log link function will be included to provide estimates of the hazard of employment retention based on an assumption of continuous time. This model estimates the same parameters as described above and takes the form of

To answer our fourth research question (about subgroup differences in outcomes), for both sets of outcomes, we will first run models without covariates to understand retention (of employment or post-secondary enrollment) in the overall sample and then run analyses including youth characteristics as covariates to understand associations between these characteristics and retention. We will graphically depict survival curves for each subgroup.

To answer our fifth research question (about COVID-19 experiences), we will use descriptive statistics, and we will add a time-varying covariate for the onset of COVID-19 to the models described above (second and third sets of analyses). We will first use frequencies to describe youth’s self-reported perceptions of COVID-19's influences on their housing, education, and employment.

Second, we will use frequencies to describe TLP-staff-reported changes in youth's housing, education, and employment goals in their case plans. This will enable us to understand the ways programs may have adjusted (e.g., de-emphasized or more strongly emphasized) education or employment expectations for youth who entered TLPs prior to and after the onset of COVID-19. Adjustments in program expectations may influence education and employment trajectories. Thus, this information can help contextualize study findings about change in these trajectories observed after TLP entry.

Third, we will assess the extent to which the onset of COVID-19 is associated with differences in youth educational and employment trajectories and retention by adding a time-varying covariate for the onset of COVID-19 to the models described above (second and third sets of analyses). The U.S. declared a state of emergency due to the COVID-19 pandemic on March 1, 2020, and nearly all programs (97 percent) are in states that implemented stay-at-home orders by April 1, 2020. Our minimum time units of analysis for trajectory outcomes are quarters for employment data and semesters for education data. We will code the time-varying variable as time-weighted exposure (meaning the COVID-19 pandemic is active in the U.S.), with a value of 0 for units of time prior to March 1, 2020 that do not contain March 1, 2020 (i.e., prior to Q1 2020), a value of 1 for time periods after March 1, 2020 that do not contain this month (i.e., Q3 2020), and the proportion of time in days on or after March 1, 2020 included in the unit of time for periods of time that include March 2020 (i.e., for Q1 2020, 31 ÷ 91 days = 0.34). This covariate can be interpreted as the difference in the average rate of change (slope) or in the survival hazard during time periods of exposure to COVID-19 compared to time periods without exposure to COVID-19. (If the COVID-19 pandemic is deemed to have ended in the U.S. by the end of study data collection, the same approach could be used to weight time periods of partial or no exposure based on an established end date.)

B3. Methods to Maximize Response Rates and Deal with Nonresponse

Expected Response Rates

All individuals who agree to participate in the evaluation will be asked to complete the BIF to participate in the study. However, it is possible that some youth respondents will not fully complete the BIF. Because of the relatively short burden to complete the BIF, we anticipate these occurrences will be rare. Participants will also be provided with a $10 gift card as a token of appreciation for completing the BIF and to acknowledge their essential contribution to the study. We therefore anticipate a 95 percent response rate for youth who complete the consent form.

The data collected via the YIF is standard program data that TLP programs collect. We anticipate TLP staff will update the information in the YIF on a rolling basis as youth enter and exit the TLP and complete aftercare services. However, it is possible that some youth will fail to provide the TLPs with this program data. A 95 percent response rate is also expected for the YIF.

Dealing with Nonresponse

The presence of non-response to select questions in the BIF can threaten the ability to conduct administrative data matching. To minimize this risk, we will also ask TLP program staff to verify each youth’s SSN as part of the administrative data collection effort built into the Youth Outcomes Study design.

Maximizing Response Rates

Collecting longitudinal data from runaway and homeless youth is a challenge because many are transitory and lack fixed addresses. To obtain adequate data, the Youth Outcomes Study will shift to an approach that relies largely on administrative data. This shift means that the evaluation will no longer be dependent on follow-up survey responses from youth, thus minimizing the risk of non-response.

B4. Tests of Procedures or Methods to be Undertaken

ACF is not proposing any substantive changes to the previously approved questions that remain in the BIF as part of this request. ACF has not conducted any additional testing of the data collection procedures or methods since receiving initial OMB approval.

B5. Individual(s) Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Consultations on the statistical methods used in this study have been undertaken to ensure the technical soundness of the research. Administration of the data collection will be overseen by Abt Associates (statistical and research contractor). The same contractor will analyze data. Members of this research team include:

Ms. Alisa Santucci

Abt Associates

6130 Executive Boulevard

Rockville, MD 20852

(301) 347-5376

Dr. Jill Khadduri

Abt Associates

6130 Executive Boulevard

Rockville, MD 20852

(301) 634-1745

Mr. Cristofer Price

Abt Associates

6130 Executive Boulevard

Rockville, MD 20852

(301) 634-1854

Dr. Jessica Thornton Walker

Abt Associates

6130 Executive Boulevard

Rockville, MD 20852

(301) 347-5622

Dr. Scott Brown

Abt Associates

6130 Executive Boulevard

Rockville, MD 20852

(301) 347-5597

Ms. Tanya de Sousa

Abt Associates

6130 Executive Boulevard

Rockville, MD 20852

(301) 347-5695

References

Browne, M. W. & Cudeck, R. (1993). Alternative ways of assessing model fit. In Bollen, K.A. & Long, J.S. (Eds.) Testing structural equation models. Newbury Park, CA: Sage, 136–162.

Huber, P. J. (1967). The behavior of maximum likelihood estimates under nonstandard conditions. In Proceedings of the Fifth Berkeley Symposium on Mathematical Statistics and Probability. Berkeley, CA: University of California Press, 1, 221–233.

Steiger, J. H. (1989). Causal modeling: a supplementary module for SYSTAT and SYGRAPH. SYSTAT; Evanston, IL.

Valentine, E. J., Skemer, M., & Courtney, M. E. (2015). Becoming adults: One-year impact findings from the Youth Villages Transitional Living Evaluation. New York: MDRC.

White, H. (1980). A heteroskedasticity-consistent covariance matrix estimator and a direct test for heteroskedasticity. Econometrica, 48, 817–830.

White, H. (1984). Asymptotic Theory for Econometricians. Orlando, FL: Academic Press.

1 Most TLPs serve a relatively small number of youth. Among the TLPs funded in September 2017 alone, the average number of youth served annually was about 10, ranging from 1 to 54 youth annually.

13