Design Report

APPENDIX S

DESIGN REPORT FOR YOUTHBUILD EVALUATION

DESIGN OF THE YOUTHBUILD EVALUATION

June 2012

MDRC

Mathematica Policy Research

Social Policy Research Associates

TABLE OF CONTENTS

1. Introduction 1

2. YouthBuild 4

   1. Program components 5

   2. Performance standards 7

3. Findings from Earlier Research 7

   1. Evaluations of other programs for disadvantaged youth 7

   2. Evaluations of YouthBuild 12

   3. Implications for the YouthBuild evaluation 16

4. Conceptual Framework 17

5. Program Selection and Expected Sample Size 20

   1. Selecting DOL-funded programs 20

   2. Selecting CNCS-only-funded programs 22

   3. Sample size and minimum detectable effects 22

6. Enrollment, Random Assignment, and Flow 25

   1. Process for random assignment 26

   2. Working with selected programs 30

7. Evaluation Overview 32

   1. Research components 32

   2. Data sources and data collection 33

8. Impact Analysis 39

   1. Key questions 39

   2. Analysis strategy 40

   3. Weighting 40

   4. Effects for subgroups 42

   5. Effects of program features 43

9. Process Analysis 46

   1. Key questions 46

   2. Data sources and data collection 48

   3. Key areas and data sources used 54

10. Cost Effectiveness Analysis 59

11. Timeline and Reports 60

    1. Timeline 60

    2. Reports 60

LIST OF TABLES, FIGURES AND APPENDICES

Figure

1 Conceptual Framework 19

2 Random Assignment Flow Chart 27

3 Evaluation Timeline 62

Table

1 Selected Rigorous Evaluations of Programs for High

School Dropouts 9

2 Minimum Detectable Effects 24

3 Survey Items for Measuring Key Program Impacts 35

4 Information from the DOL MIS to Collect at Baseline 37

5 Topics and Respondents for Process Study Site Visits 50

6 Draft YouthBuild Classroom Observation Checklist 56

Appendix

A Grantee survey 65

B Youth Development Modules for Youth Surveys 88

C Process Visit Protocols 99

D Program Quality Checklist 167

I. Introduction

National studies estimate that 3.5 million to 6 million people between the ages of 16 and 24 are high school dropouts—meaning that they have not earned a high school diploma and are not now enrolled in high school.¹ A disproportionate number of dropouts come from low-income and minority families. According to the National Center for Education Statistics (NCES), the share of 16- to 24-year olds who are out of school and lack a diploma or General Educational Development credential (GED) is four percent in the highest income quartile and 17 percent in the lowest quartile. Similarly, the dropout rate is six percent for white youth, 12 percent for black youth, and 20 percent for Hispanic youth.²

While there is some disagreement among researchers about whether dropout rates have risen over time, what is clear is that the consequences of dropping out have increased dramatically, given trends in the labor market over the recent and not so recent past. First, technological changes and other factors have increased employer demand for higher skilled workers, with the result that inflation-adjusted wages for less educated workers have fallen over the past several decades.³ On top of this, youth are finding it harder and harder to find jobs, reflecting what has been called a “collapse” in the young adult labor market. Between 2000 and 2008, for example, the share of all 16- to 24-year-olds with employment during the summer fell from 65 percent to 58 percent, falling to 49 percent during the recent recession.⁴ Employment rates are especially low for youth without a high school diploma. Indeed, among young men who were not enrolled in school and did not have a high school diploma or GED, only one in two was employed in 2010.⁵ High rates of youth unemployment are a concern given that early problems in the labor market can have lasting effects.⁶

Adding to the problems faced by these youth is that a large fraction of them do not go on for further education. A recent study documented that among dropouts from low-income families, for example, only half of them went on to earn either a high school diploma (16 percent) or a GED (34 percent) within eight years after scheduled graduation.⁷ The GED by itself has generally not been found to have much return in the labor market, although there is evidence that it may lead to earnings increases in the longer-term for some groups.⁸ The GED can also be a route to post-secondary education, although a minority of GED recipients go on to enroll in college and even fewer complete a degree.⁹

Finding ways to reengage high school dropouts and help them make a successful transition to adulthood is one of our nation’s central social policy challenges. However, the realities of today’s labor market pose a particular challenge for out-of-school youth, who are no longer connected to institutions designed to provide them with training and link them to good jobs. YouthBuild is one such “second chance” program. Targeted largely to high school dropouts, the program provides construction-related training, educational services, counseling and leadership development opportunities to low-income, out-of-school youth ages 16-to-24. Having grown from a single program in New York City to a large national network of more than 240 programs serving more than 9,000 youth per year, YouthBuild is one the most promising models for addressing the needs of out-of-school youth.

In 2007, responsibility for federal funding of YouthBuild was placed under the Department of Labor’s (DOL) Employment and Training Administration (ETA). Since that time, DOL has provided grants to over 200 programs. The Corporation for Community and National Service (CNCS) is another major funder of YouthBuild programs, providing grants to more than 50 programs through its National Direct and state grants.

In July 2010, ETA contracted with MDRC, Mathematica Policy Research and Social Policy Research Associates to conduct a rigorous evaluation of DOL-funded YouthBuild programs, in order to measure the program’s effects on the young people it serves. DOL also has an agreement with CNCS to include in the evaluation a number of YouthBuild programs that do not receive DOL funding but do receive funding from CNCS.

This report presents the proposed evaluation design. In brief, the evaluation will consist of the following key elements:

• A random assignment research design. Eligible and interested youth at select programs will be assigned at random to either a YouthBuild group, eligible to enroll in YouthBuild, or a control group, not eligible to enroll in YouthBuild but eligible for other services in the community.

• A total of 60 DOL-funded YouthBuild programs and 23 CNCS-only-funded programs. From the universe of programs awarded DOL funding in 2011, 60 programs were randomly selected for the impact component of the evaluation, with larger programs having a higher probability of selection. The 23 CNCS-only-funded programs represent the universe of programs that did not receive DOL funding but did receive grants of $95,000 or more from CNCS in 2011.

• Follow-up surveys and administrative records data to track program effects for four years. Three surveys will be administered to youth 12, 30, and 48 months after their enrollment in the study. The surveys will focus on service receipt, employment, educational attainment, criminal justice involvement, living arrangements, YouthBuild experiences, and social-emotional development. Administrative records data covering post-secondary enrollment and employment and earnings will also be obtained.

• A full sample of about 4,200 youth and a survey sample of 2,700 youth. Assuming that the average program will have 50 applicants, the total evaluation sample from 83 programs will be approximately 4,200 youth. Administrative records data will be available for the full sample. The surveys will be fielded to a randomly drawn subset of the full sample, yielding data on 2,700 youth.

• A process analysis to document variation across the sites in program components and fidelity. This analysis will explore the design and operations of the YouthBuild programs, the perceptions and experiences of the participating youth, and the local context in which each program operates. The analysis will be supplemented by a survey administered to all DOL- and CNCS-funded grantees, including programs not selected for the impact component of the evaluation.

This report discusses each aspect of the evaluation in more detail, including strategies for selecting programs into the evaluation, assessing program fidelity, collecting follow-up data, estimating program impacts in key domains, and measuring the program’s cost effectiveness. The next two sections provide background on the YouthBuild program and other research on programs serving youth, in order to provide important context for the evaluation.

II. YouthBuild

YouthBuild is a federally and privately funded program that provides construction-related training, educational services, counseling and leadership development opportunities to disadvantaged youth. As part of their training in construction, participants work on renovating or constructing new housing for low-income or homeless people. Eligibility is typically limited to out-of-school youth ages 16-to-24, who are from a low-income family, in foster care, are ex-offenders, migrants, disabled, or are children of incarcerated parents.

The eligibility requirements mean that the program serves a quite disadvantaged segment of the youth population. For example, among programs that received DOL grants in 2009, 95 percent of YouthBuild participants lacked a high school diploma or GED when they entered the program and 31 percent had a criminal record. In addition, a majority of participants were ages 18-to-21, male, and African-American, Latino or Native American.¹⁰

The more than 200 programs across the country are quite diverse in terms of structure and size. For example, some programs are community-based organizations (CBOs) or faith-based organizations, while other YouthBuild programs are run by local government agencies or educational institutions. As one example, at least 40 YouthBuild programs are now diploma-granting schools. Some programs are relatively small, serving 8-to-10 youth per year, while others serve 75 or more youth per year. Most programs are somewhere in the middle, serving between 30 and 35 youth per year.

All of these programs are modeled after the original YouthBuild program—the East Harlem Youth Action Program (YAP) founded in the late 1970s. In the late 1980s and early 1990s, other programs modeled on YAP were developed under the program’s new name, YouthBuild. To support these replication efforts, key staff from YAP founded YouthBuild USA in 1988 to provide technical assistance and training to new YouthBuild programs. Then, in 1992, under the umbrella of YouthBuild USA, a number of local YouthBuild programs came together to form the YouthBuild Affiliated Network made up of programs that agreed to uphold certain standards for performance and program design and to support advocacy efforts on behalf of the program and low-income youth.

That same year, YouthBuild supporters succeeded in passing Federal legislation to support the program’s national expansion. This legislation, part of the Housing and Community Development Act, placed responsibility for funding YouthBuild through an annual, competitive process under the auspices of the Federal Department of Housing and Urban Development (HUD). Administration of the Federal YouthBuild program by HUD continued until 2007 when Congress approved an amendment to the Workforce Investment Act to transfer responsibility for YouthBuild to the Department of Labor’s Employment and Training Administration.

In addition to funding from DOL, about 50 of the programs nationwide receive funding from CNCS through its National Direct grant to YouthBuild USA.¹¹ (The majority of these programs also received funding from DOL in 2009.) These programs –YouthBuild AmeriCorps programs—place a strong emphasis on the community service component of the program and on post-secondary enrollment. A distinguishing feature of YouthBuild AmeriCorps programs is that participants are eligible for education awards upon completion of YouthBuild, ranging from about $1,400 to $5,500 depending on their hours of service and other activities in YouthBuild.

Finally, other sources of funding include state-funding;¹² Average Daily Attendance money for those grantees that are accredited schools; grants that fund pre-apprenticeship projects, “green building,” and mentoring; and grants for the National Schools Initiative (NSI), which is a Gates Foundation initiative designed to improve educational outcomes and build links to post-secondary education. More recently, 19 programs have been part of the YouthBuild Post-Secondary Education Initiative, funded by several organizations, which aims to develop strategies to help YouthBuild participants enroll and persist in college.

1. Program components

YouthBuild is one of the set of youth programs that attempt to incorporate youth development principles into their design. Most YouthBuild participants spend from 8-to-12 months full-time in the program receiving a variety of services, including stipends, wages, or living allowances. These services typically include some kind of assessment, a “mental toughness” orientation, educational activities, vocational training, leadership training and community service, counseling, support services, job placement, and follow-up services:

• Assessment. Most programs administer basic skills tests such as the Test of Adult Basic Education (TABE) to determine the level of assistance that is needed for youth to attain a diploma or GED. In addition, many grantees conduct one-on-one interviews with youth to collect information on participant goals and barriers to success and conduct criminal background checks and test youth for drugs to determine youths’ supportive service needs. Some programs screen out substantial numbers of youth who they believe are not academically or emotionally ready for the program.

• Mental Toughness Orientation (MTO). These orientations may last up to four weeks and are intended to help prepare youth for the rigors of the program by facilitating group bonding, helping youth to develop trust in staff, and emphasizing goal setting and overcoming obstacles. This orientation also provides an opportunity for program staff to assess each youth’s readiness to participate fully in the program. Following successful completion of MTO, youth are formally enrolled in YouthBuild and begin receiving the core services of the program. (Note that some programs may not have a formal MTO component or may have a similar type of orientation but not refer to it as MTO. For simplicity, throughout the design report we refer to the upfront orientations as MTO.)

• Educational Activities. Participants must be offered educational services at least 50 percent of the time that they are in the program. These services can include basic skills instruction, remedial education, bilingual education, alternative education leading to a high school diploma or GED, counseling or assistance in attaining post-secondary education and financial aid, and alternative secondary school services.

• Vocational Training. Participants generally spend most of the remaining half of their time in the program in construction training, in which they rehabilitate or build housing for low-income or homeless people. While YouthBuild seeks to expand the supply of “permanent affordable housing” through its construction work, most programs renovate or build only a few units per year.¹³

• Leadership Training and Community Service. Every YouthBuild program that is part of the YouthBuild USA Affiliated Network is required to have a policy committee of youth elected to represent their peers. These youth meet regularly with the staff to discuss all aspects of the program and make recommendations for policy decisions. For leadership training, most programs operate a participant-run “advisory council” that provides youth with opportunities for leadership development. Youth also participate in community service activities through organized volunteering opportunities at local organizations and/or engage in public speaking events to advocate for YouthBuild.

• Counseling and Other Supportive Services. YouthBuild programs commonly provide counseling, support services, job placement, and follow-up services. Counseling and support services—either on-site or by referral—are usually available to youth to assist them in dealing with personal problems, mental health issues, substance abuse, pregnancy, violence, sexually-transmitted diseases prevention, transportation to and from the site, childcare and housing. Most programs also provide participants with a modest stipend for work at the construction site or subsidized work experience while in the program. Finally, most programs offer job development and placement services, including training in resume writing and interviewing.

2. Performance standards

Although HUD never established program performance standards required for Federally-funded YouthBuild sites, upon transfer of the program to DOL, YouthBuild grantees funded by ETA became subject to the same measures for enrollee performance used by most ETA-funded programs, referred to as the “common measures.” Specifically, YouthBuild grantees are subject to the youth common measures, which include placement in employment or education, attainment of a degree or certificate, and literacy and numeracy gains. Performance for the program is assessed on these three key measures.

In addition to these Federal standards, YouthBuild-specific program design and performance standards were first developed in 1992 by YouthBuild USA’s Affiliated Network and have been regularly updated every few years. The current standards include benchmarks for attendance, retention, job placement, educational attainment, civic engagement, and literacy gains. These measures are not formal performance measures used by external funders of the program, but do represent key areas that YouthBuild USA views as important indicators of performance.

III. Findings from Earlier Research

1. Evaluations of other programs for disadvantaged youth

Second-chance programs have long offered opportunities for young people who leave the K-12 education system without earning a diploma. Ranging from large national programs or networks like Job Corps and YouthBuild to small independent programs run by churches or community-based organizations, these programs typically provide some combination of education, training, employment, counseling, and social services. Some, like Job Corps and YouthBuild, have dedicated streams of federal funding, while others piece together funding from the Workforce Investment Act (WIA) and other state and local sources. Many programs target specific subsets of youth, such as those with disabilities or those in the foster care or juvenile justice systems, reflecting the availability of targeted funding for those groups. High school dropouts are typically overrepresented among these vulnerable populations.

Table 1 describes eleven rigorous evaluations of employment- or education-focused programs serving high school dropouts that have been conducted over the past thirty years (a few of the programs served both dropouts and in-school youth). The table focuses on major studies that used random-assignment designs.¹⁴

The evaluated programs are grouped according to their primary service approach. The first three programs—the National Supported Work Demonstration, the Youth Incentive Entitlement Pilot Projects (YIEPP), and the American Conservation and Youth Service Corps—relied heavily on paid (or stipended) work experience, while the next six—JOBSTART, the National Job Training Partnership Act, New Chance, the Center for Employment and Training (CET), Job Corps, and National Guard Youth ChalleNGe— focused more on job training or education. The last two—the Teenage Parent Demonstration and the Ohio Learning, Earning, and Parenting program (LEAP)—were mandatory, welfare-based programs that encouraged or required teenage mothers to work or go to school.

Overall, the evaluations tell a mixed story. Several of the studies found that young people in the program group were substantially more likely than their control group counterparts to earn a GED or another credential, but many of those same programs did not lead to positive effects on employment or earnings. The CET replication study, for example, found large increases in the receipt of a training certificate after four years (58 percent for the program group versus 37 percent for the control group) but had no effects on employment or earnings.¹⁵ The JOBSTART and New Chance studies had similar findings, although those programs led to increases in GED receipt rather than in vocational certificates.

Job Corps is one of the few programs that led to positive effects on employment and earnings, although those effects faded after the fourth year of follow-up. The evaluators noted that the provision of longer-term services and placement support may be important to producing lasting impacts and that many of the programs began changing in this direction towards the end of the evaluation period.¹⁶ Another explanation for the lack of sustained effects may be that the

increases in educational attainment were limited to GED receipt and short-term training certificates. For example, 42 percent of the program group earned a GED within four years after entering the study, compared with 27 percent of the control group. The program had no effects on college attendance.¹⁷

Finally, the effects on earnings may have also faded given the composition of the control group, which consisted of youth who were motivated to apply for and attend Job Corps. Their motivation levels were evidenced by their high rates of participation in education and training—72 percent participated in an activity during the four-year follow-up period. This issue is also relevant to the CET evaluation, where over 70 percent of youth in the control group participated in education or training during a four-year period. In that study, although the program group had higher rates of certificate receipt by month 48, both research groups had completed a similar number of hours in education or training.

2. Evaluations of YouthBuild

Several studies of YouthBuild have been conducted over the past twenty years. One of the first, published in 1996, was a formative evaluation conducted from 1991 to 1994 by Ferguson and others of the first five YouthBuild replication sites.¹⁸ This study found that YouthBuild participants had been relatively successful compared to those in other programs serving similar populations of youth. For example, a higher percentage of YouthBuild participants obtained a GED (20 percent) than did participants of comparable programs (nine percent, on average).¹⁹ In addition, 69 percent of YouthBuild participants who did not drop out for reasons such as illness or death earned a positive termination from the program. Of these positive terminations, the majority were for full-time or part-time employment or enrollment in school or training.²⁰

The evaluation also found that two-thirds of YouthBuild participants who were employed immediately after leaving YouthBuild had jobs in construction. However, while the YouthBuild program was generally successful in producing job-ready workers, it was not as successful in producing youth who had obtained the skill level needed to enter an apprenticeship in construction. This was an important finding because youth who complete apprenticeships are then eligible for higher paying, union construction jobs.

This high performance of YouthBuild relative to comparable programs was achieved despite some critical challenges experienced by programs that were struggling to raise sufficient funds to operate, since each of the sites were demonstration efforts that had no Federal source of funding. Further, some of the programs struggled with their inexperience in managing construction projects, which created some difficulties in finding appropriate projects and arranging projects and contracts to achieve timely completion of the work.

The study team conducted simulations and multivariate analyses to examine differences in program outcomes, and concluded that differences in performance between the programs were due in equal measure to differences in the characteristics of youth they served and to differences in the quality of program implementation. In terms of youth characteristics, a number of factors helped predict either positive terminations from the program or GED attainment, including the participant’s age, basic skill levels, goals, and his or her belief in the efficacy of knowledge and effort, as opposed to chance, in determining lifetime earnings.

Finally, the authors developed a model of the stages of development youth must pass through in order to most benefit from their YouthBuild participation. Modeled on Erik Erikson’s work, these stages include trust versus mistrust, autonomy versus shame and doubt, initiative versus guilt, industry versus inferiority, and identity versus identity confusion. Critical to success, according to this model, is that youth must build positive relationships with staff of the programs, but also that they use the staff as supports for their personal development.

Another, smaller-scale study of a YouthBuild Welfare-to-Work (WtW) program, published in 2001, also found relatively positive outcomes for YouthBuild participants.²¹ YouthBuild USA received a grant in 1998 to fund 10 YouthBuild programs to work with Welfare-to-Work eligible trainees for two program cycles. Despite having served a higher percentage of Temporary Assistance for Needy Families (TANF) program recipients than was typical for YouthBuild, participants in the YouthBuild WtW programs were more likely than participants in non-YouthBuild WtW programs to be placed in jobs at the end of the program, and they earned higher wages.

A survey of YouthBuild graduates from a subset of established YouthBuild sites also found that graduates fared relatively well after leaving the program.²² This study, published in 2004, found that 75 percent of YouthBuild graduates were either enrolled in postsecondary education or job training or were working in jobs for an average wage of at least $10 per hour. Ninety-two percent also expressed positive emotions, a solid self-image, and optimism about the future; 76 percent reported not receiving food stamps, Unemployment Insurance (UI), or TANF; and only nine percent reported being convicted of a felony after participating in the program compared with 27 percent prior to the program. These graduates also provided strongly positive overall evaluations of their YouthBuild experience. However, many expressed a need for continued access to services after graduation. The majority of YouthBuild graduates entered the program facing the challenges of poverty, substance abuse, and/or criminal records, for example, and these problems continued to weigh on them after graduation.

By contrast, the primary study funded by HUD of 20 YouthBuild programs operating from 1996 through 1999 found less positive results.²³ The report authors concluded that YouthBuild was a relatively expensive program, given that certain outcomes for program participants appeared no better on average than outcomes for control group youth in studies of other programs, including JOBSTART and Job Corps. For example, only 36 percent of all participants obtained jobs on exit, and only 13 percent of all participants found jobs in construction. The study attributed this low rate to several factors, including the fact that many participants were not interested in construction as a career and that, in many communities, few entry-level construction jobs were available. The study also reported that approximately 50 percent of participants dropped out of the program, though this drop-out rate varied somewhat by site, depending on the site’s recruitment and enrollment policy. The difference between this study’s conclusions and those of Ferguson et al, may be due to differences in the programs analyzed, samples and time periods used, and methods of comparison.²⁴ They highlight the need for a rigorous evaluation of YouthBuild’s effects.

Additional studies have focused on smaller components of the program, or only on a small subset of grantees. Drawing on a survey of 695 YouthBuild participants (primarily those who participated in YouthBuild in 2004 and 2005), one study sought to assess post-secondary education related outcomes associated with Education Awards made under the AmeriCorps program.²⁵ This study found a range of encouraging outcomes. For example, YouthBuild completers who were Americorps members were significantly more likely to have applied to and been accepted at post-secondary schools or training institutions than were non-AmeriCorps members. However, they were not more likely to have enrolled in such institutions, although most youth were surveyed within only their first year or two after leaving YouthBuild.

A more recent study (released in 2009) was a process study of the YouthBuild Youth Offender programs, including 34 grantees that had received special grants from DOL to serve these youth.²⁶ This evaluation documented broad variation across the programs in term of a number of program components, including the duration and purposes of mental toughness orientation, the types of educational services offered, and the quality of vocational training participants receive. Grantees also varied substantially in the intensity of their case management, with some providing intensive and numerous interactions and others adopting more of a laissez-faire approach.

In terms of outcomes, more than 68 percent of youth ex-offender participants completed the program either by graduating or through early job placement. While only nine percent of youth offenders entered the program with their GED or high school diploma, more than one-third (36.3 percent) obtained one of them after entering YouthBuild. Additionally, nearly two-thirds of all youth offenders were placed in unsubsidized employment, an educational activity, or occupational training. Finally, the majority of youth ex-offender participants (74.6 percent) did not re-offend or have their probation or parole revoked during the follow-up period after entering YouthBuild.

The study also found that grantees located in urban areas, with lower student-to-classroom teacher ratios, that had active Youth Policy Councils, that owned and/or managed their own worksites, and that linked vocational instruction to academic instruction tended to have higher rates of employment among their youth ex-offenders. In terms of educational attainment, grantees that were charter schools tended to have higher rates of high school or GED completion, and grantees that were independent entities, rather than part of a larger sponsoring agency, had lower rates of high school diploma or GED attainment. Participants in sites that had lower student-to-staff and lower student-to-classroom teacher ratios had better educational outcomes.

Another recent study, focusing on YouthBuild programs that receive AmeriCorps funding, also found that lower student-to-teacher ratios were strongly correlated with higher GED attainment.²⁷ In addition, offering a high school diploma option was positively related to GED attainment, as was having in-house GED programs rather than contracting for GED instruction. Further, students in programs with schedules that had used a one-week-off, one-week-on schedule or alternating construction and academics within any given week (rather than alternating every two weeks or longer) had better odds of getting their GEDs. In terms of participant characteristics, math skills at program entry were strongly correlated with the likelihood of attaining a GED.

In general, the earlier studies of YouthBuild suggest that program completers have fared well, especially considering the critical barriers they have had to overcome to find employment or increase their educational attainment. In addition, program fidelity and certain program features are associated with participant outcomes. However, the key limitation of these studies is the lack of a credible comparison group, making it difficult to determine how many participants would have achieved these outcomes even in the absence of the program.

3. Implications for the YouthBuild evaluation

Taken as a whole, the prior research on youth programs in general and YouthBuild in particular suggest a number of implications for the evaluation.

• It will be important to document the variation across programs in fidelity to the YouthBuild model and program features. The earlier research found significant variation across programs in fidelity, quality, and in certain program features. Examples include the length of MTO, quality of instruction, student-teacher ratios, the type of organization, and the extent of post program follow-up. The research cited above suggests that many of these factors may influence the effects of the program.

• It will be important to look at effects by education focus. The difficulty youth programs have had in achieving sustained increases in economic outcomes may be traced, in part, to the educational goal of most programs—to help participants pass the GED exam. Many studies have concluded that the labor market does not, in fact, view the GED as equivalent to a high school diploma. Studies have also shown that postsecondary education pays off as much for GED holders as for high school graduates, but that only a small minority of GED holders complete even one year of postsecondary education.²⁸ YouthBuild includes a mix of programs, some targeting the GED and some focused on high school diplomas (and some focused on both credentials) and, as noted earlier, is also increasingly focused on transitions to post-secondary education. The CNCS-funded, Americorp YouthBuild programs also have a post-secondary focus, providing education awards for graduates.

• Program effects are likely to vary by the types of youth served. Research on other youth programs has found differences in effects across certain characteristics, such as age and arrest history. In addition, YouthBuild studies document differences in outcomes across several youth characteristics, such as age, basic skill levels, and goals.

• It will be critical to document the services received by youth in the control group. In any voluntary program, youth who apply are likely to be a self-selected and motivated group. This issue is especially relevant to YouthBuild, given that staff often screens applicants and accepts only those who demonstrate motivation and commitment. Thus, the young people who are assigned to the control group, ineligible for YouthBuild, are likely to seek out other programs in the community and may even enroll in services that are fairly similar to YouthBuild. In this case, as was the case for the CET evaluation, it will be important to document the “treatment” difference in services between the program and control groups in order to interpret the impact findings.

• It will be important to examine effects on youth development outcomes. Although effects on educational attainment and employment and earnings will be a key focus of the evaluation, it will be important to not lose sight of other types of outcomes, such as social and emotional development, and community engagement. Earlier YouthBuild studies documented a significant transformation among graduates in their aspirations and beliefs. As noted in one study, for example, graduates now had more ability to envision a better life and the sense that they had a right to claim that life. Effects on these types of outcomes are important in their own right. They are not only explicit goals of the YouthBuild program but can be precursors to self-sufficiency.²⁹

IV. Conceptual Framework

Figure 1 presents the conceptual framework that will guide the analysis. YouthBuild generally begins with an upfront screening process to identify youth who staff feel will benefit from the program. This process includes both an initial assessment, which may involve basic skills tests and interviews, and the MTO phase. While some critics view this upfront process as “creaming,” YouthBuild staff often describe it as essential to the program’s success, since they believe that some eligible youth are simply not ready to benefit from the program. Moreover, they argue, MTO plays a critical role by building a strong, positive group identity among the cohort of youth. The screening and MTO phase are not expected, by themselves, to lead to impacts but may affect the impacts of the program by determining the types of youth who receive the core YouthBuild services.

Most youth who make it through MTO are enrolled and offered a rich and unusual mix of services and supports, including education, vocational training, leadership training, community service, counseling and supportive services, and, in the case of CNCS-funded programs, an education award to help cover the costs of post-secondary education.

The youth development services are what distinguish YouthBuild from more traditional youth employment programs. In the YouthBuild context, youth development includes a strong emphasis on leadership (youth typically play an active role in running the program), community service and activism, and intensive, personalized support and counseling from staff. Many experts believe that effective second-chance programs for youth must focus on developing the “whole person,” not just providing training and education, because disadvantaged youth often lack opportunities for positive developmental experiences in their homes, schools, and communities.

As youth near the end of the formal program, staff helps them find jobs or enroll in further education. Follow-up services, both formal and informal, may continue for many months after official participation ends. When successful, this cluster of services, opportunities, and relationships (along with stipends or wages) binds youth to the program, addressing a critical weakness in many youth programs – low levels of retention.

This lengthy, intensive, multi-faceted intervention is hypothesized to create a number of positive changes for participants. Most concretely, youth may emerge from the program with an educational credential (or at least literacy and numeracy gains), work experience (even if they are not interested in construction, the worksite experience builds generic job readiness skills), and a job that staff helped them find. They might also be enrolled in post-secondary education. Less tangibly, YouthBuild aims to stimulate lasting attitudinal changes that will keep the young person on a positive trajectory. As described in the earlier studies, the program aims to help youth see themselves differently – as young adults who have the power to affect their own circumstances and to bring about positive changes in their communities. Key effects in the short-term, then, include new educational credentials, jobs, changes in self-perception, and

possibly changes in peer groups. Also, because youth have been positively engaged in the program, one would expect to see less crime and delinquency.

Over the longer-term, one would hope to see increases in earnings (from higher employment rates and higher wages), continued increases in education (hopefully in the post-secondary system), continued desistence from crime, more stable living arrangements, improved health and well-being, and less out-of-wedlock childbearing. Eventually, outcomes like marriage and family formation might appear.

The bottom panel of the conceptual framework illustrates the importance of community context and individual factors in mediating the program’s potential impacts. In terms of community context, for example, the availability of services in the community may diminish the program’s effects if applicants who do not make it into YouthBuild can readily access other services for youth. The economy will likely also affect program impacts, although it is not always clear in what direction. Similarly, the types of youth the programs serve can have important effects on outcomes. YouthBuild might have different effects for youth who enter the program with higher basic skills, compared with effects for their counterparts with lower reading and math levels.

V. Program Selection and Expected Sample Size

The evaluation will include 83 YouthBuild programs, 60 of the 74 programs awarded grants by DOL in 2011 and 23 of the 40 programs that did not receive DOL funding in 2011 but received funding from CNCS. The latter programs will be referred to as CNCS-only-funded programs, although they may receive funding from other, non-DOL sources. DOL and CNCS chose to include the CNCS only-funded sites in the evaluation in order to examine whether program impacts vary by funding source. As noted earlier, CNCS-funded programs tend to have a greater focus on the community service component of the program and on enrollment in post-secondary education, reinforced in large part by the education awards available for youth who complete their programs.

In deciding on the total number of programs to include in the impact component of the evaluation, we attempted to balance the objectives of 1) maximizing the representativeness of the sample and the statistical power of the impact analysis and 2) ensuring high quality implementation of program enrollment and random assignment procedures.

1. Selecting DOL-funded programs

DOL awarded grants to 74 YouthBuild programs in May 2011.³⁰ Prior to the selection of 60 programs, the following types of programs were eliminated from consideration:

Programs in which youth are guaranteed entry into YouthBuild as early as 9^th or 10^th grade. Some charter schools affiliated with YouthBuild programs enroll youth in the 9^th or 10^th grade and guarantee them a later placement in YouthBuild conditional upon successful completion of the first two or three years. Random assignment at the point of entry into YouthBuild would not be possible in these programs, since youth have already been guaranteed a slot.

Programs in which youth in the control group will receive services very similar to YouthBuild. A key part of the evaluation is to measure the extent and type of participation in services by the control group, in order to place the impact findings in context. In some sites, youth who do not get into YouthBuild are expected to enroll in services that are very similar to YouthBuild’s. One obvious example is programs that recruit for YouthBuild from the Conservation Corps. Other examples might include YouthBuild programs embedded in charter schools, where control group youth would remain in the charter school and also receive some type of vocational training component. Including these programs in the evaluation would not provide a true test of YouthBuild’s effects, since the program and control groups would receive similar types of services.

None of the 74 grantees included a guarantee of entry into YouthBuild. However, three programs were dropped from the selection universe because discussions with site staff indicated that the counterfactual was very similar to YouthBuild. In these three programs, youth in the control group are highly likely to receive services and those services would look very similar to YouthBuild. According to program proposals, these three programs planned to serve a total of 133 youth in a given program year. The other 71 grantees planned to serve a total of 3,171 in a given program year. Given that the excluded sites account for only 4.1 percent of expected enrollment among DOL-funded sites, the study team’s ability to extrapolate the study findings to the universe of DOL-funded sites is not compromised.

Sixty programs from the universe of 71 programs were selected for the impact component of the evaluation, using probability-proportional-to-size sampling. Each program had a probability of selection that was equal to its expected enrollment over a given program year. This method gives each YouthBuild slot (or youth served) an equal chance of being selected for the evaluation, meaning that the resulting sample of youth who enroll in the study should be representative of youth served by these programs.

All of the 60 selected programs are required by DOL to participate in the evaluation. Should there be some sites that cannot participate the evaluation team will conduct a response analysis to assess how these sites differ from the sites that do participate.³¹

2. Selecting CNCS-only-funded programs

CNCS funds programs through its National Direct grant to YouthBuild USA. Forty YouthBuild programs received CNCS grants but not DOL funding in 2011 through YouthBuild USA. After reviewing available information and conducting phone calls with each of the 40 programs, the evaluation team determined that many of the programs, particularly those receiving small CNCS grants, were likely to shut down in 2012 or not enroll youth during the study enrollment period. For this reason, the study team, along with ETA and CNCS, opted to select the 24 programs that received a grant of at least $95,000 in 2010 from CNCS. From this 24, one program that planned to shut down was eliminated from consideration, leaving 23 selected to participate in the study.

3. Sample size and minimum detectable effects

Although it is difficult to project the exact sample size expected by the end of the intake period, a rough estimate can be obtained using information on the average program size and the targeted size of the applicant group desired for a random assignment ratio of 60 program group youth to 40 control group youth.³² For example, the average YouthBuild program serves between 30 and 35 youth per program year, and a hypothetical program that serves 30 youth might invite 40 youth to MTO, knowing that about 10 youth will not finish MTO. In this case, the targeted number of applicants would be about 60 to 65, with the result that 40 applicants would be assigned to the program group and invited to MTO and 20 applicants would be assigned to the control group. However, as described in more detail in the next section, some programs may elect to conduct random assignment during or after MTO. In this case, the hypothetical program serving 30 youth will invite a certain number of youth to MTO, expecting that about 42 youth will complete, with the result that 30 youth are assigned to the program group and 12 youth are assigned to the control group. These two scenarios suggest that the average program will randomly assign between 40 and 60 applicants. In addition, data from the 20 programs that have conducted random assignment to date indicate that the average program among this group contributed about 45 youth to sample. Using this information, the study team estimates that the average program will randomly assign 50 applicants, for a total sample size of 4,150 youth (or 83 x 50). A sample of 3,400 youth will be drawn randomly from this larger sample for the fielding of the follow-up surveys, yielding a respondent sample of about 2,700 youth (assuming an 80 percent response rate).

An analysis of Minimum Detectable Effects (MDE) suggests that these sample sizes are large enough to detect policy relevant effects for the full samples and key subgroups. MDEs indicate the size of program impacts that are likely to be observed or detected for a set of outcomes and a given sample size.³³ Table 2 presents the MDEs for educational attainment, employment, and earnings. MDEs are shown for the full sample of 4,150, for whom administrative records data on post-secondary enrollment and employment and earnings will be available, and the survey sample of 2,720.

For the survey sample, the MDE for having a high school diploma or GED is 4.6 percentage points. The MDE for a subgroup comparison, assuming the subgroup comprises half of the sample, is 6.5 percentage points. These effects are policy relevant and well below the effects on educational attainment found from several other evaluations of programs for youth. For example, impacts on GED or high school diploma attainment were 15 percentage points and 25 percentage points, in the Job Corps and ChalleNGe programs, respectively.

MDEs for short-term employment rates are shown in the second column. MDEs are similarly 4.6 percentage points for a full sample comparison and 6.5 percentage points for a subgroup comparison. Effects for the full administrative records sample are smaller. Several of the programs mentioned earlier for disadvantaged youth led to substantial increases in short-term employment rates, although these gains tended to diminish in size over time. The Job Corps evaluation found impacts on annual employment rates of 10-to-16 percentage points early in the follow-up period using Social Security Administration records. However, Job Corp’s effects on employment using survey data were substantially smaller, ranging from 3-to-5 percentage points per quarter. Other effects on employment include an impact of 4.9 percentage points after 21 months in ChalleNGe and an impact of 9.3 percentage points after 30 months for women in the high fidelity CET replication sites.

MDEs for earnings are shown in the final column. With the survey sample size, we would be able to detect as statistically significant an impact of at least $1,016 on annual earnings during a given follow-up year. Assuming a control group average of $11,000, for example, this impact would represent a nine percent increase in earnings. MDEs for a subgroup comparison are larger, at $1,436, or a 13 percent increase. Although MDEs using administrative records are smaller, these effects are large, even relative to the significant effects generated by the Job Corps program. Career Academies, however, did lead to substantially larger earnings gains than these, particularly for the men in the sample.

A final point to consider when assessing the statistical power is that random assignment in most sites will determine which youth get invited to MTO not which youth enroll in the core YouthBuild services. As noted earlier, MTO is designed to prepare youth for the rigors of the program, and many youth who start MTO may subsequently decide that they are not ready to participate. This is likely to lead to a dilution of program effects, since some youth will drop out before reaching the core components of the program. This type of attrition poses a challenge for the evaluation, since youth assigned to the program group and invited to start MTO are always considered part of the program group, even if they drop out of MTO and never enroll in YouthBuild. Is it difficult to anticipate the extent of MTO dropout, since its length and intensity will vary across programs.

Early discussions with about 20 programs, in preparation for random assignment in fall 2011, suggest a potential dropout rate of about 20 percent. On average, program staff have indicated that they would like to invite to MTO about 1.25 times the number of YouthBuild slots they have available, meaning that they expect about 80 percent of youth invited to MTO to complete MTO and enroll in YouthBuild. Assuming a dropout rate of 20 percent on average, the effect of YouthBuild on the youth who go on to participate in its core components would need to be about 1.25 times as large as those presented in Table 2. These effects are still within the range of effects found from other programs.³⁴

VI. Enrollment, Random Assignment, and Flow

Once the programs were selected for the evaluation, the evaluation team began working with them to design processes to inform applicants about the study and the implications of their participation, obtain informed consent, and implement random assignment. While it has been and will be necessary to ask some programs to make modest changes in order to accommodate the study, the site liaisons have attempted to tailor the study enrollment procedures to meet the needs of individual programs to the greatest extent possible to avoid distorting program operations in order to accommodate the study.

One example of tailoring is the question of where to place random assignment in the flow of youth into the program, i.e., to place it before or after MTO. One argument for placing random assignment after MTO is that program effects will be diluted if a large number of youth drop out of the program before reaching the core education and vocational training services. On the other hand, some program staff argue that MTO is a core program component. In this case, random assignment after MTO would result in a control group that received one part of the YouthBuild model. Given the latter concern, the team has opted to encourage programs to conduct random assignment before MTO or during the first few days of MTO, before youth receive a substantial dose of YouthBuild services.

In practice, as the study team has started working with individual sites, program staff have expressed clear and strong preferences on this issue. The majority of programs have decided to place random assignment before MTO, in part because of the ethical concerns of turning some youth away (those assigned to the control group) after they have completed a sometimes long and rigorous MTO. About one in five programs has indicated a preference for random assignment during or after MTO, in most cases because that is how they have selected youth in the past. While it may be preferable to have all sites conduct random assignment at the same point, and while the study team will encourage sites to conduct random assignment before or early in MTO, the study team has opted to allow programs the flexibility to determine what works best for them, in order to maintain programs’ good will and cooperation and, ultimately, the integrity of the random assignment process.

1. Process for random assignment

In general, the expected flow of youth into the evaluation is summarized in Figure 2 and follows several basic steps.

1. Programs will recruit and screen youth using their regular procedures. However, the recruitment materials will be reviewed to ensure that they do not guarantee youth a spot in the program.

2. Program staff will present applicants with a video prepared by the evaluation team that explains the study and reiterates the material contained in the Informed Consent Form (ICF). Both the video and the ICF will explain that applicants are required to participate in the study in order to have a chance to enroll in YouthBuild. Staff will be provided with scripts from the research team to answer any questions youth may have after the video presentation and ICF.

3. Youth who choose to participate will sign the ICF, then provide baseline and contact information. As discussed below, baseline data will be collected via the YouthBuild DOL MIS or MDRC’s random assignment website (for the CNCS-only-funded programs) while contact information will be collected on a paper form. (These steps illustrate the generic process, but there may be slight variation from program to program.)

4. Once staff have identified the full study sample and entered the required baseline data into the MIS system, program staff will log into MDRC’s web-based system to enter a few key pieces of information for each applicant. Once these data are entered, staff will inform MDRC and indicate how many youth they would like to either 1) invite to MTO, if that is their selected point of random assignment, or 2) invite to continue on with MTO or start the program, if they are conducting random assignment during or after MTO. MDRC will then conduct random assignment, assigning a certain number of applicants to the program group (invited to MTO or to the program) and the remainder to the control group.

5. Youth assigned to the program group will be invited to report for MTO or invited to continue on with MTO or start the program, depending on the point of random assignment. Those assigned to the control group will be informed of their status and given a list of alternative services in the community. Each youth who completes the intake process will receive a $10 Point-of-Sale debit card as a token of appreciation for the time spent listening to an explanation of the study and completing paperwork.³⁵

The above steps illustrate a generic process for study enrollment, but the details will need to be tailored to each participating program. For example, the above flow assumes a “batch” process, in which random assignment is conducted once for an entering cohort. Some programs enroll multiple cohorts and will go through these steps more than once. Other programs may enroll youth on a rolling basis, in which case each applicant will be informed of his or her group assignment just after they apply.

Key features of the random assignment process include the following:

• A random assignment ratio of 60 treatment youth to 40 control youth, on average, with variation across sites. The statistical power of an experiment (or the ability to detect program effects) is maximized when half of the sample is randomly assigned to the program group and half to the control group, for a 50:50 ratio. This power falls as the ratio moves away from 50:50, to 60:40, 70:30, etc. However, benefits of statistical power must be weighed against the costs to the programs of recruiting additional applicants—many programs, for example, may not have twice as many applicants as they have slots to fill. For this reason, the target random assignment ratio for the YouthBuild evaluation is 60:40 (or six youth assigned to the program group for every four youth assigned to the control group). To meet this goal, program staff will be encouraged to obtain a pool of eligible and appropriate applicants that is approximately 1.67 times the number of youth they would like to invite to MTO. If a program has 24 openings, for example, they will be asked to identify at least 40 applicants, and 16 youth would be assigned to the control group. However, the actual ratio in a given site will depend on its number of excess applicants.

• Random assignment will not take place in a given program cycle without a number of excess applicants that is at least 10 percent of the total number of applicants. A program that identifies 20 applicants for 18 slots would need to conduct random assignment using a ratio of 90:10, or 9 program youth to 1 control youth. The more skewed the ratio, the less is a sample’s contribution to the overall statistical power of the analysis. Thus, it is necessary to weigh the costs of data collection in that site against the statistical power it brings in. However, we recognize that in rare instances a program might not be able to obtain the desired number of excess applicants for a given program cycle. In these cases, we will allow the random assignment ratio to increase from the target of 60:40, but not beyond 90:10.

• Random assignment for most sites will take place as close to the start of MTO as possible, while in some sites it may take place during or after MTO. Our experiences to date with the sites indicate that most sites will want to place random assignment just prior to MTO, in which case some youth assigned to the program group may not complete MTO and never enroll in the formal YouthBuild program. In these sites, it is critical to place random assignment as close as possible to the start of MTO to minimize other forms of drop off, such as being invited to MTO but never showing up. This drop off might occur, for example, if random assignment is conducted in July and program youth are invited to an MTO that starts in mid September. We will also work with sites to consider placing random assignment early during MTO, after some youth have already dropped out by not showing up for the first or second day. As noted earlier, we will encourage programs not to place random assignment too far after the start of MTO, in order to minimize the chances that youth who are ultimately assigned to the control group receive substantial YouthBuild services.

• Random assignment will occur in multiple enrollment cycles per program. Some programs enroll youth multiple times during the year, in some cases to replenish dropouts from their original cohorts. We propose to conduct random assignment at each of these enrollment periods, covering the period September 2011 through October 2012. We will work with individual programs to document whether youth enrolled during replenishment cycles experience a notably different program from other youth, such as no MTO or a shorter educational or vocational component. Conducting random assignment during multiple cycles will help us meet our sample size goal and also treats all youth equally, e.g., youth who apply later in the year are still subject to random assignment and thus have some probability of not getting into the program.

• Programs will be allowed to let a few youth bypass random assignment. We expect that some programs may have mandatory referrals or other youth who need to enroll in the program for various reasons. Program staff will be allowed to reserve up to five percent of their program slots for mandatory referrals. These youth will not be a part of the study sample and not included in the impact analysis.

• Youth assigned to the control group will not be allowed to enroll in YouthBuild for two years after they are enrolled into the study. The evaluation team will work with programs to develop protocols for youth assigned to the control group to refer these youth to other services in the community, but not YouthBuild services or services that are very similar to YouthBuild. In addition, control group youth will not be able to enroll in YouthBuild for at least two years after study entry. In order to credibly estimate the effects of the program, it is critical to minimize the number of “crossovers,” or individuals assigned to the control group who subsequently receive program services. We will stress to participating programs the importance of enforcing this embargo and will access program MIS records periodically during this period to identify any control group members that subsequently enroll in YouthBuild. We will then work with programs for which this occurs to minimize further crossovers.

• Random assignment will take place for all of the site’s YouthBuild slots. Some programs receive more than one funding source and use each source to fund a certain number of program slots. Sites are encouraged to use random assignment to fill all participant slots, not just the number of slots that are funded by DOL or CNCS.

2. Working with selected programs

Although all selected grantees are required to participate in the evaluation, we expect that enrolling them into the study and securing their buy in will take time. The YouthBuild network is highly decentralized and the agencies that operate YouthBuild programs are quite independent. Many have Boards of Directors that would need to approve their participation in an evaluation. Moreover, early discussions with program managers and staff made clear that many are not fully comfortable with the notion of random assignment. These staff argue that their programs generally do not “say no” to many youth – rather, they feel it is the youth who remove themselves from consideration by failing to follow through on required steps in the application process or by not participating fully in MTO. When highly motivated youth cannot be admitted, programs may place them on a waiting list and accept them later to fill slots created by dropouts, or staff may actively help youth sign up for other youth programs in their communities. Some staff argued that barring a control group member from participating in YouthBuild would damage their program’s reputation in the community.

To address the concerns head-on, the evaluation team has and will continue to use a multi-pronged strategy to enroll sites and build support for the study, including letters signed by ETA, CNCS, and YouthBuild USA welcoming grantees into the study, phone calls by the evaluation team, presentations and individual meetings at grantee conferences, and, finally, visits to each participating site.

Site representatives on the evaluation team will be assigned a certain number of grantees and work with them to tailor the design and implement random assignment. This process will follow several stages:

• Discussion of recruitment strategies and screening processes for identifying eligible and appropriate applicants. Some program staff report that they do not have excess applicants. Rather, they accept all youth who are deemed suitable for enrollment and the youth self-select out by failing to show up for MTO or later dropping out of MTO. Some programs may indeed accept nearly everyone who applies, but others employ a fairly rigorous screening process to narrow down hundreds of applications in order to invite, for example, 50 youth to start MTO. In either case, the evaluation team will need to have extensive discussions with program staff either on how to scale up recruitment in order to bring in more applicants or how to modify their screening process in order to identify a suitable list of candidates that exceeds the number of youth they would like to invite to MTO.

• Determine the point of random assignment and the point at which the study is explained to youth and the baseline data are collected. There is variation across programs in the way in which youth apply for YouthBuild. Some programs might collect applications throughout the summer and invite suitable applicants to the start of MTO in the fall. Others might require all applicants to come in the week before MTO to view an orientation video and confirm their interest in participating. Random assignment should be placed as close as possible to the start of MTO. In some cases, it will be necessary to work with programs to identify a group of applicants who are still interested in MTO and likely to show up if assigned to the program group. This may take the form of an additional orientation meeting in the weeks before MTO. Beyond issues of timing, the programs might also differ in the extent to which they over-enroll in MTO in order to fill YouthBuild slots. Finally, special attention will need to be paid to YouthBuild programs that are charter schools or alternative high schools, since it may be necessary to coordinate the random assignment process with the local school district’s admissions process.

• Random assignment training. The individual program staff who will carry out the research procedures will need to be trained. Although the procedures will vary somewhat from program to program, we anticipate that all sites will need to be trained on certain key activities: showing the video and answering questions about the study; collecting the required baseline information and entering the data into the management information system (MIS); explaining the purpose of the contact sheet and helping youth complete it; using MDRC’s web-based system to conduct random assignment; communicating the results of random assignment to youth; and monitoring subsequent enrollment to ensure that no control group youth enroll in YouthBuild before the end of the embargo period.

• Monitoring random assignment. Random assignment began in September 2011 and will continue through October 2012.³⁶ During this period, the evaluation team will take several steps to monitor the process to ensure its integrity, including calling the programs to check in on the status of random assignment, making visits to some programs that might be experiencing problems with random assignment, hosting conference calls for groups of sites to share concerns and information, monitoring baseline characteristics for study youth to ensure no systematic differences between the two research groups, and checking the baseline data for the presence of “crossovers,” or control group youth who enroll in YouthBuild before the end of the embargo period.

VII. Evaluation Overview

The YouthBuild evaluation will include a comprehensive study of the program, involving about 4,200 youth across 83 programs. The study has three major components: an impact analysis, an implementation and process analysis, and a cost-effectiveness analysis, which are described below.

1. Research components

• Impact analysis. This analysis will examine the effects of YouthBuild on improving outcomes for the young people it serves.³⁷ Put simply, it will determine whether the program group had better outcomes than it would have achieved without YouthBuild. The study will track both the program and the control groups for four years using survey and administrative records data. Because random assignment, when properly implemented, helps eliminate systematic differences between the program and control groups prior to the start of the program, any subsequent differences in outcomes—for example, in GED attainment or earnings—can be attributed to the program with confidence. The impact analysis will examine the program’s effects on a wide range of outcomes, including educational attainment and enrollment, work and earnings, criminal justice involvement, family structure, and social-emotional development.

• Implementation and process analysis. This analysis will explore the operations of the YouthBuild program and the perceptions and experiences of the participating youth. It will describe how the youth are recruited into YouthBuild, their rates of participation in various services, and selected outcomes during the enrollment period. In addition, the analysis will assess each program’s fidelity to the YouthBuild model and to a youth development model in general. Finally, the process analysis will collect data capturing the local environment, including economic opportunity and the availability of alternative services for youth.

• Cost-effectiveness analysis. This analysis will estimate the cost of operating and running YouthBuild and compare these costs against any positive gains that are achieved. Costs will include administrative costs, salaries, materials, etc. These costs will be compared against selected impacts, such as the gain in GED or diploma receipt or the reduction in recidivism.

2. Data sources and data collection

Grantee survey

A grantee implementation survey will be fielded to all DOL grantees and CNCS-only-funded programs. The Grantee Survey (presented in Appendix A) will gather in-depth information about the program design and services, such as how the educational and vocational services are structured, how these two components are linked, as well as information about student-teacher ratios. The survey will be administered in Fall 2012 and will serve two key purposes. First, it will ensure uniform data on a variety of program characteristics to support the process study. These data will allow the study team to explore whether there are correlations between outcomes and program characteristics. Second, it will help place the impact analysis findings in context by allowing the team to document how the 83 programs participating in the impact component of the evaluation compare with the other DOL-funded and CNCS-only funded programs that are not participating.

Youth surveys

The follow-up surveys are the primary source of outcome data to assess program impacts. Follow-up surveys with research participants (program and control group members) will be conducted 12, 30 and 48 months after random assignment. One benefit of conducting a 12-month follow-up survey is that it will likely limit the amount of locating needed to find sample members for the survey. That early survey will also provide important information on any services that may have been obtained by youth in the control group. The surveys will be fielded to a random subset of 3,400 youth from the full evaluation sample. We expect to obtain an 80 percent response rate, for a total survey sample of 2,700 youth.

Based on the YouthBuild program, the study’s research objectives, and a review of the literature on educational and employment interventions for youth, the surveys will include five core modules, measuring service receipt and outcomes in four key areas, and one module covering YouthBuild experiences for youth in the program group. In addition, three topical modules will measure other aspects of youth development:

Core Modules

• Educational attainment: Highest level of education completed, post-secondary educational plans, post-secondary enrollment.

• Service Receipt: Participation in education, training, employment and youth development activities.

• Employment: Detailed characteristics of current or most recent job, job and earnings history.

• Criminal justice involvement and delinquency: Delinquent activities, arrests, convictions, and incarcerations.

• Current living arrangements, marital status, and fertility.

• YouthBuild experiences (for program group only). Participation in YouthBuild, rating of program quality and experience.

Topical Modules

• Social/emotional development: Self-efficacy, self-esteem, and positive relationships.

• Identity Development: Critical thinking, community service, and civic engagement.

• Health and Well-Being: Drug and alcohol use, risky behavior, and physical and mental health.

Table 3 summarizes the specific data items that we will measure in each of the core modules. Appendix B presents the proposed survey questions for the three topical modules on youth development. The estimated time for survey administration is 40 minutes.

Administrative records

We will obtain two types of data through administrative records in order to supplement the survey data:

Post-secondary education records. The follow-up surveys will include questions on respondents’ enrollment in postsecondary programs, and we expect that these items will be accurately reported. Nonetheless, we will obtain data from the National Student Clearinghouse to supplement the survey data.

The National Student Clearinghouse, a nonprofit organization, collects and distributes enrollment, degree, and certificate data from more than 3,000 colleges that enroll more than 90 percent of the nation’s college students. The data include, for example, enrollment start and end dates, full-time or part-time status, institution name and type (four-year, two-year), and the date and type of any degrees earned. The Clearinghouse will also provide data on sample members who do not respond to the surveys. MDRC will submit requests for data to NSC once per year, starting in 2014.³⁸

Earnings and Employment records. The evaluation team has submitted a request to the U.S. Department of Health and Human Services in order to obtain access to data from the National Directory of New Hires (NDNH), maintained by the Office of Child Support Enforcement (OCSE). The NDNH includes quarterly wage records that are the same as what is typically collected through state UI systems. In some areas, NDNH has better coverage of jobs, such as those in the military; in other areas, such as some agricultural labor, some domestic labor, and jobs paid by commission, NDNH has some underreporting. Nevertheless, these data would allow us to examine effects on quarterly and annual employment, employment retention, and quarterly and annual earnings.

We believe that our request will be approved, given that the YouthBuild population is very relevant to OCSE’s interests. Pending approval, we will obtain access to records data for all study participants for one year prior to random assignment and four years after random assignment. We would request data once per year starting in June 2012.

MIS data from DOL and YouthBuild USA

DOL-funded programs are required to enter data on participant characteristics, service receipt and outcomes into the DOL YouthBuild MIS, and all CNCS-funded programs are required to use an MIS developed and managed by YouthBuild USA, called WebSta. Data from these systems will be used for several purposes. First, they will provide information on a number of demographic characteristics of youth prior to the point of random assignment (at baseline). The evaluation team selected a small number of baseline variables that are obtained via the DOL MIS and were deemed important to obtain prior to the point of random assignment (see Table 4).³⁹ Program staff will be required to enter these data for all youth entering the study before

random assignment takes place. Since these data are already required by DOL and YouthBuild USA, using them will help to minimize staff burden.

CNCS-only-funded sites will most likely be asked to enter baseline data into MDRC’s web-based random assignment system, since these data will not be available in individual-level format from YouthBuild USA. In addition, staff at all sites will be required to enter the data for all eligible applicants, not just for youth who eventually enroll in YouthBuild.⁴⁰ Note that only youth who are assigned to the program group and subsequently complete MTO and enter YouthBuild will count as part of the site’s enrollment and towards DOL or YouthBuild USA performance measures. Other youth (including those in the control group and those in the program group who do not complete MTO) will not have an “enrollment” date entered for them into the MIS and thus will not be counted as part of the site’s enrollment.

In addition to these background characteristics, the MIS systems also contain a wealth of information on the services provided to participants. Though the number of data elements capturing service receipt is too numerous to list in its entirety, generally speaking the services arerecorded through the use of start and end dates (as well as expected dates of completion for activities not yet completed) in specific activities, and whether the activity has been completed.

Finally, these data contain some information on outcomes youth achieve (e.g., employment, earnings, recidivism, educational attainment and post-secondary enrollment, and literacy and numeracy gains) during and subsequent to their program participation. Although these data will not be used to measure program impacts, they will be used in the process analysis to describe outcomes for program participants. Note that the MIS systems will only capture these post-enrollment data for youth in slots funded by either DOL or CNCS. Thus, program participation data will not be available for youth in slots funded by other sources. We expect that the large majority of youth in the evaluation will be in either DOL-funded or CNCS-funded slots. Nonetheless, the survey data will capture participation in services for all types of youth, regardless of funding source.

VIII. Impact Analysis

1. Key questions

The impact analysis will determine the effects of the intervention on the overall sample and for key subgroups of youth. Key questions include the following:

What are YouthBuild’s effects on:

• Educational attainment. GED or high school diploma receipt, enrollment in post-secondary education, receipt of college credits, college degrees, and vocational training certificates.

• Employment and earnings. Employment in unsubsidized jobs, earnings, benefits and wage rates, employment stability, job finding methods.

• Criminal justice involvement. Delinquent activity, arrest rates, incarceration rates, recidivism, and types of offenses.

• Social and emotional development. Sense of empowerment, self-regulation and control, self-efficacy, leadership proclivity, and civic engagement.

• Living arrangements and family formation. Type and stability of living arrangements, marriage or cohabitation, payment of child support, involvement with non-resident children, and childbearing.

How do YouthBuild’s effects vary by:

• Youth characteristics, such as age, criminal justice history, and skill level.

• Program features, such as model fidelity, length of MTO, focus on post-secondary education, and institution type.

2. Analysis strategy

The basic estimation strategy is to compare average outcomes for the program and control groups. This is a straightforward calculation that is generally easy to explain to policymakers and other nontechnical audiences. We will use regression adjustment to increase the power of statistical tests that are performed, in which the outcome, such as “has a GED or high school diploma” is regressed on an indicator for program group status and a range of other background characteristics.

The basic impact model would be the following:

Y_i = α + βP_i + δX_i + ε_i

where: Y_i = the outcome measure for sample member i; P_i = one for program group members and zero for control group members; X_i = a set of background characteristics for sample member i; ε_i = a random error term for sample member i; β= the estimate of the impact of the program on the average value of the outcome; α=the intercept of the regression; and δ = the set of regression coefficients for the background characteristics.

We can use a linear regression framework or a more complex set of methods depending on the nature of the dependent variable and the type of issues being addressed, such as: logistic regressions for binary outcomes (e.g., employed or not); Poisson or Negative Binomial regressions for outcomes that take on only a few values (e.g., months of employment); and quantile regressions to examine the distribution of outcomes for continuous outcomes.

c. Weighting

The variation in enrollment across the 83 programs means that some programs will contribute many more sample members to the analysis than others. As noted earlier, the sampling strategy (probability proportion to size sampling) will obtain a representative sample of youth, not necessarily a representative sample of programs. We propose to give each youth equal weight in the analysis, for two reasons. First, the appropriate impact question is “what is the effect of YouthBuild for the average young person in the program?” Second, weighting each individual equally typically provides a more precise estimate of program impacts. (Impacts from larger sites are typically more precisely estimated and, as such, should be given more rather than less weight in the analysis.) Nonetheless, we will examine the sensitivity of the results to a different weighting strategy—one that gives each site equal weight in the analysis (e.g., giving small sites larger weight and large sites less weight).

As selected, the DOL-funded sites are representative of YouthBuild programs that were funded by DOL in 2011. Similarly, the CNCS only-funded sites will be representative of those sites that received at least $95,000 in CNCS funding in 2011 but not DOL funding. However, while they do represent all DOL-funded and CNCS only-funded programs, it is not clear that the 83 sites are representative of all YouthBuild programs. We will not attempt to develop a weighting scheme to try to generalize to a sample beyond the one from which these sites were drawn, but instead to make clear that our findings generalize only to DOL-funded and CNCS only-funded programs.

Weighting will be necessary, however, to account for the different random assignment ratios across programs. Recall that the program-control ratio in a given site will depend largely on the number of excess applicants each site has, relative to the number of youth staff would like to invite to MTO. For example, assume that there are two sites in the evaluation. Site 1 was able to conduct random assignment using a 60:40 ratio, with 60 youth assigned to the program group and 40 youth assigned to the control group. Site 2, in contrast, assigned 80 youth to the program group and 20 youth to the control group. With varying ratios, sample members will need to be weighted accordingly to obtain correct impact estimates. In this case, youth in site 2 to would be assigned weights in order to bring the weighted ratio in that site to 60:40. The weights for youth in site 2 are derived as the desired ratio divided by the actual ratio, or .75 for youth in the program group (.60 divided by.80) and 2 for youth in the control group (.40 divided by.20). Youth in site 1 would all receive a weight equal to 1, since the actual ratio in this site is equal to the desired ratio. Analyses would then be run on weighted data over the pooled sample.

Weighting may also be necessary if some programs, and thus the youth they serve, are over-represented in the sample. First, as noted earlier, the intake period will run from September 2011 through October 2012 and may be extended through December 2012 if the sample size goals are not met by October 2012. For programs that enroll one cohort per year each September, for example, we may capture two years of enrollment. In contrast, a program that enrolls every June would only contribute one cohort to the sample. While the study team would prefer to limit random assignment at any given program to one program year, concerns about achieving an adequate sample size may mean enrolling additional youth where possible.

Second, and related to the concern about achieving an adequate sample size, some programs may not be able to conduct random assignment during the intake period for some of their enrollment cycles if they do not have enough excess applicants at each cycle. Thus, it may be necessary to weight the sample in order to have each program contribute one year of enrollment to the analysis.

It may also be the case, however, that a few programs are never able to conduct random assignment, given a lack of excess applicants at all cycles during the intake period. It is likely that these programs, which have fewer applicants, differ in important ways from those that have many applicants, such as in size, urban-rural status, and capacity. The types of youth they serve may also be different from youth served by other programs. Thus, we will conduct an analysis comparing the characteristics of programs that were versus were not able to conduct at least one round of random assignment and will consider weighting the data to account for these differences.

Finally, comparing outcomes for all randomly assigned youth provides an estimate of the effectiveness of the intervention in a real-world setting, in which some youth do not receive YouthBuild services, because they drop out of MTO or sometime thereafter. This is likely to be an important issue in YouthBuild, since a potentially large fraction of youth drop out of MTO, by design in many programs. For this reason, it will also be important to understand the efficacy of the program, or its effects on youth who participate at some point or for some minimum duration. We will explore various nonexperimental methods to estimate the effect of the program on those who participated, including nonparticipation adjustments, instrumental variable estimation, and propensity score matching, and present these results with the appropriate caveats.

d. Effects for subgroups

The evaluation will investigate whether YouthBuild worked especially well for particular types of programs and for particular subgroups of youth. In terms of program characteristics, candidates include: fidelity to the YouthBuild model; focus on post-secondary education; type of organization; effectiveness of instructors and the extent to which the academic and skills training are integrated; whether the program owns or manages the construction site; extent of post-program services; and the strength and tenure of leadership.

In addition, there is interest in the role of MTO in YouthBuild’s effects. Some critics argue that the program achieves good outcomes because MTO serves to weed out unmotivated youth. Program staff, on the other hand, might argue that these are precisely the youth who are ready to benefit from YouthBuild. In addition, MTO serves as an important team building process for an entering group of students. Although an intensive MTO undoubtedly serves to screen out unmotivated students, whether it leads to larger or smaller impacts is an open question. We anticipate that there will be enough variation in MTO length to divide sites into less versus more intensive MTO samples.

For youth characteristics, there are number of candidates to consider, including: literacy/numeracy⁴¹; age; offender status; educational attainment; English language proficiency; number of years out of school; and family background.

Prior to the analysis, we will define a small number of confirmatory subgroups, in order to avoid the potential for data mining and the problem of multiple comparisons (e.g., if we examine impacts for a large number of subgroups, some will be statistically significant simply by chance.) Subgroups can be chosen as confirmatory because theory suggests a program difference by a given dimension, because differences in impacts by a given dimension have been found in prior youth evaluations (for example, the Job Corps’ evaluation finding that the program’s impacts were larger and longer-lasting for older youth), or because a given subgroup is of great policy interest. Similarly, the results from the process analysis will help to inform which program features and fidelity measures might be most important to examine. We expect to limit the confirmatory analysis to three subgroups for youth characteristics and three subgroups for program characteristics.

Restricting the analysis to a few confirmatory subgroups does not rule out the possibility of a more exploratory analysis of additional subgroups later in the evaluation. Findings from this analysis, however, would necessarily be more speculative and given less weight in the discussion of program impacts.

e. Effects of program features

In addition to estimating effects for subgroups of programs, defined by particular features, we plan a more formal analysis of how this variation in program components and implementation strength is correlated with impacts. A first step in conducting this analysis will be to determine whether there is significant variation in impacts across sites using standard tests from the literature on multi-level modeling. Having established variation, we propose to use multi-level methods to explore the relationship between site and program characteristics and site-level impacts. This approach would answer the question of whether the strength of implementation or whether certain program features, such as length of MTO or post-secondary services, are associated with larger impacts, holding other program features constant. Findings from the process analysis as well as discussion with DOL and reviewers will guide the selection of program features. The analysis of program features will also depend on how these features are distributed across the programs, something we will learn from the process study and the grantee survey.

In this approach the unit of analysis is the individual for Level One and the sites for Level Two. The site-level impact, then, is allowed to vary with site characteristics (e.g., implementation strength, program components, service contrast, etc.). It is important in this type of analysis to control for the characteristics of the local environment (e.g., the local unemployment rate and the state of the local housing market) and the characteristics of the program participants, since impacts may vary across sites due to these factors. The power of this type of analysis increases as the number of sites increases.⁴² Although our plan to include 83 sites in the evaluation substantially increases our ability to perform this type of analysis, we will need to limit the number of program features to a handful of the most relevant ones.

Equations (1) and (2) specify the analytical approach. In this random coefficients model, the size of the site level impact, β_m, is allowed to vary with site fidelity, the service contrast, program components, and local economic conditions.

Level 1:

(1)

where:

Y_im = outcome of interest for sample member i from site m,

= student level characteristic k for sample member i from site m,

δ_k =a regression coefficient indicating how the outcome variable varies

with sample member characteristic, k,

= dummy variable equal to 1 if sample member i in site m is a program

group member and zero otherwise,

β_m = the program impact at site m for the typical program group member

from the study sample,

α_m = mean outcome level at site m for the typical control group member

from the full study sample,

= an individual-level random error for sample member i from site m.

Level 2:

(2)

=

where:

= a measure of the level of model fidelity in site m,

= a measure of the treatment-control differential (service contrast) in site

m,

= program component variable in site m,

= the economic environment in site m,

µ_m = a random component of the program impact for site m.

In this case, τ_{1 –} τ₄ represent the association between the program’s impact and fidelity, service contrast, program component, and economic environment, respectively, controlling for other characteristics included in equation 2. If τ₁ is statistically significant and positive, for example, stronger site fidelity is associated with larger program impacts.⁴³

An important caveat to the multi-level analysis, however, is that it is non-experimental. The process by which certain sites chose to implement certain program features may not be fully understood, but it is almost certainly not random. For this reason, variation in program characteristics is likely correlated with other unobserved features of the site. Sites that offer youth the ability to earn certification in the construction industry, for example, are likely to differ from other sites in ways that influence the effectiveness of their programs. Nevertheless, we think this analysis can generate useful hypotheses about the source of impacts.

IX. Process Analysis

The process study will address several major topics important to the success of the evaluation. These lines of inquiry will help document: 1) the design of the participating YouthBuild programs, the services they offer to youth in the program groups, and the implementation of these services; 2) the key contextual factors in the local communities that may affect services and outcomes for youth in both research groups; and 3) the characteristics of the youth who participate in the study, the experiences of program group youth in YouthBuild, and the dosage of YouthBuild services they receive.⁴⁴ The process study will also provide data to measure program costs.

1. Key questions
   
   

YouthBuild design, services, and implementation

Design

• What is the organizational and leadership structure of the YouthBuild programs?

• Who are the main referral sources and other partners?

• Does the program offer a high school diploma, a GED, or both? If both, how does it decide which youth will follow each path?

• Does the program own the worksites and does it focus on rehabilitation, new construction, or both?

• How large is the program and how long does it last? When do enrollment cycles begin?

• How do the programs in the impact component of the study compare with other YouthBuild programs?

Services

• What activities occur prior to the Mental Toughness Orientation (MTO)? What services are available during MTO? Has the nature of MTO services changed as a result of the study?

• What is the range of educational and construction training services available? What is the frequency and duration of these services?

• What is the quality of classroom instruction and other key program services?

• What job placement services are available to participants?

• When does the placement process begin? Is it initiated by the participant? By the program? What types of employment are targeted? Are there specific targets for wages/earnings?

• Once an individual is placed, are follow-up services available? What services are offered/delivered? For how long?

Implementation

• What is the proportion of time spent at the construction worksite and in educational activities?

• Did the program experience any serious implementation challenges during the study period? If so, how were they addressed?

• What is the staff to participant ratio? What is the background of staff? What is the program’s general approach to counseling and case management?

• What is the range of youth leadership opportunities available? What is the structure of the Youth Policy Councils?

• To what extent is community service available and emphasized?

• How does the program approach discipline? What are the consequences if a participant fails to attend activities or acts inappropriately?

• To what extent is the program faithful to the established YouthBuild model?

  • ^{How do grantees display respect for youth’s intelligence? How do they give power to youth over their immediate environment? Do they model and instill consistently positive values?}

  • ^{Is the work they provide youth meaningful and important? Does staff exhibit real and patient care for youth’s development? Do youth feel they are protected?}

  • ^{Does the grantee provide family-like support and appreciation? Does staff have high expectations and standards? How does staff balance firmness and compassion in challenging youth?}

Contextual factors

• How does the availability of economic and educational opportunities affect the nature of the YouthBuild program’s design, implementation, and impacts?

• To what extent are there alternative services available within the community? How do they compare with YouthBuild (and, if available, how much do they cost)? Does the program actively refer control group youth to alternative services? Which ones?

Program costs and funding

• What is the cost of the major program components?

• How is the program funded, how has this changed over time, and how do funding constraints affect program services?

Youth experiences and dosage

• What are the characteristics of youth in the research sample and how do these vary by site?

• What “dosage” of the different types of services do program group members receive from YouthBuild?

• What proportion of youth achieves the key program outcomes (earning a credential, finding a job, etc.)?

• How do youth perceive YouthBuild and the relationships they build with staff and peers?

2. Data sources and data collection

The process study will use three main categories of data: site visits, MIS data, and a survey of DOL grantees (described earlier).

Site visits

The evaluation team will conduct one round of site visits to each of the 83 selected programs. These visits will start in early 2012 and take place at many programs close to the time of random assignment. Conducting the visits during this time frame will enable us to document the specific services and strategies that were in place when youth entered the programs.

Following from best practice models of site-visit research,⁴⁵ a key step before the site visits are conducted is for senior team members to develop protocols, or scripts, which will guide the data collection. Our goal for these interviews is to fully document and understand the flow of participants through the program, the range of services provided, and how staff monitors the progress of their participants. A summary list of broad topics and respondents is included in Table 5. Drafts of the protocols are included in Appendix C.

We have developed separate site visit protocols for a range of topics, including:

• Organizational Structure

• Recruitment, Intake, and Assessment

• Mental Toughness Orientation

• Case Management

• Academic Services

• Vocational and Construction Training

• Employment Services

• Youth Leadership and Community Service

• Partnerships

• Alternative Youth Services

MIS data

MIS data will enable us to examine the characteristics of the youth (e.g., volume of participants over time, their demographic attributes) enrolled in YouthBuild and the services they receive (e.g., types of services, sequence and intensity of services, etc.) while in the program. Additionally these data will contain some information on outcomes youth achieve (e.g., employment, earnings, recidivism, educational attainment and post-secondary enrollment, and literacy and numeracy gains) during and subsequent to their program participation though, as we describe below, this will not be our primary source of outcome data for the evaluation.

As noted earlier, MIS data will be available for the DOL-funded programs from DOL’s MIS system and for the CNCS-only-funded programs from YouthBuild USA’s WebSta system. The WebSta data cover the same broad areas as the DOL MIS data, although with less detail. The discussion that follows is based on the breadth of data provided by the DOL MIS. The programs that do not use the DOL MIS would be excluded from the analysis of some measures.

Background information. The MIS collects a range of demographic information. The process study will use these data in aggregate form to describe the types of participants being served across all DOL-funded grantees, as well as for the sample of grantees that is participating in the impact component of the study. By comparing the characteristics of participants in sites randomly selected to participate in the impact component to those not selected for this component, we will be able to assess the representativeness of our sample and confirm that it is reflective of all DOL-funded grantees.⁴⁶

Program participation. The ETA-designed system also contains a wealth of information on the services provided to participants. The services are recorded through the use of start and end dates (as well as expected dates of completion for activities not yet completed) in specific activities, and whether the activity has been completed. Activities include:

• Educational activities, including:

  • Math/reading remediation

  • GED preparation and high school diploma program

  • Post-secondary planning and exploration activities

  • Other education activities

• Workforce-related activities, including:

  • Vocational/occupational skills training services

  • Pre-apprenticeship program

  • On-the-Job Training

  • Other job training activities

  • Subsidized employment and internships

  • Workforce information services

  • Work readiness training

  • Career/Life skills training

  • Other workforce preparation activities

• Community activities, including community service, community involvement, and leadership development activities

• Mentoring

• Health services, including substance abuse services, mental health treatment, emergency medical care and pregnancy leave

• Other supportive services, including transportation, childcare, and follow-up services.

Participant outcomes. Finally, the database also contains substantial information on participant outcomes. We will examine the outcome information in the MIS, both for descriptive purposes and to allow us to compare outcomes for participants in the programs within our sample of 83 sites to those in YouthBuild programs not in the impact component of the study. The outcomes in the MIS include:

• Employment information, including:

• Placement into unsubsidized employment, including benefits, hourly wage, and number of hours worked

• Employment in the first, second, third, and fourth quarters after the program exit quarter

• Hours worked and hourly wages in the first week of the first, second, third, and fourth quarters after the exit

• Educational attainment and credential information, including:

• Attainment of high school diploma, GED or skill certificates

• Changes in standardized test (e.g., TABE) scores⁴⁷

• Changes in educational functioning⁴⁸

• Post-Secondary enrollment information, including:

• Date on which the participant enrolled

• Whether the youth was enrolled in post-secondary education (and the type of school in which the youth was enrolled) in the first, second, third, and fourth quarters after the exit quarter

• Recidivism information, including:

• New convictions

• Arrests

• Re-Arrests or Re-convictions

• Apprenticeship information, including the date on which the youth entered and completed the program, the hourly wage, and the number of hours worked

• Whether the youth obtained a driver’s license

3. Key areas and data sources used

The following sections describe what specific data will be used to address each of the four research topics identified above.

Design, services, and implementation

For the most part, the data needed to explore this topic will be obtained through the grantee survey and semi-structured interviews with YouthBuild managers and staff. In addition to the interviews of YouthBuild staff and their partners, three additional data collection tools will be used by the evaluation team while on site.

• Checklists to rate program quality

Site visitors will employ checklists to rate the overall quality of services, which will include indicators of the intensity and duration of the services, the level of interaction of staff with participants, and the availability and quality of training opportunities. In this instance, our checklist will begin with the sixteen core elements of YouthBuild programs, as developed by YouthBuild USA, which indicate fidelity to the overall YouthBuild model. While some of these sixteen elements can be difficult to measure (e.g., “a path to future opportunity”), incorporating as many of them as possible will enable us to document the fidelity of the various sites to the “ideal” YouthBuild model. We may also include some elements that may not be unique to the YouthBuild model but which may differentiate programs in terms of their overall quality of services. A draft of the program quality checklist is included in Appendix D.

Using a checklist format to score a site on these and other components, site visitors can provide a quantitative measure for each of the programs we visit. These scores may then be used as part of the impact analysis to explore whether better quality services, as evidenced by the presence of more expected elements, is associated with larger impacts.

• Classroom observation checklist to measure instructional quality

We propose to assess in a general way the quality of educational instruction provided in the program. YouthBuild seeks to provide participants with positive classroom environments in which to learn, from teachers who have high expectations and provide individualized attention. Considering that most YouthBuild participants have dropped out of high school, one challenge for YouthBuild teachers is how to engage students in learning the core curriculum subjects and vocational training skills that contribute to their success after graduating from high school or achieving a GED certificate. To meet this challenge, teachers rely on innovative instructional techniques and practices that differ from the more traditional approaches commonly found in students’ former schools.

A checklist will be designed specifically for the proposed study, based on classroom observation tools developed by Teach For America to help monitor its corps of teachers who work with at-risk students in low-income districts. Table 6 presents the types of questions that would be included in the YouthBuild Classroom Observation Checklist. This checklist could be used for both academic and vocational classes.

Thus, the implementation of a basic classroom observation measure would provide useful and consistent information about aspects of instructional quality that are central to YouthBuild. These data can be used to assess how well programs in the study implement their visions for their instructional component. Furthermore, obtaining more detailed information about how successful programs provide classroom instruction will be a useful resource for all the YouthBuild programs.

We should note that we will only observe classroom or vocational instruction for a very brief period (nor more than an hour) on a single day. Thus, we do not intend for the observational checklists to be thorough and exhaustive assessments of the overall quality of instruction, and we recognize their limitations especially in comparing grantees to one another based on this assessment. Rather, we view these assessments as providing some additional information, captured in a consistent format across all sites, that will allow us to develop a better understanding of the range of instructional methods in use by the grantees in our study.

• Focus groups and individual interviews

Finally, in half of the sites we propose to conduct a focus group discussion with program participants in the research sample. In the other half of the sites, we will conduct one-on-one interviews with two youth participants. Sites will be randomly selected for either focus groups or interviews, in order to ensure that the type of data collection used does not vary with site characteristics. We will ask youth about their experiences with the program, to assess its strengths and weaknesses, and about the program’s influence on their knowledge, attitudes and behaviors. Focus group sessions will be conducted with small groups (four-to-eight participants) and will typically last one hour. Site visitors will facilitate the sessions, asking the group to respond to a series of open-ended questions. We anticipate that these sessions will be relatively unstructured in order to allow for open discussion and a candid exchange of ideas. Individual interviews will last approximately 45 minutes.

Contextual Factors

As illustrated in the conceptual framework, the community context – community characteristics that are external to YouthBuild – is likely to affect both the characteristics of the research sample and the supports and opportunities available to youth in both the program and control groups. For example, we are interested in learning about the general labor market characteristics of the community, especially the condition of the construction industry, as well as the status of the local housing market. We are also interested in documenting educational opportunities and the presence of other youth development services that may be available to the control group and those in the program group who drop out of YouthBuild.

Some aspects of the context – for example, general labor market and housing market conditions – are relatively easy to explore using published data (though such data may not provide a detailed picture of the situation in low-income communities). However, other factors will need to be addressed through the unstructured interviews conducted during the site visits. For example, through interviews, we will explore how local education policies affect young people’s opportunities to obtain a high school diploma or a GED and how the local criminal justice context affects what happens if youth are arrested.

Perhaps the most critical issue to investigate is the services that are available to the control group, as well as services available to youth in the program group who drop out of YouthBuild. The 12-month survey will provide information on the types of services respondents in both groups receive, but the precise nature of these services – and, thus, the contrast between the alternative services and YouthBuild – can be difficult to interpret based on closed-ended survey questions. Thus, the process study will attempt to learn about the types of services potentially available to the control group in each community. This is a complex undertaking because the network of services available to youth in a particular community – through school districts, community colleges, faith-based and community organizations, and public agencies – may be quite extensive. In small towns or rural areas, it may be possible to interview one or two alternative service providers. In larger municipalities, it may make more sense to speak with city officials or others who can provide a broad-brush picture of the service environment. Of course, we will also talk to YouthBuild staff about any efforts they make to refer control group members to particular service providers.

Another important goal of these discussions is to obtain some general information on the costs of alternative services. As discussed further below, the cost-effectiveness analysis will need to begin by calculating the net cost of YouthBuild services; this requires some basic information on the costs of the non-YouthBuild services that youth, in particular control group members, report receiving on the 12-month survey.

Youth characteristics, experiences and dosage

Baseline demographic data will be extracted from the MIS to capture youth characteristics. In addition, the MIS data will provide information about services provided to participants and participant outcomes. The 12-month survey will also include a program-group-only module with questions on youth’s experiences in the program.

Program costs and funding

The final objective of the process study is to collect data on the costs of providing YouthBuild services. We will spend time on site collecting cost data from YouthBuild staff to determine what the overall costs are for participants. We propose to collect this information directly from sites, rather than from a single source, such as DOL, because our prior experience has taught us that site-level data collection will be far more detailed and thorough. Many YouthBuild programs combine multiple sources to provide services to youth. Thus, gathering information from only one source may dramatically underestimate the actual costs of program services because it would not include the total costs of the program.

To collect reliable cost data, we will rely upon techniques that have increased our success in the past, including holding pilot discussions with three or four grantees to discuss with them how they allocate and document costs in their program. Such discussions generally yield critical information that better enables us to tailor our cost questions to other grantees.

Based on these initial discussions, we will develop cost worksheets (and instructions for completing them) that can be shared with program administrators in advance of the site visits, so that they have ample time to review what we are seeking and identify how to complete them. Once on site, we will review the worksheets in-person with either the Director of the program, or the primary financial staff person (or both), allowing them to ask questions and to summarize any variations in their own financial systems so we fully understand what their reported costs include. (In many cases, we will need to follow up later by phone to obtain information that was not available during the visit).

After the first several of these visits, we will gather all site visitors together to discuss the primary deviations or limitations uncovered, and modify the worksheets and our approach as necessary. Doing so will better prepare the site visitors for collecting data in as similar and consistent a manner as possible across sites. We anticipate that we will collect data on the following:

• Dollar costs associated with operating the YouthBuild programs and providing program services, including costs paid from the DOL grants as well as from other sources of funding. These will include both direct and indirect expenses, such as:

• ^{Administrative expenses, such as program management, fiscal services, etc.}

• ^{Salaries and fringe benefit costs for line staff or others working directly with participants (e.g., case managers, job developers)}

• ^{Other direct costs, such as payments for supportive services and training, the cost of materials and supplies (including construction materials) directly used to benefit participants, and the cost of youth stipends}

• ^{Indirect expenses, such as facilities rental and general office expenses}

• Costs of other services accessed by program participants that are provided by community partners

• The implied dollar value of costs contributed by participants contributing to the production of housing.

In some prior studies, we have found that not all programs maintain cost data in a manner that can be successfully used for a cost-effectiveness analysis. While we are optimistic that the YouthBuild grantees will be able to provide this information, it is possible that some may not. If we encounter this challenge, we will collect cost information from a subsample of grantees.

X. Cost-Effectiveness Analysis

Assessments of the YouthBuild program will depend not only on the impacts it generates, but also on whether the benefits are deemed large enough to justify the costs. For this reason, we will conduct an analysis of the cost-effectiveness of the program. At the simplest level, a cost-effectiveness calculation can generate an estimate of the cost of serving the average program participant. However, program benefits can also be used to answer questions such as, “what is the average percentage increase in GED attainment per net (or added) dollar invested in YouthBuild; what is the average reduction in incarceration rates per net (or added) dollar invested; and what is the average increase in a measure of social engagement or pro-social behavior per net dollar invested?”

A cost-effectiveness study is warranted, but not necessarily a full-scale benefit-cost assessment, given that many of the benefits are not easily valued in dollar terms. Applying a dollar value to non-economic outcomes is challenging and controversial. In a cost-effectiveness study, in contrast, benefits and costs are both being measured and compared, but no effort is made to place a market value on the benefits that accrue.

The cost analysis will use the cost data collected in the process study to estimate the average gross and net costs of YouthBuild per program participant. Gross cost refers to the value of resources expended to operate the program. These costs will include administrative expenses, staff salaries and fringe benefits, other direct costs (such as materials and supports), and indirect expenses (such as rental and facility expenses). Additional costs will include the cost of other services used by participants, such as substance abuse/mental health counseling.

Our most refined cost-effectiveness analysis will compare the net cost of YouthBuild services (that is, the cost of producing the observed service difference) to the resulting impacts (benefits) generated by the program. Net costs refer to the expenditures on program participants that are above and beyond expenditures for alternative services and supports that would have been made on behalf of program participants in the absence of the program. As discussed earlier, youth in the control group will likely seek out education or employment services on their own. Thus, estimating net costs requires estimating the gross costs incurred by the control group—which will primarily be the costs of their use of relevant services—and subtracting that amount from the total gross costs incurred by the program group. For this purpose, we will use survey data to identify what services control group members access (e.g., education services, employment services) and how those services are paid for, and will conduct supplemental interviews with a sampling of community service organizations and other providers during the site visits to place a dollar value on the services’ costs.

XI. Timeline and Reports

1. Timeline

Figure 3 presents a timeline for the evaluation, showing the expected dates for each major component. Visits to sites and random assignment began in September 2011 and are expected to continue through October 2012. The CNCS sites were selected in January 2012 and are expected to conduct random assignment from about March 2012 through October 2012. The process study site visits are scheduled to begin in early 2012 and the first youth survey (at 12 months after study entry) will be fielded starting in September 2012.

2. Reports

The evaluation will produce three reports.

• Process report. In June 2013, a report describing the findings from the process study will be submitted to ETA. This report will document, for example, the process of recruiting sites for the evaluation, the characteristics of sites that chose to participate, and process of randomly assigning youth to either the program groups or a control group. The report will also discuss the characteristics of youth served, the flow of participants through the programs, the delivery of services, variation in program components and fidelity, youth participation rates, and any challenges to serving participants.

• Interim report. In September 2015, a report describing interim effects of YouthBuild on a range of outcomes will be submitted to ETA. This report will use data from both administrative records and the 12- and 30-month surveys to examine impacts on educational attainment, employment, job characteristics, community involvement, attitudes and aspirations, family structure and living arrangements, and involvement with the criminal justice system. We will also consider examining effects for key subgroups of youth.

• Final report. In March 2017, the final report documenting longer-term impacts of YouthBuild will be submitted to ETA. Effects will be estimated for participation in education and training, the attainment of educational credentials, employment and earnings, criminal justice involvement, family status and living arrangements, risky behaviors, health status and other measures of well-being. This report will also examine effects for key subgroups of youth and present an analysis of the effectiveness of certain program components. Finally, the report will present an analysis of the cost-effectiveness of the program.

The YouthBuild Evaluation Peer Review Panel, in addition to reviewing this design document, will review drafts of the 30- and 48-month impact reports.

1 National Center for Education Statistics, The Condition of Education (Washington: National Center for Education Statistics, 2009), table A-20-1; Center for Labor Market Studies, Left Behind in America: The Nation’s Dropouts Crisis (Boston: Northeastern University Center for Labor Market Studies, 2009).

2 National Center for Education Statistics, Digest of Education Statistics: 2007 (Washington: National Center for Education Statistics, 2008), tables 105 and 106. National Center for Education Statistics, The Condition of Education (Washington: National Center for Education Statistics, 2009), table A-20-1.

3 Economic Policy Institute, The State of Working America.

4Bureau of Labor Statistics, “Youth employment and unemployment in July 2010.” http://www.bls.gov/opub/ted/2010/ted_20100903.htm

5 Bureau of Labor Statistics, “College Enrollment and Work Activity of High School Graduates, News Release.” http://www.bls.gov/news.release/hsgec.t02.htm

6 Neumark, David. (2002). Youth Labor Markets in the United States: Shopping Around vs. Staying Put. The Review of Economics and Statistics 84(3): 462-82.

7 David Hurst, Dana Kelly, and Daniel Princiotta, Educational Attainment of High School Dropouts 8 Years Later (Washington: National Center for Education Statistics, 2004).

8 Heckman, James, John Humphries, and Nicholas Mader. (2010). “The GED.” National Bureau of Economic Research, Working Paper, No. 16064. Murnane, Richard, John Willett, and Kathryn Boudett. (1999). “Do Male Dropouts Benefit from Obtaining a GED, Postsecondary Education, and Training?” Evaluation Review, Vol 23, No. 5.

9 Heckman, James, John Humphries, and Nicholas Mader. (2010). “The GED.” National Bureau of Economic Research, Working Paper, No. 16064

10 DOL YouthBuild MIS.

11 CNCS also funds several YouthBuild programs through grants to State Service Commissions.

12For example, Massachusetts YouthBuild grantees have a line item within the state budget that provides them funding each year.

13Recognizing that a sizable proportion of YouthBuild participants are not interested in construction as a career, a growing number of YouthBuild grantees are offering other forms of vocational training as part of their programs, including landscaping, maintenance, and computer certification (i.e., Microsoft Office certification), among others. The core vocational training of the majority of the programs, however, remains in the construction field. Prior to 2012, DOL grants could only be used to fund YouthBuild slots that offered construction training.

14 The exception is the Youth Incentive Entitlement Pilot Projects, which compared YIEPP sites to other neighborhoods or cities. YIEPP served both in-school and out-of-school youth.

15 Early positive effects observed for women in the study faded by the 54-month point.

16 Schochet, Peter, John Burghardt, and Sheena McConnell. (2006). National Job Corps Study and Longer-Term Follow-up Study: Impact and Benefit-Cost Findings Using Survey and Summary Earnings Records Data. Princeton, NJ: Mathematica Policy Research, Inc.

17 The National Guard Youth ChalleNGe program, the most recent to be evaluated, led to an increase in GED receipt and in college enrollment after 21 months. It is too early to tell whether these effects will lead to impacts on employment and earings.

18Ronald Ferguson, Philip L. Clay, Jason C. Snipes, and Phoebe Roaf. YouthBuild in Developmental Perspective: A Formative Evaluation of the YouthBuild Demonstration Project. September, 1996.

19 GED attainment rates for YouthBuild participants were compared with in-program impacts from several other programs, some of which were estimated with a non-experimental comparison group.

20 A few cases were deemed positive terminations because the youth stayed for the maximum program period, and the remainder were listed as positive, but unspecified.

21 Anne Wright, The YouthBuild Welfare to Work program: Its Outcomes and Policy Implications (Somerville, MA, 2001).

22 Andrew Hahn, Thomas D. Leavitt, Erin McNamara Horvat, James Earl Davis. Life After YouthBuild: 900 YouthBuild Graduates Reflect on their Lives, Dreams and Experiences (Somerville, MA, 2004).

23 Applied Real Estate Analysis, Inc. Evaluation of the YouthBuild Program (Washington, D.C.: U.S. Department of Housing and Urban Development, 2003).

24 For example, Ferguson et al., compared YouthBuild participants’ outcomes to the in-program impacts of other programs, whereas the HUD study compared YouthBuild participants’ outcomes to outcomes for the control groups in other studies.

25 Andrew Hahn and Tom Leavitt, The Efficacy of Education Awards in YouthBuild AmeriCorps Programs (Somerville, MA, 2007)

26 Wally Abrazaldo et al., Evaluation of the YouthBuild Youth Offender Grants: Final Report (Washington, D.C.: U.S. Department of Labor, Employment and Training Administration, 2009).

27 Mike Midling and Jill Leufgen, An Analysis of GED Attainment at YouthBuild AmeriCorps Programs (Oakland, CA, 2010).

28John Tyler, “The General Educational Development Credential: History, Current Research and Directions for Policy and Practice,” in Review of Adult Learning and Literacy, volume 5 (Boston: National Center for the Study of Adult Learning and Literacy, 2005).

29 Heckman and colleagues, for example, find effects on labor market outcomes of measures of locus of control and self-esteem. Heckman, James, Jora Stixrud, and Sergio Urzua (2006). “The Effects of Cognitive and Noncognitive Abilities on Labor Market Outcomes and Social Behavior,” Journal of Labor Economics, Vol. 24 No. 3.

30 An additional two programs received funding to supplement their March 2011 grants. These two programs were not considered part of the May grantee class.

31 In rare instances, for example, a site may agree to participate in the study but ultimately not contribute any youth to the sample if it does not have excess applicants for its program in any of its enrollment cycles. As discussed more fully in section VI, random assignment will take place in a given site when there are more eligible and appropriate applicants than the program can serve.

32 The random assignment ratio will be discussed in section IV.a.

33 MDE’s, which require estimates of the mean and standard deviation of key outcomes, are based on data from similar populations, specifically data from the 34 YouthBuild Youth Offender Grantees and data from the Center for Employment and Training (CET) replication study. Since these are estimates, the actual minimum detectable effects may be smaller or larger than what is shown here.

34 In addition, the impacts reported for Job Corps are based on a program participation rate of 73 percent, meaning that the impacts per participant are somewhat larger.

35 POS cards can be used anywhere that a credit or debit card is accepted, and money can be easily added to the cards upon completion of each survey interview. Sample members keep their POS card for the duration of the study allowing it to function as both a pre-paid and promised incentive (the card can be replaced if it is lost).

36 We will extend the intake period through December 2012 if our sample size target has not been met by October 2012.

37 As discussed in Section VIII.b, because the sample consists of two separately drawn samples of DOL-funded and CNCS only-funded YouthBuild programs, the results from all sites combined may not be generalizable to the full universe of youth served by YouthBuild.

38 Because vocational colleges tend to be relatively under-reported in the NCS data, we will explore the possibility of obtaining data from the U.S. Department of Education indicating whether youth received Pell Grants.

39 In addition, a few questions on the youth’s reasons for applying to YouthBuild and measuring locus of control were added to the DOL MIS for staff to input prior to random assignment. Similar types of constructs were seen as predictive of success in the earlier Ferguson et al., study.

40 Similarly, some programs might only need to enter data into the DOL MIS for a fraction of their participants if many of their program slots are funded by other sources. In these sites, we will consider having them enter the data directly into MDRC’s web-based system.

41 Literacy/numeracy levels will be defined using the basic skills tests that all programs administer to youth. However, not all programs administer this test to all applicants prior to the start of MTO and, thus, prior to random assignment. We will examine effects by literacy/numeracy only for youth in the programs that administer the test prior to random assignment. If the sample of youth in these sites is fairly small, the estimates will be more speculative.

42 Greenberg, David, Robert Meyer, Charles Michalopoulos, and Michael Wiseman (2003). “Explaining Variation in the Effects of Welfare-to-Work Programs.” Evaluation Review, Vol 27, No. 4.

43 In the above specification, each domain (fidelity, etc.) is listed as only one variable for ease of presentation. However, each domain will consist of an array of variables.

44 For ease of presentation, the components of the study are grouped slightly differently in this section than in the conceptual framework.

45 Robert K. Yin. 2002. Case Study Research: Design and Methods (Third Edition). Thousand Oaks, CA: Sage Publications.

46 As noted earlier, our exclusion of three sites from the selection universe of 74 sites should not affect our ability to generalize to all YouthBuild DOL grantees, since these three programs represent less than five percent of programs and less than five percent of YouthBuild slots.

47 While most programs use the TABE to determine basic skill levels, some programs use the Comprehensive Adult Student Assessment Systems (CASAS) test.

48 Educational functioning is determined by test scores and includes the following categories: beginning, low intermediate, high intermediate, low adult, and high adult.