Bundling OMB Part A final

SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT SUBMISSIONS

 

A. Justification

A1.  Need for Information Collection

The Evaluation Bundling Approach

CNCS’ 2011-2015 Strategic Plan¹ emphasizes the importance of supporting evidence-based programs and increasing the capacity of the national service network as a whole to address program impacts, reflecting the broader evidence movement in the federal government and philanthropic sector. To that end, CNCS believes in developing and investing in creative strategies that establish an evidence base for national service interventions, particularly those that demonstrate impact and justify programmatic investment while maximizing financial and human capital resources.

According to authorizing legislation², CNCS’s AmeriCorps State and National (ACSN) grantees are required to conduct evaluations that assess their programs and interventions. As required by 45 CFR § 2522.710, those programs receiving $500,000 or more in funding must conduct an impact evaluation, executed by an external evaluator; programs receiving less than $500,000 per year may conduct an implementation or impact evaluation. The vast majority of the evaluations conducted by this latter category of grantees, who currently represent about 85 percent of AmeriCorps programs, are not impact evaluations. While these “small” grantees frequently cite resource constraints as a reason they avoid conducting impact evaluations, small sample sizes, low evaluation capacity, and lack of technical expertise are also barriers to impact evaluation for many of these programs. CNCS’ current evaluation capacity building initiative has generated resources and tools³ to assist grantees of any funding size in planning and conducting program evaluations, but nonetheless, limited evaluation capacity persists among these small ACSN programs.

The ACSN Opportunity Youth Evaluation Bundling project is an effort to generate rigorous outcome evidence, leverage limited evaluation resources, and build the evaluation capacity of smaller ACSN grantees. At the same time, it is an effort to test a novel evaluation approach that has the potential to enhance the ability of any small program to gather and utilize outcome data. This “bundling” approach also affords individual programs an opportunity to take advantage of pooled resources, and promotes capacity building within and among these programs. Most importantly, the bundling process can aggregate participants across programs to attain a study sample of sufficient size to support an impact evaluation with a comparison group. This can provide a rigorous assessment of evidence of effectiveness.

Evaluation “bundling” seeks to obtain strong evidence of program impacts by grouping programs with common outcomes and similar interventions together to create a larger unit based on those commonalities, which is then evaluated using an appropriate impact evaluation methodology. In this iteration, the evaluation design will be a multi-site, quasi-experimental study to draw causal conclusions about the overall impact of nine AmeriCorps State and National interventions representing 20 distinct sites. The programs selected for this evaluation were chosen because their program activities, outcomes, measures, information needs, evaluation experience and resources related to their work with opportunity youth were similar enough to suggest that individual programs could contribute to, and would gain from, a collaborative and participatory impact evaluation process.

In this evaluation, the “bundle” will be composed of ACSN grantees engaging opportunity youth (OY) as members. The sample for this study will be a cohort of those opportunity youth members who participated in one of the nine participating ACSN opportunity youth programs during a one-year enrollment period, and a group of comparison participants who either applied to the program during the enrollment year, but declined to participate or were not selected for participation due to oversubscription. Participants will be surveyed in order to assess the impacts of AmeriCorps service on opportunity youth in three key areas: educational attainment, employment, and connection to the community. A detailed schedule of activities is available in Table 2 in A.16.

Background

The Obama Administration has identified opportunity youth as a targeted population for assistance in both educational attainment and reducing unemployment, as well as supporting several initiatives to address issues facing opportunity youth (White House 2012). Opportunity youth, or disconnected youth, are defined in the Consolidated Appropriations Act of 2014 as “individuals between the ages of 14 and 24 who are low income and either homeless, in foster care, involved in the juvenile justice system, unemployed, or not enrolled in or at risk of dropping out of an educational institution.” For the purpose of the evaluation, opportunity youth are defined as economically disadvantaged individuals age 16-24 that are disconnected from school or work prior to service. This population is called opportunity youth because of the potential and opportunity they present to the nation.

It is estimated that in 2011, opportunity youth cost approximately $93 billion in lost revenues and increase social services (Bridgewater and Mason-Elder 2012).

By investing in these youth, both taxpayers and the larger economy can benefit from more opportunity youth joining the workforce and reducing reliance on government assistance. The grantees selected for this bundled evaluation are unique in that their programs intentionally engage opportunity youth as AmeriCorps members, involving them in meaningful community service work while providing critical support and development opportunities in education, employment, and community connection. This approach allows the opportunity youth members a chance to serve their communities as well as receive services from the grantees such as GED coaching or high school completion classes, and hard and soft skill training for employment. While studies of youth demonstrate that individual interventions, some involving volunteering, in the outcome areas of education, employment, and community engagement produce change⁴, we hypothesize that it is the unique synergy of our grantee programs’ multiple interventions, meaningful relationships, and engagement with the community that causes improvements in outcomes for our opportunity youth members.

Literature Review

Currently, 6.7 million young adults between the ages of 16 and 24 are neither employed in the labor market nor enrolled in school (Aspen Institute, 2014). These disconnected youth include a disproportionate number of black (32%) and Hispanic (22%) youth (Annie E. Casey Foundation, 2013). Disconnected youth are demographically diverse and include returning veterans, teen parents, immigrants, the homeless (including LGBT youth), those with mental and/or physical challenges, and those who are incarcerated (Belfield, Levin, & Rosen, 2012). Some of these disconnected youth grow discouraged and drop out of college; some have participated in multiple job-training programs but never received the support to get into better-paying jobs with a prospect for advancement. These disconnected youth are called "opportunity youth" to reflect their perseverance in seeking viable pathways to economic stability and as a reminder that supporting their goals is an important step toward a brighter future for employers and the nation (Aspen Institute, 2014; Belfield, Levin, & Rosen, 2012; Corcoran et al., 2012; Stuart Foundation, 2011).

Research on opportunity youth illustrates the system failures that inhibit reconnecting these youth to education and career resources. For instance, criminal justice policies focusing on punishment over rehabilitation often do not provide sufficient "on-ramps" for disconnected youth to get back on track (Hjalmarsson, 2008; Lochner & Moretti, 2004; Sweeten, 2006). In addition, high schools and postsecondary schools may lack services such as childcare and adequate transportation, which are vital ancillary supports necessary for these youth to successfully complete their education. There may also be a mismatch between students’ goals, the available education options, and available jobs in the students’ communities (Fernandes & Gabe, 2009). Some students may lack the relationships and encouragement, within and outside of schools, to support them in the challenges they face.

In many areas, the school, social services, and government systems, along with the philanthropic sector, operate in silos that are uncoordinated and unable to support disconnected youth in achieving their goals (Belfield, Levin, & Rosen, 2012; Corcoran et al., 2012). Thus, programs supporting opportunity youth must strengthen the connections across systems and sectors and add value to re-engage these disengaged youth (e.g., school districts, juvenile justice and foster care systems, foundations and nonprofits, postsecondary education systems). Only through this more coordinated approach can opportunity youth be supported in their continued education, including highlighting career pathways, and fostering their potential as mentors and community leaders (Belfield, Levin, & Rosen, 2012; Gewertz, 2011; Hair et al., 2009). Recently, collective impact approaches have shown promise in coordinating business, education, and philanthropic efforts to support disconnected youth (Corcoran et al., 2012). In addition to supporting community-level knowledge of opportunity youth, communities can develop resource mobilization centers where disconnected youth can be assessed and guided toward local education and employment avenues that fit with their goals and life circumstances. These centers can offer information about "on-ramps" such as apprenticeship programs and other supports (e.g., childcare, immigration support) that can be customized to support an opportunity youth's unique situation (Belfield, Levin, and Rosen, 2012; Kirsch, Yamamoto, and Sum, 2007; Vericker et al., 2009).

The nature of the AmeriCorps service experience, specifically the intensity and duration of the service period; topical focus and exposure to community problems; and supplemental member training and development can develop individuals into empowered, engaged citizens. It is likely that this will be more pronounced for opportunity youth and disadvantaged youth serving as members.

Outcome: Education Advancement

Developing pathways for the opportunity youth who are having educational difficulties or dropped out of school is a core component of education support (Belfield, Levin, & Rosen, 2012; Bird et al., 2014; Corcoran et al., 2012; Gennetian, 2012; Gewertz, 2011; Julian & Kominski, 2011; Rutschow and Crary-Ross, 2014). Alternative schools and recuperative schools have the potential to support disconnected youth more effectively than traditional high schools (Lochner & Moretti, 2004). These nontraditional schools may also serve to connect opportunity youth to employment networks that will be valuable once their education and training is complete. Education support also encompasses building flexibility into education policy so that funding for the challenges faced by opportunity youth is more easily accessed (Belfield, Levin, & Rosen, 2012; Corcoran et al., 2012). In this way, improving the education system's flexibility can facilitate a more holistic approach to supporting opportunity youth.

Studies of low-income youth note that education supports can function to lift them and their families out of poverty and offer a buffer for the next generation against the threat of poverty (Bird et al., 2014; Martin and Broadus, 2013). Bird’s work argues that educational advancement in one generation can provide momentum, knowledge, and resources for future generations to attain post-secondary degrees and succeed in the labor market (Bird et al., 2014). Kemple and Willner (2008) found strong support for the benefits of educational support for students in low-performing school districts at risk of dropping out. Their randomized trial revealed that monthly earnings, months employed, hours worked per week, and hourly wages were higher among those who received support and graduated compared to similar youth who did not graduate. Although numerous studies of youth in poverty suggest that education programs improve youth’s life chances, research on such programs specific to opportunity youth are just beginning to surface. Martin and Broadus's (2014) randomized control trial of the GED Bridge program found that program participants were more likely to complete the GED course and pass the GED exam than students who receive treatment as usual. Bridgeland, and Mason-Elder (2012) found similar results for youth in poverty who were receiving education support (many of whom were opportunity youth) were more likely to attain their high school degree and earn college credits than youth who were not getting support.

Outcome: Employment and Career Readiness

Career development among opportunity youth requires the assistance of local colleges and businesses, nonprofits, and other stakeholders in providing channels between education and labor market needs (Allen et al., 2014; Belfield, Levin, & Rosen, 2012; Mortimer, 2010; Snyder & Dillow, 2011; Vericker et al., 2009; Wallace, 2014). These entities need to develop in ways that facilitate smooth transitions from education to employment. Highlighting career pathways includes expanding career paths through addressing business leaders' negative perceptions about hiring opportunity youth (e.g., that disconnected youth do not have skills to contribute to the workforce, that they are ‘risky’ employees) (Belfield, Levin, & Rosen, 2012; Corcoran et al., 2012; Sum et al., 2014). Creating internships and apprenticeships for disconnected youth can give employers the chance to learn about a youth's likelihood of contributing to their business or organization before offering permanent employment. Collective impact approaches to support opportunity youth have carved new grooves into local economies to bridge gaps between disconnected youth and the employment opportunities they seek. These approaches utilize businesses, government, nonprofit, and local schools to work together in local communities to give disconnected youth access to credentials and connections to contribute to local economies and support themselves and their families (Allen et al., 2014; Corcoran et al., 2012; Wallace, 2014).

Opportunity youth often find themselves working in low-pay jobs with little room for advancement. Internship and training programs are avenues that have shown promise in supporting opportunity youth to improve their career prospects (Mortimer, 2010; Sum et al., 2014). The Urban Alliance has been particularly successful in supporting opportunity youth through internships (Theodos et al., 2014). A recent outcome study of the Urban Alliance’s internship program for at-risk youth showed that more than 90 percent of participants reported positive feelings about working in professional office environments, noting that the internships provided both hard and soft skill development necessary to succeed in these jobs (Theodos et al., 2014). An experimental study of the Job Corps program, an initiative that provides education and employment internships to disadvantaged youth (ages 16 to 24) found that participants had higher annual earnings than the control participants three and four years after completing the program (Schochet et al., 2008). Other programs have addressed those in need, some of whom are opportunity youth, through collaborations between local governments and businesses. The Transitional Work Corporation (TWC) participated in a randomized control trial that showed statistically significant improvement in employment in unsubsidized jobs, increases in earnings, and less reliance on food stamps for the treatment groups versus control participants who were receiving government assistance (Bloom et al., 2009).

Outcome: Community Connection

A third factor related to opportunity youth is encouraging commitment to community connection (Corcoran et al., 2012). Some of the research posits that supports that assist opportunity youth in knowing what resources are available to assist them are keys to supporting connections to the community, e.g., financial support and literacy, transitional education services (Stuart Foundation, 2011). Community reengagement success often hinges on the ways in which communities organize resources and systems (Bridgeland and Milano, 2012; Hanleybrown, Kania, and Kramer, M. 2012). A study by the Center for Law and Social Policy (2003) found that urban centers with established youth resource delivery systems had positive effects on the uptake of community services among disconnected youth. Particularly for opportunity youth who have been incarcerated. Having support in community reentry and assistance with housing and other basic needs may provide exposure to tangential education and employment supports (Allen et al., 2014; Corcoran et al., 2012; Osypuk et al., 2013; Wallace, 2014).

Research suggests that opportunity youth who have succeeded in connecting with their communities can become key resources for other disconnected youth trying to find their own way (Corcoran et al., 2012; Wallace, 2014). Thus, educators and others who support opportunity youth can encourage them to become role models and offer instructive advice for new cohorts of youth, including how to navigate the education system (Hanleybrown, Kania, & Framer, 2012). Supporting disconnected youth can inspire self-confidence and instill a sense of leadership in reconnected youth. More established opportunity youth can take their experience and inform their communities about best practices and areas of improvement for the newly disconnected youth (Bird, 2013; Corcoran et al., 2012).

Grantee Participants

Given the importance of common outcomes and similar interventions to the viability of the bundling approach, CNCS and JBS designed a rigorous grantee screening process to select and review programs for participation in the treatment group. Using a spreadsheet of grantees selecting opportunity youth performance measures, CNCS collected an initial list of possible candidates to recruit based on those grantees listing at least one opportunity youth performance measure and recording that at least 75% of the program’s members were made up of opportunity youth. This list was then vetted with the AmeriCorps State and National program officer if they were a national direct grantee, and with the state commission if they were a sub-grantee; some grantees were weeded out at this point due to lack of compliance in other areas of programming or due to program officer or commission assessment of their capacity to participate. The remaining grantees were then recruited for participation and screened for suitability using a screening rubric of criteria needed for a successful bundle (the results of this screening process are contained in Attachment B-1 in Justification Part B). Consistency in these features for programs comprising the bundle also helps to ensure that meaningful data can be collected and aggregated across the programs to measure the outcomes and impacts of these programs at a collective level. Inconsistency across the bundle would prevent aggregate assessment of outcomes/impacts.

After being recruited to the study, grantees were engaged in a series of capacity building technical assistance calls, both one-on-one with the project’s evaluation contractor, JBS International, and in group format via conference call. The purposes of these calls were to refine individual program theories of change and logic models, and to create a common theory of change and logic model. These segued into facilitated discussions to develop research questions and an evaluation design. These are each described more below.

Research Questions

Based on the body of literature reviewed in each outcome area above, and based on participating program reflection, this evaluation will seek to answer the following research question:

Following program participation, do opportunity youth participating in AmeriCorps programs as members significantly improve on various measures of educational and career attainment, and become more connected to their communities, as compared to matched opportunity youth who do not participate in national service programs?

This research question compares opportunity youth participating in AmeriCorps to those not participating in AmeriCorps. Furthermore, at follow-up, compared to matched opportunity youth who are not AmeriCorps members, do opportunity youth AmeriCorps members have:

• Increased educational attainment?

• Increased number of employment-based skills, including both hard and soft skills?

• Increased job attainment and retention?

• Improved, positive civic engagement?

• Decreased likelihood of engagement with criminal justice system?

Logic Model

To support the evaluation design process, a combined theory of change and two logic models (one comprehensive and one simplified) were created. While the comprehensive logic model speaks to local needs driving each program’s approaches to engaging their opportunity youth members, the simplified group logic model depicts a core set of program services and intended benefits that are nearly universal across the programs participating in the evaluation. These outcomes are the target of this evaluation.

This simplified logic model (Attachment A-1) facilitated the creation of a survey instrument that could be used with all the programs in the bundle, as well as with their comparison group youth, to examine the effect of service experience (or lack thereof) on opportunity youth in the outcome areas of education, employment, and community engagement. Survey items were reviewed by programs and piloted with current opportunity youth members, and assess changes in desired outcomes in knowledge, attitudes, behavior, and skills in each of the three outcome areas.

Instrument Development

The survey instrument development process began with review of the early drafts of the program logic models and by considering three primary outcome areas of interest to CNCS: educational attainment, employment attainment, and civic engagement. Key outcome domains were also identified from program logic models and the group calls related to program outcomes. Existing instruments with items relating to these domains were identified from the literature (specific sources are noted in the “Survey Items” section of this memo), and the draft items were reviewed by CNCS staff. Simultaneously, the bundle program group and the JBS evaluation team reviewed the draft instrument alongside the programs’ outcomes (as depicted in the simplified program logic model, A-1). Based on this review and CNCS input, the initial draft was revised and submitted, along with a table of specifications, to CNCS for review. JBS shared a draft of the survey instrument with the programs participating in the bundled evaluation, and collected feedback during a group conference call. During the call, JBS obtained program feedback on the draft instrument. Several programs also provided written feedback after the call.

Overall, the programs indicated that the instrument contained items consistent with their program activities and intended outcomes. Some programs expressed a desire for the survey to be shorter. Programs also provided suggestions to render specific items more relevant to their members (e.g., removing references to college major which would not apply to many respondents). Programs were in agreement that the preferred survey modality for members would be a paper form administered in a group setting proctored by program staff; a mix of modes was recommended for comparison group youth.

Survey items were created to collect two categories of data: (1) demographic characteristics to be used in propensity score matching and/or entered as covariates in the final analysis; and (2) data regarding outcomes and impacts shared by the bundle and illustrated in the simplified consolidated logic model. In developing the survey, a number of existing tools were identified and examined for potential use in collecting data on demographics, outcomes, and impacts. Instruments and resources examined or used as the basis for early drafts of the survey, and used in the final version of the survey, include the Youth Risk Behavior Surveillance System (YRBSS) survey, the Communities that Care (CTC) survey, the Sense of Community Index (SCI), the Short Grit Scale, and the CDC’s (2005) Compendium of Assessment Tools for Measuring Violence-Related Attitudes, Behaviors and Influences Among Youths.

The sources of demographic characteristics are summarized in Table 1 of Attachment A-2. The AmeriCorps application was used as the basis for items assessing age, race, criminal history, education, and area of residence. Additional sources used include the AmeriCorps Alumni survey (Gender, Caregiver status, Military status), the National Longitudinal Study of Youth (NLSY; education, employment), the Gates Foundation survey (employment), the DC Alliance of Youth Advocates survey (DCAYA; employment), and the Behavioral Risk Factor Surveillance System survey (BRFSS; physical/mental disability). Both the NLSY and BRFSS are national, federally administered surveys that have been extensively validated.

In several cases, minor adjustments were made to questions based on program feedback. In two cases (criminal history, area of residence), the questions were shortened in order to keep the survey brief, and because it was determined that the extensive information included in the AmeriCorps application was not necessary for the purposes of the survey.

New items were created to track two variables: financial support and similar services. The first question asks participants to report whether they are accessing state or government supports in multiple categories (e.g., health care, food assistance). The second asks participants whether they are participating in programs or receiving services similar to those provided by bundling grantees. This question was created in response to concerns from program participants that the study account for any potential contamination effects (e.g., if comparison group members enrolled in alternate AmeriCorps programs).

The sources of items assessing outcomes and impacts are summarized in Table 2 of Attachment A-2. In cases where the same items are used in both the matching process and to measure outcomes (e.g., education level), the process for creating or deriving items will not be described again. The remaining items were primarily taken or adapted from the following sources: the AmeriCorps Exit survey, the AmeriCorps Alumni survey, the Career Decision Self-Efficacy scale (CDSE), and the Career Competencies Indicator (CCI). The redesigned AmeriCorps Exit survey was used as a source for questions regarding sense of community, civic engagement, and self-efficacy. The latter includes a community self-efficacy scale originally presented and validated as the Competencies for Civic Action scale (CCA; Flanagan, 2007). One additional question regarding civic engagement was taken from the AmeriCorps Alumni survey.

The CDSE, a validated measure (Betz, Klein, & Taylor, 1996), was adapted based on feedback from bundling participants, in order to make it more relevant to respondents. The original scale contained several items related to self-efficacy that were applicable only to those interested in obtaining a four-year college degree (e.g., questions about choosing a college major). Since this is not necessarily the goal for opportunity youth AmeriCorps members, many of these items were removed and replaced with items more relevant to the target population, such as an item asking about vocational certification. For the sake of brevity, new additional items assessing self-efficacy/knowledge of community resources were added from the adapted CDSE, as an alternative to creating a separate self-efficacy scale related to the ability to access community resources.

The CCI, another validated measure (Francis-Smythe, Haase, & Steele, 2013), was used as the basis for questions regarding attitudes toward maintaining employment. The question used was derived from the Career Goal Setting portion of the measure, as this portion was demonstrated to be linked to later career satisfaction and income. For the sake of brevity, several items were eliminated from this sub-scale and a new item measuring attitudes toward postsecondary education was added to this scale.

Two new items were created to assess behaviors. The first was a checklist of items related to education, employment, and access to community resources; respondents are asked to indicate whether they participated in a given behavior on the list (e.g., revising a resume). The second asks participants if they’ve looked for a selection of positions (e.g., part time work) in the last six months. One additional item was added to assess access to community resources. This item asks participants to indicate if they are using or have used a checklist list of community supports (e.g., food bank, job center).

All items in the survey were pilot tested. This process is extensively described in B.4, and in the attached Pilot Testing Memo. Variability of each item and the effect of modality on survey responses were assessed. A subset of pilot survey respondents completed cognitive interviews; the resultant feedback was systematically analyzed to ensure that survey items are easy to understand and that any problems with comprehensibility are addressed.

The survey will be administered to opportunity youth AmeriCorps members and comparison group members at three points in time: first, when members enter the AmeriCorps program; second, when members exit the program; and finally, 6 months after program exit. The survey will be available in three modes: online, paper-and-pencil self-administered survey, and as a computer-assisted telephone interview (CATI). Please see Table 2 in A.16 for a timeline of activities.

CNCS recognizes that this approach to program evaluation is novel, and has kept careful records of the processes involved in planning and implementing this project, beginning with a proof of concept feasibility study conducted in 2013. Due to space considerations in this Justification, documentation of participating grantee identification, recruitment, and screening processes, as well as of theory of change and logic model development, instrument development, capacity building activities, and coordination of project logistics is available to OMB in memo format as needed. A full evaluation plan is also available, though much of the information in the evaluation plan is contained in this Justification.

A2.  Indicate how, by whom, and for what purpose the information is to be used.

It is anticipated that the primary users of this data will be stakeholders internal to CNCS. Stakeholders in the AmeriCorps program offices and in the CEO’s office will be able to use this valuable data to make decisions about program and service experience improvement for opportunity youth. Relatedly, participating programs and their State Service Commissions will be able to review their programming to make meaningful changes to their interventions based on the evaluation report and the descriptive results specific to their program.

External stakeholders, such as researchers and federal partners may also find value in the evaluation’s results as they endeavor to implement evaluations of small programs, leverage evaluation resources, and to conduct focused research on national service interventions. These groups may be supplied with data reports as is relevant and necessary. When reporting to any stakeholder group, CNCS’ Office of Research and Evaluation (R&E) and the evaluation contractor will strive to ensure appropriate use of results through explaining the strengths and limitations of the data and any analyses.

 

 A3.  Minimize Burden: Use of Improved Technology to Reduce Burden

To minimize burden, we will offer the survey in three formats: paper, online, and telephone. While a paper survey is most convenient for treatment group participants, who can easily complete the survey as part of on- and off-boarding activities, offering additional online and telephone options allows for flexibility for individual respondents. This is particularly important for comparison group members, who’s access to any given survey modality may vary substantially from person to person. Additionally email and telephone reminders will be administered to all study participants.

A4.  Non-Duplication

There are no other sources of information by which CNCS can meet the purposes described in A2. While CNCS is actively surveying members about educational plans and their experiences with career preparation provided by their programs via the recently redesigned Member Exit Survey, there is no way to extract the surveys of opportunity youth participating in any given program. Additionally, the questions in this proposed survey cover the outcome areas of interest in greater depth and breadth, and have been specifically tailored to the population under study.

 

A5.  Minimizing for economic burden for small businesses or other small entities.

This collection of information does not impact small businesses because they are not eligible to be grantees or AmeriCorps members.

A6.  Consequences of the collection if not conducted, conducted less frequently, as well as any technical or legal obstacles to reducing burden.

 

The evaluation bundling project is both an evaluation capacity building initiative and an opportunity to build rigorous impact evidence for some of CNCS’ smaller programs. An important consequence of not following through with data collection for this project would be that CNCS would lose the chance to build grantee evaluation capacity through experiential learning. Grantees would subsequently lose the opportunity to experience the process of undergoing a rigorous impact evaluation, assisted by an external evaluator many could not procure on their own. Further, they would not gain learnings from the process that could bolster their ability to complete such types of rigorous evaluations in the future. Additionally CNCS would not be informed about the efficacy of these grantees’ interventions. Given the pace of accumulation of such evidence via grantee evaluation reports, it would likely be many years before CNCS would have a comparable level of information about these interventions.

A7.  Special circumstances that would cause information collection to be collected in a manner requiring respondents to report more often than quarterly; report in fewer than 30 days after receipt of the request; submit more than an original and two copies; retain records for more than three years; and other ways specified in the Instructions focused on statistical methods, confidentially, and proprietary trade secrets.

There are no special circumstances that would require the collection of information in these ways.

A8.  Provide copy and identify the date and page number of publication in the Federal Register of the Agency’s notice. Summarize comments received and actions taken in response to comments. Specifically address comments received on cost and hour burden.

The 60 day Notice soliciting comments was published on Friday, January 30, 2015 on page 5093. One comment was received but was not responsive to the posting and therefore was not addressed.

 

 A9.  Payment to Respondents

Although participation in the project is voluntary, respondents are likely to perceive a time cost and burden associated with their participation. The use of incentives to increase response rates has been well documented by Dillman (2000), whose findings indicate that the use of incentives has a positive impact on increasing response rates, with no adverse effects on reliability (Dillman, 2000). Maximizing response rates is a key priority for this project, particularly given the disengaged, hard-to-reach nature of the population being studied. Gift cards from a major retailer (e.g. Target) will be used as an incentive for opportunity youth to complete the survey. Both treatment and comparison group respondents will be given a $10 gift card after completing the pre-test and again after completing the post-test survey. Given the difficult-to-track nature of the population under study, as well as the length of time between post-test and follow-up post-test, a $20 gift card will be given after completing the six month follow-up post-test to increase the likelihood of obtaining adequate response rates.

  A10.  Assurance of Confidentiality and its basis in statute, regulation, or agency policy.

Members’ responses to this information collection will remain private to the extent permitted by law.

We will make clear that we intend to keep information private and not share individual responses; the basis for the assurance of privacy is from the privacy statement in the survey invitation, consent forms, and instrument (see Attachments A-1, B-4-6, and B-12 in Justification Part B). Measures will be taken by the evaluation contractor to remove key identifiers prior to data analysis, so that individual responses cannot be linked to a specific individual. Additionally, all analyses, summaries, or briefings will be presented at the aggregate level and it will not be possible to identify individual respondents in any material that is presented.

The survey data will be stored on the evaluation contractor’s secure servers, which are protected by a firewall that monitors and evaluates all attempted connections from the Internet. Access to any data with identifying information will be limited only to evaluation contractor staff directly working on the survey.

A11.  Sensitive Questions

The information collection does not include questions of a sensitive nature.

 

 A12. Hour burden of the collection

We expect approximately 1266 respondents to take the survey. The survey will be administered to respondents at three points in time: at the beginning of their term of service, at the end of their term of service, and six months after completing their term of service; for comparison group respondents, the survey will be administered at an equivalent time period. The frequency of response will not be greater than three times in a calendar year, and should amount to approximately 20 minutes of effort per respondent. The estimated total burden hours across the life of the project are 1266 hours.

Table 1: Total Project Burden Hours

Time per response	Number of responses per individual	Total time per individual	Number of individuals
.33 hours	3	1 hour	1266
		Total burden	1266 hours

 A13. Cost burden to the respondent

There are no direct costs to respondents other than their time to participate in the study.

 

A14. Cost to Government

This study involves a one-time cost to the Federal Government totaling approximately $550,000. The cost of the existing contract for the project planning year is $96,142.07. The anticipated cost of contracts to cover the remaining two and a half years of the project are estimated to total $400,000. The first year contract includes recruiting and screening bundle participants (ACSN grantees), logic model development, drafting and piloting the survey instrument, recruiting the comparison group, completing an evaluation plan, and one-on-one technical assistance with grantees. The second year contract includes fielding the instrument for both the pre-test and the initial post-test and data collection. The third year contract includes fielding the instrument for the follow-up post-test and data collection, as well as analysis and reporting. A six month contract may be added if follow up post-tests and analysis extend beyond the third year contract timeframe.

A15. Reasons for program changes or adjustments in burden or cost.

Not applicable.

 

 A16. Plans for Publication, Analysis, and Schedule

Time Schedule

The bundling project is composed of a planning period (roughly September, 2014-July, 2015), a data collection period (roughly August, 2015-December, 2016), a follow up period (roughly January, 2017-June, 2017), a data analysis period (roughly June, 2017-August, 2017), and a reporting period (August, 2017). Table 2 shows an overall project timeline tailored to each evaluation stakeholder’s activities.

Table 2: Opportunity Youth Evaluation Bundling Project: Overall Timeline for Grantee, CNCS and Evaluation Contractor Activities

Evaluation Step	Month	Activity
Evaluation planning and design	November (2014)	Participate in informational call; confirm participation and organize materials
	December	Participate in 4 evaluation planning calls
	January (2015)	Participate in 1 evaluation planning call; review evaluation plan and instrument
Pilot testing	February-June	Participate in pilot testing; participate in additional evaluation technical assistance (both one-on-one and with the larger group); complete individualized comparison group recruitment plans; finalize survey instrument and administration protocol; finalize evaluation plan; begin clearance process; IRB review
Prepare for data collection	July	Prepare for data collection and comparison group recruitment
	August	Prepare for data collection and comparison group recruitment; Implement pre-test*
Implement evaluation	September	Regular programming
	October	Participate in reflection and lessons learned process; Regular programming
	November 2015-March 2016	Regular programming
	April	Prepare for data collection
	May	Implement post-test*
	June-July	Participate in reflection and lessons learned process
	August-December	Regular programming
	January-June (2017)	Prepare for and implement follow up test data collection
Analysis	June-July	Conduct and review analysis
Reporting, dissemination, and next steps	August	Reporting to stakeholders; identifying areas for program improvement

*Timing may differ for some programs.

Publication Plans

The results of the survey will be tabulated and analyzed by the evaluation contractor selected to implement the evaluation plan designed in the planning year; individual respondents will not be identified by their responses in analyses.

Reporting and dissemination of results will be mainly in the form of written memos and reports with supporting graphs and tables on key data points. For internal stakeholders at CNCS and the selected grantee programs, these will be focused on information relevant to program improvement and enhancement, or on topics relevant to the particular stakeholder group. For external stakeholders, an evaluation report may be made available detailing the steps taken to conduct the evaluation and presenting the results; an emphasis will be made on the strengths and limitations of the data and corresponding analyses to ensure appropriate use of results. The data gathered in this survey may be utilized in analysis and planning work for other program evaluations and research projects conducted by CNCS as applicable. 

Analysis Plan

A detailed plan describing the analysis to be conducted on study data is described in Justification Part B. Quantitative analyses of survey data will be conducted in order to document the impacts of AmeriCorps service on opportunity youth specifically in regards to educational attainment, employment, and increased connection with the community. The results of these findings are primarily for internal use, but may be shared with key government policy and management officials, CNCS and program staff, and the public.

Data Set Up, Cleaning, and Non-Response Bias Analysis

• Data set up and cleaning. Once each survey period has closed for a particular site, the data will be downloaded and cleaned, applying any post-coding as needed for the analysis. Data files will be produced in restricted and public-use formats.

• Response rates and nonresponse bias analysis. The data analysis will calculate response rates (per OMB’s Standards and Guidelines for Statistical Surveys) for the overall sample and for all relevant subgroups to identify any limits of the study’s representativeness of opportunity youth serving in AmeriCorps programs. If the response rate is below 80 percent, a nonresponse and response bias analysis for individual survey items will be conducted.

• Response frequencies. Response frequencies (i.e., counts and percentages) for each item and the mean, median, mode, standard deviation, and minimum and maximum values for every survey questions will be calculated. Frequencies and descriptive statistics will be calculated for all participants combined and for each cohort and program separately.

• Analysis for reporting: Basic descriptive analyses (including frequencies, percentages, means, and standard deviations) will be calculated, along with impact analyses using regression analysis. Table 3 details how the survey data will be used to assess the impact of AmeriCorps service on opportunity youth in three key areas defined by the project’s research questions. A thorough description of these analyses is provided in Justification Part B.

Table 3. Research Questions and Proposed Analysis

Research Question	Relevant survey items (See Attachment A-1 in Part B for survey)	Analytical methods
In what ways does participating in AmeriCorps impact educational attainment for opportunity youth members?	Educational Attainment Support: 7, 8, 9, 17 j-o, 21 f-j, Changing Behaviors and Attitudes: 20 c Demographics: 7, 8, 9	Regression analysis
In what ways does participating in AmeriCorps impact employment for opportunity youth members?	Hard and Soft Skill Support: 10, 11, 12, 15 a-i, 19 a-e, 20, Changing Behaviors and Attitudes: 18 a-b, d Demographics: 10, 11a-c, 12, 13	Regression analysis
In what ways does participating in AmeriCorps impact the connection to the community for opportunity youth members?	Increased Civic Engagement: 30, 31, 32 Changing Behaviors and Attitudes: 18, 19, 26, 27 Knowledge of Community Resources: 17 p-r, 24, 25	Regression analysis

A17.  Explain the reason for seeking approval to not display the expiration date for OMB approval of the information collection.

Not applicable.

 

 A18.  Exceptions to the certification statement

There are no exceptions to the certification statement in the submitted ROCIS form.

Works Cited

Aspen Institute (2014, December). Opportunity Youth Network. Retrieved from http://aspencommunitysolutions.org/press-release-aspen-institute-convening-explores-educational-and-employment-opportunities-for-opportunity-youth/.

Allen, L., Miles, M., and Steinberg, A. (2014). Achieving collective impact for opportunity youth: emerging lessons on using data and resources to improve the prospects of young people. Stanford Social Innovation Review. Fall 2014.

Belfield, C.R., Levin, J. M., & Rosen, R. (2012). The economic value of opportunity youth. Corporation for National and Community Service. http://www.civicenterprises.net/MediaLibrary/Docs/econ_value_opportunity_youth.pdf

Bird, K. (2013) Opportunity says it all: how five communities are supporting the transformation, education, and employment success of young black men. Center for Law and Social Policy. http://www.clasp.org/resources-and-publications/publication-1/Opportunity-Says-It-All.pdf

Bird, K., Foster, M., and Ganzglass, E. (2014). New opportunities to improve economic and career success for low-income youth and adults: key provisions of the Workforce Innovation and Opportunity Act (WIOA). Center for Law and Social Policy. http://www.clasp.org/resources-and-publications/publication-1/KeyProvisionsofWIOA-Final.pdf

Bloom, D., Rich, S., Redcross, C., Jacobs, E., Yahner, J., Pindus, N. (2009). Alternative Welfare-to-Work Strategies for the Hard-to-Employ: Testing transitional Jobs and Pre-Employment Services in Philadelphia.” MDRC.

Bridgeland, J. and Mason-Elder, T. (2012). National roadmap for opportunity youth. Civic Enterprises.

Bridgeland, H., & Milano, J. (2012). Opportunity Road: The Promise and Challenge of America’s Forgotten Youth 2012, Civic Enterprises & America’s Promise Alliance.

Corcoran, M., Hanleybrown, F., Steinberg, A., & Tallant, K. (2012). Collective impact for opportunity youth. FSG Social Impact Consultants.

Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York, NY: John Wiley and Sons.

Fernandes, A. and Gabe, T. (2009). Disconnected Youth: A look at 16-to 24-year olds who are not working or in school, U.S. Congressional Research Service.

Gennetian L., Sciandra M., Sanbonmatsu L., Ludwig J., Katz L., Duncan G., Kling J., Kessler R. (2012). The Long-Term Effects of Moving to Opportunity on Youth Outcomes. Cityscape [Internet]. 14(2):137-68. http://scholar.harvard.edu/lkatz/publications/long-term-effects-moving-opportunity-youth-outcomes

Gewertz, C. (2011). GED revision opens path to higher education. Education Week, 31(12) 1, 16-17.

Hair, E.C., Moore, K.A., Ling, T.J., McPhee-Baker, C., & Brown, B.V. (2009). Youth who are ‘disconnected’ and those who then reconnect: Assessing the influence of family, programs, peers, and communities. Child Trends Research Brief 2009-37, www.childtrends.org

Hanleybrown, F., Kania, J., & Kramer, M. (2012). Channeling change: making collective impact work.

Hjalmarsson, R. (2008). Criminal justice system involvement and high school completion. Journal of Urban Economics, 63, 613-630.

Julian, T.A., and Kominski, R.A. (2011). Education and Synthetic Work- Life Earnings Estimates. American Community Survey Reports, ACS-14. U.S. Census Bureau, Washington, DC.

Kemple J. and Willner C. (2008). Long-term impacts on labor market outcomes, educational attainment, and transitions to adulthood.” New York, NY: MDRC.

Kids Count Data Book, Annie E. Casey Foundation, 2009.

Kirsch, I., Braun, H., Yamamoto, K., & Sum, A. (2007). America’s Perfect Storm. Three Forces Facing Our Nation’s Future. ETS Policy Brief.

Lochner, L., & Moretti, A. (2004). The effect of education on crime: Evidence from prison inmates, arrests, and self-reports. American Economic Review, 94, 155-189.

Mannino, C. A., Snyder, M., & Omoto, A. M. (2011). Why Do People Get Involved? Motivations for Volunteerism and Other Forms of Social Action. Social Motivation, 127.

Martin, V., Broadus, J., (2013). Enhancing GED instruction to prepare students for college and careers, early success in Laguardia Community College’s Bridge to Health and Business Program. http://www.mdrc.org/sites/default/files/Enhancing_GED_Instruction_brief.pdf

Metz, E. (2014). State of the Field: Youth Community Service in the USA. In Handbook of Child Well-Being (pp. 977–997). Netherlands: Springer.

Mortimer,J.T. (2010). “The Benefits and Risks of Adolescent Employment.” Prev Res 17(2): 8–11, http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2936460

Osypuk, T. L., Tchetgen, E. J. T., Acevedo-Garcia, D., Earls, F. J., Lincoln, A., Schmidt, N. M., & Glymour, M. M. (2012). Differential mental health effects of neighborhood relocation among youth in vulnerable families: results from a randomized trial. Archives of general psychiatry, 69(12), 1284-1294.

Rutschow, E. Z., & Crary-Ross, S. (2014). Beyond the GED: promising models for moving high school dropouts to college.

Schochet, P. Z., Burghardt, J., & McConnell, S. (2008). Does job corps work? Impact findings from the National Job Corps Study. The American economic review, 1864-1886.

Snyder, T.D., & Dillow, S.D. (2011). Digest of Education Statistics 2010 (NCES 2011-015). National Center for Education Statistics, Institute of Education Sciences, U.S. Department of Education. Washington, DC. http://nces.ed.gov/pubs2011/2011015.pdf

Sum, A., Khatiwada, I., Trubskyy, M., & Palma, S. (2014). The plummeting labor market fortunes of teens and young adults. Brookings Institution.

Stuart Foundation (2011). Promising Strategies from the California Connected by 25 Initiative: Tips and Resources to Improve Outcomes for Transition Age Foster Youth 2011.

Sweeten, G. (2006). Who will graduate? Disruption of high school education by arrest and court involvement. Justice Quarterly, 23(4), 462-480.

The White House, We Can't Wait: The White House Announces Federal And Private Sector Commitments To Provide Employment Opportunities For Nearly 180,000 Youth. 2012. See http://www.whitehouse.gov/the-press-office/2012/01/05/we-cant-wait-white-house-announces-federal-and-private-sector-commitment

Theodos, B., Pergamit, M., Edelstein, S., George, T., Freiman, L. 2014. Preparing youth for college and career: a process evaluation of urban alliance. Urban Institute.

U.S. Census Bureau, Current Population Survey, 2014 Annual Social and Economic Supplement (2013). See <http://www.census.gov/hhes/www/cpstables/032014/perinc/pinc04_000.htm>

Vericker, T., Pergamit, M., Macomber, J., & Kuehn, D. (2009). Vulnerable Youth and the Transition to Adulthood. ASPE Policy Brief (Washington DC: U.S. Department of Health and Human Services).

Wallace, J. (2014). Jobs for Disconnected Youth and Talent for Suppliers: Insights from an innovation lab dedicated to bridging the talent divide. Stanford Social Innovation Review. Winter 2014.

Attachments

Attachment A-1: Simplified Group Logic Model

Inputs	Activities	Outputs	Outcomes	Impacts
Program structure and opportunity to serve Recruitment and outreach to target populations AmeriCorps funding and education awards Program staff experienced in working with at-risk populations Positive role models / mentors Program partnerships with community agencies , employers, and educational institutions	High school completion or GED support (e.g., mentors, classes) Postsecondary education preparation Soft skill instruction and support (e.g., resume writing) Hard skill instruction and support (e.g., construction) Counseling and work/career exploration Connect and integrate into the community; show members where to go for services and resources Community service activities Leadership activities (e.g., leadership skills training)	Members without diplomas or GEDs receive high school completion or GED support Members receive postsecondary education preparation Members receive training on job search (soft) skills Members receive training in trade/job (hard) skills Members receive work and career exploration support or counseling Members receive information on community resources and how to access them Members engage in service activities Members engage in leadership activities Members gain practical job experience.)	During the program Members increase desire/expectations for postsecondary education Members gain knowledge about postsecondary education ( Members gain knowledge of job search skills Members increase positive attitudes about obtaining and maintaining employment Members gain knowledge of community resources Members increase positive attitudes / sense of community Members increase self-efficacy Up to 3 months after the program Members complete job search components Up to 12 months after the program Members gain job experience Members complete coursework and/or take GED test.	During the program through 6 or more months afterward Members or alumni increase in GED certificates / H.S. diplomas Members or alumni increase access to community products (e.g., housing applications, TANF) By 3 to 6 months after the program Members or alumni obtain a job / internship / apprenticeship Members increase civic engagement (registered to vote, engaged in the community.) During the program or up to 12 or more months after the program Members or alumni increase in completed college/trade school applications Members or alumni decrease recidivism / interaction with the criminal justice system 3 to 12 or more months after the program Members or alumni increase in college enrollment 6 to 12 or more months after the program Members or alumni maintain employment

Attachment A-2: Item Sources, by Outcome Area

Table 1: Matching Variables

Variable	Question(s)	Source(s)
Age	1	AmeriCorps application
Gender	2	AmeriCorps Alumni survey
Race	5	AmeriCorps application (adapted)
Criminal history	26, 27	AmeriCorps application (adapted)
Physical/mental disability	28	Behavioral Risk Factor Surveillance System [BRFSS]
Education level/degrees	7, 8, 9	AmeriCorps application (adapted)(7), National Longitudinal Survey of Youth [NLSY] (8), NLSY (adapted) (9)
Prior employment history and income	10, 11a-c, 12, 13	NLSY (adapted)(10, 11), Gates (12), DC Alliance of Youth Advocates [DCAYA] (adapted)(13)
Financial support	25	New item
Caregiver status	3a, 3b	AmeriCorps Alumni survey
Military status	4	AmeriCorps Alumni survey
Area of residence	6	AmeriCorps application (adapted)
Similar services	33	New item

Table 2: Outcomes and Impacts

Outcome	Question(s)	Source(s)
Members increase desire/expectations for postsecondary education	20	New item (Career Competencies Indicator [CCI] adaptation)
Members gain knowledge about postsecondary education	17 (j, k, l, m, n, o)	Career Decision Self-Efficacy scale [CDSE] (adapted)
Members gain knowledge of job search skills	17 (a, b, c, d, e, f, g, h, i)	CDSE (adapted)
Members increase positive attitudes about obtaining and maintaining employment	20 (a, b, d)	CCI (adapted)
Members gain knowledge of community resources	17 (p, q, r)	New items (CDSE adaptation)
Members increase positive attitudes / sense of community	18, 19	AmeriCorps Exit survey
Members increase self-efficacy	15, 16	AmeriCorps Exit survey (originally from Competencies for Civic Action scale)
Members complete job search components	21 (a, b, c, d, e), 22	New items
Members gain job experience	10, 11a-c, 12, 13	NLSY (adapted)(10, 11), Gates (12), DCAYA (13)
Members complete coursework and/or take GED test	8, 9, 21 (f, g)	NLSY (8) NLSY (adapted) (9), New items (21)
Members or alumni increase in GED certificates / H.S. diplomas	7	AmeriCorps application (adapted)
Members or alumni increase access to community products (e.g., housing applications, TANF)	24, 25	New items
Members or alumni obtain a job / internship / apprenticeship	10, 11a-c	NLSY (adapted)
Members increase civic engagement (registered to vote, engaged in the community)	30, 31, 32	AmeriCorps Exit survey, (30, 31) AmeriCorps Alumni survey (32)
Members or alumni increase in completed college/trade school applications	21 (h, i)	New items
Members or alumni decrease recidivism / interaction with the criminal justice system	26, 27	AmeriCorps application (adapted)
Members or alumni increase in college enrollment	7, 8, 9, 19(j)	AmeriCorps application (adapted) (7),NLSY (8) NLSY (adapted) (9), New item(19 j)
Members or alumni maintain employment	10, 11a-c	NLSY (adapted)

1 http://www.nationalservice.gov/about/strategic-plan

2 The National and Community Service Act as Amended by The Edward M. Kennedy Serve America Act, 42 U.S.C. 12501 Sec.131(b)1A See http://www.nationalservice.gov/sites/default/files/documents/1990_serviceact_as%20 amended%20through%20pl%20111-13.pdf.

3 See, for example, the Evaluation Core Curriculum webinar series available on CNCS’ Knowledge Network page for AmeriCorps State and National grantees.

4 Studies of youth demonstrate, for example, that volunteering reduces the likelihood of school truancy and drug abuse (Metz, 2014), and that civic engagement improves self-esteem and life satisfaction (Mannino, Snyder, & Omoto, 2011).

17