YouthBuild_OMB_Package_Part_A_10.27.2011

Part a: Supporting statement for YouthBuild Evaluation
paperwork reduction act submission

The Impact Evaluation of the YouthBuild program is a seven-year experimental design impact evaluation funded by the U.S. Department of Labor (DOL), Employment and Training Administration, and the Corporation for National and Community Service (CNCS). YouthBuild is a youth and community development program that addresses several core issues facing low-income communities: education, employment, crime prevention, leadership development and housing. The program primarily serves high school dropouts and focuses on helping them attain a high school diploma or general educational development (GED) certificate and teaching them construction skills geared toward career placement. The evaluation will measure core program outcomes including educational attainment, postsecondary planning, employment, earnings, delinquency and involvement with the criminal justice system, and youth social and emotional development.

The evaluation started in July 2010 and is scheduled to continue until June 2017. MDRC is the prime contractor; Mathematica Policy Research and Social Policy Research Associates are subcontractors that will assist MDRC with designing the study, implementing random assignment, analyzing the findings and reporting the results. This request for clearance from the Office of Management and Budget (OMB) is for the initial information data collection: a census of all 2011 DOL- funded YouthBuild grantees and CNCS-funded grantees who do not also receive DOL funding. The request includes the following documents:

1. Introductory Letter for Grantee Survey

2. Grantee Survey

In a separate request, to be submitted later, DOL will seek OMB clearance for future data collections for the evaluation, specifically interview protocols and cost data collection forms for the collection of information during site visits to select YouthBuild sites, and follow-up surveys for study participants in a select group of sites. The request for approval for this study is being submitted in separate parts because data collected through the evaluation’s initial stages will inform the qualitative, cost and follow-up data collection instruments.

A. Justification

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

The evaluation design includes three components: impact, implementation, and cost-effectiveness. All 2011 grantees will participate in the implementation component, while a random selection of grantees will participate in the impact and cost-effectiveness components. This information collection will produce a substantial amount of information needed for the implementation component. The web-based survey will be administered to all 2011 DOL-funded and CNCS-funded YouthBuild grantees and is designed to provide uniform data on a variety of program characteristics, providing the first glimpse into many of these characteristics across the full universe of funded programs. In addition, this information will help place the impact component’s findings in context by allowing the evaluation team to document how the universe of programs receiving 2011 grants compares to the programs participating in the impact component. Finally, the survey will provide uniform information about all funded YouthBuild programs in order to explore correlations between outcomes and program characteristics. This evaluation of the YouthBuild program will be carried out under the authority of the Workforce Investment Act, Section 172, which states that “for the purpose of improving the management and effectiveness of programs and activities…the Secretary shall provide for the continuing evaluation of the programs and activities.” (WIA, Sec. 172(a) 1998).

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

Clearance by OMB is currently being requested to administer the grantee survey. The data therein will be used to assist in the implementation and impact analyses. The grantee survey is described in detail below, along with how, by whom, and for what purposes the information collected will be used. Subsequent OMB clearance packages will include requests for clearance for four additional data collection instruments: site visit protocols and three participant follow-up surveys. It is understood that approval to administer the grantee survey does not imply approval to collect data not included in this clearance request, i.e. site visit data or participant follow-up data.

All 74 programs that received DOL funding in 2011¹ and the 40 programs that receive CNCS but not DOL funding in 2011 will participate in a web-based grantee survey. The introductory letter (Appendix A) will provide information about the content of the survey and average administration time, and explain how to access the web-based instrument. The grantee survey (Appendix B) will gather more in-depth information about the program design and services, such as how the educational and vocational services are structured, how these two components are linked, as well as information about student-teacher ratios. Data from the survey will serve two key purposes. First, it will ensure uniform data on a variety of program characteristics to support the process study and impact study. As mentioned earlier, one goal of the impact analysis is to examine how program impacts vary across program features. Second, of the universe of 114 grantees, approximately 77 YouthBuild grantees will be randomly selected to participate in the impact component of the evaluation. This grantee survey will help place the study’s impact findings in context by allowing the evaluation team to document how the sites selected for the impact component of the evaluation compare to sites from the same universe that were not selected for that component.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

Advanced technology will be used in data collection efforts to reduce burden on site staff. The grantee survey will be conducted on the web to facilitate quick completion and submission. A web survey will reduce the burden on site staff by only asking those questions that are appropriate for the organization. Data from web surveys is immediately available and more accurate than self-administered questionnaires administered via paper and pencil. Web surveys also reduce the amount of interviewer labor necessary to complete a data collection, and allow respondents to complete the questionnaire on their own schedule, in multiple sittings, without having to return any forms by mail.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in item 2 above.

The evaluation strives to minimize data duplication. Information collected in the grantee survey, including data on total enrollment and program characteristics (such as intensity of orientation procedures, work site ownership and management, and tenure of program directors) is not routinely collected, nor is it available from any other source.

5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize burden.

Administration of this survey will create a minimal burden on YouthBuild programs. The grantee survey will be administered over the internet and can be completed at the responding staff member’s convenience. If preferable and more convenient, a member of the evaluation team will follow up to obtain the information by telephone.

6. Describe the consequences to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles in reducing burden.

The evaluation represents an important opportunity for DOL to add to the growing body of knowledge about the impacts of second-chance programs for youth who have dropped out of high school, including outcomes related to educational attainment, postsecondary planning, employment, earnings, delinquency and involvement with the criminal justice system, and youth social and emotional development. The evaluation will include a random assignment component, with applicants in selected sites randomly assigned to one of two research groups. Random assignment is generally viewed as the best and most feasible design for credibly and reliably answering questions about the effectiveness of social programs and policy interventions. When implemented properly, this approach creates groups that are, on average, identical in their characteristics before the intervention. Any differences in outcomes of youth in the two groups can be confidently attributed to differences in the service intervention. Without random assignment, it is very difficult to provide study findings that policymakers and other stakeholders will believe.

If the evaluation is not conducted, Federal program or policy activities will not be informed by high quality information upon which to base critical decisions regarding the YouthBuild program, nor will DOL know whether this program is one that has substantial impacts upon its participants. Given the program continues to be funded through various organizations across the country, and DOL may wish to continue funding programs targeted to high school dropouts, it is imperative that rigorous information on the impacts of this program is obtained.

Not collecting data on the universe of grantees through the grantee survey will limit the evaluation team’s ability to fully understand the context of the YouthBuild program and place the evaluation’s impact findings into context.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

• requiring respondents to report information to the agency more often than quarterly;

• requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

• requiring respondents to submit more than an original and two copies of any document;

• requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years;

• in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;

• requiring the use of statistical data classification that has not been reviewed and approved by OMB;

• that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

• requiring respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

There are no special circumstances surrounding the requested data collection. All data will be collected in a manner consistent with Federal guidelines. There are no plans to require respondents to report information more than quarterly, to prepare a written response to a collection of information within 30 days of receiving it, to submit more than one original and two copies of any document, to retain records, or to submit proprietary trade secrets. The grantee survey will produce valid and reliable results that can be generalized to the universe for the study, since the entire universe will be surveyed. DOL has indicated that completing the survey is required for its grantees and indicated this requirement in its Solicitation for Grant.

8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency’s notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.

a. Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995, the public was given an opportunity to review and comment through a 60-day Federal Register Notice, published on May 11, 2011 (FR, Vol. 76, No. 91 pp. 27363-27365). No comments were received in response to that notice.

b. Consultations Outside of the Agency

No such consultations have taken place.

9. Explain any decision to provide any payment or gift to respondents, other than re-numeration of contractors or grantees.

The evaluation team does not plan to offer an incentive to respondents for completing the grantee survey.

10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

Grantees will be informed that information gathered will be presented in summary format and used only for statistical purposes. Respondents’ privacy will be protected to the fullest extent permitted by law. While this information collection makes no express assurance of confidentiality, ETA is responsible for protecting the privacy of the respondents’ information and will maintain the data in accordance with all applicable Federal laws, with particular emphasis on compliance with the provisions of the Privacy and Freedom of Information Acts. All contractors that have access to individually identifying information are required to provide assurances that they will respect and protect the privacy of the data.

A range of measures described below will be followed to protect and safeguard the data that are collected.

a. Protection of Personal Information

It is Mathematica and MDRC policy to efficiently protect sensitive information and data, in whatever media they exist, in accordance with applicable Federal and state laws and contractual requirements. All sample members will receive unique identification codes which will be stored separately from personally identifying information. In conjunction with this policy, we require all staff members to:

• Comply with company Confidentiality Pledge and Security Manual procedures to prevent the improper disclosure, use, or alteration of confidential information. Staff may be subjected to disciplinary and/or civil or criminal actions for knowingly and willfully allowing the improper disclosure or unauthorized use of confidential information.

• Access materials the company considers to be confidential and proprietary information only on a need-to-know basis when necessary in the performance of assigned duties.

• Notify their supervisor, the YouthBuild evaluation Project Director, and the organizational Security Officer if sensitive information has either been disclosed to an unauthorized individual, used in an improper manner, or altered in an improper manner.

• Report immediately to both the YouthBuild evaluation Project Director and the organizational Security Officer all contacts and inquiries concerning sensitive or proprietary information from unauthorized staff and non-research team personnel.

b. Protection of Data

The security protocols also cover all aspects of protection for hard copy and electronic data. The grantee survey is a web-based survey; however, hardcopies may be made available upon request. All completed hardcopy materials will be shipped to the evaluation team using Federal Express or an equivalent system that allows for package tracking; if any item is delayed or lost it will be investigated immediately. All completed hardcopy documents and other survey materials will be shipped to the Survey Operations Center (SOC), a secure facility that can only be accessed by a key card. SOC staff will receipt the hardcopy documents into a secure database and store all documents containing sensitive information in locked file cabinets or locked storage rooms when not in use. Unless otherwise required by DOL, these documents will be destroyed when no longer needed in the performance of the project. All SOC staff members are required to comply with security policy and complete yearly refresher trainings.

In addition, Mathematica has developed a Disaster Recovery Plan that provides a full contingency/disaster recovery plan for major systems outages. Data use agreements (DUAs) are negotiated on a case-by-case basis. DUAs are tracked in a database on a project-by-project basis to ensure, among other things, that the data collected during the project are destroyed at the end of the project in accordance with the DUA.

All of the major software systems that will be used on the project guarantee the security of the data they collect and store. All systems and their associated databases are secured behind the firewall between the local area network and any external internet connection. To the extent that the databases must be accessed outside this local area network, or when data must be shared across the different organizations that comprise the evaluation team, this access will be across secure point-to-point connections, and data will not be sent across the internet.

c. Background checks and security

Evaluation team members working with this data will have previously undergone background checks. These checks may include filling out an SF-85 or SF85P form, for example, authorizing credit checks, and having fingerprints taken.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers these questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

The grantee survey does not include any questions of a sensitive nature.

12. Provide estimates of the hour burden of the collection of information. The statement should:

• Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. Unless directed to do so, agencies should not conduct special surveys to obtain information on which to base hour burden estimates. Consultation with a sample (fewer than 10) of potential respondents is desirable. If the hour burden on respondents is expected to vary widely because of differences in activity, size, or complexity, show the range of estimated hour burden, and explain the reasons for the variance. Generally, estimates should not include burden hours for customary and usual business practices.

• If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 13 of OMB Form 83-I.

• Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage and rate categories. The cost of contracting out or paying outside parties for information collection activities should not be included here. Instead, this cost should be included in Item 13.

MPR pretested the grantee survey during March and April of 2011 by administering the survey to select 2010 grantees in three modes: paper and pencil interview (PAPI) with a phone debriefing, phone interview with a phone debriefing, and cognitive in-person interviewing. Each mode was intended to closely mimic the experience of completing a survey on the web or by telephone. In total, six pretests took place, including two in each mode. On average, respondents required 30 minutes to complete the survey. Representatives of all 114 sites participating in the evaluation will be surveyed (74 DOL-funded sites and 40 CNCS-only funded sites). It is expected that each respondent will require 30 minutes to complete the survey, for a total of 57 hours [(114 x 30) ÷ 60 = 57].

The hour burden of data collection for the YouthBuild grantee survey is outlined below.

Table: Burden Estimates for YouthBuild Sites and Participants

Data Collection Instrument	Number of Respondents/Instances of Collection	Frequency of Collection	Average Time Per Response	Burden (Hours)
Grantee Survey	114	Once	30 minutes	57

13. Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden shown in Items 12 and 14).

The proposed data collection will not require the respondents to purchase equipment or services or to establish new data retrieval mechanisms. Survey content is based on estimates and factual information available to the respondents. Therefore, the cost to respondents solely involves answering the questions on the survey. The evaluation team does not expect grantee respondents to spend extensive time generating, maintaining, disclosing or providing the information.

14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information. Agencies may also aggregate cost estimates from Items 12, 13, and 14 in a single table.

The estimated cost to the Federal government for the study design and survey components discussed in this Supporting Statement can be seen below.

The total cost to the Federal government of carrying out this study is $14,957,969, to be expended over the seven-year period of the study. Of this, $306,677 is for the collection and analysis of grantee information from the Grantee Survey. The remaining $14,651,292 is for other costs of the study, including design, implementation and monitoring of random assignment, design and administration of participant surveys, outcome analysis, and reporting.

An estimated $200,000 (two staff-year equivalents) will be spent by DOL staff managing the study and overseeing the contractor. Since the study will last seven years (including follow-up data collection, analysis and reporting), the annualized staff cost to the Federal government is $28,571.

Table: Estimated Annualized Cost to the Federal Government

Task	Total
Grantee Survey (including breakouts below)	$297,008
Design Survey	$94,409
Programming	$98,179
Data Collection	$68,318
Data file	$36,102
Analysis of Grantee Survey	$9,669
Annualized DOL staff cost	$28,571
Total Cost for this Data Collection Request	$335,248

15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-I.

This is the first submission for data collection for the evaluation of YouthBuild programs. It is a one-time request and will count as +57 hours toward ETA’s Information Collection Burden.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and end dates of the collection of information, completion of report, publication dates, and other actions.

The evaluation plan includes a range of deliverables and reports. The following table shows an outline of these deliverables, followed by a fuller explanation of each item.

1. Table: Deliverable Timeline

   Deliverable	Date
   Design Report	Fall 2011
   Process Report	Summer 2013
   30-Month Interim Report	Fall 2015
   48-Month Final Impact Report	Spring 2017

3. Design report. In November 2011, the evaluation team will complete a design report describing in detail the proposed design for the evaluation. The report will discuss expected sample sizes, research groups, the random assignment process, and site selection and recruitment. Based on a conceptual model of how YouthBuild might affect youth outcomes, key administrative data to be collected and major topics to be addressed in each of the follow-up surveys will be outlined. Finally, the report will outline the proposed analysis plan for the process, impact, and cost-effectiveness studies.

Process report. In the summer of 2013, the evaluation team will complete a report describing the findings from the process study. This report will document, for example, the process of recruiting sites for the impact component of the evaluation, the characteristics of sites that participate, and the process of randomly assigning youth in those sites to either the program group or a control group. The report will also discuss the characteristics of youth served by all YouthBuild grantees in the universe, the flow of participants through the programs, the delivery of services, youth participation rates, and any challenges experienced by sites in serving participants.

30-month interim report. In September 2015, the evaluation team will complete a report describing interim effects of YouthBuild on a range of outcomes. This report will use data from both administrative records and the 30-month survey to examine impacts on educational attainment, employment, job characteristics, community involvement, attitudes and aspirations, family structure and living arrangements, and involvement with the criminal justice system. The evaluation team will also consider examining effects for key subgroups of youth.

48-month impact final report. In March 2017, the evaluation team will complete the final report documenting longer-term impacts of YouthBuild. Likely outcomes will include participation in education and training, the attainment of educational credentials, employment and earnings, criminal justice involvement, family status and living arrangements, positive behaviors and activities, risky behaviors, health status and other measures of well-being. This report will also examine effects for key subgroups of youth and present an analysis of the effectiveness of certain program components. Finally, the report will present an analysis of the cost-effectiveness of the program.

17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

We are not seeking approval to not display the expiration date.

18. Explain each exception to the certification statement identified in Item 19, “Certification for Paperwork Reduction Act Submissions,” of OMB Form 83-I.

Exception to the certification statement is not requested for the data collection.

1 DOL awarded grants to 76 programs in May 2011 from 2011 fiscal year appropriations. However, two of these programs received partial funding to supplement grants they received from 2010 appropriations. They are not considered 2011 grantees and, accordingly, are not included in the evaluation.

17