The U. S. Department of Labor (DOL) contracted with Abt Associates (in partnership with MDRC) to conduct an evaluation of the Cascades Job Corps College and Career Academy Pilot program. As required under the Paperwork Reduction Act, DOL is seeking approval from the Office of Management and Budget (OMB) for data collection instruments associated with the evaluation. The Job Corps program is the Federal government’s largest investment in residential job training for disadvantaged youth. The pilot program will test innovative and promising models that could improve outcomes for students; particularly youth, ages 16-21. The evaluation, funded by the DOL Chief Evaluation Office, will use multiple approaches including an impact study and implementation analysis of the Cascades Job Corps College and Career Academy (CCCA) pilot program.
This supporting statement is the first in a series of OMB submissions that correspond to an array of data collection activities for the evaluation of the CCCA pilot. CEO is seeking clearance in this submission for: the baseline information form to support the impact study, tracking data to support the planned 18-month follow-up survey, and stakeholder interview and student focus groups to support the implementation study.
Subsequent OMB submissions will seek clearance for additional evaluation data collection activities, including a follow-up survey (tentatively planned for 18-months post program entry). This subsequent submission will more fully describe the plans for analysis of impact data (via treatment/control comparisons, adjusting for covariates).
The Job Corps program is authorized by the Workforce Innovation and Opportunity Act (WIOA) of 2016 (P.L. 113-128). The Consolidated Appropriations Act in 2016 (P.L. 114-113) appropriated about $1.7 billion to fund Job Corps for Program Year 2016. The program aims to address the multiple barriers to employment faced by low-income youth ages 16-24 throughout the United States. Research has shown that while the program increases education and earnings of students, it is more beneficial for youth over age 20 than for younger students (Schochet, Burghardt, and McConnell, 2008).
To strengthen the program, DOL has committed to using Job Corps’ demonstration authority to begin the process of testing and evaluating innovative and promising models that could improve outcomes for these youth. As such, the contract to operate the CCCA pilot (DOL-ETA-16-H-0010) required participation in an independent, third-party evaluation. These evaluation activities will assist DOL in identifying evidence-based strategies that are the most promising for Job Corps students. The strategies tested by this evaluation will be those implemented by the CCCA contractor and will likely include career pathways frameworks, standards-based instruction, and sectoral training.
This evaluation is designed to answer research questions such as the following:
How were the components of the pilot program operationalized and the program implemented? (e.g.: How were youth recruited and screened for the program? How did staff interact with students, and how were students involved in the operations of the CCCA pilot program? What factors influenced program implementation? What challenges did the program face in implementation and how were those challenges overcome? What implementation practices appear promising for replication?)
What impact did CCCA have on education, employment, and earnings outcomes?
Does CCCA improve critical social-emotional skills, such as self-efficacy, future orientation, and reaction to challenges?
What strategies or program components appear promising?
The research design is a randomized controlled trial with assignment to either a treatment or control group. Treatment group members will be offered a slot at the CCCA Job Corps center. Control group members will not be offered a slot at CCCA, but will also not be prevented from enrolling in other available training programs, including at other Job Corps centers.
This configuration – a comparison of access to the focal program’s services to access to other services – is a common design for random assignment studies of training programs. It is also one that answers the relevant policy question: Do the services delivered in the CCCA pilot program improve student outcomes relative to existing Job Corps and non-Job Corps program services available in the area?
Through the Job Corps application and admission process to the CCCA center, random assignment of individuals to the treatment or control group will occur for approximately [up to] three years from the time we receive OMB approval. Roughly 1,100 students will be assigned to each group, for a total of approximately 2,200 study members overall (Exhibit A.1).
Exhibit A.1: Size of Study Groups
Services Offered to Participants |
Treatment Group Members |
Control Group Members |
CCCA |
1,100 |
0 |
Not CCCA |
0 |
1,100 |
Total |
1,100 |
1,100 |
During the evaluation intake and enrollment period, all persons who apply for the program and are determined to be eligible will be told about the study (including random assignment) and asked to sign a form confirming they have been informed about and understand the study (see Informed Consent Form). Since many of the (prospective) students will be minors, both parental consent and youth assent will be required in these cases. The forms they will be asked to sign are included in this package (see Parental Consent Form and Youth Assent Form).
Everyone who consents to participate will be asked to complete the Baseline Information Form (BIF) (included in this package) which will be accessible on-line through the Participant Data System (PDS) – the web-based system that will be custom-built for the study. Job Corps staff will then use the PDS to randomly assign each participant to either the treatment or control group, notify the individual of his or her assignment, and complete various next steps tailored to the individual enrollees based on their group assignment.
To address the research questions listed above, the evaluation of the CCCA pilot will include the following data collection activities:
Baseline data (for treatment and control group members) (clearance requested in this package)
Implementation site visits (two rounds of site visits) (clearance requested in this package)
Follow-up tracking forms to collect updated contact information (for treatment and control group members) (clearance requested in this package)
An 18-month follow-up survey (for treatment and control group members) (clearance will be requested in a future package)
Existing administrative data from the Job Corps Management Information System(s), including the Administrative data from CCCA
Other administrative data from the National Student Clearinghouse (NSC) and the National Directory of New Hires (NDNH)1 (for treatment and control group members)
With the submission of this justification, DOL’s Chief Evaluation Office (CEO) requests clearance for the first, second, and third data collection components listed above (i.e., the baseline data collection, implementation site visits, and follow-up tracking forms). DOL anticipates submitting a future OMB package to request permission to conduct the fourth component (i.e., the 18-month follow-up survey). The fifth and sixth components are all existing administrative data and their use greatly reduces the response burden on study participants by decreasing the length of the baseline and 18-month follow-up surveys.
The baseline data elements to be collected are described below. We anticipate that some of this information can be obtained through the Job Corps program’s MIS. However, for those data elements not available through existing administrative records, we will capture the information in the BIF.
Identifying Information. This includes complete name, birth date, and social security number (SSN) – enough information to ensure that each individual is randomly assigned only once. Identifiers are also necessary for ensuring that participant information can be accurately matched with administrative records (e.g., in the Job Corps MIS and the NDNH).
Contact Information. Achieving response rate targets on follow-up surveys is critically dependent on the availability of accurate contact information for sample members. The BIF will capture each applicant’s cellular telephone number, email address, and social media contact information. The form will also capture the contact information for the participant’s mother/female guardian (or father/male guardian if mother/female guardian is not present or living), one grandmother (or grandfather if both grandmothers are not living or present in the participant’s life), and one additional friend or family member who is likely to know how to contact the participant. (Past studies of disadvantaged youth have found that grandparents are often the best contacts for locating participants for follow up). This contact information will include street address, cell and landline phone numbers, and email address. We will not collect participants’ street address (except for those that live with their mother or grandmother) because youth are very mobile and are thus unlikely to remain at that address by the time of the follow-up survey.
Demographic and Socioeconomic Characteristics and Employment History. This information will be necessary to ensure that the random assignment process was conducted properly (by confirming that the research groups have similar characteristics at baseline). We will also use the information to describe the study samples, conduct subsequent analyses of subgroups, improve statistical precision of impact estimates, and build the statistical weights for the follow-up survey. Examples of this information include ethnicity, parental education, and 12-month work history.
The web-based PDS will record and compile the baseline data on all study sample members. The development and use of this customized PDS ensures that participant data are consistently recorded and available for both treatment and control group members.
In addition to gathering the baseline data using the BIF, we will address the research questions listed above with two rounds of site visits to CCCA. While on site we will conduct interviews with key CCCA staff and their external partners, hold a focus group with students, and observe the center’s operations. These visits will focus primarily on documenting program implementation, especially those components which are unique to the CCCA model. To understand how CCCA differs from the regular Job Corps model, we also will conduct a round of phone interviews with leadership in other nearby Job Corps centers.
DOL’s Chief Evaluation Office requests clearance to collect: 1) baseline data that will be used to perform and monitor random assignment, allow for future locating of participants, and inform the subsequent analysis; 2) qualitative data through monitoring and implementation site visits, and; 3) updated contact information for study participants through periodic follow-up tracking activities. The BIF, site visit interview protocols, and follow-up tracking form are described in detail below, along with how, by whom, and for what purposes the information collected will be used.
Baseline and tracking data will be collected from everyone who is determined to be eligible and has given signed consent to participate in the study. The informed consent form will be administered to all eligible individuals who apply to CCCA. Job Corps staff will either ask the applicant (and for minors, their parent) to read the form, or will read the form to the applicant before answering any questions. The consent form ensures that the potential study participant (and for minors, their parent) has been fully informed about the study, including random assignment, data collection, and privacy of the data. It ensures that all key parties are fully aware in advance of the participation requirements of the study and know that they can decline to participate or drop out at any time.
For those study participants who are minors at the time of enrollment, their parent or legal guardian will be required to give consent to participate, and the minor will be asked to complete a youth assent form. All study participants who are not legal minors will themselves be required to give consent.
The BIF will collect basic demographic, socioeconomic, and psychosocial characteristics on all consenting study participants prior to random assignment. As discussed above, it will also collect key contact information, including participant’s name, cell phone number, and email address, and contact information for the participant’s mother, grandmother, and one other friend or family member who would likely have knowledge of his or her whereabouts at the time of follow-up data collection.
Baseline and contact data are needed for the following purposes, to:
Conduct random assignment. Basic identifying information (name, gender, and SSN) is needed to conduct random assignment. As such, we will use SSNs to ensure that no one goes through the random assignment process more than once. The PDS will track and alert Job Corps staff when a duplicate SSN is entered.
Monitor random assignment. We will use baseline data to detect statistically significant differences in the characteristics of members of the treatment and control groups, since such differences would suggest a problem with the assignment process.
Locate participants for surveys. The BIF will collect detailed identifying and contact information for each participant, including contact information for people who may know the future whereabouts of the participant. This information is crucial for locating study participants for the follow-up surveys and thereby increasing the response rates.
Collect administrative data. Names and SSNs are required to access administrative data in the NDNH and NSC.
Describe the sample at random assignment. Baseline data will allow the research team to describe in more detail the population being served by the CCCA pilot program as well as the control group.
Conduct the impact analysis. Baseline data will be used to increase precision of the impact estimates and adjust for potential bias that may arise from survey nonresponse.
The BIF included in this package includes all of the baseline data that will be needed for the evaluation. In order to reduce respondent burden and minimize duplicative efforts, the evaluation team will work with National Job Corps staff to auto-fill elements on the BIF that are also collected via the Job Corps’ national MIS. Items on the BIF that the evaluation team understands to be collected via the MIS are indicated with asterisks on the version of the BIF included for clearance.
Job Corps staff will be available to assist study participants with completion of the BIF, including answering any questions about the survey for participants with low literacy or other reading barriers. The form will be translated into Spanish and other languages as necessary.
The follow-up tracking form, which is included in this package, will be provided to all study participants through a variety of methods:
A hard-copy form will be sent twice to each study participant via U.S. Postal Service mail; included in the mailing will be an introductory letter (see first page of the tracking form included in this package) and an addressed, stamped return envelope.
A postcard will be sent one time to each study participant via U.S. Postal Service mail; this postcard (which is included in this package) will include both a web address via which study participants can securely access and submit the tracking form, as well as a toll-free telephone number that study participants can call to complete the tracking form verbally over the phone.
All study participants will be provided with a unique study PIN upon entry into the study. Participants will use this PIN to access the web-based form and/or when they call the telephone number. The web-based form will be hosted on a secure website, which will only be accessible to evaluation team members who are currently working on the project.
Text messages and email messages will be sent to each study participant who provided permission to do so (via the BIF) three times (two email messages and one text message); these short messages will include a link to the web-based tracking form (same link as provided in the postcard).
Respondents to the tracking efforts will be provided with a $2 thank you incentive for each response. This incentive and the proposed methods for collecting updated contact/tracking information have proven to be effective in increasing response rates for the follow-up survey of sample members, which will be included in a subsequent OMB package.
The research team conducting site visits for the implementation sub-study will use interview protocols to guide conversations with key informants. These guides are included in this package. The information collected through these interviews, observations and focus groups will be used to describe how the CCCA operated. These initial site visit activities will allow the research team to document what services were actually in place, what were some of the challenges of operating this model, and how the program addressed these challenges.
Study participants will complete the BIF directly in the web-based PDS. This system will reduce the burden of data collection and enhance data quality in several ways. First, the system includes drop-down menus and response categories to minimize the data entry decisions placed on study participants. Second, the computerized questionnaire allows for the inclusion of efficient skip patterns that minimize respondent burden by not asking non-applicable questions. The research team will train the CCCA staff to be able to guide study participants through the online data entry system as well as on how to use the random assignment function.
This evaluation will benefit from the pre-enrollment information available for all Job Corps applicants captured in the Job Corps’ national MIS. To the extent feasible, questions on the BIF that are available from the Job Corps’ national MIS will be removed from the BIF (and PDS) (pending confirmation of availability of these data from the MIS) so that respondents will not have to provide these data twice. The key exception to this design is the inclusion of several individual identifiers (name, date of birth, and social security number) that are also included in the Job Corps national MIS. This duplication allows us to confirm a proper merge of the national MIS data with baseline data collected in the PDS, and to ensure access to data from the NDNH and NSC.
The BIF has been designed as a supplement to the national MIS since the MIS lacks sufficient identifiers and contact information for future data collection and does not document many important characteristics of the sample (e.g., family background and psychosocial measures). These data are needed to conduct subgroup analyses, to monitor the precision of random assignment, to generate more accurate non-response bias sample weights, and provide for more precise estimates of impacts. Moreover, the information that will be collected from the BIF and recorded in the PDS for the evaluation is not otherwise available in the format required for conducting private and accurate random assignment in a manner that: (1) is systematic for all consenting eligible applicants, (2) maintains integrity of the process, and (3) ensures privacy of information.
To document the information collected from Job Corps staff during site visits, we have designed the protocols to complement information provided by the CCCA contractor in its application, contract, and annual reports. There should be no adverse impact for Job Corps staff participating in the study.
The evaluation of the pilot program will impose no burden on this sector of the economy.
This is a one-time data collection activity, and the data collection plan described here will be a first step in allowing the research team to generate precise, unbiased estimates of the impacts of the pilot program’s services. In addition, the data will provide the inputs needed to describe how the individual components of the CCCA program were implemented including the initial design, challenges encountered and perceived quality.
Without collecting necessary baseline information on study participants (to complement the data available in the Job Corps MIS), the study could not implement random assignment correctly or ensure that it had been conducted appropriately. The lack of baseline information would limit the ability to describe the research sample’s characteristics. Without baseline data, impact estimates would also be less precise (so that small impacts would be less likely to be detected), and adjustments for nonresponse to the follow-up surveys would have to be based on less-detailed administrative data.
The collection of the follow-up tracking form will enable the evaluation team to increase response rates to the planned 18-month follow-up survey (which will be the focus of a future OMB submission). Increased response rates and thus sample sizes will allow for the evaluation to more reliably detect program impacts.
Site visits are critical to the impact and implementation study. The visits will allow the research team to monitor the evaluation and document the CCCA pilot program as implemented. This information provides a valuable context for presenting and interpreting the findings from the impact study. Without the site visits, the evaluation team would not be able to adequately characterize the nature of the program changes brought about in the pilot program. This inability to describe changes resulting from the pilot would reduce the generalizability and replicability of the findings. Without this information, the team could still report on the impacts of the CCCA pilot, however the broader utility of the data would be limited. Specifically, the study would lack the operational insight need to fully replicate a successful model or avoid challenges faced by CCCA practitioners.
There are no special circumstances for the proposed data collection.
In accordance with the Paperwork Reduction Act of 1995 (P.L. 104-13) and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995), DOL published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on Monday, June 27, 2016, Volume 81, Number 123, pages 41598-41599, and provided a 60 day period for public comment. A copy of this notice is included in this package. During the notice and comment period, the government did not receive any comment or request for copies of the instrument.
The following people were consulted in developing the design, the data collection plan, and the forms for which clearance is requested.
Abt Associates
Mr. Jacob Klerman (617) 520-2613
Ms. Julie Williams (301) 634-1782
Dr. Jane Leber Herr (617) 520-3042
MDRC
Dr. Jean Grossman (609) 258-6974
We will also assemble a technical working group consisting of three experts in the following areas: (1) experience with Job Corps and/or disconnected youth; (2) experience with workforce development and job training; (3) random assignment evaluation; and (4) survey methods. These experts will review and comment on the evaluation study design and data collection procedures.
Many of the items in the BIF are from previously approved baseline data collection instruments, including those from the following studies: the Ready to Work (RTW) Evaluation (OMB No. 1205-0507), the National Evaluation of Youth Corps (OMB No. 3045-0119), the Evaluation and System Design for Career Pathways Programs: 2nd Generation of HPOG/HPOG Next Gen Design (OMB No. 0970-0462), the Successful Transitions to Adulthood Research Study (STARS) (OMB No. 0970-0383), and the National Impact Study of the National Guard Youth Challenge Job Challenge Evaluation (OMB No. 1291-0008).
The evaluation team consulted with the representatives of the Office of Job Corps (OJC) within DOL. OJC administers the Job Corps program and is in contact with the operator for the CCCA site. The evaluation team also attended a kickoff meeting for the CCCA pilot contractor and delivered a presentation to inform them about the evaluation and offer them the opportunity to ask questions.
Respondents to the tracking efforts will be provided with a $2 thank you incentive for each response. Providing this small incentive for responses to the tracking efforts has proven to be effective in encouraging sample members to submit updates to their contact information. In turn, collecting this updated contact information has been proven effective in increasing response rates for the follow-up survey of sample members (which will be included in a subsequent OMB package).
No payments to respondents are involved in the collection of baseline data through the BIF or qualitative data through the site visit discussion guides.
The evaluation’s baseline data collection will include personally identifiable information (PII) collected through the BIF.
Respondent privacy will be protected to the extent allowed by law. DOL recognizes that the Job Corps serves vulnerable populations, and that centers must protect study participants from any risks of harm from evaluation activities. Accordingly, all evaluation staff will sign a privacy/non-disclosure agreement. Moreover, the CCCA evaluation will obtain informed consent from all study participants. Administering a consent form will ensure that participants understand the nature of the research and data collection. The informed consent form will be administered directly to participants 18 years of age or over; for participants under 18, a parental consent form will be administered to the parent or legal guardian and a youth assent form will be administered to the participant (these forms are included in this request).
As a part of informed consent, the following rationale for data collection and privacy assurances will be provided to study participants:
Research is being conducted to test innovative adjustments to the Job Corps program model. This program and research are funded by the U.S. Department of Labor.
The program will collect some personal information from them, such as their name, date of birth, Social Security Number, and contact information. The researchers studying the program for the government need this information. All of the information collected about the students for the program or for the evaluation will be kept completely private to the extent allowed by law, and no one’s name will ever appear in any report or discussion of the evaluation results.
As part of the study, we will inform participants that the researchers will contact them a few times over the next two to five years for contact information updates and for (tentatively planned) one or more follow-up surveys to ask questions about educational progress, work, and other topics. We will inform them that they can refuse to participate in the interviews or to answer any specific question(s).
Researchers and program staff using the information collected will take strong actions to protect their information.
The informed consent forms (i.e., the informed consent for applicants 18 years of age or over, the parental informed consent for applicants under 18 years of age, and the youth assent for applicants under 18 years of age) will be submitted to and reviewed by Abt Associates’ Institutional Review Board (IRB). The IRB will also review the random assignment procedures and information provided to potential study participants. Enrollment of individuals into the study will not occur until IRB approval has been received. IRB review is in process. In addition, upon receipt of IRB approval, Abt Associates will seek a Certificate of Confidentiality from the National Institutes of Health.
Data collected from the BIF will be entered by Job Corps program applicants into the secure web-based system (the PDS), which will only be accessible to evaluation team members who are currently working on the project. Hard copies of the informed consent forms will be shipped to the evaluation team via Federal Express to enable tracking of the packages. Forms will be stored in locked file cabinets at Abt Associates.
The follow-up tracking data will be collected via a secure website, phone calls, or through the U.S. Postal Service. Study participants may choose through which method(s) they respond. No sensitive PII will be collected on the tracking form: the form includes name, address, and phone number.
To protect the privacy of key informants interviewed on site, all hard copies of site visitor notes will be stored in a locked file cabinet when not in use. Electronic versions of site visitor notes will be stored on a secure server. Access to this drive will be limited to research staff members who are working on the project and have signed a privacy/non-disclosure agreement.
DOL is planning to produce a Public Use File (PUF) at the end of the evaluation contract that will include data collected from the BIF and 18-month survey. PII will be removed, and appropriate measures will be undertaken to reduce the risk of re-identification. The PUF will contain administrative data from the Job Corps MIS and the CCCA MIS but will not contain any administrative data from other sources.
The BIF will collect background information on students who have consented – or have been given parental consent – to participate in this evaluation. The need for personal information is discussed above. Program staff will notify potential study participants during the enrollment process that they may refuse to answer any individual BIF question. Potential study participants will also be notified that their responses will be kept private, to the extent allowable under law, to encourage their candid responses.
Much of this information is generally collected as part of the enrollment process in most programs, and is therefore not typically considered sensitive. The exception is PII and especially SSN. However, due to the fact that many of the CCCA pilot program students will be minors at the time of enrollment, the evaluation team considers questions of a sensitive nature to include those related to health (e.g., learning disabilities), criminal background (e.g., arrests and convictions), and self-efficacy measures.
SSNs are needed for two important purposes. First, SSNs will be used to avoid duplication of random assignment; as a completely distinct form of identification, checking for matching SSNs is the only completely dependable method for ensuring that participants are not randomly assigned twice. The PDS is designed to securely check for duplicate SSNs without every displaying the SSN to the user (once the SSN is entered into the system, it becomes and remains hidden to PDS users).
Secondly, SSNs will be used so that the researchers can match program records to NDNH records and NSC records. These secondary data sources will be used to measure a key outcome of interest to the evaluation: the impact of the pilot program on employment, education and earnings. NDNH requires matching on SSN. An advantage of NDNH data is that they can be collected for a large sample for a long period of time. Thus, such secondary data sources allow the researchers to estimate precisely, for the whole sample and for subgroups, the net impacts on employment and earnings over a multi-year period. Compared to survey data, these data have the advantage that they are not subject to potential biases that can occur because of recall error and survey non-response.
In educational programs, such as the CCCA pilot, an important policy question is how to best serve students with handicaps, such as learning disabilities or those who are not native English speakers. Obtaining baseline information in sensitive areas allows us to address these issues. By forming subgroups based on students’ responses, we can estimate the CCCA’s impact to determine if the program is differentially effective. We can also explore if these groups of individuals have different experiences in the CCCA pilot program than other students and how different their CCCA experience is to what they would have received otherwise.
Similarly, there is much policy interest in the employment and training field in how best to serve young people who have been arrested and/or convicted of a crime. Again, to address this issue, we need to have information on an applicant’s prior criminal history.
One’s belief that individual effort can lead to change (self-efficacy) is fundamental to the willingness of a young person to succeed. Many studies have shown that that these developmental outcomes affect education and employment outcomes (Eccles and Gootman, 2002 or Gambone, Klem and Connell, 2002). CCCA explicitly is attempting to affect critical social-emotional factors that impact a young person’s ability to succeed in education and employment. However, self-efficacy varies considerably across people; thus, we are measuring it at baseline to control for much of the variation we will see in the sample so that we will have more power to detect an impact of CCCA on self-efficacy.
Baseline data collection will begin shortly after OMB approval is received for this package. Exhibit A.12.1 presents the estimated respondent burden on study participants for baseline data collection. It shows the average time, in hours, that the evaluators estimate will be required for study participants and other respondents to complete the tasks listed in the exhibit. These tasks are: completion of the BIF, qualitative interviews with students, CCCA staff, and other individuals, and completion of the follow-up tracking form. Response times are estimated based on the research team’s prior experience with similar data collection.
The evaluators estimate that the BIF will take study participants on average 20 minutes to complete. The team assumes that 25% of the data elements on the BIF included in this submission will be auto-filled from the Job Corps’ national MIS; thus, the 20 minute completion time is estimated based only on data elements that cannot be auto-filled from the MIS. The evaluation team projects that 2,200 study participants randomized for the impact study. The total burden hours for baseline data collection are therefore (2,200 x 20)/60 = 733 hours for the BIF. Because data collection will occur approximately over the course of three years, the annual data collection burden is 244 hours. These estimates do not include the burden associated with any potential follow-up surveys completed by study participants (which would be covered by a future PRA submission).
The evaluators also estimate completion of the follow-up tracking form will take about 6 minutes each. The estimated burden associated with completion of the tracking form assumes that 25% of study participants will respond to each of the six tracking attempts. This 25% response rate to six tracking attempts equates to (6 * 0.25) = 1.5 average responses per respondent. Exhibit A.12.1 also shows the estimated burden on program staff and partners participating in key informant interviews for the implementation study. Evaluation team members will interview up to a total of 16 staff across both the CCCA center and nearby Job Corps centers; the 10 CCCA staff members will be chosen based on their job title, out of approximately 138 people. Most interviews will take approximately 1 to 1.5 hours. At the CCCA center, the evaluation team plans to interview 8 key CCCA center staff (the Academy Director, the College and Career Dean, the Admission and Placement Director, the College Prep Manager, the IT Pathway Manager, the Healthcare Pathway Manager, the Social Development Director, and the Placement Officer). The team will also conduct a series of focus groups with a total of 30 students. We will also interview 3 employers and 2 program partners and conduct interviews with 3 staff members from other Job Corps centers.
The expected response rate by the centers is 100 percent. Participation in evaluation activities on the part of the CCCA contractor is required as a condition of their contract. We anticipate that other Job Corps centers will also participate in requested qualitative data collection through phone interviews. The primary contact at the CCCA and other Job corps centers will assist the research team to identify appropriate contacts at the centers and schedule interviews at least a month prior to the visit. The time allowed for each interview may vary based on respondents’ knowledge, but we expect to stay close to the average based on previous experience with similar site visits. Total projected burden across respondents for the site visits is 58.5 hours.
Exhibit A.12.1: Estimated Annualized Respondent Burden Hours
Instrument |
Number of Respondents |
Number of Responses Per Respondent |
Total Number of Responses |
Average Burden Hours Per Response (in Hours) |
Total Burden Hours |
Baseline Information Form (BIF) |
733 a |
1 |
733 |
0.33 b |
242 |
Interview Protocol – Academy Director, College and Career Dean, and College Preparation Manager |
1.00 |
2 |
2 |
1.5 |
3 |
Interview Protocol – Admission and Placement Director and Placement Officer |
0.67 |
1.5 |
1 |
1.33 |
1 |
Interview Protocol – IT Pathway Manager and Healthcare Pathway Manager |
0.67 |
2 |
1 |
1 |
1 |
Interview Protocol – Social Development Director |
0.33 |
2 |
1 |
1 |
1 |
Interview Protocol – Skagit Valley College Liaison and Sedro-Wooley School District Liaison |
0.67 |
2 |
1 |
0.5 |
1 |
Interview Protocol – Employer Calls |
1 |
1 |
1 |
0.5 |
1 |
Interview Protocol – Other Center Directors |
1 |
2 |
2 |
1 |
2 |
Interview Protocol – Student Focus Group |
10 |
1 |
10 |
1 |
10 |
Follow-up Tracking Form |
733 a |
1.5 c |
1100 |
0.1 |
110 |
Total |
--- |
--- |
1852 |
--- |
372 |
a This includes 2,200 participants (1,100 treatment and 1,100 control group members) enrolled into the impact study over a three year period. A response rate of 100% is expected for the BIF.
b We assume that 25% of the BIF data elements will be provided by the Job Corps’ national MIS and thus create no burden for respondents. These elements are indicated with asterisks (*) on the accompanying BIF. Thus, the burden estimate applies only to new elements that are not available from the Job Corps’ national MIS.
c We will request tracking information six times for each study participant; based on previous experience, we expect 25% of study participants will respond to each tracking attempt. This 25% response rate to 6 tracking attempts equates to (6 * 0.25) = 1.5 average responses per respondent.
The cost to respondents for the burden of information collection estimated above is based solely on the time involved for the informed consent or youth assent, BIF, interviews, and follow-up postcards, which are captured in the burden estimates in Exhibit A.12.2.
Exhibit A.12.2: Estimated Annualized Respondent Cost Burden
Instrument |
Total Burden Hours |
Hourly Wage Rate |
Total Burden Costs |
Baseline Information Form (BIF) |
242 |
$7.25 a |
$1,755 |
Interview Protocol – Academy Director, College and Career Dean, and College Preparation Manager |
3 |
$36.99 b |
$111 |
Interview Protocol – Admission and Placement Director and Placement Officer |
1 |
$36.99 b |
$37 |
Interview Protocol – IT Pathway Manager and Healthcare Pathway Manager |
1 |
$36.99 b |
$37 |
Interview Protocol – Social Development Director |
1 |
$36.99 b |
$37 |
Interview Protocol – Skagit Valley College Liaison and Sedro-Wooley School District Liaison |
1 |
$25.80 d |
$26 |
Interview Protocol – Employer Calls |
1 |
$50.21 c |
$50 |
Interview Protocol – Other Center Directors |
2 |
$36.99 b |
$74 |
Interview Protocol – Student Focus Group |
10 |
$7.25 a |
$73 |
Follow-up Tracking Form |
110 |
$7.25 a |
$798 |
Total |
372 |
-- |
$2,998 |
a The median hourly wage for Job Corps applicants was assumed to be $7.25, the federal minimum wage (effective July 24, 2009) http://www.dol.gov/dol/topic/wages/minimumwage.htm.
b The median hourly wage for CCCA and other Job Corps center staff is calculated based on information from the Bureau of Labor Statistics; May 2015 National Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/oes_nat.htm: Education Administrators, All Other postsecondary (SOC code 11-9039).
c The median hourly wage for employer and program partners is calculated based on information from the Bureau of Labor Statistics; May 2015 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/oes_nat.htm: Human Resource Managers (SOC code 11-3121).
d The median hourly wage for employer and program partners is calculated based on information from the Bureau of Labor Statistics; May 2015 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/oes_nat.htm: Human Resource Managers (SOC code 21-1012).
There are no capital and start-up costs borne by respondents for this information collection. There are no costs to respondents associated with operation, maintenance, or the purchase of services.
The information collection activity and associated instruments have been developed by the evaluation contractor, Abt Associates, in performance of Contract Number: DOLQ129633231. The Chief Evaluation Office is funding the costs of the study. The proposed baseline and site-visit data collection will take place beginning as early as possible on or after January 1, 2017 and will end no later than December 31, 2019.
The estimated annualized cost to the Federal government is $233,184. The estimated total cost to the Federal government is $932,735. This cost estimate is comprised of two components:
1. The estimated total cost to the Federal government for the contractor to carry out the tasks for which this justification seeks approval is $448,900. This is comprised of $26,700 for instrument design and $422,200 for training staff (on guiding study participants through the online data entry system and random assignment) and data collection. Annualized over three years, the estimated annual cost is $149,633.2
2. DOL expects the annual level of effort for Federal government technical staff to oversee the contract will require 200 hours for one Washington D.C.-based l GS-14, Step 4 employee earning $57.39 per hour.3 To account for fringe benefits and other overhead costs the agency applies a multiplication factor of 1.6. Thus, the estimated annual cost borne by DOL for these duties is $18,365.4 The data collection period covered by this justification is three years, so the estimated total cost for performance of these duties is $55,095.
This is a new data collection.
For the evaluation of the CCCA pilot program, CEO is seeking OMB approval beginning January 1, 2017 and ending December 31, 2019.
Exhibit A.16 presents an overview of the project schedule for information collection. It also identifies deliverables associated with each major data collection activity.
Exhibit A.16 Overview of Requested Data Collection Schedule
Data Collection Activity |
Timing |
Associated Publications |
1. Baseline data collection (for treatment and control group members)
|
Beginning as early as January 1, 2017, or when OMB approval is received, if later. Baseline data collection is expected to end by December 31, 2019, or three years after OMB approval is received, if later. |
Final report |
2. Follow-Up Tracking Form |
Beginning three months after random assignment/study intake begins; taking place for three years |
Final report |
3. Site visits, observations, staff and management interview |
Fall 2017- Spring 2019 |
Final report |
All instruments created for the CCCA pilot evaluation will display the OMB approval number and the expiration date for OMB approval.
No exceptions are necessary for this information collection.
Eccles, J. and Gootman, J.A. (Eds.) (2002). Community Programs to Promote Youth Development.
Washington, DC: National Academies Press.
Gambone, M.A., Klem, A.M. and Connell, J.P. (2002). Finding Out What Matters for Youth: Testing Key Links in a Community Action Framework for Youth Development. Philadelphia: Youth Development Strategies, Inc., and Institute for Research and Reform in Education.
Schochet, Peter Z., Burghardt, J. and McConnell, S. (2008). "Does Job Corps Work? Impact Findings from the National Job Corps Study." The American Economic Review 98.5: 1864-1886.
1 The primary purpose of the NDNH, operated by the Office of Child Support Enforcement (OCSE), Agency for Children and Families, HHS is to assist state child support agencies in locating parents and enforcing child support orders; however, Congress has authorized specific state and Federal agencies to receive information from the NDNH for authorized purposes. http://www.acf.hhs.gov/programs/css/resource/a-guide-to-the-national-directory-of-new-hires
2 ($26,700+$422,200)/3=$149,633
3 See Office of Personnel Management 2016 Hourly Salary Table: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2016/DCB_h.pdf
4 200 x $57.39 x 1.6 = $18,365
Abt
Associates Inc. Supporting
Statement for OMB Clearance Request ▌pg.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 0000-00-00 |