The U. S. Department of Labor (DOL) contracted with Abt Associates (in partnership with MDRC) to conduct an evaluation of the Cascades Job Corps College and Career Academy Pilot program. As required under the Paperwork Reduction Act, DOL is seeking approval from the Office of Management and Budget (OMB) for data collection instruments associated with the evaluation. The Job Corps program is the Federal government’s largest investment in residential job training for disadvantaged youth. The pilot program will test innovative and promising models that could improve outcomes for students; particularly youth, ages 16 to 21. The evaluation, funded by DOL, will use multiple approaches including an impact study and implementation analysis of the Cascades Job Corps College and Career Academy (CCCA) pilot program.
OMB approved initial data collection activities for the CCCA Evaluation under OMB control number 1290-0012 (approved on February 6, 2017). Those approved data collection activities included the Baseline Information Form to support the impact study, tracking data to support the 18-month follow-up survey, and stakeholder interview and student focus groups to support the implementation study. OMB issued a second approval for data collection activities including the 18-month follow-up survey under OMB control number 1290-0023.
This supporting statement is the third OMB submission regarding data collection activities for the evaluation of the CCCA pilot. DOL is seeking approval to renew the previously approved information collection request (OMB control number 1290-0012) in order to allow for:
An extension of the follow-up tracking effort approved under OMB control number 1290-0012 to support the 18-month follow-up survey, and
Additional stakeholder interviews and student focus groups to support the implementation study, using slightly modified protocols from those approved under OMB control number 1290-0012.
The evaluation team will send the tracking form to study participants randomized between November 1, 2017 and December 31, 2018 (n=620) six times. For each tracking attempt, the evaluation will attempt to contact every study participant, unless the participant has withdrawn from the study or requested not to be contacted again.
This submission also seeks approval for a modest increase in burden to conduct site visits to two traditional Job Corps centers and to conduct one additional round of interviews with CCCA staff and students. These are in addition to the two visits to the CCCA center already approved (under OMB control number 1290-0012). The expanded effort at traditional Job Corps centers will include twenty-two additional interviews with staff and partners and four additional focus groups of approximately ten students each. The expanded effort at CCCA will include thirteen additional interviews with staff and partners and two additional focus groups of approximately ten students each.
The two non-pilot sites will be chosen from among the universe of 123 Job Corps Centers currently in operation by first identifying a small group of centers that have similar components to those at the CCCA center. Location, availability, and feasibility of conducting a visit within the study period will be considered in making a final selection for which two sites to visit.
Respondents for the staff and partner interviews will be chosen based on their job title and responsibilities. For the student focus groups, students will be randomly selected to receive an invitation to participate. The number of students in the universe of potential respondents will depend on final site selection but in general centers have slots for an average of 300 students. Using this average, it is estimated that the total universe of potential student respondents for the focus groups will be 900 (calculated using three site visits times the average number of slots of 300).
When applicable, students in the Healthcare track will be placed in one focus group and students in the Information Technology track will be placed in a separate focus group to allow the researchers to delve into greater depth about each of the separate program offerings. No statistical methods will be used in the implementation analysis, and discussions of the results will be carefully phrased to make clear that no generalization is intended. Based on the research team’s experience conducting past focus groups at the Cascades Job Corps center, we expect 80 to 100 percent participation in focus groups during the proposed site visits.
The data will be collected through semi-structured interviews held at the CCCA center and at two non-CCCA Job Corps centers. The interview protocols, which include minor updates from those approved under OMB control number 1290-0012, are included in this package.
No statistical estimation on a sample will be done. The forms provide updated contact information for study participants who are included in the 18-month follow-up survey cohort.
The site visits are designed to provide in-depth qualitative information about the CCCA pilot and nearby Job Corps programs; no estimation procedures will be used. . Information gathered and detailed notes taken during site visits will be used to create a picture of each site and its program. The research team will organize information into topics such as site management and operations, staffing structure, culture, services, partnerships, implementation challenges, and student experiences, among other topics. The data will then be analyzed to compare and contrast the CCCA program and its key implementation components with other Job Corps programs. Specifically, the research team will use data from visits to other Job Corps centers to highlight some of those comparisons.
Additionally, the research team will identify how the CCCA program and its services changed over time and will identify and categorize the implementation challenges that the CCCA program faced and any remedies that were implemented to overcome those challenges. To investigate the factors that affect implementation, the research team will use the information from the various respondents to develop hypotheses about what factors influence the program. Researchers can then investigate hypotheses by reviewing the site visit narrative and the site notes to see how often the hypothesis holds.
The evaluators will contact study participants six times through four methods to request completion of the tracking form. The evaluation team will (1) mail a hard-copy of the form twice to each study participant via U.S. postal service (an addressed, stamped return envelope will be enclosed); (2) the evaluation team will send a postcard once via U.S. postal service, on which a web link and toll-free phone number will be provided to complete the form; (3) the evaluation team will send a one text message containing a link to the form to those study participants who opted to be contacted via text in the Baseline Information Form (BIF); and (4) at two different times, the evaluation team will send email messages containing a link to the form to those study participants who opted to be contacted via email in the BIF.
The data will be collected through semi-structured interviews held at the CCCA center and at the non-CCCA Job Corps centers. Two-person teams of experienced researchers will conduct one additional site visit to CCCA and two site visits to non-CCCA Job Corps centers.
Not applicable.
Using a variety of methods (so as to reduce burden on respondents), the evaluation team will contact study participants with the follow-up tracking form once per calendar quarter during the 18-month follow-up period. The evaluation team will conduct three additional site visits and will schedule them with sufficient advanced notice so as to reduce the burden on respondents.
While completion of follow-up tracking forms is not a requirement of the study, the evaluation team will utilize different approaches to maximize the collection of any updates to participants’ contact information. We have planned for quarterly outreach activities, alternating both hard-copy and electronic outreach. For example, in one quarter we will send participants an email invitation to update info on a secure web portal, and in the next quarter, we will send either a postcard or a tracking form, both of which would invite them to provide updated contact information. We also plan to communicate with participants via text message (if they permitted) with a link to the same web portal.
CCCA participation in the evaluation—including site visits—was a condition of the pilot contract. It is anticipated that other Job Corps centers selected for site visits will agree to participate since the national Job Corps office will request their compliance.
Site visitors will work closely with the primary contact for each program to help in scheduling the site visit. One member of the two-person site visit team will take responsibility for working with the primary contact person to handle the scheduling and logistics, e.g., identifying appropriate interview respondents. Dates for site visits will be set approximately one month in advance to allow ample time to schedule interviews. Interview appointments will then be confirmed via e-mail the week prior to the visit. Should a potential respondent not be available during the visit, the research team will follow up with a time to interview the person by phone.
In order to maximize attendance of the participant focus groups, the research team will work with the centers to communicate invitations and reminders to the students. The research team will begin the invitation process about three weeks prior to the execution of the focus group and will seek preliminary commitment from the desired number of students. In addition, the research team will seek preliminary commitment from additional students as a backup.
The general discussion topics for the additional site visits have not changed from the original interview protocols which were approved under OMB control number 1290-0012. Minor revisions, additions, and deletions were made to the organization of the protocols and to specific questions in order to account for different department structures and staffing configurations as well as to account for things that the implementation team learned about the Cascades Job Corps program and the changes that took place since the inception of the program.
The individuals listed in Exhibit B.5 below made a contribution to the design of the evaluation. The data collected for the Impact Study will be analyzed under the direction of Mr. Klerman. Both the conduct and analysis of data for the Implementation Study will be under the direction of Dr. Grossman.
Exhibit B.5: Individuals Consulted
Name |
Telephone Number |
Role in Study |
Dr. Correne Saunders |
(301) 347-5657 |
Project Director |
Ms. Julie Williams |
(301) 347-5657 |
Former Project Director |
Mr. Jacob Klerman |
(617) 520-2613 |
Co-Principal Investigator |
Dr. Jean Grossman |
(609) 258-6974 |
Co-Principal Investigator |
Inquiries regarding the study’s planned analysis should be directed to:
Jacob Klerman |
Co-Principal Investigator |
301-347-5953 |
Deborah Martierrez |
Senior Evaluation Specialist, Chief Evaluation Office, DOL |
202-693-5091 |
Abt
Associates Inc. Supporting
Statement for OMB Clearance Request ▌pg.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Katheleen Linton |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |