PRA 1205-0426 Supporting Statement A (11-29-19) revised

Post Enrollment Data Collection for Job Corps Participants

OMB Control No. 1205-0426

October 2019

Supporting Statement

Post Enrollment Data Collection for Job Corps Students OMB Control No. 1205-0426

This ICR seeks OMB approval for a revision of this currently approved ICR (expiration date 10/31/19). The collection of information through this survey is necessary to provide for program evaluation and to meet WIOA reporting requirements.

A. JUSTIFICATION

The 2014 Workforce Innovation Opportunity Act (WIOA) required the Office of Job Corps to collect and report specific post enrollment outcomes for eligible Job Corps participants beginning in Program Year (PY) 2016. The WIOA performance reporting requirements, which replaced those of the 1998 Workforce Investment Act (WIA), are designed to provide a common set of metrics to be reported by similar programs. WIOA substantially changed many outcome metrics for Job Corps compared to those required under WIA and provided new guidance on the definition of high performing and low performing centers. In order to collect the necessary information to meet the new WIOA reporting requirements, the Office of Job Corps revised its post enrollment data collection system (PEDC), which primarily collects data through survey instruments, and obtained OMB approval under the Paperwork Reduction Act (PRA) for this modified system (OMB # 1205-0426, expiration date: 10/31/2019).

After reviewing the initial results of this WIOA-based PEDC, Job Corps determined that revisions to the survey instruments were necessary to increase the accuracy of the data collected, improve the survey flow, and reduce participant burden. Specifically, the following revisions are proposed:

• We reorganized the survey order into three categories: employment, education and satisfaction with the Job Corps experience. This change improves the survey flow, reduces redundancy, reduces the duration of the survey, and provides better data quality.

• We revised the survey questions in each of the three categories identified above to reduce respondent burden. We added probes and additional questions to minimize missing values on key data elements. By changing the flow of the survey, the revised survey reduces burden by eliminating redundant questions and, therefore, reducing the duration of the survey.

• In addition, we revised some questions on student safety and satisfaction and added others to improve feedback on safety culture at Job Corps centers and better focus the satisfaction questions by service area.

The additional revisions to the survey are primarily to accommodate the adjusted timeframe. These include interviewer/programming instructions to reference the new second and fourth quarter (Quarter 2 and Quarter 4) periods and revising related questions to ensure that we measure employment and school/training outcomes for the appropriate periods.

In addition to these changes, DOL/ETA required Job Corps to shift the timing of the survey periods of this collection to align with other ETA programs subject to WIOA that rely primarily on Unemployment Insurance (UI) wage record matches. This alignment uses the same uniform data elements and data definitions as all the other DOL/ETA WIOA subject programs. Job Corps will process the post enrollment performance data in this collection as a supplemental source for reporting WIOA performance, with UI wage records serving as the primary data source. This includes WIOA employment and earnings performance measures. As with other ETA programs subject to WIOA, Job Corps’ WIOA performance reports will automatically be generated by DOL in aggregate form after the performance results that incorporate UI wage record matches are returned to and calculate by the Workforce Integrated Performance System (WIPS). This method ensures data accuracy and consistency with other ETA programs subject to WIOA and completes the OMB requirement for the use of administrative data as a primary source for WIOA performance reporting. Job Corps will continue to use the post enrollment performance data as the primary data source for the calculation of various metrics and reports for managing the program.

The rationale for conducting the PEDC is explained in further detail in Section A1 of this Supporting Statement (Consequences of Less Frequent Data Collection). This collection is necessary to obtain the data required under Sections 159(c) and (d) of WIOA. Section 159(c) requires the Secretary of Labor to collect information on performance of Job Corps participants, centers, recruiters, and career transition service providers. It requires the information to be consistent with the WIOA performance indicators as amended by Section 116(b) (2) (A) (ii). It also includes additional metrics that Job Corps is required to track. In addition, Section 159(c) (1) of WIOA includes reporting requirements for primary indicators of performance that are applicable to the youth formula programs, which are found at Section 116(b) (2) (A) (ii).

The regulatory citations are detailed below:

686.945 Provides the procedures for the management of Job Corps student records. These can be found in the Department of Labor’s Privacy Act System of Records Notice (SORN) DOL/GOVT-2, Job Corps Student Records. The Secretary issues guidelines for a system for maintaining records for each student during enrollment and for disposition of such records after separation.

686.955 Establishes procedures to ensure the timely and complete reporting of necessary financial and program information to maintain accountability. Center operators and operational support service providers are responsible for the accuracy and integrity of all reports and data they provide.

686.1000 Describes how the performance of the Job Corps program as a whole, and the performance of individual centers, outreach and admissions providers, and career transition service providers, is assessed in accordance with the regulations in this part and procedures and standards issued by the Secretary, through a national performance management system, including the Outcome Measurement System (OMS).

686.1010 Describes the primary indicators of performance for Job Corps centers and the Job Corps program as outlined in this submission.

686.1020 Describes the indicators of performance for Job Corps outreach and admissions providers.

686.1030 Describes the indicators of performance for Job Corps career transition service providers.

686.1040 Details the information that will be collected for use in the Annual Report.

The Job Corps program is responsible for collecting and reporting on the six WIOA-required primary performance measures. The information needed to report on one of those six measures, Measurable Skills Gain, is covered under a separate collection. Information needed to report on the remaining five WIOA measures is addressed in this collection.

Those five measures are:

• the percentage of program participants who are in education or training activities, or in unsubsidized employment during the second quarter after exit from the program;

• the percentage of program participants who are in education or training activities, or in unsubsidized employment during the fourth quarter after exit from the program;

• the median earnings of program participants who are in unsubsidized employment during the second quarter after exit from the program;

• the percentage of program participants who obtain a recognized postsecondary credential, or a secondary school diploma or its recognized equivalent (and, for those who obtain a secondary school diploma or its recognized equivalent, have also obtained or retained employment or are in an education or training program leading to a recognized postsecondary credential), during participation in or within one year after exit from the program; and,

• the indicator(s) of effectiveness in serving employers.

The modules included in the revised collection instruments are similar to the approved data collection and are as follows: 1. Appendix A is an overview of the collection instruments. 2. Appendices B, C and D are the scripts used in the primary and secondary collections.

1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

Job Corps is the nation's largest and most comprehensive residential education and job training program for at-risk youths, ages 16 through 24. The Economic Opportunity Act of 1964 originally established Job Corps. The program was transferred from the Office of Economic Opportunity to the Department of Labor in 1969. Job Corps was later authorized as part of the Comprehensive Employment and Training Act (1978) under Title IV-B, and this authorization continued as part of the Job Training Partnership Act (JTPA) of 1982 and the Workforce Investment Act (WIA) of 1998. Now authorized by the Workforce Innovation and Opportunity Act (WIOA), Job Corps is administered federally through the Department of Labor, Employment and Training Administration, Office of Job Corps. The Job Corps program comprises 6 regional offices and 121 Job Corps centers nationwide.

Job Corps participants are typically high school dropouts in need of further education and technical training. Most Job Corps participants live on campus, but in some centers, youth from local areas can commute to centers for training. As an open-entry, open-exit program, Job Corps admits and graduates students throughout the year. Generally, these students are economically disadvantaged ethnic minorities who come from communities that do not experience favorable education or employment outcomes. Job Corps offers these youths academic and career technical training, social skills training, personal and career counseling, health care, and other supportive services.

During their Job Corps tenure, students can obtain a secondary school credential, learn a business or trade through the career technical training program, earn an industry-recognized credential and participate in work-related training that is intended to make them marketable in jobs that pay a living wage. Students enrolled for at least 60 days who obtain their high school diploma (or its equivalent) or complete their career technical training and are not separated due to a Level 1 Zero Tolerance infraction are considered program graduates. A non-graduate Job Corps participant is classified as either a former enrollee or an uncommitted student. A former enrollee is a student who remained in Job Corps for at least 60 days but left the program before obtaining a high school diploma (or its equivalent) or completing career technical training. An uncommitted student is one who left Job Corps in less than 60 days (regardless of achievement), or who separated due to a Level 1 Zero Tolerance infraction at any time. For purposes of WIOA reporting, any student who completes the Career Preparation Period (CPP) or is enrolled in Job Corps for at least 60 days is deemed a Job Corps participant, regardless of the reason for separation.

Upon separation from Job Corps, youth are prepared to pursue employment opportunities, additional educational and training experiences, or enter the Armed Forces. Job Corps career development specialists are responsible for assisting graduates and former enrollees with finding a job or enrolling in a school or training program. Career development specialists also help students create resumes and prepare for interviews. WIOA authorizes Job Corps to provide placement services to graduates for up to 12 months after they exit the program and to former enrollees for 3 months after exit.

The revised PEDC will continue to collect information on five of the six WIOA required primary performance measures. These are:

• the percentage of program participants who are in education or training activities, or in unsubsidized employment during the second quarter after exit from the program;

• the percentage of program participants who are in education or training activities, or in unsubsidized employment during the fourth quarter after exit from the program;

• the median earnings of program participants who are in unsubsidized employment during the second quarter after exit from the program;

• the percentage of program participants who obtain a recognized postsecondary credential, or a secondary school diploma or its recognized equivalent (and, for those who obtain a secondary school diploma or its recognized equivalent, have also obtained or retained employment or are in an education or training program leading to a recognized postsecondary credential), during participation in or within one year after exit from the program; and

• the indicator(s) of effectiveness in serving employers.

Job Corps may also use the information collected from the proposed PEDC to measure and report participant performance under the following metrics mandated by Section 159(d) of WIOA:

• the number of graduates who entered the Armed Forces;

• the number of graduates who received a regular secondary school diploma;

• the number of graduates who received a State recognized equivalent of a secondary school diploma;

• the percentage and number of graduates who enter postsecondary education;

• the average wage of graduates who enter unsubsidized employment—

  1. on the first day of such employment; and

  2. on the day 6 months after such first day

The second and fourth quarter data collection effort also provides a means to verify work and school placements for all placed graduates and former enrollees. In addition, it provides information responsive to the WIOA measure on the effectiveness of serving employers.

The effort to verify work and school placements, and obtain information on program effectiveness is consistent with recommendations by the Office of the Inspector General in their 1988 report that Job Corps "monitor the services provided by placement contractors to ensure that contract requirements are being met and students are receiving quality placement services."

In addition to being key for compliance with the WIOA reporting requirement of effectiveness in serving employers, customer satisfaction data is important and useful to effect program improvement. Although this survey only serves as a backup source, results can be compared with internal outcome measurement programs to ensure they agree with actual customer sentiment. Customer feedback can help identify areas for improvement and lead to new initiatives. Data suggests that outbound phone interviews are easier to complete and provide the strongest response rates.

To fulfill these information collection objectives, Job Corps will use the two primary instruments identified in Appendices B and C and the secondary instruments identified in Appendix D to collect data. The Job Corps Data Center (JCDC) will provide information about eligible participants for all surveys. The information will include contact information in addition to the work or school placement status of graduates and former enrollees after exiting Job Corps.

2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

Congress and the Secretary of Labor will use the data collected with these instruments to assess the effectiveness of the Job Corps program in meeting its objectives under WIOA. The data collected are used to fulfill the reporting requirements under Section 116(b) (2) (A) (i) and Section 159(c) (4) of WIOA. The Office of Job Corps has also incorporated these data into the Outcome Measurement System to evaluate the post-center outcomes of graduates and former enrollees. Job Corps will continue to use the information about student outcomes and satisfaction with specific services to develop or refine policies to improve the delivery of educational and job training services to at-risk youth.

3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g., permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also, describe any consideration of using information technology to reduce burden.

The data collection instruments are programmed into a Computer-Assisted Interviewing system (CAI). This system supports both interviewer-assisted and self-administered online survey modes. As is the case in the ongoing survey administration, the CAI will perform a variety of response checks and include programming to support complex skip patterns and checks for internal consistency. The system supports automated scheduling, callback, and online data storage, which makes it easier to control the sample, monitor the study, and reduce data entry costs. In addition to the functionality provided by the current system, the CAI can be enhanced to offer a secure self-administered web-based option for completing the surveys online. Automated online surveys reduce the respondent burden by allowing data collection to be completed at a time and pace that is most convenient for the respondent. Online surveys also have the potential to speed up the collection of data by reducing the number of surveys that need to be completed by an interviewer-assisted telephone mode. Students that participate via the self-administered online module would not need to be subsequently contacted by telephone to complete the survey with an interviewer.

Data obtained using the combined online and telephone survey modes is compared with historical data from the telephone-only survey to ensure the introduction of a new completion mode does not adversely impact survey results.

4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

No other existing source of data can meet the specific data reporting requirements of WIOA. Section116 (i) (2) of WIOA notes that the Secretary of Labor may collect information on former Job Corps students by using quarterly wage records and Job Corps will use this Unemployment Insurance (UI) wage match process as soon as it becomes available through the Workforce Integrate Performance Systems (WIPS); however, Job Corps will not have access to individual-level results from the UI wage record data matching. Instead, UI wage matches will be incorporated into the Job Corps datafile and used to calculate the aggregated outcomes for WIOA measures by a third-party entity, and these outcomes will be returned to WIPS and reported for Job Corps. In addition, UI wage records do not provide any information about enrollment in school or training programs or attainment of secondary or post-secondary credentials, which are key program outcomes, and are needed for accurately calculating several of the six primary WIOA measures. In spite of the limitations of UI wage records, on July 1, 2018 Job Corps began participating in the Common Reporting Information System (CRIS) data sharing process through WIPS. Although the WIOA performance metrics are not yet available through CRIS as data sharing agreements are still being negotiated, it is anticipated that wage matches for WIOA performance metrics will begin by July 2020. Once UI wage data matches are available the information captured in this collection will be considered supplemental data when reporting performance measures, as required by Section 159 of WIOA. However, post-enrollment data will remain necessary for program management and reporting of additional Job Corps specific WIOA metrics. Once the information is available from UI wage matches, the information from the Employer/Educational Institution survey (Appendix D) will be used to verify or corroborate results. The instruments presented in this application are designed to collect this information.

5. If the collection of information impacts small businesses or other small entities, describe any methods used to minimize burden.

This data collection will place minimal burden on small businesses as the participant interviews will all be conducted with individual former program participants. Respondents will not be contacted at their place of employment unless they have provided that information for contact purposes. Businesses and educational institutions will be contacted through a short phone interview and/or by an online method. The brief employment/educational institution verification and satisfaction survey instrument is included in Appendix D and the post enrollment surveys for students upon initial separation and at 13 weeks after separation are included in Module 6 of Appendix B.

6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

Since the information from eligible participants must be obtained according to the requirements of WIOA, less frequent data collection will not allow the Office of Job Corps to obtain eligible participant outcomes to meet reporting requirements under the performance measures specified in the law. WIOA requires the collection of information about outcomes in the second and fourth quarters after exit quarter from the program. The PEDC contact with eligible students during these two periods will collect information as required under the law.

7. Explain any special circumstances that would cause an information collection to be conducted in a manner inconsistent with OMB guidelines:

These data collection efforts do not involve any special circumstances.

8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, required by 5 CFR 1320.8(d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.

Describe efforts to consult with persons outside the agency to obtain their views on the availability of data, frequency of collection, the clarity of instructions and record keeping, disclosure, or reporting format (if any), and on the data elements to be recorded, disclosed, or reported.

Consultation with representatives of those from whom information is to be obtained or those

who must compile records should occur at least once every 3 years—even if the collection of information activity is the same as in prior periods. There may be circumstances that may preclude consultation in a specific situation. These circumstances should be explained.

In accordance with the Paperwork Reduction Act of 1995, the public was allowed 60 days to comment through the Federal Register notice posted on July 3, 2019 (84 FR 3197). No public comments were received.

9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

Consistent with WIOA, the Job Corps program provides ongoing services to graduates for 12 months and former enrollees for 90 days after exit. A key element of the provision of these services involves regular contact between the student and his or her career transition specialist after program separation. To encourage students to maintain contact with Job Corps, the program provides graduates and former enrollees with a transitional allowance. In addition, post-separation incentives are provided for taking the surveys: $10 after completion of the 2^nd quarter survey and $20 after completing the 4^th quarter survey. Studies found a dose-response relationship between monetary incentives and response rate by survey mode. These findings suggest that the size of the offered incentive will impact response rate and that tailoring estimates to specific survey modes and timing taking into account the target population is recommended (Mercer, et al. 2015).¹ Although higher incentive amounts would be potentially advantageous for increasing survey cooperation for these surveys, the availability of even these lower amounts are helpful in keeping participants engaged and in contact with the follow-up transition activities, including the survey.

Incentive approaches, rooted in the best evidence from the research literature, are incorporated into the survey methodology. Specifically, evidence has shown that including prepaid monetary incentives in initial survey mailings has long-proven to be successful in increasing respondent’s motivation to complete a survey (Church, 1993)². As a result, the mailed invitation letter will include a modest pre-paid unconditional cash incentive of $1 - $2. Also, for the online survey mode, we plan to incorporate what is called an “early bird” incentive to increase likelihood of respondents using the most efficient and cost-effective response mode. Research has shown that an early bird incentive model (in which an increased incentive is given for responding to a survey request by web before a specified deadline) is quite effective in improving completion rates (LeClere, F.S et al, 2012)³. To further encourage students to complete the JC WIOA web survey within the two week web push, the use of an “early-bird” incentive is only paid if the respondent completes the survey within a designated time period. These additional incentive amounts are not explicitly budgeted but can be accommodated within the existing costs of the survey effort.

10. Describe any assurance of privacy provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

This survey is conducted in accordance with all relevant regulations and requirements, including the Privacy Act of 1974 (5 USC 552a), the Privacy Act Regulations (34 CFR Part 5b), the Freedom of Information Act (5 CFR 552) and its related regulations (41 CFR Part 1-1, 45 CFR Part 5b, and 40 CFR 44502). In addition, the procedures for the management of Job Corps student records can be found in the Department of Labor’s Privacy Act System of Records Notice (SORN) DOL/GOVT-2 (Job Corps Student Records) 67 FR 16815. The Secretary issues guidelines for maintaining records for each student during enrollment and for disposition of such records after separation.

In May 2007, in an effort to support Job Corps’ commitment to securing Personal Identifiable Information (PII), all aspects of the data collection process were consolidated under the secure network environment provided and maintained by the JCDC. The application is accessible via Citrix, JCDC's preferred secure method.

JCDC takes several steps to insure the safety and integrity of the student data housed at its facilities in Austin, Texas. All student data is housed on a central server, in a secured computer room in a locked building on the fenced and guarded facility. There are several layers of system security. The server network is a private network. The network is also protected from the Internet by a series of layered firewalls, access control lists, and intrusion detection systems.

All Job Corps employees and contractors enter data into the system through the private network.

All users must have a valid user ID and password (which is changed every 90 days) to enter the system. Additional security is obtained by using two-factor authentication, application level user IDs, passwords, and specific permission applied at the database level. The integrity of the data is insured by running daily validation programs that use a set of pre-approved business rules established by the Office of Job Corps. In order to secure the integrity of data transmission, JCDC has established 256-bit encrypted Citrix sessions between the Job Corps center operators who provide the data and the JCDC. All of the survey contractor’s staff have completed security awareness training and have access to the Citrix environment and the CDSS suite of applications.

Additionally, the contractor survey sites are maintained in accordance with the Guide for Security and Privacy Controls for Federal Information Systems and Organizations (SP 800-53 Revision 4), at the moderate level, posted by The National Institute of Standards and Technology's Computer Security Resource Center. Internal access to individual records within the database at the Job Corps Data Center follow the principles of least privilege required and are controlled by all appropriate security measures, including controlled usernames, passwords, profile names, host names, firewall security IDs, and crypto cards. Input of data occurs only via a secure Citrix tunnel maintained by JCDC, ensuring a secure means of communication for the data collection contractors and hosting facility.

Additionally, the personal computers (PCs) are accessible only after providing individual user passwords that must be changed on a regular basis. Screensavers are installed on all PCs, set to activate after ten minutes, and can only be released through the individual user's password. PC passwords are not physically documented; therefore, loss of access may be recouped only by the reassignment of a password by the information systems staff.

All Job Corps survey respondents are told that completing the interview is voluntary and that the information they provide will not be shared with anyone outside of the Job Corps community. Answers to some questions in the survey instrument will be provided to appropriate agency staff (for example, information that results in questioning the validity of the initial placement will be reported to Job Corps Regional Office staff for follow-up). Answers to other questions will be available at the individual level in reports for Job Corps service providers and at the aggregated level for reports about Job Corps’ performance (for example, the Job Corps Outcome Measurement System). Therefore, it is not possible to guarantee to respondents that their answers will not be released to anyone. However, respondents will be assured that Job Corps will use the collected data only for program evaluation purposes and that their answers will not be shared with anyone outside of Job Corps without their written approval.

The policies and procedures maintained at any facility operated by service providers under contract are designed to protect the privacy of data. All contract personnel are required to sign pledges to protect respondents' privacy. Respondent names and contact information are stored in a computer database that is separate from the database that stores the survey responses. Additionally, the issues of privacy will be addressed in training sessions for all staff involved with the administration of the data collection instruments. Individual identifying information will be maintained separately from hardcopy collection forms and from computerized data files prepared for conducting the analysis.

To keep the data private, it is important not to release data into public use files or to present results in reports so that individuals can be identified. Data collected through the survey instruments will not be made publicly available. Job Corps staff and contractors will analyze data for purposes of program management and quality improvement. Nonetheless, reports that use these data will be handled in a manner that eliminates the possibility of compromising privacy. Job Corps staff and contractors will follow commonly accepted guidelines (what the Federal Committee on Statistical Methodology calls the Threshold Rule) and display only aggregated data when there are more than three cases in a cell table. When there are three or fewer cases in a cell table, tabular data will be presented by combining categories or suppressing cells to ensure the elimination of possible individual identities. Given the size of Job Corps centers and the monthly year-to-date aggregation of data, the data presented in any cell table will most likely represent information from at least four observations.

11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior and attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

These data collection instruments being submitted for approval do not include questions of a sensitive or personal nature. All respondents are assured that their responses will be kept private at the outset of the self-administered web survey or the interviewer-assisted telephone interview unless they specifically authorize its release in writing. In addition, they will be informed that they do not have to answer questions with which they are uncomfortable.

All questions concerning wages and income have been successfully pretested, and similar questions have been used extensively in previous instruments with no evidence of adverse consequences. Additionally, the survey instrument is equipped to accommodate respondents who are uncomfortable providing a specific wage by allowing their answers to be coded in ranges. Specifically, similar questions about wages and income from employment were used in the "Evaluation of the Impact of the Job Corps Program on Participants' Post-Program Labor Market and Related Behavior" study (CMB 1205-0351, expiration date 10/96).

12. Provide estimates of the hour burden of the collection of information.

This data collection is an annual process that supports the administration of approximately 93,400 Job Corps self-administered online surveys and interviewer-assisted telephone interviews. This total is in line with the burden estimate of the currently approved version. Taking into account the changes made in the instruments to reduce participant burden, the self-administered surveys require an average of 12 minutes to complete and the interviewer-assisted telephone interviews 17 minutes. These administration times are based on experience in using the interviewer-assisted telephone instruments with participants, the interviewer-assisted employer verification interviews, and the estimated time for a self-administered online instrument.

Table1: Participant Survey and Employer Educational Institution Sample Sizes (Annual Estimated)*
Survey Status	Annual Totals
Population: Job Corps exiters.	44,680
Sample: All Job Corps exiters who are assigned online surveys on initial separation and at 13 weeks after separation.	44,680
Population: Job Corps participants (second and fourth quarter)	43,720
Sample: Participants who are assigned for survey attempts (second and fourth quarter).	43,720
Population: Employers and educational Institutions of placed students.	10,000
Sample: Employers who are assigned for survey attempts.	10,000

* Based on Program Year 2017 Data

The combined reporting burden for respondents associated with this data collection effort is estimated at approximately 21,700 hours, as shown in Table 2. The response rate has fallen for the second and fourth quarter surveys (as with all telephone survey response rates) to an average of about 40 percent of participants responding in the past six months. However, we anticipate higher response rates with the revised survey, which is projected to take noticeably less time than the prior version and to be much more straightforward for respondents, which will help to make the interviewing experience more attractive to respondents. We will incorporate the use of an online survey approach, which the research and our survey contractor have found to be much more attractive to this younger age population, who can complete the survey at times convenient for them while using computers, tablets, or smartphones. The online survey is currently approved under this collection. However, it has not yet been incorporated as the implementation of the WIOA revised call in survey has required more time than earlier anticipated. We are also considering testing several alternatives to our current incentive amounts and distribution schedules to more align with survey best practices. Finally, we are investigating the prospect of using text messages for participant contact and reminders—again a more popular option with this population. In other studies conducted by our survey contractor with young and disadvantaged populations, they have been able to achieve close to or at a 60 percent completion rate using online and phone administrations. We anticipate that successful implementation of these additional innovations will facilitate moving the response rate for this survey into the 60 percent range over time. As a result, a 60% response rate is estimated for the second quarter participant survey and a 40% response rate for the 4^th quarter survey for a combined 50% rate. The historical response rate for the Employer/Educational Institution Survey is 50%.

The combined reporting burden for respondents associated with this data collection effort is estimated at approximately 21,700 hours, as shown in Table 2. Costs to respondents are limited to the time they will spend either answering the self-administered online survey or the interviewer-assisted telephone survey.

Table 2: Estimates of Respondent Burden Average Time
Respondent Category	Total Responses	Hours per Respondent	Estimated Hours
Online survey of students upon initial separation, at 13 weeks after separation, and participants after the second and fourth quarters after exit quarter.	52,680	0.2	10,579
Telephone interview of eligible participants after the second and fourth quarter after exit quarter	35,720	0.28	10,121
Employer/Institution Verification Satisfaction Survey	5,000	0.2	1,000
Total	93,400		21,700

Costs to respondents are limited to the time they will spend either answering the self-administered online survey or the interviewer-assisted telephone survey.

Table 3 shows the estimated costs to respondents, based on the Job Corps national average wage of placed graduates and former enrollees for Program Year 2017. A standard hourly rate of $25.00 per hour is used for estimating the costs of responses for employers and educational institutions. The total estimated cost of the burden for respondents is approximately $260,411 per year. This burden is offset by the incentive payments totaling approximately $612,080 (see Section A14 and Table 4 for calculation) that will be provided to many of the respondents for completing the follow-up surveys.

Table 3: Estimated Cost to Respondents for Data Collection number of responses based on estimated response rates.

Activity	Number of Respondents	Frequency	Total Annual Responses	Time Per Response (Hours)	Total Annual Burden (Hours)	Hourly Rate*	Monetized Value of Respondent Time
Online survey of students upon initial separation and at 13 wks.*	22,340	2	44,680	0.2	8,979	$11.32	$2.29
Online survey of Job Corps participants who did not qualify as graduates during the second and fourth quarter after exit quarter	2,000	2	4,000	0.2	800	$11.32	$2.26
Online survey of graduates during the second and fourth quarter after exit quarter	2,000	2	4,000	0.2	800	$11.42	$2.28
Telephone interviews with Job Corps participants who did not qualify as graduates after the second and fourth quarter after exit quarter.	3,118	2	6,237	0.283	1,765	$11.42	$3.23
Telephone interviews with graduates after the second and fourth quarters after exit quarter	14,742	2	29,483	0.283	8,356	$11.42	$3.24
Employer/Institution Verification/Satisfaction survey	5,000	1	5,000	0.2	1,000	$25.00	$5.00
	49,200		93,400		21,700
* Hourly Rate based on Monthly Center Summary Report (MPO35) for PY 2017; run on 8/29/18. The same rate as former enrollees was used for participant who did not graduate. * Online surveys at initial separation and 13 weeks of 44,680 estimated separations.
* Employer/Institution wage rates based on BLS data (pay and benefits).

13. Provide an estimate for the total annual cost burden to respondents or record keepers resulting from the collection of information. (Do not include the cost of any hour burden already reflected on the burden worksheet).

There are no additional costs to the respondents for participating in this survey. All telephone or postage costs for contacting the respondents are borne by the Federal government through the data collection contractors.

14. Provide estimates of annualized costs to the Federal government. Also, provide a description of the method used to estimate cost, which should include quantification of hours, operational expenses (such as equipment, overhead, printing, and support staff), and any other expense that would not have been incurred without this collection of information.

The estimated cost of funding this data collection effort annually will be $3,333,000 in 2019 dollars. This estimate includes the ongoing maintenance of the infrastructure needed to administer the CATI system, ongoing data communication to and from the Job Corps Data Center, labor costs for telephone interviews from trained staff who are not Job Corps employees, data processing including coding of occupational and industry information, and preparation of summary data tabulations. Supervision of this total system is also included here. This cost does not include one-time expenditures that will not recur in the future.

Approximately 40% of the cost is budgeted for labor. Approximately 5% of the cost is budgeted for long distance phone service, and approximately 55% of the cost is budgeted for software and equipment including the CATI facilities.

Additionally, the incentive system used to help ensure that eligible participants remain in

contact with the Job Corps system and, in turn, complete the follow-up surveys will cost approximately $612,080 annually (Table 4). The incentive system allots $10 payments to participants completing the second quarter survey and $20 payments for those completing the fourth quarter survey after exit.

Table 4: Incentive Payments
Surveys	Respondents	Incentive	Amount
Eligible Participants at second quarter after exit quarter	26,232	$10	$262,320
Eligible Participants at fourth quarter after exit quarter	17,488	$20	$349,760
Total	43,720		$612,080

The total combined annual cost to the Federal Government including the data collection effort and incentive payments is $3,737,000.

15. Explain the reasons for any program changes or adjustments reported on the burden worksheet.

Based on actual survey response data from the existing collection, Job Corps estimates it will have 43,720 eligible participant respondents in the second and fourth quarter surveys, plus 5,000 employer/institution respondents, and 44,680 online surveys of students upon separation and at 13 weeks after separation, yielding 93,400 total responses This results in a burden of 21,700 hours. This represents a projected reduction in the participant behavior compared to the WIOA survey that began after OMB approval in August 2016 however the addition of the post placement online module to the participant survey means that the total burden remains unchanged.

16. For collections of information whose results will be published, outline plans for tabulation and publication. Address any complex analytical techniques that will be used. Provide the time schedule for the entire project, including beginning and ending dates of the collection of information, completion of report, publication dates, and other actions.

The data collected serves four primary purposes. First, the data is used to fulfill part of the performance measurement and reporting requirements for the Job Corps program specified under WIOA. Secondly, the data is used to determine the long-term placement and wage results for the Outcome Measurement System (OMS), which is used to manage many aspects of the program including center-level performance. Additionally, the data is used for independent verification of contractor reported outcomes regarding initial placement and wages. The 13-week exiter survey will be used for this purpose and to improve contact information. Having an alternate verification source is important, as these metrics are included in the assessment of the performance of the very contractors who report the outcomes; additionally, the metrics are part of the system used for determining incentive and award fees for performance-based service contracts. Finally, the information supports the continuous program improvement activities regularly conducted by the Office of Job Corps and program operators. Although the same data collection instruments support all these purposes, the analysis plan, reporting plan and time schedule for each does differ.

WIOA Performance Reporting: Job Corps is required to report WIOA outcomes quarterly and annually. As such, Job Corps is required to provide raw student-level data to the Workforce Integrated Performance System (WIPS), which will be used by ETA as a data repository for WIOA and related data of all ETA programs. These data will be used to produce aggregated quarterly and annual reports for each program. No individual-level results will be published.

• Six Primary WIOA Measures. The individual-level information obtained from the participant surveys will be used in the calculations of five of the six primary measures required under WIOA:

1. the percentage of program participants who are in education or training activities, or in unsubsidized employment during the second quarter after exit from the program;

2. the percentage of program participants who are in education or training activities, or in unsubsidized employment during the fourth quarter after exit from the program;

3. the median earnings of program participants who are in unsubsidized employment during the second quarter after exit from the program;

4. the percentage of program participants who obtain a recognized postsecondary credential, or a secondary school diploma or its recognized equivalent (and, for those who obtain a secondary school diploma or its recognized equivalent, have also obtained or retained employment or are in an education or training program leading to a recognized postsecondary credential), during participation in or within one year after exit from the program; and,

5. the indicator(s) of effectiveness in serving employers.

Individual-level variables will be calculated for each of these measures and included in the Participant Integrated Record Layout (PIRL) that is used to calculate the Quarterly Performance Report (QPR) for WIOA.

• WIOA Annual Reporting Requirements. WIOA also requires that Job Corps report on the wage graduates earned on the day six months after the first day of unsubsidized employment. Prior to the implementation of WIOA, Job Corps relied on surveys of initially placed graduates that were administered six and 12 months after initial placement. With the transition to WIOA, and with measures based on outcomes two and four calendar quarters after exit, Job Corps re-designed the surveys to collect data on a calendar quarter after exit timeline and no longer had a tool to directly report on wages on the day six months after initial placement. As a result, Job Corps has developed a proxy for calculating this measure using the subset of participants with quarterly employment data that overlap the period five-to-seven months after initial placement. An average hourly wage is calculated for placed graduates that includes regular wages, overtime wages and tips (as applicable) and a mean value is calculated nationally and by center and CTS provider.

Job Corps Management and Performance Reporting: Job Corps also utilizes the data captured from the surveys for internal program management and data integrity monitoring, and performance reporting and evaluation of its program operators.

• Outcome Measurement System. The surveys also provide critical data to the Job Corps Outcome Measurement System (OMS), a key management tool used by Job Corps to assess overall program performance against goals. The PY 2019 OMS incorporates four key measures that align with those required under WIOA. Job Corps calculates the mean values of the placement rates in Quarter 2 and in Quarter 4, mean earnings in Quarter 2, and the employer retention rate based on the results of the surveys for inclusion in the OMS. These average values are calculated nationally and by centers and CTS providers to assess program performance. Performance measurement results are calculated and published by Job Corps on a monthly basis at the center, regional, and national levels.
  

• Verification of initial placement results and management/continuous improvement activities: The surveys are also the primary source for independent re-verification of initial placements reported by CTS providers to ensure the integrity of the underlying data. Answers to the questions in the surveys about their initial job or schooling placements are compared to values reported by CTS providers to identify situations where the reported placement may ‘potentially’ be invalid (e.g., employer/school name different, hours worked or in school different, wage different). Information of these types of differences are then provided to the Job Corps Regional Offices as ‘questionable placements’ for potential follow-up actions with the CTS provider to determine if the initial reported placement is valid. In addition, average values of such differences by type are calculated nationally and by center operator and included in reports that show patterns of potential data integrity problems across a wide variety of performance metrics to identify patterns of potential data integrity issues. These questionable placement (QP) reports are for internal contractor monitoring and management purposes only and are not published or made public.

As part of the assessment of each contractor's performance, Job Corps is able to use survey results to verify the placement outcomes of former enrollees and graduates who were reported by service providers as having entered a job or school program. Reports of questionable placements (QPs), i.e., where there is a discrepancy between the placement reported through the PEDC and the placement reported by the service provider, are generated based upon either results from the survey or the employer/school verification. QP reports are provided to the Job Corps Regional Offices for adjudication. These QP reports are for internal contractor monitoring and management purposes only and are not published or made public.

17. If seeking approval not to display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.

The instruments are primarily administered as computer-assisted online surveys or computer-assisted telephone interviews. The OMB expiration date will be displayed on the survey screens for the online survey. However, as are there are no hardcopy or online instruments for telephone interview participants to receive or view, a printed expiration date cannot be displayed for this method of data collection. Interviewers will be able to provide the OMB number and expiration date at the request of any participant. In conducting follow-up activities with employers and schools, it is anticipated that respondents will sometimes prefer to complete a hardcopy version of the instrument rather than an online survey or telephone interview. In such cases, the instrument will be mailed or sent via facsimile to the contact at the employing organization or the school and it will display the OMB control number and expiration date.

18. Explain each exception to the topics of the certification statement identified in “Certification for Paperwork Reduction Act Submissions,”

No exceptions are requested in the "Certification of Paperwork Reduction Act Submissions."

1 Mercer, A., Caporaso, A., Cantor, D., and Townsend, R. (2015).  How much gets you how much? Monetary incentives and response rates in household surveys.  Public Opinion Quarterly, 79, 105-129.

2 Church, Allan H. 1993. Estimating the Effects of Incentive on Mail Survey Response rates: A Meta-Analysis. Public Opinion Quarterly 57: 62-79.

3 LeClere, F., S. Plumme, J. Vanicek, A. Amaya, and K. Carris. 2012. “Household Early Bird Incentives: Leveraging Family Influence to Improve Household Response Rates.” In American Statistical Association Joint Statistical Meetings, Section on Survey Research.

13