Supporting Statement A
For the Paperwork Reduction Act of 1995: Approval for the 18-Month Follow-up Survey for the Ready to Work Partnership Grants Evaluation
October 2016
The U.S. Department of Labor (DOL) has contracted with Abt Associates and MEF Associates to conduct an evaluation of the Ready to Work (RTW) Partnership Grants (hereafter, RTW Evaluation). DOL is seeking approval from OMB under the Paperwork Reduction Act of 1995 for an 18-month follow-up survey associated with the RTW Evaluation. DOL funded the RTW Grant program in 2014 in order to improve the economic well-being of Americans facing long-term unemployment. It has awarded $169 million to 23 grantees, with individual awards ranging from $3 to $10 million. DOL also sponsored an evaluation, which includes: (1) an implementation study that examines the operation of the programs and participation patterns of program enrollees in key program activities, and (2) an impact study that uses a random assignment research design to determine whether selected grantee programs increased participants’ employment, earnings, and other outcomes. In the selected study sites, sample members will be randomly assigned to either a treatment group that will be offered the RTW-funded services, or a control group that cannot receive these services. The four RTW programs selected for the evaluation are:
Anne Arundel Workforce Development Corporation’s (AAWDC) Maryland Tech Connection (MTC) program operating in the Baltimore, Maryland and Washington, DC metropolitan region.
Jewish Vocational Service’s (JVS) Skills to Work in Technology program operating in the San Francisco Bay Area.
RochesterWorks!’s Finger Lakes Hired program operating in Monroe County, New York, which includes the city of Rochester.
Worksystems Inc.’s (WSI) Reboot Northwest program operating in the Portland, Oregon and the Vancouver, Washington metropolitan region.
Across the four sites, the evaluation will collect data from a projected 2,510 treatment group members and 2,510 control group members, for total of 5,020 individuals.
OMB approved initial data collection activities for the RTW Evaluation under OMB control number 1205-0507 (original approval on March 1, 2013; two approvals for nonsubstantive changes were approved on July 31, 2013 and March 13, 2015; and an extension with revisions request is currently under review at OMB). Those approved data collection activities included telephone interviews and site visits for the purpose of selecting study sites; study participant consent and Baseline Information Forms (BIF); and implementation study site visit protocols.
This submission seeks OMB approval for an additional data collection instrument, the 18-month follow-up survey of study participants. The survey will provide critical information on the experiences and educational and economic outcomes of study for both treatment and control members. Specific outcomes to be considered include the receipt of training and related supports, receipt of credentials, factors that may affect the ability to work (such as attitudes about work and self, and availability of work supports), employment, earnings, job characteristics, receipt of public benefits, and household income. For treatment group members, the survey will also collect opinions on the RTW services provided.
The RTW Evaluation addresses one of the central challenges facing policymakers and program administrators in the current economic environment: how to develop effective strategies to help individuals, particularly those facing long-term unemployment, obtain and succeed in training for high-growth industries. DOL is addressing this critical issue by providing resources to develop training programs for unemployed individuals that will upgrade their skills to meet the needs of American businesses, and to conduct a multi-site evaluation to build a strong knowledge base to inform future policy.
The RTW grant program focuses on training for “middle-skill” jobs, those that generally require education beyond high school but not a four-year degree. These types of jobs make up the largest share of the U.S. labor market; however, employers in key industries often cannot find sufficiently-trained workers to fill these jobs (Holzer and Lerman, 2007). Indeed, the H-1B visa program, which allows foreign individuals to temporarily work in the U.S., has been an important means for U.S.-based businesses to address skills gaps.
The RTW grant program is funded by fees paid by employers seeking to hire foreign individuals requiring an H-1B visa. Consistent with this funding source, the RTW grant program focuses on training individuals for middle-skill positions in high-growth industries that require relatively advanced levels of education and experience. Thus, the target population is generally individuals who can qualify for training programs that prepare them for positions higher than entry level and generally requiring at least a high school diploma. Of particular interest to DOL is serving the long-term unemployed who, depending on their work history, may be well positioned to prepare for and pursue emerging middle-skill jobs.1
The RTW grant program funded partnerships of workforce agencies, training providers, employers, and other organizations. As specified in DOL’s Solicitation for Grant Application for RTW (DOL 2014), grant-funded services are provided through several tracks: (1) coaching and other direct job placement services; (2) short-term training that leads to direct job placement; and/or (3) longer-term training that culminates in an industry-recognized credential and employment. Given these specifications, grantees have designed training programs that include services customized to the individual needs of participants through personalized assessments, staff guidance and assistance, and educational and work-based training options.
As noted above, the RTW Evaluation is using an experimental design to estimate program impacts on participants’ education, economic outcomes, and other measures of well-being. For each grantee program included in the evaluation, program applicants who meet the program’s eligibility requirements are randomly assigned to one of two groups: a treatment group that can access the RTW-funded services, and a control group that cannot, but that can access other services in the community. Random assignment ensures that the characteristics of the two groups have no systematic (i.e., non-random) differences at baseline. This design makes it possible to estimate program impacts without bias by comparing average outcomes for treatment and control group members over time, with any differences in outcomes being directly attributable to the program services. Overall, the impact study will examine the following research questions. Separately for each program, what is the impact on:
Individuals’ participation in education and training services?
The range of supports received, specifically receipt of advising, financial aid assistance, and employment assistance?
Educational attainment, including the receipt of credentials from training? To what extent do individuals progress to the next step of training on a career ladder?
Participants’ ability to work, such as their attitudes towards work and self, and access to transportation or childcare?
Employment and earnings?
The characteristics of jobs, including wages, work hours and schedule, benefits, and sector of employment?
Participants’ total household income and receipt of public assistance benefits?
While each of the four RTW grantees in the evaluation shares a common approach for conducting random assignment, staff have tailored the process to reflect local practices for recruitment, eligibility determination, and enrollment. Program staff recruit potential participants and determine their eligibility. As part of the intake process, program staff administer the informed consent form, which describes the study, the data collected, and the rights and responsibilities of the participant. Those who consent to participate in the study complete the Baseline Information Form (BIF), which collects demographic information, employment and education history, and contact information. Applicants choosing not to sign the informed consent form are excluded from the study and from participating in the RTW training program for the duration of the study enrollment period. Program staff then conduct random assignment using a web-based tool (the Participant Tracking System) designed for this evaluation. Individuals assigned to the treatment group are offered the program while those assigned to the control group receive information about other services in the community. OMB (Control No. 1205-0507) previously approved data collection for these initial phases.
Many of the employment outcomes for the study will be measured using quarterly Unemployment Insurance (UI) records maintained by the National Directory of New Hires (NDNH). The NDNH, which is compiled and maintained by the Office of Child Support Enforcement (OCSE) in the Department of Health and Human Services (HHS), is a national database of new hire data, quarterly wages, and unemployment insurance data that State Directors of New Hires, employees, and state workforce agencies submit to OCSE on a quarterly basis. NDNH captures information for all jobs covered by unemployment insurance, and thus will provide quarterly employment and earnings data for the vast majority of study participants and of jobs (these records will not, however, include information for jobs that are “off the books”). The evaluation will use NDNH quarterly data to determine sample members’ earnings, employment status (i.e., non-zero earnings), and job tenure.
For the impact study, this employment and earnings data from the NDNH data is complemented by the 18-month follow-up survey. The 18-month follow-up survey, which is the subject of this PRA request, will provide critical information to measure the impact of the RTW-funded services on training and related services, receipt of credentials, factors that may affect the ability to work, job characteristics (e.g. sector, wages, hours, and benefits), receipt of public benefits, and household income.
There are no legal or administrative requirements that necessitate the collection.
The follow-up survey will be conducted approximately 18 months after random assignment and will include all sample members in the four grantee programs included in the evaluation. The follow-up survey will collect data on training and related services, receipt of credentials, factors that may affect the ability to work (such as attitudes about work and self, and availability of work supports), job characteristics (e.g., sector, wages, hours, and benefits), household composition, receipt of public benefits, and household income. The expected length of the survey is 40 minutes and sample members will receive a $25 token of appreciation (in the form of a money order or gift card) for completing the survey.
Abt SRBI, a subsidiary of Abt Associates, will administer the 18-month follow-up survey. Two modes will be used to administer the survey: telephone and in-person follow up, as needed. The evaluator is seeking an 80 percent response rate for each grantee program, with approximately half of the respondents interviewed by phone, and about half interviewed in person.
The purpose of the 18-month follow-up survey is to understand critical information in terms of service receipt, employment, and other outcomes. The survey is an important source for documenting the type and duration of training and other services received; receipt of credentials; factors that may affect individuals’ ability to work; information on employment that is not available from other sources; and information on income and public benefit receipt, including determining income relative to the poverty level. In addition, for treatment group members, the survey will collect opinions on the RTW services provided. Exhibit A.1 summarizes covered measures for each of the domains included in the 18-month follow-up survey. Appendix A.1 includes the follow-up survey.
Exhibit A.1: Key Domains for RTW Evaluation 18-Month Follow-up Survey
Domain |
Rationale |
|
A |
Introduction and confirmation of identity |
|
B |
Training and education:
|
Captures receipt of training activities provided by RTW grant programs, as well as from other available programs, and educational attainment. The treatment-control service differential in training receipt is critical for interpreting employment and earnings impacts. For both treatment and control study participants, we will collect data on the range of training provided (by the grant-funded programs or other programs); data will be collected on the length and hours of the training programs, as well as completion status, and receipt of degrees and certificates. |
C |
Training-related supports:
Treatment participants only:
|
Captures receipt of training supports and ancillary activities provided by RTW grant programs, as well as from other available programs. Questions on opinions about the RTW services provided (asked of treatment participants only) captures information on participants’ views of the effectiveness and appropriateness of services offered by the RTW programs. |
D |
Factors affecting ability to work:
|
Measures of short-term outcomes associated with training-related supports, particularly the job-readiness services (e.g., provided through the boot camps), such as: sense of self; self-evaluation of own work-related characteristics (e.g., diligence, tendency to stay on task, and being a hard worker) and life skills (e.g., coping capacity); sense of control over life; motivation to work (e.g., measured by respondent’s reservation wage); and availability of work supports (e.g., transportation, child care). |
E |
Employment:
|
Employment outcomes not available through NDNH, particularly those focused on job quality, including wages; benefits; work schedule and flexibility; occupation and industry of employment. |
F |
Household composition |
Information on number of household members, combined with earnings measures, will allow calculation of income relative to the poverty level. |
G |
Income and public benefits |
Additional employment-related outcomes not available in the NDNH, including total household income and receipt of public benefits such as Unemployment Insurance, Temporary Assistance for Needy Families (TANF), and Supplemental Nutrition Assistance Program (SNAP). |
The evaluator will use the survey results to determine the impact of the RTW-funded services on participants’ education, economic, and other related outcomes. The primary beneficiaries of the evaluation results and planned data collection effort will be DOL, training program administrators, other state and local policymakers, and other federal agencies and policy makers. DOL will use the information to understand what training strategies and supports are effective in helping unemployed individuals and those facing barriers to work increase their employment levels and earnings. This will be important information in guiding the operation of state and local workforce programs, particularly given the recent Workforce Innovation Opportunity Act (WIOA) that includes a focus on career pathways training programs such as those under study in the RTW Evaluation. Secondary beneficiaries of this data collection will be those in public policy and program administration who are interested in understanding effective job training strategies more broadly.
The follow-up survey administration will use Computer-Assisted Telephone Interviewing (CATI) technology. CATI technology reduces respondent burden, as interviewers can proceed more quickly and accurately through the survey instruments, minimizing the interview length. Computerized questionnaires ensure that the skip patterns work properly, minimizing respondent burden by not asking inappropriate or non-applicable questions. For example, respondents who have held no job since randomization will skip out of the questions about current or most recent job characteristics. Computer-assisted interviewing can build in checkpoints, which allow the interviewer or respondent to confirm responses, thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values.
There is minimal duplication of data collection in the evaluation. The follow-up survey will ask about employment status despite the availability of some employment information in the NDNH data. This survey question is necessary as a screener to asking about job characteristics, information not available through NDNH. Additionally the follow-up survey will ask about participation in training and credentials. For the treatment group, some data on service receipt are available in each site’s administrative data. However, this data is of varying quality and the survey questions collect this information systematically across sites. Importantly, service receipt data for control group members is only available through the survey, as the evaluation team will not be able to determine if and where they receive non-RTW-funded services.
The data collection does not involve small businesses or other small entities.
Abt SRBI will contact participants approximately 18 months after the participants are randomly assigned at each of the four grantees. Currently, this is the only follow-up survey that the evaluation team will administer to sample members. It is a one-time data collection activity, so it would not be possible to collect this data less frequently. The timing of the survey will provide enough time for sample members to participate in RTW training programs and services (or for the control group other non-RTW funded services they may pursue), and potentially obtain employment in the field of training. While the evaluation could still report on the employment outcomes of RTW funding using NDNH data without the survey, the evaluation could not determine the service and credential receipt differential between the treatment and control groups that drives any observed earnings impacts. Moreover, the survey allows for a look at a broader set of employment-related outcomes, including the characteristics of the job, total income, and public benefit receipt.
There are no special circumstances for the proposed data collection.
In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13 and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995)), DOL published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on April 25, 2016, (81 FR 24131) and provided a 60-day period for public comment. A copy of this notice is included as Appendix A.2. During the notice and comment period, the government received 1 comment. The comment did not include an affiliation, and generally stated support for closing down the program. The comment is attached.
Incentive payments are a powerful tool for maintaining low attrition rates in longitudinal studies. The use of incentive payments for the RTW Evaluation can help maximize response rates, which is necessary to ensure unbiased impact estimates. Three factors helped to determine the incentive amounts for each survey:
Respondent burden, both at the time of the interview and over the life of the study
Costs associated with participating in the interview at that time
Other studies of comparable populations and burden
Previous research has shown that sample members with certain socio-economic characteristics are significantly more likely to become survey respondents when incentive payments are offered. In particular, sample members with low incomes and/or low educational attainment have proven responsive to incentives (Duffer et al. (1994); Educational Testing Service (1991)).
Sample members that complete the 18-month follow-up interview will receive a money order or gift card for $25 as a token of appreciation for their time spent participating in the survey. The RTW 18-month follow-up survey is comparable to other surveys of similar populations conducted by Abt Associates, such as the Pathways for Advancing Careers and Education (PACE) Evaluation 15-month follow-up survey and the Health Profession Opportunity Grants (HPOG) Evaluation 15-month follow-up survey (both conducted for the Administration of Children and Families at HHS), and the Green Jobs and Health Care (GJ-HC) Impact Evaluation 18-month follow-up survey (conducted for DOL).
Abt Associates, Abt SRBI, and MEF Associates are very cognizant of and committed to maintaining federal, state, and DOL data security requirements. All Abt Associates, Abt SRBI, and MEF study staff will comply with relevant policies related to secure data collection, data storage and access, and data dissemination and analysis.
The research team developed strong protocols to help maintain the privacy of respondents to the extent permitted by law. All research staff working with personally identifiable information (PII) will follow strict procedures to protect private information and they will sign a pledge stating that they will keep all information gathered private to the extent permissible by law. All papers that contain participant names or other identifying information will reside in locked areas and passwords will protect any computer documents containing identifying information.
The 18-month follow-up survey is voluntary. Prior to the start of each survey, researchers will inform sample members that all of their responses will be kept private, their names will not appear in any written reports, and that responses to the questions are voluntary. Specifically, the research team will take the following specific measures to protect respondents’ privacy:
Use rigorous security measures for survey data. Abt Associates and Abt SRBI have established safeguards that provide for the privacy of data and the protection of the privacy of the sampled individuals on all of its studies. All data users are aware of and trained on their responsibilities to protect participants’ personal information, including the limitations on uses and disclosures of data. Signed data privacy agreements are also required. All personal data (identifiable and de-identified data analyses files) will reside on a secure workstation or server that is protected by a firewall and complex passwords, in a directory that can only be accessed by the network administrators and the analysts actively working on the data. Survey data collected are stored in secure CATI servers. Data transfer to and from Abt and Abt SRBI will occur through Abt’s secure online file transfer platform that utilizes FIPS 140-2 validated cryptographic modules. Researchers assign generic study identifiers – not based on PII – for each study participant to link participant data. PII is removed from all electronic files prior to analysis.
Notification of data security breaches. All study partners, including the sites that are participating in the evaluation, are aware that they must notify Abt Associates within one hour of a breach of PII privacy per OMB rules. Study partners are also aware that they must notify Abt within 24 hours from the time any study partner knows of a breach/deviation from the data security plan. Researchers will notify DOL of any data security breaches, including breaches of protocol no later than 24 hours after Abt staff is made aware of the breach.
Restricting access to the study network folder. Secure servers will store all data collected that contains PII for the RTW Evaluation. Access to the study network will be restricted by assigning a password to each relevant staff member.
In addition to these study-specific procedures, the evaluator has extensive corporate administrative and security systems to prevent the unauthorized release of personal records. These systems include state-of-the-art hardware and software for encryption that meets federal standards and other methods of data protection (e.g., requirements for regular password updating), as well as physical security that includes limited key card access and locked data storage areas.
None of the survey questions for the RTW Evaluation are sensitive in nature. The most sensitive questions relate to income, public benefit receipt, and factors that affect their ability to work, including perceptions of one’s own skills (e.g., perseverance, self-efficacy). Since RTW programs should result in increased income and reduced public benefit receipt, these are important domains to measure. The other items are necessary to evaluate the mediating role of these factors that could potentially be improved by the RTW services or play a role in improving education and economic outcomes. Interviewers will remind study participants during the interviewing process that they may refuse to answer individual items. Interviewers will also provide assurances to participants that their responses will be kept private to encourage candid responses.
The evaluator estimates that it will take respondents approximately 40 minutes (0.67 hours) on average to complete the RTW 18-month follow-up survey. The evaluation team also used estimates from similar surveys, for the PACE Evaluation 15-month follow-up survey and the HPOG Impact Evaluation 15-month follow-up survey, both conducted for HHS, and the GJ-HC Impact Evaluation 18-month survey, conducted for DOL. The burden estimate is based on a total sample of 4,016 respondents (based on an 80 percent response rate for 5,020 field surveys). The burden is annualized by three years, which is the length of random assignment.
To place a value on respondents’ time, the evaluation team calculated the average hourly wage for respondents based on the average state-level minimum wage rates in the RTW Evaluation sites (California, Washington, New York, and Maryland). We multiplied this average minimum hourly wage ($9.18) by 1.4 to account for the value of fringe benefits (estimated to equal 40 percent of the hourly wage), providing an hourly value of $12.85.2
Exhibit A.2: Estimated Annual Burden for the 18-Month Participant Follow-Up Survey
Total number of respondents over entire evaluation |
Annual number of responses |
Number of responses per respondent |
Average burden time per response |
Total burden hours over entire evaluation |
Annual burden hours |
Time value |
Monetized burden hours (rounded) over entire evaluation |
Annual monetized burden hours |
4,0163 |
1,3394 |
1 |
0.67 |
2691 |
897 |
$12.85 |
$34,579.355 |
$11,526.456 |
This data collection effort involves no recordkeeping or reporting costs for respondents other than those described in Exhibit A.2 (above).
The total annualized cost to the federal government is $593,365. Costs result from the following two categories:
• The estimated cost to the federal government for the contractor to carry out this study is $1,725,000 for survey data collection. Annualized over three years of data collection, this comes to $575,000
• The annual cost borne by the DOL for federal technical staff to oversee the contract is estimated to be $18,365. The annual level of effort expected to perform these duties will require 200 hours for one Washington, D.C., based federal GS 14 step 4 employee earning $57.39 per hour.7 To account for fringe benefits and other overhead costs, the agency has applied a multiplication factor of 1.6 (200 hours x $56.57 x 1.6 = $18,365).
This evaluation involves new data collection.
The timing of the data collection and deliverables for the RTW Evaluation is driven largely by the period needed to conduct random assignment and meet the target sample sizes. The key dates for data collection activities for the RTW Evaluation are as follows:
Random assignment will occur from July 2015 to July 2018 (approved under OMB No. 1205-0507).
Implementation study site visits will occur in spring of 2016, 2017, and 2018 (approved under OMB No. 1205-0507).
The follow-up period (18 months after random assignment) will span January 2017 through January 2020. The 18-month survey is the subject of this PRA request. The 18-month follow-up survey data will therefore be available in spring/summer 2020. The 18-month NDNH data should be available by late summer 2020.
Deliverables for the RTW Evaluation will be completed on the following schedule:
Early Implementation Study Report (based on first round of site visits): January 2017.
Interim Report, including 18-month survey, 18-month NDNH data, and implementation study (all data sources and site visits): March 2021.
Final Report, including 30-month NDNH data and implementation findings: March 2022.
Public Use Data Set: April 2022.
All instruments created for the RTW Evaluation will display the OMB approval number and the expiration date for OMB approval.
No exceptions are necessary for this information collection.
Burtless, Gary. 2014. Unemployment and the “Skills Mismatch” Story: Overblown and Unpersuasive. Washington, DC: The Brookings Institution.
Duffer, A., J. Lessler, M. Weeks, & M. Mosher 1994. Effects of Incentive Payments on Response Rates and Field Costs in a Pretest of a National CAPI Survey. Proceedings of the Survey Research Methods Section of the American Statistical Association, Vol. 2, pp. 1386-1391.
Educational Testing Service. 1991. National Adult Literacy Survey Addendum to Clearance Package, Volume II: Analyses of the NALS Field Test (pp. 2-3).
Holzer, Harry J., and Robert I. Lerman. 2007. America’s Forgotten Middle-Skill Jobs: Education and Training Requirements in the Next Decade and Beyond. Washington, DC: Workforce Alliance, Skills2Compete Campaign.
U.S. Department of Homeland Security. 2013. Characteristics of H-1B Specialty Occupation Workers, Fiscal Year 2012 Annual Report to Congress. Washington, DC: U.S. Department of Homeland Security, U.S. Citizen’s and Immigration Services.
1 For this grant program, DOL defines the long-term unemployed as those who have been jobless for 27 weeks or more.
2 Minimum wages for these states (accessed in March 2016) are available from the U.S Department of Labor at http://www.dol.gov/whd/minwage/america.htm: California ($10), Washington ($9.47), New York ($9), and Maryland ($8.25). The average of $9.18 was then multiplied by 1.4 to account for fringe benefits.
3 Assumes a sample of 5,020 with an 80 percent response rate.
4 Data collection will take place over three years.
5 The monetized burden hours is the product of the total burden hours (2691) and the time value ($12.85).
6 The annual monetized burden hours is the product of the annual burden hours (897) and the time value ($12.85)
7 See Office of Personnel Management 2015 Hourly Salary Table, available at: https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2016/DCB_h.pdf
Supporting
Statement for OMB Clearance Request Part A ▌pg.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Abt Single-Sided Body Template |
| Author | Jan Nicholson |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |