baseline data collection request for ARRA-funded grants; job training evaluation (OMB 1205-0481)
The U.S. Department of Labor’s Employment and Training Administration (ETA) seeks to extend OMB approval for collecting participant baseline data from organizations that received grants under the Green Jobs, Health Care High Growth Training Program.
ETA is undertaking the Green Jobs and Health Care Impact Evaluation of the Pathways Out of Poverty (POP – Green Jobs) and Health Care and High Growth Training grant initiatives. The overall aim of this evaluation is to determine the extent to which enrollees achieve increases in employment, earnings, and career advancement as a result of their participation in the training provided by Pathways and Health Care grantees and to identify promising best practices and strategies for replication. Although the full evaluation involves three data collection efforts, this current request if for extending approval for the baseline data collection using a web-based Participant Tracking System (PTS).
As part of a comprehensive economic stimulus package funded under the 2009 American Recovery Reinvestment Act (ARRA), DOL funded a series of grant initiatives to promote training and employment in selected high-growth sectors of the economy. Individuals facing significant barriers to employment, as well as those who were recently displaced as a result of the economic downturn, are the high-priority labor pools targeted by these ARRA initiatives. High-growth and emerging industries are emphasized as part of the ARRA’s focus on labor demand, with a particular focus on emerging “green” sectors of the economy and pressing skill shortages in health care fields. These grant programs are consistent with ETA’s emphasis on more “customized” or “sector-based” labor market solutions, and job seekers, including incumbents, facing significant barriers to economic self-sufficiency become a resource to targeted growth sectors facing skill shortages or anticipating hiring needs.
ARRA’s focus on the needs of high-growth and emerging industries to hire additional workers comes at a critical time. During periods of both recession and expansion, it is important that employers remain attentive to the challenge of building and maintaining a productive workforce to ensure their long-term competitiveness. This applies particularly in industries such as health care, education, and energy, in which the Bureau of Labor Statistics projects significant job growth over an extended time (Bureau of Labor Statistics 2010). However, several factors, including declines in educational attainment among American workers, a skilled workforce that is aging and in need of replacement for retiring workers, and continued immigration are affecting workforce skill levels and employers’ ability to remain competitive and increase productivity (Dohm and Shniper 2007). Training programs like those funded by ARRA are designed either to provide these skills or to begin an entry-level career path toward acquiring them.
ETA’s grant programs represent an important step to increasing postsecondary education and training in high-growth areas, particularly health and green jobs. They provide needed resources to provide training, encourage partnerships between different service delivery systems, feature strong employer involvement, and focus on the provision of innovative and promising training strategies. To learn about the impacts of this significant investment of resources in training programs, ETA has funded a rigorous evaluation using a random assignment research design.
Previous research in the training field has provided insight into the educational and economic effects of training on participants. However, many of the studies did not use random assignment, which leaves them open to concerns about selection bias and makes it difficult to determine what outcomes would have been in the absence of the training services. To assess the impacts of these training programs effectively, a rigorous design and implementation of random assignment are required.
The overriding goals of the evaluation are to determine the extent to which enrollees achieve increases in employment, earnings, and career advancement as a result of their participation in training provided by the Pathways and Health Care grantees and to identify promising practices and strategies for replication. The study will use an experimental design involving random assignment to measure the impact of the program, as well as a process study to examine implementation and operations. The random assignment study will be conducted in six grantee programs. ETA will select grantees based primarily on the perceived strength and scale of their intervention. Therefore, we will not supply estimates of the impact of the grant programs as a whole, but rather will provide results on interventions operated by selected grantees.
The evaluation will address the following research questions:
What is the impact of the programs on the receipt of education and training services, in terms of both the number who receive these services and the total hours of training received?
What is the impact of the programs on the completion of educational programs and the receipt of certificates and credentials from the training?
What is the impact of the program on the employment levels, earnings, and career advancement of participants?
To what extent do the programs result in any employment (regardless of sector)? To what extent do the programs result in employment in the specified sector in which the training was focused?
What features of the programs are associated with positive impacts, particularly in terms of target group, curricula and course design, and additional supports?
What are the lessons for future programs and practices?
For this evaluation, the treatment condition is defined as having the opportunity to enroll in training funded by either the Pathways or the Health Care grants. The treatment condition will vary from site to site depending on the grantees selected for the evaluation and the nature and context of the training programs those organizations choose to implement with their grant funds. The control condition, or counterfactual, is defined as not having the opportunity to enroll in training funded by Pathways or Health Care grants. However, control group members will not be prevented from enrolling in other locally available training programs or services in the community. We recognize that some people assigned to the control group will find opportunities to receive some form of training. This configuration—a comparison of access to the focal program’s services to other services in the community—is a common design for random assignment studies of training programs. It is also one that answers the relevant policy question: Does adding the program services funded by the Pathways and Health Care grants to the configuration of training services already available in the community improve participant outcomes?
At each selected site, individuals will be randomly assigned to a treatment or control group. The target sample size is roughly 600 in each experimental group in four large sites, and 300 in each experimental group in two small sites, for a total of 6,000 sample members overall (Table A.1).
Table A.1. Sample Sizes
Site |
Treatment Group Members |
Control Group Members |
Total Sample |
Large 1 |
600 |
600 |
1,200 |
Large 2 |
600 |
600 |
1,200 |
Large 3 |
600 |
600 |
1,200 |
Large 4 |
600 |
600 |
1,200 |
Small 1 |
300 |
300 |
600 |
Small 2 |
300 |
300 |
600 |
Total |
3,000 |
3,000 |
6,000 |
During the intake period, all persons who apply for the program and are determined to be eligible will be told about the study (including random assignment) and asked to sign a form confirming they have been informed about and understand the study. (The form they will be asked to sign is shown in Appendix A.) Everyone who consents to participate will be asked to complete the Baseline Information Form. Program staff will then enter the person’s data into the web-based Participant Tracking System. The system will randomly assign each participant to either the treatment or the control group. Staff will then notify the individual of his or her assignment. People who do not consent will not be randomized or served with grant funding, but they may obtain training and employment help from other sources on their own.
Addressing the research questions adequately requires collecting detailed data from multiple sources. We propose to collect a rich set of baseline, service, and outcome data on treatment and control group members. The baseline data covered by this emergency clearance will enable the team to describe the characteristics of study participants at the time they are randomly assigned to the treatment or control group, ensure that random assignment was conducted properly, create subgroups for the analysis, provide contact information to locate individuals for follow-up surveys, and improve the precision of the impact estimates. The documentation of the services the participants report receiving after random assignment and key outcomes of interest to the ETA (certificate attainment, employment attainment and retention, earnings and career advancement) will be part of a future clearance package.
This request for extension is limited to the baseline information that must be collected at the outset of the study for people who are randomly assigned. The discussion below addresses those elements. Baseline data elements to be collected include:
Identifying Information. This includes complete name, address, telephone number, email, birth date, gender, and social security number (SSN)—enough information to ensure that each individual is randomly assigned only once. Identifiers are also necessary for tracking and locating sample members for follow-up surveys and for ensuring that we can obtain and accurately match administrative records on sample members.
Demographic and Socioeconomic Characteristics and Employment History. Baseline data in these areas are required to ensure that the random assignment process was conducted properly (by confirming that the research groups have similar characteristics at baseline) and to monitor random assignment. We will also use the information to describe the study sample and to document differences in the populations served across the study sites. This information will allow us to conduct subsequent analyses of subgroups.
Barriers to and Attitudes Toward Work. The baseline data collection effort will also ask questions about specific barriers that study participants may face (health, child care, and transportation) and attitudes toward work. We will use this information (1) to describe the sample and understand issues that could affect their ability to participate in training and work, and (2) to conduct subsequent subgroup analyses (for example, to determine whether the program was effective for those with specific barriers).
Locating Information. Accurate locating information is crucial to achieving high survey response rates. As mentioned above, the BIF will capture each applicant’s landline and cellular telephone numbers and email address. The form will also capture Facebook Name and alternative contact information for up to three relatives or friends who might know how to contact the sample member.
A web-based PTS will execute the random assignment procedures and compile baseline data on study sample members. This PTS will assure that participant data will be in a consistent format across sites. The PTS will also perform random assignment.
ETA requests approval to extend the expiration date for collecting baseline data that will be used to perform and monitor random assignment, allow for location of participants in the future, and inform the later analysis. The consent form and BIF are described in detail below, along with how, by whom, and for what purposes the information collected will be used.
The informed-consent form will be administered to all eligible individuals at the selected sites. The grantee staff will ask the person to read the form, or will read the form to the person, and will then answer any questions. The consent form ensures that the potential study participant has been fully informed about the study, including random assignment, data collection, and confidentiality of the data. It ensures that people are aware of the participation requirements of the study and know that they can decline to participate or drop out at any time.
The BIF will collect basic demographic, socioeconomic, and barrier-related characteristics on all consenting persons prior to random assignment. It will also collect the name, address, phone number, and email address of up to three of the participant’s close friends or relatives who would likely have knowledge of his or her whereabouts at the time of follow-up data collection.
Baseline and contact data are needed for the following purposes:
To Conduct Random Assignment. Some basic identifying information (name, address, date of birth) is needed to conduct random assignment.
To Monitor Random Assignment. We will use baseline information to ensure that no one goes through the random assignment process more than once. Grantee staff at each selected site will enter identifying information about individuals into the PTS, and the system will alert the staff if the person has already been randomly assigned. Checking for previous randomization ensures that people always remain in the same research group and that the research sample does not include duplicate cases. We will also use baseline data to detect differences in the characteristics of members of the treatment and control groups, since such differences would suggest a problem with the assignment process.
To Locate Participants for Surveys and Collect Administrative Data. The BIF will collect detailed identifying information for each participant, including people who may know the whereabouts of the participant. This information is crucial for locating study participants for the follow-up surveys and thereby increasing the response rates. The participant’s SSN is essential for obtaining administrative data, such as Unemployment Insurance quarterly wage records that will be used to measure employment outcomes, since agencies typically provide data by matching on SSN. The SSN is also helpful in locating participants for follow-up surveys, as many of the locating services search for current address and telephone number using the SSN.
To Describe the Sample at Random Assignment. Baseline data will allow researchers to describe in more detail the population being served by each program.
To Define Subgroups for the Impact Analyses. Baseline data on the characteristics of sample members are essential to define subgroups for which impacts can be estimated. These include characteristics such as sex, race/ethnicity, health status, employment history, barriers to employment, and attitudes toward work.
To Increase the Precision of Impact Estimates. By including information on the baseline characteristics of study participants in regression models, the precision of impact estimates can be improved.
To Adjust for Nonresponse Bias. With random assignment, simple differences in the mean outcomes between the treatment and control groups provide unbiased estimates of the impacts. However, systematic differences between the characteristics of members of the two groups might occur because of differential rates of survey nonresponse across the groups. To the extent that these characteristics are correlated with the outcome variables, this may lead to biased impact estimates. Baseline characteristics can be used to adjust for potential bias that may arise from survey nonresponse.
The BIF will be completed by everyone who has been found eligible for Pathways or Health Care grant training and has given signed consent to participate in the study. As with the consent forms, grantee staff administering the BIF will be available to answer questions. For people with low literacy or other reading barriers, the grantee’s staff can provide assistance.
The grantee staff in the evaluation will use the PTS to enter participant data at the time of intake to randomly assign participants into treatment and control groups; all baseline items for the study will be entered. The system will ensure that each site collects valid and comparable data required for random assignment, and once data are entered into the PTS, the system has an algorithm that will automatically randomly assign each entered individual into the control or treatment group. Hence, the PTS will ensure that rigorous and unbiased procedures are used to assign people according to the experimental design being used to evaluate net impacts.
The PTS is a web-based system for gathering background information on participants as well as for executing random assignment for the evaluation. Its drop-down menus and response categories minimize the data entry burden on grantee staff, and the research team will train them all on how to use the system to randomly assign participants.
The information being recorded in the PTS for the evaluation is not otherwise available in the format required for conducting accurate random assignment in a manner that: (1) is systematic across sites, (2) maintains integrity of the process, and (3) ensures the privacy of information. The research team reviewed the quarterly reports each grantee submits to ETA and found that they present financial information and minimum participant-level data. The reports fail to provide key identifiers and contact information for future data collection, to document the characteristics of the sample in terms of education and employment history and work-related barriers, or to supply key data needed to create meaningful subgroups.
This data collection does not involve small businesses or other small entities.
The data collection plan will enable the Pathways and Health Care evaluations to generate precise, unbiased estimates of the impacts of the training services offered. Results from this rigorous evaluation will inform policymakers about net impacts for participants and the context within which the programs operate.
Without collecting baseline information on study participants, the study could not implement random assignment correctly or ensure that it had been conducted appropriately. The lack of baseline information would limit the ability to describe the population of training program participants and would limit the analysis of impacts of the program on subgroups, hence limiting the ability to determine the groups for which the program is most effective. Without baseline data, impact estimates would be less precise (so that small impacts would be less likely to be detected), and adjustments for nonresponse to the follow-up surveys would have to be based on less-detailed administrative data.
Finally, if it did not collect detailed contact information for study participants, the study would not be able to track them and administer the 18- and 36-month follow-up surveys. This would likely lead to a higher nonresponse rate and thus pose a greater risk to the quality of survey data and, in turn, the impact estimates.
This data collection effort does not involve any special circumstances.
The sixty day notice soliciting comments on extending approval for this collection was published in the Federal Register on September 28, 2011 (Vol. 76, p. 60084). No comments were received.
There are no payments to study participants for completing the consent form or the BIF.
Abt Associates and Mathematica Policy Research have a strong set of methods in place to ensure the protection of data collected from study participants, even at baseline. This includes policies and procedures related to privacy, physical and technical safeguards, and approaches to the treatment of personally identifiable information (PII). (Additional information on these procedures is in Appendix D.)
Confidentiality Policy
Abt and Mathematica are very cognizant of federal, state, and DOL data security requirements. All Abt and Mathematica study staff will comply with relevant policies related to secure data collection, data storage and access, and data dissemination and analysis. All staff working with PII sign data confidentiality agreements. Abt’s and Mathematica’s security policies meet the legal requirements of The Privacy Act of 1974; the “Buckley Amendment,” Family Education and Privacy Act of 1974; the Freedom of Information Act; and related regulations to assure and maintain the confidentiality of program participants.
Prior to random assignment, all potential program participants will receive an information brochure about the evaluation that will include answers to frequently asked questions. The consent form, which will be signed by all study participants, informs them that all information they provide will be treated confidentially to the maximum extent allowed by law and used for research purposes only. Further, participants will be assured that they will not be referred to by name or in any other way that could identify them in reports or communications with DOL.
Confidentiality Safeguards
Abt and Mathematica have established safeguards that provide for keeping data private to the extent of the law for the sampled individuals in all of its studies. At baseline, all information on individual participants is entered into the PTS. The web application will reside on a Microsoft Windows server running an IIS web server that will be physically located at MaximumASP, a monitored access-controlled secure data center. The system is being developed and managed by a subcontractor, Relyon. The web server has been hardened using a best-practices security hardening checklist (NIST). Administrator access to the database server will be restricted to an authorized Abt Associates server administrator and Relyon. Accounts on the web server will be protected with passwords that are at least 8 characters long, that contain at least 1 special character and number, and that will not contain dictionary words. These requirements are enforced upon account creation. Passwords will expire every 90 days, and users will have to create new passwords that fulfill the requirement of the password policy. All sensitive data will be encrypted and protected by Secure Socket Layer (SSL). Logging or output files will not contain confidential data and will be limited to generic system and error data. Furthermore, data extracts for use by the project team will be available in files encrypted and available to project team members on Relyon’s File Transfer Protocol over SSL (FTPS) site.
The system will segregate user data into confidential data, user-identifiable data, and project-specific data. Confidential data such as SSNs will be stored in a separate database table containing a system-generated ID number with the SSN stored in encrypted form. Confidential data will be entered into the system but will at no point be displayed. Confidential data will only be downloadable by two system users who will protect and secure the data according to federal, state, and DOL data security requirements, and will encrypt the data following download. User-identifiable data will be stored separately from project-specific data and will be available for updating only by designated administrators on the research team. Baseline data from the study will be available to the project team in specific extracts and reports.
The PTS will be accessible only to staff who are currently working on the project. Staff access to participant-level data will be restricted. To access the PTS, users will first log on to their workstations and then to the database using a separate log-in prompt. The database will be removed and securely archived at the end of the baseline data collection period. For additional detail on the system security safeguards, please see Appendix D.
The BIF will collect background information on participants who have consented to participate in this evaluation. The BIF collects individual identifying information of a sensitive, personal, and private nature, including (1) last and first name; (2) Social Security number; (3) date of birth; (4) participant contact information; (5) ethnicity and race; (6) citizenship status; (7) marital status; (8) number of children; (9) whether the individual is a TANF or SNAP recipient; (10) name and contact information for three individuals who can be contacted if the program cannot locate the individual; (11) education level; (12) employment history; and (13) work-related barriers.
The reasons why these questions are considered necessary and specific uses to be made of this information:
The last and first names are needed by site staff to be able to effectively use and update the data system in a reliable, efficient, and user-friendly manner (e.g., to be able to easily locate an individual’s record and update the record with service and outcome data).
The Social Security Number (SSN) of each participant is needed for two important purposes. First, SSNs will be used avoid duplication of random assignment; as a completely distinct form of identification, checking for matching SSNs is the only completely dependable method for ensuring that participants are not randomly assigned twice, either in the same site or at different sites. The PTS is designed to securely check for duplicate SSNs within and across sites, without ever displaying the SSN to the user (once the SSN is entered into the system, it becomes and remains hidden to PTS users). Secondly, SSNs will be used so that the researchers can collect critical administrative data—Unemployment Insurance (UI) records—that will be used to measure the primary outcome of interest to the evaluation: the impact of the training programs on employment and earnings. The wage records collected by state UI agencies consist of quarterly earnings, by employer, for all UI-covered employees in the state. The only way to accurately access an individual’s UI data is through their SSN; other identifiers such as name and date of birth are not unique enough to ensure that the correct data will be obtained. An advantage of UI data is that they can be collected for a large sample for a long period of time. Thus, they allow the researchers to estimate precisely, for the whole sample and for subgroups, the net impacts on employment and earnings over a three-year period. They are also fairly uniform across states and over time, a characteristic that facilitates a straightforward approach to analysis and allows for cross-site comparisons. Compared to survey data, these data have the advantage that they are not subject to the potential biases that can occur because of recall error and survey non-response.
Other data which are of a sensitive nature – including date of birth, ethnicity, race, citizenship status, marital status, number of children, education level, employment history and whether the individual is a TANF or SNAP recipient, work-related barriers -- are needed to support detailed analyses of the types of participants in the treatment and control groups receiving training services, as well as to conduct analyses of services received and educational/employment outcomes by various subgroups.
Participant contact information and name and contact information for up to three individuals who know the participant are necessary in order for the study staff to locate the participant should they lose contact in order to complete the follow-up surveys.
The time burden for administering the consent form and the BIF is estimated to be 13 minutes for each study participant. Based on the number of items on the form and previous experience on the Young Parents Demonstration Program conducted for ETA, we expect the consent form to take an average of 3.5 minutes to complete and the BIF an average of 9.5 minutes.
Again based on the number of items on the form and previous experience, the time for each staff member to process each intake form is also estimated to be 13 minutes. This will consist of two types of activities: (1) review of the information provided by the study participant on the BIF (5 minutes), and (2) entering of data from the BIF into the study’s PTS (8 minutes).
The estimated total hour burden of collecting information at intake is 2,600 hours (Table A.4). About 6,000 customers will complete the two intake forms (informed consent and BIF), and each customer will take an average of 13 minutes to complete the forms. Hence, the total time for customers to complete the forms is 6,000 × (13/60) hours, which equals 1,300 hours. About 24 staff will work with sample members to complete the BIF for about 250 individuals each. A grantee staff person will take about 13 minutes per study participant. The total staff time is 6,000 × (13/60) = 1,300 hours, or about 54 hours per staff person.
Table A.4. Burden Estimates for Study Participants and Staff
Respondents |
Number
of Respondents/ |
Frequency of Collection |
Average
Time |
Burden |
Intake Forms |
|
|
|
|
Study Participants |
6,000 |
Once |
13 minutes |
1,300 |
Staff
|
24 |
Once |
13 minutes per customer, with an average of 250 customers per staff respondent |
1,300 |
Total for Intake |
6,024 |
-- |
-- |
2,600 |
The estimated total burden for the data collection included in this request for clearance is 2,600 hours, which equals the sum of the estimated burden for study participants and grantee staff to complete the intake forms.
The estimated total burden for the data collection included in this request for clearance is 2,600 hours, which equals the sum of the estimated burden for study participants and grantee staff to complete the intake forms. Financial costs incurred by the survey respondents are minimal for completing the BIF as it is incorporated into the enrollment process for the training program. To calculate the 13 minute cost for each of the 6,000 participants to provide data as part of the enrollment process and for the staff to collect the data is estimated at $18.76 per person. The total annualized cost to participants and staff for the baseline data collection is presented below in Table A.5. The total estimated costs for these data collection activities are $24,388 x 2 = $48,776. The average hourly wage in that table for the PTS and baseline data collection is $18.76, based on the BLS average hourly earnings of production and nonsupervisory employees on private, service providing, nonfarm payrolls (September 2010 National Industry-Specific Occupational Employment and Wage Estimates, from the U.S. Department of Labor, Bureau of Labor Statistics and available on the department’s website).1
Table A.5(A). Total Annualized Cost Estimates for Baseline Data Collection
Data Collection Activity |
Total Burden Hours |
Average Hourly Wage |
Total Annualized Cost |
PTS – Baseline Data Collection and entry into the PTS |
|
|
|
Staff Participants
|
1,300 1,300 |
$18.76 $18.76 |
$24,388 $24,388 |
Financial costs incurred by the survey respondents are minimal for completing the BIF as it is incorporated into the enrollment process for the training program. The 13 minute cost for the staff to collect the data is estimated at $18.76 per person. The total annualized cost to staff for the baseline data collection is presented below in Table A.5(B). The average hourly wage in that table for the PTS and baseline data collection is $18.76, based on the BLS average hourly earnings of production and nonsupervisory employees on private, service providing, nonfarm payrolls (September 2010 National Industry-Specific Occupational Employment and Wage Estimates, from the U.S. Department of Labor, Bureau of Labor Statistics and available on the department’s website).2
Table A.5(B) Total Annualized Cost Estimates for Baseline Data Collection
Data Collection Activity |
Total Burden Hours |
Average Hourly Wage |
Total Annualized Cost |
PTS – Baseline Data Collection and entry into the PTS |
|
|
|
Staff
|
1,300 |
$18.76 |
$24,388 |
Following is the annual cost for the entire Green Jobs and Health Care Impact Evaluation of the Pathways Out of Poverty and Health Care and High Growth Training grant initiatives to the federal government:
Table A.6. Annual Costs
Year |
Dates |
Cost |
1 |
2010-2011 |
$1,466,492 |
2 |
2011-2012 |
$1,012,994 |
3 |
2012-2013 |
$1,690,244 |
4 |
2013-2014 |
$1,997,998 |
5 |
2014-2015 |
$1,825,124 |
Total |
|
$7,992,852 |
None
Time |
Activity |
Summer 2011 |
Baseline data collection began |
Winter 2013 |
Baseline data collection ends |
Fall 2014 |
Interim report summary published |
Summer 2015 |
Survey data collection ends |
Fall 2015 |
Final report summary published Public use data file available |
The expiration date for OMB approval will be displayed on all forms associated with this data collection.
Exception to the certification statement is not requested for the data collection.
1 U.S. Department of Labor, Bureau of Labor Statistics, Table B-8. Average hourly and weekly earnings of production and nonsupervisory employees on private nonfarm payrolls by industry sector, seasonally adjusted (accessed from the following website as of September 2010: http://www.bls.gov/webapps/legacy/cesbtab8.htm)
2 U.S. Department of Labor, Bureau of Labor Statistics, Table B-8. Average hourly and weekly earnings of production and nonsupervisory employees on private nonfarm payrolls by industry sector, seasonally adjusted (accessed from the following website as of September 2010: http://www.bls.gov/webapps/legacy/cesbtab8.htm)
| File Type | application/msword |
| File Title | Abt Single-Sided Body Template |
| Author | Melissa O'Connor |
| Last Modified By | naradzay.bonnie |
| File Modified | 2011-12-29 |
| File Created | 2011-12-29 |