Emergency submission; site visit data collection request for ARRA-funded grants; job training evaluation
Supporting Statement Part B
Part B: Collection of Information Involving Statistical Methods 1
1. Respondent Universe and Sampling 1
2. Analysis Methods and Degree of Accuracy 3
3. Methods to Maximize Response Rates and Data Reliability 3
The Employment and Training Administration (ETA) of the U.S. Department of Labor (DOL) is undertaking the evaluations of four Solicitations for Grant Applications (SGAs) that were issued under the American Recovery and Reinvestment Act (ARRA): Pathways Out of Poverty (POP), Energy Training Partnership (ETP), State Energy Sector Partnership (SESP), and Health Care and Other High Growth and Emerging Industries Training.
The overall aim of these evaluations is to determine the extent to which enrollees achieve increases in employment, earnings, and career advancement as a result of their participation in the training provided by Green Jobs and Health Care grantees and to identify promising best practices and strategies for replication. While the full evaluations involve several data collection efforts including surveys, this emergency clearance is requested only for site visit data collection for each of the studies.
Process Study Site Visits.
Implementation Evaluation:
For the implementation evaluation, one round of site visits is conducted to 36 grantees. The information collected through in-depth interviews during site visits, will be used to provide important contextual information on the effectiveness of the grants in different environments. It will also help ETA to assess whether the program is particularly effective in certain types of communities, with specific populations and in certain environments.
Impact Evaluation:
This research activity involves conducting two rounds of site visits to the grantees in the impact evaluation for the purpose of documenting the program environment, participant flow through random assignment and program services, the nature and content of the training provided, the control group environment and grantee perspectives on implementation challenges and intervention effects.
During the visits, site teams will interview key administrators and staff (including program partners and employers) using a semi-structured interview guide and focus groups held with participants (first round only). The interview guides are presented in Appendix A.
ETA is seeking emergency clearance at this time only for the site visit data collections. The request for clearance for the surveys will be submitted at a later date using the regular clearance process.
1. Respondent Universe and Sampling
Site Selection for the Implementation Study
Selecting a meaningful and representative sample of sites from each SGA is a critical design component associated with this study. Across the four SGAs, there are at total of 152 grantees. The research team plans to conduct site visits at 36 of these grantees. One approach to site selection would have been to randomly select these 36 sites. However, there is a possibility that doing so would result in a skewed sample which would over-represent or under-represent the 152 grantees along some dimension. For example, the 36 randomly selected sites might, by chance, contain a very small number – possibly even zero – of grantees from one of the SGAs. To address this concern, we adopted a methodology that included several steps.
To choose the 36 sites to visit, a matrix was developed which ranked the 152 grantees on five different categories based on guidance received from DOL/ETA. These five categories are listed below in descending order of importance:
Industry – Nursing/Allied Health, Other Healthcare/Emerging Industry, Renewable Electric Power, Energy Efficiency and Green Construction, and Other Green Industry
Grantee Organization Type – Community-Based Organization, Union, Education, Workforce Investment Board, and State Workforce Agency
SGA Type – SESP, HHG, POP, or ETP
Grantee Level – Local, Regional, State, or National
Geographic Diversity – the six DOL regions.
The site visits will involve semi-structured interviews with administrators and staff, key program partners, and employers. All individuals who serve these roles at each site, are able and willing, and consent to be interviewed will be included in the qualitative data collection; therefore, no sampling will be used. For the focus groups, site staff assist us in identifying approximately eight students who would be available to participate in a focus group; this will be a convenience sample and will not be intended to represent the broader group of participants.
DOL is selecting four sites from the universe of POP-Green Jobs and HCHGT grantees to participate in the evaluation based on their likely numbers of applicants, quality of implementation, early placement information, program service strategies, targeted industries, and appropriateness of implementing a random assignment design.
The site visits will involve semi-structured interviews with administrators and staff, key program partners, and employers. All individuals who serve these roles at each site, are able and willing, and consent to be interviewed will be included in the qualitative data collection; therefore, no sampling will be used. For the focus groups, site staff assist us in identifying approximately eight students who would be available to participate in a focus group; this will be a convenience sample and will not be intended to represent the broader group of participants.
Table B.1. Sample Size Requirements for Impact Study Visits
Sample Size Requirements: Process Study Visits |
|
Number of participating sites |
4 |
Number of participating individuals |
84 (average of 21 per site) |
Anticipated response rates |
100 percent
|
Anticipated number of respondents |
84
|
2.Analysis Methods and Degree of Accuracy
Statistical methods will not be used to select the sample of sites. Sites will be selected based on their likely numbers of applicants, quality of implementation, early placement information, program service strategies, targeted industries, and appropriateness of implementing a random assignment design.
The universe of individuals admitted by the selected sites into their grant programs during the study intake period will be included in the study. Statistical analysis of the baseline data will consist solely of descriptive tabulations (to profile the population participating in the grant programs examined). Because there will be no sampling variation in the data for this purpose, the data analysis will have a high degree of accuracy.
No statistical analysis will be performed on the qualitative data collected during the site visits for either study.
3. Methods to Maximize Response Rates and Data Reliability
The methods to maximize response rates and data reliability are discussed below for the site visit protocols.
Response rates .The strategy to collect qualitative data during site visits will ensure that response rates are high and that the data are reliable. Site visit team members will begin working with site staff well in advance of each visit to ensure that the timing of the visit is convenient.. Because the visits will involve several interviews and activities each day, there will be flexibility in the scheduling of specific interviews and activities to accommodate the particular needs of respondents and site operations.
Data Reliability. Strategies to ensure that the data are reliable and as nearly complete as possible include flexibility in scheduling of visits and the assurance given to respondents of privacy of the information that they provide. Furthermore, the neutral tone of the questions in the data collection protocols and the absence of sensitive questions, along with the training of the site visitors, will facilitate a high degree of accuracy in the data. In addition, shortly after each site visit, the site visit team members will synthesize the data from each interview, observation, and group discussion to the requirements of a structured write-up guide. Because most questions will be asked of more than one respondent during a visit, the analysis will allow for the triangulation of the data so that discrepancies among different respondents can be interpreted.
4. Tests of Procedures or Methods
To ensure that the site visit protocols are used effectively as field guides and that they yield comprehensive and comparable data across the study sites, senior project team members will conduct internal pilot tests of the interview guides with other research team members. The purposes of these tests are to ensure that the site visit protocols, which will guide researchers as they collect data on site, include appropriate probes that assist researchers in delving deeply into topics of interest and that the protocols do not omit relevant topics of inquiry. Furthermore, testing of the protocols can enable the staff leading this task to assess that the site visit agenda that the site visit team develops—including how data collection activities should generally be structured during each site visit—is practical given the amount of data that is to be collected and the amount of time allotted for each data collection activity. Adjustments to the site visit guides will be made as necessary.
5. Individuals Consulted on Statistical Methods
The data collection effort for the Implementation Study is primarily qualitative in nature; as such this study will not be using any statistical consultants for the project.
As noted earlier, consultations on the research design, sample design, and data collection procedures are part of the impact study, design phase of the evaluation. The purposes of these consultations are to ensure the technical soundness of the study and the relevance of its findings and to verify the importance, relevance, and accessibility of the information sought in the study.
Peer Review Panel Members of the Impact Study
1. Maureen Conway, [email protected]
2. Harry J. Holzer, [email protected]
3. Robert J. LaLonde, [email protected]
4. Larry Orr, [email protected]
5. Burt S. Barnow, [email protected]
6. Mindy Feldbaum, [email protected]
Bureau of Labor Statistics. An Occupational Analysis of Industries with Employment Gains: Occupational Employment Statistics (OES) Highlights.Washington: BLS, 2010.
Dohm, Arlene, and Lynn Shniper. “Occupational Employment Projections to 2016.”Monthly Labor Review, vol. 130, no. 11
| File Type | application/msword |
| File Title | Abt Single-Sided Body Template |
| Author | Melissa O'Connor |
| Last Modified By | Swick.Savi |
| File Modified | 2011-12-27 |
| File Created | 2011-12-27 |