May 2015
The request for approval of this Information Collection Request (ICR), entitled “Evaluation of the Trade Adjustment Assistance Community College Career Training (TAACCCT) Grants Program,” is for the site visit and survey data collection for the Rounds 1-3 TAACCCT grants, for which a 60-day public comment period was initiated through a Federal Register Notice (FRN) dated 8/1/2014. However, two other ICRs for the TAACCCT grants exist. The first ICR is to collect baseline data for an impact study and conduct implementation site visits for the Round 4 TAACCCT grants, entitled “Evaluation of Round 4 Trade Adjustment Assistance Community College Career Training (TAACCCT) Grants Program,” for which a 60-day public comment period was initiated through an FRN dated 2/26/2015. The second ICR is for an extension of the collection of performance information for the TAACCCT grants (Control No. 1205-0489), for which FRNs were published on 12/30/2014 and 3/30/2015.
The U.S. Department of Labor’s Chief Evaluation Office seeks to document and assess the overall Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant program through a national program evaluation. The evaluation encompasses an evaluability assessment, a formal implementation analysis, a performance assessment, and an outcome assessment. The national TAACCCT evaluation is designed to present a national view of the effectiveness of the grants in building capacity in community colleges across the nation that result in improved employment outcomes for participants, the challenges encountered in the implementation of the grants and ways to improve outcomes. To achieve this goal, the national evaluation conducted by the Urban Institute will collect survey and site visit data across grantees in a systematic manner and with similar protocols to maximize data comparability across grant sites.
The TAACCCT program is authorized by Division B of the American Recovery and Reinvestment Act of 2009 (P.L. 111-152), and the Health Care and Education Reconciliation Act of 2010 provided the program with $500,000,000 annually from Fiscal Years 2011-2014 for competitive grants to eligible institutions of higher education. The program aims to improve education and employment outcomes for TAA-eligible workers and other adults attending community college and other higher education institutions by helping these institutions build capacity to provide effective occupationally-focused education and training programs of two years or less in the 50 States, the District of Columbia and Puerto Rico. Funding for evaluation activities under this program have been designated by the Employment and Training Administration (ETA) for third-party evaluations of each grant and by the Chief Evaluation Office for a national evaluation. These evaluation activities will assist DOL in identifying evidence-based programs that are the most promising for TAA-eligible workers and other adults, examining how capacity building and systems reform in community colleges can be achieved, and developing strategies for research and evaluation of similar interventions.
The Solicitation for Grant Applications (SGAs) for the program include wording that alerted applicants to the possibility of a national evaluation. The Department of Labor Solicitation for Grants Application for Rounds 1-3 (SGA/DFA PY 11-08, SGA/DFA PY 10-03, and SGA/DFA PY 12-10) established that awarding of funds may require the cooperation of the grantee in an evaluation of overall performance of the TAACCCT grants.1 The SGA stated that by accepting grant funds, grantees must agree to participate in such an evaluation should they be selected to participate. The SGA also stated that DOL intends to select some portion of grantees to participate in a rigorous evaluation and depending on the evaluation design, grantees must be prepared to share records on participants, employers, funding, and outcomes.
Both proposed data collection activities – the college survey and site visits – are necessary to ensure that the national evaluation can adequately document and assess the overall TAACCCT program. While grantees are required to submit quarterly reports to ETA with aggregate participant and outcome data, these reports will not capture the expected variation in training and capacity-building approaches and these additional data collection activities are needed to provide a detailed yet comprehensive view of the activities funded by TAACCCT. In addition, many of the grantees are consortia made up of a group of colleges and no separate data are reported to ETA by consortia members. A standardized survey of all colleges of Rounds 1, 2 and 3, such as the one proposed, is demonstrably the best method to gather comprehensive information about grant activities undertaken by participating TAACCCT colleges, as well as assessing the extent to which grantees have achieved the main goals under the initiative.
The second data collection activity proposed is site visits to 20 grantees, 10 each for Rounds 2 and 3. The site visits will provide more in-depth qualitative information on the grants that cannot be captured through a survey. Semi-structured interviews will be held with a variety of stakeholders in the TAACCCT grants that will allow us to gain multiple perspectives on the grant activities, outcomes, and sustainability. In addition, focus groups with program participants will be held in the sites.
The college survey and site visit data will be analyzed as a part of the implementation analysis and evaluability assessment for the national evaluation of the TAACCCT program. The survey and site visit data will be important to developing the comprehensive understanding of the TAACCCT grants. The site visits will also be used to better understand how these types of training interventions, for which funding for capacity-building and systems reform is the primary activity – can be more rigorously evaluated.
Analysis based on these data will be used by DOL to understand how well these grants were implemented and how similar efforts can be evaluated in the future. Community colleges seeking to serve a range of students and tie their program to local labor demand, and students looking for information about effectiveness of available programs can also make use of other analysis based on these data and its tabulated form. All materials developed from the analyses of these data collection efforts are intended to reach multiple audiences including:
DOL and other federal leaders
Institutions of higher education and their trade groups
State and local workforce agencies and organizations
Industry groups
Researchers
Policymakers at the state and federal levels of government looking to design similar programs and
Others interested in understanding the experiences and lessons from the community college training and capacity-building programs
Survey participants will respond online to an electronic version of the survey. The web-based survey has been created and tested in Qualtrics, a commercial software application for development and administration of online surveys. The main advantage of the online survey is the automatic tabulation of responses that reduces both the hours of staff time needed for survey processing and the possibility of introducing human error into the data. The automated skip patterns embedded in the online survey also place less of a burden on the respondent than the customary “if-then go to” instructions of a paper and pencil questionnaire. (See Attachment 2 for the draft survey instrument). The web and paper versions of the questionnaire will both be in modular formats that allow the primary respondent to pass sections or questions on to other staff members who may be better equipped to address particular topics. Survey data will be stored on a confidential drive that only research team members can access.
Interviews during the site visits will be conducted face-to-face, although researchers may follow up with interviews by telephone if there is a need for further clarification after the visit. While on site, interviews will be conducted by teams of two researchers, one who will guide the discussion and one who will primarily take notes on a laptop computer. All hard copies of site visitor notes and audiotapes will be stored in a locked file cabinet when not in use. At the Urban Institute, electronic versions of site visitor notes will be stored on a confidential drive. Respondents will not interface with any automated, electronic, mechanical, or other technological collection techniques or other forms of information technology during this portion of the proposed data collection.
In the focus groups, researchers will take detailed notes during discussion groups and, with participants’ consent, record the discussions. The recordings will serve as a check against handwritten notes to ensure completeness and accuracy but will not be transcribed. Handwritten notes will be reviewed and cleaned the day of the discussion group. Researchers will type up interview notes either while on site, if possible, or shortly after returning from the site visit. The notes will be organized by topic to ensure efficient analysis and ease of access when writing the project report.
The information we propose to collect from TAACCCT grantees is not otherwise available. There is neither a systematic survey nor other qualitative assessment of the overall TAACCCT program being conducted. The information currently being collected from grant recipients through the narrative quarterly reports to ETA is not standardized in a way that allows data analysis and does not present the detail needed for a national evaluation.2 To the extent feasible, we will integrate the participant reports submitted by grant recipients into the analysis for this evaluation. However, those reports lack the kind of in-depth information on grant activities and programs that the survey and site visits will provide. We will also synthesize the findings from the third-party evaluations to support our analysis of the overall TAACCCT program.
The TAACCCT online college survey will not impact small businesses or other small entities. All survey respondents are community or technical colleges, community college districts, state community college systems or college consortia.
There is a small chance that the data collection during the site visits may impact a small business should one of the grantee partners be a small business. For grantee partners, participation in interviews is voluntary and a small business may choose not to be interviewed. In addition, interviews with partners will last no longer than one hour and can be done in person or by phone as is the most convenient for the small business (or any other) respondent.
Given the significant expenditures involved in the TAACCCT grants, and the role that this and similar grant programs are intended to play in shaping the nation’s workforce system, it is critical to document the different models and projects that are operating under the initiative, examine and assess the implementation to date, and identify innovative features and potentially promising strategies. The grantee survey and site visits are critical to this evaluation project, as they represent the only opportunity to gather comprehensive and in-depth information on implementation from all grantees in the first two grants. The data will be collected once.
There are no special circumstances that would cause this information collection to be conducted in a manner that would:
require respondents to report information to the agency more often than quarterly;
require respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
require respondents to submit more than an original and two copies of any document;
require respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records, for more than three years;
be in connection with a statistical survey, that is not designed to produce valid and reliable results that can be generalized to the universe of study;
require the use of statistical data classification that has not been reviewed and approved by OMB;
include a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
require respondents to submit proprietary trade secrets, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality .
Notification of this survey was published in the Federal Register on August 1, 2014 (79 FR 44868), a copy of which is in Attachment 1. The public was given 60 days from the date of publication to submit comments. No comments from the public during this time period were received.
A representative of the American Association of Community Colleges was consulted to review the survey instrument to ensure that the questions and response options were relevant to the experience of the TAACCCT institutions and could be understood by community colleges, the respondent type for the survey. The representative made suggestions for questions and wording that could improve the survey. These comments were similar to those received during the pretesting process (see B.4). One comment that was not incorporated into the revised survey instrument was to address the relationship between the grantee institution and member institutions for consortia grantees. This topic will be explored during the site visits, as it would be challenging to capture well from survey data.
Focus group participants will receive a payment of $20 in the form of cash or a gift card to help defray costs of transportation, child care, or other costs associated with attending the session. We will take into consideration community preferences regarding the form of the incentive (e.g., cash, gift card).
The TAACCCT colleges to which the survey is distributed, as well as any respondents interviewed during the site visits, will be assured that their responses will be kept private. Steps will be taken, in accordance with the Urban Institute Institutional Review Board (IRB) guidelines, to offer respondents the assurance that the information they provide is considered private and will not be shared with anyone outside of the research team in a manner that would allow respondent identification unless the research team is legally ordered otherwise. All findings from the survey will be presented at the aggregate level and with a minimum cell size of 3. Findings from the site visits will be presented at the organizational level, in order to provide detail and illustrative examples, but no individual respondents will be identified or quoted in any publication. Prior to collecting data, each survey and interview respondent will be given the pertinent privacy information, an explanation of the nature of the study, and a description of the time necessary to participate. However, no binding guarantees of confidentiality will be offered as privacy laws do not apply to this data collection. Please see the first pages of each data collection instrument – Attachments 2, 3, and 4 – for the respondent privacy statements imbedded in the informed consent procedures.
To protect survey respondent privacy, survey data will be secured (procedures are described in the response to item 3 above). While the survey is still active, access to any data with identifying information will be limited only to contractor staff directly working on the survey and will require special usernames and passwords. Once the survey is closed to respondents, responses will be downloaded for analysis from the SQL server database and kept on a controlled access, encrypted network drive. Hard copies of the survey will be entered into the electronic format and kept in a locked file cabinet in a designated Urban Institute employee’s office. All survey hard copies will be shredded upon completion of the evaluation.
To protect site visit respondent’s privacy, all hard copies of site visitor notes and audiotapes will be stored in a locked file cabinet when not in use. At the Urban Institute, electronic versions of site visitor notes will be stored on a confidential drive set up by its IT department. Access to this drive will be limited to research staff members who are working on the project and have signed the confidentiality pledge. A similar data security procedure will be followed for information obtained from the follow-up telephone interviews with program staff. Three years after the project is completed, notes will be shredded and electronic files securely deleted.
There are no questions of a sensitive, personal, or private nature included in the survey or the site visit interview guides.
Hour burden of the collection of information
All hour burdens refer to annual burden on respondents as this is a one-time data collection effort.
Survey
The survey will be fielded to all 867 colleges across all 178 Rounds 1-3 grant recipients. Respondents will be primary representatives of the participating colleges (including the grant organization) deemed to have sufficient knowledge of the TAACCCT grant activities to complete the survey. Specifically, the respondent will most likely be the program coordinator or an administrator at the college.
The estimated response rate for the one-time survey is 90 percent. Although participation in evaluation activities is required as a condition of the grant award, we expect that due to changes in staffing, about 10 percent of grantees will not respond to the survey.
The response time will average 90 minutes and will vary with the complexity of the grant activities. Table 1 provides an estimate of the respondent burden for completing the survey.
Table 1. Estimated Hour Burden for TAACCCT College Survey (Rounds 1-3)
Category |
Universe of Respondents |
Response Rate |
Number of Respondents |
Frequency |
Average Time Per Respondent |
Total Burden Hours |
Primary contact at participating colleges
|
867 |
90% |
780 |
Once |
90 minutes |
1,170 Hours |
Total |
867 |
|
780 |
|
|
1,170 hours |
Site Visits
Researchers propose to visit 20 grantee organizations and their partners for the site visit portion of the data collection. At each of these sites, we plan to interview the program coordinator and one other most knowledgeable individual about the grant activities. We will also interview approximately 10 additional respondents from the grantee organization depending on the program, including a college administrator, grant coordinator, instructional and support staff, data and financial management staff, and curriculum developers. In addition, we will interview industry and community partners such as the local workforce investment board or trade association and representatives from partner employers. We will also conduct focus groups with approximately 8 participants at each site, with a recruitment pool of 200. See Table 2 for more detail on the types of respondents and expected burden during the year of the site visits
The expected response rate by the grantees is 100 percent. Participation in evaluation activities is required as a condition of the grant award. The research team will schedule interviews in advance of arriving on site. We expect that 80 percent of the contacted potential participants for the focus groups will participate.
The primary contact at the grantee organization will assist the Urban Institute research team to identify appropriate contacts at partner organizations and schedule interviews. Since the research design only requires three partner interviews per site, and almost all grantees have at least three partner organizations, we anticipate little difficulty in recruiting the necessary number of partner respondents.
The primary contact at each site will spend an estimated four hours to complete the interview and assist the research team with site visit preparation. All other interviews will last approximately one hour. This time allowed for each interview may vary based on respondents’ knowledge but we expect to stay close to the average based on previous experience with similar site visits. The focus groups are expected to take no more than 75 minutes.
Table 2. Estimated Time Burden for Respondents to Interviews and Focus Groups for TAACCCT Fieldwork (20 visits total to Rounds 2 and 3)
Category |
Universe of Respondents |
Response Rate |
Number of Respondents |
Frequency |
Average Time Per Respondent |
Total Burden Hours |
Grantee institution primary contact |
20 |
100% |
20 |
Once |
4 hours |
80 hours |
College dean and other administra-tors |
20 |
100% |
20 |
Once |
60 minutes |
20 hours |
Staff at institution receiving grant funds |
200 |
100% |
200 |
Once |
75 minutes |
250 hours |
Staff at employer partner
|
60 |
100% |
60 |
Once |
60 minutes |
60 hours |
Staff at workforce investment partner
|
60 |
100% |
60 |
Once |
60 minutes |
60 hours |
Staff at other partners |
60 |
100% |
60 |
Once |
60 minutes |
60 hours |
Students at participating colleges (10 recruited per site for focus groups) |
200 |
80% |
160 |
Once |
75 minutes |
200 hours |
Total |
620 |
|
580 |
|
|
730 hours |
Table 3 presents the annualized costs to survey respondents, which occur within one year as this is a one-time data collection. This estimated cost for the staff at grant recipient organizations is based on median hourly wages for education administrators in post-secondary institutions, as listed in the May 2013 National Industry-Specific Occupational Employment and Wage Estimates, from the U.S. Department of Labor, Bureau of Labor Statistics and available on the Department’s website.3 The cost burden for site visit respondents was estimated using the median wages in May 2013 of the occupation most closely linked to the expected respondent.
Survey
The total annualized cost to respondents for the TAACCCT college survey is presented in Table 3. This estimated cost for the staff at participating colleges is based on median hourly wages for education administrators, post-secondary, at colleges, universities and professional schools, as listed in the May 2013 National Industry-Specific Occupational Employment and Wage Estimates, from the U.S. Department of Labor, Bureau of Labor Statistics and available on the Department’s web site. Please note that the costs to the respondents for completing the survey and participating in the site visit interviews are expected to come out of their grant funds since they are required to participate in evaluation activities as a condition of the grant award.
Table 3. Estimated Annualized Costs to TAACCCT College Survey Respondents Based on Hour Burden
Category |
Estimated number of respondents |
Total hours |
Median Hourly Wage |
Total Annualized Cost |
Education administrators, post-secondary (Grantee institution primary contact)4 |
780 |
1,170 hours |
$48.37 |
$56,592.90 |
Total |
780 |
1,170 hours |
|
$56,592.90 |
Site Visits
The total annualized cost to respondents for the TAACCCT site visits is presented in Table 4. The estimated cost to employer partner respondents is based on median hourly wages for administrative service managers in the manufacturing industry. The estimated cost to workforce investment system partner respondents is based on median hourly wages for local government managers, and the estimated cost to other partner agency respondents is based on median hourly wages for civic and other social organization managers. Students participating in focus groups will be compensated via a $20 gift card and this cost will be borne by the Urban Institute.
Table 4. Estimated Annualized Costs to TAACCCT Fieldwork Respondents Based on Hour Burden
Category |
Estimated number of respondents |
Total hours |
Median Hourly Wage |
Total Annualized Cost |
Education administrators, post-secondary (Grantee institution primary contact and college leaders) |
40 |
100 hours |
$48.37 |
$4,837.00 |
Education, Training and Library Occupations (Staff at institution receiving grant funds)5 |
200 |
250 |
$24.76 |
$6,190.00 |
Human resources managers (Employer partner respondents)6 |
60 |
60 hours |
$53.45 |
$3,207.00 |
Local government managers, excluding schools and hospitals (Workforce investment system partner respondents)7 |
60 |
60 hours |
$43.95 |
$2,637.00 |
Civic and social organization managers (Other partner respondents)8 |
60 |
60 hours |
$32.60 |
$1,956.00 |
Students at participating colleges (for focus groups) |
160 |
200 hours |
$0.00 |
$0.00 (no occupational code; will be compensated via incentive) |
Total |
580 |
730 hours |
|
$18,827.00 |
The total (one-time) annual cost of the data collection to the respondents is $75,419.90. Neither the survey nor the site visits will require respondents to purchase equipment or services or to establish new data retrieval mechanisms. There are no capital/start-up or ongoing operation/maintenance costs associated with this information collection. The content of the survey and the site visit instrument is based on the respondents’ experiences, opinions, and factual information. Therefore, the cost to respondents solely involves the time in answering the questions on the survey and the time to complete the interview. This is captured in the burden estimates provided in A.12.
The estimated cost of the proposed data collection effort to the Federal government is $665,484 and will be borne by the Department of Labor’s Chief Evaluation Office. The cost estimate is based on data collection tasks that cover implementation site visits and the college survey.
This is a new request.
After collecting survey and field data, the Urban Institute team will present it in summary formats that allow DOL and other stakeholders to better understand the variety of TAACCCT programs and their implementation. Details of the programs will be summarized and tables, charts, and graphs will be used to illustrate the results. A statistical software package, most likely STATA, will be used to conduct the analyses. The analysis will also integrate the findings from the document review and performance data to fully document and assess the overall TAACCCT grant program.
The analysis of survey data will immediately follow its collection. All analysis files will be downloaded from the SQL server database and kept on a controlled access, encrypted network drive. Qualitative and quantitative analytic activities related to the site visits will begin after the completion of all the site visits. The Urban Institute research team will prepare individual site summaries and perform a cross-site analysis of key topics such as program design and development, training and service delivery models, training typologies, capacity-building efforts, participant characteristics and progress, partnerships, participant outcomes, resource leveraging, potential for replication, and implementation lessons. The cross-site analysis will focus on the same key topic areas covered in the site summaries but will also capture similarities and differences between the sites in key programmatic and operational features and implementation experiences and challenges. Other existing data – grant applications, performance data, and third-party evaluation findings - will be analyzed along with the primary data from the survey and site visits. A TAACCCT database has been created to compile extant data in one place and for which the data can be tabulated and analyzed.
Once the data analysis is completed, the Urban Institute research team will prepare a final report and submit it to DOL. The report will include a stand-alone summary, an executive summary, the main body, and appendices with additional analyses from the survey. We anticipate that main sections of the final report will describe key findings, promising practices, and implementation challenges. The first, interim version of the report will include an analysis of the Round 1 and 2 grants and will be submitted in September 2016. The Round 3 grants will be incorporated into the findings of the interim report and a final report for the overall grant program will be issued. If agreed to by DOL and resources allow, the Urban Institute will also produce a short policy brief, geared toward a practitioner audience and highlighting key lessons and challenges in developing community college career training programs for TAA-eligible workers and other adults in need of skills and credentials.
In addition to the final report, the Urban Institute research team will produce two memos – one after each set of site visits. These memos will provide an assessment of the feasibility to use more rigorous evaluation methods for the TAACCCT and other capacity-building focused grants. The first memo will be submitted in summer 2015 and the second in summer 2016.
The OMB approval number and expiration date will be displayed or cited on all information collection instruments.
There are no exceptions to the certification statement.
1 All three SGAs can be found at http://www.doleta.gov/taaccct/applicantinfo.cfm.
2 OMB No. 1205-0N465.
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT FOR PAPERWORK REDUCTION ACT 1995 |
| Author | Administrator |
| Last Modified By | Javar, Janet O - ASP |
| File Modified | 2015-06-08 |
| File Created | 2015-06-08 |