final TAACCCT R4 OMB Package Supporting Statement Part A 7 10 2015

The Department of Labor (DOL) contracted with Abt Associates (in partnership with the Urban Institute) to conduct an evaluation of Round 4 of the Trade Adjustment Assistance Community College and Career Training (TAACCCT) grant program and is seeking approval from OMB under the Paperwork Reduction Act for data collection instruments associated with the evaluation. The TAACCCT grant program provides community colleges and other eligible institutions of higher education with funds to expand and improve their ability to deliver education and career training programs that can be completed in two years or less and are suited for workers who are eligible for training under the Trade Adjustment Assistance for Workers (TAA) Program. TAACCCT-supported programs prepare participants for employment in high-wage, high-skill occupations. Through these multi-year grants, DOL is helping to ensure that America’s institutions of higher education are helping adults succeed in acquiring the skills, degrees, and credentials needed for high-wage, high-skill employment while also meeting the needs of employers for skilled workers. DOL is implementing the TAACCCT program in partnership with the U.S. Department of Education. A total of 49 grants were awarded in Round 1 in FY 2011, 72 in Round 2 in FY 2012, 57 in Round 3 in FY 2013, and 71 in Round 4 in FY 2014. Some of the grants were awarded to single institutions and others to consortia of institutions.

Data collection described in this submission pertains only to Evaluation of Round 4 Trade Adjustment Assistance Community College and Career Training Grants. Note that separate OMB packages are currently being drafted to obtain clearance for two other information collection requests (ICRs). The first ICR is for site visits and survey data collection for the Rounds 1-3 TAACCCT grants, entitled “Evaluation of the Trade Adjustment Assistance Community College Career Training Grants Program” (Federal Register Notice dated 8/1/2014). The second ICR is for an extension of the collection of performance information for the TAACCCT grants (Control No. 1205-0489), for which FRNs were published on 12/30/2014 and 3/30/2015.

The evaluation, funded by the DOL Chief Evaluation Office, will use a multi-pronged approach including 1) an impact study of selected grantees, 2) an implementation analysis, and 3) a synthesis of the independent third-party evaluations that all Round 4 grantees are required to procure. The Round 4 national evaluator, through collaboration with grantees and their evaluators, also will provide an assessment of the overall initiative.

In this document, we request OMB clearance for the first in a series of data collection activities for the evaluation of Round 4 TAACCCT grants. This package requests clearance for 1) collecting baseline data to support the impact study of up to five selected Round 4 grantees and 2) semi-structured fieldwork in the form of site visits to up to nine Round 4 grantees as part of the implementation analysis.

To support the impact study, two data collection instruments will be administered to all study participants in the selected grantees. One is a short baseline information form (BIF) that will collect basic identification, demographic, and contact information. The second is a self-administered questionnaire (SAQ), covering information related to TAACCCT goals.

The semi-structured fieldwork will involve interviews with college administrators, program coordinators, faculty and instructional staff, industry and community partners, and employers. Field researchers will use a modular interview guide that can be adapted based on the respondent’s knowledge base.

Subsequent OMB submissions will seek clearance for additional evaluation of Round 4 TAACCCT data collection activities.

A1: Necessity for the Data Collection

The TAACCCT program is authorized by Division B of the American Recovery and Reinvestment Act of 2009 (P.L. 111-152), and the Health Care and Education Reconciliation Act of 2010 appropriated $500,000,000 annually from Fiscal Years 2011-2014 for competitive grants to eligible institutions of higher education. The program aims to improve education and employment outcomes for TAA-eligible workers and other adults attending community college and other higher education institutions by helping these institutions build capacity to provide effective occupationally-focused education and training programs of two years or less in duration in the 50 States, the District of Columbia and Puerto Rico. Funding for evaluation activities under this program have been designated by the Employment and Training Administration for third-party evaluations of each grant and by the Chief Evaluation Office for a national evaluation. These evaluation activities will assist DOL in identifying evidence-based programs that are the most promising for TAA-eligible workers and other adults, examining how capacity building and systems reform in community colleges can be achieved, and developing strategies for research and evaluation of similar interventions.

The Department of Labor Solicitation for Grant Applications for Round 4 (SGA/DFA PY 13-10) established that awarding of funds may require the cooperation of the grantee in an evaluation of overall performance of the TAACCCT grants, including an impact study.¹
DOL’s Chief Evaluation Office seeks to document and assess the overall TAACCCT grant program through a national program evaluation. For Round 4, the evaluation encompasses an impact study of selected grantees, an implementation analysis, and a synthesis of third-party evaluations. Building on work of the national evaluations of Rounds 1-3, the evaluation of Round 4 grants is designed to present a national view of the effectiveness of the grants in building capacity in community colleges across the nation that result in improved employment outcomes for participants, the challenges encountered in the implementation of the grants, and ways to improve outcomes.

This evaluation is designed to answer the following research questions:

What service delivery and/or system reform innovations resulted in improved employment outcomes and increased skills for participants?
Under what conditions can these innovations most effectively be replicated?
What impacts were achieved by the participants of the TAACCCT Round 4 grantees’ training programs?
What are the types of emerging ideas for service delivery change and/or system reform that seem the most promising for further research? Under what conditions are these ideas most effective?
What directions for future research on the country’s public workforce system, and workforce development in general, were learned?

To address these research questions, the evaluation of Round 4 TAACCCT will include the following:

Baseline data collection (for treatment and control group members in impact study sites)

Implementation site visits (up to 9 sites)
A 12-month follow-up study participant survey (for treatment and control group members in impact study sites)
A college survey (of all colleges participating in Round 4 TAACCCT grants)
An employer study
A match to the National Directory of New Hires (NDNH) to obtain earnings histories

This submission is to request clearance for the first and second data collection components listed above. Both proposed data collection activities – the baseline information data collection for the impact study and site visits – are necessary to ensure that the national evaluation can adequately document and assess the overall TAACCCT program. DOL anticipates submitting a single additional OMB package to request permission to conduct the third, fourth and fifth study components. The sixth component is not subject to PRA review, but it is an important part of the evaluation design that greatly reduces the response burden on study participants by decreasing the length of the baseline and 12-month follow-up surveys.

As mentioned in the overview, baseline data collection will involve two forms: the baseline information form (BIF) (Appendix C.1) and the self-administered questionnaire (SAQ) (Appendix C.2). The primary purpose of the information collected with both forms is to provide a rich set of covariates that the evaluators can use to boost power through regression-based estimation of treatment effects and for survey nonresponse adjustment. A secondary purpose of the BIF is to gain an improved understanding of the population served by TAACCCT-supported grants. The evaluators might also use the information to define subgroups if sample sizes allow sufficient power for such analyses.

The second data collection activity proposed is site visits to the (up to five) impact study grantees and up to four additional Round 4 grantees. The site visits will provide more in-depth qualitative information on the grants that cannot be captured through the college survey. Semi-structured interviews will be held with a variety of stakeholders in the TAACCCT grants that will allow the research team to gain multiple perspectives on the grant activities, outcomes, and sustainability.

A2: Purpose and Users of Information

The immediate users of data collected in the BIF and SAQ will be the Abt Associates evaluation team, who will use the data to reduce follow-up non-response biases and improve the power of the impact study to detect the effects of participation in TAACCCT-funded programs. Immediate users of data collected in site visits will be the Abt evaluators, who will use the information to improve the generalizability and replicability of report findings.

Indirect users of both types of data will be policy analysts at DOL, Congress, and elsewhere who may base future policies, at least in part, on the findings of the evaluation. Program designers will also be indirect users of the information through the findings of the evaluation. Community colleges seeking to serve a range of students and tie their program to local labor demand, and students looking for information about effectiveness of available programs can also make use of other analysis based on these data and its tabulated form. All materials developed from the analyses of these data collection efforts are intended to reach multiple audiences including:

DOL and other federal agency staff
Institutions of higher education and their trade groups
State and local workforce agencies and organizations
Industry groups
Researchers
Policymakers at the local, state and federal levels of government looking to design similar programs
Others interested in understanding the experiences and lessons from the community college training and capacity-building programs

Finally, a Public Use File will be prepared and made available to future researchers.

A3: Improved Information Technology to Reduce Burden

Depending on the grantee’s intake procedures, each Round 4 TAACCCT impact study site will determine whether completion of the BIF is most appropriate by electronic means or on paper. If by electronic means, a site staff member will ask the applicant questions from the BIF and input answers directly into the secure, web-based database designed to collect this information. Or, the participant will fill out a paper BIF and staff will enter the data into the system at a later time. In either scenario, the web-based system technology enhances data quality in several ways. First, computerized questionnaires ensure that the skip patterns work properly. Including efficient skip patterns minimizes respondent burden by not asking inappropriate or non-applicable questions (or in the case of staff entering data at a later time, mitigating staff data entry burden). Second, computerized questionnaires can build in checkpoints which allow the interviewer or data entry staff to confirm responses thereby minimizing data entry errors. Finally, automated survey administration can incorporate hard edits to check for allowable ranges for quantity and range value questions, minimizing out of range or unallowable values.

The SAQ will always be administered by paper to encourage more honest reporting. The psycho-social nature of the questions may be considered by some personal. By ensuring that staff will not see these responses, as study participants will complete the form and seal it in an envelope that will be returned unopened to the research team, participants will be encouraged to answer the questions truthfully.

The BIF data will be entered into a secure website maintained by Abt’s subcontractor, Relyon Media. Staff at the grantee or sub-grantee level who are granted authorization to access the system will receive a secure password and will be able to enter and/or view data on their program’s participants (but not those in programs operated by other grantees in the study).

A4: Efforts to Identify Duplication

This will be the first impact evaluation of the TAACCCT grant program. No prior parallel research exists. DOL fully expects the services funded with TAACCCT grants to serve a wide range of workers beyond veterans and TAA-eligible workers. The current research will focus on the benefits of these newly created programs to these broader service populations.

Regarding the information collected in the BIF and SAQ, there is no other source of the data than the program applicants themselves. Grantees may collect similar information as part of their application process. However, it is likely that the content of college-specific admission or enrollment forms will differ by study site. For the impact study, it is critical that the evaluation team collect identical information from study participants across all sites.

Regarding information collected from grantees during site visits, we have designed the protocols to complement information provided by grantees in their grant applications and annual reports. In addition, grant agreements include funds and mandates that grantees participate in data collection. There should be no adverse impact for any grantees participating in the study.

A5: Involvement of Small Organizations

None of the TAACCCT grantees are small businesses or entities. The evaluation of TAACCCT grant program will impose no burden on this sector of the economy.

A6: Consequences of Less Frequent Data Collection

This is a one-time data collection activity. Appendix E provides rationales for items in the BIF and SAQ. Consequences for not collecting BIF and SAQ data at baseline are:

Obtaining baseline data allows the TAACCCT evaluation team to describe the research sample’s descriptive characteristics as well as confirming balance between the treatment and control groups. Without baseline data, the evaluation team would lack the ability to confirm that random assignment produced statistically similar treatment and control groups. Equally important, the team would not be able to describe in more detail the population these innovative programs are serving.
Without baseline values of measures correlated with outcomes, estimates for the impact analysis would be less precise. The more precise the estimates are, the more confident evaluators can be in determining which strategies improve outcomes for individuals.
Lack of baseline data would mean the TAACCCT evaluation team would be less able to detect and correct survey nonresponse biases.
Lack of baseline data would mean the TAACCCT evaluation team would be unable to estimate impacts for subgroups of interest to provide insights for improved program targeting.

Site visits are critical to the implementation study, which will document the TAACCCT-funded programs as implemented, help to explain impact study findings, and promote deeper learning from the impact study. Without the site visits the evaluation team would not be able to adequately characterize the nature of the program changes brought about with the TAACCCT funding. This would reduce the generalizability and replicability of the findings. Without this information, the team could still report on the effects of TAACCCT funding to five grantees which would be of interest to federal policy analysts, but it would be very difficult for program designers to learn anything from the evaluation. Specifically, they would not learn how to replicate any successes or avoid the same problems as they design similar programs for other community college systems.

A7: Special Circumstances

There are no special circumstances for the proposed data collection.

A8: FRN and Consultation

Federal Register Notice and Comments

In accordance with the Paperwork Reduction Act of 1995 (Pub. L. 104-13 and Office of Management and Budget (OMB) regulations at 5 CFR Part 1320 (60 FR 44978, August 29, 1995)), DOL published a notice in the Federal Register announcing the agency’s intention to request an OMB review of this information collection activity. This notice was published on Thursday, February 26, 2015, 80 FR 10515, and provided a 60 day period for public comment. A copy of this notice is included as Appendix D. During the notice and comment period, the government did not receive any comment or request for copies of the instrument.

Outside Consultation

The evaluation team consulted with the representatives of the Employment and Training Administration (ETA) within DOL. ETA administers that grant program and therefore keeps in contact with the grantees. The evaluation team also hosted a webinar for grantees to inform them about the impact study and offer them the opportunity to ask questions.

In addition, many of the items in the Self-Administered Questionnaire are from previously approved data collection instruments for Innovative Strategies for Increasing Self-Sufficiency study (now known as the Pathways for Advancing Careers and Education study) (clearance number 0970-0343).

A9: Payment of Respondents

No payments to respondents are involved in the collection of baseline data through the BIF or SAQ.

A10: Confidentiality of Respondents

The Round 4 TAACCCT impact study baseline data collection will include personally identifiable information (PII) collected through the BIF. The SAQ will not include any PII, but will include a unique study identifier.

Respondent privacy will be protected to the extent allowed by law. DOL recognizes that TAACCCT grantees serve vulnerable populations (per the authorizing legislation), and that grantees must protect study participants from any risks of harm from evaluation activities. Accordingly, the TAACCCT impact study will obtain informed consent from all study participants. Administering an Informed Consent Form will ensure that participants understand the nature of the research and data collection. The Informed Consent Forms are included as Appendices B.1 (Informed Consent Form for Randomized Program Applicants) and B.2 (Informed Consent Form for Auxiliary Data Collection on Veterans and TAA-Eligible Workers).

As a part of informed consent, the following rationale for data collection and privacy assurances will be provided to TAACCCT participants by grantees selected for the impact study:

Research is being conducted to see how well various approaches to training work. This program and research are funded by the U.S. Department of Labor.
The program will collect some personal information from you, such as your name, date of birth, Social Security Number, and your involvement in other programs. The researchers studying the program for the government also need this information. All of the information about you collected for the program or for the research studies will be kept completely private to the extent allowed by law, and no one’s name will ever appear in any report or discussion of the evaluation results.
As part of the study, the researchers will contact you approximately one year from now to ask questions about your educational progress, work, and other topics. They may also contact you for up to two additional surveys. You can refuse to participate in the interviews or answer any of the questions.²
Researchers and program staff using the information collected will take strong actions to protect your information.

BIF data will be entered by staff into the secure web-based system. Hard copies of the BIF, and sealed envelopes with the SAQs, will be shipped to the evaluation team via Federal Express to enable tracking of the packages. Forms will be stored in locked file cabinets at Abt Associates.

To protect site visit respondents’ privacy, all hard copies of site visitor notes will be stored in a locked file cabinet when not in use. Electronic versions of site visitor notes will be stored on a password protected drive set up by Abt’s IT department. Access to this drive will be limited to research staff members who are working on the project and have signed the confidentiality pledge.

The consent form will hold open the possibility of two additional rounds of follow-up, although these are not currently planned. This possibility is mentioned in order to maintain flexibility for DOL to assess longer-term outcomes of study participants. The Berk (2012) evaluation of TAA training found that it took four years for negative effects on employment and earnings to fade out. Three years after the project (or last round of followup) is completed, hard copies of forms and site visit notes will be shredded and electronic files securely deleted.

DOL is planning on producing a Public Use File (PUF) at the end of the evaluation contract that will include data from the BIF, SAQ and 12-month follow-up questionnaire. PII will be removed, and appropriate measures will be undertaken to reduce the risk of re-identification. The PUF will not contain any administrative data from any source.³

A11: Sensitive Questions

Basic Information Form and Self-Administered Questionnaire Questions

There are no sensitive questions on the BIF that are commonly considered private other than the request for PII. There is a question about felony convictions, but this information is a matter of public record. Program staff will notify potential study members during the enrollment process that they may refuse to answer any individual BIF or SAQ items. Potential study members will also be notified that their responses will be kept private, to the extent allowable under law, to encourage their candid responses.

The SAQ includes questions that some respondents may consider sensitive, which is why the form will be administered on paper, sealed in an envelope by the study participant and not viewed by staff. The literature provides ample support for including these items as factors associated with success in education and employment as outlined in Appendix E. Including these items will boost statistical power, enabling the evaluation team to reduce sample sizes below the levels that would otherwise be required. The Round 4 TAACCCT evaluation team will treat all information collected on the BIF and the SAQ as private.

A12: Estimation of Information Collection Burden

Baseline data collection will start as soon as possible in 2015 (preferably October 1^st, pending OMB approval) and continue through May 2017. Exhibit A12 presents the estimated respondent burden on study participants for baseline data collection. It shows the average time, in hours, that the evaluators estimate completing intake surveys for participants. Response times were estimated based on pre-testing of the BIF and SAQ, as well as prior experience of the evaluators with similar data collection. Each study participant will sign the informed consent and fill out the BIF and SAQ at intake.

As discussed in Part B of this request (under special populations), the evaluators propose administering the BIF and SAQ to some applicants who are ineligible for the study: veterans and TAA-eligible individuals who are target populations for the grants but cannot be randomly assigned per DOL policy. Although information collected from veterans and TAA-eligible workers cannot be used to assess impacts of the individual grant program, DOL currently lacks information about the extent to which TAACCCT grantees serve these target populations, the characteristics of those who do participate, and the outcomes of training. The baseline data collection provides DOL an opportunity to collect a rich set of economic and demographic information about key target populations, and, if the sample size is large enough to warrant the effort, matching this sample to the NDNH to learn about employment-related outcomes of training.

The evaluators estimate that the entire baseline data collection process will take study participants on average 22 minutes to complete. The team estimates that completing the BIF will take 15 minutes and the SAQ 7 minutes. The evaluation team anticipates 5,500 study participants (5,000 participants randomized for the impact study, and 500 TAA-eligible workers and veterans). The total burden hours for baseline data collection is therefore 5,500 x 22/60 = 2,018.5 hours, or 1,375 hours for the BIF and 643.5 hours for the SAQ. Because this is a one-time data collection that will occur in each grantee site approximately over the course of a year, the annual data collection burden also is 2,018.5 hours. These estimates do not include the burden associated with any potential follow-up surveys completed by program participants.

Exhibit A12 also shows the estimated burden on program staff and partners participating in key informant interviews for the implementation study. TAACCCT researchers will interview up to 30 staff and partners at up to 9 grants. Each interview will take approximately 1.5 hours. At each of these sites, the evaluators plan to interview the program coordinator and other knowledgeable individuals about the grant activities. They will also interview approximately 12 additional respondents from the grantee organization or consortium partner depending on the program, including a college administrator, grant coordinator, instructional and support staff, data and financial management staff, and curriculum developers. In addition, the evaluators will interview industry and community partners such as the local workforce investment board or trade association and representatives from partner employers.

The expected response rate by the grantees is 100 percent. Participation in evaluation activities is required as a condition of the grant award. The primary contact at the grantee organization will assist the TAACCCT research team to identify appropriate contacts at partner organizations and schedule interviews at least a month prior to the visit. This time allowed for each interview may vary based on respondents’ knowledge but we expect to stay close to the average based on previous experience with similar site visits. Total projected burden across respondents for the site visits is 405 hours.

Exhibit A12: Annual Information Collection Activities and Cost (Baseline Data Collection for Randomized Evaluation)

Instrument		Annual Number of Respondents	Number of Responses Per Respondent	Average Burden Hours Per Response	Total Burden Hours	Average Hourly Wage	Total Annual Cost
Baseline Information Form (BIF)		5500^a	1	0.25	1,375	$7.25^b	$9,969
Self-Administered Questionnaire (SAQ)		5500^a	1	0.117	645	$7.25^b	$4,676
Interview Protocol – Grant Director		9^c	1	1.5	14	$43.83^d	$614
Interview Protocol –Community College Leadership		54^c	1	1.5	81	$43.83^e	$3,550
Interview Protocol – Community College Staff		108^c	1	1.5	162	$29.96^f	$4,854
Interview Protocol -Employer Partners		36^c	1	1.5	54	$47.81^g	$2,582
Interview Protocol- Workforce investment system partners		36^c	1	1.5	54	$38.25^h	$2,066
Interview Protocol- CFBOs, Social Service Partners		27^c	1	1.5	41	$33.45ⁱ	$1,371
	Total				2,423.5		$29,682

^a This includes 5,000 participants (2,500 treatment and 2,500 control group members) in the impact study, and 500 TAA-eligible workers and veterans.

^b The hourly wage of $7.25 is the federal minimum wage (effective July 24, 2009) http://www.dol.gov/dol/topic/wages/minimumwage.htm.

^cThe evaluation team plans to interview up to 30 staff and partners in each of 9 grants: one grant director in each of 9 grants; six community college leaders in each of 9 grants; an additional 12 staff in each of 9 grants; four employer partners in each of 9 grants; four workforce investment system partners in each of 9 grants; and three CFBOs or social service partners in each of 9 grants.

^dThe median hourly wage for Grant Directors was calculated based on information from the Bureau of Labor Statistics; May 2014 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/naics4_611300.htm: Education administrators, postsecondary (NAICS 611300, SOC code 11-9033) wage rate of $43.83.

^eThe median hourly wage for Community College Leadership was calculated based on information from the Bureau of Labor Statistics; May 2014 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/naics4_611300.htm Education administrators, postsecondary (NAICS 611300, SOC code 11-9033) wage rate of $43.83.

^fThe median hourly wage for Community College Staff was calculated based on information from the Bureau of Labor Statistics; May 2014 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/naics4_611300.htm: Education, Training and Library Occupations (NAICS 611300,SOC code 25-0000) wage rate of $29.96.

^gThe median hourly wage for Employer Partners was calculated based on information from the Bureau of Labor Statistics; May 2014 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/naics2_31-33.htm#11-0000: Human Resources Managers (manufacturing, SOC code 11-3121,) wage rate of $47.81.

^hThe median hourly wage for Workforce Investment System Partners was calculated based on information from the Bureau of Labor Statistics; May 2014 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/naics4_999300.htm: Local government managers, excluding schools and hospitals (NAICS 999300, SOC code 11-0000) wage rate of $38.25.

ⁱThe median hourly wage for CFBO, Social Service Partners was calculated based on information from the Bureau of Labor Statistics; May 2014 National Industry-Specific Occupational Employment and Wage Estimates found at http://www.bls.gov/oes/current/naics4_813400.htm: Civic and social organization managers (NAICS 813400, SOC code 11-0000) wage of $33.45.

A13: Cost Burden to Respondents or Record Keepers

There are no direct costs to respondents, and they will incur no start-up or ongoing financial costs. The cost to respondents involves solely the time involved for the interviews, BIFs, SAQs, and informed consents which are captured in the burden estimates in Exhibit A12.

A14: Estimate of Cost to the Federal Government

The information collection activity and associated instruments have been developed by the evaluation contractor, Abt Associates, in performance of Contract Number: GSA MOBIS 874-01, Order Number: DOL-ETA-14-F-00013. The Chief Evaluation Office is funding the costs of the study. The proposed baseline and site-visit data collection will take place beginning as early as possible on or after October 1, 2015 and will end no later than May 31, 2017.

The total annualized cost to the federal government is $145,529. Costs result from the following two categories:

1. The estimated cost to the federal government for the contractor to carry out this study is $161,383 for baseline data collection (including forms design, forms, and data entry) and $257,101 for the implementation site visits (including instrument design). Annualized over three years, this comes to $139,494 (($161,383+$257,101)/3=$139,494).

2. The annual cost borne by DOL for federal technical staff to oversee the contract is estimated to be $18,102.40. DOL expects the annual level of effort to perform these duties will require 200 hours for one Washington D.C. based Federal GS 14 step 4 employee earning $56.57 per hour. (See Office of Personnel Management 2015 Hourly Salary Table at http://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/salary-tables/pdf/2015/DCB_h.pdf ). To account for fringe benefits and other overhead costs the agency has applied multiplication factor of 1.6: 200 hours x $56.57 x 1.6 = $18,102.

TOTAL FEDERAL COST is $145,529 ($418,484 + $18,102 = $436,586).

A15: Change in Burden

This is a new data collection.

A16: Plan and Time Schedule for Information Collection, Tabulation and Publication

16.1 Analysis Plan

The evaluation will cover both implementation and impact analysis. Because implementation data collection will conclude prior to impact data collection (Project Year 3 versus Project Year 4) the team will begin implementation analysis first. This lag will allow the implementation findings to help inform impact analyses (e.g., they will suggest impact study exploratory questions).

Implementation Analysis. The evaluation team will analyze 1) data collected as part of the site visits to the five impact study sites; 2) data collected from four additional grantees for special topic studies; and 3) data collected from the college survey (a subsequent OMB package will include this instrument). For the five impact study sites, following the site visits the team will organize data collected into short summaries that align with the TAACCCT conceptual framework (see Appendix A). This will provide a portrait of each grantee in terms of identified workforce needs, grant activities, capacity building operations, expected outcomes, and program context. This summary will also describe treatment/control group differences in experiences. These summaries will suggest exploratory questions for the impact study and will help the team interpret impact study findings. The evaluation team will also synthesize the findings to describe the similarities, differences, implementation successes and challenges across the five grantees. Special topics analyses will follow a similar model; the evaluators will create summary reports that highlight key findings.

Impact Analysis. The evaluation team anticipates analyzing each study site separately, with a single confirmatory outcome for each. The team will estimate the impact on those offered each treatment (referred to as Intention to Treat or ITT) as well as the impact on those actually receiving the treatment (the effect of the Treatment on the Treated or TOT). The single confirmatory outcome will be related to credential attainment, with the exact details of which credentials to be considered depending on the site. Other outcomes that the team examines will be treated as exploratory in the presentation of results. If it turns out that it appears to be essential to have multiple confirmatory outcomes, then the evaluation team will apply standard corrections for multiple comparison testing.

Although it is possible to estimate the ITT estimate of the impact of treatment for a site by simply computing the difference in means between the two groups at the site, more precise estimates can be obtained by regression analysis to compensate for chance imbalances in prespecified covariates such as baseline education between the treatment and control groups. Such regression analysis is more typically used on continuous outcomes such as earnings, but it also works well for binary outcome variables such as credential attainment (e.g., Judkins and Porter, 2013). Blank survey items and attrition can cause bias in the ITT estimates. The evaluation team will reduce this risk through a mixture of techniques known as multiple imputation and nonresponse weighting that exploit what was known about the study participants at baseline (from the BIF and SAQ) and what is learned about them post-randomization from the NDNH (Molenberghs and Kenward, 2007). Estimation of TOT requires stronger assumptions than estimation of ITT. As is standard in the industry to produce such estimates, the evaluation team will assume that there was no effect on participants who were randomly assigned to the treatment group but did not avail themselves of the opportunity in some meaningful way (Angrist, Imbens and Rubin, 1996).

16.2 Time Schedule and Publications

For the evaluation of the Round 4 TAACCCT grants, CEO is seeking OMB approval beginning October 1, 2015 and ending September 29, 2018.

Exhibit 16.2 presents an overview of the project schedule for information collection. It also identifies deliverables associated with each major data collection activity.

Exhibit A16.2 Overview of Project Data Collection Schedule

Data Collection Activity	Timing	Associated Publications
Baseline data collection (for program participants and control group members)	Beginning as early as October 1, 2015, or when OMB approval is received, if later. Baseline data collection is expected to end by May 31, 2017.	Final report
12-month follow up survey (for program participants and control group members)*	From December 2016 – Fall 2018.	Final report
College survey*	Fall 2017	Final report
Site visits, observations, staff and management interview	Fall 2016- Spring 2017	Final report
Employer survey*	TBD	TBD

*DOL will request permission for these data collection activities (2, 3, and 5) in a subsequent package.

A17: Reasons not to Display OMB Expiration Date

All instruments created for the TAACCCT Round 4 impact study and national evaluation will display the OMB approval number and the expiration date for OMB approval.

A18: Exceptions to Certification for Paperwork Reduction Act Submissions

No exceptions are necessary for this information collection.

References

Alexander, L. and Martray, C. (1989). The development of an abbreviated version of the mathematics anxiety rating scale. Measurement and Evaluation in Counseling and Development, 22, 143-150.

Angrist, J. D., Imbens, G. W., and Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91, 444-455.

Berk, J. (2012). Understanding the Employment Outcomes of Trainees in the Trade Adjustment Assistance (TAA) Program Under the 2002 Amendments: Final Report – Prepared as Part of the Evaluation of the Trade Adjustment Assistance Program. Princeton: Mathematica Policy research.

Cohen, S., Kamarck, T., and Mermelstein, R. (1983). A global measure of perceived stress. Journal of Health and Social Behavior, 24(4), 385-396.

Duckworth, A.L., Peterson, C., Matthews, M.D., and Kelly, D.R. (2007). Grit: Perseverance and passion for long-term goals. Journal of Personality and Social Psychology, 92(6), 1087-1101.

Heidkamp, M., Mabe, W., and DeGraaf, B. (2012). The Public Workforce System: Serving Older Job Seekers and the Disability Implications of an Aging Workforce. The NTAR Leadership Center: Rutgers, NJ. http://www.dol.gov/odep/pdf/NTAR_Public_Workforce_System_Report_Final.pdf

Hopko, D.R., Mahadevan, R., Bare, R.L., and Hunt, M.K. (2003). The Abbreviated Math Anxiety Scale (AMAS): Construction, validity, and reliability. Assessment, 10(2), 178-182.

Jameson, M.M. and Fusco, B.R. (2014). Math anxiety, math self-concept, and math self-efficacy in adult learners compared to traditional undergraduate students. Adult Education Quarterly, 64(4), 306-322.

Judge, T.A. (2009). Core self-evaluations and work success. Current Directions in Psychological Science, 18(1), 58-62.

Judkins, D.R., and Porter, K.E. (2013). The remarkable robustness of ordinary least squares in randomized clinical trials. In JSM Proceedings, Social Statistics Section. Alexandria, VA: American Statistical Association. 1235‑1247.

Levine, L. (2010). Older displaced workers in the context of an aging and slowly growing population. Washington, D.C.: Congressional Research Service. Retrieved from: http://aging.senate.gov/crs/aging22.pdf.

Molenberghs, G., and Kenward, M.G. (2007). Missing Data in Clinical Studies, Chichester: John Wiley & Sons.NCES (2012). http://nces.ed.gov/surveys/piaac/questionnaire.asp.

Plake, B.S. and Parker, C.S. (1982). The development and validation of a revised version of the Mathematics Anxiety Rating Scale. Educational and Psychological Measurement, 42, 551-557.

Richardson, F.C. and Suinn, R.M. (1972). The Mathematics Anxiety Rating Scale. Journal of Counseling Psychology, 19, 551-554.

Snyder, T.D. and Dillow, S.A. (2010). Digest of Education Statistics 2009 (NCES 2010-013). National Center for Education Statistics. Institute of Education Sciences. U.S. Department of Education. Washington, DC.

Woodward, T. (2004). The effects of math anxiety on post-secondary developmental students as related to achievement, gender, and age. Inquiry, 9(1). http://www.vccaedu.org/inquiry/inquiry-spring2004/i-91-woodard.html

1 The Round 4 SGA (as well as SGAs for prior rounds) can be found at http://www.doleta.gov/taaccct/applicantinfo.cfm.

2 As described in A12 and in Part B, auxiliary data collection on TAA-eligible workers and veterans will focus on baseline data, and should sample be large enough, matches to National Directory of New Hires to examine employment-related outcomes. Follow-up surveys will be limited to the impact study sample.

3 The PUF will not include information collected from TAA-eligible workers or veterans.

Abt Associates Inc. Contents ▌pg. iv

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	Abt Single-Sided Body Template
Author	Katheleen Linton
File Modified	0000-00-00
File Created	2021-01-25

final TAACCCT R4 OMB Package Supporting Statement Part A 7 10 2015