OMB part A R5 DEI Revised - SMP

Paperwork Reduction Act Submission
Supporting Statement Part A for the

Office of Disability Employment Policy (ODEP)

October 19, 2016

ICR # 201609-1230-001

Evaluation of the Disability Employment Initiative Evaluation (DEI)

Table of Contents

Part A: Justification for the Study 4

A.1 Circumstance Necessitating the Collection of Information 4

A.1.1 Necessity of the Data Collected through Participant Tracking

System (PTS) and Comparison Group Survey 5

A.1.2 Necessity of the Data Collected through Site Visits 6

A.2 Purposes and Use of the Data 6

A.2.1 PTS and Survey of Comparison Group Individuals 7

A.2.2 WIOA, W-P and Qualitative Data 7

A.2.3 Site Visits 8

A.3 Use of Technology to Reduce Burden 9

A.3.1 WIOA and W-P Data to Reduce Burden on AJC Staff 9

A.3.2 Site Visit and Telephone Interview Data Collection Technology 10

A.4 Efforts to Identify and Avoid Duplication 10

A.5 Efforts to Minimize Burden on Small Entities 10

A.6 Consequences of Less Frequent Data Collection 11

A.7 Special Circumstances 12

A.8 Federal Register Comments and Efforts to Consult with Persons

Outside the Agency 12

A.9 Payments to Respondents 13

A.10 Assurances of Confidentiality 14

A.11 Questions of Sensitive Nature 15

A.12 Estimates of Respondent Burden 16

A.13 Estimates of Other Annual Costs to Respondents 21

A.14 Estimates of Annualized Government Costs 21

A.15 Changes in Hour Burden 22

Table of Contents (continued)

A.16 Time, Schedule, Publications, and Analysis Plan 23

A.16.1 Time, Schedule and Publication 23

A.16.2 Analysis Plan 23

A.16.2.1 Tabulation 24

A.16.2.2 Analyses of Quantitative Data 24

A.16.2.3 Analyses of Qualitative Data 24

A.16.2.4 Publication 26

A.17 Display of Expiration Date of OMB Approval 26

A.18 Exceptions to Certification Statement 26

References – Part A 27

List of Exhibits

Exhibit A.1: Annual Participant Focus Group Burden Rounds 5-6 16

Exhibit A.2: Annual Site Visit Participant Interviews: DRC and Job

Center Manger Burden Rounds 5-6 17

Exhibit A.3: Annual Site Visit Participant Interviews: Employment

Counselor Burden Rounds 5-6 18

Exhibit A.4: Annual Comparison Group Survey and PTS (Web Portal)

Data Collection Rounds 5-6 19

Exhibit A.5: DEI Round 5 and 6 Grantees and Expected Total and Career

Pathways Enrollment 21

Exhibit A.6: Data Collection and Publication Timeline 23

Part A. Justification for the Study

A.1 Circumstances Necessitating the Collection of Information

In 2010, ETA and ODEP joined efforts and released a Solicitation for Grant Applications (SGA) that made provisions for approximately $21,276,575 million in funds for cooperative agreements to go to the first round of state Workforce Investment Act- Workforce Innovation and Opportunities Act (WIA-WIOA) entities. This funding was a product of the Consolidated Appropriations Act of 2010,¹ which allocated one half of these funds to ETA and one half to ODEP for the two agencies to develop and implement an approach that would increase the effective and meaningful participation of job seekers with disabilities (JSWDs) in the workforce.² To accomplish this goal, ODEP and ETA designed the DEI to (a) refine and verify service delivery strategies for employing JSWDs that have proven effective or promising, and (b) foster the replication of those strategies across public workforce development systems. DEI is designed to help JSWDs find a way to the middle class through model service delivery.³ Nine states were awarded grants in 2010 to implement Round 1 DEI projects. In 2011 and 2012, ETA and ODEP awarded $21,166,560 and $20,654,352 to 14 new states, and in 2013 a total of $18,597,758 was awarded to five Round 1 states, which received Round 4 continuation grants, and three new DEI states. In total, ETA and ODEP have allocated $81,695,245 in funds to state LWIA entities to fund systems change efforts for the employment of JSWDs in local areas around the country. In 2014, ETA and ODEP provided $14,837,785 to six Round 5 grantees to implement career pathways and related workforce development programs. Round 6 grantees were awarded cooperative agreements in October 2015.

Twelve DEI grantees will be included in the Rounds 5-6 evaluation. Six Round 5 grantees were awarded in 2014 (California, Kansas, Illinois, Massachusetts, Minnesota and South Dakota) and six Round 6 grantees were awarded in 2016 (Washington, New York, Alaska, Georgia and Hawaii). For the matched comparison group study design, comparison group participants (who enrolled in services in surrounding Local Workforce Investment Areas (LWIAs) were selected based on the demographic characteristics of their participant populations using Workforce Innovation and Opportunities Act (WIOA) data.

The DEI Rounds 5-6 impact evaluation includes two distinct study research designs. The first study design is a matched comparison group design, with the treatment and comparison groups established at the LWIA level. Treatment LWIAs were selected by the grantees as having the capacity to implement the Rounds 5-6 DEI intervention. The comparison LWIAs were selected to align with the demographic and economic characteristics of the treatment LWIAs; discussions with each grantee about their comparison LWIAs were used to finalize the list of comparison LWIAs. In this quasi-experimental design (QED), a matched comparison group of individuals with disabilities provides the counterfactual so that there are no systematic differences between the two groups of participants that may influence program outcomes. The unit of analysis is the LWIA and inference is at the LWIA level. Because each treatment LWIA implements the intervention and because some of the DEI strategies are implemented at the LWIA level rather than at the individual level, it is necessary for the LWIA to be the unit of analysis for the evaluation.

The other QED will evaluate the impact of the specific career pathways component that is part of the Rounds 5-6 intervention. In this design, we will match similar participants within the Round 5 and 6 grantee treatment LWIAs, with the only primary difference being enrollment in the career pathways component versus enrollment in other programs and services, such as WIOA staff assisted core, intensive and/or training services

A key element in the implementation of the DEI initiative is the American Job Center’ (AJC’s) participation as Employment Networks (ENs) in the Social Security Administration’s (SSA’s) Ticket to Work (TTW) program designed to support beneficiaries with disabilities in their efforts to become economically self-sufficient.⁴ While the site visits to the Round 5 and 6 DEI grantees described above will include basic questions about TTW implementation that will be asked over the next several years, we will also conduct a single round of targeted site visits for a more in-depth study of TTW implementation. This will involve a sample of six workforce ENs paired with six non-workforce ENs serving the same communities. The purpose of this additional data collection is to assess how workforce ENs operate TTW and how they integrate ticket holders into workforce service provision. The study will also assess how workforce ENs compare with other types of ENs in their organizational characteristics, whom they serve, and the outcomes they achieve. These visits will allow us to gather foundational evidence and knowledge of lessons learned and promising practices that can be used to strengthen and improve TTW implementation throughout the workforce system.

.

The purpose of this submission is to create new data requests.⁵

New Data Requests:

1. Amend the fields collected in the DEI Participant Tracking System (PTS) by removing the existing fields and adding fields that identify DEI participants and their use of DEI service delivery strategies, a question on disability categories and questions on activities of daily living;

2. Add a survey of comparison group individuals to collect information on disability categories, activities of daily living, DEI service delivery strategies used and related outcomes measures.

1. DEI participant identifier;

2. Type of disability

3. Activities of daily living

4. DEI service delivery strategies used;

5. Receipt of training related certifications and/or diplomas;

3. Adjust the DEI Annual Site Visit and Focus Group Protocols to include questions on career pathways design, implementation and utilization.

4. Add a new set of Site Visit Protocols for in-depth study of the implementation of the TTW program, to be used for conducting interviews in LWIAs and in TTW programs outside the workforce system that serve the same communities.

A.1.1 Necessity of the Data Collected through PTS and Comparison Group Survey

For the evaluation of DEI Rounds 1-4, WIOA and Wagner-Peyser (W-P) data were used to obtain information on employment outcomes and customer characteristics. They were also used to identify participants with disabilities and their disability type. However, WIOA and W-P are limited in terms of the range of disability types that can be collected using the existing workforce data systems. WIOA field 202 asks each AJC customer if she/she has a disability (1 = Yes, 0 = No, 9 = Participant did not self-identify) and in Data Element 203, it asks if the customer has a Physical Impairment, Mental Impairment or Both Physical and Mental Impairments. Interviews conducted during the DEI Rounds 1-4 evaluation identified numerous disabilities that affect DEI participants’ return to work and employment related activities. However, they are not included in WIOA or W-P. In addition, WIOA and W-P do not include questions on activities of daily living (ADL). ADLs are basic tasks, such as dressing, transferring, walking, shopping, cleaning etc., that individuals do on a daily basis. Research shows that the likelihood of a successful job placement is mitigated by the activities (both work and leisure) that an individual can perform, the number of hours that an individual can work and the types of job tasks that are associated with a job (Loisel & Anema, 2013). These characteristics are essential for conducting a match between treatment and comparison group individuals and therefore, it is necessary to add to the Rounds 5-6 evaluation a survey of comparison group individuals, completed one time that captures information on a much broader list of disabilities and a short list of ADLs. This information will be collected by DRCs for treatment group individuals using the DEI web portal so that both treatment and comparison group individuals have an opportunity to provide this information. The Participant Tracking System (PTS) and telephone survey of comparison group individuals will allow us to determine the type of disabilities and level of functioning of each treatment and comparison group individual.

A.1.2 Necessity of the Data Collected through Site Visits

DEI Rounds 5-6 added the collection of information on Career Pathways and new DEI Administrative Data Elements. The site visit protocol was amended to include questions on the design, implementation and utilization of Career Pathways and related training programs. We also developed an additional site visit protocol with an in-depth focus on the implementation of the Ticket to Work program in a small sample of LWIAs and of TTW programs outside of the public workforce system, operating in the same communities. The DEI Administrative Data Elements, which are collected by DEI grantees using the DEI Web Portal or state WIOA data systems.

A.2 Purposes and Use of the Data

Information will be collected from Local Work Force Investment Area (LWIA) staff and DEI participants by a team of trained researchers using a structured interview protocol. The data will be analyzed to provide a thorough understanding of the implementation, utilization and effectiveness of policies, practices and strategies that comprise the DEI. The data will be analyzed to describe the DEI participant recruitment and enrollment process, the services DEI participants receive, their outcomes and the overall impact of the program. In addition, qualitative data will be collected to identify systems change in treatment LWIAs. Workforce Innovation and Opportunities Act (WIOA) and Wagner-Peyser data will be used to measure participant outcomes, including the DEI employment rate, employment retention rate and average wages compared to a matched comparison group. The data will also be used to develop a thorough understanding of the services and supports needed by individuals with disabilities seeking employment and to identify the factors that contribute to employment, employment retention and receipt of wages. The results of the DEI evaluation will also be used to inform future research activities on the influence of disability type and severity, socioeconomic and economic factors, and job readiness among individuals with disabilities. They will also be used to develop and refine existing DEI service delivery strategies and technical assistance and training curriculums that increase the capacity of LWIAs to provide services and supports to individuals with disabilities seeking employment.

A.2.1 PTS and Survey of Comparison Group Individuals

The PTS will provide information that will ensure that each program participant’s use of DEI services and related outcomes can be accurately documented. The PTS is a secure web based system that collects participant level information. It will collect information that will track program enrollment, use of DEI service delivery strategies, type of disability, activities of daily living and receipt of training related certifications and/or diplomas.

A.2.2 WIOA, W-P and Qualitative Data

The DEI Evaluation also relies on WIOA and Wagner-Peyser (W-P) data for the measurement of participant outcomes. The purpose of this information is to determine the efficacy of the DEI based on customer outcomes, describe customer characteristics and patterns of service utilization. The DEI grant period for Rounds 5-6 is 42 months with 36 months of program operation. For these Rounds, the evaluation will receive uploads from each state on a quarterly basis for the fourteen quarters during which the grantees are operating, plus four follow-up quarters. In addition, four additional quarters will be submitted by each DEI grantee to allow for the collection of outcomes that occur after the grant period.

Combined, these data will allow DOL to assess the short and long-term DEI impact on:

• Number and percentage of DEI customers (WIOA new administrative data elements or PTS);

• Number and percentage of individuals with disabilities served at AJCs in core, staff-assisted core, WIA-WIOA intensive and WIA-WIOA training;

• Number and percentage of DEI customers in career pathways programs (WIOA new administrative data elements or PTS);

• Characteristics of customers with disabilities who are served at each participating LWIA (WIOA);

• Through a separately administered survey, information on type of disability disclosed by customers, with delineation on categories of disability and ADL, degree and training completion rates (Degree/Diploma/Certification Completion) (SURVEY);

• Through interviews and site visits, program implementation, information on the availability and nature of services and support for AJ C customers with disabilities, and systems change (site visits);

• Employment rates and employment retention rates as of the third quarter after exit quarter (WIOA);⁶ and

• Earnings after exit for four quarters (WIOA).

These data are necessary for DOL officials to make informed decisions about program effectiveness, replication, continued funding, and new initiatives. Without these data, it will be impossible to determine whether the DEI met its goals and warrants further replication and/or revisions to its design. The risks that would result from such a knowledge-gap include future misuse of resources (funding of an ineffective initiative), missed opportunities (failure to fund an effective initiative), and harm to individuals with disabilities who seek employment services (in the event of misused resources or missed opportunities).

A.2.3 Site Visits

In years 2015-2017 and 2016-2018, the DEI Evaluation Team will conduct one site visit to each of the six Round 5 and Round 6 grantees. In each of the treatment LWIAs participating in the DEI Evaluation, on-site or if necessary, telephone interviews will be conducted with the DEI state lead, DRC, WIB director, AJC managers, AJC staff members, and agency partners and selected employers. A site visit to one comparison LWIA and AJC in close proximity to each treatment LWIA will also be conducted. In treatment and comparison LWIAs, approximately eight to ten AJC DEI participants will be asked to participate in a customer focus group. The site visits will be tailored to fit each grantee as well as each DEI or comparison site. Variations across sites are likely to mean that the number and types of individuals to be interviewed will vary. The general domains to be investigated through collecting and analyzing site visit data include: the current status at baseline and change in grantees’ workforce development system at follow-up; grantee customer characteristics; implementation of the five grant requirements; implementation of grantee selected DEI strategies; program implementation challenges; and system change. The study team developed respondent-specific questions to be asked in the interviews and focus groups. The site visits for other rounds of the DEI will follow the same protocol.

The decision to conduct site visits in the first, second year and third years of DEI Rounds 5-6, allows the DEI Evaluation Team to collect extensive baseline (year 1), mid-term (year 2) and follow-up (year 3) information including observations of WIA-WIOA intensive and training programs, career pathways programs, state-assisted core activities and to conduct customer focus groups. Follow-up site visits in the third year will allow for the collection of information on systems change.

The site visits are three-to-five days in length, approximately one day for the state and one day each for the participating LWIAs and at least one comparison LWIA, visiting at least one AJC in each LWIA in addition to meeting with partners, employers and conducting customer focus groups. The evaluation team will ensure that at least two members of each site visit team are senior staff, and at least two are experienced focus group facilitators, to maximize the value of the visit and minimize the burden on the respondents at each site.

In addition to the site visits to Round 5 and 6 DEI grantees and two or more LWIAs for each, additional site visits will be conducted to six LWIAs with strong Ticket to Work programs along with TTW programs outside the public workforce system that serve the same six communities. This in-depth study of the TTW program will provide information on how implementation of TTW through the workforce system compares with implementation through other kinds of community agencies, and allow the workforce system to learn lessons and promising practices from effective programs outside the workforce system.

A.3 Use of Technology to Reduce Burden

Every effort has been made to structure the DEI Evaluation to minimize the burden on state, LWIA and AJC staff and DEI customers. The DEI Evaluation will collect participant-level information in three ways: collection of WIOA and W-P data from state agencies, entry into a PTS that identifies DEI participants and collects information on disability categories, activities of daily living and related outcomes. In addition, a Survey of Comparison Site Individuals will be used to collect information from comparison group individuals on type of disability, activities of daily living, and related outcomes that would be difficult to gather with precision from the WIOA system, such as degree or certificate completion from a postsecondary institution. To collect information on comparison group individuals, we will use a computer-assisted telephone interviewing (CATI) system, instead of a more costly and time-consuming field survey. The CATI approach is expected to be less than half the cost of the field approach, per completed survey.

As potential candidates enter AJCs in both the treatment and comparison LWIAs, we will seek to collect informed consent so that AJC entrants would know why they could potentially be contacted for the survey. At this point, we would collect contact information and verify that the information is correct from the AJCs.

To reduce burden on survey respondents, survey experts will work to help screen survey candidates (e.g., screening for mental capacity to appropriately complete the survey), send an advanced letter to potential candidates (by email where possible), and train phone interviewers in advance of survey implementation. As the DEI program is a rolling enrollment of services and does not follow a strict cycle or calendar, we plan to cluster the fielding of the telephone survey by quarter (four times a year). The survey itself will be relatively short, at approximately 9 minutes in length, and completely entirely by phone. Using the contact information first collected at entry into the AJC, in addition to any updates in contact information from the AJC itself, researchers will use the information to contact treatment and comparison group respondents to collect information on disability type, activities of daily living, and academic outcomes. The evaluation will use the latest survey technology to facilitate the collection of the survey data in standardized and accurate ways, and will ensure the privacy and protection of the information collected.

We anticipate that the first implementation of the survey would begin 18 months after the start of the program, which would occur approximately fall 2016 with the final deployment of the survey in 2018. Round 6 will begin in 2017 and will continue through 2019.

A.3.1 WIOA and W-P Data to Reduce Burden on AJC Staff

Use of WIOA and W-P data will significantly reduce the burden on AJC data collection activities. WIOA and W-P data collection is already integrated into each AJC. Therefore, there is no additional burden placed on the staff.

A.3.2 Site Visit and Telephone Interview Data Collection Technology

The site visits are designed to collect information on grantee start-up, implementation and systems change. This data collection effort will rely on 45 to 60-minute interviews with DEI State Leads, DRCs, LWIB Directors, AJC managers and staff, agency partners and employers and focus groups with customers. To minimize burden on respondents during this process, the evaluators will utilize the following technology:

• An Internet based calendar will remind all respondents of the date and time of their interview. For site visit interview respondents, the calendar will provide location and driving directions, if necessary. The calendar will provide automated reminders to each respondent 24 hours and 30 minutes prior to their scheduled interview. This technology will provide updates if dates and times of interviews are changed.

• The DEI Evaluation Webpage (http://socialdynamicsllc.com/deigranteessecure.html) provides information on the DEI Evaluation as well as contact information for each Evaluation Liaison, a document archive that includes documents related to the DEI Evaluation and a Frequently Asked Questions archive that respondents can access to learn about the DEI Evaluation and minimize time on site discussing the evaluation methodology prior to each interview.

• Privacy and informed consent forms will be available online for respondents that have a certified digital signature.

A.4 Efforts to Identify and Avoid Duplication

The evaluation will provide a rigorous assessment of the impact of the DEI, using a quasi-experimental matched comparison group design. The need for this research, and the fact that it is not duplicating existing information, has been confirmed by background research conducted by members of the DEI Evaluation Team, all of whom have extensive experience evaluating DOL disability employment initiatives (e.g., Livermore & Goodman, 2009, Klayman & Bleimann, 2014). Recent literature reviews and attendance at national meetings by research partners also provided confirmation that the information obtained through the DEI Evaluation will not be redundant.

Additional safeguards have been built into the evaluation plan with the express purpose of avoiding duplication of effort. Researchers will access and use administrative data that is regularly reported by states, including WIOA data. DOL consulted with state DEI representatives and information technology (IT) specialists to learn about any state-specific collection of customer data that is currently underway and have made efforts to integrate DEI Evaluation data collection activities with current data collection efforts at the grantee level.

A.5 Efforts to Minimize Burden on Small Entities

The “small entities” involved in the DEI Evaluation include employers and non-profit partner agencies that will participate in the site visit interviews. These respondents will participate in interviews that last 45 minutes rather than the 60 minute interviews being conducted with government agency staff and AJCs. In addition, site visitors will travel to each respondent’s location to minimize the time dedicated to the interview process for small entity respondents. As shown in the site visit data collection matrix, these respondents have fewer questions assigned to their interviews and therefore a shorter period of time will be needed to conduct these interviews. Based on internal pilot tests, these interviews will be approximately 45 minutes in duration.

A.6 Consequences of Less Frequent Data Collection

Without this study, DOL will not have information critically needed for understanding the factors that contribute to gainful employment at a living wage or higher, among individuals with disabilities, including Social Security Disability Income and Social Security Income beneficiaries. This understanding is needed to inform training and technical assistance, as well as policy and practice in the areas of disability employment, reemployment, utilization of Social Security Administration work incentives and employment supports. Furthermore, without the DEI evaluation and the collection of workforce and related data, the public workforce investment system will not have up to date information on the strategies that lead to employment and wages. Although similar studies have provided information on similar workforce strategies, such as Wachen’s (2010) “How I-BEST Works: Findings from a Field Study of Washington State’s Integrated Basic Education and Skills Training Program,” few cover the depth and geography of the DEI. Lastly, if the TTW portion/data collection of this study were to be omitted, we would not be able to address research questions pertaining to how workforce ENs  compare to non-workforce ENs in serving SSI/SSDI beneficiaries with disabilities under SSA’s TTW Program.

Some of the technical challenges that may reduce the number of participating states and/or the frequency of data collection create limitations related to the reliability and validity of study findings, as many of the technical obstacles to reducing burden are related to the study design. For example, a relatively large sample size of LWIAs is needed to conduct the planned analyses and to identify significant estimates between treatment and comparison groups. Quarterly collection of WIOA and Wagner-Peyser data and annual site visits are needed to measure outcomes and program impact, and describe behaviors and attitudes prospectively and with a short enough recall period to enable accurate reporting. The burden will on both staff and participants will be limited to one 45 to 60 minute interview per year for approximately 8-10 staff and one 60 minute focus group per year for 8-10 participants.

Neglecting to collect the data specified in this document would prevent DOL from making fully-informed policy decisions regarding the delivery of employment services to people with disabilities. Collecting data on the status of unemployed or underemployed individuals with disabilities who are at AJCs is necessary to determine whether the DEI had the desired impact. The DEI Data System contains information that is essential to the DEI Evaluation. We hypothesize that the DEI will affect the number and characteristics of customers with disabilities seeking services at the treatment LWIAs (e.g. disability type).

The collection of interview and focus-group data during annual site visits will ensure that the DEI Evaluation Team collects detailed information on program implementation/start-up and program design. The site visit data will enable the evaluation team to answer the “how” and “why” questions that in turn support meaningful system change and/or successful replication of the DEI. Specifically, by collecting data to document the challenges, solutions, and best practices of DEI grantees, additional grantees can implement the DEI approach more efficiently and effectively. Finally, the DEI intervention is comprehensive, requiring collaboration among a diverse group of stakeholders. Collection of data through interviews and focus groups ensures that DEI implementation and impact is understood from multiple perspectives, not the least of which are those of the program staff and the individuals whom the intervention is meant to assist.

Based on our prior experience conducting research on disability employment programs, conducting site visits on a one-year interval allows enough time for complex systems changes to unfold, but not so much time that retrospective data becomes unreliable.

The present data collection schedule includes quarterly collection of customer-level data WIOA data and annual collection of site-visit data. Two criteria were considered in determining this frequency: the methodological rigor of the evaluation and burden on the DEI grantees. The quarterly data collection of customer-level data will support a rigorous analysis of program impact by ensuring that enough data points are available to detect change in customer level outcomes. It will also allow the DEI Evaluation Team to monitor the submission of data to ensure that the incoming data is of appropriate quality and includes all required data elements and both baseline and follow-up intervals. In the case of missing data, analysts will be better equipped to fill in gaps or reconfigure existing quarterly data than if they were working with semi-annual or annual data. A quarterly schedule is also intended to streamline reporting requirements for states. This frequency is synchronized with the states’ existing federal reporting schedule.

A.7 Special Circumstances

All data collection activities comply fully with regulations. There are not special circumstances associates with these data collection activities.

A.8 Federal Register Comments and Efforts to Consult with Persons Outside the Agency

The 60-day notice to solicit public comments was published in the Federal Register on January 12, 2016 (81 FR 1446, 2016-00460). A 60-day correction notice was published in the Federal Register (81 FR 36350, 2016-1333) on May 26, 2016. No public comments were submitted during the notice period.

In planning this evaluation, DOL consulted with disability employment experts and grantees who will be data providers. DEI Grantee Discussions were conducted with each Round. Individuals at the state level who had been assigned the role of DEI State Lead or DEI Primary Contact participated in these calls, as well as AJC managers and staff from participating LWIAs, since their involvement and understanding of the data collection activities is integral to the successful implementation of the DEI evaluation. In addition, grantees discussed their existing administrative data systems and customer registration procedures. During these conference calls, administrative, program and IT staff were educated on the use of a QED and what it would entail. This allowed the grantees to explain what data collection procedures would be easiest for their AJC staff. Leaders from each grantee explained potential data sharing procedures, including the feasibility of sharing data and related challenges. These meetings ensured that the DEI Evaluation minimizes the data-collection burden on all stakeholders; asking only for those data not available elsewhere (i.e., avoiding duplication of effort), and ensuring the simplest delivery method possible. Consultations with data-providers have suggested that quarterly collection of customer data will not be burdensome, as it matches their existing reporting schedules for DOL. All agree that the data collection plans are clear and concise. No major concerns arose during these calls.

In addition, the DEI Evaluation Team consulted with several experts on the QED design. This information was particularly important to the identification of the data elements necessary to conduct an impact study of the DEI.

Persons Consulted
Jason Albert, MA DEI State Lead	[email protected]	P: 617-626-5190 F: 617-727-8671
Diane Hurley, MA Manager	[email protected]	P: 617-626-5701 F: 617 626 6661
Bill Molseed, SD Administrator	[email protected]	P: 605-773-5017 F: 605-773-6184
Bill McEntaffer, SD Program Manager	[email protected]	P: 605-773-4548
Kay Tracy, MN Youth Unit	[email protected]	P: 651-259-7555 F: 651-215-3842
Susan Weidenbach, MN	[email protected]	P: 785-296-7842
David Mayer, CA Project Advisor	[email protected]	P: 916-654-7422 F: 916-654-9821
LaJuana Thompson, CA Staff Service Manager	[email protected]	P: 916-657-5280 F: 916-654-9821
Lisa Jones, Policy Manager	[email protected]	P: 217-558-2443 F: 217-557-5506
Lynn Douma, MN	[email protected]	P: 651-259-7536

A.9 Payments to Respondents

Payments will be made to focus group participants and survey respondents. The evaluation includes annual site visits to DEI implementers at both the state and local levels. Site visits in spring 2016, spring 2017 and spring 2018 for Round 5 and spring 2017-2019 for Round 6, will include one focus group of AJC customers. The evaluators will work with AJC staff to identify between 8-10 customers to participate in each focus group. The focus group will receive $25.00 at the conclusion of the 60-minute session. The money provided is intended as an incentive, to ensure adequate participation among customers. Survey respondents will receive $15 for completing the Survey of Comparison Site Individuals. We anticipate approximately 2,460 customers enrolled in Rounds 5-6 with approximately the same amount in the comparison sites. The total cost to the evaluation for Round 5 will be $16,860 for the treatment site customers and $16,860 for the comparison group customers for a total of $33,720. Rounds 5-6 combined will total approximately $67,440. These payments were approved by DOL in 2012.

Ensuring that focus groups include a minimum of 8-10 participants is an important part of the methodology. Groups smaller than this not only lack representativeness, but also are at greater risk of stifled or stagnant inter-personal dynamics (Kreuger, 1994). There is substantial research indicating that incentives increase response rates in social research (Ryu, Cooper, & Marans, 2006), including focus group research (Singer, Van Hoewyk, Gebler, Raghunathan, & McGonagle, 1999; Willimack, Schuman, Pennell, & Lepkowski, 1995). A snow-ball sampling procedure will be used to create the focus group samples. The DEI Evaluation Team will prepare flyers containing information on the focus group and the incentive of $25.00, as well as refreshments. Evaluation Liaisons will work closely with AJC Managers and DRCs to ensure that focus group flyers are posted throughout the AJC, posted online, distributed through RSS and listservs and emailed to customers. $25.00 180) participants for a total cost of

In addition to the research on incentives, all research partners in the DEI Evaluation have extensive experience conducting focus groups, and can attest to the importance of providing a participant incentive. Focus groups using a cash incentive of similar value with a similar population have produced useful data in the following studies: Independent Studies of Start-Up USA Self-Employment Program and the WIRED Evaluation. Finally, researchers have found that providing cash at the time of the focus group is greatly appreciated by participants, and presenting the incentive at the conclusion of the group encourages participants to stay for the 90-minute duration, rather than leave part way through. According to the National Science Foundation, monetary incentives improve study participation and offset the costs of follow-up and recruitment of non-respondents (Zhang, 2010).

A.10 Assurances of Privacy

Each staff, focus group and comparison group survey respondent will receive a brief statement of confidentiality and asked to sign a consent form. We will seek to collect consent so that all respondents know why they are being asked to participate in the DEI evaluation and can make an informed decision on whether or not they wish to participate.

To respect and uphold the privacy of DEI participant information, all contractors involved in the evaluation will operate within the guidelines established by the federal Privacy Act of 1974 to protect respondents’ privacy and the privacy of all data collected. The Privacy Act of 1974 states that “micro-data files prepared for purposes of research and analysis are purged of personal identifiers and are subject to procedural safeguards to assure anonymity” (5 U.S.C. § 552a, Public Law No. 93-579). The evaluation contractors hired for this project have extensive experience with handling sensitive data. These contractors will have routine procedures in place to ensure the privacy of computerized and paper records, including the use of passwords and encrypted identifiers to prevent direct or indirect disclosures of information. Furthermore, the information management systems of these contractors will fully comply with the Government’s system requirements. The system of records includes the WIOA, the W-P administrative database and the DEI Data System. The DEI Data System was developed specifically for the DEI Evaluation.

Two safeguards will be implemented to ensure privacy for this study: 1) data transferred electronically will be password encrypted prior to transmission and transmitted on secured access lines and 2) staff access to data storage and files will be limited to authorized personnel who have passwords.

Information or copies of information may be released only to authorized individuals and the organization must ensure that unauthorized individuals cannot gain access to, or alter participant information. Participant information must be released in accordance with federal or state laws, court orders, or subpoenas.

Interview notes, and, to a lesser extent, focus group notes, may still contain personally-identifiable information. Researchers will work to report data in an aggregate manner. If there are data to be shared that are intrinsically identifiable, researchers will check first with the respondent(s) for permission to share these results. Participation in all interview and focus group activities is voluntary. Although DEI state leaders have committed to assisting with the evaluation, and therefore may feel pressure to participate, researchers will make it clear to them that they may decline answering any question, and may stop the interview at any time.

Focus group participants will be told the following: The information you provide will be kept private, and will not be disclosed to anyone, but the researchers conducting this investigation, except as otherwise required by law.

Certain disclosure limitations will be used to maintain the privacy data and findings. The data file for the DEI Evaluation will be comprised of WIOA data from each DEI state. Although individual customer records will be included, no geographic identifier other than the state in which services were provided, will be included. In addition, findings based on data from WIOA will be presented only in the aggregate (program-wide) to represent the entire DEI, rather than individual states or LWIAs. In addition, table cells in reports with standard errors larger than 50 percent and table cells with values less than 5.0 will not be disclosed. Random rounding will be used to suppress complementary cells if their value can be calculated by subtraction from the marginal total.

A.11 Questions of Sensitive Nature

Site visits to selected Workforce Development Areas (WDAs), as well as TTW ENs, will not require responses to questions about sensitive issues. The study will ask the respondents at each site about: customer characteristics that impede and/or facilitate employment success; implementation of DEI; DEI implementation differences when serving youth; challenges that need to be addressed to improve employment outcomes of individuals with disabilities; DEI effects on system change and customer outcomes; and best practices for facilitating changes in outcomes.

A survey will collect information about the nature of a customer’s disability and activities of daily living. This potentially sensitive information is not available from other sources and is considered critical to the evaluation of the employment outcomes that will be analyzed in the evaluation. Disability and health variables have been shown to be significant determinants of employment outcomes. Without the collection of this information, the evaluation could not move forward. This information will provide a greater descriptive understanding of the types of disabilities that participants have when they enter an AJC and ensure a more accurate match to a comparison site individual, which will improve the accuracy of the QED.

A.12 Estimates of Respondent Burden

Exhibit A.1 Estimated Annualized Respondent Burden Hours and Costs for Participant (Grantees) Focus Group Burden Rounds 5-6

Participant (Grantees) Focus Group Burden Rounds 5-6
States	Number of Respondents	Number of Responses per Respondent	Total Number of Responses	Average Burden per Respondent (in Hours)	Total Burden Hours	Hourly Wage Rate*	Total Burden Costs
California	20	1	20	1	20	$22.30	$446
Kansas	20	1	20	1	20	$22.30	$446
Illinois	20	1	20	1	20	$22.30	$446
Massachusetts	20	1	20	1	20	$22.30	$446
Minnesota	20	1	20	1	20	$22.30	$446
South Dakota	20	1	20	1	20	$22.30	$446
Alaska	20	1	20	1	20	$22.30	$446
Georgia	20	1	20	1	20	$22.30	$446
Hawaii	20	1	20	1	20	$22.30	$446
Iowa	20	1	20	1	20	$22.30	$446
New York	20	1	20	1	20	$22.30	$446
Washington	20	1	20	1	20	$22.30	$446
Total	240	--	240	--	240	--	$5,352
All Occupations	$22.30*
*Source- http://www.bls.gov/oes/current/oes_nat.htm

Exhibit A.2: Estimated Annualized Respondent Burden Hours and Cost for Annual Site Visit Participant Interviews: DRC and Job Center Manager

Burden Rounds 5-6 and TTW

Staff Interview Burden: (DRCs, One-Stop Managers, TTW)
	Number of Respondents	Number of Responses per Respondent	Total Number of Responses	Average Burden per Respondent (in Hours)	Total Burden Hours	Hourly Wage Rate*	Total Burden Costs
California	7	1	7	1	7	$55.30	$387.00
Kansas	7	1	7	1	7	$55.30	$387.00
Illinois	6	1	6	1	6	$55.30	$332.00
Massachusetts	7	1	7	1	7	$55.30	$387.00
Minnesota	8	1	8	1	8	$55.30	$442.00
South Dakota	4	1	4	1	4	$55.30	$221.00
Alaska	7	1	7	1	7	$55.30	$387.00
Georgia	8	1	8	1	8	$55.30	$442.00
Hawaii	5	1	5	1	5	$55.30	$277.00
Iowa	9	1	9	1	9	$55.30	$498.00
New York	7	1	7	1	7	$55.30	$387.00
Washington	6	1	6	1	6	$55.30	$332.00
TTW	36	1	36	1	36	$55.30	$1990.00
Total	117.0	--	117.0	--	117.0	--	$6,469.00
Management Occupations	$55.30*
*Source -http://www.bls.gov/oes/current/oes110000.htm

Exhibit A.3: Estimated Annualized Respondent Burden Hours and Cost for Site Visit Participant Interviews: Employment Counselor Burden Rounds 5-6

	Other Staff Interview Burden: Employment Counselors
		Number of Respondents	Number of Responses per Respondent	Total Number of Responses	Average Burden per Respondent (in Hours)	Total Burden Hours	Hourly Wage Rate*	Total Burden Costs
	California	9	1	9	1	9	$33.80	$304.00
	Kansas	9	1	9	1	9	$33.80	$304.00
	Illinois	7	1	7	1	7	$33.80	$237.00
	Massachusetts	6	1	6	1	6	$33.80	$203.00
	Minnesota	8	1	8	1	8	$33.80	$270.00
	South Dakota	4	1	4	1	4	$33.80	$135.00
	Alaska	4	1	4	1	4	$33.80	$135.00
	Georgia	6	1	6	1	6	$33.80	$203.00
	Hawaii	4	1	4	1	4	$33.80	$13500
	Iowa	7	1	7	1	7	$33.80	$237.00
	New York	9	1	9	1	9	$33.80	$304.00
	Washington	5	1	5	1	5	$33.80	$169.00
	TTW	28	1	28	1	28	$33.80	$946.00
	Total	106.00	--	106.00	--	106	--	$3,582.00
	Social and Community Service	$33.80*
	*Source- http://www.bls.gov/oes/current/oes119151.htm

Exhibit A.4: Annual Comparison Group Survey and PTS (Web Portal)

Data Collection Rounds 5-6

States			Number of Respondents for DEI Rounds 5-6 Grantees (Treatment)			Number of Respondents for the IWD Contact Points					Number of Responses per Respondent			Average Burden per Respondent (in Hours)			Total Burden Hours		Hourly Wage Rate*		Total Burden Costs
						(Comparison)		Total Contacts
California			375			248		623			1			5/60			52		$22.30		$1,160
Kansas			140			92		232			1			5/60			19		$22.30		$424
Illinois			320			211		531			1			5/60			44		$22.30		$981
Massachusetts			165			109		274			1			5/60			23		$22.30		$513
Minnesota			155			102		257			1			5/60			21		$22.30		$468.
South Dakota			75			50		125			1			5/60			10		$22.30		$223.00
Alaska			600			396		996			1			5/60			83		$22.30		$1,851
Georgia			215			142		357			1			5/60			30		$22.30		$669
Hawaii			170			112		282			1			5/60			24		$22.30		$535
Iowa			300			198		498			1			5/60			42		$22.30		$937
New York			215			142		357			1			5/60			30		$22.30		$669
Washington			436			288		724			1			5/60			60		$22.30		$1,338
Total			3,166			2,090		5,256			--			--			438		--		$9,768
All Occupations			$22.30*

*Source- http://www.bls.gov/oes/current/oes_nat.htm

Summary Table of the Total Estimated Annualized Burden Hours and Costs by Category

	Total Number of Respondents	Total Burden Hours	Total Number of Responses	Total Burden Costs
Participant Focus Groups	240	240	240	$5,352.00
Site Visit Staff Interviews	117	117	117	$6,469.00
Site Visit Other Staff Interviews	106	106	106	$3,582.00
PTS and Comparison Survey	5,256	438	5,256	$9,768.00
Total	5,719	901	5,719	$25,171.00

The Paperwork Reduction Act (PRA) requires the agency to account for the amount of burden that it is placing on the public when seeking information on behalf of the Federal government. This burden is measured in terms of hours and includes the following activities:

• R6 Site Visits/Telephone Interviews/Focus Groups Labor

• R5 Site Visits/Telephone Interviews/Focus Groups Labor

• Sustainability Interviews with Rounds 5-6

• Qualitative Analysis Rounds 5-6

• Site Visit Travel Expenses

• DEI Participant Tracking System Maintenance

• Training and Technical Assistance 5-6 DEI Sites

• Programming

• Monitoring Response/Completion Feedback to Sites

• T&TA Expenses

• Merging Datasets and Processing Data, Uploading/Downloading, Flattening

• Maintenance of FTP Site

• Outcome Analysis

• DEI Synthesis Reports

• Survey of Comparison Site Individuals

• Follow-Up to Non-Response

In September 2015 and 2016, a total of twelve states received grants to implement Round 5 and Round 6 of the DEI (See Exhibit A.5). Exhibit A.5 provides the expected number of enrollees for WIOA and Career Pathways services for these states awarded DEI grants.

Exhibit A.5: DEI Rounds 5 and 6 Grantees and Expected Total and Career Pathways Enrollment

California Adult	Kansas Adult	Illinois Youth	Massachusetts Adult	Minnesota Adult	South Dakota Adult	TOTAL Expected Enrollment	Expected CP Enrollment
Expected Enrollment Total/CP	Expected Enrollment Total/CP	Expected Enrollment Total/CP	Expected Enrollment Total/CP	Expected Enrollment Total/CP	Expected Enrollment Total/CP
375/245 CP	140/120 CP	320/195 CP	165/140 CP	155/120 CP	75 CP	1,230	895
LWIAs = 3	LWIAs = 3	LWIAs = 2	LWIAs = 3	LWIAs = 3	LWIA = 17

Alaska	Georgia	Hawaii	Iowa	New York	Washington	TOTAL Expected Enrollment	Expected CP Enrollment
Expected Enrollment Total/CP	Expected Enrollment Total/CP	Expected Enrollment Total/CP	Expected Enrollment Total/CP	Expected Enrollment Total/CP	Expected Enrollment Total/CP
600/300 CP	215/140 CP	170/120 CP	Estimate = 300/140 Not enough information	215/185 CP	Estimate = 436/100 CP Not Enough Information	1,936*	985*
LWIAs = 1	LWIAs = 3	LWIAs = 1	LWIAs = 5	LWIAs = 2	LWIA = 2

* Theses totals were estimated based on limited information provided by the grantee.

A.13 Estimates of Other Annual Costs to Respondents

There are no costs to the respondents other than their time.

A.14 Estimates of Annualized Government Costs

Annualized Costs to the Federal Government: Site Visits, DEI Data System, Data Processing, Survey
								2017-2018	2018-2019	2019-2020	2020-2021
	Site Visits/Telephone Interviews/Focus Groups
								12 states	12 states	6 states	Final Rounds 5-6 Analysis
								R 5, 6	R 5, 6	R 6
	R6 Site Visits/Telephone Interviews/Focus Groups Labor							$80,394.00	$70,982.00	$60,949.00
	R5 Site Visits/Telephone Interviews/Focus Groups Labor							$110,293.00	$110,293.00
	Sustainability Interviews with Round 5-6							$25,934.00			$25,934.00
	Qualitative Analysis Rounds 5-6							$132,093.00	$132,093.00	$90,394.00	$90,394.00
	Site Visit Travel Expenses							$70,293.00	$80,293.00	$80,293.00
	TTW Site Visits							$72,650.00
	Subtotal							$491,657.00	$393,661.00	$231,636.00	$116,328.00
	DEI Data System							2017-2018	2018-2019	2019-2020*	2020-2021
								12 states	12 states	6 states	Final Rounds 5-6 Analysis
								R 5, 6	R 5, 6	R 6
	DEI Participant Tracking System Maintenance							$65,650.00	$65,650.00	$45,948.00	$30,485.00
	Training and Technical Assistance 5-6 DEI Sites							$70,394.00	$70,394.00	$70,394.00	$30,485.00
	Programming							$60,989.00	$60,989.00	$60,989.00	$60,989.00
	Monitoring Response/Completion Feedback to Sites							$70,594.00	$70,594.00	$50,394.00	$50,394.00
	T&TA Expenses							$70,493.00	$70,493.00	$50,493.00
	Subtotal							$338,120.00	$338,120.00	$278,218.00	$172,353.00
	Data Processing							2017-2018	2018-2019	2019-2020	2020-2021
								Rounds 5-6	Rounds 5-6	Rounds 5-6	Final Rounds 5-6 Analysis
	Merging Datasets and Processing Data, Uploading/Downloading, Flattening							$45,069.00	$45,069.00	$45,069.00	$60,393.00
	Maintenance of FTP Site							$18,283.00	$18,283.00	$18,283.00	$18,283.00
	Outcome Analysis							$90,283.00	$90,283.00	$90,283.00	$90,283.00
	DEI Synthesis Reports							$102,394.00	$102,394.00	$102,394.00	$102,394.00
	Subtotal							$153,635.00	$153,635.00	$153,635.00	$271,353.00
	Survey of Types of Disability, ADL and Selected Outcomes Labor							2017-2018	2018-2019	2019-2020	2020-2021
											Final Rounds 5-6 Analysis
	Survey of Types of Disability, ADL and Selected Outcomes Labor							$330,494.00	$330,494.00	$330,494.00	$78,765.00
	Follow-Up to Non-Response							$70,293.00	$70,293.00	$70,293.00	$0.00
	Subtotal							$400,787.00	$400,787.00	$400,787.00	$78,765.00
	TOTAL							$1,384,199.00	$1,286,203.00	$1,064,276.00	$638,799.00
	5-Year Total		$4,373,477.00
	*If additional grantees are added, this estimate of burden will be revised.

Annualized Cost to the Federal Government: $874,695

A.15 Changes in Hour Burden

This is a new data collection.

A.16 Time, Schedule, Publication, and Analysis Plan

A.16.1 Time, Schedule and Publication

The DEI Evaluation is designed to collect information on program start-up, challenges to implementation, program impact and customer outcomes. Site visit data collection will occur in fall and spring 2016-2017 and 2017-2018 for Round 5 and Round 6. Uploading of WIOA and W-P data from DEI grantees will occur on a quarterly basis. Publications based on the analysis of DEI data are scheduled for 2017, 2018, 2019 and 2020.

Exhibit A.6: Data Collection and Publication Timeline

					Qtr 1 2017		Qtr 2 2017	Qtr 3 2017	Qtr 4 2017	Qtr 1 2018
Comprehensive Site Visits Year 1
Site Visit Synthesis Report Year 1 (Publication)
Comprehensive Site Visits Year 2
Site Visit Synthesis Report Year 2 (Publication)
Comprehensive Site Visits Year 3
Final Site Visit Synthesis Report Year 3 (Publication)
DEI Data System Quarterly Uploads
					Qtr 2 2018		Qtr 3 2018	Qtr 4 2018	Qtr 1 2019	Qtr 1 2019
Comprehensive Site Visits Year 1
Site Visit Synthesis Report Year 1 (Publication)
Comprehensive Site Visits Year 2
Site Visit Synthesis Report Year 2 (Publication)
Comprehensive Site Visits Year 3
Final Site Visit Synthesis Report Year 3 (Publication)
DEI Data System Quarterly Uploads

A.16.2 Analysis Plan

The DOL requires information on the impact of the DEI and customer outcomes, including changes in the employment rate, employment retention rate and employment preparation of customers with disabilities. DOL plans to use the data to provide technical assistance and support to the nation’s local workforce development systems and assist DEI grantees in improving their system change efforts and services to customers with disabilities. The analytical approaches outlined below are designed to describe the DEI, measure its impact, and document customer outcomes.

A.16.2.1 Tabulation

The following sections provide analysis of quantitative data from the DEI data system, qualitative site visit data and the publication of reports on the DEI for DOL.

A.16.2.2 Analyses of Quantitative Data

The data analyses will begin with developing basic descriptive statistics in tabular and graphical form to provide information about the characteristics and outcomes of the DEI treatment and comparison groups, using a pooled (overall) sample, as well as by state. For example, descriptive analyses will focus on customer and site characteristics, providing a profile of customers in each of the six states. Outcomes of interest will be measured at the individual level and include measures of service utilization and academic and employment outcomes. One utilization measure is the proportion of individuals registering for services in an AJC who report having a disability. Employment outcomes include: whether an individual with a disability earns a credential, whether they enter employment, whether they retain employment, and their reported earnings.

Using a quasi-experimental matched comparison group design, we will aim to answer two distinct research questions:

1) What is the impact of DEI Round 5-6 services on treatment participants’ outcomes, as compared to similar participants in non-DEI LWIAs?, and

2) What is the impact of the career pathways intervention component of the DEI Round 5-6 intervention on participants’ outcomes, as compared to similar participants within the same LWIAs who did not participate in career pathways components? We will control for a variety of demographic characteristics through the use of the WIOA administrative data and the participant survey.

Impacts will be estimated using regression procedures that control for baseline covariates. The covariates can adjust for the presence of any differences between observable baseline characteristics across treatment and comparison sites, as well as differences between participants who do and do not enroll in the career pathways intervention component at the treatment sites. We will use a multi-level model to account for inherent differences between the treatment and comparison LWIA sites. The regression model will include individual characteristics such as gender, race/ethnicity, age, educational attainment, marital status, employment status and earnings at time of registration, employment history variables, race, ethnicity, gender, disability type, and severity of impairment. Indicator variables for the LWIA sites will also be included. The analysis will be performed separately for each state. In addition, a combined analysis will be performed in which the impact of DEI Round 5-6 interventions is estimated for all six grantees.

A.16.2.3 Analyses of Qualitative Data

Site visit data will provide a wealth of information on Round 5 DEI program start-up, implementation and system change. Site visits will be conducted in spring 2016, 2017 and 2018 in order to collect information on start-up and program implementation, system change and how it affects service availability, accessibility and customer outcomes. The DEI system change framework will focus on the particular strategies and outcomes of each grantee. It will identify the pathways by which change is expected to occur and the role(s) that the DEI is expected to play in producing that change. The DEI system change framework will identify organizational and individual members of each grantee’s workforce development system and its contributions to system change. Follow-up site visits will examine how the selected strategies produced the intended system and customer-level outcomes by linking DEI customer service utilization to services provided by the agencies and organizations that comprise the workforce development system in each state.

The qualitative data for this evaluation are derived from the in-depth interviews, customer focus groups, document collection, and event/activity observations conducted during the site visits. The data sources will include site visit write-ups, interview and focus group transcripts, and write-ups of site visitor observations of trainings, collaborative meetings, etc., that are related to the site’s implementation of the DEI. We will also analyze information obtained from customers that participate in the focus groups on their disability, employment status/history, education and training, and self-perceived barriers to employment, and degree or training completion. In addition, we will perform a comprehensive content analysis of program materials, grant applications, and other relevant documents.

The credibility of an evaluation’s findings ultimately stems from the quality, transparency, and rigor of methods used to collect, analyze and write about the data. In conducting qualitative research, it is critical to adhere to quality standards and to employ tools for rigorous and systematic processes that compare well to the data management processes and the level of analytic transparency found in the quantitative realm. Our approach to the management of qualitative data and to the standards and processes for rigorous qualitative data analysis is based on practical, tried-and-true strategies for ensuring quality in data acquisition, data management, and data analysis, and will ensure that qualitative analyses of program implementation, use of DEI strategies and system change are rigorous and replicable.

We will conduct our qualitative data analysis using principles of Grounded Theory modified to fit the scope and purpose of the study (Glaser & Strauss, 1994). As is the case with most qualitative studies, the data are voluminous and require data reduction. Key to this effort is the development of a coding scheme that is aligned with the conceptual framework of the overall evaluation. We will create a hierarchy of conceptual categories linked to the research questions and system change framework. The categories, or “codes,” at the top of the hierarchy will, at a minimum, align with the six project components required of all nine DEI grants (State Level DEI Project Lead; DRC; AJC accessibility; participation in TTW as an EN; sustainability; and support for the DEI Evaluation). Also included as “top codes” will be the major strategies selected by the grantees to achieve the objectives of the DEI: integrated resource teams; integrated resources; customized employment; self-employment; Guideposts for Success; asset development; and partnerships and collaboration.

During Year 1, the focus of the coding scheme for the qualitative data analysis will be implementation start-up issues and documentation of program design. In subsequent years, the focus of the coding and analysis effort will shift so that our qualitative analyses will allow us to document the ongoing implementation of the program, refinements or shifts in design, and system change. In particular, the site visit data will be used to document the implementation of the specific DEI program strategies each DEI grantee is implementing.

A.16.2.4 Publication

The evaluation team will provide analyses of program impact and customer outcomes in annual and final reports, as well as descriptions of program start-up, implementation issues and procedures grantees use to resolve program challenges. In particular, the evaluation will look into the impact of the DEI on the employment, employment retention and employment preparation rates of customers with disabilities, as well as the completion of training programs, certifications, high school/GED completion and college degrees. The evaluation will also examine the utilization of AJC services by customers with disabilities. The DEI Evaluation will allow DOL to identify best and promising practices and replicate system change activities that successfully lead to improvements in customer outcomes. The study will also provide direction for other states that seek to replicate the DEI. The data resulting from this study is the property of DOL.

DOL will utilize information from the DEI Evaluation to inform their strategic planning process, promote best and promising practices and help other states’ workforce development systems become more integrated and efficient, leading to better access and availability of services for customers with disabilities. Any published information will be used to inform stakeholders of the findings and how they can inform workplace development systems and policies. A final report will be made available on DOL’s website. Additionally, materials will be prepared for journals, conferences and associates interested in the subject. Finally, findings will be applied to the development of training modules on workplace development systems.

A.17 Display of Expiration Date of OMB Approval

The DEI Evaluation will show the OMB expiration date on all data collection instruments.

A.18 Exceptions to Certification Statement

There are no exceptions to the certification statement

References – Part A

Creswell, J. W. (2007). Qualitative inquiry & research design: Choosing among five approaches. Thousand Oaks, CA: Sage Publications.

Department of Labor. (2010). Notice of availability of funds and solicitation for grant applications (SGA) for cooperative agreements under the Disability Employment Initiative. Federal Register, 75(152),

Glaser, B. G., & Strauss, A. L. (1994). The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, IL: Aldine Publishing Company, 1967. Kreuger, R.A. Focus Groups: a practical guide for applied research. 2^nd Ed. Thousand Oaks, CA: Sage Publications.

Livermore, G., & Goodman, N. (2009). A review of recent evaluation efforts associated with programs and policies designed to promote the employment of adults with disabilities. Ithaca, New York: Rehabilitation Research and Training Center on Employment Policy for Persons with Disabilities.

Livermore, G., & Coleman, S. (2010). Use of AJCs by Social Security Disability beneficiaries in four states implementing Disability Program Navigator initiatives. Washington, DC: Mathematica Policy Research.

Office of Management and Budget (1999/2010). Estimating paperwork burden. Retrieved from http://www.whitehouse.gov/omb/fedreg_5cfr1320

Ryu, E., Couper, M, & Marans, R. (2006) Survey incentives: Cash vs. in-kind; Face-to-face vs. mail; Response rate vs. nonresponse error. International Journal of Public Opinion Research, 18 (1): 89-106.

Singer, E., Groves, R., & Corning, A. (1999). Differential incentives: Beliefs about practices, perceptions of equity, and effects on survey participation. Public Opinion Quarterly, 63, 251-260.

Singer, E., Van Hoewyk, J., Gebler, N., Raghunathan, T., & McGonagle, K. (1999). The effect of incentives in telephone and face-to-face surveys. Journal of Official Statistics, 15, 217–230.

United States Department of Labor, Bureau of Labor Statistics. (2011, January). Economic News Release: December 2010 employment status of the civilian population by sex, age, and disability status, not seasonally adjusted. Retrieved from http://www.bls.gov/news.release/empsit.t06.htm

Willimack, D., Schuman, H., Pennell, B., & Lepkowski, J. (1995). Effects of a prepaid non-monetary incentive on response rates and response quality in a face-to-face survey. Public Opinion Quarterly, 59, 78–92.

Zhang, F. (2010). Incentive experiments: NSF experiences. (Working Paper SRS 11-200). Arlington, VA: National Science Foundation, Division of Science Resources Statistics.

1 Pub. L. No. 111-117

2 U.S. Department of Labor, Office of Disability Employment Policy. (2010). Disability Employment

3Initiative fact sheet October 2010. Retrieved from http://www.dol.gov/odep/topics/DEI.htm

U.S. Department of Labor, Office of Disability Employment Policy. (2010). Disability Employment

4 As certified ENs, individual workforce centers, LWIAs and states can receive SSA payments for each beneficiary who assigns them their “ticket” who enters employment and earns substantial income through their employment. ENs developed through the DEI initiative represent the majority of ENs operated by the public workforce system.

5 Information collection for the first 3 data requests relied heavily on previous control number 1230-0006.

6 These impacts will reflect both impacts for treatment and comparison group customers.

7