DEI Eval OMB Package Supporting Statement Part A_FINAL_03.30.12

Paperwork Reduction Act Submission
Supporting Statement Part A for the Office of Disability Employment Policy (ODEP)

2010-2013 Disability Employment Initiative Evaluation

March 29, 2012

Table of Contents

Part A: Justification for the Study 3

A.1 Circumstances Necessitating the Collection of Information 3

A.1.1 Background Information: The Disability Employment

Initiative (DEI) 3

A.1.2 Purpose of the DEI Evaluation 4

A.1.3 Data Collection Activities 4

A.1.3.1 Annual Site Visits 4

A.1.3.2 DEI Data System 5

A.1.3.3 Timeline for DEI Evaluation Data Collection Activities 5

A.2 Purposes and Uses of the Data 6

A.2.1 DEI Data System 6

A.2.2 Annual Site Visits 7

A.3 Use of Technology to Reduce Burden 7

A.3.1 DEI Data System to Reduce Burden 8

A.3.2 Site Visit and Telephone Interview Data Collection

Technology 9

A.4 Efforts to Avoid Duplication 9

A.5 Methods to Minimize Burden on Small Entities 9

A.6 Consequences of Not Collecting Data 10

A.7 Special Circumstances 11

A.8 Federal Register Announcements and Consultation 11

A.8.1 Federal Register Announcement 11

A.8.2 Consultation Outside of Agency 11

A.9 Payments of Gifts 12

A.10 Assurance of Confidentiality and Disclosure Limitation

Methodology 13

A.11 Justification for Sensitive Questions 14

A.12 Estimates of Hours Burden 15

A.12.1 Annual Site Visits 15

A.12.2 DEI Data System 19

A.13 Annualized Cost to Respondents 22

A.14 Annual Cost to the Federal Government 22

A.15 Reasons for Program Changes 24

Table of Contents (continued)

A.16 Plans for Tabulation and Publication 24

A.16.1 Tabulation 24

A.16.1.1 Analyses of Quantitative Data 24

A.16.1.2 Analyses of Qualitative Data 25

A.16.2 Publication 26

A.17 Approval Not to Display the Expiration Date for

OMB Approval 27

A.18 Exceptions to the Certification Statement 27

References – Part A 28

Appendices

Appendix 1: Seven DEI Strategies

Appendix 2: Site Visit Question Matrix

Appendix 3: DEI Data System Data Elements

List of Figures

Figure A-1: Data Collection and Publication Timeline 6

List of Tables

Table A-1: Estimated Annual Hours of Burden and Cost Estimate Due to

Site Visits (Round 1 States) 17

Table A-1.1: Estimated Annual Hours of Burden and Cost Estimate Due to

Site Visits (Round 2 States) 18

Table A-2: DEI Data System Burden Estimates for Round 1 and

Round 2 States 20

Table A-3: DEI ROCIS Crosswalk 22

Table A-4: Breakdown of Costs for Site Visits/Telephone Interviews and

the DEI Data System 23

PART A. JUSTIFICATION FOR THE STUDY

A.1CIRCUMSTANCES NECESSITATING THE COLLECTION OF INFORMATION

This OMB package is being submitted to obtain clearance for two new data collections associated with the Disability Employment Initiative (DEI) evaluation: 1) Annual Site Visits/Telephone Calls to sixteen DEI grantees and 2) implementation of the DEI Data System. These data collection efforts are essential to the measurement of program implementation, system change, program impact, and customer outcomes and will provide the U.S. Department of Labor (DOL) with important information for strategic planning, program replication and development of disability employment policy.

A.1.1 Background Information: The Disability Employment Initiative (DEI)

At present, the employment rate of people with disabilities is just 18 percent, compared with a rate of 64 percent among individuals without disabilities (DOL, Bureau of Labor Statistics [BLS], 2011). The DEI is designed to reduce this discrepancy by helping states: 1) Improve educational, training, and employment opportunities and outcomes of youth and adults with disabilities who are unemployed, underemployed, and/or receiving Social Security disability benefits; and 2) Help individuals with disabilities find a path to the middle class through exemplary and model service delivery by the public workforce system (DOL, 2010).

The Combined Appropriation Act of 2010, Division D of Public Law 111-117, includes $12 million in funds to the Employment and Training Administration (ETA) and $12 million to the Office of Disability Employment Policy (ODEP) to develop and implement a plan for improving effective and meaningful participation of persons with disabilities in the workforce and to evaluate the impact of the Disability Employment Initiative (DEI). The DEI evaluation is a comprehensive outcome, impact and implementation study that includes the random assignment of pilot and comparison group Local Workforce Investment Areas (LWIAs) in each of the nine DEI grantee states.

In September 2010, Alaska, Arkansas, Delaware, Illinois, Kansas, Maine, New Jersey, New York and

Virginia (Round 1 Grantees) and in September 2011, California, Hawaii, Ohio, South Dakota, Tennessee, Washington and Wisconsin (Round 2 Grantees) received grants to implement the DEI. These DEI grantees have agreed to implement five program requirements in demonstration areas selected by DOL:

1. Hire a State DEI Project Lead;

2. Hire a Disability Resource Coordinator (DRC);

3. Maintain One-Stop Career Center accessibility;

4. Participate in the Ticket to Work program as an Employment Network; and

5. Plan for sustaining DEI activities after the three-year grant period.

In addition, DEI grantees have agreed to incorporate at least two of the following seven program design strategies in the same areas:

1. Integrated Resource Teams;

2. Integrated Resources;

3. Customized Employment;

4. Self-Employment;

5. Implementation of the Guideposts for Success;

6. Asset Development Strategies; and

7. Partnerships and Collaboration.

See Appendix 1 for descriptions of the seven DEI strategies.

A.1.2 Purpose of the DEI Evaluation

The purpose of the DEI evaluation is to report to DOL on DEI program start-up, how the 9 Round 1 grantees, funded in 2010 and the 7 Round 2 grantees, funded in 2011, resolve challenges to start-up and program implementation, approaches to create effective system change in workforce development systems, and measures of program impact and customer outcomes. This information will be shared with components within DOL (including the Employment and Training Administration [ETA] and the Office of Disability Employment Policy [ODEP]), DEI grantees, and other organizations involved in disability employment policy and practice. This information will help these stakeholders make necessary adjustments during the grant period, replicate successful program strategies and approaches used by DEI grantees, and support improvements in workforce development systems nationwide. DEI evaluation findings also will be used to improve program activities and services to customers. It supports DEI grantees and their partners in the development of workforce development systems that increase access and availability to employment and employment preparation services for customers with disabilities, including Ticket to Work participants and other Social Security disability beneficiaries.¹

Data collected over the life of this evaluation will catalog systemic challenges and effective solutions, and will help grantees and stakeholders understand the impacts of the DEI on service utilization and employment of customers with disabilities. This kind of strategic programming or data-based decision making underlies the DEI project itself --its parameters are based on the systematic data collection and findings from previous DOL programs.

A.1.3 Data Collection Activities

The DEI evaluation includes two types of data collection activities: 1) Annual Site Visits or Telephone Interviews and 2) Implementation of the DEI Data System, involving the collection of customer-level data.

A.1.3.1 Annual Site Visits

In years 2012 and 2013, the DEI Evaluation Team will conduct one site visit or a round of telephone interviews to the sixteen DEI grantees. In each of the randomly assigned LWIAs participating in the DEI evaluation, on-site or telephone interviews will be conducted with the DEI state lead, disability resource coordinators, workforce investment board (WIB) directors, One-Stop managers, One-Stop staff members, and agency partners and employers. Additionally, eight to ten One-Stop customers will be asked to participate in a customer focus group. The site visits will be tailored to fit each grantee as well as each DEI or comparison site. Variations across sites are likely to mean that the number and types of individuals to be interviewed will vary. The general domains to be investigated through collecting and analyzing site visit data include: the current status at baseline and change in grantees’ workforce development system at follow-up; grantee customer characteristics; implementation of the five grant requirements; implementation of grantee selected DEI strategies; program implementation challenges; and system change. The study team developed respondent-specific questions to be asked in the interviews and focus groups. These questions are attached as Appendix 2.

In spring 2012 and 2013, the DEI Evaluation Team will conduct four-person site visits or a round of telephone interviews. The decision to conduct site visits in the first year of the DEI grants, telephone interviews in the second year and a second round of site visits in the third year, allows the DEI Evaluation Team to collect extensive baseline (year 1) and follow-up (year 3) information including observations of intensive and training programs and customer focus groups. The telephone interview data collection conducted in the second year of the DEI grants is a low-cost approach to collecting information on the progress made by grantees since the first year of the grant. Without follow-up site visits in the third year of the round 1 and round 2 DEI grants, the DEI Evaluation Team will not be able to collect information on systems change. This ordering should not have an appreciable impact on the quality of the data collected because each round of DEI grantees have the same data collection requirements at the same intervals during their respective grant periods.

The site visits are five days in length, approximately one day for the state and one day each for the participating LWIAs, visiting at least one One-Stop in each LWIA in addition to meeting with partners, employers and conducting customer focus groups. In 2012, site visits will be conducted with Round 2 Grantees and telephone interviews will be conducted with Round 1 Grantees. In 2013, site visits will be conducted to Round 1 Grantees and a round of telephone interviews will be conducted with Round 2 Grantees. For states with more than two DEI sites and two comparison sites, additional trained site visitors will be used. The evaluation team will ensure that at least two members of each site visit team are senior staff, and at least two are experienced focus group facilitators, to maximize the value of the visit and minimize the burden on the respondents at each site. The telephone interviews will include interviews with the same staff that will be included in the site visits. However, focus groups will not be conducted for these grantees.

A.1.3.2 DEI Data System

The purpose of the DEI data system is to collect information not contained in the Workforce Investment Act Standardized Record Data (WIASRD) and Wagner-Peyser administrative data systems, including additional demographic, outcome and service utilization data. This information will be used to profile DEI customers with disabilities receiving services at each participating One-Stop, their disability characteristics, service utilization and outcomes. The DEI Data System will provide important information that will allow the evaluation team to document the services received by customers and their employment outcomes (See Appendix 3: DEI Data System Data Elements). The DEI Evaluation Team and DOL have already consulted with each DEI grantee about their preferred mode of data collection to support the DEI Data System. The system will be integrated with the usual data collection processes at each One-Stop. (See section A.3 for descriptions of the three modes of DEI Data System data collection offered to DEI grantees.)

A.1.3.3 Timeline for DEI Evaluation Data Collection Activities

The DEI Evaluation is designed to collect information on program start-up, challenges to implementation, program impact and customer outcomes. Site visit data collection will occur in Spring of 2012 and 2013. Uploading of DEI Data System data from DEI grantees will occur on a quarterly basis. Publications based on the analysis of DEI data are scheduled for 2012 and 2013 (See Figure A-1).

Figure A-1: Data Collection and Publication Timeline

A.2 PURPOSES AND USES OF THE DATA

Data will be collected from two sources: 1) The DEI Data System and 2) Annual Site Visits to the nine DEI grantees.

A.2.1 DEI Data System

The purpose of the information collected by grantees and entered into the DEI Data System will be used by DOL to determine the efficacy of the DEI based on customer outcomes, describe customer characteristics and patterns of service utilization. Specifically, these data will allow DOL to assess the DEI’s impact on:

• Number and percentage of individuals with disabilities served at One-Stop career centers;

• Characteristics of customers with disabilities who are served;

• Availability and nature of services and support for One-Stop customers with disabilities;

• Degree and training completion rates of customers with disabilities;

• Employment rates and employment retention rates as of the third quarter after the exit quarter among customers with disabilities;²

These data are necessary for DOL officials to make informed decisions about program effectiveness, replication, continued funding, and new initiatives. Without these data, it will be impossible to determine whether the DEI met its goals and warrants further replication and/or revisions to its design. The risks that would result from such a knowledge-gap include future misuse of resources (funding of an ineffective initiative), missed opportunities (failure to fund an effective initiative), and even harm to individuals with disabilities who seek employment services (in the event of misused resources or missed opportunities).

DOL and state officials, LWIB directors, Disability Resource Coordinators, and One-Stop Career Center staff will use data from the DEI Evaluation to continually refine services that improve employment preparation and employment rates of customers with disabilities.

A.2.2 Annual Site Visits

The purposes of the Annual Site Visits and Telephone Interviews are to document grantee start-up issues and the implementation of the five requirements and strategies selected by each grantee, the context in which each grant is being implemented; program implementation challenges; and systems change. For the youth programs, site visits and telephone interviews will provide information on the approaches DEI grantees use to identify the needs of youth with disabilities, including the strategies they pursue in order to prepare youth with disabilities for employment, identify employment training opportunities, and collaborate with local employers, community-based agencies, employer organizations and community colleges. In addition, the site visits will include information on how grantees assist youth who are transitioning out of foster care or the juvenile justice system, among other topics. The second round of site visits/telephone interviews will allow the DEI evaluation team to measure system change based on the baseline information collected during the first round of site visits. It will also allow for the collection of information on strategies for achieving system change.

Data from the annual site visits/telephone interviews to the sixteen DEI grantees also will address the following issues:

• How specific content of a particular strategy influences or interacts with contextual and organizational factors within LWIAs and one-stop centers;

• Challenges to systems integration across the workforce development system;

• The nature of the relationship between the LWIAs and mainstream (e.g., education, mental health) agencies;

• How collaborations and partnerships facilitate the achievement of employment outcomes for customers with disabilities, including those co-enrolled with WIA;

• Challenges in collaborating with key agencies, WIA formula and discretionary grants and priority initiatives;

• Specific challenges serving customers with disabilities;

• Factors associated with partnership achievement in terms of service integration and access;

• Specific challenges that slow or inhibit services and systems integration, blending and braiding of resources

• Innovative promising practices used to serve youth and adult customers with disabilities

A.3 USE OF TECHNOLOGY TO REDUCE BURDEN

Every effort has been made to structure the DEI evaluation to minimize the burden on One-Stop Career Center staff and customers. The DEI Evaluation Team will design a DEI Data System that will capture customer-level information. The DEI Data System includes 11 data elements for employed adults, 12 for unemployed adults, 16 for employed youth, and 17 for unemployed youth. The DEI Evaluation Team and DOL have communicated with each DEI grantee about the implementation of the data system. In addition, codebooks from existing systems being used by the grantees have been reviewed to ensure that the DEI Data System data elements do not duplicate any of the information already being collected. In the event that data elements are already being collected by a DEI grantee, the duplicated elements will be removed from the DEI Data System and collected through the state’s existing administrative systems.

A.3.1 DEI Data System to Reduce Burden

The DEI Data System will be integrated with current One-Stop data collection activities. Based on discussions with DOL officials and grantees, three modes of data collection are being provided to the One-Stops in order to assist them in collecting data for the DEI Data System:

1. Internet Link on One-Stop Desktops: This mode of data collection will allow One-Stops to place an Internet Link (e.g. DEI Data System Entry Screen) on any windows-based computer. The link will open the DEI Data System website with “Baseline” and “Follow-Up” screens. One grantee (VA) selected this option.

2. Integration into an Existing Management Information System: Some state grantees use sophisticated management information systems such as the One-Stop Operating System (OSOS) or America’s Job Link while others use a customized system developed by their state workforce development agency. The DEI Evaluation Team assisted five grantees in comparing the DEI Data System to their existing system so that data elements not currently collected can be added. This process has been completed and duplicate data elements have been removed for the six states, including Arkansas, Delaware, Illinois, Maine, New Jersey and Delaware, that selected this option.

3. Hardcopy Form: Two grantees, Arkansas and Maine, requested the use of hardcopy DEI Data System forms rather than an Internet link or the integration of DEI data elements with an existing information system. These grantees reported that the use of hardcopy forms would be considerably more efficient than the addition of an Internet link or adding to their existing systems since they often use “group registration” procedures that are more amendable to paper and pencil data collection. For the two grantees that chose this option, the DEI Evaluation Team will collect the hardcopy forms from each grantee through their State DEI Lead on a quarterly basis and enter the information into the DEI Data System.

Regardless of the mode of data collection, the DEI Evaluation Team will customize the DEI Data System mode of data collection for each grantee and if necessary, individual One-Stops. “Technical calls” with each grantee have been conducted to discuss the advantages and disadvantages of the three modes of data collection, how to integrate DEI Data System data collection activities with existing One-Stop data collection systems and the customer registration process. In addition, a toll-free technical assistance telephone number is available for assistance with the DEI Data System, as are quarterly webinars and site visits from the DEI Evaluation Team and DEI Data System staff.

The new data elements contained in the DEI data system will require an estimated 4.8 minutes to be completed by each customer with the help of a One-Stop staff. This estimate is based on a pilot test of nine One-Stop customers.

A.3.2 Site Visit and Telephone Interview Data Collection Technology

The site visits and telephone interviews are designed to collect information on grantee start-up, implementation and systems change. This data collection effort (for both the site visit interviews and telephone interviews) will rely on 45 to 60-minute interviews with DEI State Leads, DRCs, LWIB Directors, One-Stop Managers and staff, agency partners and employers, and for site visits only, focus groups with customers. To minimize burden on respondents during this process, the evaluators will utilize the following technology:

• An Internet based calendar will remind all respondents of the date and time of their interview. For site visit interview respondents, the calendar will provide location and driving directions, if necessary. The calendar will provide automated reminders to each respondent 24 hours and 30 minutes prior to their scheduled interview. This technology will provide updates if dates and times of interviews are changed.

• The DEI Evaluation Webpage (http://socialdynamicsllc.com/deigranteessecure.html) provides information on the DEI Evaluation as well as contact information for each Evaluation Liaison, a document archive that includes documents related to the DEI Evaluation and a Frequently Asked Questions archive that respondents can access to learn about the DEI Evaluation and minimize time on site discussing the evaluation methodology prior to each interview.

• Privacy and informed consent forms will be available online for respondents that have a certified digital signature.

A.4 EFFORTS TO AVOID DUPLICATION

The DEI evaluation includes only new data collection activities and does not include information that has been collected previously. The evaluation will provide a rigorous assessment of the impact of the DEI, which has not been possible for previous DOL disability employment initiatives. The need for this research, and the fact that it is not duplicating existing information, has been confirmed by background research conducted by members of the DEI Evaluation Team, all of whom have extensive experience evaluating DOL disability employment initiatives (e.g. Livermore and Goodman 2009). Recent literature reviews and attendance at national meetings by research partners also provided confirmation that the information obtained through the DEI evaluation will not be redundant.

Additional safeguards have been built into the evaluation plan with the express purpose of avoiding duplication of effort. Researchers will access and use administrative data that is regularly reported by states, including the Wagner-Peyser Labor Exchange and WIASRD. DOL consulted with state DEI representatives and information technology (IT) specialists to learn about any state-specific collection of customer data that is currently underway and have made efforts to integrate DEI Evaluation data collection activities with current data collection efforts at the grantee level. State IT specialists have shared the code-books or data-dictionaries for any current data collection effort so that the evaluators can cross-check each state’s data elements with the DEI Data System data elements. This process allowed the evaluators to customize the collection of data for each grantee, asking only for those data elements that are not being collected by any other means.

A.5 METHODS TO MINIMIZE BURDEN ON SMALL ENTITIES

The “small entities” involved in the DEI Evaluation include employers and non-profit partner agencies that will participate in the site visit interviews. These respondents will participate in interviews that last 45 minutes rather than the 60 minute interviews being conducted with government agency staff and One-Stops. In addition, site visitors will travel to each respondent’s location to minimize the time dedicated to the interview process for small entity respondents. As shown in the site visit data collection matrix (Appendix 2), these respondents have a fewer questions assigned to their interviews and therefore a shorter period of time will be needed to conduct these interviews. Based on internal pilot tests, these interviews will be 45 minutes in duration.

A.6 CONSEQUENCES OF NOT COLLECTING DATA

Neglecting to collect the data specified in this document would prevent DOL from making fully-informed policy decisions regarding the delivery of employment services to people with disabilities. Collecting data on the status of unemployed or underemployed individuals with disabilities who are One-Stop customers is necessary to determine whether the DEI had the desired impact. The DEI Data System contains information that is essential to the DEI Evaluation. We hypothesize that the DEI will affect the number and characteristics of customers with disabilities seeking services at the pilot locations (e.g., disability type, employment history, education). Hence, the additional variables being collected through the DEI data system are necessary to measure any such effects and to control for them in conducting the impact analysis.

The collection of interview and focus-group data during annual site visits will ensure that the DEI Evaluation Team collects detailed information on program implementation/start-up and program design. The site visit data will enable the evaluation team to answer the ‘how’ and ‘why’ questions that in turn support meaningful system change and/or successful replication of the DEI. Specifically, by collecting data to document the challenges, solutions, and best practices of DEI grantees, additional grantees can implement the DEI approach more efficiently and effectively. Finally, the DEI intervention is comprehensive, requiring collaboration among a diverse group of stakeholders. Collection of data through interviews and focus groups ensures that DEI implementation and impact is understood from multiple perspectives, not the least of which are those of the program staff and the individuals whom the intervention is meant to assist.

Based on our prior experience conducting research on disability employment programs, conducting site visits on a one-year interval allows enough time for complex system changes to unfold, but not so much time that retrospective data becomes unreliable.

The present data collection schedule includes quarterly collection of customer-level data and annual collection of site-visit data. Two criteria were considered in determining this frequency: the methodological rigor of the evaluation and burden on the DEI grantees. The quarterly data collection of customer-level data (e.g. DEI Data System) will support a rigorous analysis of program impact by ensuring that enough data points are available to detect change in customer level outcomes. It will also allow the DEI Evaluation Team to monitor the submission of data to ensure that the incoming data is of appropriate quality and includes all required data elements and both baseline and follow-up intervals. In the case of missing data, analysts will be better equipped to fill in gaps or reconfigure existing quarterly data than if they were working with semi-annual or annual data. A quarterly schedule is also intended to streamline reporting requirements for states. This frequency is synchronized with the states’ existing federal reporting schedule. Since researchers will use the same data, states can, in effect, send their quarterly reports along with the DEI Data System elements.

A.7 SPECIAL CIRCUMSTANCES

All data collection activities comply fully with regulations. There are no special circumstances associated with these data collection.

A.8 FEDERAL REGISTER ANNOUNCEMENTS AND CONSULTATION

A.8.1 Federal Register Announcement

The 60-day notice to solicit public comments was published in the Federal Register June 20, 2011.

A.8.2 Consultation Outside the Agency

Randomization was carried out and assigned sites either pilot or control status. In an effort to carry out public consultation, states were contacted, and PRA may have been implicated during those discussions.

In planning this evaluation, DOL consulted with disability employment experts and grantees who will be data providers. ‘Random Assignment,’ ‘DEI Evaluation Orientation’ and ‘DEI Data System Technical’ conference calls were conducted with each DEI grantee. Individuals at the state level who had been assigned the role of DEI State Lead or DEI Primary Contact participated in these calls, as well as managers and staff from the Local Workforce Investment Boards representing an LWIA, and staff from One-Stop Career Centers, since their involvement and understanding of the data collection activities is integral to the successful implementation of the DEI evaluation. In addition, grantees discussed their existing administrative data systems and customer registration procedures. During ‘Technical Calls’ IT staff were informed of the plan to add variables (depending on those already collected) to their existing data collection procedures, provide an Internet link to the DEI Data System and/or use a paper and pencil data collection form. This allowed the grantees to explain what data collection procedures would be easiest for their One-Stop staff. IT experts from each grantee explained potential data sharing procedures, including the feasibility of sharing data and related challenges. These meetings ensured that the DEI Evaluation minimizes the data-collection burden on all stakeholders; asking only for those data not available elsewhere (i.e., avoiding duplication of effort), and ensuring the simplest delivery method possible. Consultations with data-providers have suggested that quarterly collection of customer data will not be burdensome, as it matches their existing reporting schedules for DOL. All agree that the data collection plans are clear and concise. No major concerns arose during these calls.

In addition, the DEI Evaluation Team consulted with experts from Mathematica Policy Research on the random assignment and sampling of LWIAs. This information was particularly important for the site selection process and to identify the data elements necessary to conduct an impact study of the DEI.

Persons Consulted

1. Joseph Ashley, VA DEI State Lead

804-662-7624

[email protected]

2. Georgia Coopersmith, Manager

804-640-1774

[email protected]

3. Brian Burns, NJ DEI Primary Contact

609 633-6420

[email protected]

4. Shawna Harper, AK DEI Primary Contact

907-269-5703

[email protected]

5. Elaine Kost, NY DEI Primary Contact

518-485-2151

Elaine [email protected]

6. Lily Kersh, AR DEI Primary Contact

[email protected],

7. Auta Main, ME DEI Primary Contact

207-623-7975

[email protected]

8. Rich Sanders, AK DEI State Lead

[email protected]

9. Jeff Schroeder, KS DEI State Lead and Primary Contact

785-296-0658

[email protected]

10. Glenn Sergeant, Sr., AR DEI State Lead

501-371-1016

[email protected]

11. Ed Tos, NY DEI State Lead

302-761-8275

[email protected]

12. Jonathan Wheat, IL DEI State Lead

312-494-6702

[email protected]

A.9 PAYMENTS OR GIFTS

Payment will be made only to focus group participants in the DEI evaluation. The evaluation includes annual site visits to DEI implementers at both the state and local levels. Site visits in spring 2012, and spring 2013 will include one focus group of One-Stop customers in each LWIA. The evaluators will work with One-Stop center staff to identify between 8-10 customers to participate in each focus group. All customers who participate in the focus group will receive $25.00 at the conclusion of the 90-minute session. The total cost to the evaluation will be $26,550 (118 LWIAs x 9 participants x $25.00). The money provided is intended as an incentive, to ensure adequate participation among customers.

Ensuring that focus groups include a minimum of 8 participants is an important part of the methodology. Groups smaller than this not only lack representativeness, but also are at greater risk of stifled or stagnant inter-personal dynamics (Kreuger, 1994). There is substantial research indicating that incentives increase response rates in social research (Ryu, Cooper, & Marans, 2006), including focus group research (Singer, Van Hoewyk, Gebler, Raghunathan, & McGonagle, 1999; Willimack, Schuman, Pennell, & Lepkowski, 1995). A snow-ball sampling procedure will be used to create the focus group samples. The DEI Evaluation Team will prepare flyers containing information on the focus group and the incentive of $25.00, as well as refreshments. Evaluation Liaisons will work closely with One-Stop Managers and Disability Resource Coordinators to ensure that focus group flyers are posted throughout the One-Stop Career Centers, posted online, distributed through RSS and listservs and emailed to customers.

In addition to the research on incentives, all research partners in the DEI evaluation (Social Dynamics, Berkeley Policy Associates, Altarum, and Mathematica) have extensive experience conducting focus groups, and can attest to the importance of providing a participant incentive. Focus groups using a cash incentive of similar value with a similar population have produced useful data in the following studies: Independent Studies of Start-Up USA Self-Employment Program and the WIRED Evaluation. Finally, researchers have found that providing cash at the time of the focus group is greatly appreciated by participants, and presenting the incentive at the conclusion of the group encourages participants to stay for the 90-minute duration, rather than leave part way through. According to the National Science Foundation, monetary incentives improve study participation and offset the costs of follow-up and recruitment of non-respondents (Zhang, 2010).

The only exception to conducting one focus group per LWIA will be in New York, where there are 20 LWIAs participating in the DEI Evaluation. In this case we will randomly select 14 of the sites (7 pilot sites and 7 comparison sites) in which to conduct focus groups.

A.10 ASSURANCES OF CONFIDENTIALITY AND DISCLOSURE LIMITATION METHODOLOGY

While this information collection makes no express assurances of confidentiality, considerable efforts will be undertaken to ensure the privacy of respondents is protected from improper disclosure. Individually-identifiable data will be collected in three ways for the DEI evaluation:

• Site Visit interview notes;

• Focus group notes and recordings;

• DEI Data System.

To respect and uphold the privacy of DEI participant information, all contractors involved in the evaluation will operate within the guidelines established by the federal Privacy Act of 1974 to protect respondents’ privacy and the privacy of all data collected. The Privacy Act of 1974 states that “micro-data files prepared for purposes of research and analysis are purged of personal identifiers and are subject to procedural safeguards to assure anonymity” (5 U.S.C. § 552a, Public Law No. 93-579). The evaluation contractors hired for this project have extensive experience with handling sensitive data. These contractors will have routine procedures in place to ensure the privacy of computerized and paper records, including the use of passwords and encrypted identifiers to prevent direct or indirect disclosures of information. Furthermore, the information management systems of these contractors will fully comply with the government’s system requirements. The system of records includes the Workforce Investment Act Standardized Record Data (WIASRD), the Wagner-Peyser administrative database and the Disability Employment Initiative (DEI) Data System. The DEI Data System was developed specifically for the DEI evaluation.

Two safeguards will be implemented to ensure privacy for this study: 1) data transferred via modem will be password encrypted prior to transmission and transmitted on limited access lines and 2) staff access to data storage and files will be limited to authorized personnel who have passwords.

Information or copies of information may be released only to authorized individuals and the organization must ensure that unauthorized individuals cannot gain access to, or alter participant information. Participant information must be released in accordance with Federal or State laws, court orders, or subpoenas.

Interview notes, and, to a lesser extent, focus group notes, may still contain personally-identifiable information. Researchers at Social Dynamics and Berkeley Policy Associates will work to report data in an aggregate manner whenever possible. If there are data to be shared that are intrinsically identifiable, researchers will check first with the respondent(s) for permission to share these results. Participation in all interview and focus group activities is voluntary. Although DEI state leaders have committed to assisting with the evaluation, and therefore may feel pressure to participate, researchers will make it clear to them that they may decline answering any question, and may stop the interview at any time.

Focus group participants will be told the following: The information you provide will be kept private, and will not be disclosed to anyone but the researchers conducting this investigation, except as otherwise required by law.

Certain disclosure limitations will be used to maintain the privacy data and findings. The data file for the DEI evaluation will be comprised of the DEI data system, WIASRD and Wagner-Peyser data systems from each DEI state. Although individual customer records will be included, no geographic identifier other than the state in which services were provided, will be included. In addition, findings based on data from the DEI Data System will be presented inly in the aggregate (program-wide) to represent the entire DEI, rather than individual states or LWIAs. In addition, table cells in reports with standard errors larger than 50 percent and table cells with values less than 5.0 will not be disclosed. Random rounding will be used to suppress complementary cells if their value can be calculated by subtraction from the marginal total.

A.11 JUSTIFICATION FOR SENSITIVE QUESTIONS

Site visits to selected LWIAs and One-Stops will not require responses to questions about sensitive issues. The study will ask the respondents at each site about: customer characteristics that impede and/or facilitate employment success; implementation of DEI; DEI implementation differences when serving youth; challenges that need to be addressed to improve employment outcomes of individuals with disabilities; DEI effects on system change and customer outcomes; and best practices for facilitating changes in outcomes.

The DEI data system will collect information about the nature of a customer’s disability. This potentially sensitive information is not available from other sources and is considered critical to the evaluation of the employment outcomes that will be analyzed in the evaluation. Disability and health variables have been shown to be significant determinants of employment outcomes. Without the collection of this information, the evaluation could not move forward.

A.12 ESTIMATES OF HOURS BURDEN

The Paperwork Reduction Act (PRA) requires the agency to account for the amount of burden that it is placing on the public when seeking information on behalf of the Federal government. This burden is measured in terms of hours and includes the following activities:

1. Reviewing instructions

2. Using technology to collect, process and disclose information

3. Adjusting existing practices to comply with requirements

4. Searching data sources

5. Completing and reviewing the response; and

6. Transmitting or disclosing information (Office of Management and Budget, 1999/2010).

Under the DEI Evaluation contract, there are two types of data collection activities:

1. Annual Site Visits;

2. DEI Data System.

A.12.1 Annual Site Visits

There are currently two rounds of grantees participating in the DEI. Round 1 includes 9 states that were funded in 2010. Their grants will continue through 2013. Seven states were funded in 2011. Their grants will continue through 2014. On an annual basis, researchers will conduct site visits to one round of states and telephone interviews with the other round of states. In 2012, researchers will conduct site visits to round 2 states and telephone interviews with round 1 states. In 2013, round 1 states will receive site visits and round 2 sites will receive telephone interviews. . Whether site visits or telephone interviews are used, in each grantee workforce development system, interviews will be conducted with the DEI State Lead, Disability Resource Coordinator(s), One-Stop Staff, Partners & Employers, WIB Directors and Focus Group Customers. It is important to mention that DOL is considering funding a 3^rd round of states in late 2012.

Respondents will be selected for telephone interviews and site visit interviews using a respondent-drive sampling methodology (RDS). The DEI Evaluation Team has already collected information on the key personnel (and their organizations) in each LWIA participating in the DEI evaluation. Prior to scheduling the interviews, the assigned evaluation liaison will prepare an initial list of respondents that will be shared with the DEI state leads. The DEI state leads will be asked to provide additional names and contact information for key personnel not included on the list. As interviews are scheduled, the site evaluation liaison will ask each respondent to provide additional names and contact information for key personnel along with their organizational affiliations. This process will begin with a list of approximately ten key stakeholders, including the DEI state lead, Disability Resource Coordinators, WIB directors, One-Stop managers and state officials, as the central points of contact for the sample. Each will be asked to provide names and telephone numbers (or e-mail addresses) of people from five stakeholder groups: state officials, WIB staff, One-Stop staff, employers, service providers, and local government. Each referred person will be asked to refer additional informants. The chain of recommended informants will initially diverge, as many potential respondents will be recommended, then converge as a few key names are repeatedly mentioned. Thus, the final list of informants will include the most frequently named individuals, who will be asked to participate in a 45-60 minute interview with a member of the DEI evaluation team.

As shown in Table A-1, the number of respondents per Round 1 Grantee depends on the number of LWIAs participating in the DEI Evaluation. The cumulative hours of burden due to the site visits to DEI grantees for the entire project period for these grantees is 2,183 for two rounds of site visits or telephone interviews in years 2012 and 2013. As shown in Table A-1.1, the cumulative hours of burden due to the site visits/telephone interviews to Round 2 DEI grantees for the entire project period is 839. For Round 1 grantee it is 1,344. Each DEI State Lead, DRC and WIB Director will be interviewed in each state and two One-Stop and partners/employers will be interviewed per LWIB. A minimum of three customer focus groups (during site visits only) will be conducted in each state. In states that have one LWIB, one focus group will be conducted. In states with more than one LWIB, three focus groups will be conducted.

A.12.2 DEI Data System

To determine the number of customers with disabilities from whom data will be collected via the DEI Data System, the numbers of FY 2009 WIASRD and Wagner-Peyser services users with disabilities were obtained from the DEI grant applications. The numbers of customers with disabilities reported in Table A-2 include data only from the LWIAs that were selected to participate in the DEI evaluation, including both pilot and comparison sites.³ These numbers were then reduced by 11 percent (based on information reported in Livermore, & Coleman 2010) to obtain an approximate unduplicated count of customers with disabilities (See Table A-2 Column 1).

Because the DEI Evaluation includes baseline and follow-up data collection, burden on customers and staff will occur twice. Therefore, multipliers are included in Table A-2 to account for these two points of contact for customers and staff. For each data collection point, customers with disabilities and staff will spend on average 4.82 minutes (or .08333 hours) (Column 6) completing the DEI data collection form per point of contact (baseline or follow-up), as determined by a pilot test with 9 One-Stop customers. As shown in Table A-2, the annual burden estimate for the DEI Data System 11,647 hours for Round 1 and Round 2 Grantees combined.

The ROCIS crosswalk in Table A-3 provides a step-by-step explanation of how the ROCIS burden estimates were calculated for the DEI Evaluation. There are four crosswalks: 1) Partners and Employers; 2) Consumer Focus Groups; 3) State Government Employees; 4) DEI Data System. Each crosswalk includes the same fields with different values in each field that pertain to the size of the respondent group, the estimated amount of time the group will be engaged in data collection activities for the DEI Evaluation and the cost per response (which varies by respondent type). With this information, the annual burden of respondent time (Annual Burden (Hours) and cost (Annual Cost) is calculated as identified below:

1. Partners and Employers -- Partners and employers include all non-government service providers and employers that work with the One-Stops to recruit and hire customers with disabilities. The annual frequency of response events is 1 for this respondent category. The estimated number of hours per response is 45 minutes and the estimated number of respondents is 174. Because the hourly rate estimate (from the Department of Labor, Bureau of Labor Statistics) is $40.29, the cost per response is $30.22 (.75*$40.29). The annual burden (hours) is .75*174 = 130.5. The annual cost is calculated as follows: $40.29 (hourly rate) * 130.5 (annual burden hours) = $5,257.85.

2. Consumer Focus Groups -- Consumers are the One-Stop customers with disabilities who register for One-Stop intensive and/or training services during the grant period. The annual frequency of response events is 1 for this respondent category. The estimated number of hours per response is 90 minutes and the estimated number of respondents is 324. Because the hourly rate estimate (from the Department of Labor, Bureau of Labor Statistics) is $21.35, the cost per response is $32.03 (1.5*$21.35). The annual burden (hours) is 1.5*324 = 486. The annual cost is calculated as follows: $21.35 (hourly rate) * 486 (annual burden hours) = $10,376.10.

3. State Government Employees -- State government employees include state agency staff, One-Stop staff and Disability Resource Coordinators. The annual frequency of response events is 1 for this respondent category. The estimated number of hours per response is 1.63 and the estimated number of respondents is 368. Because the hourly rate estimate (from the Department of Labor, Bureau of Labor Statistics) is $21.35, the cost per response is $34.80 (1.63*$21.35). The annual burden (hours) is 1.63*368 = 599.84. The annual cost is calculated as follows: $21.35 (hourly rate) * 599.84 (annual burden hours) = $12,806.58.

4. DEI Data System -- The DEI Data System is completed by customers with support from a One-Stop staff member. The number of responses per respondent is calculated by adding the total of number of estimated respondents from the Data System Burden Spreadsheet for Round 1 and Round 2 grantees and then dividing by 16 (the number of grantees to arrive at the number of respondents per grantee (9,061.65). However, the burden is calculated by the total number of responses rather than respondents. To arrive at the total of number of responses the total number of respondents for Round 1 (38,441) and Round 2 (34,052) was multiplied by 2 and then added together as follows: R1 (38,441*2) + R2 (34,052*2) = 144,986.34, which is the total number of responses. The estimated number of hours per response is .08033 (hours) and the estimated number of responses is 144,986.34. Because the hourly rate estimate (from the Department of Labor, Bureau of Labor Statistics) is $21.35, the cost per response is $1.72 (.080333*$21.35). The annual burden (hours) is .080333*144,986.34 = 11,647.24 hours. The annual cost is calculated as follows: $21.35 (hourly rate) * 11,647.24 (annual burden hours) = $248,668.49.

5. Table A-2: DEI Data System Burden Estimates

(Round 1 Grantees)

(Round 2 Grantees)

Table A-3: DEI ROCIS Crosswalk

A.13 ANNUALIZED COST TO RESPONDENTS

This information collection imposes no capital, start up, maintenance, or operations costs on respondents beyond the value of time identified in item 12.

Table A-4: DEI Data System

Annualized Cost to Respondents

(Round 1 Grantees)

(Round 2 Grantees)

A.14 Annual Cost to the Federal Government

Contractor costs associated with the site visits to 7 Round 2 grantees and telephone interviews with 9 Round 1 grantees are estimated to be $499,748 in 2012. Contractor costs associated with the development, implementation and ongoing data monitoring of the DEI data system are estimated to be $521,485 in 2012. The total annual cost is $971,233 in 2012. These figures were estimated as the sum of the anticipated direct labor; fringe and burden on direct labor; other direct costs associated with staffing the site visits (e.g. computers, telephone, reproduction, shipping, mail, travel and per diem; general and administrative costs, and contractor fee) and staffing the design, implementation, maintenance and grantee training related to the DEI data system, including all equipment and supplies (e.g. computers, software, telephone, reproduction, shipping, mail, travel and per diem; general and administrative costs, and contractor fee).

Table A-4: Breakdown of Costs for Site Visits/Telephone Interviews and the DEI Data System

A.15 REASONS FOR PROGRAM CHANGES

The DEI Evaluation is a new data collection.

A.16 PLANS FOR TABULATION AND PUBLICATION

The DOL requires information on the impact of the DEI and customer outcomes, including changes in the employment rate, employment retention rate and employment preparation of customers with disabilities. DOL plans to use the data to provide technical assistance and support to the nation’s local workforce development systems and assist DEI grantees in improving their system change efforts and services to customers with disabilities. The analytical approaches outlined below are designed to describe the DEI, measure its impact, and document customer outcomes.⁴

A.16.1 Tabulation

The following sections provide analysis of quantitative data from the DEI data system, qualitative site visit data and the publication of reports on the DEI for DOL.

A.16.1.1 Analyses of Quantitative Data

The data analyses will begin with developing basic descriptive statistics in tabular and graphical form to provide information about the characteristics and outcomes of the DEI pilot and comparison samples, overall and by state and LWIAs. For example, descriptive analyses will focus on customer and site characteristics, providing a profile of customers at each of the nine sites. Outcomes of interest will be measured at the individual level and include measures of service utilization and employment outcomes. The key utilization measure is the proportion of individuals registering for services in a One-Stop center who report having a disability. Employment outcomes include whether an individual with disability enters employment, employment retention, and earnings.

With a random assignment design (described further in Section B.1), there should be no systematic observable or unobservable differences between LWIAs in the treatment and comparison groups except for the availability of the DEI services, which are determined randomly. Thus, simple differences in the mean values of utilization measures between LWIAs assigned to the two research groups will yield unbiased impact estimates of program effects, and the associated t-tests (adjusted appropriately for design effects due to weighting and clustering) can be used to assess statistical significance. Effect sizes will also be calculated.

As indicated, the analysis will measure the differences in the mean values of employment and other outcomes for customers with disabilities who use the treatment and comparison LWIAs. Because the availability of the DEI services in pilot LWIAs may influence the customers that choose to use One-Stop services and those who self-identify as having a disability, the difference in mean outcomes reflects both any impact of the DEI services on outcomes for those who would have used the treatment One-Stop centers in the absence of the DEI services as well as the impact of the DEI services on One-Stop utilization by customers with disabilities. Measuring an unconditional impact of the DEI on employment outcomes of customers with disabilities would have required random assignment at the customer level. This option was not feasible because the systemic nature of some components of the DEI (for example, maintaining One-Stop accessibility and the requirement that the One-Stop become an Employment Network). Random assignment at the customer level also would have imposed a much greater burden on the One-Stop centers.

Impacts will also be estimated using regression procedures that control for baseline covariates. This approach will improve the precision of the impact estimates, because the covariates will explain some of the variation in outcomes both within and between sites (thereby reducing the effects of clustering within the LWIAs in the study). Also, covariates can adjust for the presence of any differences between observable baseline characteristics across pilot and comparison sites.

Hierarchical Linear Models (HLM) will include customer-level control variables that pertain to the period prior to random assignment and may be correlated with key outcome measures. These include individual characteristics such as educational attainment, marital status, employment status and earnings at time of registration, employment history variables, race, ethnicity, gender, disability type, severity of impairment, or Ticket assignment. Indicator variables for the random assignment strata will also be included. The analysis will be performed separately for randomly assigned sites in states with a youth focus and in states with an adult focus. In addition, a combined analysis will be performed in which the impact of the DEI is estimated for all randomly assigned sites, regardless of youth or adult focus. Additional detail on the HLM analysis is provided in Section B.

A.16.1.2 Analyses of Qualitative Data

Site visit data will provide a wealth of information on DEI program start-up, implementation and system change. Site visits will be conducted in Spring of 2012 and 2013 in order to collect information on start-up and program implementation, system change and how it affects service availability, accessibility and customer outcomes. The DEI system change framework will focus on the particular strategies and outcomes of each grantee. It will identify the pathways by which change is expected to occur and the role(s) that the DEI is expected to play in producing that change. The DEI system change framework will identify organizational and individual members of each grantee’s workforce development system and its contributions to system change. Follow-up site visits will examine how the selected strategies produced the intended system and customer-level outcomes by linking DEI customer service utilization to services provided by the agencies and organizations that comprise the workforce development system in each state.

The qualitative data sources for this evaluation derive from the in-depth interviews, customer focus groups, document collection, and event/activity observations conducted during the site visits, and will include site visit write-ups, interview and focus group transcripts, and write-ups of site visitor observations of trainings, collaborative meetings, etc., that are related to the site’s implementation of the DEI. We will also analyze information obtained from customers that participate in the focus groups on their disability, employment status/history, education and training, and self-perceived barriers to employment, degree or training completion. In addition, we will perform a comprehensive content analysis of program materials, grant applications, and other relevant documents.

The credibility of an evaluation’s findings ultimately stems from the quality, transparency, and rigor of methods used to collect, analyze and write about the data. In conducting qualitative research, it is critical to adhere to quality standards and to employ tools for rigorous and systematic processes that compare well to the data management processes and the level of analytic transparency found in the quantitative realm. Our approach to the management of qualitative data and to the standards and processes for rigorous qualitative data analysis is based on practical, tried-and-true strategies for ensuring quality in data acquisition, data management, and data analysis, and will ensure that qualitative analyses of program implementation, use of DEI strategies and system change are rigorous and replicable.

We will conduct our qualitative data analysis using principles of Grounded Theory modified to fit the scope and purpose of the study (Glaser & Strauss, 1994). As is the case with most qualitative studies, the data are voluminous and require data reduction. Key to this effort is the development of a coding scheme that is aligned with the conceptual framework of the overall evaluation. We will create a hierarchy of conceptual categories linked to the research questions and system change framework. The categories, or “codes,” at the top of the hierarchy will, at a minimum, align with the six project components required of all nine DEI grants (State Level DEI Project Lead; Disability Resource Coordinator; One-Stop accessibility; participation in Ticket to Work as an EN; sustainability; and support for the DEI evaluation). Also included as “top codes” will be the major strategies selected by the grantees to achieve the objectives of the DEI: integrated resource teams; integrated resources; customized employment; self-employment; Guideposts for Success; asset development; and partnerships and collaboration.

During Year 1, the focus of the coding scheme for the qualitative data analysis will be implementation start-up issues and documentation of program design. In subsequent years, the focus of the coding and analysis effort will shift so that our qualitative analyses will allow us to document the ongoing implementation of the program, refinements or shifts in design, and system change. In particular, the site visit data will be used to document the implementation of the specific DEI program strategies each DEI grantee has selected to implement.

A.16.2 Publication

The evaluation team will provide analyses of program impact and customer outcomes in annual and final reports, as well as descriptions of program start-up, implementation issues and procedures grantees use to resolve program challenges. In particular, the evaluation will look into the impact of the DEI on the employment, employment retention and employment preparation rates of customers with disabilities, as well as the completion of training programs, certifications, high school/GED completion and college degrees. The evaluation also will examine the utilization of One-Stop services by customers with disabilities. The DEI Evaluation will allow DOL to identify best and promising practices and replicate system change activities that successfully lead to improvements in customer outcomes. The study will also provide direction for other states that seek to replicate the DEI. The data resulting from this study is the property of DOL.

DOL will utilize information from the DEI evaluation to inform their strategic planning process, promote best and promising practices and help other states’ workforce development systems become more integrated and efficient, leading to better access and availability of services for customers with disabilities. Any published information will be used to inform stakeholders of the findings and how they can inform workplace development systems and policies. A final report will be made available on DOL’s website. Additionally, materials will be prepared for journals, conferences and associates interested in the subject. Finally, findings will be applied to the development of training modules on workplace development systems.

The timetable for the evaluation’s data collection activities and the production of publications is shown in Figure A-1 (above).

A.17 APPROVAL NOT TO DISPLAY THE EXPIRATION DATE FOR OMB APPROVAL

The DEI evaluation will show the OMB expiration date on any written instrumentation. The following Public Burden Statement will appear:

An agency may not conduct or sponsor and you are not required to respond to, a collection of information unless it displays a currently valid OMB control number. Public reporting burden for this collection of information is estimated to average XXX, including time for reviewing instructions, searching existing data sources, gathering and maintaining data, and completing and reviewing the collection. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to Richard Horne, U.S. Department of Labor, Office of Disability Employment Policy. [email protected] or 202-693-4923.

Focus group participants will be told the following:

This information collection could not take place without OMB authorization obtained through OMB Control Number: XXXX-XXXX

A.18 EXCEPTIONS TO THE CERTIFICATION STATEMENT

There are no exceptions to the certification statement.

References – Part A

Creswell, J. W. (2007). Qualitative inquiry & research design: Choosing among five approaches. Thousand Oaks, CA: Sage Publications.

Department of Labor. (2010). Notice of availability of funds and solicitation for grant applications (SGA) for cooperative agreements under the Disability Employment Initiative. Federal Register, 75(152),

Glaser, B. G., & Strauss, A. L. (1994). The Discovery of Grounded Theory: Strategies for Qualitative Research. Chicago, IL: Aldine Publishing Company, 1967.Kreuger, R.A. Focus Groups: a practical guide for applied research. 2^nd Ed. Thousand Oaks, CA: Sage Publications.

Livermore, G., & Goodman, N. (2009). A review of recent evaluation efforts associated with programs and policies designed to promote the employment of adults with disabilities. Ithaca, New York: Rehabilitation Research and Training Center on Employment Policy for Persons with Disabilities.

Livermore, G., & Coleman, S. (2010). Use of One-Stops by Social Security Disability beneficiaries in four states implementing Disability Program Navigator initiatives. Washington, DC: Mathematica Policy Research.

Office of Management and Budget (1999/2010). Estimating paperwork burden. Retrieved from http://www.whitehouse.gov/omb/fedreg_5cfr1320

Ryu, E., Couper, M, & Marans, R. (2006) Survey incentives: Cash vs. in-kind; Face-to-face vs. mail; Response rate vs. nonresponse error. International Journal of Public Opinion Research, 18 (1): 89-106.

Singer, E., Groves, R., & Corning, A. (1999). Differential incentives: Beliefs about practices, perceptions of equity, and effects on survey participation. Public Opinion Quarterly, 63, 251-260.

Singer, E., Van Hoewyk, J., Gebler, N., Raghunathan, T., & McGonagle, K. (1999). The effect of incentives in telephone and face-to-face surveys. Journal of Official Statistics, 15, 217–230.

United States Department of Labor, Bureau of Labor Statistics. (2011, January). Economic News Release: December 2010 employment status of the civilian population by sex, age, and disability status, not seasonally adjusted. Retrieved from http://www.bls.gov/news.release/empsit.t06.htm

Willimack, D., Schuman, H., Pennell, B., & Lepkowski, J. (1995). Effects of a prepaid non-monetary incentive on response rates and response quality in a face-to-face survey. Public Opinion Quarterly, 59, 78–92.

Zhang, F. (2010). Incentive experiments: NSF experiences. (Working Paper SRS 11-200). Arlington, VA: National Science Foundation, Division of Science Resources Statistics.

APPENDIX 1

SEVEN DEI STRATEGIES

1. Integrated Resource Teams: The IRT is a promising practice identified by the DPN Initiative whereby a team comprised of representatives from different agencies and service systems (both generic and disability specific) coordinate services and leverage funding to meet the employment needs of an individual job seeker with a disability. The job seeker is a member of the IRT working with providers (e.g., interpreter service, community college, etc.) to identify and strategize how their combined services and resources can benefit and support an individual job seeker’s education, training or employment goals. IRTs lead to improved communication and impacted by multiple systems and benefit variables.

2. Integrating Resources and Services: Blending and Braiding Funds, Leveraging Resources, Integrating services and the blending and braiding of funds from multiple funding sources are strategies that are often incorporated into IRT, Guideposts, Vocational Rehabilitation, customized employment, self-employment and other employment models. Leveraging different Federal and state program funds involve two or more agencies contributing to the individual job seeker’s education, training or employment goals. For example, supportive services may be covered by one provider, while training costs are covered by another agency or program. The job seeker with a disability may have multiple challenges that are best addressed through a diversified funding strategy. For the purposes of this SGA, the term ‘‘blended funding’’ is used to describe mechanisms that pool dollars from multiple sources and make them in some ways indistinguishable. ‘‘Braided funding’’ utilizes similar mechanisms, but the funding streams remain visible and are used in common to produce greater strength, efficiency, and/or effectiveness.

3. Customized Employment: Customized employment involves individualizing the relationship between job seekers and employers in ways that meet the needs of both. It is based on an individualized determination and discovery of the strengths, requirements, and interests of a person with multiple challenges. The process is designed to meet the workplace needs of the employer and the discrete tasks of the position identified for the job seeker. When a customized relationship is developed, it results in a shared employment alliance. Customized employment offers the chance for a job to fit the individual, meet individual needs, and match what s/he has to offer. Customized employment provides an avenue to employment for job seekers who feel that traditional job search methods do not meet their needs.

4. Self-Employment: Self-employment has long been an employment alternative for individuals seeking a new or better career. Today, many job seekers with disabilities are turning to the flexibility of self-employment to meet both their career aspirations and financial goals. Self-employed persons have increased latitude in determining the hours they work, the type of work they do, and how much money they make. Self-employment strategies for youth and adults with disabilities are consistent with ETA’s policy guidance in this area.

5. Guideposts for Success: Based on an extensive literature review of research, demonstration projects and effective practices covering a wide range of programs and services, including youth development, quality education, and workforce development programs, ODEP, in collaboration with the National Collaborative on Workforce Disability for Youth, identified Guideposts for Success. The Guideposts reflect what research has identified as key educational and career development interventions that can make a positive difference in the lives of all youth, including youth with disabilities.

6. Asset Development Strategies: Asset development strategies include various approaches to enhance long-term economic self-sufficiency, including use of individual development accounts, implementation of financial literacy training for youth and adults, incorporation of SSA pass plans and other work incentives, utilization of the Earned Income Tax Credit (EITC) and other tax provisions, and self-directed benefit and resource accounts, among others. Asset development strategies include benefits and services that are funded through resources other than those made available under the WIA, such as tax filing assistance, housing, nutrition, health care, or child care assistance. Information on asset development strategies and tax credits, including their relevance for the workforce development system.

7. Partnerships and Collaboration: Applicants must demonstrate that the proposed project will include coordination with a variety of partners that impact the ability of adults and youth with disabilities to successfully participate in education, training and employment opportunities. A description of coordination plans and strategy for partnerships must be provided in the project proposal. Coordination across multiple agencies includes outreach to the partner’s clients/consumers, co-location and integration in One-Stop Career Centers, and leveraging available funds, resources and organizational expertise. Partnering across multiple systems programs is often a pre-requisite to providing employment-related supports that are needed to successfully address multiple challenges to employment. Potential partners include, but are not limited to: State and local Vocational Rehabilitation; Medicaid/Medicare; Medicaid Infrastructure Grant Projects, Mental Health; Developmental Disability/Mental Retardation Agency; State and local Education Departments; SSA programs, such as Work Incentives Planning and Assistance (WIPA) Projects and SSA Area Work Incentive Coordinators (AWICs); Temporary Assistance for Needy Families (TANF); community colleges, colleges, and universities; community services organizations including employment service providers; and national, state and local financial literacy and asset development programs and resources; among many others. Critically important to the Department is the inclusion of DOL programs such as adult, dislocated worker and youth training opportunities, Veterans Employment Training Services (VETS) programs, Job Corps, YouthBuild, ex-offender programs, Senior Community Employment Service Program (SCSEP), registered apprenticeship programs, American Indian and Native American education, post-secondary, vocational training, health and education programs, Migrant and Seasonal Farm Workers Programs, and other relevant DOL Federal/State-grant activities, as fundamental.

APPENDIX 2

SITE VISIT QUESTION MATRIX

ATTACHED

APPENDIX 3

DEI DATA SYSTEM DATA ELEMENTS

ATTACHED

1 DOL is considering funding a 3^rd round of grantees in 2012.

2 These impacts will reflect both impacts for customers with disabilities who would have used the treatment One-Stop centers in the absence of the DEI as well as changes in the composition of the customers served by the treatment One-Stop centers as a consequence of the DEI.

3When only state-wide data were available, counts were scaled down by the fraction of the population who resided in the participating LWIAs (where available) or the fraction of the state’s LWIAs participating in the evaluation.

4 Information on the LWIAS included in each Round 1 and Round 2 Grantees’ selection pool and their random assignment results, is provided in Section B of this OMB clearance package.