1230-0NEW CUSSAT Supporting Statement Part A_final_updated

A Study of Customer Satisfaction with Office of Disability Employment Policy (ODEP) Technical Assistance (TA) Centers

OMB No. 1230-0NEW

April 2018

A Study of Customer Satisfaction with ODEP Technical Assistance (TA) Centers

Supporting Material for OMB Clearance

Part A. Justification

1. Circumstances Making the Collection of Information Necessary

The Department of Labor’s (DOL) Office of Disability Employment Policy (ODEP) established five Technical Assistance (TA) Centers to serve a diverse set of purposes, functions, and customers. Operating with grants funded by ODEP, these Centers assist employers, federal agencies, state governments, non-profits, individuals with disabilities, and others with technical assistance and policy development concerning the integration of people with disabilities into employment. Currently, the five grantees operating ODEP’s TA Centers are:

• The Viscardi Center, which supports the Employer Assistance Resource Network (EARN);

• The Institute of Educational Leadership, which houses the National Collaborative on Workforce and Disability for Youth (NCWD/Y);

• West Virginia University, which maintains the Job Accommodation Network (JAN);

• Rehabilitation Engineering and Assistive Technology Society of North America, which supports the Partnership on Employment and Accessible Technology (PEAT); and

• The National Disability Institute, which houses the National Center on Leadership for the Employment and Economic Advancement of People with Disabilities (LEAD).

DOL’s Chief Evaluation Office (CEO) in partnership with ODEP will methodically study the level of customer satisfaction with the TA Centers. CEO undertakes a learning agenda process each year to identify Departmental priorities for program evaluations. This evaluation was prioritized as part of that process in FY2014. Division H, Title I, section 107 of Public Law 114-113, the “Consolidated Appropriations Act, 2016” authorizes the Secretary of Labor to reserve not more than 0.75 percent from special budget accounts for transfer to and use by the Department’s Chief Evaluation Office (CEO) for departmental program evaluation. Further, 29 USC 3224a (1), authorizes the Secretary of Labor to conduct ongoing evaluation of programs and activities to improve the management and effectiveness of these programs. CEO contracted with Westat to conduct this five-year study. The overarching objectives are to determine the extent to which customers are satisfied with the TA provided by the Centers and to document the processes and methods used by the TA Centers to encourage the adoption and implementation of ODEP’s policies and practices by targeted and untargeted customers. DOL requests clearance to collect the information needed to meet the study’s objectives.

2. Purpose and Use of the Information Collection

The information collection will answer research questions regarding the quality and utility of the services the TA Centers provide, and the degree to which Center programs and services have led to use of ODEP’s recommended policies and practices. To do so, DOL will collect data annually from a sample of TA Center customers for three years to assess customer satisfaction and document the processes by which ODEP policies and practices are delivered to and adopted by TA Center customers. The data collection activities are described below.

• Pulse Survey. The study team will invite customers who request TA from the Centers to complete a web-based Pulse Survey approximately 48 hours after initiating such requests. This brief survey (no more than 10 questions) will collect customers’ initial impressions of responsiveness, clarity, relevance, and perceived value. Because the sampling unit for the Pulse Survey is a TA event, this design will allow us to collect comparable data on customer satisfaction across all Centers from the greatest cross-section of customers and the greatest range of TA.

• In-Depth Survey. The study team will administer an annual web-based In-Depth Survey to a subsample of customers receiving TA and policy dissemination of special interest to DOL. They will subsample customers receiving intensive TA provided over time, customers participating in networking and collaborative activities; and frequent customers. Whereas the Pulse Survey will ask about customer satisfaction with specific TA events, the In-Depth Survey will assess overall satisfaction with the Centers and the extent of policy dissemination.

• Qualitative Interviews. Based on responses to the In-Depth Survey, the study team will select a small subsample of customers to complete qualitative telephone interviews that will allow us to drill down further in our examination of satisfaction with and utilization of TA services and policy dissemination. Because they plan to select purposively three types of customers for these interviews – employers, government agencies, and community based organizations – we will be able to assess the utility and implementation of TA and policy dissemination within specific settings of different organizations. Furthermore, to explore how TA and policy dissemination can be improved in the future, the study team will select customers reporting both high and low levels of customer satisfaction.

In addition, the study team will conduct additional rounds of interviews with Center directors and staff to get their perspective on issues associated with perceived adoption and implementation of ODEP’s policies and practices. Furthermore, this study will collect relevant administrative data from each of the Centers such as the number of monthly hits on the Center’s website and the number of direct requests for consultation services.

The study team will conduct both qualitative and quantitative analyses of the data collected to answer the research questions. The two customer satisfaction surveys will provide data on overall satisfaction with the TA services, perceived quality of services, customer expectations, as well as information on perceived adoption and implementation. They will examine these measures in the aggregate, as well as explore subgroup differences (e.g., by type of user, type of service, etc.). The qualitative telephone interviews will provide richer data on two important dimensions of the customer experience: 1) satisfaction with the quality, relevance and usefulness of the received TA to accomplish Center objectives and 2) the extent to which the TA received results in perceived adoption and implementation of ODEP policies.

3. Use of Information Technology

The TA Centers currently have varying types and amounts of customer information. Most of the Centers have some kind of customer database, but they are not always comprehensive with some databases containing more information about customers than others. Successful implementation of the surveys hinges on development of a complete sample frame of all customers. Furthermore, a robust system containing comprehensive customer information can enhance Centers’ knowledge of their audience.

To identify and gain access to all types of customers, the study team will develop a customer database that each TA Center will use to record prospectively basic information on all customers to the greatest extent possible. The study team will embed sampling capability within the database such that the system will appropriately administer the data collection instruments to sampled customers upon OMB approval.

In addition to serving as the sampling frame for this study, the database will also create an enduring mechanism that TA Centers can use in the future to report out to ODEP, in a standardized fashion, how many and what types of customers are being served during any given period. Secondary advantages of this system include ongoing maintenance of comparable data across Centers; ease of data reporting to ODEP; and a standardized organizational method for all incoming, future grantees.

The study management system housing the database will deploy both the Pulse Survey and In-Depth Survey to sampled customers using web-based technology. We anticipate that the online survey will be less burdensome to respondents, as it will offer easy access and submission, while also allowing respondents to complete the survey at a time convenient to them and at their own pace. An online survey has the additional advantages of reducing the potential for errors by checking for logical consistency across answers, accepting only valid responses and enforcing automated skip patterns.

4. Efforts to Identify Duplication

The proposed data collection will allow DOL to assess customers’ perceptions of the quality and utility of the services the TA Centers provide, and the degree to which customers perceive that Center programs and services have led to use of ODEP’s recommended policies and practices. Although some of the TA Centers engage in activities to assess customer satisfaction with specific aspects of the TA they provide, there is currently no comparable data available related to customer satisfaction for the TA Centers. While these specific data related to an individual TA Center are important and useful data sources, they do not provide comparable information that DOL can use to assess customer across the TA Centers. In addition, DOL is not aware of any previous or planned effort to collect information concerning the extent to which customers perceive that Center programs and services have led to use of ODEP’s recommended policies and practices. Hence, the proposed data collection is needed to gather the information necessary to address the study research questions.

5. Impact on Small Businesses or Other Small Entities

The TA Centers serve a variety of customer types, including employers that may be classified as small businesses. Hence, it is likely that such employers will be sampled for both survey activities. However, only the employee initiating contact with the TA Center will be invited to complete the survey. Furthermore, the estimated length of the Pulse Survey will be no more than 5 minutes, and the In-Depth Survey will average no more than 15 minutes. Because the number of employees within an organization that we will invite to complete these brief surveys is relatively small, we expect the impact on small business to be minimal.

6. Consequence to Federal Program or Policy Activity if Collection is not Conducted

If the proposed data collection activities are not conducted, detailed information to answer the research questions otherwise would not be available.

7. Special Data Collection Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances that would require this information collection to be conducted in any manner other than that described above. The data collection will be performed in a manner consistent with federal guidelines.

8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

The notice of the proposed information collection activity was published in the Federal Register on 82 FR 27080 on 6/13/2017. There were no public comments.

The following individuals from outside the agency were consulted to provide input and recommendations on the study design and data collection methods: Dr. Brenda Turnbull, Senior Researcher at Policy Studies Associates, Inc.; Professor John Kregel of Virginia Commonwealth University; and Ms. Judy Young, Associate Director of Training and Development at Cornell University Industrial and Labor Relations School.

9. Explanation of Any Payment or Gift to Respondents

Data collection activities will not include any incentive payments or gifts to respondents.

10. Assurance of Privacy Provided to Respondents

Respondents will be informed that their responses will be kept private to the extent permitted by law. All participants will be informed that the information collected will be reported in aggregate form only and no reported information will identify any individuals. Terms of the DOL contract authorizing data collection require the contractor to maintain the privacy of all information collected, unless written permission is provided by the respondent. The study team will protect personal information in accordance with Federal and state laws and contractual requirements.

The security procedures the study team will implement cover all aspects of data handling for hard and electronic data. When not in use, all hardcopy documents (e.g., interview notes or transcribed interviews) will be stored in locked file cabinets or locked storage rooms. Unless otherwise required by DOL, these documents will be destroyed when no longer needed for the project.

Qualitative interviews will be recorded, subject to respondent approval. Interviewers will ensure a private meeting space for any in-person interviews. Written materials and analyses from the interviews used as part of study reports will be prepared in such a way as to protect the identity of individuals. Only the project staff present at the interviews, the principal investigator, project director, and selected staff helping transcribe the recordings will have access to the notes. Notes will be securely stored in protected electronic files or locked cabinets. Only the staff members present at the interviews or transcribing the recordings will have access to the recordings.

Access to the web surveys will require a unique PIN provided to the respondent. Survey data collection will use secure sockets layer encryption technology to ensure that information is secure and protected.

11. Justification for Sensitive Questions

There are no questions in the interviews or surveys that are of a sensitive nature.

12. Estimates of Annualized Burden Hours and Costs

Table A-1 provides the number of respondents, frequency of responses, and the burden hours for the data collection.

Table A-1. Estimated Annualized Respondent Hour and Cost Burdens

Data Collection Activity	Total Number of Respondents	Annual Number of Respondents	Number of Responses per Respondent	Total Annual Responses	Average Burden Hours per Response (In Hours)	Total Annual Burden Hours	Total Burden Hours	Average Hourly Rate1	Annual Cost Burden
Pulse Survey	33,900	11,300	1	11,300	5/60	942	2,825	$23.86	$22,476
In-Depth Survey	9,855	3,285	1	3,285	15/60	821	2,464	$23.86	$19,589
Qualitative Interviews with Customers	72	24	1	24	1	24	72	$23.86	$573
Qualitative Interviews with Center staff	30	10	1	10	1	10	30	$56.74	$567
TOTAL	43,857	14,619	--	14,619	--	1,793	5,391	--	$43,205

13. Estimates of Other Total Annual Cost Burden to Respondents and Record Keepers

There are no additional cost to the respondents other than their time.

14. Annualized Cost to the Federal Government

1. The estimated cost to the federal government for the contractor to carry out this study is $1,356,081. The estimated cost was calculated by estimating labor hours, operational expenses, and additional costs associated with the design of study’s qualitative protocols; sample development; development and implementation of the pulse and in-depth surveys; development and maintenance of the study database; quantitative and qualitative analysis; and the preparation of interim and final reports of evaluation findings. The data collection period covered by this justification is three years, so annualized, the estimated cost to the federal government for the contractor to carry out this study is $452,027 ($1,356,081/ 3= $452,027).

2. In addition, DOL expects the annual level of effort for Federal government technical staff to oversee the contract will require 200 hours for one Washington D.C.-based GS-14, Step 4 employee earning $59.04 per hour. To account for fringe benefits and other overhead costs the agency applies a multiplication factor of 1.6. The annual cost for Federal government technical staff to oversee the contract is $18,893 ($59.04 x 1.6 x 200 = $18,893).

The total annualized cost is $470,920 ($452,027 + $18,893= $470,920).

15. Explanation for Program Changes or Adjustments

This is a new data collection.

16. Plans for Tabulation and Publication and Project Time Schedule

To address the research questions, the study team will use a combination of quantitative analysis (e.g., descriptive statistics of survey data, significance testing of differences in estimates between subgroups, regression analysis) content analysis of the qualitative data, and a review of administrative documents.

Summary of Survey Results. The study team will administer the customer satisfaction survey annually for three years. Upon the completion of each annual survey, the study team will prepare a comprehensive summary of the results. These annual summaries will include an explanation of the methodology used as well as descriptive statistics.

Interim and Final Reports. The interim and final reports will provide key documentation of the data collection activities and findings. The reports will include an overview of the study, a description of data collection, a summary of the data used in the analyses and methodology, major findings, and implications of the findings. In addition, both the interim and final reports will also include a 150-word abstract. The study team will prepare a detailed outline of the interim report in Year 3, based on the findings from the data collected up to that point. The study team will prepare a final draft report describing the overall study findings and methodology. The final report will build upon the interim report, providing updated findings based on new data and comparisons over time. The process for the final report will be similar to that for the interim report with a detailed outline submitted by month 55 (Year 5), a draft report submitted by month 57, and the revised final version submitted by month 59 (at least 30 days prior to contract end).

17. Reason(s) Display of OMB Expiration Date is Inappropriate

It is appropriate for the proposed data collection materials to display the OMB expiration date.

18. Exceptions to Certification for Paperwork Reduction Act Submissions

We are not requesting any exceptions to certification under the Paperwork Reduction Act (PRA). This package is our submission to comply with the PRA.

1 The average hourly wage rates were obtained using the latest Occupational Employment Statistics data on wages, adjusted for inflation. Bureau of Labor Statistics, “Occupational Employment Statistics—May 2016 National Occupational Employment and Wage Estimates” (https://www.bls.gov/oes/current/oes_nat.htm#00-0000)