EN Report Card Survey Description

DOCUMENTATION FOR THE GENERIC CLEARANCE

OF CUSTOMER SATISFACTION SURVEYS

T ITLE OF INFORMATION COLLECTION: Employment Network Report Card Survey

SSA SUB-NUMBER: I-01

DESCRIPTION OF ACTIVITY (give purpose of activity, provide specific information; i.e., date(s) of survey, number of focus groups, locations, etc.):

Background

Ticket to Work (TTW) is a Social Security Administration (SSA) program that offers adult beneficiaries receiving Social Security Disability Insurance or disability/blindness-related Supplemental Security Income benefits more choices for receiving employment services. Under this program, SSA issues tickets to eligible beneficiaries who, in turn, may choose to assign those tickets to an Employment Network (EN) or to their State Vocational Rehabilitation Agency (SVRA) to obtain employment services, vocational rehabilitation services, or other support services necessary to achieve a vocational (work) goal. The EN or SVRA will coordinate and provide appropriate services to help the beneficiary find and maintain employment.

To ensure that ENs are providing effective and quality service, SSA needs to assess beneficiary satisfaction with EN services. In this survey, SSA will conduct a customer satisfaction assessment of select beneficiaries who are currently participating in the Ticket to Work program and have had their tickets assigned with a specific EN for three months or longer. The beneficiary survey will serve two purposes: 1) it will assess EN satisfaction for the participants and 2) will test a potential vehicle for annually capturing beneficiary satisfaction with EN performance.

Description of Pilot Study

We are planning to survey approximately 1081 beneficiaries located within the California region who assigned their Tickets to the following two ENs (excluding the SVRA acting as an EN) as of December 31, 2008: Service First of Northern California and the Tulare County Office of Education. We will conduct this survey by mail; an explanation of the survey follows.

This proposed survey will be a quantitative survey using the uSPEQ Consumer Experience Survey questionnaire (with customizations made specifically for the SSA Ticket to Work program). Our Ticket to Work Operations Support Manager (contractor), MAXIMUS, will mail the surveys out within weeks of receiving OMB approval.

The survey responses will be strictly voluntary and anonymous, and we will not provide any payments/stipend to participants. The survey responses will be returned to MAXIMUS, who will forward them to the Commission on Accreditation of Rehabilitation Facilities (CARF) International who administers the uSPEQ survey. They will analyze the results and prepare a report for SSA.

uSPEQ^® (pronounced you speak) is a confidential, anonymous, and scientifically-tested consumer reporting system that gives persons served a voice in their services. The uSPEQ Consumer Experience Survey is a subjectively measured, self-administered instrument consisting of a 20-item questionnaire (version 2.0) with five domains or subscales. The primary purpose of uSPEQ is to gather feedback from consumers or persons served regarding their perceptions of the quality of services they are currently receiving or have received in the past.

Providers across the spectrum of health and human services can use the uSPEQ feedback for quality improvement and outcomes management. As a uniform survey tool for all health and human services fields, uSPEQ has the ability to assess performance across diverse populations and settings. This survey tool is designed to provide data for benchmarking and comparative analysis of the consumer experience. The survey instrument was field tested (in 2004 and 2005) to ensure its psychometrical soundness as well as feasibility for data collection in the fields. In 2008, the survey instrument went through another round of testing and refinement. uSPEQ was developed and is administered under the auspices of CARF, a leading international accreditation body for over 40 years. The CARF family of organizations currently accredits more than 5,000 providers at more than 18,000 locations in the United States, Canada, Western Europe, and South America. More than 7.2 million persons of all ages are served annually by providers of CARF-accredited programs and services.

The cover sheet for the survey questionnaire will include the name of the employment network to which the ticket is assigned. In addition, to enhance the analysis, the cover sheet will be coded with the letters “A, B, C” to distinguish three key levels of employment the EN has assisted the beneficiaries in achieving as indicated by payments the agency has made to the EN. The coding is as follows: A = No payment to EN, B = EN has received the first milestone payment and not the second milestone payment, and C = EN has received all 4 milestone payments. A non-coded cover sheet will be used for beneficiaries achieving employment equivalent to that required for the second or third milestone payment but not the fourth milestone payment. We will send the survey questionnaire to a beneficiary with the code appropriate to his or her employment level.

Statistical Information

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Provide, in tabular form, data on 1) the number of entities in the universe covered by the collection, 2) the corresponding sample for the universe as a whole, and 3) each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If you have conducted the collection previously, include the actual response rate achieved during the last collection.

This proposed survey will be a quantitative survey using the uSPEQ® Consumer Experience Survey questionnaire (with customizations made specifically for the SSA Ticket to Work program). The target population is the current SSA beneficiaries from two ENs in California. We will divide a total of 1081 beneficiaries who assigned their Tickets to the two ENs in the region. The beneficiaries will be divided into three groups by their employment levels (see Table below for details).

The cover page of the survey will be coded appropriately before being mailed to each beneficiary. For the scope of this pilot or “Proof of Concept” study, we plan to conduct a census survey, i.e., to send the survey questionnaire by mail to all of the 1081 beneficiaries. With a typical mail survey response rate at 15 to 20%, we expect to receive approximately 200 completed surveys. uSPEQ response rates ranged from 71% in residential settings to 20% for mailed questionnaires, with an average response rate at 60.9%.

Employment level	Number of beneficiaries in target population	Sampling method
A. No employment	749	Census
B. Attained first milestone	103	Census
C. Attained all four milestones	58	Census
Total*	1081	Census

* Includes beneficiaries at other unspecified employment levels.

2. Describe the procedures for the collection of information:

We describe the methodology for conducting this mail survey above. For future surveys, we plan to take a stratified random sampling (SRS) approach for ENs serving over 100 beneficiaries annually. Specifically, a census survey will be conducted for ENs serving 100 or fewer beneficiaries. For larger ENs, we will divide the target population into strata, groups of individuals that are similar in some way that is important to the response (e.g., race, age, education, and or health status). Then a separate SRS is chosen from each stratum to form the full survey sample that is representative of the target population. We expect the results from the current pilot study will help us understand and determine the key demographic subgroups within the target population.

3. Describe methods to maximize response rates and to deal with the issues of non-response.

To maximize the response rate for this survey, we plan to do a 3-wave survey. Wave One consists of an advance letter from MAXIMUS to each of the prospective survey respondents, stressing the importance of completing and returning the upcoming uSPEQ survey. One week later, the survey will be mailed to them with self-addressed and stamped envelopes. Follow-up letters or postcard reminders will be sent out as Wave Three to promote response rate.

Once the survey data are collected from the pilot study, we will analyze the demographic characteristics of the respondents and compare them to those of the target population. Our goal is to secure a survey sample that is representative of the target population.

4. Describe any tests of procedures or methods to be undertaken.

As we described above, the uSPEQ test has been previously assessed.

After all the data are collected, uSPEQ will compile a statistical report for SSA. A standard uSPEQ report consists of the following sections:

• Technical notes about the survey instrument and the response rate for the survey

• Survey results highlights: Top five survey items with positive responses and potential areas for improvement

• Survey results by time period: Frequency distribution by rating category for each survey item and comparison over time if applicable

• Survey results by site: Summary statistics for each survey item by site (EN)

• Demographic characteristics of respondents: For overall and by site (EN)

• Custom items: Summary statistics for each of the custom survey items

• Survey item response summary by rating category and by site (EN)

Post hoc comparisons for statistical significance (e.g., Bonferroni test) will be performed on the key survey items to be determined by the SSA team. For example, are the differences among the employment levels (A, B, and C) statistically significant? Are the differences between the two ENs on some survey items statistically significant? Are the differences among the age groups in their responses to some items significant? Post hoc comparisons will be performed only after obtaining significant differences from one-way Kruskal-Wallis ANOVA (Analysis of Variance).

With a full scale survey for SSA in the future, uSPEQ would prepare an overall report for SSA, aggregating survey data from all the participating ENs all over the country. The overall report would highlight strengths as well as areas for potential improvement for the ENs as a whole. It would also compare the performance of each EN to the aggregates for each survey item by questionnaire domain. In addition, the report would list the best practice facilities by questionnaire domain and for overall satisfaction. Further custom reporting would be available.

5. Provide the name and telephone number of individuals you consulted on statistical aspects of the design. If you are using a contractor who will actually collect and/or analyze the data, provide their name as well.

Di Shen, Ph.D. is the Program Manager for this contract. He is the Chief Research Officer of CARF International. His phone number is: (520) 325-1044 ext. 140; and his email is: [email protected].

The Questionnaire

See the survey questionnaire, attached. Since it may not be obvious why some questions are included in the questionnaire, below are explanations for several questions.

Non-Demographic Section

Questions on Participation:

ENs under the Ticket to Work program assist individuals with disabilities to become self-sufficient through work. Most of the survey items under Participation are related to “outcomes”, i.e., what happened to the person served as a result of the service.  Can they do better in the community?  Also, these questions tell us whether beneficiaries believe their ENs are helping them become more independent in the community.

Questions on Employment:

For individuals with disabilities, one of the primary outcomes is finding employment. These employment questions show whether beneficiaries who are current clients of an EN believe the services the EN is providing are effective in helping them develop appropriate skills, finding work related to the skills they were trained in, and if the jobs are meaningful to them. These items address the key components or outcome of ENs delivering quality services to beneficiaries.

Questions on Assistive Technology:

The intent of this question is to help determine if the ENs are helping beneficiaries gain access to needed assistive technology and to use these devices.

Demographic Section

Demographic Section:

This section describes the demographic characteristics of the beneficiaries who participated in the survey. These items generally help explain response patterns.  For example, an EN may be doing very well with some groups of beneficiaries but not with other groups.  The feedback from the survey will indicate potential areas for improvement for the ENs service to certain demographics. The responses to this question may also help determine if there is bias in responses or if the results can be generalized. In addition, the demographic characteristics of the survey respondents will help determine if the survey result is representative of the target survey population.

Primary Occupational Status:

This question describes the employment situation of the beneficiaries, whether they are employed, have supported employment, etc. Since we are evaluating the EN’s ability to help beneficiaries go to work or increase earnings, this is a relevant question.

Health Status:

The health status question describes the health condition of the beneficiaries. If they report poor health, the EN may refer the beneficiaries to health care services.  It is possible, by analyzing the results of the survey, to find out some beneficiaries were not able to find employment because of their health issues.

Primary means of Communication:

In an effort to provide competent staff to work with the beneficiaries, it is important for ENS to understand how they can communicate most effectively with their beneficiaries. This question helps determine whether ENs have appropriate staff needed to work with their beneficiaries.

IF FOCUS GROUP MEMBERS WILL RECEIVE A PAYMENT, INDICATE AMOUNT:

We will not compensate participants.

USE OF SURVEY RESULTS:

SSA will use this beneficiary survey for two purposes: 1) to assess EN satisfaction for the participants and 2) to test a potential vehicle for annually capturing beneficiary satisfaction with EN performance.

BURDEN HOUR COMPUTATION (Number of responses (X) estimated response time

(/60) = annual burden hours):

Based on 60 percent response rate:

Number of Respondents: 649

Estimated Response Time: 10 minutes

Burden Hours: 108 hours

0960-0526

EN Proof of Concept Survey

6/14/09