Survey Description TTW EN Beneficiary Satisfaction Survey

DOCUMENTATION FOR THE GENERIC CLEARANCE

OF CUSTOMER SATISFACTION SURVEYS

TITLE OF INFORMATION COLLECTION: 2015 Ticket to Work Employment Network (EN) Beneficiary Satisfaction Survey

SSA SUB-NUMBER: B-01

DESCRIPTION OF ACTIVITY (give purpose of activity, provide specific information; i.e., date(s) of survey, number of focus groups, locations, etc.):

Background

Ticket to Work (TTW) is a Social Security Administration (SSA) program that offers adults receiving Social Security disability insurance (SSDI) benefits or Supplemental Security Income (SSI) payments due to disability or blindness choices for receiving employment services. Under this program, SSA issues tickets to eligible recipients, who in turn, may choose to assign those tickets to an employment network (EN) or to their state vocational rehabilitation agency (SVRA) to obtain employment services, vocational rehabilitation services, or other support services necessary to achieve a vocational (work) goal. The EN or SVRA coordinates and provides appropriate services to help recipients find and maintain employment.

The Ticket to Work and Work Incentives Improvement Act of 1999, part of the Social Security Act, mandates a periodic survey to measure the satisfaction of beneficiaries receiving Employment Network services. Using the Beneficiary Satisfaction Survey, Social Security will conduct a customer satisfaction assessment of select recipients from two groups of beneficiaries who are participating in the TTW program. The first group will include beneficiaries who have assigned their tickets with a specific EN for three months or longer. The second group will include beneficiaries who decided to unassign their Ticket from their EN within the past year. The purpose of the survey is to assess participants’ satisfaction with their EN.

To ensure ENs are providing effective and quality service, SSA needs to assess SSDI beneficiaries and SSI recipients’ satisfaction with their ENs. In this survey, SSA proposes to conduct a customer satisfaction assessment of select recipients who are participating in the TTW program. The purpose of the survey is to assess participants’ satisfaction with their EN.

Description of Survey

We are planning to survey approximately 31,200 SSDI and SSI beneficiaries or their representative payees. This sample will include 25,300 beneficiaries or beneficiaries who assigned their Tickets with a specific EN for three months or more between January 1, 2015 and December 31, 2015; as well as 5,900 beneficiaries who decided to unassign their Ticket from their EN between January 1, 2015 and December 31, 2015.

The purpose of the EN Beneficiary Satisfaction Survey is to (1) collect information from beneficiaries who participate in the Ticket program concerning their satisfaction with their ENs; and (2) provide feedback to ENs regarding the strengths and weaknesses of their operation, staff, and the services they provide, as perceived by their consumers. We will provide this information via the program’s website to help beneficiaries make informed choices when they select an EN from which they wish to receive services. The degree of accuracy needed is sufficient to identify meaningful differences among ENs’ ratings. For this purpose, we consider a difference of 10 percent sufficient to potentially affect a consumer’s decision to choose one EN over another.

The 2015 TTW Employment Network Beneficiary Satisfaction Survey is an updated version of a survey of the same population conducted by Social Security in previous years. The survey questionnaire covers a variety of topics related to satisfaction with a current or former EN, areas of strength and weakness of the EN, service expectations, service needs, service fulfillment, the beneficiaries’ current employment situation, and an open-ended item examining areas for program improvement.

Social Security, its contractor, MAXIMUS, and MAXIMUS’ subcontractor, JBS International, Inc. updated and edited this, the 2015 survey, by to improve the validity of the “satisfaction” measurement, to solicit new information from beneficiaries, and to improve usability by participants with a range of disabilities. In order to help ensure the Satisfaction Survey questions will be universal or address accommodation challenges of all TTW beneficiaries, input was obtained from individuals with a range of disabilities during cognitive testing focus groups. Participants provided recommendations on each question’s language, content, and format. Below is a summary of the changes we made to the survey as a result.

• New Question 1: We added a new question to assess whether or not the beneficiary has received Social Security Insurance, Social Security Disability Insurance, Medicaid, or Medicare benefits in the past 12 months. It is possible that receiving such benefits could affect an individual’s relationship with, or dependency on, the TTW program. Results from the survey will explore whether participants’ responses vary by benefit status.

• New Question 2: We added to capture the status of the beneficiary’s relationship with the TTW Program – whether they previously received services from an Employment Network (EN) but are no longer, or if they are currently receiving services.

  • It is important to capture the intricacy of a beneficiary’s relationship with their EN, as it allows analysis of the responses of those who are currently receiving services differently than those who are no longer receiving services.

• We provided the definition of an EN. Many focus group participants had a difficult time understanding exactly what an EN was, and how it related to the TTW Program.

• Question 3: asks beneficiaries why they are not currently using an EN. We worded this similar to a 2014 survey question. A “Not Applicable – I am currently using an EN” option was added, instead of using skip patterns (as was done in 2014), which can be more complicated.

  • The question language was changed from “being with an EN” to “receiving services from an EN” for clarity and to be consistent with Question 2.

• Based on feedback, we made changes to the language used in several response options to increase readability and comprehension (e.g., changing ‘Dissatisfied with my EN’ to ‘Not happy with my EN.’)

• We broke “Employment Network terminated their services or unassigned my Ticket” into two separate response options (and language was changed).

• We added options for “Unable to work because of transportation issues,” “Not happy with the job my EN found for me,” and “EN was unable to find me a job.”

• Based on the input session feedback, participants had a difficult time with the concept of “assigning a Ticket.” Therefore, that phrasing was changed.

• Feedback indicated that the response options of “Do not want to lose my SSI or SSDI benefits” and “Medicaid or Medicare benefits” infer that working with an EN would cause someone to lose such benefits. As this isn’t always the situation (e.g., it can be the fear of losing such benefits that people make decisions based off of), the wording was changed to “Worried about losing my…benefits.”

• For consistency and clarity, we changed the “check all that apply” options to blank lines and put the “select one” options as check boxes throughout the survey.

• Before Question 4: we added instructions which instruct beneficiaries to “Think about the EN you currently use. If you are no longer using one, think about the one you most recently used.” Since participants may have used more than one EN in their past, this standardizes which EN they should be thinking of in answering the questions that follow.

• Question 4 : will have the respondent’s current or most recent EN in a prefilled box. We worded this question similarly to a 2014 item.

• New Question 5: was added to capture the most frequent method (i.e., phone, online, in-person) and direction of communication between the beneficiary and his/her EN.

• Question 6: asks beneficiaries how many ENs they contacted before selecting their current/most recent EN. This is similarly worded to a 2014 survey question.

  • Based on input session feedback, we broke out the numbers (0, 1, 2, 3, 4, 5, 6+) instead of grouping them.

• Question 7: asks beneficiaries why they selected their EN. We worded this question similarly to a 2014 item.

• We reworded some response options to simpler language, based on input session feedback.

• “The only provider nearby/closest provider” option from the 2014 survey version was broken up into two separate options (“Only choice where I live” & “Closest to where I live”).

• Several other response options were added: “Willing to address my employment needs,” “Recommended by a Vocational Rehabilitation Counselor,” “Recommended by another person (Please specify who this person is to you, like a friend or nurse: ___),” “Only EN that returned my call,” and “The EN contacted me.”

• We revised the language from “Only provider willing to accept my ticket” to “Only provider that invited me to join an EN.”

• Questions 8-15: ask the beneficiary how satisfied or dissatisfied they are about a number of statements regarding their EN staff. Questions 8-12 and 15 are similar to the 2014 survey questions.

  • The response options were previously a 5-item Likert-type scale (Completely Satisfied, Somewhat Satisfied, Neither Satisfied nor Dissatisfied, Somewhat Dissatisfied, and Completely Dissatisfied). We changed this to a 3-item Likert-type scale (Satisfied, Neither Satisfied nor Dissatisfied, and Dissatisfied) scale. This was based on feedback from input session respondents who were overwhelmed by the 5-item option and had difficulty understanding the intricacies of what was ‘Completely’ vs. ‘Somewhat.’

• Questions 13 and 14 are new questions: “I am ____ with my EN’s ability to help me reach my financial goals,” and “I am ____ with my EN’s understanding about the types of jobs I want.”

• The wording of Question 9 was changed to increase readability. For instance, instead of “The respectfulness of the staff at my EN” (2014 survey), the 2015 version reads “I am _____ with how my EN respected me.” While ‘respectfulness’ is an abstract concept,’ beneficiaries are likely to better understand whether or not they were respected.

• Questions 16-17: ask how much the beneficiary agrees, disagrees, or is undecided, with two statements about using the money the beneficiary earns from working to replace Social Security cash benefits. This is very similar to a 2014 survey question.

  • In the 2014 survey question, the wording was, “It is my goal to substitute or reduce my need for…” This was broken up into two questions in the 2015 version – “It is my goal to replace all of my benefits” and “It is my goal to replace some of my benefits.”

• In 2014 survey, responses were a 5-item Likert scale (Strongly Agree, Somewhat Agree, Neither Agree nor Disagree, Somewhat Disagree, Strongly Agree). Similar to Questions 8-15, these were simplified to a 3-item Likert scale (Agree, Neither Agree nor Disagree, Disagree).

• Question 18: asks what was good or bad about the beneficiary’s EN. We worded this similar to a 2014 survey question.

  • In the 2014 survey version, the question asked which “Areas I liked” and “Areas I think need to improve.” However, based on input session feedback that this question was confusing, the options were changed to “Good,” “Fair,” “Bad,” and “Not Applicable.” “Not Applicable” was added as an option because some respondents may not have experience with a particular characteristic, and therefore don’t feel comfortable rating their satisfaction with it.

• The simplified language was based on feedback (e.g., “Staff responsiveness in returning phone calls and e-mails” was changed to “How quickly staff returned phone calls/e-mails.”

• Addition of options were added: “Physical accessibility, such as wheelchair ramps” and “Staff Knowledge.”

• “Respectfulness of staff services” was removed from the 2014 survey version because it was asked in Question 8 and “Quality of services” was removed because it was too general.

• Question 19: lists several services that an EN could provide, and asks for respondents to identify which services they wanted to get, which they actually got, and which they hope to get in the future. This is very similar to a 2014 survey question.

  • Based on input session feedback that the column titles were confusing and not clear, they were renamed. “Service you and your EN agreed were needed” became “Services you wanted to get.” “Services you actually received” became “Services you actually got.” And “Services you expect to receive in the future” became “Services you hope to get in the future.”

• For clarity, the columns were also labeled 1, 2, and 3, and were described more in detail in the instructions prior to the table.

• Based on feedback, a definition was provided for job planning and job coaching (participants were confused about the difference between the two).

• We removed, “Ongoing support to keep a job”, because it is part of job coaching.

• We revised “Benefits counseling,” to provide an explanation, as it could refer to the EN’s benefits, job benefits, etc. to “Help in understanding how having a job could affect my Social Security and/or Medicaid/Medicare benefits.”

• We added four items to capture additional services provided by ENs: “Help building my resume,” “Help learning a skill,” “Help getting accommodations” and “Job interview preparation”.

• Question 20: asks whether the beneficiary’s EN provided services that helped the beneficiary get their current job. We worded this similar to a 2014 survey question.

  • An option was added for “Not Applicable – I am not currently working,” as opposed to the 2014 survey version, which uses skip patterns.

• New Question 21: asks whether the beneficiary’s EN provided services that helped the beneficiary keep their current job. This question was added to capture situations where the respondent may have already had a job and the EN was helping them keep it, or if the EN helped them get and keep a job.

• Question 22: asks since the beneficiary started with the TTW program, has the beneficiary regularly worked 30 hours or more/week? We worded this similar to a 2014 survey question.

  • We revised the language “assigned a ticket” to “since choosing your EN” for clarity purpose.

• Question 23: asks when the beneficiary began working at their current job. We worded this similar to a 2014 survey question.

• We added option for “Not Applicable – I am not currently working,” as opposed to the 2014 survey version, which uses skip patterns.

• Question 24: asks how many hours the beneficiary usually works per week at their current job. A 2014 survey question asked whether the respondent worked full-time (30 hours/week or more), part time (less than 30 hours/week) or were unemployed. This revised question (along with Question 25) will provide SSA with a more accurate measure of how many hours beneficiaries are working and how much beneficiaries are making on a weekly, monthly, and yearly basis.

• Question 25: asks for the beneficiary’s current hourly wage. In the 2014 survey, a similar question asked for an annual salary. However, input session feedback indicated that most participants knew what they earned in terms of an hourly pay, as opposed to an annual salary.

  • Also added an option for “Not Applicable – I am not currently working,” as opposed to the 2014 survey version, which uses skip patterns.

• The 2014 survey version also had respondents check which of several provided groups of salary amounts (e.g., “$20,000 - $39,999) they fell into. To allow SSA to calculate a more accurate annual salary and understand exactly where the beneficiary lies in relation to Substantial Gainful Activity (SGA) standards, this question was changed to a fill-in-the-blank, where the respondent writes in a specific dollar amount that they make per hour.

• Prior to Question 26: there are instructions to skip to Section D if the respondent works at least 30 hours a week. Thus, questions 26-28 are to only be answered by beneficiaries who work part-time or are unemployed.

• Based on feedback we removed the term “full-time” and replaced with “working at least 30 hours a week” and the phrase “part-time” was changed to “working less than 30 hours a week”. The phrase “full-time” made certain respondents think of a 40 hour work week, and therefore SSA’s definition of full-time (> 30 hours) would have to be written at every instance where “full-time” was written. To reduce redundancy, we removed all instances of “full-time” and “part-time” and replaced with the specific weekly hours.

• Question 26: asks why the beneficiary is working part time. We worded this similar to a 2014 survey question.

  • Based on feedback, we revised the language in specific response options to increase participant comprehension (e.g., “health or disability considerations” to “health or disability reasons”).

• We added three additional response options to capture other reasons why a beneficiary would only be working part-time: “Physical accessibility reason,” “Problems with a supervisor or coworker,” Lack of reliable transportation.”

• We removed “Personal choice”, because it was too broad and respondents can provide a more specific answer in the ‘Other’ response option.

• Based on feedback that some respondents might not know what “laid off” meant, “Laid off by employer” was changed to “Terminated or laid off from previous job”

• New Question 27: asks if the respondent had ever worked 30 hours a week or more. This allows SSA to understand the employment history of the beneficiary.

• New Question 28: asks if the beneficiary is interested in working full time. This allows SSA to understand whether working full time is something the beneficiary is actually interested in.

• Question 29: is an open-ended item that seeks information from the beneficiary about comments/suggestions to improve the TTW program. We worded this question similarly to a 2014 item.

• New Questions 30 and 31: will help SSA determine whether TTW participants are aware of TTW terminology.

The data collection process will be mixed-mode. Prior to data collection commencing, beneficiary contact for the survey will be initiated by sending a message via social media (e.g., Facebook, Twitter), which will inform participants that postcards with information for taking the survey electronically will be arriving in their mail soon, as well as hard copies of the survey soon thereafter (Appendices A & B, Volume II). Follow-up social media blasts will act as reminders to beneficiaries to take they survey sent to them (Appendices M & N, Volume II). When data collection begins, first, we will mail a postcard (Appendices C & D, Volume II) in an enclosed envelope to the entire sample. The postcard will contain the web address and an individualized password to the web version of the survey. All beneficiaries invited who do not complete the survey online within 10 days of receiving the initial postcard mailing will be mailed a paper copy (e.g., self-administered) of the survey. The survey mailing will include an introductory letter (Appendices E &F, Volume II), which will communicate the objectives of the study and invite the individual to either complete the survey online or complete the paper survey enclosed in the mailing and return it in the postage-paid envelope included in the mailing. Approximately two weeks later, those invited beneficiaries who do not respond to the pre-survey postcard or to the first mailing of the survey will receive a second survey mailing, containing a follow-up letter (Appendices K & L, Volume II). We will mail surveys in the language of preference (Spanish or English) on record with Social Security.

Beneficiaries from ENs with close to 10 complete responses (threshold for individual EN-level reporting), who did not respond to the initial or second mailing of the survey, will be selected for phone call reminders. The anticipated size of the telephone follow-up reminder call sample is approximately 1,000 non-respondents. The phone contacts will primarily be used to encourage completion of the paper or web-based surveys, but interviewers will administer the survey telephonically, if a respondent prefers. A telephone script (Appendices G & H, Volume II) will be used. Up to three attempts will be made to contact those beneficiaries. Again, as noted above, this directed follow-up effort to beneficiaries in ENs which are close to the reporting threshold of 10 completed cases, is being conducted with the sole purpose of increasing the number of “reportable” ENs.

JBS International, Inc., the Ticket Program Manager’s (MAXIMUS Federal) subcontractor, will initiate the data collection within one month of receiving OMB approval.

The survey responses will be strictly voluntary. We will not provide any payments or stipends to participants.

USE OF SURVEY RESULTS:

Social Security will use the results of the study to monitor and evaluate beneficiaries’ satisfaction with their EN’s performance. Where sufficient data from the survey is available, Social Security will also incorporate customer satisfaction information derived from the survey into the EN Profiles. The EN Profiles are online resources used by prospective and current TTW participants to provide them with general information about specific ENs.

BURDEN HOUR COMPUTATION (Number of responses (X) estimated response time

(/60) = annual burden hours):

Number of Responses: 31,200

Estimated Response Time: 15 minutes

Annual Burden hours: 7,800

NAME OF CONTACT PERSON: Renee Clarke, COR

Quantitative Surveys:

For quantitative surveys, you will need to complete the questions below.

A. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Provide, in tabular form, data on 1) the number of entities in the universe covered by the collection, 2) the corresponding sample for the universe as a whole, and 3) each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If you have conducted the collection previously, include the actual response rate achieved during the last collection.

Sampling Plan:

We are planning to survey two groups within population of SSDI and SSI beneficiaries or their representative payees. Similar to the 2014 survey, we plan to survey approximately 25,300 SSDI and SSI beneficiaries or their representative payees who assigned their tickets to ENs as of December 31, 2015. In addition, we are planning to survey approximately 5,900 SSDI and SSI beneficiaries or their representative payees who unassigned their ticket from an EN between December 31, 2014 and December 31, 2015.

We will apply two sampling approaches to select samples from these two groups: a stratified random sampling of Ticket-assigned beneficiaries and random sampling of Ticket-unassigned beneficiaries.

The sampling universe for Ticket-assigned beneficiaries includes all SSDI and SSI beneficiaries served by ENs under the TTW program for three months or more between January 1, 2015 and December 31, 2015. A mixed sampling method is proposed; we will survey (1) beneficiaries served by all ENs meeting the criteria below that have 500 or fewer beneficiaries, and (2) a randomly selected 1,400 beneficiaries from the 14 ENs that have more than 500 beneficiaries assigned to them.

	N (approximate)
Number of (Ticket-assigned) TTW beneficiaries	53,699
Sample	25,300
Strata	By EN size. We will divide the sample into 3 strata based on grouping ENs by number of beneficiaries with assigned tickets. The first strata will consist of 84 large ENs defined as those with 101 beneficiaries or more (these ENs work with 41,614 beneficiaries in total) . The second strata, 178 mid-sized ENs, are defined as having between 26 and 100 beneficiaries (these ENs work with 9,213beneficiaries in total). The third strata, 139 small ENs, consist of those having between 10 and 25 beneficiaries (these ENs work with 2,292 beneficiaries in total). At the strata-level, we will sample approximately 13,800 of beneficiaries from the large ENs to ensure that we are able to report on the maximum number of ENs for which the reporting threshold of 10 completed surveys can be met. Approximately 9,200 and 2,300 beneficiaries, respectively, will then be selected from the second and third strata (mid-sized and small ENs). This will allow a sufficient number of completed cases to make-group level (i.e., sized-based comparisons) in levels of satisfaction across large, mid-sized and small ENs.

The expected response rate is 28%, which is based on the number of completed surveys during 2014 data collection. While the research plan includes some elements shown to increase return rate, i.e., multiple formats for completion, advance notice, follow-up (mail and phone) and postage-paid, addressed envelopes; it does not include any incentives for respondent participation.

A simple random sampling technique will be used to obtain a representative sample of beneficiaries who chose to unassign their Ticket in 2015.

	N (approximate)
Number of (Ticket-unassigned)TTW beneficiaries	9,122
Sample	5,900

Based on results of the 2014 survey of the expected response rate among beneficiaries who unassigned their Ticket is 23 %. To perform needed statistical analyses related to the unassigned Ticket population and to collect 1,000 completed survey responses, a sample size of 5,900 unassigned beneficiaries is required.

B. Describe the procedures for the collection of information:

The collection uses a mixed-mode, multi-stage approach to data collection that draws on the Dillman Tailored Design Method (Dillman, D. A., Smyth, J. D., & Christian, L. M., 2014). Beneficiary contact for the survey will be initiated by sending a message via social media (i.e., Facebook, Twitter), which will inform participants that postcards with information for taking the survey electronically will be arriving in their mail soon, as well as hard copies of the survey soon thereafter. Follow-up social media blasts will act as reminders to beneficiaries to take the survey that we sent to them. Next, beneficiaries will receive an initial pre-notification postcard contained within a sealed envelope to ensure that the information contained on the post-card (e.g., beneficiary’s EN name, PIN and link to an online survey administration option) is maintained confidential for the intended recipient. All invited beneficiaries who do not complete the survey online within 10 days of receiving the advance postcard will be mailed the paper (self-administered) survey. The self-administered survey mailing will include an introductory letter, which will communicate the objectives of the study and invite the individual to either complete the survey enclosed in the mailing and return the paper survey in a postage-paid envelope that will be provided or, to complete the survey online. Those invited respondents who do not respond to the pre-survey postcard or to the first mailing of the survey will receive a second survey mailing identical to the first. The second survey mailing will be sent approximately two weeks after the initial mailing.

A sub-sample of the larger non-respondent population will receive follow-up telephone call . This effort will assist in increasing the total number of reportable ENs (i.e., those with a minimum of 10 responses) for inclusion in the EN Profile, an online resource guide that contains performance-related data about all ENs.

C. Describe methods to maximize response rates and to deal with the issues of non-response.

There are multiple efforts used to maximize response rates. First, we will post social media notices to alert beneficiaries of the upcoming postcards. Then, we will mail potential recipients a pre-notification postcard in an enclosed envelope explaining the purpose of the survey and encouraging them to participate online. To those who did not complete the survey online, we will also mail a paper version of the survey with an addressed, postage-paid envelope, as well as the URL and PIN for an online response option for the survey.

Approximately two weeks after the original mailing, we will mail non-respondents a reminder letter, another copy of the survey, a postage-paid envelope, and a URL and PIN for optional online completion. We will mail surveys in the language of preference (Spanish or English) on record with Social Security.

In addition, to boost response rates, we will follow-up the second mailing of the self-administered survey to non-responders with a telephone reminder/outreach effort. The phone contacts will primarily be used to encourage completion of the paper or web-based surveys, but respondents will be provided the option to complete the survey by phone. Up to three telephone attempts will be made to contact beneficiaries.

There are no unusual problems requiring specialized sampling procedures. The Ticket to Work and Work Incentives Improvement Act of 1999, part of the Social Security Act, mandates a periodic survey to measure the satisfaction of beneficiaries receiving Employment Network services.

D. Describe any tests of procedures or methods to be undertaken. When possible, OMB encourages testing of procedures as an effective means of refining collections of information to minimize burden and improve utility. However, this is not always necessary.

JBS International, Inc., a Social Security subcontractor, has revised last year’s survey to better address beneficiaries’ satisfaction with ENs. The current survey has been reviewed by individuals with expertise with the TTW program, including contractor staff who are supporting the TTW program, and Social Security staff. Prior to deployment of the survey, JBS International conducted individual input sessions with individuals with disabilities, as well as a cognitive in-person focus group testing with individuals familiar with the TTW program, to pre-test the survey and make any revisions to the survey instrument.

E. Provide the name and telephone number of individuals you consulted on statistical aspects of the design. If you are using a contractor who will actually collect and/or analyze the data, provide their name as well.

Consulted on statistical aspects of the design and to collect and analyze the data:

Eileen Elias, ME.d. (240) 380-0431

Manu Singh, Ph.D. (301) 323-8324

Erika Tait Olson (240) 645-4854

JBS International, Inc. (Subcontractor to MAXIMUS)

REFERENCES:

Dillman, D. A., Smyth, J. D., & Christian, L. M. (2014).  Internet, Phone, Mail

and Mixed-Mode Surveys: The Tailored Design Method (4th ed.). New York, NY: John Wiley and Sons.

11