Supporting Statement B - 0960-NEW (TTW Evaluation)

Supporting Statement B for Ticket to Work Program Evaluation

OMB No. 0960-New

B: Collections of Information Employing Statistical Methods

In compliance with the Ticket to Work and Work Incentives Improvement Act of 1999 (P.L. 106-170, Ticket Act), SSA is undertaking an evaluation of the programs authorized by the Ticket Act, including the TTW program offered by Employment Networks (ENs) and Vocational Rehabilitation (VR) agencies, the Work Incentives Planning and Assistance (WIPA) program offered by WIPA projects, and the Protection & Advocacy for Beneficiaries of Social Security (PABSS) program offered by Protection & Advocacy (P&A) agencies. The Ticket Act requires SSA to provide for ongoing, independent evaluation to assess: (1) the effects of the program on work outcomes and self-sufficiency, and (2) their cost effectiveness (Section 101(d)(4)(A)). In compliance with P.L. 106-170, SSA is undertaking this evaluation to assess the effectiveness of the programs the Ticket Act authorized, both in terms of program outcomes and cost (efficiency). On September 29, 2023, SSA awarded a contract to Mathematica, a research organization, to conduct the independent evaluation.

The evaluation will provide SSA with evidence about Ticket Act program effectiveness, along with the factors that drive the effectiveness of the programs including provider participation, service availability, Ticketholder awareness, service access, and service use. The structure of the analytic approach is designed to help SSA understand program effectiveness, as well as the potential reasons why the programs do or do not achieve their legislative intent: to allow individuals with disabilities to seek the services necessary to obtain and retain employment and reduce their dependency on cash benefit programs. The evaluation will provide SSA with information about the program’s effectiveness as well as actionable information that SSA can use to promote programmatic improvements. The evaluation will also document the cost effectiveness of Ticket Act programs as currently structured, allowing SSA to identify opportunities to deliver the same outcomes at lower costs and/or improve outcomes with additional investments.

SSA requests clearance for the following data collection efforts to support the evaluation:

1. Surveys of the Ticket Act service providers (“provider surveys”).

Mathematica will field three concurrent surveys, each focusing on a specific type of Ticket Act service provider. Specifically, Mathematica will field one survey to every provider. As of 2024, this population consists of 441 ENs and VR agencies, 74 WIPA projects, and 57 P&A agencies. Mathematica will invite one person from each of the 572 organizations to respond as a representative on behalf of the organization. Each organization’s representative will complete a self-administered online survey. The surveys will ask about program operations and effectiveness, characteristics of the people receiving services, service provision, and opportunities for program improvements.

Qualitative interviews with Ticketholders (“qualitative data collection”). These interviews will provide a platform for open-ended guided discussions during which interviewees can share their experiences with the Ticket Act programs. The interviewees will include TTW participants (who assigned their Ticket to an EN or used it with VR agency) and non-participants (Ticketholders who are working but did not participate in the TTW program) as well as WIPA service users and non-users (Ticketholders who did not use WIPA services). Mathematica will use an interview guide to structure the discussions. Attachment B contains the outreach materials, screening questions, and interview topics.

We included more information about the timing, burden, and cost of these data collection activities in Part A of this package.

1. Statistical Methodology

Provider surveys

Mathematica will implement three surveys of Ticket Act service providers nationwide. Mathematica does not plan to employ sampling for these surveys but instead field the survey to the 572 organizations that provide TTW, WIPA, and PABSS services in the United States as of 2024¹. Mathematica will invite one person from each of the 572 organizations to respond as a representative on behalf of the organization, Mathematica will field the provider surveys over an eight-week period beginning within three months of receiving PRA clearance. The three surveys all address the same core topics, specifically organizational characteristics, the provider’s approach to service delivery, challenges in administering Ticket Act services, and considerations for program improvement. Mathematica will offer the surveys as online surveys and expects to achieve an 80 percent response rate for each survey based on its experience with prior SSA evaluations. Of the expected 458 completed surveys, Mathematica anticipates 353 from ENs and VR agencies, 59 from WIPA projects, and 46 from P&A agencies.

Assumptions for universes and sample sizes in the survey data collection

Ticket Act service providers	Number in total population universe	Number included in survey	Expected number of completed interviews
ENs and VR agencies	441	441	353
WIPA projects	74	74	59
P&A agencies	57	57	46
Total	572	572	458

Though SSA requests data from Ticket Act providers each year as part of program monitoring, SSA has not systematically surveyed the Ticket Act providers in this way before. If the response rate for any of the surveys is lower than 80 percent, Mathematica will conduct a nonresponse bias analysis and take the results into account during weighting procedures.

Assumptions for universes and sample sizes in the qualitative data collection

Population	Universe	Number selected for outreach	Target number of completed interviews
Ticketholders
TTW-only participants	300,000	1,200	60
WIPA-only participants	40,000	200	10
TTW and WIPA participants	Unknown	400	20
Non-participants of either program	Unknown	200	10
Subtotal		2,000	100
Total
Total		2,000	100

Qualitative interviews with Ticketholders

Mathematica will conduct up to 100 qualitative interviews via telephone with Ticketholders. The goals of the interviews include: understanding the Ticketholders’ experiences learning about, accessing, and using Ticket Act programs and services; and gathering any related successes, challenges, or suggestions for improvement. Attachment B includes the topics for the interviews.

The universe of TTW participants is about 300,000, and the universe of WIPA participants is about 40,000. As the table above indicates, Mathematica will select and reach out to 2,000 Ticketholders with a goal of completing interviews with 100 of them. Of the 100 interviews, Mathematica will conduct 60 with those who participated in TTW services only, 10 with those who participated in WIPA services only, 20 with those who participated in both TTW and WIPA services, and 10 with Ticketholders who are eligible for but have not participated in TTW or WIPA services.

Mathematica will recruit Ticketholders who reflect varied program involvement and individual characteristics to learn about a range of experiences. Mathematica will use SSA administrative data to identify those who have participated in TTW or WIPA services in the last year. Mathematica will select three purposive samples of Ticketholders: TTW participants who assigned their Ticket or put it in use with an EN or VR agency or who recently unassigned their Ticket from an EN; WIPA participants who received a referral to a WIPA project; and those who participated in both TTW and WIPA services. Mathematica will also recruit a convenience sample of working Ticketholders who did not participate in TTW or WIPA services. Mathematica will administer a brief screening questionnaire with interested Ticketholders to confirm that their experiences will provide relevant information for the interview.

2. Procedures for Collecting the Information

Statistical methodology for sample selection

The provider surveys will include all providers and therefore we will not conduct sampling for these surveys.

The qualitative interviews will use purposive and convenience sampling strategies to identify interviewees. We will use SSA administrative data to identify a purposive Ticketholders who have recent experiences with TTW and/or WIPA programs, to collect insights on their ability to find and use services. The sample we select will include a range of experiences based on Ticketholder characteristics such as type of disabling condition and rural/urban status, as measured in administrative data. We will use a convenience sample drawn from known contacts to identify beneficiaries who opted to work without using TTW or WIPA services.

Qualitative interviews, by nature, are not meant to be statistically representative of the universe of all Ticketholders. Such interviews, however, provide rich and meaningful insights beyond what can be obtained in a survey with pre-identified response options. By sampling beneficiaries for interviews based on broad geographic and disability experiences, we will understand potential differences in barriers and facilitators to service access and take-up. This information, in turn, can inform strategies that SSA can use to increase the awareness, availability, use and effectiveness of Ticket Act program services.

Estimation procedure

Provider surveys. Mathematica will produce descriptive statistics to describe the experiences and perspectives of providers in each program separately (that is, TTW, WIPA, and PABSS). Analyses will also compare statistics across groups of providers within each program (as sample sizes allow) based on characteristics such as provider type, geographic service area, tenure in the program, number of clients served, and other measures derived using administrative data. Statistical analyses will document differences in means and distributions across provider groups, as necessary. Findings will complement the qualitative and administrative data analyses to develop a comprehensive understanding of the experiences of Ticket Act program providers and develop suggestions for ways to improve the programs. Mathematica will document the results in the reports and briefs described in Part A.

Qualitative interviews. As discussed in Part A, Mathematica will use qualitative interview data to provide a detailed description of the experiences of Ticketholders. Mathematica will code and analyze the data systematically to avoid bias in interpreting findings. Mathematica will document the results in the reports and briefs described in Part A.

Degree of accuracy needed for the purpose described in the justification

Each provider survey represents the maximum available sample, as it includes the full universe of Ticket Act service providers. As such, we did not calculate minimum detectable effects to determine the sample size. Mathematica’s analysis will summarize the experiences of all providers in each program separately (that is, TTW, WIPA, and PABSS). Mathematica will also consider differences across subgroups of providers within each program (for example, comparing ENs that have a large number of Ticket assignments with ENs that have relatively few assignments or comparing ENs based on the business model registered with SSA). SSA is interested in understanding the differences across provider groups and the extent to which differences are statistically meaningful; Mathematica will develop the stratification approach based on the aims of the evaluation research questions.

For the qualitative interviews, Mathematica determined the 100 interviews are sufficient to canvas a range of experiences with Ticket Act programs based on the experience of reaching saturation (the point at which we no longer learn new findings) in qualitative data collection on past SSA evaluations. As noted, we do not expect that the findings will be statistically representative of the universe of all Ticketholders but expect that this number will be sufficient to canvass a wide range of beneficiary experiences.

3. Methods to Maximize Response Rate

Provider surveys

Designing and fielding surveys. Mathematica designed the provider survey instruments to address the evaluation’s objectives and research questions. Mathematica developed a separate instrument for each provider type based on the unique characteristics and terminology for each. The length of each instrument balances the competing demands of including enough questions to satisfy the evaluation’s need for information about a variety of topics but also limiting the number of questions to avoid compromising the quality of the responses obtained from the survey. After Mathematica drafted the instruments, SSA and consultants knowledgeable about the Ticket Act programs reviewed and provided input on them. Mathematica then tested the instruments in pretests involving no more than nine respondents per provider type, as described in Section 4. Mathematica will field the three instruments concurrently in a single survey data collection effort. Mathematica’s survey fielding methods are designed to maximize response rates and the quality of response data. Offering the surveys for self‑administration online, for example, enables respondents to complete the survey when they choose. In addition, Mathematica will assure respondents of the privacy of their responses, which should yield higher quality data.

Response rates. Mathematica’s approach to the provider surveys addresses several challenges that can depress response rates. First, SSA and Mathematica will send providers an initial letter and follow-up emails that contain a personalized link to access the online survey. Second, Mathematica kept the survey instruments brief to encourage providers to respond and answer all questions. Third, as discussed in greater detail in Part A, Mathematica will offer a $40 incentive payment after providers complete the survey. Fourth, Mathematica will conduct telephone follow-up to ensure outreach is routed to the correct contact person, promote survey completion, and address providers’ concerns. Fifth, SSA and Mathematica will proactively address concerns about legitimacy by using the SSA and TTW logos on survey materials. Sixth, SSA and Mathematica will ask professional organizations that are relevant to the providers to help promote awareness of the surveys and validate the surveys’ importance and legitimacy by endorsing the surveys and announcing them to their members before the field period.

Data reliability. Mathematica developed the provider survey instruments and contact materials using materials developed and fielded on recent similar SSA demonstrations, such as the Retaining Employment and Talent after Injury/Illness Network (RETAIN), OMB No. 0960-0821. SSA and evaluation consultants knowledgeable about the Ticket Act programs reviewed the draft instruments and contact materials and helped refine them further. Mathematica also tested the instruments with a small number of providers (fewer than 10), as described in Section 4.

Item nonresponse. Although Mathematica’s experience conducting surveys for similar evaluations suggests that rates of item nonresponse on the provider surveys will be very low, some item nonresponse is inevitable. Mathematica will review missing data on each item and report results that document the proportion of missing responses. Because of the small number of providers and important differences across provider experiences, Mathematica does not intend to impute missing responses.

Individual-level nonresponse. As with almost any survey, some nonresponse in the provider surveys is inevitable. Because these surveys are voluntary, some providers might decline to participate. Mathematica expects to attain a response rate of at least 80 percent based on its experience with prior SSA demonstrations. If response rates are lower, Mathematica will analyze nonresponse using various data items from SSA’s administrative data records. The nonresponse bias analysis will consist of the following steps:

• Compute response rates for key subgroups. Mathematica will compute the response rate for the subgroups using the American Association for Public Opinion Research definition of participation rate, which is the number of respondents that provided a usable response divided by the total number of entities the survey requests participation from (American Association for Public Opinion Research 2023). Mathematica will calculate response rates across key subgroups, such as provider size, business model, or service region. The goal is to identify whether response rates in specific subgroups differ systematically from those of other subgroups or from the overall response rate to determine provider groups that might not be represented sufficiently in the analysis. This could inform the development of nonresponse weights for use in the analysis.

• Compare the distributions of respondents’ and nonrespondents’ characteristics. Using data from SSA’s administrative records, Mathematica will compare the characteristics of respondents and nonrespondents. Characteristics could include information about the ENs such as provider size or business model, as well as information about the clients they serve such as the percentage of SSDI-only, SSI‑only, or concurrent beneficiaries. Mathematica will assess the statistical significance of the differences between these groups using t-tests or chi-squared tests which calculate the likelihood that a difference of at least the size observed could have occurred due to chance. The less likely a difference is to have occurred due to chance, the more confidence we have that it reflects real differences. This can help identify patterns of differences in observable characteristics that might suggest nonresponse bias. This approach, however, has low power to detect substantive differences when sample sizes are small, and the large number of statistical tests conducted can also result in high rates of Type I error. Consequently, Mathematica will interpret the results of this item-by-item analysis cautiously.

• Identify the characteristics that best predict nonresponse and use this information to generate nonresponse weights. Mathematica will use logistic regression models to assess the partial associations between each characteristic and response status; propensity scores obtained from such models provide a concise way to summarize and correct for initial imbalances (Särndal et al. 1992). Because of the rich administrative data available for this analysis, Mathematica will use a mixture of substantive knowledge and automated machine-learning methods to identify covariates to include in the final weights. Examples of automated procedures Mathematica could use to produce these weights efficiently include: (1) using prespecified decision rules, such as those described by Imbens and Rubin (2015) and Biggs et al. (1991) to select covariates and interactions between them; and (2) identifying and addressing outliers by, for example, trimming weights in a way that minimizes the mean-square error of the estimates (Potter 1990). The resulting nonresponse weights would serve to conduct analyses that are representative of ENs and as a component of the weights used to conduct analyses representative of Ticket assignments.

Qualitative interviews with Ticketholders

Response rates. Because Mathematica will draw interviewees from purposive and convenience samples of volunteers, target response rates to ensure a representative population of Ticketholders are not at issue. Mathematica anticipates a cooperation rate of around five percent based on a similar survey of TTW service users. SSA and Mathematica will reach out to the potential interviewees with a mailed invitation letter and follow up via telephone thereafter, as needed. To proactively address concerns about legitimacy, the mailed invitation letter will include the SSA and TTW logos. To mitigate interview nonresponse, Mathematica will offer to interview people immediately or schedule an interview at the time most convenient for the interviewee and contact the interviewee by telephone on the day before the interview. Because Mathematica will conduct the interviews by telephone, interviewees will not face barriers related to transportation to an interview location. Mathematica will limit the interviews with nonparticipants to 30 minutes and with participants to 40 minutes to minimize burden on interviewees. Finally, Mathematica will provide a $40 gift card to interviewees to encourage interview participation and mitigate the risk of attrition after scheduling.

Data reliability. Mathematica interviewers will use an interview guide, based on the interview topic list provided in Attachment B, to support reliability while conducting the qualitative interviews. In addition, Mathematica will train all interviewers on the guide, the appropriate use of unbiased probes, and best practices for interviewing people with disabilities. The interviewers will take notes and obtain permission to record each interview.

4. Tests of Procedures

Provider surveys

Mathematica pretested the TTW, WIPA, and PABSS provider survey instruments with convenience samples of nine ENs and VR agencies, four WIPA projects, and four P&A agencies, respectively. After the providers completed the surveys, Mathematica asked debriefing questions to gather feedback on the instruments, assess flow, and measure respondent comprehension. Pretest participants received an incentive for their participation. Mathematica revised each instrument in collaboration with SSA based on findings from the pretests. The pretest interviews provided an informed estimate of respondent burden for each survey (32 minutes for TTW, 23 minutes for PABSS, and 33 minutes for WIPA), as required by the Office of Management and Budget.

Qualitative interviews with Ticketholders

Mathematica modeled the interview guide on guides used in similar evaluations, and senior research staff assessed the interview guide to confirm that it will yield the desired information. In addition, Mathematica will use the initial interview to test the interview guide and identify any needed revisions.

5. Statistical Agency Contact for Statistical Information

Below lists the TTW evaluation team members providing input on the technical issues discussed in this information clearance request.

Eleanor Stinnett

Telephone: Not available

Affiliation: SSA

Seth Hartig

Telephone: Not available

Affiliation: SSA

Denise Hoffman

Telephone: 202-554-7517

Affiliation: Mathematica

Holly Matulewicz

Telephone: 617-674-8364

Affiliation: Mathematica

Sarah Croake

Telephone: 734-205-3083

Affiliation: Mathematica

Jody Schimmel Hyde

Telephone: 202-554-7550

Affiliation: Mathematica

Noémie Sportiche

Telephone: 510-768-1269

Affiliation: Mathematica

Isabel Musse

Telephone: Not available

Affiliation: Mathematica

Gina Livermore

Telephone: 202-264-3462

Affiliation: Mathematica

Diane Beaver

Telephone: 202-484-4247

Affiliation: Mathematica

Joanna Nevins

Telephone: 202-838-3589

Affiliation: Mathematica

John Connelly

Telephone: Not available

Affiliation: Council of State Administrators of Vocational Rehabilitation

Cheryl Bates-Harris

Telephone: Not available

Affiliation: Retired

Larrisa Cummings

Telephone: Not available

Affiliation: University of Alaska Center for Human Development

Gloria Freeney

Telephone: Not available

Affiliation: Freeney Rehabilitation Career Services and the National Employment Network Association

References

American Association for Public Opinion Research. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. Tenth edition. Oakbrook Terrace, IL: AAPOR, 2023.

Biggs, David, Barry de Ville, and Ed Suen. “A Method of Choosing Multiway Partitions for Classification and Decision Trees.” Journal of Applied Statistics, vol. 18, no. 1, 1991, pp. 49–62.

Damschroder, L.J., D.C. Aron, R.E. Keith, S.R. Kirsh, J.A. Alexander, and J.C. Lowery. “Fostering Implementation of Health Services Research Findings into Practice: A Consolidated Framework for Advancing Implementation Science.” Implementation Science, vol. 4, no. 50, 2009.

Imbens, Guido, and Donald Rubin. Causal Inference in Statistics, Social, and Biomedical Sciences. New York: Cambridge University Press, 2015.

Potter, Francis J. “A Study of Procedures to Identify and Trim Extreme Sampling Weights.” In Proceedings of the American Statistical Association, Section on Survey Research Methods. Alexandria, VA: American Statistical Association, 1990, pp. 225–230.

Särndal, Carl-Erik, Bengt Swensson, and Jan Wretman. Model-Assisted Survey Sampling. New York: Springer-Verlag, 1992.

1 To the extent that the universe of service providers changes between the time of drafting of this document and the survey fielding period, we will field the survey to the population of services providers as of a date as close to the beginning of survey fielding as practicable.

1