Supporting Statement--0687
Supporting Statement--0687.doc
Youth Transition Process Demonstration Evaluation Collection
OMB: 0960-0687
Supporting Statement for Paperwork Reduction Act Submission
Karen CyBulski
Anne Ciemnecki
Submitted to:
Social Security Administration Office of Program Development and Research Suite 700, 400 Virginia Avenue, SW Washington, DC 20024 Telephone: (202) 358-6509 Facsimile: (202) 358-6505
Project Officer:
|
Submitted by:
Mathematica Policy Research, Inc. P.O. Box 2393 Princeton, NJ 08543-2393 Telephone: (609) 799-3535 Facsimile: (609) 799-0005
Project Director: |
|
|
CONTENTS
Page
A. JUSTIFICATION 1
1. Circumstances that Make the Data Collection Necessary: Legal or Administrative Requirements 1
2. How, by Whom, and for What Purpose the Information Will Be Used 3
3. Use of Improved Information Technology 6
4. Efforts to Identify Duplication 7
5. Involvement of Small Entities 7
6. Consequences if Information Is Not Collected or Collected Less Frequently 7
7. Special Circumstances 8
8. Adherence to Guidelines in 5 CFR 1320.5(d) (2) and Consultation outside the Agency 8
9. Remuneration of Respondents 10
10. Assurance of Confidentiality 10
11. Questions of a Sensitive Nature 11
12. Estimates of Annualized Hour Burden 12
13. Estimates of Annualized Capital Burden 12
14. Estimates of Annualized Cost to the Government 12
15. Explanation for Program Changes or Adjustments 15
16. Plans for Tabulation and Publication and Project Time Schedule 15
17. Expiration Date for OMB Approval 17
18. Exceptions to the Certification Statement 17
B. COLLECTIONS
OF INFORMATION EMPLOYING STATISTICAL
METHODS 18
1. Respondent Universe and Sampling Methods 18
2. Procedures for the Collection of Information 19
3. Methods to Maximize Response Rates 21
4. Tests of Procedures 23
5. Statistical Consultants and Persons Collecting and Analyzing the Data 23
LIST OF TABLES
Table Page
A.1 OUTCOMES FOR WHICH IMPACTS ARE ANTICIPATED IN THE
LONGER
TERM 6
A.2 ANNUALIZED BURDEN 13
A.3 ANNUAL COSTS TO THE FEDERAL GOVERNMENT 15
B.1 CHARACTERISTICS OF EXISTING YTD PROJECTS PARTICIPATING
IN
THE RANDOM ASSIGNMENT EVALUATION 19
B.2 MINIMUM DETECTABLE IMPACTS FOR THE YTD EVALUATION,
ASSUMING
INDIVIDUALIZED RANDOM ASSIGNMENT 21
Supporting
Statement For
YOUTH TRANSITION PROCESS DEMONSTRATION
EVALUATION
OMB CONTROL No. 0960-0687
The Social Security Administration (SSA) is requesting clearance for the collection of data needed to implement and evaluate the Youth Transition Demonstration (YTD) projects. YTD projects intend to help young people with disabilities make the transition from school to work. By waiving certain disability program rules and offering services to either youth who are receiving disability benefits or at risk of receiving them, we expect these projects to encourage youth to work and/or continue their education. A random assignment evaluation of YTD projects will take place in 6 sites across the country. The evaluation will produce empirical evidence on the impacts of the waivers and project services on not only educational attainment, employment, earnings, and receipt of benefits by youth with disabilities but also on the Social Security Trust Fund and federal income tax revenues.
Given the importance of estimating YTD impacts as accurately as possible, the evaluation uses rigorous analytic methods based on the random assignment of youth to a treatment or control group. Several data collection efforts are planned or underway. These include (1) baseline interviews and informed consent with youth and their parents or guardians prior to random assignment; (2) follow-up interviews at 12 and 36 months after random assignment; (3) interviews and/or roundtable discussions with local program administrators, program supervisors, and service delivery staff; (4) focus groups of youth, their parents, and service providers; and (5) in-depth interviews with youth and/or their parents or guardians within three months of completing the 12 month follow-up interview.
OMB has granted clearance for the baseline interviewing, 12 month follow-up interviewing, in-depth interviewing, focus groups, and discussions with program staff and service providers under OMB # 0960-0687, which expires November 30, 2011. In this package, SSA requests clearance of the 36-month follow-up interviewing.
A. JUSTIFICATION
1. Circumstances that Make the Data Collection Necessary: Legal or Administrative Requirements
a. Circumstances
The transition to adulthood for youth with disabilities can be difficult. SSA is sponsoring the YTD projects, and the related evaluation of those projects, to examine the effectiveness of providing services to youth with disabilities during their transition to adulthood. In addition to the host of issues facing all transition-age youth, those with disabilities have special issues related to health, social isolation, multiple service needs, and lack of access to supports. This set of challenges complicates their planning for future education and work and often leads to poor educational and employment outcomes, and high risk of dependence. SSA is investing considerable resources in developing and evaluating strategies to maximize the economic self-sufficiency of youth with disabilities, focusing on youth at the ages of 14 to 25 years, as they transition from school to work. Hallmark features of the YTD evaluation include carefully designed and targeted demonstration projects that are policy-relevant and can operate at a scale required by the evaluation and a rigorous random assignment evaluation design.
b. Legal or Administrative Requirements
Since 1980, Congress has required the SSA to conduct demonstration projects to test the effectiveness of possible program changes that could encourage individuals to work and decrease their dependence on disability benefits. In fostering work, we intend these demonstrations and the program changes they test to produce savings in the trust funds or improve program administration.
To achieve these objectives, SSA’s demonstration authority contains several key features that provide SSA with a potentially valuable tool for assessing the effectiveness of policy alternatives. One of these features is SSA’s authority to waive certain disability insurance and Medicare program rules. For example, when conducting demonstrations, SSA is permitted to exempt certain beneficiaries from requirements that workers with disabilities earn below a certain amount to remain eligible for benefits. Another key aspect of SSA’s demonstration authority is the requirement that demonstration projects be of sufficient scope and conducted on a wide enough scale to ensure a thorough evaluation and results that are applicable to the program as a whole.
In addition, the legislation authorizes SSA to use trust fund monies to pay for the demonstrations and requires SSA to periodically report to Congress on its demonstration activities, providing, when appropriate, recommendations for legislative or administrative changes.
Sections 234 and 1110 of the Social Security Act (Appendix A) direct the Commissioner of SSA to carry out experiments and demonstration projects to determine the relative advantages and disadvantages of the following:
Various alternative methods of treating the work activity of individuals receiving benefits, including such methods as a reduction in benefits based on earnings designed to encourage these beneficiaries to return to work
Altering other limitations and conditions, such as lengthening the trial work period or altering the 24 month waiting period for Medicare
Implementing a sliding scale benefit offset
The Act requires that we design these demonstration projects to show that savings will accrue to the trust funds, or will otherwise promote or facilitate the administration of the program. Section 234 also provides that we conduct these projects must in a manner that will allow SSA to evaluate the appropriateness of implementing such a program on a national scale.
To overcome the barriers to employment for beneficiaries, YTD provides individualized work-based experiences, youth empowerment, family supports, system linkages, social and health services, and benefits counseling. We randomly assign enrollees in the demonstration to either the treatment or the control group. Enrollees in the control group have access to the traditional services and existing work incentives available, while enrollees in the treatment group receive the demonstration’s enhanced services as well as waivers of certain disability insurance rules to strengthen work incentives. The evaluation will assess the impact of these services and waivers on educational attainment, employment, earnings, and reduced use of disability benefits. The demonstration and planned evaluation meet SSA’s legislative and congressional mandates.
2. How, by Whom, and for What Purpose the Information Will Be Used
Information collected will answer three key questions central to assessing the effectiveness of YTD projects:
How Are the YTD Projects Implemented and Operated? What are the important issues and challenges in designing, implementing, and operating YTD projects, and what lessons can we draw from the experience? What approaches do we take to provide services to promote self-sufficiency among youth with disabilities? What are the characteristics of the interventions and the context of their provision? Who participates in the YTD projects, for how long, and what services do they get? Who provides these services? How do those services differ from those received by members of the control group? To what extent do the youth use the SSA waivers? How does participation in YTD differ for population subgroups?
What Are the Short Term and Longer Term Impacts of the Projects? How effective are the projects in increasing employment and earnings and reducing dependence on disability benefits? Do the projects affect educational attainment or other intermediate outcomes, such as work attitudes or work experience? Do they improve social-psychological well being? Do they increase the likelihood that disabled youth will be able to live independently as adults? Do these impacts differ across subgroups of the population of youth with disabilities?
What Are the Costs of Operating the Projects, and Do the Benefits Outweigh the Costs? What are the projects’ operating costs? What other costs are incurred because of the YTD projects? To what extent do the projects lead to net changes in disability benefit receipt? Are there any induced entry effects? How do the projects affect income and payroll tax receipts, benefit outlays, and the status of the Social Security Trust Funds? From the perspectives of key stakeholders, do the benefits of the projects exceed their costs?
To address these three sets of questions, the evaluation includes process, impact, and benefit-cost analyses. This supporting statement requests clearance only for the 36-month follow-up data collection, which we will need for the impact and benefit-cost analyses.
Process Analysis. The process analysis will document the delivery of the intervention services, including information provided to participating youth on SSA waivers and the extent of utilization of those waivers. It will identify implementation successes, issues, and challenges and will examine program costs. It also will provide details on the nature of each YTD intervention, and the ways in which the projects have achieved the observed results. Data for this analysis will come primarily from site visits, project records and documents, and the projects’ management information systems. Site visits will include discussions with staff of the YTD projects and partner organizations, SSA field office staff, and other youth service providers; focus group discussions with participating youth and their families; case reviews; and program observations. We use baseline survey data to describe the youth enrolled in the study. We use MIS data to describe and analyze service receipt and utilization among treatment group members. We use data from the first follow-up survey to examine participant experiences and satisfaction with YTD services.
Impact Analysis. A rigorous random assignment design is being used to determine the differences these YTD projects make in educational attainment, employment, earnings, and reduced use of disability benefits as well as such outcomes as living arrangements, quality of life, and other measures of well being among the transition-age youth enrolled in the study. Under this design, we will randomly assign youth eligible for YTD services to a treatment group (offered YTD waivers and services), or to a control group (not offered YTD waivers or services but may use existing SSA work incentives and services available in the community). Outcomes for the two groups will be compared using data collected in follow-up interviews, conducted 12 and 36 months after youth enter the demonstrations, as well as data obtained from SSA program files, administrative files of state and local agencies, and possibly SSA summary earnings records (SER). Based on these comparisons, we will assess the net effects of the YTD intervention approaches for the youth enrolled in the study and the differential effectiveness of YTD services for members of certain subgroups. We will use administrative data to address impacts on SSA disability benefits receipt and the use of SSA waivers, earnings, and other public assistance. We will use the more comprehensive data from the follow-up surveys to examine employment and other outcomes such as education, income, health, and measures of life quality and well-being.
Benefit-Cost Analysis. A key element of the YTD evaluation is to measure the costs necessary to operate the YTD projects. For policy action, cost information is essential, because legislators and administrators will not be able to proceed with a program or policy unless they have a good idea of the program costs. The evaluation will conduct a comprehensive benefit-cost analysis of the YTD projects. We will start with a comprehensive cost analysis of each project; the goal is to construct an estimate of overall project costs as well as estimates of average unit costs, such as the cost per participant and cost per program component. Drawing on data reports from project records and on information from program staff interviews; we will build up an estimate of the cost of each project. In addition, information from in-depth interviews with youth or their guardians about service utilization will provide information needed for the service cost analysis. In addition to its usefulness as an adjunct to the process study’s description of program operations, the cost analysis will provide important input for the benefit-cost analysis. For purposes of this analysis, key costs include operating and administrative costs. Benefits include, but are not limited to, net increased earnings and tax payments, net reduced disability benefits, and net reductions in the receipt of public assistance. The benefit-cost analysis also will examine net changes in services used because of the YTD projects. The benefit-cost analysis will examine the extent to which the projects lead to net increases or reductions in SSI benefit receipt (and, hence, the cost or savings to SSA) as well as assess the extent to which there are any induced entry effects as a result of the waivers and services offered by the YTD projects. The analysis will examine the costs and benefits of the projects from the perspectives of a variety of stakeholders—including SSA, other government agencies, the YTD participants, and society as a whole—and will be produced in a format consistent with the requirements of SSA’s actuaries.
Under this design, we randomly assigned youth eligible for YTD services to a treatment group (offered YTD waivers and services) or to a control group (not offered YTD waivers or services but may use existing SSA work incentives and services available in the community). Outcomes for the two groups will be compared using data collected in follow-up interviews, conducted 12 and 36 months after youth enter the demonstrations, as well as data obtained from SSA program files, administrative files of state and local agencies, and possibly SSA summary earnings records (SER). Based on these comparisons, we will assess the net effects of the YTD intervention approaches for the youth enrolled in the study and the differential effectiveness of YTD services for members of certain subgroups. We will use administrative data to address impacts on SSA disability benefits receipt and the use of SSA waivers, earnings, and other public assistance. We will use the more comprehensive data from the follow-up surveys to examine employment and other outcomes such as education, income, health, and measures of life quality and well-being.
If effective, the most immediate impacts of the interventions should reflect in increased employment-focused services and work-related experiences for those in the treatment group. We also expect more paid employment, greater income resulting from increased employment and more generous work incentives offered by the waivers, attitudes that are more positive and expectations about the future, and continued progress in education for projects that emphasize education.
In the intermediate and longer terms, we expect treatment group youth in the projects to increase their employment and earnings, have greater income, reduce risky behaviors, and have greater self-determination and self-efficacy and move toward independent living. Furthermore, we expect that, in the considerably longer term, particularly after the waivers are no longer in effect, the projects will reduce youths’ dependence on disability programs.
We have grouped the key domains for which we will examine YTD impacts in the longer run into five areas, as summarized in Table A.1.
Data from the 36-month follow-up survey will supplement administrative files that SSA maintains, and will provide data on outcomes not available from administrative sources such as attitudes and expectations.
Table A.1
Outcomes for Which Impacts Are Anticipated in
the Longer Term
Outcome Measure |
Description of Measure |
Employment and Earnings |
|
Primary outcomes |
Fraction of time employed in past year, earnings in past year |
Supplementary outcomes |
Ever employed during follow-up period, earnings patterns, hours worked, full-/part-time work, number of jobs held, wage rates, benefits, accommodations, impacts from administrative data (over the three-year period following random assignment) |
Youth Income |
|
Primary outcomes |
Total income from earnings and benefits during prior 12 months or entire follow-up period |
Supplementary outcomes |
Type and amount of earnings and benefits received, amount of SSA benefits, use of SSA work incentives and Individual Development Accounts (IDAs) |
Engagement in Gainful Activity |
|
Primary outcomes |
Fraction of youth either employed or participating in an education or training program |
Supplementary outcomes |
Time spent engaged in gainful activities, educational attainment |
Reduction in Criminal Justice System Contact |
|
Primary outcomes |
Contact with criminal justice system (arrests, incarcerations, other involvement with the criminal/juvenile system) |
Supplementary outcomes |
Types of criminal activity, drug and alcohol treatment |
Self-Determination and Self-Efficacy |
|
Primary outcomes |
Self-determination and self-efficacy scale, independence (such as traveling, having a bank account, living arrangements) |
Supplementary outcomes |
Items comprising the scales, other measures of independence, attitudes and expectations, self-esteem, social interactions |
Other Exploratory Analyses |
|
Medicaid utilization |
Number and total amount of Medicaid paid claims during a calendar year |
Health status |
SF-12 health scale; self-reported health status: excellent, very good, good, fair, poor; self-perceptions of health conditions and disabilities |
Quality of life |
Selected components of the Instrumental Activities of Daily Living (IADL) that are relevant to youth; limitations in mental, emotional, and social functioning |
3. Use of Improved Information Technology
For the YTD evaluation, MPR and its partner, Social Solutions, have implemented a management information system (MIS), the Efforts-to-Outcomes (ETO) database, to facilitate the real-time exchange of data between MPR’s survey division and the YTD projects. The ETO database draws information from several data sources including SSA administrative data, respondent survey data, claims and utilization data, and data entered by YTD project staff. We designed the ETO database so that data from all sources are linkable so that it fully supports drawing extracts and generating reports and summaries to facilitate administering, monitoring, and evaluating the study.
We use computer-assisted interviewing to collect data for the baseline and follow-up surveys. We administer the baseline and follow-up surveys as both computer-assisted telephone (CATI) and face-to-face interviews. Both applications incorporate standard checkpoints to assess each respondent’s level of fatigue and to provide the respondent with an opportunity to take a break, if necessary. Both the baseline and 12-month follow-up interviews use Computer Assisted Telephone Interviewing (CATI) software. We will use computer-assisted interviewing to collect data for the 36-month follow-up survey. We have programmed a questionnaire into the software application. The software is able to customize the flow of the questionnaire based on the answers provided, as well as information already known about the sample member such as their gender, treatment or control status, or state of residence. Interviewers read questions that appear on their computer screen and enter the respondents’ answers. In this sense, it is similar to SSA’s MCS/MSSICS systems.
Telephones equipped with amplifiers will be available for use as needed to accommodate sample members who are hearing impaired. In addition, we will use TTY and Relay technologies to facilitate participation in the telephone survey. A TTY is a special device that lets people who are deaf, hard of hearing, or speech-impaired use the telephone to communicate by allowing them to type messages back and forth to one another instead of talking and listening. A TTY is required at both ends of the conversation in order to communicate. MPR’s telephone operations center is equipped with TTY technology. We will use the Telecommunications Relay Service (TRS) for sample members who are deaf, hard of hearing or speech-impaired but who do not have a TTY. With TRS, a special operator types whatever the interviewer says so that the person called can read the interviewer’s words on his or her telephone display. He or she will type back a response, which the TRS operator will read aloud for the interviewer to hear over the phone. Both methods, TTY and TRS, increase survey administration times but enable us to conduct interviews with sample members who, without the help of these technologies, would not be able to participate. Forms are not available electronically because they are not self-administered.
4. Efforts to Identify Duplication
The surveys will only ask respondents about information that is not available in SSA’s administrative records. We have reviewed administrative records in detail to limit repetition. Some information about treatment group members that we collect through the surveys may be redundant with data that could also be available from the ETO database. However, this duplication is necessary to collect comparable data from sample members in the control group.
5. Involvement of Small Entities
The Youth Transition Process Demonstration Evaluation does not involve small entities.
6. Consequences if Information Is Not Collected or Collected Less Frequently
The baseline survey is a one-time collection and is necessary to conduct a credible evaluation. We need the baseline survey to identify and select sample members into the study groups, assure that the treatment and control groups are comparable, and obtain important covariates for subsequent analyses. The data collected during the baseline interview are not available from other sources.
The first follow-up survey which was conducted at 12 months after random assignment collected information on short term outcomes regarding education, earnings, employment, living arrangements, health, and quality of life. We planned the second follow-up survey for 36- months after random assignment. It will collect information about longer-term outcomes in the same domains as the 12-month follow-up survey. The questions focus on the past 12 or 24 months so as not to overlap with the 12-month survey recall period. The follow-up survey will collect a richer set of information than can be gathered from administrative records. For example, administrative records might have data on earnings from jobs but would not have detail about the jobs such as rates of pay, hours worked, or if the job was competitive or supported employment. We cannot conduct the impact analysis without the 36-month follow-up survey.
7. Special Circumstances
There are no special circumstances related to the collection of information required to carry out the evaluation of YTD.
8. Adherence to Guidelines in 5 CFR 1320.5(d) (2) and Consultation outside the Agency
a. Federal Register Notice
The 60-day advance Federal Register Notice published on February 17, 2009, at 74 FR 7506, and SSA has received no public comments. The second Notice published on May 20, 2009 at 74 FR 23764. There have been no outside consultations with members of the public.
b. Consultation with Outside Agencies
We need an interdisciplinary project team of economists, disability policy researchers, survey researchers, and information systems professionals to carry out the design and implementation of the evaluation. MPR is the prime contractor with overall responsibility for implementing and evaluating the demonstration. However, staff members from three other organizations are integral members of the study team. The participating organizations include the following:
Mathematica Policy Research, Inc.
600 Maryland Ave., SW
Suite 550
Washington, DC 20024-2512
(202) 484-4698
600 Alexander Park
Princeton, NJ 08540
(609) 799-3535
MDRC
19th Floor
16 East 34 Street
New York, NY 10016-4326
(212) 532-3200
Social Solutions, Inc.
2400 Boston St.
Suite 360
Baltimore, MD 21224
(410) 732-3560
TransCen, Inc.
451 Hungerford Drive
Suite 700
Rockville, MD 20850
(301) 424-2002
We have listed key staff from these organizations, their roles, and contact information below:
Tom Fraker
MPR-DC
Project Director
(202) 484-4698
David Butler
MDRC-NY
Leader of task force on program development
(212) 340-8621
Karen CyBulski
MPR-NJ
Survey Director
(609) 936-2797
Anu Rangarajan
MPR-NJ
Leader of task force on evaluation
(609) 936-2765
John Martinez
MDRC-NY
Leader of task force on pilot sites
(212) 340-8690
Anne Ciemnecki
MPR-NJ
Senior Methodologist
(609) 275-2323
Richard Luecking
TransCen
President, Programmatic TA
(301) 424-2002 x230
Matt Schubert
Social Solutions
CEO, ETO
(410) 732-3560
Kelli Crane
TransCen
Director, programmatic TA
(240) 418-2684
In addition, SSA has convened a technical working group with these members:
Harold Beebout
Independent Consultant
(540) 987-3320
Michael Callahan
Marc Gold & Associates/Employment for All
(228) 497-6999
(228) 497-6966
Nancye Campbell
DHHS/ACF
(202) 401-5760
Elizabeth McGuire
DHHS/HRSA
(301) 443-9290
Alexandra Kielty
USDOL/ETA
(202) 693-3730
(202) 693-3818
K. Charlie Lakin
University of Minnesota
(612) 624-5005
(612) 625-6619
Rebecca Maynard
University of Pennsylvania
(215) 898-3558
Howard Rolston
Abt Associates
703-532-4195
Betsy Valnes
Black Hills Special Services
(605) 224-5336
Mary Wagner
SRI International
(650) 859-2867
c. Consultation with Beneficiaries
Beneficiaries and their parents have participated in the pretest of the follow-up survey.
9. Remuneration of Respondents
At baseline, we offer beneficiaries a $10 gift for returning a completed consent form. The form of the payment varies by location. In New York City, for example, it is a $10 MetroCard. In other places, it is a $10 Target or Wal-Mart gift card. Another $10 gift will be offered to beneficiaries completing the 12 month follow-up interview and again, for the 36 month follow-up interview. The incentive will increase interview response rates and reduce sample attrition between the baseline and 12 month interviews and between the 12 month and 36 month interviews.
We offer all focus group participants $40 to cover their time, transportation, or other costs of participating.
We do not offer program staff members remuneration for completing interviews because they will do this as part of their job responsibilities.
10. Assurance of Confidentiality
The information provided for this project is protected and held confidential in accordance with 42 U.S.C. 1306, 20 CFR 401 and 402, 5 U.S.C. 552 (Freedom of Information Act), 5 U.S.C. 552a (Privacy Act of 1974), and OMB Circular No. A-130. We will treat data in a confidential manner unless otherwise compelled by law.
The study team takes seriously the ethical and legal obligations associated with the collection of confidential data. We ensure the secure handling of confidential data via several mechanisms. These include obtaining suitability determinations for designated staff, training staff to recognize and handle sensitive data, protecting computer systems from access by staff without favorable suitability determinations, limiting access to secure data on a “need to know” basis and only for staff with favorable suitability determinations, and creating data extract files from which identifying information has been removed.
We will take several steps to assure sample members that we will treat the information they provide confidentially and used for research purposes only. We will make clear the assurances and limits of confidentiality and the Paperwork Reduction and Privacy Act statements appear on the advance letter in all advance materials sent participants and restated at the beginning of each interview session.
Subcontractors, consultants, and vendors will be required to establish confidential information safeguards that meet prime contract security requirements. The project director or task leader will take action to ensure that any confidential information provided to or generated by a subcontractor, consultant, or vendor is properly disposed of at the completion of the agreement between the parties.
11. Questions of a Sensitive Nature
The purpose of the study is to test the effects of waivers of SSA program rules and an innovative array of enhanced employment and educational services for youth with disabilities. Therefore, obtaining information about potentially sensitive topics, such as the health status and the disabling condition of sample members, is central to the intervention. The surveys will not collect data that we can obtain directly from other sources (for example, we can obtain information about receipt of disability benefits directly from SSA administrative records).
The survey will include questions about the following topics that one may consider potentially sensitive:
Health status, including disability information and severity of disabling condition
Assistance needed with Instrumental Activities of Daily Living (IADLs)
Mental health status
Participation in drug and alcohol treatment programs
Involvement with the criminal justice system
Many of the questions were adapted without modification from other national surveys of similar populations, such as the National Longitudinal Transition Survey (NLTS), the National Beneficiary Survey (NBS), and the Evaluation of Job Corps (JC). The instrument also contains items from the Short Form 12 (SF-12).
12. Estimates of Annualized Hour Burden
Table A.2 shows the annualized number of expected participants in the data collection, the number of interviews, hours per response, and the total associated response burden. Burden was determined though experience administering similar items on the 12-month follow-up interview and in pretest-based estimates of the 36-month follow-up questionnaire. All eligible youth who consented to participate in the YTD demonstration are eligible to participate in the 36-month follow-up survey. We expect to obtain responses from 3,962 respondents for the 36-month follow-up interview. We estimate 50 minutes (0.83 hour) per completed interview. Respondents will incur no monetary costs for completing the interview.
13. Estimates of Annualized Capital Burden
There are no direct costs to respondents other than their time to participate in the study, as described above. We will not ask beneficiaries to maintain any new records. The evaluation contractor will collect and maintain all survey data. Costs for data collection, storage, processing, and other functions related to these data are borne solely by the contractor. We summarize these costs below and consider costs to the federal government, paid through SSA contracts.
14. Estimates of Annualized Cost to the Government
The total cost to SSA of conducting the YTD evaluation is $46,829,767. Table A.3. shows the costs by year. Labor costs are budgeted by estimating the number of hours of required staff at the various wage levels, multiplying by the applicable wage rates, and multiplying the resulting subtotals by factors to cover fringe benefits and burden expense. The basis for estimating other direct costs varies with the type of cost estimated. For example, we based the estimates of survey telephone expense and computer expense for CATI on the estimated hours of interviewer time, while we based reproduction expense on the number of pages of material reproduced.
Finally, we summed and multiplied by a factor to cover general and administrative expenses, and to obtain the total of labor costs, other direct costs and the added fee.
TABLE A.2
ANNUALIZED BURDEN
Data Collection |
Collection |
Number of Respondents |
Responses Per Respondent |
Average Burden |
Total Response
Burden |
2007 |
Baseline |
962 |
1 |
0.55 |
529 |
|
Informed Consent |
962 |
1 |
.083 |
80 |
|
12-month follow-up |
437 |
1 |
0.83 |
363 |
|
Focus group |
140 |
1 |
1.5 |
210 |
|
Program staff/service provider |
32 |
1 |
1 |
32 |
Total 2007 |
|
|
|
|
1,214 |
2008 |
Baseline |
2,531 |
1 |
0.55 |
1,392 |
|
Informed Consent |
2,531 |
1 |
.083 |
210 |
|
12 month follow-up |
1,502 |
1 |
0.83 |
1,247 |
|
In-depth interviews |
120 |
1 |
.42 |
50 |
|
Focus group |
60 |
1 |
1.5 |
90 |
|
Program staff/service provider |
32 |
1 |
1 |
32 |
Total 2008 |
|
|
|
|
3,021 |
2009 |
Baseline |
1,895 |
1 |
0.55 |
1,042 |
|
Informed Consent |
1,895 |
1 |
.083 |
157 |
|
12 month follow-up |
1,518 |
1 |
0.83 |
1,260 |
|
In-depth interviews |
120 |
1 |
.42 |
50 |
|
Focus group |
150 |
1 |
1.5 |
225 |
|
Program staff/service provider |
80 |
1 |
1 |
80 |
|
36-month follow-upa |
364 |
1 |
0.83 |
302 |
Total 2009 |
|
|
|
|
3,116 |
2010 |
Baseline |
263 |
1 |
0.55 |
145 |
|
Informed Consent |
263 |
1 |
.083 |
22 |
|
12 month follow-up |
1,137 |
1 |
0.83 |
944 |
|
Focus group |
90 |
1 |
1.5 |
135 |
|
Program staff/service provider |
48 |
1 |
1 |
48 |
|
36-month follow-up |
1,252 |
1 |
0.83 |
1,039 |
Total 2010 |
|
|
|
|
2,333 |
2011 |
12 month follow-up |
158 |
1 |
0.83 |
131 |
|
36 month follow-up |
1,265 |
1 |
0.83 |
1,050 |
Total 2011 |
|
|
|
|
1,181 |
2012 |
36 month follow-up |
949 |
1 |
0.83 |
788 |
Total 2012 |
|
|
|
|
788 |
2013 |
36 month follow-up |
132 |
1 |
0.83 |
109 |
Total 2013 |
|
|
|
|
109 |
Grand Total |
Baseline |
5,651 |
1 |
0.55 |
3,108 |
|
Informed Consent |
5,651 |
1 |
.083 |
469 |
|
12 month follow-up |
4,752 |
1 |
0.83 |
3,944 |
|
In-depth interviews |
240 |
1 |
.42 |
101 |
|
Focus group |
440 |
1 |
1.5 |
660 |
|
Program staff/service provider |
192 |
1 |
1 |
192 |
|
36 month follow-upa |
3,962 |
1 |
0.83 |
3,288 |
Grand Total |
|
|
|
|
11,762 |
a We are requesting an additional 3,288 hours of burden for this clearance.
TABLE A.3
ANNUAL COSTS TO THE FEDERAL GOVERNMENT
Year |
Cost |
2006 2007 2008 2009 2010 2011 2012 2013 2014 Total |
|
15. Explanation for Program Changes or Adjustments
Even though we have lowered the number of respondents, and, therefore, the burden for the baseline study/initial informed consent, we intend to have more respondents complete the 12 Month Follow-Up and participate in the Focus Groups. Due to that increase, we are also increasing the number of Program Staff/Service Providers. Finally, we are adding a 36 Month Follow-Up study to the program, which increases the total annual burden. In oncoming years, while we will systematically decrease the baseline study, we will be increasing the burden for the 12 and 36 Month Follow-Up interviews (see Table A.2 of the Supporting Statement for the chart of annual burden information). The burden shown on ROCIS is for FY2009 only; however, we show the total burden for the collection in Table A.2 of the Supporting Statement.
16. Plans for Tabulation and Publication and Project Time Schedule
Baseline data collection began in July 2006 and will continue through 2010. The 12-month follow-up data collection began in November 2007 and will continue through 2011. Likewise, the 36-month data collection, for which we are requesting clearance, will begin in August 2009 and continue through 2014.
We planned a series of reports throughout the life of the demonstration. We scheduled to produce project-specific early assessment reports 8 months after the demonstrations enroll their first youth, beginning in the spring of 2007. We randomly assigned process and implementation, and early impact reports are due 18 months after the last youth at a project site, beginning in September 2009. We will produce the final report and public use data files by October 2014. We may produce up to three reports on special topics over the life of the demonstration by October 2014.
The process and implementation reports will document and describe how we planned and implemented the demonstration, explain program processes, document beneficiary experiences with the demonstration and describe outcomes or results. The following distinct components of program implementation will be addressed: (a) outreach, recruitment, and participation; (b) the intervention, including whether each component was implemented as planned, differences in implementation across subgroups, existing service systems, and the use of services; (c) organizational arrangements, communication, and coordination; (d) coordination with SSA field offices; and (e) experiences and satisfaction of beneficiaries and other stakeholders. We will explicitly document implementation issues we encountered as well as how we addressed them. We will also document how major features of the program change over the course of the evaluation, the reasons for the changes, and the implications for program outcomes measured in the evaluation.
The impact reports will investigate the demonstration’s effects on a wide array of education, earnings, and self-determination outcomes; the amount of benefits the beneficiary receives from SSA; and the beneficiary’s quality of life, both overall and for meaningful subgroups. Our proposed methodological approach combines a random assignment design with regression adjustment to improve the precision of our estimates. Because we randomly assign individuals to the control group and to the treatment group, the impact analysis will focus on differences in the outcomes of beneficiaries between these two groups using a regression framework to control for other explanatory variables. We will use regression-adjusted comparison of randomly assigned treatment group to control group for the full sample to address the impact of the intervention on beneficiaries’ education, labor market, and other outcomes. We will use regression-adjusted comparison of randomly assigned treatment group to control group for subgroups defined by pre-randomization values of age, race, gender, and type of disability.
The exact statistical technique used to estimate regression-adjusted impacts will depend on the nature of the dependent variable and the type of issues addressed. For example, if the dependent variable is continuous, then ordinary least squares regression produces estimates of impacts that are unbiased. For binary outcome variables (such as whether or not the beneficiary is employed), logistic regression models generate estimates that are consistent and efficient if the parametric assumptions underlying those models are correct. If the dependent variable is a count variable then we will use an ordered logit model. If the dependent variable is ordinal, we will first reduce the measure to binary outcomes and then estimate a logit model. To account for the fact that we will observe sample members for different lengths of time, we will also consider using event-history or hazard models for binary outcome measures. These models provide unbiased estimates of program effects on binary outcomes when participants’ data are truncated.
The purpose of the benefit-cost analysis is to determine whether the program impacts of the YTD demonstration are sufficiently large to justify the costs of providing program services. The results of this analysis will play an integral part in the decision to expand the demonstration to the larger population. We will base the analysis on an accounting framework that summarizes the intervention’s effects and resource use from the perspective of SSA and other key stakeholder groups, including society as a whole.
To ensure that the benefit-cost findings are as helpful as possible to SSA, we plan to present the information in a way that has proven useful for communicating this type of information to the SSA Office of the Actuary and to OMB. First, we will summarize all of the information based directly on data collected during the demonstration period. The second set of estimates will present the size of future effects (if any) that would be required for the program to generate benefits that exceed costs along with an analysis of how likely it seems that future effects of that size will occur. In this way, SSA actuaries will be able to see the net value generated during the observation period and then use the more speculative analysis of possible future benefits and costs to draw conclusions about whether the YTD projects would ultimately pay for themselves. In addition to using this general presentation format, we will work with the actuaries during the evaluation to ensure that the other assumptions used in the analysis—the discount rate, correction for inflation, and projections about potential productivity growth—are consistent with the ones they are using to assess other potential SSA initiatives. This consistency will go a long way in ensuring that comparisons of the various options are accurate and useful.
17. Expiration Date for OMB Approval
We will display the OMB expiration date on all survey materials sent to respondents, including the advance. It will be accessible in the computer-assisted instruments when a respondent requests the information.
18. Exceptions to the Certification Statement
We are not requesting any exceptions. The data collection will conform to all provisions of the Paperwork Reduction Act.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
1. Respondent Universe and Sampling Methods
YTD projects intend to improve long-term employment outcomes for youth ages 14-25 with disabilities. Both current SSI beneficiaries and youth who are at risk of receiving benefits as adults comprise the respondent universe for YTD services. In April 2005, approximately 776,000 youth ages 14 to 25 years old received SSI benefits. In addition, 320,000 youth were at risk of receiving benefits as adults, even though they did not qualify to receive benefits as children.1
YTD projects deliver services to youth with disabilities in their jurisdictions. On September 30, 2003, SSA awarded five-year cooperative agreements to seven state agencies and universities to implement YTD projects. We selected three of these projects for the national random assignment evaluation. In addition, we selected five new projects (out of 13 that applied) for a limited pilot phase. We selected three of the pilot projects in the fall of 2007 to join the national random assignment evaluation, for six random assignment projects. The respondent universe for this evaluation is youth who are willing and eligible to participate in the YTD services of the six random assignment projects.2
Baseline data collection began in July 2006 and will continue through 2010. The 12-month follow-up data collection began in November 2007 and will continue through 2011. Likewise, the 36-month data collection, for which we are requesting clearance, will begin in July 2009 and continue through 2014. All randomly assigned youth who consented to participate in the demonstration and completed a baseline interview comprise the sample for the 36-month follow-up survey.
a. Selection of Youth
Each of the random assignment projects will serve 400 treatment group youth. To allow for attrition, we generated a treatment group of 480 youth who we may serve, and a control group of 400 youth. This results in a total of 5,280 youth in the study (880 youth in each of 6 projects) recruited between July 2006 and October 2010. We obtain baseline information and written consent to participate in the evaluation for all youth participating in the YTD demonstration. We expect that 90 percent (approximately 4,752) will complete the 12-month follow-up interview. At 36 months, we expect to obtain responses for 3,962 youth: 82 percent of those who responded to the 12-month interview and 12.5 percent of those who were non-respondents at 12 months for a longitudinal response rate of 75 percent three years after random assignment.
Table B.1 provides descriptions of the populations and our best estimates of the numbers of youth who meet the eligibility criteria for each project.
TABLE B.1
CHARACTERISTICS OF YTD PROJECTS PARTICIPATING
IN THE RANDOM ASSIGNMENT EVALUATION
Project Name |
Location |
Description of Youth Served |
Population Estimates |
Colorado’s Youth Work Incentive Network of Support (WINS) |
Boulder, El Paso, Larimer, and Pueblo counties, CO |
14- to 25-year-old current SSI beneficiaries |
2,750
|
New York’s Transition WORKS |
Erie County, NY |
16- to 25-year-old current SSI beneficiaries in Erie County |
3,300 |
New York’s CUNY Youth Transition Demonstration |
Bronx, NY |
15- to 19-year-old current SSI beneficiaries |
4,533 |
Abilities, Inc. |
Miami-Dade County, FL |
16- to 22-year old current SSI beneficiaries in Miami-Dade County |
6,952 |
Career Transition Program (CTP) |
Montgomery County, MD |
High school juniors and seniors with with severe emotional disturbances—15 percent current SSI beneficiaries, 85 percent at risk of becoming beneficiaries |
1,650a
|
Human Resources Development Foundation (HRDF) |
WV (19 counties)b |
16- to 25-year-old current SSI beneficiaries |
4,702 |
a Based on high school juniors and seniors residing in Montgomery County who are receiving special education services and have a code of severe emotional disturbance (SED).
b Monongalia, Preston, Marion, Taylor, Harrison, Barbour, Lewis, Upshur, Randolph, Wood, Jackson, Mason, Cabell, Wayne, Putnam, Kanawha, Fayette, Raleigh, and Mercer counties.
2. Procedures for the Collection of Information
a. Recruiting Study Participants at Baseline
Different recruitment strategies are necessary depending on whether a project serves only youth who are in SSA records or whether it also or only serves youth who identified by other means, such as referrals. We first discuss recruiting procedures for youth with presence in SSA records. For projects serving current beneficiaries or at-risk youth who can be identified through SSA records (for example, youth whose applications were denied), MPR conducts baseline interviews, gathers written informed consent, and randomly assigns consenting youth into the treatment or control group. After random assignment, we only share the names of treatment group members with the YTD projects for enrollment and services. We list the specific steps in the recruitment process below:
1. Obtain a list of beneficiaries (or denied applicants) from SSA for the relevant catchment areas
2. Check the list to exclude ineligible youth based on age, place of residence, or disabling condition(s)
3. Randomly sort the list into batches of youth (also called replicates)
4. Send letters to a batch of youth informing them about YTD program services to recruit them into the study
5. Place telephone calls to determine interest in YTD services
6. Gather baseline and re-contact data by telephone, and obtain written informed consent from youth/parent by mail or in person. Appendix G contains the consent forms for the three existing sites. Consent forms for newly selected will be similar.
7. Randomly assign youth to the treatment or control group
8. Provide YTD project staffs with information on treatment youth so they can contact them and start providing program services
9. Continue to release cases in batches until we reach the desired enrollment for the project
We will identify at-risk youth who have not applied for SSI benefits through referrals from local organizations, including schools and other agencies that work with youth with disabilities. We have not yet recruited at-risk youth into the study; however, our design for the recruitment procedures is as follows:
The YTD project determines whether a youth who has been referred to it meets the project’s eligibility criteria.
If a youth meets the criteria, the project obtains a completed application form, collects baseline and re-contact data, and obtains written informed consent.
The project transmits this information to MPR.
MPR conducts random assignment and immediately provides the YTD project with information on the treatment/control status of the case.
YTD project staff informs the youth of his or her random assignment status and commences services to youth in the treatment group.
This process continues until we reach the desired enrollment target.
We designed hybrid procedures for projects that serve both youth who can be identified in SSA records as well as youth who must be identified through other sources.
For either recruitment method, MPR (or the YTD project) contacts parents or legal guardians of youth under age 18 and gains consent to speak with youth. For youth over age 18 with legal guardians, MPR (or the YTD project) gains permission from the legal guardians before approaching the youth. Both the baseline and 12-month follow-up interviews contain a parent module consisting of questions that youth may not be able to answer reliably.
Most YTD projects have limited samples in their intended catchment areas and are planning expansions to reach enough youth to generate the 880 sample members needed to make statistically significant comparisons between treatment and control group members. For example, the New York City YTD project is now considering expanding from the Bronx into areas of Manhattan to target its services to 17- and 18-year-olds. Likewise, the Montgomery County, Maryland, project is considering an expansion into Prince Georges County and has expressed a willingness to expand to Frederick if necessary so that it can target services to youth with severe emotional disorders.
b. Study Procedures for 12- and 36-Month Follow-Up Interviewing
We will use neither stratification nor sampling in conducting the 36-month follow-up survey. We will attempt to contact all individuals who agreed to participate in the demonstration. We will mail an advance letter to all sample members. We will then attempt to contact sample members by telephone. We will assign non-respondents to field interviewers who will first attempt to locate and then interview study participants.
We will mail an advance letter to sample members advising them of the upcoming survey about one week prior to their 12-month anniversary and again one week prior to their 36-month anniversary. These letters will contain a toll-free number that the youth or his or her parent may call if they have questions or wish to set an interview appointment. Next, MPR will telephone the last known number for the youth and/or his or her parent or guardian. If the number is disconnected, MPR will attempt to locate an address or telephone number. MPR will use CATI as the primary mode of data collection for the follow-up survey. We will interview in person all sample members who do not respond by telephone, or whose disabilities prevent them from being able to complete the interview via telephone. However, before conducting an in-person interview, we will attempt to use TTY, computers, and other technologies that might enable an interview without field follow-up, similar to the procedures used for the 12-month data collection. It is important that we conduct follow-up interviews at the appropriate interval following random assignment, which is 36 months or shortly thereafter. Given that the sample intake period will be over a long period for most projects, the number of in-person interviews required per month at a site may be too few to justify the cost of computer-assisted personal interviewing (CAPI) data collection. Thus, MPR expects to use more cost-effective in-person data collection methods, such as providing the field interviewers with cell phones they can use to call in to MPR. For youth who decline to participate, MPR will identify why they are reluctant to participate and will send a letter that addresses their concerns and encourages participation.
c. Study Procedures for Process Visits and Focus Groups
A major source of information for the process analysis will be two comprehensive visits to each random assignment project. The exact timing of the process visits to a specific project will depend on how long youth enroll in the project as well as the duration of intervention services. However, we expect that the first visit to most projects will be within the first two years of demonstration startup (that is, the start of random assignment), and the second visit will be approximately a year later.
Staff Interviews. During the site visits, the evaluation team will conduct individual and group interviews with management and staff of various stakeholders in the local YTD project such as the following:
Project directors and site managers will offer insights into the history of each project’s sponsoring organization and its experience in serving youth with disabilities; an overview of the conception, development, and implementation of the program model and the organizational and management structure for the project, including the project budget and key project partners; and the roles and qualification of staff members, their caseloads, and the supervisory structure of the primary service providers. We designed interviews at this level to highlight some of the major challenges that service providers have encountered.
Project line staff, who are in direct contact with the youth being served, will provide insight into how the youth are identified and recruited, the methods used to assess a youth’s needs and the project’s approach to serving them, the way appropriate services are selected and delivered, and the extent to which youths’ families are involved with project services. These staff will also provide insight into how much structure or flexibility staff members have in performing their jobs, the extent to which clients’ experiences diverge from the program model, and the reasons behind such variation.
Staffs of partner organizations can provide information on linkages between the project and other services providers as well as on the successes or challenges of the collaborations. They will provide perspectives on the nature of the agreements, how effectively they function, and the ways in which project services complement or are integrated with the services of partner organizations. These might include interviews with the staff of direct service partner organizations as well as with the staffs of schools, vocational rehabilitation agencies, mental retardation and developmental disabilities agencies, and other agencies that serve persons with disabilities.
Staffs of local SSA field offices can provide insights into the broad context of services available in the area and the local implementation of SSA’s waivers for YTD participants.
Separate protocols will be developed to provide structure for the each of the types of data to be collected during the site visits. We will create a master protocol that will include the items to be covered during the visits and will identify their relationships to the objectives or key questions for the process analysis. Items from the master protocol will be selected, tailored, and used with appropriate follow-up probing and elaboration depending on the specific project and the person being interviewed. Similarly, we will create focus group guides, as well as structured protocols to record data from case reviews and observations of project activities.
Focus Groups with YTD Participants. To capture critical qualitative information about the experiences of YTD participants (and where relevant, their families), we will conduct two focus groups in each project with participating youth and their families. These focus groups will be discussions to gather information on participants’ experiences while participating in the project and their awareness and utilization of services. The focus group discussions will cover the perceived quality of project services, perceptions of gaps in activities or services, and how the SSA waivers were explained and offered to participants. Each group will include 8 to 12 youth or parents. The focus groups will complement the information collected in the follow-up surveys, providing a more in-depth and qualitative understanding of their experiences. They will help the evaluation team assess whether and how the projects did or did not meet participants’ expectations. We will also try to conduct a focus group in each project with treatment group members who did not participate in services to understand their reasons for nonparticipation. Project staff members will recruit youth and parents to participate in the focus groups. The discussions will be held at project facilities that are well known in the community and are accessible to persons with disabilities.
d. Statistical Power/Precision Estimates
For this evaluation to be useful to policymakers, it needs to have a sample that is large enough to allow us to detect policy-relevant impacts. The design of the YTD evaluation calls for the random assignment of 880 youth with disabilities to either a treatment or a control group for each of six projects. Table B.2 presents estimates of the minimum treatment-control differences that we could detect for three types of outcomes that the evaluation will examine. First, for outcomes that can be expressed in binary terms, such as the likelihood of becoming employed or of leaving the SSI rolls, we present estimates for outcomes centered on 50 percent (the most conservative assumption), as well as on 30 or 70 percent. Second, we examine annual earnings based on SER data. Third, we consider monthly SSA benefit amounts. The earnings and benefit outcomes will be critical in determining the cost effectiveness of YTD services. We presented the minimum detectable treatment-control differences for these outcomes under the assumption that we use a two-tailed test and 90 percent confidence levels to determine impacts. The table shows minimum detectable differences at 80 percent power (that is, the ability to detect true differences 80 percent of the time). We assume a reduction in variance of 10 percent owing to the use of regression models.
The numbers in the table indicate that, with sample sizes of 480 treatment group members and 400 control group members, we could detect impacts on employment and benefit receipt of 7 to 8 percentage points, impacts on earnings of $489 annually, and impacts on SSI benefits of $42 per month. For example, if the likelihood of being employed one year after random assignment were 30 percent in the absence of YTD services, and if YTD services raised this to 38 percent, then we would have an 80 percent chance of detecting this impact with our sample.
We confirm the adequacy of samples of 480 treatments and 400 controls by several studies of people with disabilities. For example, we use the evaluation of the Transition Employment Training Demonstration study, which used samples of about 375 recipients each in the treatment and control groups. The study estimated that transitional employment services for SSI recipients with mental retardation increased earnings during the second year after random assignment by $835 and the probability of employment increased at the end of that year by 12 percent. Similarly, the evaluation of the Structured Training and Employment Transitional Services demonstration, which targeted youth with mental illness, found an increase of more than 9 percentage points in employment for treatment group youth 15 months after random assignment.
TABLE B.2
MINIMUM DETECTABLE IMPACTS FOR THE YTD
EVALUATION,
ASSUMING INDIVIDUALIZED RANDOM ASSIGNMENT
|
Employment Rate or SSI Receipt Rate |
|
|
|
Sample Size (Treatment/Control) |
50 Percent |
30 or 70 Percent |
SER Annual Earnings
|
Monthly SSI Benefits
|
Full Sample |
|
|
|
|
480/400 |
8.0 |
7.3 |
$489 |
$42 |
Subgroup Sample |
|
|
|
|
240/200 |
11.7 |
10.3 |
$690 |
$60 |
Note: The calculations assume (1) a 90 percent level of confidence for a two-tailed test and an 80 percent level of power, (2) a standard deviation of $267 for the monthly SSI benefits amount and $3,069 for annual earnings, and (3) a reduction in variance of 10 percent owing to the use of regression models. We derived the standard deviations from Mathematica’s Ticket to Work Evaluation Summary Earnings Records data and SSI benefits data for youth ages 18 to 25 in 2001.
3. Methods to Maximize Response Rates
We will use the following procedures to maximize the response rates.
Effective and targeted advance materials
Collecting and updating of contact data for the sample member at 12 months for someone who would know how to reach the sample member at the 36-month interview.
Multiple methods for tracking and locating beneficiaries, including the use of extracts from SSA administrative data to capture address updates during the course of the survey, the use of an independent vendor providing commercially available contact information, and MPR’s internal respondent tracking efforts
At the 12- and 36-month follow-up interviews, the use of a combination of telephone and in-person interviewing to maximize our ability to contact sample members
Interviewer training that includes instruction on motivational interviewing, that stresses the importance of respondent cooperation, and that develops interviewer skills for averting and converting refusals
Interviewer training on when and how to select an appropriate proxy to conduct an interview
A bilingual module to help bilingual interviewers assess whether to conduct an interview in Spanish or English and to cover differences in dialects
Protocols for breaking off and then resuming interviews to accommodate beneficiaries who may become fatigued during the interview
The focus of all respondent materials (letters, brochures, and consent forms) will be to secure cooperation through the clarity, simplicity, and thoroughness of the materials, which we will write at a sixth grade reading level. Despite this, locating participants will be a challenge. While SSA has contact information for all current beneficiaries, that information is not always accurate, and at follow up some sample members will no longer be receiving benefits from SSA. Telephone numbers can be particularly problematic because there is no administrative reason to keep them updated in SSA records. Addresses are more reliable because they sometimes use them for mailing checks. However, these might be post office boxes, addresses of guardians, financial institutions, or other individuals and organizations that may be of only limited use in locating a beneficiary. Further, since many beneficiaries now receive their checks via direct deposit, SSA address information is less accurate now than it once was.
To improve the contact information, we will mail an advance letter to each sampled person prior to each survey, using the most recent address of record. The letter will describe the survey, provide a toll-free number to contact Mathematica, and indicate that we will contact the beneficiary regarding it. We will send the letter “address service requested,” which results in (1) the mail being forwarded to recipients who have a forwarding address and (2) a notice of the new address being sent to the sender. If the forwarding authorization has expired, we return the letter to the sender with the new address attached.
When an address is available but a phone number is not, we will conduct a directory search to obtain a number. For cases where neither SSA records nor the directory search yields a telephone number, MPR will use alternative locating strategies, including online nationwide databases to verify or update addresses and other information. During the 12-month interview, we request the name, address, and telephone number of two people who are likely to know how to contact the sample member in the future. If we lose contact with the sample member, we will contact these individuals to obtain the sample member’s most recent information. At the 36-month follow-up, if locating contacts are exhausted and no current phone number is available, we will conduct a field search, starting with any available information. This will usually involve a contact with the addressee for the beneficiary’s monthly check, which may be the beneficiary or their representative payee. If the addressee is not the beneficiary, we would expect that individual to have the contact information that we are seeking. Some sources might be reluctant to provide that information, and in such an instance, we would ask the source to pass on a written request to the beneficiary to send us the information on a postage-paid card, to call a toll-free number, or to contact us by email.
When a phone number is available or we obtain it, we will attempt to contact the beneficiary by telephone to conduct the interview. We will make attempts on different days and times. If we make successful contact and the beneficiary consents to the interview, we will conduct the interview using CATI technology. As indicated above, we will make multiple accommodations to increase response and encourage participation by sample members in the interview. For respondents who are deaf or hard of hearing we will use amplified telephones, TTY, and Relay technologies. For respondents who speak Spanish, advance materials will be available in Spanish, and we will develop a Spanish-language version of the survey instruments that we will administer by Spanish-speaking interviewers. We will use interpretation services for other non-English speakers. For respondents who fatigue easily, we will use structured checkpoints during the interview so that interviewers can assess whether a respondent is becoming too fatigued to continue with the interview and schedule a convenient time to complete the interview. A ten-dollar post-paid incentive at baseline and after each follow-up interview will keep sample members engaged over time.
4. Tests of Procedures
The procedures for the 36-month follow-up are identical to those on the 12-month interview. Most questions in the 36-month follow-up questionnaire appear on other studies of youth or persons with disabilities. These include our 12-month interview National Longitudinal Transition Survey (NLTS), the National Beneficiary Survey (NBS), the Short Form 12 (SF12), the Canadian Youth in Transition Survey (YITS), Opening Doors, and the Job Corps Evaluation (JC).
In addition, we conducted nine telephone pretests among youth with disabilities selected for the pilot programs. We used the pretest to evaluate the clarity of the questions asked, identify possible modifications to either question wording or question order that could improve the quality of the outcome data, and estimate respondent burden. We scrutinized new questions (not used in previous, similar surveys) especially closely during the pretest. The interviews took, on average, 50 minutes to administer.
5. Statistical Consultants and Persons Collecting and Analyzing the Data
Mathematica Policy Research, Inc. (MPR) is conducting this study, including collecting and analyzing the survey data, under contract to SSA (Contract No. SS00-05-60084). MDRC is a subcontractor to MPR on this study. Thomas Fraker of MPR (202-484-4698) is the project director and has overall responsibility for the project. Anu Rangarajan (609-936-2765) and John Martinez of MDRC (212-340-8690) are the principal investigators. Karen CyBulski (609-936-2797) and Anne Ciemnecki (609-275-2323) direct the data collection effort. Joyanne Cobb of SSA (202-358-6509) is the technical Project Officer.
1 Our definition of at-risk youth includes denied child SSI applicants age 16 to 25 and youth with serious emotional disturbances age 14 to 17.
2 All seven of the existing projects and the three new ones selected for the random assignment study will be included in a process study of the implementation of YTD. That study will include discussions with project staff and service providers.
| File Type | application/msword |
| File Title | MEMORANDUM |
| Author | Dawn L. Patterson |
| Last Modified By | 177717 |
| File Modified | 2009-05-21 |
| File Created | 2009-02-19 |
© 2024 OMB.report | Privacy Policy