AB Project Addendum to the Supporting Statement

ACCELERATED BENEFITS DEMONSTRATION PROJECT

ADDENDUM TO THE SUPPORTING STATEMENT

ISSUES OF SPECIAL INTEREST TO OMB

NOTE: SSA and its contractors conducted several briefings with OMB about this project. During the last briefing, OMB raised several issues of particular interest to them. To help ease OMB’s review of this study, SSA is addressing these issues in this addendum.

1. Explain the reasons for removing the pilot survey, being specific about the time and other restrictions which prevent its use. Discuss the types of information we could have obtained from the pilot, whether they will be obtained at other points during the project, and the potential impact of not collecting this information.

When we published the federal register notice for this project on January 8, 2007, we had planned to conduct a pilot of the baseline survey to collect information on the recruitment process and the characteristics of likely study participants. We have since decided not to conduct the pilot. Instead, at the request of the Social Security Administration, we will begin recruitment for the demonstration in July and use the early months of recruitment to learn what would have been learned through the pilot study.

The proposed pilot would have resulted in 250 baseline surveys completed by uninsured beneficiaries. The pilot would have provided information about a number of factors that will affect our ability to recruit individuals for the study, including (1) the proportion that can be contacted by phone, (2) the proportion that will be willing to answer questions to determine whether they have insurance, (3) the proportion who are uninsured and therefore eligible for the demonstration project, and (4) the proportion of eligible participants who would complete the baseline survey.

We now plan to collect similar information during the first phase of recruitment. During that first phase, we plan to recruit 20 new individuals in each month from July through September. As discussed in Section B of the supporting statement, this first phase will provide reasonably precise estimates of each of the quantities mentioned above. In addition, by beginning recruitment without a pilot, we will also learn whether eligible beneficiaries will provide consent to be in the study and to be randomly assigned. We would not have received credible information on this matter through the pilot study.

The pilot would also have given us information on characteristics of uninsured new DI beneficiaries, particularly impairment type. We no longer believe that information is needed before we begin the intervention. Information we have received from SSA since we filed the federal register notice suggests the sample will include a broad range of impairments. We are therefore now planning an intervention that will be appropriate for individuals with many different impairments. Information from the pilot study would therefore not have been used to design the intervention.

2. Discuss the differences between the early-use survey and a pilot, and why we cannot use the early-use survey as a sort of small-scale pilot.

In other MDRC studies, a pilot has often been conducted at the beginning of the study to ensure that recruitment is proceeding as expected and the intervention is being delivered as designed. If early results from the pilot indicate some problems with either recruitment or implementation of the intervention, those can be modified and further studied before we begin recruiting the evaluation sample.

We will be conducting a similar assessment of the intervention during the three-month initial phase of recruitment, and we will modify recruitment or the intervention as needed. If we find that the intervention is not being delivered as intended, we would not include the sample enrolled during the first phase in conducting the impact analysis.

The primary difference between this assessment and a formal pilot is that the early enrollment phase will be shorter than a formal pilot, and we will have less time to make adjustments before the second phase begins. This short period is needed to stay within the project’s overall five-year schedule. In addition, if the intervention is delivered as expected, we will use this portion of the sample in conducting the impact analysis, while the sample recruited through a pilot is typically not used in subsequent research.

We also plan to conduct an early-use survey with 480 study participants about six months after they enter the study. The primary purpose of this survey will be to compare health insurance status and health care use between the three research groups (AB Plus, AB Basic, and control). This will help us interpret findings on health outcomes, employment, and benefit payments that are derived later using the 15-month follow-up survey and administrative records. If, for example, we see no effects on the later outcomes, it would be important to know whether individuals were using more health care because of the intervention or, by contrast, whether the control group found other forms of health care coverage and health insurance. Although we plan to collect information on health insurance status and health care use in the 15-month survey, we are concerned that respondents will be unable to reliably remember this information going for a 15-month period going back to random assignment.

The early-use survey will also ask members of the AB Plus about their experience with the care manager, with the PGAP program, and with the AB Plus employments and benefits counselors. If the surveys uncover substantial problems, this might lead us to modify the intervention. However, this is not the primary purpose of the early use survey, nor will it be the primary source of information on whether the intervention is being delivered well. The primary source of that information will be the assessment during the first phase of enrollment and the process study.

In order to use the early-use survey as a type of pilot, we would have to stop recruitment after the early-use sample is recruited, conduct and analyze the early-use survey, modify the intervention if necessary, and then begin recruitment again. This process would not be feasible because it would run up against the project’s end date and budget. Recall that the project is scheduled to end in January 2011. The early-use survey will be fielded with all individual recruiting during the first six months and it will be fielded six months after individuals enter the study. We therefore do not expect results to be available until about 18 months after the beginning of recruitment, or January 2009. This would only leave two years (from January 2009 through January 2011) to complete recruitment, deliver the intervention to the remaining sample, and collect follow-up data. Since individuals can receive the intervention for up to two years, there would not be enough time to complete all evaluation activities. In addition, the project does not have a big enough budget to provide benefits to a substantial number of people who would be in a pilot but who would not be included in the impact analysis.

3. Discuss how the findings of the first few months of the baseline survey could impact methodology in later parts of the project.

As discussed above, we plan to randomly assign 20 individuals per month in July, August, and September 2007 and assign the remaining 1,940 sample members from October 2007 through September 2008.

The estimates derived form the first phase of enrollment will be used to determine how many metropolitan areas to include in the second phase of recruitment. If response rates are high and insurance rates are low, we might be able to recruit the sample from 10-15 metropolitan areas. If the response rate is lower or the insurance rate is higher, we might have to enroll participants from 100 metropolitan areas or more.

In addition, results from these first few months might lead us to alter our recruitment methods to increase our ability to find potential participants, screen them for eligibility, and conduct the baseline survey. For example, if we are able to locate enough people, but they are unwilling to respond to the baseline survey, we might revise our respondent materials such as the answers to Frequently Asked Questions, to more effectively address sample members’ concerns about participating. It’s likely that we would revise our interviewer training materials and conduct booster training sessions to further refine our approach. If we find that beneficiaries are unwilling to participate because they doubt the validity of the study, we might undertake outreach to local SSA field offices to try to increase knowledge about the demonstration project. If we find that many of the phone numbers are unusable, we might propose in-person tracking of some individuals. Because each of these options would increase the cost of conducting the baseline survey beyond the project’s budget, we have not proposed them at this point.

The other purpose of the first phase of enrollment is to gradually roll out the intervention. During these three months, we will closely monitor how the intervention is being delivered through regular phone calls with POMCO and CareGuide and by monitoring the use of health care by analyzing claims filed through POMCO. Our goal will be to resolve any early implementation difficulties before phase two of enrollment begins. We will also monitor take up of the health care plan. As discussed in the supporting statement, individuals assigned to the AB Plus and AB Basic groups will be sent a description of the health care plan and asked to sign and return a form that acknowledges they understand the plans and its limitations. If we find that many individuals are not returning these forms, we will investigate additional means of encouraging them to return the forms, such as incentives for doing so.

We plan to resolve these issues before the second, more intensive phase of enrollment begins. If substantial changes are made to the intervention, we would consider dropping the 60 individuals enrolled during the first phase in conducting the impact analysis. That would leave us with 1,940 study participants and cause only slight reductions in the statistical precision of the impact estimates.

3. Discuss how the specific intervention approaches were developed, being specific about the types of health care treatment included for the intervention group.

Here is a brief description of how the intervention was developed. The intervention has two components: (1) health care benefits, which will be offered to both the AB Basic and AB Plus groups, and (2) additional services offered to only the AB Plus group. The additional services in AB Plus include care management to help participants make appropriate decisions about their health care, a behavioral component to help motivate individuals to want to return to work, and employment and benefits counseling.

In developing the health care plan, we relied heavily on the expert panel described in Part A of the supporting statement. The panel includes a variety of experts: two individuals with disabilities who represent major advocacy groups; the president of an organization that provides benefits counseling and other services to people with disabilities; several experts in the design of health insurance and health care plans; several economists to provide advice on the likely costs of the plan; and a physician and psychologist working with people with disabilities. Advice from the expert panel was supplemented by the expertise of several members of the team, including a psychiatrist and physician at Group Health Cooperative in Seattle and health experts at the subcontractors that will be administering health care benefits and providing care management.

The expert panel met on three occasions. On the first occasion, they offered opinions about the general characteristics of the health care benefits. At the second meeting, they responded to several detailed options, and those options were amended to take into account the panel’s recommendations. The panel reviewed two final options at the third meeting and recommended we go with one of the two options. That option was designed to look similar to a Medicare Advantage plan in that it covers all standard types of health care and medications within a provider network (Multiplan), subject to modest copayments (for example $12 for a physician visit). In addition, the expert panel recommended that the health care plan include vision and dental benefits since these might be major impediments to the return to work for some groups of SSDI beneficiaries. Since some types of care might be difficult to find in the proposed network – particularly mental health treatment – and because some study participants might have already established relationships with health care providers that are not part of the proposed network, the panel recommended that the plan cover some out-of-network providers. Finally, they recommended that the plan help pay for devices that might be needed by certain beneficiaries, such as sturdy wheelchairs and scooters. Each of these recommendations was incorporated into the plan.

The original vision of AB Plus was that it would include care management to help people make appropriate health care choices. In early team meetings, our consultants from GHC suggested that helping study participants return to work would depend at least as much on motivation as on receiving appropriate health care. They recommended we include in our expert panel Dr. Michael Sullivan, a professor of psychology at McGill University and founder of the PGAP program, which is designed to motivate people with lower back pain to return to daily activities and ultimately return to work. When we met with our expert panel, we received unanimous endorsement of the idea that motivation would be key to success of the intervention. This idea was especially endorsed by the two panel members who are advocates with disabilities. We therefore added a version of PGAP to the AB Plus intervention.

The original version of AB Plus was designed to connect study participants with employment supports and benefits counseling offered through local SSA field offices. In our early discussions with SSA and the expert panel, however, we became convinced that local field offices were not well enough staffed to provide extra counseling to AB study participants. We therefore contracted with Trans Cen, Inc. to provide employment and benefits counseling.

Because of the excellent advice we have received from the expert panel, expert consultants, and subcontractors, we decided not to conduct the focus groups that were described in the federal register notice in November 2006. In particular, the two disability advocates reached out informally to members of their organizations, and we believe that outreach serves the same purpose as the focus groups. We also decided not to conduct the focus groups because doing so would have pushed the project beyond its five-year horizon. After the study began, SSA began to require stricter data security from its contractors, and this delayed our ability to obtain names of beneficiaries who might be appropriate participants in the focus groups. As a result, we were not able to conduct the focus groups early enough to help develop the intervention while still staying completing the project’s within five years.

3. Disclose the findings of the IRB on the final survey design.

In September 2006, MDRC’s IRB reviewed the baseline survey and the plan for acquiring consent for individuals to be in the study. They approved the survey and agreed that the demonstration project presents minimal risk, which is a requirement for them to waive written consent. Before the IRB formally agrees to waive written consent, they asked the team to make a formal presentation regarding the intervention and recruitment process. This presentation will take place at the next meeting of the IRB in early June, although we will try to maximize the chance the IRB approves our plan by sharing materials with the chair of the IRB before then. After the IRB has made a decision, we will share that decision with OMB. Since we expect this to happen in early June, it should take place in time to receive OMB approval for a July start to recruitment.

3. Discuss pretesting of the consent script and process for providing OMB with a revised script.

An important element of the enrollment process is the consent script. The consent script has to communicate the goals of the research and the implications of enrolling in the study in a way that is easily understood by sample members. When the consent is being obtained by telephone, as in AB, the language has to be especially clear and concise. It also helps if the script has built in stops where the interviewer pauses and provides the sample member an opportunity to ask questions.

We developed a consent script with these considerations in mind, and tested it internally involving a small number of MPR staff members. The feedback received from both respondents and interviewers was that the script is too long. Members of the AB expert consultant group, especially those who have disabilities, expressed similar concerns about the length of the script, and noted that some of the wording could raise unnecessary concerns among sample members.

The early feedback suggested that further testing would help identify the optimal balance between the keeping the script short to avoid break-offs and describing the study in sufficient detail to allow sample members to make informed decisions about participating. We are currently in the process of recruiting participants for cognitive interviews that will focus on their reactions and examine their understanding of the consent script in further detail. While we would have preferred to use SSA’s administrative records to select a pretest sample with comparable characteristics to that of our target population, delays associated with revised data security requirements have made this unfeasible. We are, instead, focusing on recruiting a convenience sample of current SSDI beneficiaries, and will use their input to assist us in revising the consent script.

We expect the revisions to focus on increasing the clarity and efficiency of the wording. No substantive changes that could alter the meaning of the consent script are expected. We will share a revised version of the consent script with OMB as soon as it is available, but no later than April 30, 2007. Members of MDRCs IRB will review the revised consent script prior to its submission to OMB, although it will not receive formal approval from the next full Board meeting in early June.

Accelerated Benefits Demo Project Addendum

2/5/2021

7