Response Memo

MEMORANDUM

1100 1st Street, NE, 12th Floor

Washington, DC 20002-4221

Telephone (202) 484-9220

Fax (202) 863-1763

www.mathematica-mpr.com

TO: Michel Smyth

FROM: Anu Rangarajan, Julita Milliner-Waddell, and Jillian Berk DATE: 9/13/2012

CBRA #015

SUBJECT: Responses to OMB Questions about the OMB Supporting Statement for the Evaluation of the COBRA Subsidy Provision in the American Recovery and Reinvestment Act of 2009

During a teleconference call on September 5, 2012, OMB staff requested additional information about four topics related to the OMB Supporting Statement for the Evaluation of the Consolidated Omnibus Budget Reconciliation Act (COBRA) Subsidy provided by the American Recovery and Reinvestment Act of 2009 (“the COBRA study”). The four topics are as follows:

1. Additional information on the Interactive Voice Response (IVR) system.

2. Justification of the response rate assumptions.

3. Discussion of incentive payments.

4. Discussion of the limitations of the quasi-experimental design.

A. Additional information on the Ivr system

Using computer-assisted telephone interviewing (CATI) as the only mechanism for screening the high volume of sample members (an estimated 22,000 to 26,000 interviews) would be cost prohibitive and would require an extended field period. In addition, it only offers one mode for responding to the initial screener. As a result, we have proposed using an Interactive Voice Response (IVR) system to screen sample members for eligibility to save costs and complete the survey in a timely manner. Screening using a web-based system was also considered, but it introduced high programming and maintenance costs and does not provide the ability to immediately transfer eligible sample members to a live interviewer. We believe that the combination of IVR and CATI for screening provides the best strategic mix for boosting response and reducing burden. The IVR screener can be completed at any time (24 hours per day/7 days per week), and requires approximately two minutes to complete.

Each sample member will receive a targeted invitation letter that references the purpose of the study, the sponsor, their separating employer and job separation date of interest, and the potential to receive an incentive payment if they are deemed study eligible. The letter will explain the two ways to screen for eligibility, provide toll-free telephone numbers for both screening options, and explain the differential incentive offer. A unique personal identification number for each sample member will also be provided.

When a sample member calls into the IVR, they will hear a brief pre-recorded introduction to the study and instructions about using their touch tone telephone to respond to the IVR, including pressing the star button to be transferred to an interviewer at any time.

Respondents will first be asked to enter information to confirm their identity— the unique personal identification number provided in their letter and the last four digits of their social security number. Once their identity is confirmed, they will immediately be asked to enter their telephone number in case of connection problems. This will help us to reach them in the future, if needed. The respondent will then answer as few as two questions (if eligible) or at most four questions (if not eligible). Respondents who pass this preliminary screening will be transferred to an interviewer at our Survey Operations Center (SOC) for a more refined screening and survey completion. If the screener is completed outside of SOC operating hours, the sample member will receive a message that an interviewer will call them back when the SOC re-opens, and will be asked to enter the best time when they can be contacted. The SOC will also receive daily data transfers from the IVR system that will allow for identification of sample members who completed the screener and were deemed eligible, but did not connect with an interviewer, as well as respondents who began but did not complete the screener. Interviewers will use this information to contact eligible sample members to try and complete interviews. If someone does not complete the screener interview, they will be sent a reminder postcard urging them to do so to take advantage of the potentially higher incentive offer.

By automating the preliminary screening and thus reducing to some extent the number of respondents who require in-depth screening, the IVR option will help us manage the large volume of screening interviews necessary. Other studies have found that IVR is an efficient method for data collection. Kashner et al. 2009 used IVR to collect healthcare utilization and cost information, and Kim et al. 2007 used an IVR to screen for depression among pregnant women. IVR may also increase response rate among some sample members who prefer this option and might not respond to interviewer-initiated contact attempts by giving them one more option to respond to.

We plan to closely monitor respondent use of the IVR system, tracking the characteristics of respondents and monitoring data quality. We can use data generated by the study to understand if the characteristics of respondents who use IVR differ from the respondents who directly call the SOC. This analysis will allow us to add to the body of knowledge on a relatively new technology.

B. justification of our response rate assumptions

The response rate assumptions set forth for the COBRA study were based on our experience on the Accelerated Benefits (AB) Demonstration. For the AB Demonstration conducted for the Social Security Administration (OMB No. 0960-0747), Mathematica undertook a large screening effort to identify persons who would be eligible to participate in the demonstration. While the stakes were higher for that demonstration—a chance to receive health insurance benefits for the uninsured—sample members were not told the criteria for eligibility until they called in. Similarly, sample members for the COBRA study, will not know the criteria for their eligibility until they call in and complete the screener. Although the stakes of the demonstration are lower, we anticipate that the offer of the $50 incentive for eligible completers will encourage sample members to call in.

The screening for AB was conducted using CATI and was much more complex than the simplified two-minute screener that is proposed for the COBRA study. In AB, Mathematica had a total sample size of 21,109 of which we were able to contact 17,512 sample members (83 percent contact rate). Of those contacted, 17,010 completed their screening interview (97 percent cooperation rate). Of those screened, 1,979 beneficiaries were uninsured and therefore eligible for the demonstration (12 percent eligibility rate), and 1,952 of the eligible sample members consented to participate (99 percent consent rate) and 1,939 completed their interviews and were randomly assigned (99 percent response rate). The screening rates achieved on the AB study show that high call-in rates are feasible among targeted populations and that once they are deemed eligible, survey completion rates among eligible sample members will be high.

C. discussion of incentive payments

Our incentive offer strategy for the COBRA study is designed to maximize the response rate, encourage early response, and motivate sample members to use the cost-saving IVR to complete eligibility screening. The amount of $50 is based in large part on our experience with the TAA Experiment conducted for DOL. We believe that this offer will appeal to more people and produce timelier responses.

We are requesting OMB clearance to offer a payment of $50 to respondents who use the IVR for initial screening and complete the survey within four weeks of receiving the invitation; $40 will be offered to IVR completers who complete the survey more than four weeks after receiving the invitation. For respondents who are determined eligible for the study through screening completed by an interviewer (call-ins and call-outs), we are seeking clearance to offer $40. We believe that this differential offer is needed to promote use of the IVR over CATI. Mathematica has successfully used differential incentives on two rounds of the National Survey of Recent College Graduates (NSRCG) conducted for the National Science Foundation.

D. discussion of limitations of the quasi-experimental design

The study’s sampling approach is designed to yield, in a cost-effective manner, a national probability sample from the study populations and periods in the UI sample frame. This approach will enable estimation of characteristics and impacts of the subsidy for the populations described in the previous section. Study estimates and measures of their precision will be design-based—that is, based explicitly on the sampling design and the sampled populations. However, the findings will be indicative of the potential effects of similar programs implemented on similar populations at some future time.

The study’s estimates of the impact of the ARRA subsidy on COBRA take-up will be based on a quasi-experimental design. The limited eligibility period of the ARRA COBRA subsidy provides a nice natural experiment that we can use to look at the impact of the subsidy on take-up. We will compare outcomes for a sample of subsidy-eligible individuals to a sample of otherwise similar individuals who were not subsidy-eligible. People in the latter group, called the subsidy comparison group, resemble subsidy-eligibles because they experienced an involuntary job termination and were not eligible for another group health insurance plan, but their date of job loss did not occur during the qualification period. We will use state-of-the-art propensity score matching methods to select the study comparison group ensuring that the treatment and comparison groups are as similar as possible. Even with the natural experiment, a quasi-experimental study will have limitations. While the analysis will ensure that the two groups appear similar on observable characteristics, including demographics and pre-job loss characteristics, subsidy-eligible workers and the subsidy comparison workers lost their jobs during different time periods. Our analysis will control for the economic conditions, but unobservable differences between the two groups of workers may still bias the impact estimate. We will conduct extensive sensitivity tests to examine the robustness of our findings and explore potential sources of bias.

References

Kashner, T.M., H. T. Madhukar, A. Wicker, M. Fava, J. Greist, J. Mundt, K. Shores-Wilson, A. J. Rush, and S. Wisniewski. “Voice Response System to Measure Healthcare Costs: A STAR*D Report.” American Journal of Managed Care, vol. 15, no. 3, March 2009, pp. 153-162.

Kim, H., Y. Bracha, and A. Tipnis. “Automated Depression Screening in Disadvantaged Pregnant Women in an Urban Obstetric Clinic.” Archives of Women’s Mental Health, vol.10, no. 4, June 2007, pp. 163-169.

cc: Jonathan Simonetta

An Affirmative Action/Equal Opportunity Employer