Findings from a Field Test Experiment on a New Approach

Attachment L

Findings from a Field Test Experiment on a New Approach to

Measuring Health Insurance in the Current Population Survey

Joanne Pascale and Amy Steinweg

US Census Bureau

4600 Silver Hill Road Suitland, MD 20233

[email protected]

September 26, 2011

Keywords: CPS, ACS, health insurance, measurement error, redesign, field test

1. Introduction

The Census Bureau’s Current Population Survey Annual Social and Economic Supplement (called the CPS hereafter) is a key data source for health insurance estimates, but under-reporting of coverage has been a persistent problem, prompting research on improving the data quality of the CPS (DeNavas-Walt et al, 2011). New questions are also arising regarding the relative data quality across surveys and interpretation of the estimates as the American Community Survey (ACS) now collects data on health insurance, and estimates were released for the first time in fall of 2009. In an attempt to understand and reduce measurement error associated with these surveys, a series of research projects has been underway at the Census Bureau, the most recent component of which is the Survey of Health Insurance and Program Participation (SHIPP) – a split-panel field test of three different question series, each designed to measure health insurance coverage. Two of the panels mimicked the health insurance questions from the CPS and ACS, and the third panel included an experimental questionnaire on health insurance coverage (the “Redesign” or EXP for short). The EXP was developed primarily to reduce measurement error in the CPS and the focus of this report is limited to that comparison. Future reports will examine differences between the EXP and ACS, and differences between the ACS and CPS.

Past research has indicated that particular survey design features of the CPS are associated with measurement error, and among them the calendar year reference period has probably received the most attention (Bhandari, 2004; Bennefield, 1996; Davern, 2009; Lewis, Ellwood and Czajka, 1998; Marquis and Moore, 1990; Ringel and Klerman, 2005; Rosenbach and Lewis, 1998; Swartz, K., 1986). Results from cognitive testing of the CPS show that some respondents ignore the calendar year reference period and instead report on their current status or their most recent spell of coverage, and that those with recent coverage are more likely to report accurately than those with coverage in the more distant past (Pascale, 2008/2009), and related research shows similar results (Resnick et al, 2004; Lynch, 2006). Nevertheless, providing data on calendar year coverage is a goal of the CPS. Thus the EXP takes a new approach to questions on time period of coverage, beginning by asking about current coverage status, and then asking about duration of coverage (at the month-level) during the past calendar year.

The household-level CPS design has also been shown to risk underreporting for certain household members (Hess et al, 2001; Pascale, Roemer and Resnick, 2009), and yet a person-level design lengthens the survey, inducing respondent fatigue and underreporting (Blumberg et al, 2004). The EXP employs a hybrid approach. It begins by asking questions at the person-level and if a particular plan type is identified, questions are asked to determine whether other household members are also covered by that same plan. Subsequent people on the roster are then asked about by name, one at a time, and for those who had been reported as covered under a previously-reported plan, that coverage is simply verified and a question is asked to determine if they had any additional plans.

A third problematic feature of the CPS is the way in which plan type is determined – through a series of eight fairly detailed questions on source of coverage – which often challenges respondents’ sometimes limited knowledge of the complex maze of health insurance plans and programs (Cafferata, 1984; Cantor et al, 2008; Davern et al, 2008; Loomis, 2000; Pascale, 2009c; Roman, Hauser and Lischko, 2002; Walden et al, 1984). This routine may also contribute to the persistent problem of Medicaid under-reporting (Blumberg and Cynamon, 1999; Call et al, 2008; Card et al, 2001; Eberly, Pohl and Davis, 2008; Klerman, Ringel and Roth, 2005; Lynch and Resnick, 2009; Research Project (aka SNACC), 2008; Roemer, 2007). The EXP takes a different approach, first asking about any coverage at all, then identifying general source (job, government or some other way) and then following up with tailored questions to elicit the necessary detail.

Due to these measurement issues, a comprehensive research agenda has been underway at the Census Bureau for several years to both examine better ways of collecting retrospective data on health insurance coverage and, more generally, to detect other survey design features that could be contributing to measurement error. There are, however, certain fixed constraints regarding any kind of redesigned questionnaire. For example, in spite of the mounting evidence that the calendar year reference period (perhaps compounded by the 3-month lag time) is problematic, the CPS is nevertheless still charged with collecting data on the entire calendar year, and it has the constraint of being fielded in March of the subsequent year. Thus the research agenda included an exploration of ways of asking about both current and past calendar year coverage within the same set of questions. The rationale was two-fold: research suggests current status estimates are more accurate than calendar year estimates (at least those generated under current CPS methodology), and it was also hoped that a revised set of retrospective questions could improve on the calendar year estimates (Blair and Ganesh, 1991; Loftus et al, 1990). Indeed the new questions on current status may be able to be leveraged to serve as an anchor which may help elicit reports of past year coverage more accurately than the standard methodology (Crespi and Swineheart, 1982; Pascale, 2009b).

Thus far the overall research tasks have included an extensive and ongoing literature review (Czajka and Lewis, 1999; ASPE, 2005; Pascale, 1999), multiple rounds of cognitive testing (Pascale, 2008/2009, Pascale, 2003), several split-ballot experiments (Pascale, 2007; Pascale, 2004; Pascale, 2001), development of a redesigned questionnaire including both current and calendar year questions, cognitive testing of the redesign (Pascale, 2009b), a pretest of the redesign fielded in March 2009 (Pascale, 2009a) and, most recently, a large-scale split-ballot field test conducted in the spring of 2010 (the SHIPP). Results from the first several stages of this research have been reported elsewhere, as noted above. The main focus of this report is the SHIPP field test.

2. Methods Overview

The SHIPP survey was carried out from March 22 through May 10 of 2010 by the Census Bureau’s telephone interviewing staff in Hagerstown, Md., via three discreet but consecutive 10-day field periods. The survey was administered over the telephone using a CATI instrument and took an average of 17 minutes per household to complete (see Appendix A for details on the methodology). The content of the survey included basic demographics of all household members, disability, labor force participation and earnings, participation in government programs (such as Food Stamps), health insurance, a respondent debriefing, and a request for consent to link data to administrative records. The sample was drawn from two sources – a Random Digit Dial (RDD) frame and Medicare enrollment files (MCARE). The goal was to complete 3,000 household interviews from the RDD sample and 2,000 interviews from the Medicare sample. That goal was exceeded for both sample types: there were 3,081 (57%) completed interviews from the RDD sample and 2,295 from the Medicare sample. In total these 5,376 households represented 12,743 people. Because average household size was larger among the RDD sample than the Medicare sample, at the person-level 59 percent of the interviews pertained to people from the RDD sample and the remaining 41 percent pertained to people from the Medicare sample. Response rates (based on the AAPOR RR4 definition) were 47.6 percent for the RDD sample and 61.4 percent for the Medicare sample. See Appendix A for a more complete summary of the SHIPP field test methods.

3. Results

1. Demographic Profile Across Panels

Though independent samples were drawn for each treatment (within sample type and even field period), the demographic profile of respondents across treatments was different, and the extent of these differences depends on which comparisons one is making. For the CPS and EXP RDD sample, most demographic characteristics were fairly well-balanced, with the exception of race (see Table 1a). The EXP treatment resulted in more white non-Hispanics than the CPS treatment, by almost three and a half percentage points, and the CPS in turn had higher levels of both black non-Hispanics (by almost one a half percentage points), and those in the “other” race category (by more than two percentage points). The MCARE sample showed a similar imbalance on race (though the EXP had more in the “other” race category than the CPS), and there were also more Hispanics in the EXP treatment than in the CPS. The CPS also had more people under 18 and fewer people over 65 than the EXP. And finally, the CPS had more people below the household income threshold¹, more people not in the work force and fewer non-full-time workers (see Table 1b). When both the RDD and MCARE samples are combined, some of these differences are reduced (see Table 1c), though the race and employment status differences remain. While many of these demographic characteristics are correlated with key outcome measures on health coverage (such as public coverage and uninsured rates), as a first step we present preliminary results on unweighted, unadjusted estimates across treatments. Forthcoming versions of this paper will adjust for the demographic imbalances across treatments and address statistical issues involved in combining the RDD and MCARE samples.

2. RDD Sample Estimates

Overall, there were very few significant differences between estimates from the CPS and EXP panels for the RDD sample – across plan types and even within subgroups (see Table 2, excel attachment). The rate of uninsured was virtually the same (EXP was 0.01% higher than CPS) and there were no significant differences in the uninsured rate across subgroups. For public and private coverage overall, and within each plan type (employer-sponsored insurance or ESI, Medicaid, etc.) there were no significant differences except in the “other coverage” category, where the CPS estimate was 2.45 percentage points higher than the EXP. The only other notable finding is within the Medicare category, where the EXP resulted in significantly higher estimates than the CPS among those under 18 and over 65 years old, those in non-full-time employment, and those below the income threshold.

3. MCARE Sample Estimates

There were a fair number of significant differences among the Medicare sample (see Table 3, excel attachment). The overall uninsured rate in the EXP was 2.22 percentage points lower than in the CPS (and significant), and the direction of the gap was consistent across all subgroups. That is, the EXP uninsured estimate was lower than the CPS estimate for all subgroups. Among certain subgroups the CPS-EXP difference in the uninsured was particularly pronounced and significant – those 18-64, black non-Hispanics, and those below the income threshold. Among non-Hispanics the CPS-EXP gap was 2.11 percentage points and significant, and among Hispanics the gap was 9.38 percentage points but did not reach statistical significance.

For public coverage overall (Medicaid and Medicare combined) the EXP estimate was 2.34 percentage points higher than CPS. This difference did not reach statistical significance but for all subgroups the EXP estimate was higher than the CPS, and among those 65 and older, black non-Hispanics and those below the income threshold the difference was significant. For Medicaid there was virtually no difference overall (0.08 percentage points), and for most subgroups the CPS-EXP gap was not statistically significant and went in different directions – that is, for some subgroups the CPS estimate was higher than the EXP estimate, and for some subgroups the reverse was true. But for Hispanics and those below the income threshold the EXP estimate was significantly higher than for the CPS estimate. For Medicare the only significant difference was among those 65 and older, where the EXP estimate was 3.11 percentage points higher than the CPS.

For private coverage overall (ESI and directly purchased combined), there were no significant differences overall and among subgroups only one significant difference – the CPS estimate was higher for those 65 and over. For ESI coverage there were no significant differences, overall or by subgroup, and the magnitude of the differences was rather low and went in both directions across subgroups. For directly purchased coverage, however, there were a number of differences. Overall the CPS estimate was higher than the EXP, the direction of this difference was consistent across all subgroups, and for some subgroups the difference was significant – those 65 and over, non-working, and those above the income threshold.

4. Overall Sample Estimates

The RDD and MCARE samples were each drawn from different universes, so statistical inferences cannot be made with regard to significance levels. However, for purposes of examining differences by subgroup, the samples were combined to examine differences in levels and patterns of reporting.² For the most part results show similar patterns as those found for the RDD and MCARE samples (see Table 4, excel attachment). The EXP estimate of the uninsured was slightly lower than the CPS (by 0.87 percentage point), and across all subgroups the EXP estimate was lower than CPS. For some subgroups in particular the difference was especially pronounced – black non-Hispanics, those below the income threshold, Hispanics and those in non-full-time employment.

For public coverage the EXP estimate was 2.43 percentage points higher than the CPS and across all subgroups the EXP estimate was higher. The gap was especially pronounced among those 65 and older, those not in full-time employment, black non-Hispanics and those in the “other” race category, Hispanics, and those below the income threshold. For Medicaid, the CPS-EXP gap among all subgroups was less that a percentage point except for Hispanics, where the gap was 8.88 percentage points, those in the “other” race category (3.19 percentage points) and black non-Hispanics (1.25 percentage points). For Medicare the EXP estimate was 2.06 percentage points higher than CPS, and for almost all subgroups the EXP was higher (for “other” race and Hispanics the CPS was higher but by 0.06 and 0.14 percentage point respectively). For most other subgroups the EXP estimates was 3 to 4 percentage points higher than the CPS.

For private coverage overall, the CPS estimate was higher than the EXP and for all subgroups CPS was higher. For some subgroups the difference was especially pronounced – Hispanics and those 65 and over. For ESI coverage the CPS estimate was slightly higher, by 0.71 percentage point. For most subgroups differences were small except for those 65 and over, where CPS was almost three percentage points higher than EXP, and among Hispanics, where the gap was over six percentage points. For directly purchased coverage the overall difference, and among all subgroups, was very small – less than a percentage point in most cases.

4. Summary

Successful fielding of SHIPP indicated that this instrument can be used to capture multiple time points of coverage, vastly expanding the utility of the data from the current CPS module which capture’s only coverage ‘at any point in the past year’. SHIPP provided coverage estimates for current point-in-time, over a year’s worth of month-level data that could capture gaps in coverage, and of course, coverage at any point in the past year.

Estimates for the RDD sample indicate virtually no difference between the EXP and CPS designs in the uninsured rate or private coverage, and higher reporting of public coverage in the EXP for disadvantaged and elderly subgroups. For the Medicare sample the EXP results in a lower estimate of the uninsured for the overall sample, higher reporting of public coverage among disadvantaged and elderly subgroups, and virtually no change in private coverage reporting compared to the CPS. When both sample types are pooled the same general pattern emerges – under the EXP design the uninsured rate is lower overall and for all subgroups (and the magnitude of the gap is particularly pronounced among disadvantaged subgroups), reporting of public coverage is higher overall and for all subgroups (again the gap is higher among elderly and disadvantaged subgroups), and private coverage estimates are lower overall and among all subgroups (with the gap being higher among elderly and disadvantaged subgroups). These patterns suggest that the EXP design is more effective than the CPS at eliciting public coverage reporting for the subgroups most likely to be eligible for public coverage, and hence these subgroups are less likely to be misclassified as uninsured. The reduced reporting of private coverage among disadvantaged subgroups suggests there may be some degree of swapping going on – that is, under the CPS public coverage may be mistakenly reported as private coverage for certain subgroups.

The SHIPP was entirely telephone-based and did not include a cell-phone-only or face-to-face component. Individuals missed through this methodology tend to be young adults, minorities and low income individuals – in other words, people with characteristics highly associated with public coverage eligibility and being uninsured. Thus, while the patterns observed in the SHIPP experiment are promising, the observed differences would likely be more pronounced if the subgroups most affected by the differences in questionnaire design were represented properly in the sample. If these patterns of reporting do hold up under more intense scrutiny – that is, if the EXP design really does prompt more accurate reporting, specifically by capturing more public coverage and reducing misreporting of public coverage as private – it will be important to be able to disentangle methods effects from real change that can be attributed to health reform when the Affordable Care Act (ACA) is implemented in 2014.

SHIPP is designed to capture coverage even when specific plan-type is unclear to respondents- by first determining that there is coverage, and then funneling to more specific questions to piece out coverage type. This instrument structure may confer a distinct advantage in coming years as health insurance sources change in response to the ACA. Our next step is to test the incorporation of SHIPP into the broader CPS instrument to ensure it functions correctly as an integrated instrument before we switch away from the older CPS ASEC health insurance module. We anticipate instrument testing for this will be facilitated by multiple test scenarios already detailed from our last round of testing.

REFERENCES

Bennefield, R. 1996. A Comparative Analysis of Health Insurance Coverage Estimates: Data

from CPS and SIPP. Presentation at the 1996 Joint Statistical Meetings of the American

Statistical Association.

Bhandari, S. 2004. People with Health Insurance: A Comparison of Estimates from Two Surveys. Survey of Income and Program Participation (SIPP) Working Paper No. 243. Washington, D.C.: U.S. Census Bureau.

Blair, Edward A. and Gopala K. Ganesh. 1991. “Characteristics of Interval-Based Estimates of Autobiographical Frequencies.” Applied Cognitive Psychology 5:33, 237-250, Wiley.

Blumberg, S. J., and M. L. Cynamon. 1999. Misreporting Medicaid Enrollment: Results of Three Studies Linking Telephone Surveys to State Administrative Records. Proceedings of the Seventh Conference on Health Survey Research Methods, pp. 189–195.

Blumberg, S. J., L. Osborn, J. V. Luke, L. Olson, and M. R. Frankel. 2004. Estimating the Prevalence of Uninsured Children: An Evaluation of Data from the National Survey of Children with Special Health Care Needs, 2001. Vital Health Statistics 2(136). Hyattsville, Md.: National Center for Health Statistics.

Cafferata, Gail Lee. 1984. “Knowledge of Their Health Insurance Coverage by the Elderly.”

Medical Care 22:835–47.

Call, K. T., G. Davidson, M. Davern, and R. Nyman. 2008. Medicaid Undercount and Bias to Estimates of Uninsurance: New Estimates and Existing Evidence. Health Services Research 43(3):901–914.

Cantor, J. C., A. C. Monheit, S. Brownlee, and C. Schneider. 2007. The Adequacy of Household

Survey Data for Evaluating the Nongroup Health Insurance Market. Health Services Research 42(2):1739–1757.

Card, David, Andrew K.G. Hildreth, and Lara D. Shore-Sheppard. 2001. “The Measurement of

Medicaid Coverage in the SIPP: Evidence from California, 1990–1996.” National Bureau of

Economic Research, Working Paper 8514, Cambridge, MA.

Crespi, Irving and James Swineheart. 1982. “Some Effects of Sequenced Questions Using Different Time Intervals on Behavioral Self-Reports: A Field Experiment,” Paper presented at the American Association for Public Opinion Research.

Czajka, J., and K. Lewis. 1999. Using Universal Survey Data to Analyze Children’s Health Insurance Coverage: An Assessment of Issues. Washington, D.C.:Mathematica PolicyResearch, Inc.

Davern, Michael. 2009. “Unstable Ground: Comparing Income, Poverty & Health Insurance Estimates from Major National Surveys.” Paper presented at the AcademyHealth Annual Research Meeting, June 29, 2009. Chicago.

Davern, M., K. Call, J. Ziegenfuss, G. Davidson, T. Beebe, and L. Blewett. 2008. Validating

Health Insurance Coverage Survey Estimates: A Comparison of Self-Reported Coverage and

Administrative Data Records. Public Opinion Quarterly 72(2):241–259.

DeNavas-Walt, Carmen, Bernadette D. Proctor and Jessica C. Smith. 2011. “Income, Poverty and Health Insurance Coverage in the United States: 2010.” U.S. Census Bureau Current Population Report P60-239. Washington, DC.

Eberly, T., M. Pohl, and S. Davis. 2008. Undercounting Medicaid Enrollment in Maryland: Testing the Accuracy of the Current Population Survey. Population Research and Policy Review

online April 30. http://www.springerlink.com/content/d56459733gr81vu2/

Hess, Jennifer, Moore, J. Pascale, J. Rothgeb, J. and Keeley, C. (2001), "The Effects of Person level vs. Household-level Questionnaire Design on Survey Estimates and Data Quality." Public

Opinion Quarterly, Winter 2001, 65:574-584.

Klerman, J. A., J. S. Ringel, and B. Roth. 2005. Under-Reporting of Medicaid and Welfare in the

Current Population Survey. RAND Working Paper WR-169-3. Santa Monica, Calif.: RAND.

Lewis,K.,M. Ellwood, and J.Czajka. 1998. Counting the Uninsured: A Review of the Literature.

Washington, D.C.: The Urban Institute. Inquiry/Volume 45, Winter 2008/2009 436

Loftus, Elizabeth F., Mark R. Klinger, Kyle D. Smith and Judith Fiedler. 1990. “A Tale of Two Questions: Benefits of Asking More Than One Question.” Public Opinion Quarterly, Vol. 54, No. 3. (Autumn, 1990), pp. 330-345.

Loomis, L. 2000. Report on Cognitive Interview Research Results for Questions on Welfare

Reform Benefits and Government Health Insurance for the March 2001 Income Supplement

to the CPS. July 25. Washington, D.C.: Center for Survey Methods Research, Statistical Research Division, U.S. Census Bureau.

Lynch, Victoria. 2006. “Causes of Error in Survey Reports about Who in the Household Gets

Welfare.” Unpublished paper. Joint Program in Survey Methodology, College Park, MD.

Lynch, Victoria and Dean Resnick, 2009. “Misreporting Health Insurance Status: Medicaid Enrollees in the Medical Expenditure Panel Survey, 2003." Paper presented at the American Association for Public Opinion Research, Hollywood, Fla.

Marquis, Kent H., and Jeffrey C. Moore. 1990. “Measurement Errors in the Survey of Income and Program Participation (SIPP) Program Reports.” Proceedings of the Bureau of the Census 1990 Annual Research Conference, Arlington, VA.

Office of the Assistant Secretary for Planning and Evaluation, Health and Human Services. 2005. "Understanding Estimates of the Uninsured: Putting the Differences in Context." Updated September 5. (http://aspe.hhs.gov/health/reports/05/uninsuredunderstanding-

ib/#estimates);

Pascale, Joanne, Marc I. Roemer, and Dean M. Resnick. 2009. Medicaid Underreporting in the CPS: Results from a Record Check Study. Public Opinion Quarterly. 73: 497-520.

Pascale, Joanne (2009a), "Findings from a Pretest of a New Approach to Measuring Health Insurance in the Current Population Survey." Paper prepared for the Federal Committee on Statistical Methodology Research Conference, November 2-4, 2009.

Pascale, Joanne. 2009b. “Survey Measurement of Health Insurance Coverage: Cognitive Testing Results of Experimental Questions on Integrated Current and Calendar Year Coverage.” Unpublished Census Bureau report submitted to the Housing and Household Economic Statistics Branch, February 2009

Pascale, Joanne. 2009c. “Health Insurance Measurement: A Synthesis of Cognitive Testing Results.” Paper presented at the Questionnaire Evaluation Standards (QUEST) meeting, Bergen, Norway.

Pascale, Joanne 2008/2009. “Measurement Error in Health Insurance Reporting.” Inquiry

45(4):422–37.

Pascale, Joanne. 2007. “Questionnaire Design Experimental Research Survey (QDERS) 2004 Results Overview.” Unpublished Census Bureau report.

Pascale, Joanne. 2004. “Medicaid and Medicare Reporting in Surveys: An Experiment on Order Effects and Program Definitions.” Proceedings of the American Association for Public Opinion Research, American Statistical Association.

Pascale, Joanne (2003), “Questionnaire Design Experimental Research Survey (QDERS) 2004:

Cognitive Testing Results on Health Insurance Questions.” Unpublished Census Bureau Report,

November 5, 2003.

Pascale, Joanne (2001). Measuring Private and Public Health Coverage: Results from a Split-Ballot Experiment on Order Effects. Paper presented at the American Association for Public Opinion Research, Montreal, Quebec, Canada.

Pascale, J. (1999). Methodological Issues in Measuring the Uninsured. Proceedings of the Seventh Health Survey Research Methods Conference, pp. 167–173.

Research Project to Understand the Medicaid Undercount: The University of Minnesota’s State

Health Access Center, the Centers for Medicare and Medicaid Services, the Department of

Health and Human Services Assistant Secretary for Planning and Evaluation, The National

Center for Health Statistics, and the U.S Census Bureau. “Phase II Research Results: Examining

Discrepancies between the National Medicaid Statistical Information System (MSIS) and the

Current Population Survey (CPS) Annual Social and Economic Supplement (ASEC).” March

12, 2008. http://www.census.gov/did/www/shadac/shadac.html.

Resnick, D., S. Love, C. Taeuber, and J. M. Staveley. 2004. Analysis of ACS Food Stamp

Program Participation Underestimate. Paper presented at the 2004 Joint Statistical Meeting,

Toronto, Canada.

Ringel, J. S., and J. A. Alex Klerman. 2005. Today or Last Year? How Do Interviewees

Answer the CPS Health Insurance Questions? RAND Labor and Population working

paper series WR-288. Santa Monica, Calif.: RAND.

Roemer, Marc. 2007. Presentation on preliminary results from “Research Project to Understand

the Medicaid Undercount: Phase 2 Research Results: Examining Discrepancies between the

National MSIS and CPS March Supplement (ASEC); July 17.” Washington, DC.

Roman, A. M., A. Hauser, and A. Lischko. 2002. Measurement of the Insured Population: The

Massachusetts Experience. Paper presented at the 2002 Annual Meetings of the American

Association for Public Opinion Research, St. Pete’s Beach, Fla.

Rosenbach, M., and K. Lewis. 1998. Estimates of Health Insurance Coverage in the Community

Tracking Study and the Current Population Survey. Document no. PR98-54. Washington, D.C.: Mathematica Policy Research, Inc.

Swartz, K. 1986. Interpreting the Estimates From Four National Surveys of the Number

of People Without Health Insurance. Journal of Economic and Social Measurement 14:

233–242.

Walden, Daniel C., Constance M. Horgan, and Gail Lee Cafferata. 1984. “Consumer Knowledge of Health Insurance Coverage.” Proceedings of the Fourth Conference on Health Survey Research Methods, Washington, DC.

Appendix A: SHIPP Methodology Summary

A. QUESTIONNAIRE

1. Content:

1. Demographics

2. Disability

3. Labor force

4. Unearned income

5. Health insurance (three treatments)

CPS ASEC: Current Population Survey Annual Social and Economic Supplement

ACS: American Community Survey

EXP: Experimental version

6. Respondent Debriefing, Linking Request, Address

2. Experimental version captures coverage for job-based and directly-purchased private plans (policyholder and dependent), Medicare, Medicaid, other government programs, military coverage, school-based coverage, coverage from somebody outside the household, and a residual ‘other’ category. In future iterations we would like to add Indian Health Services (HIS) to this list.

3. Mode: CATI

4. Length: 17 minutes

B. SAMPLE

The sample unit is phone number, drawn from two sources: (1) RDD and (2) addresses of people enrolled in Medicare as of May, 2009, from the Centers for Medicare and Medicaid Services (CMS). Because the Medicare files contain address but not phone number, Telematch was used to search for a phone number for these addresses.

C. EXPERIMENTAL DESIGN

1. Interviewers

7. Census Bureau’s Telephone Facility, Hagerstown, Md., all with experience on health surveys

8. Three groups of seven interviewers per group, balanced by experience and skill levels

9. Four main supervisors, two supervisory assistants, and eight monitors, all cross-trained on all questionnaire versions at commencement of project

2. Field Periods

In order to allow each questionnaire version “equal access” to fresh sample and fresh interviewers, the field period was divided into three time periods of two weeks each, and within each time period all three questionnaire versions were worked evenly.

3. Rotation of Interviewers Through Questionnaire Versions and Field Periods

Each interviewer group was first assigned to one questionnaire version, and the group worked on only that version during the first time period (Weeks 1-2). At the end of Week 2, each interviewer group was rotated off of that first questionnaire version and on to a different questionnaire version. They received a brief training on the new questionnaire version, focusing just on the differences between their previous questionnaire and the new one. They then worked on just that second questionnaire version throughout the second time period (Weeks 3-4). At the end of Week 4 interviewers were again rotated to their third and final questionnaire version and received a brief training on the new version, and during the last time period (Weeks 5-6) they worked on only that version. Over the course of the survey, all interviewers worked in all 3 time periods and all 3 questionnaire treatments. In any given time period, an interviewer worked on a single questionnaire treatment.

4. Samples

In order to accommodate the assignment of interviewer groups described above, there were nine independent samples – one for each unique combination of interviewer group/questionnaire version/field period. For example, Interviewer Group 1 was assigned to work on CPS during Weeks 1-2. At the end of Week 2, that particular sample was closed out for good. Interviewer Group 1 then moved on to the ACS questionnaire and a new sample was released for them to work on that version during Weeks 3-4. At the end of Week 4 this sample was closed out for good and Interviewer Group 1 moved on to the EXP questionnaire and another new sample was released for them to work on that version during Weeks 5-6. This same routine was repeated for Interviewer Groups 2 and 3, for a total of 9 independent samples.

5. Training and Field Period

Time Period	Training			Data Collection
	Content	Dates	Time
1	Base training (all interviewer groups together)	March 18 (a.m.)	4 hours	March 22-April 6
	ACS health section (interviewer group A)	March 18 (p.m.)	3 hours
	CPS health section (interviewer group B)	March 19 (a.m.)	3 hours
	EXP health section (interviewer group C)	March 19 (p.m.)	3 hours
2	ACS health section (interviewer group B)	April 7 (a.m.)	2 hours	April 9-23
	CPS health section (interviewer group C)	April 7 (a.m.)	2 hours
	EXP health section (interviewer group A)	April 8 (a.m.)	4 hours
3	ACS health section (interviewer group C)	April 24 (a.m.)	2 hours	April 26-May 10
	CPS health section (interviewer group A)	April 24 (p.m.)	2 hours
	EXP health section (interviewer group B)	April 24 (p.m.)	4 hours

6. Interviewer Groups and Field Periods

	Interviewer Group 1	Interviewer Group 2	Interviewer Group 3
Field period 1 (weeks 1-2)	CPS Health Qs	ACS Health Qs	EXP Health Qs
Field period 2 (weeks 3-4)	ACS Health Qs	EXP Health Qs	CPS Health Qs
Field period 3 (weeks 5-6)	EXP Health Qs	CPS Health Qs	ACS Health Qs

D. PRODUCTION

1. Advance Letters: mailed in all households where we had an address (56% of households).

2. Completed Interviews

	CPS		ACS		EXP		TOTAL
	HHs	People	HHs	People	HHs	People	HHs	People
RDD	1,059	2,640	1,033	2,483	989	2,370	3,081	7,493
Medicare	747	1,757	774	1,747	774	1,746	2,295	5,250
TOTAL	1,806	4,397	1,807	4,230	1,763	4,116	5,376	12,743

3. Response Rates (preliminary) AAPOR RR4

	CPS	ACS	EXP	TOTAL
RDD	48.96%	47.91%	45.51%	47.46%
Medicare	63.19%	60.02%	61.03%	61.37%

Appendix B: Demographic Profile Across Panels and Samples

Table 1a: Demographics Across Treatments: RDD Sample CAL
Table of panel by ager1 (p=0.81)
panel	ager1						Total
	< 18	18-24	25-34	35-44	45-64	65+
CPS	825	129	149	260	714	563	2640
	31.25	4.89	5.64	9.85	27.05	21.33
EXP	736	112	144	216	673	489	2370
	31.05	4.73	6.08	9.11	28.4	20.63
Total	1561	241	293	476	1387	1052	5010
Table of panel by ager2 (p=0.78)
panel	ager2			Total
	< 18	18-64	65+
CPS	825	1252	563	2640
	31.25	47.42	21.33
EXP	736	1145	489	2370
	31.05	48.31	20.63
Total	1561	2397	1052	5010
Table of panel by senior (p=0.55)
panel	senior		Total
	< 65	65+
CPS	2077	563	2640
	78.67	21.33
EXP	1881	489	2370
	79.37	20.63
Total	3958	1052	5010
Table of panel by race (p=0.01)
panel	race			Total
	blknohis	other	whtnohis
CPS	214	354	2072	2640
	8.11	13.41	78.48
EXP	159	269	1942	2370
	6.71	11.35	81.94
Total	373	623	4014	5010

Table of panel by hispan (p=0.67)
panel		hispan				Total
		No		yes
CPS		2481		159		2640
		93.98		6.02
EXP		2234		136		2370
		94.26		5.74
Total		4715		295		5010
Table of panel by educ (p=0.74)
panel		educ												Total
		AA		BA		HSgrad		Prof		lessHS		smcoll
CPS		163		448		649		268		275		360		2163
		7.54		20.71		30		12.39		12.71		16.64
EXP		144		371		614		240		231		327		1927
		7.47		19.25		31.86		12.45		11.99		16.97
Total		307		819		1263		508		506		687		4090
Frequency Missing = 920
Table of panel by sex (p=0.55)
panel		sex				Total
		Female		Male
CPS		1373		1262		2635
		52.11		47.89
EXP		1254		1114		2368
		52.96		47.04
Total		2627		2376		5003
Frequency Missing = 7
Table of panel by hinc (p=0.58)
panel		hinc				Total
		Above		below
CPS		1819		704		2523
		72.1		27.9
EXP		1618		649		2267
		71.37		28.63
Total		3437		1353		4790
Frequency Missing = 220
Table of panel by empstat (p=0.20)
panel		empstat						Total
		FT-FY		NotWr		Other
CPS		716		893		519		2128
		33.65		41.96		24.39
EXP		624		753		504		1881
		33.17		40.03		26.79
Total		1340		1646		1023		4009
Frequency Missing = 1001
Table 1b: Demographics Across Treatments: MCARE Sample CAL
Table of panel by ager1 (p=0.19)
panel	ager1												Total
	< 18		18-24		25-34		35-44		45-64		65+
CPS	373		63		82		81		575		583		1757
	21.23		3.59		4.67		4.61		32.73		33.18
EXP	324		71		84		98		548		621		1746
	18.56		4.07		4.81		5.61		31.39		35.57
Total	697		134		166		179		1123		1204		3503
Table of panel by ager2 (p=0.10)
panel	ager2						Total
	< 18		18-64		65+
CPS	373		801		583		1757
	21.23		45.59		33.18
EXP	324		801		621		1746
	18.56		45.88		35.57
Total	697		1602		1204		3503
Table of panel by senior (p=0.14)
panel	senior				Total
	< 65		65+
CPS	1174		583		1757
	66.82		33.18
EXP	1125		621		1746
	64.43		35.57
Total	2299		1204		3503
Table of panel by race (p=0.00)
panel	race						Total
	blknohis		other		whtnohis
CPS	225		174		1358		1757
	12.81		9.9		77.29
EXP	160		187		1399		1746
	9.16		10.71		80.13
Total	385		361		2757		3503
Table of panel by hispan (p=0.06)
panel	hispan				Total
	no		yes
CPS	1674		83		1757
	95.28		4.72
EXP	1638		108		1746
	93.81		6.19
Total	3312		191		3503
Table of panel by educ (p-0.32)
panel	educ												Total
	AA		BA		HSgrad		Prof		lessHS		smcoll
CPS	120		184		630		147		209		273		1563
	7.68		11.77		40.31		9.4		13.37		17.47
EXP	112		215		630		128		189		295		1569
	7.14		13.7		40.15		8.16		12.05		18.8
Total	232		399		1260		275		398		568		3132
Frequency Missing = 371
Table of panel by sex (p=0.99)
panel	sex				Total
	Female		Male
CPS	920		835		1755
	52.42		47.58
EXP	914		830		1744
	52.41		47.59
Total	1834		1665		3499
Frequency Missing = 4
Table of panel by hinc (p=0.12)
panel	hinc				Total
	above		below
CPS	1002		677		1679
	59.68		40.32
EXP	1038		628		1666
	62.3		37.7
Total	2040		1305		3345
Frequency Missing = 158
Table of panel by empstat (p=0.10)
panel	empstat						Total
	FT-FY		NotWr		Other
CPS	229		1020		312		1561
	14.67		65.34		19.99
EXP	229		982		363		1574
	14.55		62.39		23.06
Total	458		2002		675		3135
Frequency Missing = 368

Table 1c: Demographics Across Treatments: Entire Sample CAL
Table of panel by ager1 (p=0.71)
panel	ager1						Total
	1	2	3	4	5	6
CPS	1198	192	231	341	1289	1146	4397
	27.25	4.37	5.25	7.76	29.32	26.06
EXP	1060	183	228	314	1221	1110	4116
	25.75	4.45	5.54	7.63	29.66	26.97
Total	2258	375	459	655	2510	2256	8513
Table of panel by ager2 (p=0.27)
panel	ager2			Total
	1	2	3
CPS	1198	2053	1146	4397
	27.25	46.69	26.06
EXP	1060	1946	1110	4116
	25.75	47.28	26.97
Total	2258	3999	2256	8513
Table of panel by senior (p=0.34)
panel	senior		Total
	0	1
CPS	3251	1146	4397
	73.94	26.06
EXP	3006	1110	4116
	73.03	26.97
Total	6257	2256	8513
Table of panel by race (p=0.00)
panel	race			Total
	blknohis	other	whtnohis
CPS	439	528	3430	4397
	9.98	12.01	78.01
EXP	319	456	3341	4116
	7.75	11.08	81.17
Total	758	984	6771	8513
Table of panel by hispan (p=0.40)
panel	hispan		Total
	0	1
CPS	4155	242	4397
	94.5	5.5
EXP	3872	244	4116
	94.07	5.93
Total	8027	486	8513

Table of panel by educ (p=0.59)
panel	educ						Total
	AA	BA	HSgrad	Prof	lessHS	smcoll
CPS	283	632	1279	415	484	633	3726
	7.6	16.96	34.33	11.14	12.99	16.99
EXP	256	586	1244	368	420	622	3496
	7.32	16.76	35.58	10.53	12.01	17.79
Total	539	1218	2523	783	904	1255	7222
Frequency Missing = 1291
Table of panel by sex (p=0.65)
panel	sex		Total
	Female	Male
CPS	2293	2097	4390
	52.23	47.77
EXP	2168	1944	4112
	52.72	47.28
Total	4461	4041	8502
Frequency Missing = 11
Table of panel by hinc (p=0.70)
panel	hinc		Total
	above	below
CPS	2821	1381	4202
	67.13	32.87
EXP	2656	1277	3933
	67.53	32.47
Total	5477	2658	8135
Frequency Missing = 378
Table of panel by empstat (p=0.04)
panel	empstat			Total
	FT-FY	NotWrkg	Other
CPS	945	1913	831	3689
	25.62	51.86	22.53
EXP	853	1735	867	3455
	24.69	50.22	25.09
Total	1798	3648	1698	7144
Frequency Missing = 1369

1 A single household income question was asked in which respondents were asked if their total combined household income was above or below a certain threshold. The dollar amount of that threshold was determined by the number of household members and the presence of children under 18 and was meant to loosely approximate the poverty level.

2 Standard errors and p-values are shown in the tables. However, these values should be disregarded for tables with the full sample since statistics from the combined RDD and Medicare sample reflect two different universes.

19