CMS-10458_Supporting_Statement_Part_B_Extension 6-24-16

SUPPORTING STATEMENT – PART B SUPPORTING STATEMENT FOR

PAPERWORK REDUCTION ACT SUBMISSIONS CONSUMER RESEARCH SUPPORTING OUTREACH AND EDUCATION FOR THE EXCHANGES INDIVIDUAL AND SHOP TRACKING SURVEYS

Table of Contents

B Collection of information employing statistical methods 2

B.1 Respondent universe and sample 2

B.2 Information collection procedures 4

B.3 Methods to maximize response rates 5

B.4 Tests of procedures or methods 7

_{B.5 Statistical and questionnaire design consultant 8}

1

COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1 Respondent universe and sample

The target population is the set of population units about which the survey data are to be used to make inferences. Two distinct surveys will be conducted; one for individual consumers who will be eligible to purchase health insurance in the new marketplace and one for small businesses that will be able to offer their employees coverage through the new marketplace. For the individual consumer survey, the target population consists of US adult citizens 18-64 years of age who are without health insurance (est. 44,000,000); , or who have health insurance coverage in the non- group market (est. 25,000,000). For the business survey, the target audience consists of small business [less than 50 full-time employees] decision-makers (est. 27,000,000). For a portion of the individual consumer audience the survey will be conducted in Spanish to insure coverage for Spanish speaking individuals in our work. Together these individuals define the key target audiences for the new health insurance marketplace established under the Affordable Care Act.

The sampling frame for a survey is the list used to enumerate these population units for sample selection purposes. Below we address the use of two different frames – one for the consumer component of the information collection and one for the small business component of the collection.

_{Sampling Plan: Consumers}

The individual customer survey will use a dual Random Digit Dial (RDD) sampling frame of landline and wireless/mobile phone numbers. The sampling frames for the English survey of US consumers are the set of all US landline telephone exchanges, and the set of all US cellular telephone exchanges. These two sampling frames are provided by Marketing Systems Group (M-S-G), a reputable provider of RDD samples. From each frame a Random Digit Dial (RDD) sample will be drawn by randomly generating telephone numbers from the sets of telephone exchanges. Screening questions will be asked to determine survey eligibility for each contacted US consumer so that respondents who respond to surveys are part of the appropriate target audience i.e., either currently uninsured or self-insured through the individual [non-group] market.

Using this probability-based sampling approach, CMS should be able to produce four separate cross-sectional characterizations of the knowledge gaps among the uninsured. However, because our response rates are expected to be low (about 20%) nonresponse bias analysis will be necessary to determine the extent to which inferences can be made to the entire subpopulation (see subsequent section). Furthermore, CMS acknowledges that certain subpopulations will not be captured (e.g., non-English speakers and non-Spanish speakers who do not live in high density “Hispanic” phone exchanges). However, because the survey is being used to identify knowledge gaps, and thus inform the shape and direction of subsequent communication efforts, a strictly representative sample is not necessary.

Items measuring awareness, knowledge, and intent at or above the margins of error indicted by audience will be used to better understand barriers, perceived benefits, and unmet needs from the perspective of the respondents who comprise the target audience for the new health insurance marketplace. Our goal in this research is to enhance our understanding of the audiences that we will be charged with engaging, as opposed to developing formal official statistical descriptions of the populations.

Considering the entire consumer survey sample, the margin of error for English-speaking customers for this study should be no higher than +/- 3.5% at the 95% confidence level. Using this, it estimated that an overall sample size of 900, which yields a margin of error of +/-3.3%, will be sufficient for the purposes of the survey. Thus CMS will be able to approximate target population levels of key measures such as consumer awareness, benefit

knowledge, and potential behavioral intent as a gauge of communication needs specific to these domains within the target population within a relatively narrow range.

Sub-group analysis will be conducted between insured and uninsured individuals in the sample. It was determined the margin of error associated with these subgroups is +/- 3.8% (n=675) and +/- 6.5% (n=225), respectively at the 95% confidence level. Because the makeup of the potential Health Insurance Marketplace target population is skewed largely toward individuals who are currently uninsured, having an unequal distribution of uninsured to insured in the survey sample is appropriate. Screening quotas will be placed on the sample so that 675 interviews will be conducted with consumers who are uninsured and 225 interviews will be conducted with consumers who pay for their insurance without assistance (i.e., self-pay). We expect that this will roughly match the target audience composition. Differences will be measured on key variables such as awareness, benefit knowledge, and behavioral intent. Power to detect a between group difference in proportions of 13 percentage points (e.g. .50 vs. 0.63), which would be regarded as an actionable difference for our communication efforts, exceeds 0.91, assuming a two-sided

alpha = 0.050.

Subgroup analyses will also be conducted between CMS segments. The segments are anticipated to be represented approximately as follows, based on their prevalence in the uninsured population: Sick, Active, and Worried (29%)=196; Healthy and Young (48%)=384; Passive and Unengaged (15%)=101. Power to detect between segment differences in proportions of between 13 and 18 percentage points, depending on the specific segments compared, exceeds 0.80, assuming a two-sided alpha = 0.05. Between segment differences of smaller magnitude will be interpreted cautiously; even substantial differences that do not reach conventional levels of statistical significance will be considered directional.

The sampling frames for the Spanish survey of US consumers are a set of high-density Hispanic landline telephone exchanges and a set of high density Hispanic cellular telephone exchanges from which random samples will be drawn. Some limitation in national representation will be introduced by this approach; however, the approach is sufficient for the purpose of identifying knowledge gaps. These two sampling frames are also provided by Marketing Systems Group (M-S-G).

The Spanish language survey will be conducted with a total of 600 participants per fielding. Data from the Spanish language survey will be reported separately from the English language data and the margin of error will be +/- 4% for the Spanish survey sample. A sample size of 600 will allow CMS to identify gaps in awareness, knowledge, and behavioral intent (as measured by the key variables) within a range that will be useful in planning Spanish language outreach and messaging.

Sampling Plan: Small Business Decision-Makers

The sampling plan for the small business survey will not be random, but will be accessed via a panel that takes steps to improve the sample’s quality and representative nature. Steps involved in the sample design and implementation include: 1) definition of the target and sampled populations, 2) construction of the sampling frame, 3) specifications of the sample selection procedures and 4) creation of the sampling weights and adjustments for non-response and under- coverage.

For the survey of business leaders, participants will be selected from Survey Sampling Incorporated’s (SSI) online sample, a consistently managed, diverse and large frame that draws on SSI’s proprietary panels, online communities, social media and affiliate partnership and is managed to allow the selection of samples to reflect populations.

SSI’s 34 proprietary panels across the globe are at the core of their online sample. Since SSI’s research shows that only certain types of people want to join an online panel, SSI improves the quality and diversity of its online sample by incorporating participants from online communities, social networks, and websites of all types. SSI’s sample recruitment is quite different from the simple “river” approach: participants are invited via banners, invitations and messaging of all types, but then go through rigorous quality controls before being included in any sample. SSI can potentially access anyone online via a network of relationships with websites, panels, communities and social media groups. By selecting from a universe including SSI's proprietary panels, as well as from partnerships with web sites and online sources of subpopulations who may not wish to join a research panel may be included.

The panel sample will be selected using randomization, and contact made with community and panel members via e-mail invitation. For the internet sample, invitations are placed on thousands of web sites to provide maximum diversity of the frame, and respondents are invited to come in to "take a survey." Once within SSI's system they are matched with an available survey using multiple points of randomization.

Online samples, if recruited, managed and selected correctly, have been shown to effectively reflect the variety expected across the universe. Such an approach is expected to provide sufficient accuracy for guiding customer service decisions like the ones of interest here, and nonresponse bias analysis will be conducted to understand the potential limitations of the results.

The margin of error for the total sample size included in this study (n=300) would be 5.7% at a

95% confidence level if we were to treat the SSI panel as our target population.

B.2 Information collection procedures

^Consumers

The data collection for consumers will consist of telephone interviewers administering the survey over the telephone. The business leaders will be recruited via online panel invitation.

For the survey of US consumers, a sufficiently large landline RDD and cell phone RDD sample will be selected to achieve 1,500 completed interviews (900 English, 600 Spanish). An approximate 60/40 mix of landline/cell phone interviews will be collected for the consumer telephone surveys. This mix is based on 2011 CDC report estimates from the National Household Interview Survey (http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201112.pdf). For purposes of estimation, cross-sectional weights will be developed that account for the probability of selection from each sample frame, the eligibility rate within each sample frame, levels of non-response within each sample frame, and finally differential nonresponse by age, gender, and geographic region. Standard errors will be produced using software packages such as PASW/SPSS complex survey statistics to appropriately account for the survey design.

_{Small Business Decision-Makers}

SSI uses invitations of all types to bring in people with a diversity of motivations to take part in research and be a part of their online research panel. These include e-mail invitations, SMS and text messages, telephone alerts, banners and messaging on web sites and online communities. The messages themselves are also varied, including invitations to give your opinion, win a prize, earn cash or prizes or let your voice be heard. For the current data collection, panelists who participate will receive an incentive of rewards points worth approximately four dollars. A diversity of motivation contributes to more varied sample. To avoid self-selection bias, specific project details are not generally included in the invitation. Rather, participants are invited to “take a survey.” The details are disclosed later, when a survey has been selected for them to take within the system.

To minimize bias, a three-stage randomization process is used in matching a participant with a survey they are likely to be able to complete. First, participants are randomly selected from SSI’s panels to be invited to take a survey after responding to online messaging. A set of profiling questions is randomly selected for them to answer. (These are methodologically correct questions, never affirmation questions) and upon completion, participants are matched with a survey they are likely to be able to take, using a further element of randomization.

Multi-stage randomization is incorporated into the sample platform routing. Participants are randomly assigned to a series of profiling questions. Based on their answers, they are assigned, again using a randomization factor, to a survey they are likely to be able to take. Other factors considered in the assignment include the likelihood that they will be able to complete the survey and the characteristics of the specific study, including factors such as field time and incidence.

B.3 Methods to maximize response rates and non-response analysis plan

Survey response rates express completed interviews as a percentage of the estimated eligible units, but note that it can be decomposed into three rates if we assume that the working residential and eligible rate of unresolved cases is equal to that of resolved cases:

Response rate = Working residential number resolution rate * Household screening completion rate * Survey interview completion rate.

This rate is the so-called CASRO (Council of American Survey Research Organizations) response rate, or American Association for Public Opinion Research’s (AAPOR’s) third response rate definition with the working residential and eligible rate of unresolved cases equal to that of resolved cases. We use this definition of response rates here.

Even though our surveys focus on communication and marketing issues and are not intended to provide prevalence estimates for the characteristics of the uninsured, we note that our approaches are based on well-established methods for telephone RDD sampling and data collection. Established operational protocols are in effect that have been shown to minimize sampling and measurement errors in the survey process. Our use of a dual frame RDD approach helps to address coverage issues that have often caused problems in representativeness of telephone surveys (see, e.g., Blumberg et al., 2010). In addition, we will apply responsive design approaches by suing survey paradata and related information (e.g., contact patterns, length of interview, number and mode of respondent contact) to manage the survey operations, gain efficiencies, and enhance response rates. We typically monitor real-time information on sample outcomes (e.g., rates of nonresidential telephone numbers, disconnected lines, refusals, completed interviews) to track progress and can make adjustments to enhance survey operational processes. For example, landline and cellular phone samples often have different performance characteristics due to differential rates of nonworking and business numbers. Ongoing monitoring can suggest changes in allocation between landline and cellular calling to optimize data collection efficiency and permit adaptive responses to potential issues with respondent selection, refusal conversion strategies, and related issues.

The methods described above have been shown to yield response rates of at least 20 percent with US consumers and business leaders when the survey is of reasonable length and on a salient, non- threatening topic, as is the case in the present work. This rate is consistent with rates in typical health policy and marketing research survey studies and can be used to establish reasonably representative samples. The following procedures, for consumers, will also encourage response:

 At least three callbacks at various times, so every case will have a day, night and weekend attempt. We will also be doing refusal conversion attempts.

_{ Interviewer training will review refusal avoidance and second calls to dead dispositions.}

_{ A toll-free number is available at Market Strategies International to answer respondents’}questions.

 Calls to the toll-free number will be returned to address respondents’ concerns.

Despite the best efforts of the marketing research industry and the survey research community, there is clear evidence of declining response rates in both telephone and face-to-face surveys (see, e.g., NORC, 2007; Peytchev et al., 2009). Unit nonresponse is a source of particular concern because is often regarded as a boundary condition for nonresponse bias that can limit the utility of survey results for actionable guidance. Although a meta-analysis of nonresponse issues in survey research (Groves & Peytcheva, 2008) has reinforced the finding that response rate is not generally predictive of nonresponse bias, there is no doubt that steps taken to assess and limit such biases can result in surveys of higher quality. Unit non-response has two negative consequences for the quality of the estimates derived from the data. First, nonresponse reduces the sample size and, as the number of responses decreases, the variability of survey estimates increases. Second, and more importantly, nonresponse has the potential to cause bias in the estimates. For means and proportions, the bias depends on two factors: the response rate, and the difference in the means or proportions of the respondents and non-respondents. Therefore, bias can be expressed as follows:

Bias = (1 – RR) * (S_r – S_n),

Where RR = the unit response rate, S_r = the mean or proportion for respondents, and S_n = the mean of proportion for non-respondents.

Thus, bias increases as the difference in means/proportions increases between respondents and non- respondents, or as the unit nonresponse rate increases. Unfortunately, while the response rate can be calculated, we do not know the mean or proportion for the non-respondents. The best strategies for combating unit non-response bias on a CATI survey, like the ones proposed for consumers in this work, are multiple re-contact attempts for non-responders (as noted above) and a robust non- response weighting scheme. Both will be implemented in this work.

The potential detrimental effect of unit nonresponse can be reduced through the use of population- based weighting that adjusts not only for under-coverage but also for non-response. This weighting approach controls the weighted sample counts to population totals for characteristics presumed to be correlated with non-response, under-coverage, and/or the survey variables of interest. Analyses for the total population as well as population subgroups based on the resultant survey weights should thus produce accurate and reliable results.

The surveys described here will make use of Census-based population totals for race/ethnicity, gender, age, income, and geography in deriving the survey weights. We expect that these geographic and demographic groups would be most appropriate for increasing sample representativeness of the population, thereby reducing the potential for bias in the resultant survey estimates.

In order to assess the above weighting scheme and potential non-response bias, we will be comparing demographic profiles and income distribution derived from our data against several sources, including those published by the Census Bureau for the American Community Survey.

For purposes of estimation, cross-sectional weights will be developed that account for the probability of selection from each sample frame, the eligibility rate within each sample frame, levels of non-response within each sample frame, and finally differential nonresponse by age, gender, and geographic region. Standard errors will be produced using software packages such as SPSS/PASW complex survey statistics to appropriately account for the survey design (see, e.g., Heeringa et al.,

2010).

B.4 Tests of procedures or methods

The instruments were tested with small samples (cognitive testing with 5-8 individuals for marketplace-specific items) to verify general comprehension among the target audience. Many of the more general questions have been used successfully in other surveys for screening and classification of general consumer audiences, and so did not require cognitive testing. This type of pretesting with consumers serves to reduce the potential for confusion. We do not anticipate a need to develop any new questions as part of this work. However, if new questions were required, they would also be tested using cognitive testing methods prior to data collection. The data collection procedures have been well tested in surveys of US consumers and business leaders.

B.5 Statistical and questionnaire design consultant:

Robert Bailey Director of Research Salter Mitchell, Inc.

703.683.2240

Christine Brittle, Ph.D. Senior Researcher Salter Mitchell, Inc.

703.683.2240

Jack Fyock, Ph.D. Vice President

Market Strategies International

410.203.1245

The project officer who will receive the deliverables from the supplement is: Clarese Astrin, Ph.D.

Division of Research

Office of Communications – Strategic Marketing Group

Centers for Medicare and Medicaid Services

410.786.5424

References

The American Association for Public Opinion Research (AAPOR) (2011). “Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys.” 7th edition. Lenexa, Kansas: AAPOR.

Blumberg, Stephen J. & Luke, Julian V. (2010). “Wireless Substitution: Early Release of Estimates from the National Health Interview Survey, July–December 2009” http://www.cdc.gov/nchs/data/nhis/earlyrelease/wireless201005.pdf.

Curtin,Richard, Presser, Stanley & Singer, Eleanor. (2005). Changes in telephone survey non- response over the past quarter century. Public Opinion Quarterly, 69: 87-98.

Groves, Robert M. & Peytcheva, E. (2008). The impact of nonresponse rates on nonresponse bias: A meta-analysis. Public Opinion Quarterly, 72(2), 167-189.

Heeringa, Steven G., West, Brady T., & Berglund, Patricia A. (2010). Applied Survey Data

Analysis. Boca Raton, FL: CRC Press.

Lepkowski, James M. (1988). “Telephone Sampling Methods in the United States,” pp. 73–98 in R.

M. Groves et al. (eds.), Telephone Survey Methodology. New York: Wiley.

NORC. (2007) National Immunization Survey: A user’s guide for the 2006 Public Use Data File.

Peytchev, Andy, Baxter, Rodney K., Carley-Baxter, Lisa R. (2009). Not all survey effort is equal: Reduction of nonresponse bias and nonresponse error. Public Opinion Quarterly, 73(4):

785-806.

The Pew Internet and American Life Project (2012). – Pew Internet Summer Tracking Survey, August 7 – September 6, 2012.