The American Community Survey

Attachment V

Task Order 002
Affordable Care Act Pretesting
American Community Survey Cognitive Testing

Final Report
Submitted Friday 11/16/2012

Prepared For
U.S. Census Bureau
Prepared By
Research Support Services, Inc.
Alisu Schoua-Glusberg
Katherine Kenward
Lisa Andrews
Center for Survey Research, UMass Boston
Carol Cosenza

Research Support Services, Inc.
906 Ridge Ave
Evanston, IL 60202

Attachment V

Table of Contents
Executive Summary _____________________________________________________ 3
Introduction ___________________________________________________________ 9
Time Constraints _____________________________________________________ 10
Methodology _________________________________________________________ 10
Recruiting ___________________________________________________________ 12
Participant Characteristics _____________________________________________ 13
Findings and Analysis __________________________________________________ 15
Question Version Findings _____________________________________________ 17
Question 16 and 17 Combined __________________________________________ 23
Recommendations _____________________________________________________ 25
Conclusions ___________________________________________________________ 25
Appendices included as a separate file

Attachment V

I.

Executive Summary

Introduction: In March 2010, the Affordable Care Act (ACA) was passed, ushering in a series of
reforms of the U.S. health care system to be implemented over the course of four years. These
reforms involve expanding public coverage (Medicaid) and targeting particular populations, such
as young adults, as well as reducing health coverage disparities for racial and ethnic minorities.
One of the most significant components of the ACA is that in 2014 there will be fully
implemented “Health Insurance Exchanges” in each state. These are joint federal-state
partnerships (or, in some states, just federal government programs) designed to create a
marketplace of health insurance options for individuals and small businesses or State established
exchanges that are designed within the guidelines of the ACA. The CBO estimates that 81% of
participants in an Exchange will receive a government subsidy to lower the cost of their
premium. 1These Exchanges and products therein are still in development and states have broad
flexibility in designing their programs.
These reforms will present challenges for measuring the many different sources of health
insurance coverage. The U.S. Census Bureau recognized that there was a need to explore the
terms and concepts that various targeted groups use to refer to these new avenues for obtaining
coverage. These data will be crucial for assessing the effects of the exchanges on overall
coverage.
Census partnered with Research Support Services, Inc. and its subcontractor, the Center for
Survey Research at UMass Boston, to study the issue. The state of Massachusetts was the ideal,
and only, venue for exploratory research, having passed a law in 2006 that includes most of the
features of the ACA, including an Exchange. Findings from this research were intended to guide
the design of questions that measure health insurance in the Census Bureau’s demographic
surveys, specifically the Current Population Survey and the American Community Survey. This
report focuses on the testing conducted for the American Community Survey.
In addition to testing a new ACS question on health insurance coverage, this test included
language proficiency questions for Spanish language participants. This contributes to the Census
Bureau’s examination of language barriers in accessing health coverage.
Time Constraints: Collaboration between the RSS/CSR and Census teams began in the fall of
2011 with the detailed exploration of the implications of the Affordable Care Act nationwide,
identifying exchange concepts and terms including premium subsidies. This research included
consultation with experts in ACA and Massachusetts coverage issues, four focus groups, and
1

http://www.kff.org/healthreform/upload/8147.pdf accessed 11/15/2012

Attachment V

four iterative rounds of testing for the CPS. In July 2012, after the outcome of the Supreme
Court decision on the ACA, it was decided to add the ACS project as a piggyback to this effort.
It was agreed that the current coverage question, question 16, would be cognitively tested and an
additional question would be added and tested to determine if the respondent paid a subsidized
premium. This portion of the project began in mid August 2012. Because the final question
wording had to be submitted by mid October, it was only feasible to accommodate two rounds of
testing with 60 interviews in total.
Methodology: The RSS-CSR developed the initial cognitive protocol based on bulleted notes
provided by the ACS census team. The initial questions and translations were also provided by
the census team. Two questions were tested, 16 & 17, as well an abbreviated version of the ACS
questionnaire in order to test the coverage questions in context. Protocol development, as well as
testing, was conducted in parallel in English and Spanish. After the questionnaire was
administered, retrospective probing followed first about the respondent then about other
household members. The cognitive testing protocol was developed to assure that questions were
tested thoroughly, and that respondents were probed consistently across interviewers.
After the first round of cognitive interviews, the project team decided to split question 17 into
two parts, as discussed later under Analysis and Findings.
Recruiting: ACS recruiting was designed to minimize new recruiting efforts while maximizing
the quality of respondents chosen to participate. The ACS was able to piggyback on the
concurrent CPS project and thus could access the most useful potential respondents by recruiting
in large part from those who called in to be screened after receiving the Health Connector
mailings for participation in the CPS study.
For the ACS project recruiting, we were able to contact the respondents who called in from the
Health Connector mailings but who were not interviewed in the four CPS rounds of testing.
These potential respondents were re-screened to make sure that they still had the same type of
coverage that made them eligible for CPS testing. The remaining Spanish recruits and the
English MassHealth recruits were identified using traditional recruiting methods.
Participant Characteristics: There were two rounds of testing and a total of 60 cognitive
interviews. Because of the difficulty in recruiting Spanish monolingual Commonwealth Care
Premium respondents, and the lack of any Spanish monolingual Commonwealth Choice
participants, it was decided at the start of the rounds to conduct 35 English interviews and 25 in
Spanish. Among Spanish language respondents, during the interview 20 reported speaking
English not well or not at all. All Spanish respondents reported speaking English Not Well or
Not at all in screening.

Marital status and the presence of children in the household were also tracked. Half of the

Attachment V

respondents reported being not married with no children under 18 living in their household. Onethird of respondents reported being married and 30% reported having children under the age of
18 living in their household. Respondents reported a variety of races, age and education.
Findings and Analysis
The first question on coverage, ACS question 16 is currently in use in the American Community
Survey. In both rounds, therefore, the question was asked without changes. The purpose of
testing it in this study was to see how it worked in a state where Health Reform has already been
implemented, and in particular, where exchange programs are in operation. Because the question
was tested without changes across rounds, the findings from both rounds are discussed together.
A number of basic concepts were probed on, to make sure respondents understand them as
intended. The concept of 'health insurance or health coverage plans' was well understood across
languages. The concept of 'current' coverage was also universally well understood and
interpreted as intended.
Exchange-covered respondents did not find a response choice that directly referred to the
exchange. Those who purchase a private plan through the Exchange (Commonwealth Choice
participants), tended to answer Direct Purchase or Other and specify the plan name. There were
a variety of responses for those who have Commonwealth Care (exchange plan fully or partially
subsidized). Those who pay a premium tended to answer Direct Purchase, while those with full
subsidy tended to choose Medicaid. Among those respondents who answered Yes to Other in
round 1, the largest group (n=9) indicated they chose it because they did not feel their coverage
fit anywhere else. In round 2 only three respondents who answered Direct Purchase selected
Other..
Among those saying Yes to multiple answers, there appeared to be confusion as to what
coverage they have, mostly stemming from lack of knowledge or clarity, and not because of
issues interpreting the question. For instance, some respondents who were uncertain if their
coverage was Medicare or Medicaid chose both. The definition of Medicaid offered to them
appeared to help some of the Commonwealth Care and some of the MassHealth respondents
answer Yes, as the mention of low income suggested this fit their situation. In addition, some
respondents answered Yes to Other as well as another category as a way to write in more
specifically what they had marked above.
Alternative wording was tested for question 16b: "Purchased directly" [comprado directamente,
for Spanish] without mention of insurance company. There was no strong evidence suggesting
this change would offer a better alternative.

Attachment V

The second question tested was question 17. In each round, two versions were tested. In Round
1, the two questions were: "Is the cost of your health insurance reduced based on your family
income?" (Version A) and "Is the cost of the health insurance premium reduced based on your
family income?" (Version B).
These questions were problematic for respondents who pay no premium or out of pocket
expenses at all. The question asks about a cost they do not incur. For those who pay no
premium but do pay out of pocket expenses, Version B was more problematic than Version A.
Also, when probed in Version B, most respondents showed they understood the term 'premium',
but not everyone did.
The concept of 'is reduced' presented a number of problems. First, it was interpreted differently
by different respondents. While some thought it meant that the cost had already been reduced
and they were paying a smaller amount, others interpreted the phrase as a hypothetical 'would it
be reduced' should their income change. Some respondents simply did not know the answer,
despite interpreting the question as intended, that is, they did not know if the amount they were
paying was or was not based on their income. There was also lack of consistent interpretation as
to what the cost was reduced from.
What about accurate responses based on the respondent's reality? Most of those who pay no
premium answered Yes, but the ones who do pay answered Yes and No in similar numbers.
Based on the ACS team’s decisions after considering the round 1 findings, in round 2, question
17 became a two-part question, first filtering out those respondents who do not pay a monthly
premium. First, question 17a asked:
Is there a monthly premium for this plan? A monthly premium is a fixed amount of
money people pay each month to have health coverage. It does not include copays or
other expenses such as prescription costs.
Two versions of 17b were tested for those who answered Yes at 17a:
Version C: Is the cost of the premium reduced based on family income?
Version D: Is the cost of the premium subsidized based on family income?

Question 17a proved to be a welcome addition. It worked as intended: respondents who pay a
premium answered Yes, and no false positives were detected. This was true in both languages.
The soft edit (READ IF NECESSARY) in the interviewer-administered modes proved to be
necessary and useful for some respondents.

Attachment V

For 17b, in Version C the term 'reduced' presented problems for respondents whose premium had
recently gone up (due to the recent beginning of the new plan year for exchange programs, this
had happened for most respondents within the last quarter). It also presented problems in
Spanish as the translation asks about the premium having been reduced, rather than paying a
reduced premium. Some respondents think 'reduced' means that the premium is lower, not that
someone else pays part of it. Findings similar to those in round 1 were also seen in this round
with the term ‘reduced.’
In Version D, 'subsidized' was not clear to all. To one respondent, the question is asking people if
they agree to pay a monthly payment for their coverage.` Others who understood the term and
whose coverage is actually subsidized, did not feel their coverage is subsidized because they pay
a premium and do not know of anyone else paying part of it. Additionally, in one case at least it
was received as a term with negative connotations ("handout").
In addition to looking at each question individually, it is important to consider the coverage
questions in tandem and the extent to which appropriate coverage coding rules can be determined
when all available data is taken together.
Recommendations and Conclusions
Question 16 is in use currently by ACS and therefore will remain as is, unchanged, despite
findings from testing. However, since this is the first time the question has been tested with
respondents with exchange coverage, we recommend on the basis of our findings and for future
testing that Census look into the possibility of adding one or more response choices that may be
more appropriate to exchange-covered individuals. Until more is known about how different
states will publicize or name their exchanges, this should be tabled.
For question 17, given the impossibility of further testing before making a decision for a version
to field in ACS, we recommended two possibilities in advance of the final meeting: 1) Is your
premium reduced or subsidized based on family income?, and 2) Is your premium subsidized
based on family income? To this version a definition would be included for subsidized. Several
definitions were discussed but there was no consensus. Thus, we recommend census continue to
consider the possibility of adding and testing one in the future.
For the future, new questions may need to be designed or current ones revised as states start
giving tax credits. As that happens, the constructs behind the questions that will need to be
asked may not correspond to anything that will be known by many respondents, as it will happen
"behind the scenes."
This study accomplished its goals: to test the ACS question 16 in Massachusetts with a varied
population primarily consisting of individuals who have coverage through the state Exchange
and to test a new question about subsidies with the same population. Across two rounds of
cognitive testing, the new question was refined to better capture the realities of exchange

Attachment V

coverage, while still adequately capturing the realities of any other type of coverage. It also
helped to throw additional light on language access issues in obtaining coverage through the
State programs.
The questions worked well with the limited number of non-exchange covered respondents,
thereby suggesting no new error was introduced with these questions.
As the 2014 national rollout of the ACA in all states approaches, further research should test the
questions in states with different models of exchange programs as they begin operating.

Attachment V

II.

Introduction

In March 2010, the Affordable Care Act (ACA) was passed, ushering in a series of reforms of
the U.S. health care system to be implemented over the course of four years. These reforms
involve expanding public coverage (Medicaid) and targeting particular populations, such as
young adults, as well as reducing health coverage disparities for racial and ethnic minorities.
One of the most significant components of the ACA is that in 2014 there will be fully
implemented “Health Insurance Exchanges” in each state. These are joint federal-state
partnerships (or, in some states, just federal government programs) designed to create a
marketplace of health insurance options for individuals and small businesses or State established
exchanges that are designed within the guidelines of the ACA. The CBO estimates that 81% of
participants in an Exchange will receive a government subsidy to lower the cost of their
premium. 2 These Exchanges and products therein are still in development and states have broad
flexibility in designing their programs.
These reforms will present challenges for measuring the many different sources of health
insurance coverage. For example, do Exchange participants report having their coverage through
the government, directly purchased or something else? Exchange participation will be a way to
access some conventional sources, possibly including sources such as individually purchased
coverage, means-tested coverage, and coverage offered through employers who participate in the
Small Business Health Option Program (SHOP).
The U.S. Census Bureau recognized that there was a need to explore the terms and concepts that
various targeted groups use to refer to these new avenues for obtaining coverage. These data will
be crucial for assessing the effects of the exchanges on overall coverage. Census partnered with
Research Support Services, Inc. and its subcontractor, the Center for Survey Research at UMass
Boston, to study the issue. The state of Massachusetts was the ideal, and only, venue for
exploratory research, having passed a law in 2006 that includes most of the features of the ACA,
including an Exchange. Residents in Massachusetts have experience in interacting with several
of the new vehicles of coverage, and are thus a rich source of information and can provide
insight into how respondents may answer standardized questions about coverage.
Findings from this research were intended to guide the design of questions that measure health
insurance in the Census Bureau’s demographic surveys, specifically the Current Population
Survey and the American Community Survey. This report focuses on the testing conducted for
the American Community Survey.

2

http://www.kff.org/healthreform/upload/8147.pdf accessed 11/15/2012

Attachment V

In addition to testing a new ACS question on coverage, this included language proficiency
questions for Spanish language participants. This contributes to the Census Bureau’s
examination of language barriers in accessing health coverage.

Time Constraints
Collaboration between the RSS/CSR and Census teams began in the fall of 2011 with the
detailed exploration of the implications of the Affordable Care Act nationwide, identifying
exchange concepts and terms including premium subsidies. This research included consultation
with experts in ACA and Massachusetts coverage issues, four focus groups, and four iterative
rounds of testing for the CPS. In July 2012, after the outcome of the Supreme Court decision on
the ACA, it was decided to add the ACS project as a piggyback to this effort. It was agreed that
the current coverage question, question 16, would be cognitively tested and an additional
question would be added and tested to determine if the respondent paid a subsidized premium.
This portion of the project began in mid August 2012. Because the final question wording had to
be submitted by mid October, it was only feasible to accommodate two rounds of testing.
Though there were only two rounds of testing, the strong working relationship of the team
established during testing for CPS and a tightly controlled schedule which required unusually
fast turnaround times on the part of both the RSS/CSR and the census teams enabled the
successful completion of sixty cognitive interviews in just over two months.

Methodology
The RSS-CSR team took the lead in developing the initial cognitive protocol based on bulleted
notes provided by the ACS census team. The initial questions and translations were also
provided by the census team and included the two questions to be tested, 16 & 17, as well an
abbreviated ACS version of the questionnaire that provided context for the coverage questions.
Every effort was made to ensure that the protocol would elicit the information on questionnaire
items that was necessary for thoroughly assessing how the two items were performing and if the
coverage type could be accurately identified in a production ACS format. Protocol
development, as well as testing, was conducted in parallel in English and Spanish.
The interview protocol included an introduction where the respondent learned about the
purpose of the study, what was expected of him/her in this type of interview, and a consent
administration step. The respondent then completed the ACS questionnaire either as a selfadministered questionnaire or in a CATI or CAPI simulation. Respondents were asked to
complete the ACS for three household members, however, if they were unable to complete
the questionnaire within thirty minutes, interviewers stopped the interview once the
respondent had completed the test questions for him or herself. After the questionnaire was
administered, retrospective probing followed first about the respondent then about other
household members.

Attachment V

In developing the protocols, the team kept in mind that the purpose of the testing was to elicit
information on interpretation patterns and to make sure the questions were asking what was
intended. The team also worked to make sure that the protocols were clear and understandable in
both languages, that respondents were able to answer them, and that they did not yield false
positives or false negatives. The cognitive testing protocol was developed to assure that
questions were tested thoroughly, and that respondents were probed consistently across
interviewers.
Cognitive testing for the ACS questionnaire took place over two rounds. Each round consisted
of 30 interviews each for a total of 60 cognitive interviews. English and Spanish language
protocol development and testing occurred concurrently in each round just as in the CPS
cognitive testing rounds. Each round lasted a little over a week with interview summaries
submitted as they were completed after each interview. In order to accommodate the
compressed schedule, less than a week was available between the final summary submissions in
round 1 and the review, analysis, and initial recommendations for changes for round 2. The
following table shows the distribution of coverage type per language in each round of testing.
Round 1
Commonwealth Choice
Commonwealth Care Premium
Mass Health
ESI

English
9
6
3
0

Spanish
0
7
4
1

Round 2
Commonwealth Choice
Commonwealth Care Premium
Mass Health

English
6
6
5

Spanish
0
6
7

After the first round of cognitive interviews, a meeting of the entire team was held to discuss the
findings. The meeting focused on review of the protocol to assess what worked well, what
portions of the questionnaire needed new question wording, and if there was need for additional
areas of probing. It was decided after the first round to split the new question, question 17, into
two subparts as discussed later in analysis. Translation of the new question 17 into Spanish was
a collaborative effort between RSS and the Census team.
In translating the protocol into Spanish, we always aimed to produce instruments that
maintained equivalence of measurement across languages in order to achieve a functionally
equivalent and culturally appropriate version of the original instruments. We also prioritized
producing a translated version that worked equally well for people speaking different national
varieties or dialects of Spanish and different educational levels. Translations were carried out
under the supervision of Alisú Schoua-Glusberg, an expert in instrument translation into
Spanish for health research. She worked closely with Census linguist, Jenny Leeman.

Attachment V

Because all cognitive interviewers had already conducted four rounds of CPS interviews, they
were familiar with the goals and needs of the ACS project. Training of the interviewers was
conducted by teleconference with interviewers from RSS and CSR together, for Round 1. Alisu
Schoua-Glusberg and Carol Cosenza trained each of their teams separately for Round 2.

Recruiting
ACS recruiting was designed to minimize new recruiting efforts while maximizing the quality of
respondents chosen to participate. Because the ACS was able to piggyback on the concurrent
CPS project, the team was able to access the most useful potential respondents by recruiting in
large part from those who called in to be screened after receiving the Health Connector mailings
for participation in the CPS study. In this way, the ACS project was able to save resources and
cut down on recruiting time for 60 interviews in a compressed schedule. Like the CPS study,
ACS required participants that had used the Exchange in order to properly test its questions.
Prior to the start of any recruiting for the CPS study, it was agreed that the best source of
information for Exchange participants was the Health Connector itself. The experts we consulted
universally agreed that many Exchange participants do not realize they get their insurance
through the Exchange nor do they even know what kind of insurance they have. Using the Health
Connector rolls for the CPS study recruiting allowed us to select specific respondents with
absolute assurance that we knew their health coverage type rather than rely solely on the reports
of the individual respondent, and ensured that we were including appropriate candidates.
In May 2012, as part of the recruiting effort for the CPS project, the Massachusetts Health
Connector mailed approximately 2,000 letters to Commonwealth Care Premium members and
4,000 letters to Commonwealth Choice members. The Health Connector does not keep track of
language skills or preferences of the individuals they cover; therefore, mailing areas were
selected to include every Connector household in heavily Hispanic areas in the state. Letters
went out with English on one side and Spanish on the other directing potential participants to call
an 800 number to be screened for inclusion in the study. The English response was high, with
over 125 Commonwealth Care Premium and nearly 300 Commonwealth Choice members
calling to be screened. While response from English speakers was more than sufficient, very low
numbers of Spanish speakers called in for screening.
The lack of Commonwealth Choice Spanish monolingual participants was disappointing but not
surprising since previous efforts to recruit this population had resulted in just one Spanishspeaking bilingual participant. Individuals covered through Commonwealth Choice have a
higher level of income, which among Spanish-speaking immigrants in Massachusetts is typically
associated with bilingual language skills. This suggests to us, in the face of the evidence from
the mailing response, that this population, if it exists at all, is extremely small. Likewise,
Commonwealth Care Premium respondents also meet minimum income standards and thus
again, monolinguals are likely to be a small percent of the overall immigrant community that

Attachment V

would qualify for this type of coverage. The earnings opportunities for Spanish monolinguals are
limited.
Because of the low response rate to the Spanish Health Connector mailings, there were only two
Commonwealth Care Premium Spanish monolinguals available for inclusion from the mailing
effort. All English Commonwealth Choice and Commonwealth Care Premium respondents were
recruited through the Connector mailings.
For the ACS project recruiting, we were able to contact the respondents who called in from the
Health Connector mailings but who were not interviewed in the four CPS rounds of testing.
These potential respondents were re-screened to make sure that they still had the same type of
coverage that made them eligible for CPS testing. The remaining Spanish recruits and English
MassHealth recruits were identified using traditional recruiting methods. These methods
included the distribution of flyers in both English and Spanish and for English participants the
use of Craigslist.com. We also worked with community organizations, particularly those serving
the Spanish language community. Contacts with the YWCA of Greater Lawrence and attending
community events and festivals where contact with community leaders could be made were
fruitful in recruiting some respondents. Most helpful for Spanish monolingual recruitment was
the enthusiastic assistance of Spanish Helpline Counselor at Health Care for All, Carlos Solís,
who was critical to the successful recruiting of respondents with Commonwealth Care Premium
coverage for both rounds of ACS. As with the Health Connector, Mr. Solís was able to confirm
coverage type so even for respondents who were unsure of their type of coverage, premium, or
subsidy, we were able to distinguish their coverage type based on their recruitment source.

Participant Characteristics
There were two rounds of testing and a total of 60 cognitive interviews. Because of the difficulty
in recruiting Spanish monolingual Commonwealth Care Premium respondents, and the lack of
any Spanish monolingual Commonwealth Choice participants, it was decided at the start of the
rounds to conduct more ten more English interviews than Spanish so a total of 35 English and 25
Spanish interviewers were conducted.
Aiming for demographic and geographic diversity among respondents was a component of
recruitment. However, due to the challenges in identifying respondents that met coverage
requirements it was not always possible to select a diverse range of respondents, especially for
Spanish monolinguals which were all recruited from the limited urban areas where larger
Hispanic populations live.
Among Spanish language respondents, most reported speaking English not well during the
interview. All Spanish respondents reported speaking English Not Well or Not at all in
screening. The difference in reporting between the telephone screener and the in-person

Attachment V

cognitive interview for respondents who later reported they could speak English ‘well’ was not
clearly identified, however, it could be attributed to the comfort level of the respondents in the
in-person setting. Nothing in the recruiting process would have suggested to respondents that
they needed to be monolinguals to qualify.

Spanish
English - Very well
English - Well
English - Not well
English - Not at all

25
0
5
14
6

Marital status and the presence of children in the household was also tracked. Half of the
respondents reported being not married with no children under 18 living in their household. Onethird of respondents reported being married and 30% reported having children under the age of
18 living in their household
Marital and Parental Status
Not married/no kids < 18
Not married/kid(s) < 18
Married/no kids < 18
Married/kid(s) < 18
Refused

29
11
11
8
1

Number of Children < 18 in Household
0
1
2
3
4 or more

41
9
7
0
3

Ethnicity of respondents was also tracked in the two ACS rounds of cognitive interviewing. The
split between Hispanic and not Hispanic was roughly half because some of the English language
interviews were with Hispanic respondents. Twenty-eight respondents reported that they were
Hispanic compared to 32 who were not. The chart below shows the breakdown of respondents by
ethnicity.

Attachment V

Race
White
Black or African American
American Indian or Alaska native
Asian
Native Hawaiian or Other Pacific
Islander
[not codeable/responded Hispanic]

29
11
2
3
0
15

Finally, age and education level were also tracked. Roughly half of the respondents were in the
35-54 age range. There was one interview with a respondent over the age of 65. At initial
recruiting, it was reported that this respondent was under the age of 65. Once this error was
discovered the team decided to proceed with the interview because of the unique dual coverage
of the respondent who had both MassHealth and Medicare.
Age
18-34
35-54
55-64
65 or older
Refused
Education
Less than BA
BA or higher
Refused

III.

13
29
16
1
1
44
15
1

Findings and Analysis

The first question on coverage, ACS question 16 is currently in use in the American Community
Survey. In both rounds, therefore, the question was asked without changes. The purpose of
testing it in this study was to see how it worked in a state where Health Reform has already been
implemented, and in particular, where exchange programs are in operation. Would respondents
with exchange coverage (fully subsidized, partially subsidized, self-purchased) be able to select a
response they felt fits their situation from among the choices offered? Would any of the types of
coverage listed in the question receive false positives or false negatives? Because the question
was tested without changes across rounds, the findings from both rounds are discussed together.

Attachment V

A number of basic concepts were probed on, to make sure respondents understand them as
intended. The concept of 'health insurance or health coverage plans' was well understood across
languages. The Concept of 'current' coverage was also universally well understood and
interpreted as intended, that is, as coverage in effect at the time of interview. No problems or
issues were encountered in English or Spanish.
None of the sixty respondents interviewed reported having military or VA coverage, or coverage
from the Indian Health Service, in question 16 or in the original screening.
In the first round there were 8 Commonwealth Choice (exchange - private plan - no subsidy)
respondents and in the second round there were 6. In round 1 they divided almost evenly
between those who answered Yes to 'Other' (16h) and specified Commonwealth Choice or the
plan name, and those who answered Yes to 'Purchased directly from ...' (16b) (though one
respondent clarified that the purchase is not directly from an insurance company). One
respondent left the question blank because he was not sure where to classify coverage and did
not use Other, and another respondent gave multiple answers (Yes to 16a-ESI, Yes to 16h
(Other: Tufts), and Don't Know to 16b (direct purchase)). This latter respondent was screened as
having Commonwealth Choice but said she gets Commonwealth Choice through her employer.
In round 2, three answered direct purchase, two answered Other and specified the plan name, and
one answered Don't Know at direct purchase.
There were a variety of responses for those who have Commonwealth Care (exchange plan fully
or partially subsidized). In round 1, the largest group (n = 8) answered Yes to 16d (Medicaid...).
The others answered with multiple responses. Several respondents who are on MassHealth did
not feel any of the options listed/read included their plan and said Yes to Other, while others
answered yes to 16d (Medicaid...). In round 2, two-thirds of those who pay a premium under
Commonwealth Care answered Yes to Direct Purchase, while the remainder answered Medicaid,
MassHealth, or provided the plan name. There were no respondents in round 2 who had fully
subsidized Commonwealth Care.
There were a few other varied issues in question 16. First, among those saying Yes to multiple
answers, there appeared to be confusion as to what coverage they have, mostly stemming from
lack of knowledge or clarity, and not because of issues interpreting the question. For instance,
some respondents who were uncertain if their coverage was Medicare or Medicaid chose both.
The definition of Medicaid offered to them appeared to help some of the Commonwealth Care
and some of the MassHealth respondents answer Yes, as the mention of low income suggested
this fit their situation.

Attachment V

In the self-administered version, the instruction at Q16 to Mark Yes or No for EACH type of
coverage was disregarded (not read, not understood, or forgotten after marking a Yes) in most
Spanish cases and few English cases. Among those who understood this instruction well, not all
complied, as once they found a response that fit them, they left the rest blank. In a few cases, if
anything was left blank, it was because the respondent was not sure what to answer.
Alternative wording was tested for question 16b: "Purchased directly" [comprado directamente,
for Spanish] without mention of insurance company There was no strong evidence suggesting
this change would offer a better alternative. For some respondents this alternative begs the
question "From whom?" or sounds like it is missing something. Worthy of mention is the fact
that the Spanish self-administered version has one difference in wording with the CATI/CAPI
version: for direct purchase, the paper copy uses the term 'adquirido' [purchased/acquired]
instead of 'comprado' [purchased/bought]. Probing showed that not all respondents are familiar
with the term 'adquirido,' while all understood 'comprado' as intended.
In the self-administered version, the phrase in parentheses at 16a and 16b that says "(this person
or another family member)" was probed in round 1 among the few cases that chose one of those
answers. No evidence of difficulty interpreting this phrase was identified in either language.
Among those who answered Yes to Other in round 1, the largest group (n=9) indicated they
chose it because they did not feel their coverage fit anywhere else (2 had MassHealth, 3 had
Commonwealth Care, and 4 had Commonwealth Choice). Some used 16h to specify the name of
the plan they had already reported under the right type of program above. In round 2 only three
respondents selected Other, and they did so either because they did not find their program listed
(Commonwealth Choice; MassHealth) or wanted to specify the plan name.

Question Version Findings
Question 17
In each round, there were two versions tested for question 17. In the first round, the two versions
were:

Attachment V

Version A
English: Is the cost of your health insurance reduced based on your family income?
Spanish: ¿Se ha reducido el costo del seguro de salud suyo basado en el ingreso de la familia?
Version B
English: Is the cost of the health insurance premium reduced based on your family income?
Spanish: ¿Se ha reducido el costo de cuota mensual del seguro de salud basado en el ingreso de
su familia?

All round 1 respondents answered this question except for two who in the self-administered
version skipped it because they did not read or properly understand the instruction above it. The
two versions differed in terms of what may be reduced based on family income: the cost of
health insurance versus the cost of the health insurance premium.
A main issue with these questions, in either version, was that respondents who pay no premium
or out of pocket expenses for their coverage were confused as to how to respond. The question
asks about a cost they do not incur. For those who pay no premium but do pay out of pocket
expenses, Version B was more problematic than Version A. Also, when probed in Version B,
most showed they understood the term 'premium', but not all.
Whose income did respondents consider in formulating their answer? This depended on the size
of the household. Respondents in one-person households mostly only thought about their own,
not their extended family living elsewhere. Respondents living with family members for the
most part considered the full family income. However, the mention of family income was
confusing to some respondents in one-person households and to some who may live with family
but keep separate finances (an adult child, for instance).
How did each version perform? More of the MassHealth-covered respondents answered No to
Version B than to Version A. Commonwealth Choice respondents tended to answer No in both
versions; many indicated that income is not asked by the program. Commonwealth Care
respondents answered Yes or No in about the same proportions to both versions. Regardless of
Version, when respondents were offered the alternate version A or B, the majority felt it asked
the same question as they had just responded to in question 17.

Attachment V

The concept of 'is reduced' presented a number of problems. First, it was interpreted differently
by different respondents. While some thought it meant that the cost had already been reduced
and they were paying a smaller amount, others interpreted the phrase as a hypothetical 'would it
be reduced' should their income change. Some respondents simply did not know the answer,
despite interpreting the question as intended, that is, they did not know if the amount they were
paying was or was not based on their income. There was also lack of consistent interpretation as
to what the cost was reduced from, that is, whether it was reduced compared to what others paid
or to what they themselves were paying before. Finally, for those who do not pay a premium at
all, the idea of paying a reduced cost when coverage is fully subsidized was odd to some.
There were some alternate interpretations of the questions or to some aspects of them. When
cost of coverage typically goes up every year, some respondents had a difficult time thinking
about it as being reduced. In addition, in Massachusetts, Commonwealth Care respondents can
choose among several plans; those who selected a plan with lower or no premium, did not feel
the cost was reduced based on their income; they saw the cost reduced based on their choice of
plan and benefits.
What about accurate responses based on the respondent's reality? Most of those who pay no
premium answered Yes, but the ones who do pay answered Yes and No in similar numbers.
A third version was tested during probing, "Does the cost of the premium change if
family income changes?" This version was well received and appeared to do away with the
problems of 'reduced'. It was generally understood more as a hypothetical rather than as
something already in place.
Based on the round 1 findings, a number of recommendations were made for round 2, as follows:
1. Simplify the testing protocol by reducing probes that were not productive in round 1, or
where enough information was collected in round 1.
2. For question 17, if the intent of the question is to have respondents think ONLY of the
cost of the premium, the wording should refer to the premium specifically as it was in
version B. However, if the intent of the item is to include also co-pays or deductibles,
then the wording used in version A 'the cost of health insurance' should be used.
3. To avoid the confusion to some respondents of mentioning family income, consider using
phrase 'you or your family's income'.
4. For some respondents, a change in income would not imply a reduction or an increase in
premium, but rather having to change programs or coverage altogether. For instance, a

Attachment V

person covered under MassHealth or Commonwealth Care whose (family) income
increases may grow out of the program and qualify for Commonwealth Care or
Commonwealth Choice respectively. The reverse is also true should income decrease.
This makes a question about premium reduction not quite applicable in some cases.
5. Because of the issues with 'reduction' and the fact that the version "Does the cost of the
premium change if family income changes?" presented fewer problems, we recommend
testing this question with some changes. First, consider if it should be with or without the
mention of 'premium' depending on intent (see point 2 above). Second, consider change
mentioned in point 4 above.
Thus, the question could read:
A. Does the cost of the premium change if family income changes?
B. Does the cost of the health insurance change if family income changes?
C. Does the cost of the premium change if your or your family's income changes?
D. Does the cost of the health insurance change if your or your family's income
changes?
We proposed to test an alternative version to compare with whichever of the four options
above is tested in the questionnaires. This could be done either as a version B
questionnaire or by providing a showcard . We proposed to have that version read:
If there is a change in your or your family's income, would you have to pay more
or less than you currently pay for coverage?

Based on the ACS team’s decisions after considering the round 1 findings, in round 2, question
17 became a two-part question, first filtering out those respondents who do not pay a monthly
premium. The two versions tested were:

Attachment V

Version C
English:
17a. Is there a monthly premium for this plan? A monthly premium is a fixed amount of
money people pay each month to have health coverage. It does not include copays or other
expenses such as prescription costs.
Yes
No

SKIP to question 18

17b. Is the cost of the premium reduced based on family income?
Yes
No

Spanish:
17a. ¿Tiene este plan una cuota mensual? Una cuota mensual es una cantidad fija de dinero
que se paga todos los meses por la cobertura de salud. No incluye los copagos u otros gastos,
tales como los costos de las medicinas recetadas.
Sí
No --> pase a la pregunta 18

17b. ¿Se ha reducido el costo de su cuota mensual debido al ingreso de la familia?
Sí
No

Attachment V

Version D
English:
17a. Is there a monthly premium for this plan? A monthly premium is a fixed amount of
money people pay each month to have health coverage. It does not include copays or other
expenses such as prescription costs.
Yes
No
SKIP to question 18
17b. Is the cost of the premium subsidized based on family income?
Yes
No
Spanish:
17a. ¿Tiene este plan una cuota mensual? Una cuota mensual es una cantidad fija de dinero
que se paga todos los meses por la cobertura de salud. No incluye los copagos u otros gastos,
tales como los costos de las medicinas recetadas.
Sí
No --> pase a la pregunta 18

17b. ¿Está subsidiada su cuota mensual debido al ingreso de la familia?
Sí
No

How did these new versions work? First, question 17a was a welcome addition. It worked as
intended: respondents who pay a premium answered Yes, and no false positives were detected.
This was true in both languages. Also, while some respondents in English did not interpret the

Attachment V

meaning of 'premium' as intended (instead thinking of 'high quality coverage'), others knew what
it meant even if they could not adequately define it. Therefore, the soft edit (READ IF
NECESSARY) in the interviewer-administered modes proved to be necessary and useful for
some respondents.
For 17b, in Version C the term 'reduced' presented problems for respondents whose premium had
recently gone up (due to the recent beginning of the new plan year for exchange programs, this
had happened for most respondents within the last quarter). It also presented problems in
Spanish as the translation asks about the premium having been reduced, rather than paying a
reduced premium. Some respondents think 'reduced' means that the premium is lower, not that
someone else pays part of it. Findings similar to those in round 1 were also seen in this round
with the term reduced.
In Version D, 'subsidized' was not clear to all. To one respondent the question is asking people if
they agree to pay a monthly payment for their coverage. Others who understood the term and
whose coverage is actually subsidized, did not feel their coverage is subsidized because they pay
a premium and do not know of anyone else paying part of it. Additionally, in one case at least it
was received as a term with negative connotations ("handout").
Questions 16 and 17 Combined
The performance of question 16 and question 17 individually, while important and the focus of
our cognitive testing effort, do not tell the complete story. To ascertain if a respondent is
covered through an exchange and whether that coverage is fully, partially, or not subsidized at
all, it is necessary to combine the responses to both questions. While some of the story is told
simply by combining the responses, in other cases the answer(s) to question 16 will need to be
recoded based on ancillary information, either from other questions in the ACS or from the
combination of answers to 16 and 17.
The following charts for round 1 and round 2 were compiled by Carla Medalia at Census, and
show how the combination of the responses can be used to elucidate the coverage situation for
each respondent.
In addition to looking at each question individually, it is important to consider the coverage
questions in tandem and the extent to which appropriate coverage coding rules can be determined
when all available data is taken together. Carla Medalia of the Census team, reviewed the round
2 data and determined that it is possible to improve the overall accuracy of coverage reporting
when both questions 16 and 17 are considered together. In nine cases of the thirty round 2
cases, a respondent’s response for him/herself could be imputed using the two questions
combined to determine a correct code when the respondent’s actual responses were inaccurate.

Attachment V

Reviewing the two questions in tandem was especially valuable when examining the
Commonwealth Care Premium cases where respondents gave a variety of responses as they tried
to fit their situation into the categories available. Two-thirds of the time, respondents included
Direct Purchase in the response and 25% of the time they marked 16d, which includes the words
‘government assistance’ that they keyed into since they were aware that it did include a
government subsidy. In 75% of the cases respondents marked 17b, indicating that their premium
was subsidized/reduced. Interestingly, two of the three cases that did not believe they received a
subsidy/paid a reduced premium, marked 16d (government assistance). As the table illustrates
below, without using question 17 to clarify the type of coverage they have, it would be difficult
to infer that these 12 respondents all have the same type of coverage.
Commonwealth Care Premium

N=12

Direct Purchase (16b)

5

Medicaid (16d)

2

Other (16 h)

1

Direct Purchase (16b) & Other (16h)

2

Direct Purchase (16b) and Medicaid
(16d)

1

Medicaid (16d) and Other (16h)

1

Likewise, Commonwealth Choice participants were more easily identified when both questions
were taken in tandem.
Commonwealth Choice

N=6

Direct Purchase (16b)

3

Other (16 h)

2

Direct Purchase (16b) & Other (16h)

1

Participants were split between responding ‘Yes’ to 16b, Direct Purchase and 16h, Other.
However, all respondents knew they paid a premium and that the premium was not reduced.

Attachment V

IV.

Recommendations

Question 16 is in use currently by ACS and therefore will remain as is, unchanged, despite
findings from testing. However, since this is the first time the question has been tested with
respondents with exchange coverage, we recommend, on the basis of our findings and for future
testing, that Census look into the possibility of adding one or more response choices that may be
more appropriate to exchange-covered individuals. Until more is known about how different
states will publicize or name their exchanges, this should be tabled.
For question 17, given the impossibility of further testing before making a decision for a version
to field in ACS, we recommended two possibilities in advance of the final meeting:
1) Is your premium reduced or subsidized based on family income?
However, this option has never been tested with both terms, only with each separately.
2) Is your premium subsidized based on family income? To this version a definition would be
included for subsidized. Several definitions were discussed but there was no consensus. Thus,
we recommend census continue to consider the possibility of adding and testing one in the
future.
This option has been tested but without a definition. The risk here is that some respondents will
not know the meaning of subsidized, unless it is defined.
For the future, new questions may need to be designed or current ones revised as states begin
giving tax credits. As that happens, the constructs behind the questions that will need to be
asked may not correspond to anything that will be known by many respondents, as it will happen
"behind the scenes".

V.

Conclusions

This study accomplished its goals: to test the ACS question 16 in Massachusetts with a varied
population primarily consisting of individuals who have coverage through the state Exchange
and to test a new question on subsidy with the same population.
Across two rounds of cognitive testing, the new question was refined to better capture the
realities of exchange coverage, while still adequately capturing the realities of any other type of
coverage. It also helped to shed additional light on language access issues in obtaining coverage
through the State programs.
As is often the case in cognitive testing projects, regardless of the number of rounds conducted,
some questions remain at the end of the final round. In this case, a few issues remained
unresolved at the end of data collection. To find out about subsidies, the two-part question

Attachment V

approach worked best – asking first about paying a premium and then asking about whether it is
subsidized. While question 17a worked well and can be used as tested, question 17b will require
adding a soft edit (or READ IF NECESSARY definition).
The questions worked well with the limited number of non-exchange covered respondents,
thereby suggesting no new error was introduced with these questions.
As the 2014 national rollout of the ACA in all states approaches, further research should test the
questions in states with different models of exchange programs as they begin operating.