Download:
pdf |
pdfVERSION: July 18, 2014
State-Based Marketplace Website Usability
Testing Guide
Testing Materials
•
Laptop for testing
•
Laptop for note-taking
•
Audio recorder and extra batteries
•
Interviewer clock
•
Pens and notepads in each testing room
•
Information sheet
•
Interview guide
•
Incentive payments
Procedures for Obtaining Informed Consent
As interviewees arrive, greeter should have them read the information sheet (if not enough time,
interviewer should do this prior to starting). Give each person a copy of the form to keep.
Testing Goals
1. Trust of website – Who do respondents think is providing the information on the website? Do
respondents trust the source of the information? Do they trust the information? Do they trust the
processes? What would improve their trust of the web site? What would improve their trust of the
processes (e.g., applying for a subsidy, enrolling in a plan, comparing plans)?
2. Navigation of site - Is it easy for respondents to find the information they need? What changes
would improve their ability to navigate the site?
3. Understanding and application of content – How well can respondents understand and use the
information as it is presented? How well do they understand and interpret the information? How
well can they apply the information to their personal situation and help them achieve their
purpose or goal (e.g., applying for a subsidy, comparing and choosing a plan)? What changes to
display or text would improve their understanding?
1
VERSION: July 18, 2014
4. Ability to act – What do respondents plan to do with the information? What influences their
ability to act on the information presented in the website? What other information or supports do
they need to take action?
5. General response to the site – What information on the site is of greatest interest to respondents?
What do they like about the site? What don’t they like? What would they want to know or do on
the site?
Timing: 90 minutes total
Approxi
mate time
Topic
Tasks
Elaps
ed
time
5
A. Background
NA
5
5
B. Opening
NA
10
10
C. Home page
Task 1: Purpose of the website.
20
Task 2: Sponsor of the website.
20
D. Eligibility
Task 3: List of information or documents needed to apply for health
insurance.
40
Task 4: Steps to apply for health insurance.
Task 5: Appeals.
Task 6: Eligible for health plan and/or financial subsidy.
Task 7: Online calculator (if not found in Task 6).
20
E. Compare and
Choose a
Health Plan
Task 8: Find health plans.
20
F.
Task 10: Apply for health insurance.
80
5
G. Help
Function
Task 11: Get help.
85
Closing
Task 13: Word search
5
Enroll in a
Health Plan
60
Task 9: Select health plan.
Task 12: Use search bar.
Key Interview Questions and Probes
Think aloud reminders:
•
Remember to tell me your thoughts and reactions as you’re looking at the webpages.
•
Can you tell me what you’re thinking as you are working on the task?
2
90
VERSION: July 18, 2014
Track where and what participants are looking at:
•
What do you see first?
•
Can you show me which part you were looking at when you got that reaction?
•
Where are your eyes going? What are you looking at?
•
What are you looking at now? What are your thoughts?
To elicit further information:
•
And you say that because…
•
How so?
•
In what way?
•
Tell me more about that
•
Remember, there isn’t any right or wrong answer. I just want to know your honest opinion. That’s
what will help in making improvements to this website.
{Probe on significant non-verbal communication, smiles, eye-rolling, etc. Don’t over-probe nonverbal communication—i.e., don’t probe to the point that it makes participant self-conscious}
3
VERSION: July 18, 2014
[Interviewer begins reading intro here]
Background
5 minutes
•
Thank you for agreeing to do this interview. My name is [NAME] and I’ll be talking with
you today. [INTRODUCE NOTE-TAKER]
•
I work for a non-profit research organization, [American Institutes for Research].
•
We’ll be here about 90 minutes today. We won’t be taking any formal breaks, but feel free
to let me know if you need a break to go to the restroom [DESCRIBE LOCATION] or to get
something to drink.
•
Our discussion today is part of a project sponsored by the Centers for Medicare &
Medicaid Services, a federal government agency that coordinates with States to set up
Health Insurance Marketplaces, expand Medicaid, and regulate private health
insurance plans.
•
Because of the Affordable Care Act, which you may know as Obamacare, people can
purchase health insurance plans through what’s called a Health Insurance Marketplace.
Today we’ll be talking about a website developed by the [STATE NAME]. The website is
designed to help some individuals purchase health insurance.
•
We would like to learn from you how the website design can be improved.
•
We will ask you to perform a series of tasks. We will observe you as you are doing the
tasks on the website to get a sense of how well the website is working. I’ll ask you to tell me
what you are thinking as you work on each task.
•
I will try to keep silent when you are doing the task, to avoid influencing what you do. So if
you ask me questions, I may not be able to answer, though I will try to help if you get really
stuck.
•
Once you are finished with the task, we will ask you questions so we can understand how to
make the website better.
•
•
We will ask you questions to better understand your motivation behind making selections
on the website. We are not interested in your (or your family’s) personal health
information. We just want to understand your thought process when viewing information
on the website.
Remember that we are not testing you—we are testing the website. My job is just to
collect comments about the website. If something is unclear or confusing to you, it’s bound
to be confusing to other people too. If tasks are confusing, it shows problems with the design
of the website (not you).
4
VERSION: July 18, 2014
•
I would like to record our discussion today so that I can make sure I capture all of your
feedback. As I mentioned before, your full name or identity will not be associated with your
comments.
•
Everything you tell me will be confidential. We won’t connect your personal contact
information with anything that you say. While you will see us taking notes to record our
discussion, within our notes and our final report, you will be known as “Participant n.”
•
Any questions?
•
Before we begin I need to obtain verbal consent.
•
Did you have the opportunity to look over the information sheet? (If no, review the
information sheet with the participant).
Please answer yes or no to each of the following questions.
Do you understand the described project and agree to be interviewed as I guide you through
[State’s] health insurance Marketplace website?
Yes
No
Do you agree to have the interview recorded?
Do you understand that your name will not be associated with reports or documents related to
this project?
Do you understand that you can withdraw your consent at any time and stop participating in
the interview without any penalty to you?
Ground Rules
•
Please give us your honest feedback. Our goal is to make recommendations on how make
the website more easy to use, so the more feedback you offer; the more you are helping us
with improve the website.
•
We have a lot to talk about today, so there may be times when I need to move the discussion
along. Please understand that when I ask that we move to a new task, I don’t mean to
hurry you. If there is time later we can go back to pages you would like to explore more.
•
Because we’re recording, please try to speak in a voice at least as loud as the one I’m
using now so that we can make sure the tape is picking up our voices.
•
Before we begin, do you have any questions?
5
VERSION: July 18, 2014
Opening
5 minutes
1. Where, if anywhere, have you ever looked for information about health insurance?
a. Where did you go or who did you talk with to get that information?
b. What kind of information did you look at?
c. What was most important to you?
2. Before today, what did you know about state health insurance Marketplaces?
a. Have you ever gone to this website before?
b. If so, what did you use it for?
Homepage
5 minutes
As I mentioned, today we’ll be talking about a website that provides information to help people
compare and make decisions when selecting health insurance. One way that we figure out how to
improve websites is to have a small group of people look at them first so we can find out about
any potential problems and fix them.
The way we do this is by having you look at the website and tell us what you are thinking as you
look at it. In other words, we want you to talk out loud about any thoughts, feelings, reactions or
questions you have.
For example, if I asked you how many windows are in your living room you might think: “Well,
my living room and dining room are connected, so there are 6, but I guess the question is just
about my living room, so I will say 4.” Today, we’d like to hear the thoughts going through your
mind. As you do each task, go ahead and talk out loud and tell me what you’re looking at,
thinking, and what you are reading so we can understand your thought process.
Now, let’s begin with the first task.
Task 1: Find and tell me the purpose of this website.
(ALT Description: What is the website designed to do? What can this website help you with?)
[GO TO SBM HOME PAGE AND OBSERVE]
3. What do you think are the main things you do or can learn about on this website?
6
VERSION: July 18, 2014
4. Who do you think this site is for? (e.g., someone like you? Older persons? Younger
persons? Someone with a health problem?)
a. What made you think that?
5. What were your general thoughts on the homepage?
6. What are some words that describe the webpage to you? E.g., clean, simple, confusing.
[note: we will give them the word card near the end of the session]
7. Where would you want to go to next from this webpage?
8. What, if anything, on this page did you find to be confusing or unclear?
a. What could improve it?
Task 2: Find out who is running this website (or what organization is in charge of
the website)?
(ALT Description: who or what organization is putting this website together?)
[GO TO SBM HOME PAGE AND OBSERVE]
9. Who do you think is running this Web page?
a. How did you figure that out?
b. What do you think this organization does?
c. Do you feel like the information on this page can be trusted? Tell me more about
that.
Eligibility
20 minutes
Task 3: Find a list of the types of information or documents you would need to
apply for health insurance for you (and/or your family) through the website.
(Alt. Description: Before sitting down at the computer to apply for health insurance, let’s say you wanted
to know what type of information you would need to apply. See if you can find a list of the type of
information or documents you would need in order to apply.)
[OBSERVE]
10. To what extent was the information located where you expected?
7
VERSION: July 18, 2014
11. What, if anything, was confusing or unclear?
a. What could improve it?
Task 4: Identify the steps (or process) to apply for a health insurance plan through
the Marketplace website.
(Alt. Description: Before sitting down to apply for health insurance, let’s say you wanted to understand
all of the steps of the application process. For example, what would you do first to apply and what would
you need to do last. See if you can identify the steps of this process or what the process entails from start
to finish.)
[OBSERVE]
12. To what extent was information on the steps to apply for insurance located where you
expected?
13. What, if anything, was confusing or unclear?
a. What could improve it?
Task 5: Find information about how to appeal a decision from the Marketplace.
(Alt. Description: Let’s say that you have applied for health insurance through the website, and
the Marketplace tells you that you are not eligible for financial assistance. However, you believe
this is incorrect and that you really are eligible. You want to know how to appeal (or challenge)
the Marketplace’s decision and let them know you don’t agree. Now try to find information on
how you would go about this appeal.
To appeal means to tell someone at the Marketplace that you think the decision is wrong, and ask
for a fair review of the decision about how much you would have to pay for your health
insurance.)
[OBSERVE]
14. To what extent was this information located where you thought it would be?
15. What, if anything, was confusing or unclear?
a. What could improve it?
8
VERSION: July 18, 2014
Task 6: Find out if you (and your family) are eligible to:
• purchase health insurance through the website, and
• receive financial assistance to purchase the health insurance (get help paying
for your health insurance).
(Alt. Description: let’s say that you want to get health insurance through this website. How
would you find out if you could purchase it through the website? How would you find out if you
could get financial help paying for your insurance through the website?)
[OBSERVE]
16. To what extent was this information located where you thought it would be?
17. What, if anything, was confusing or unclear?
a. What could improve it?
18. Do you feel like the information on this page can be trusted? Tell me more about that.
Task 7: Use the online eligibility calculator to determine whether you could receive
financial assistance to purchase health insurance through the Marketplace.
[This task should only be done if the respondent did not find the eligibility calculator from the prior task]
19. What, if anything, is unclear or confusing?
a. What could improve it?
Compare and Choose a Health Plan
20 minutes
Now, we want to think about comparing the health plans available on the website. You want to
be sure that there are health plans that fit your needs.
Task 8: Find out what health insurance plans are available to you through this
website and compare the features of at least two of the health insurance plans that
you think might be a good fit for you.
(Alt. Description: Let’s say you want to review the health plan options and compare at least two
of the plans to see which one you may prefer to purchase. Show me at least two health plans that
you think may be a good fit for you.)
[OBSERVE]
9
VERSION: July 18, 2014
We are now going to ask you some questions to better understand your motivation behind
comparing and choosing a health plan. We are not interested in your (or your family’s) personal
health information. We just want to understand your thought process when viewing these plans
on the website.
20. To what extent was this information located where you thought it would be?
21. What, if anything, was confusing or unclear?
22. What were some of the main differences between the health insurance plans?
23. What features were the most important to you? What makes them important to you?
24. What types of information about the health insurance plans was missing or not included?
25. What terms or words about the health insurance plans were unclear?
26. What could make it easier for you to compare at least two health plans?
27. At what points did you feel like you needed assistance?
a. How would you prefer to receive this assistance (phone, Web, in-person)?
b. What would you ask?
c. What makes you prefer to get the information in this way?
Task 9: Now that you have reviewed the different health plans, let’s say that you are
ready to make a decision and select a health plan. Select the health plan that would
be best for you [and your family].
(Alt. Description: think about the health plans that you just looked at and consider which one of
them you would select for your insurance. Show me which plan you would pick.)
[OBSERVE]
28. Help me understand why you chose that plan?
29. What health plan features were most important to you (and/or your family) and why?
a. Doctors or hospitals
b. Costs (probe on what types of costs)
10
VERSION: July 18, 2014
c. Benefits, such as:
i. Prescription medicines
ii. Physical, occupational, or speech therapy
iii. Home health care or assistance
30. What, if anything, was confusing or unclear?
31. Do you feel like the information on this page can be trusted? Tell me more about that.
Enroll in a Health Plan
20 minutes
[NOTE: Some sites require user name and passwords to enroll in a plan. If this is the case,
we will ask the states for dummy test accounts for the purpose of testing. If we have dummy
test accounts, please complete this section with the participant. If not, please proceed to the
next section (Help Function).]
Now that you determined whether you are eligible for health insurance, you have selected a
health plan, go ahead and …
Task 10: Apply for health insurance using the Marketplace website.
(Alt. Description: Go ahead and get (or sign up for) health insurance.)
Don’t worry, the State has given us a fake username and password and you do not need to
provide your real personal information.
[OBSERVE]
32. To what extent was this information located where you thought it would be?
33. What, if anything, was confusing or unclear?
a. What could improve it?
34. How did you feel going through this process?
35. What, if anything, did you feel like you needed assistance with?
a. How would you prefer to receive this assistance (phone, Web, in-person)?
b. What would you ask?
11
VERSION: July 18, 2014
c. What makes you prefer to get the information in this way?
36. Do you feel like the information on this page can be trusted? Tell me more about that.
Help Function
5 minutes
Now we want to focus on the help that is available on the website.
Task 11: Find out where you could get help if you were using this website and had
questions.
(Alt. Description: Let’s say you are using the website and you run into any type of question,
where or how would you get help.)
[OBSERVE]
37. To what extent was this information located where you thought it would be?
38. What, if anything, was confusing or unclear?
39. How else would you have liked to have been able to get help?
40. How would you prefer to get help? (website, phone, in-person). Please explain.
a. What would you ask?
b. What makes you prefer to get the information in this way?
Now, please think back to a specific topic that you would have liked to get more information
about. This may have been a term that you saw that was unfamiliar or one of the questions that
you asked me while you were performing the tasks.
Task 12: Imagine you want more information on a specific topic. Pick the topic
(e.g., health plan) and use the search bar to find information on the topic.
(Alt. Description: First locate the search bar, if it is available. Then use the search bar to find
information on a topic. You may search for anything you would like. You may want to search for
something you couldn’t find earlier).
[OBSERVE]
41. How close is what you found to what you hoped to find?
42. What do you think of what you’re seeing?
12
VERSION: July 18, 2014
43. How would you use this?
44. What, if anything, was confusing or unclear?
45. What else, if anything, could help you find that topic more easily?
Closing
5 minutes
Task 13: Now that you have used the website, can you circle [or select] 5
words that describe your experience using the website [hand word card sheet
or cards]?
Before we end, I’d like to give you a chance to share any additional thoughts or comments about
the items we talked about today. Was there anything that you didn’t have a chance to say during
our discussion today or something that we didn’t talk about that you wish we had?
Thank you very much for participating in this discussion today. We appreciate your time.
13
VERSION: July 18, 2014
Please circle the 5 words that best describe your experience using the website 1. Thank you!
Accessible
Desirable
Gets in the way
Patronizing
Stressful
Appealing
Easy to use
Hard to use
Personal
Time-consuming
Attractive
Efficient
High quality
Predictable
Time-saving
Busy
Empowering
Inconsistent
Relevant
Too technical
Collaborative
Exciting
Intimidating
Reliable
Trustworthy
Complex
Familiar
Inviting
Rigid
Uncontrollable
Comprehensive
Fast
Motivating
Simplistic
Unconventional
Confusing
Flexible
Not valuable
Slow
Unpredictable
Connected
Fresh
Organized
Sophisticated
Usable
Consistent
Frustrating
Overbearing
Stimulating
Useful
Customizable
Fun
Overwhelming
Straight Forward
Valuable
Developed by and © 2002 Microsoft Corporation. All rights reserved.
1
An alternative approach we may use is to instruct the respondent to do the following: Please select five cards
from this pile that best describe your experience using the website.
14
OVERVIEW
Recruit to have a minimum of 3 interviews per target population per SBM website (2
target populations x 3 interviews per target population = minimum of 6 respondents
per SBM website). Max of 90 interviews total.
Each interview will last 90 minutes
Participants will get $75 for the website usability testing.
General screening criteria for all interviews 1:
1. Must be a U.S. citizen or national;
2. Must have used the internet in the past three months and have access to the internet
either at home or through a public venue such as the library
3. Must not have applied or enrolled via the SBM Website being tested during the
fall 2014/winter 2015 open enrollment period for 2015 coverage
4. Must be responsible for selecting health insurance for themselves and family, if
applicable
5. Must be comfortable reviewing a website in English and answering questions in
English
6. Must be between the ages of 18-64 2
7. Mix of geographic representation (e.g., people living in urban, suburban, and rural
areas)
8. Mix of genders
9. Mix of race/ethnicity
10. Mix of education (high school degree and some college)
11. Mix of family size (single and families)
12. Mix of eligibility for subsidies (eligible for premium subsidies) and no subsidy 3
(see criteria in Figure 1 for number of people in household and income)
13. Mix of enrolled and not enrolled for the 2014 coverage year
1
Screening criteria may be revised as needed based on recruitment experiences to ensure respondents are
comparable across sites.
2
According to the MIDAS data on online applicants and enrollees from the 36 states that used healthcare.gov,
the median age for enrollees is 45. One person per state that is under 26 will be accepted and one per state
that is over 50 will be accepted.
3
One person per state who is eligible to purchase through the Marketplace but not eligible for subsidies will be
accepted.
1
Figure 1. Health Exchange Tax Credits Eligibility (source: https://www.healthcare.gov/how-can-i-save-money-onmarketplace-coverage-chart/)
Additional screening criteria specific to each group
a. Group 1: Healthy (no chronic conditions)
•
Must not have been diagnosed with a chronic condition such as asthma,
diabetes, hypertension, etc.
b. Group 2: Individuals with at least one chronic condition
•
Must not have been diagnosed with a chronic condition such as diabetes,
asthma, heart disease, depression, congestive heart failure, cancer, arthritis,
COPD, kidney disease, etc.
2
RECRUITMENT SCREENER
State Based Marketplace Consumer Usability Testing
INTRO WHEN PERSON ANSWERS THE PHONE:
Hello, may I please speak with [FIRST AND LAST NAME]? My name is [RECRUITER
FIRST AND LAST NAME], and I'm calling on behalf of the American Institutes for Research,
a nonprofit research institute.
IF RESPONDENT NOT AVAILABLE EITHER:
DETERMINE A GOOD TIME TO CALL BACK OR
LEAVE MESSAGE ON ANSWERING MACHINE
ANSWERING MACHINE SCRIPT:
Hello, my name is [RECRUITER FIRST AND LAST NAME]. I'm calling on behalf
of the American Institutes for Research, a nonprofit research institute. We would
like for you to participate in an interview to test and provide feedback on a website
developed by [STATE BASED MARKETPLACE NAME]. Please call {INSERT
PHONE NUMBER} at your convenience.
IF RESPONDENT IS AVAILABLE, IDENTIFY REASON FOR CALLING.
I’m calling today about a research project sponsored by the Centers for Medicare & Medicaid
Services, a federal government agency that coordinates with States to set up Health Insurance
Marketplaces, expand Medicaid, and regulate private health insurance plans.
Because of the Affordable Care Act, which you may know as Obamacare, people can
purchase health insurance plans through what’s called a Health Insurance Marketplace. The
Marketplaces have developed websites that are designed to help some individuals purchase
health insurance.
We are looking for individuals, such as yourself, to use a state’s Health Insurance
Marketplace website and describe your experience while using the website. We will ask you
to perform a series of tasks. We will observe you as you are doing the tasks on the website to
get a sense of how well the website is working. We’ll ask you to explain what you are
thinking as you work on each task. There will be no right or wrong answers. We just want to
observe and hear about your experience.
Based on the feedback we get from you and others who use the website, we will make
recommendations about how to improve the website.
If you are interested in helping with this project, and you meet the requirements for
participation, we will invite you to come to our facility in [INSERT LOCATION NAME] for
a focus group discussion. It would take 90 minutes of your time, and we would give you $75
to thank you for participating.
3
Would you potentially be interested in testing the website?
IF YES, CONTINUE TO QUESTION 1. IF NO, THANK & END (END SCRIPT A).
May I ask you a few questions to see if you have the characteristics that AIR is looking for in
participants to test the websites?
IF NEEDED, EXPLAIN FURTHER: We need to include people with specific
characteristics and experiences and who are from a mix of different backgrounds. So I
have to ask you some questions to see if you fit the profile of participants that AIR is
looking for to use the websites.
IF YES, CONTINUE TO QUESTION 1. IF NO, THANK & END (END SCRIPT A).
Before we begin, I’d like to let you know that all information you provide will be held in
confidence. We are asking for this information to understand whether you are eligible for the
testing. We won’t connect your personal contact information with anything that you say
during the testing. Within our notes and our final report, you will be known as “Participant
n.”
Your help is voluntary, and you may ask me to skip any questions that you do not wish to
answer. You can stop at any time. None of the information that you provide to us will be
used for any purpose outside of this study.
1. Are you a US citizen or national?
YES
NO
IF NO, USE END SCRIPT B
2. Have you used the internet in the past three months and do you have access to the internet
either at home or through a public venue such as the library?
YES
NO
IF NO, USE END SCRIPT B
3. Have you applied or enrolled in a health insurance plan via the Health Insurance
Marketplace website in [INSERT STATE NAME OF POTENTIAL RESPONDENT
HERE] during the fall 2014/winter 2015 open enrollment period for 2015 coverage?
YES
NO
IF YES, USE END SCRIPT B
4
4. Are you responsible for obtaining health insurance for you or your family (if respondent
has family)?
YES
NO
IF NO, USE END SCRIPT B
5. Are you comfortable reviewing a website and English and answering questions about your
experience using the website in English?
YES
NO
IF NO, END SCRIPT B
6. What is your age?
years old
IF YOUNGER THAN 18 OR OLDER THAN 64, END SCRIPT B
{Recruit 1 per state that is under 26 and one per state that is over 50]
/-------------END OF RESPONDENT ELIGIBILITY CRITERIA--------------/
/-----------------------------------BEGIN MIX CRITERIA-----------------------/
7. How would you describe the place where you live?
Urban
Suburban
Rural
{Recruit a mix of participants who live in urban, suburban, and rural areas.}
8. What is your gender?
Male
Female
{Recruit a mix of men and women}
5
9. Are you of Spanish, Hispanic, or Latino background?
YES
NO
{Recruit a mix of races/ethnicity using Questions 10 and 11}
10.
How would you describe your race? (check all that apply)
American Indian or Alaskan Native
Asian or Pacific Islander
Black or African American
White
Another race
Specify other _______________________________
{Recruit a mix of races/ethnicity using Questions 9 and 10}
11. What is the highest grade or level of school you have completed?
Less than high school graduate
High school diploma
GED
Technical or vocational school or certificate
program
Associate’s Degree (2 year college graduate)
Some college (1 – 3 years of college, no degree)
Bachelor’s Degree
Graduate degree (Master’s or Doctorate)
{Recruit a mix of educational levels, preferably skewed to the low end of the
educational spectrum}
/-----------------------------------START SUBSIDY SCREENER QUESTIONS--------------/
12.
How many people, including yourself, currently live in your household?
People
6
Now, we want to ask you some questions about your household income. Please think about
your total household income (not just your income if you have additional members in your
household).
Which of the following categories best describes the total estimated income of your household
in 2014, before taxes? Please include before-tax income from all sources such as salaries and
wages, Social Security, retirement income, investments, and other sources.
GO TO THE FOLLOWING QUESTION BASED ON HOUSEHOLD SIZE
1 PERSON: GO TO QUESTION 13
2 PEOPLE: GO TO QUESTION 14
3 PEOPLE: GO TO QUESTION 15
4 PEOPLE: GO TO QUESTION 16
5 PEOPLE: GO TO QUESTION 17
6 PEOPLE: GO TO QUESTION 18
CHECK “YES” IN THE CATEGORY THAT APPLIES TO THE RESPONDENT. DO
NOT REPORT ELIGIBILITY TO RESPONDENT.
13.
/-----------------------------------1 PERSON IN HOUSEHOLD-----------------------/
Would you say…
YES
READ
Less than $16,105
If yes, then stop and go to
Question 19
DO NOT READ
Medicaid-eligible (if state expanded
Medicaid) or may be eligible for Qualified
Health Plan (QHP) subsidies [QHPs are the
health plans provided through the
Marketplace]
May be eligible for subsidies with QHPs
Less than $45,960
If yes, then stop and go to
Question 19
More than $45,960
If yes, then stop and go to
Question 19
Not sure/don’t know/doesn’t
want to answer
If yes, then stop and go to
Question 19
No subsidies
{Recruit a mix of participants who qualify for financial subsidies (Medicaid or QHPs).
Only include one person per state who is not eligible for subsidies }
GO TO QUESTION 19
7
/-----------------------------------2 PEOPLE IN HOUSEHOLD-----------------------/
14.
Would you say…
YES
READ
Less than $21,707
If yes, then stop and go to
Question 19
Less than $62,040
If yes, then stop and go to
Question 19
More than $62,040
If yes, then stop and go to
Question 19
Not sure/don’t know/doesn’t
want to answer
If yes, then stop and go to
Question 19
DO NOT READ
Medicaid-eligible (if state expanded
Medicaid) or may be eligible for QHP
subsidies
May be eligible for subsidies with QHPs
No subsidies
{Recruit a mix of participants who qualify for financial subsidies (Medicaid or QHPs).
Only include one person per state who is not eligible for subsidies }
GO TO QUESTION 19
/-----------------------------------3 PEOPLE IN HOUSEHOLD-----------------------/
15.
Would you say…
YES
READ
Less than $27,310
If yes, then stop and go to
Question 19
Less than $78,120
If yes, then stop and go to
Question 19
More than $78,120
If yes, then stop and go to
Question 19
Not sure/don’t know/doesn’t
want to answer
If yes, then stop and go to
Question 19
DO NOT READ
Medicaid-eligible (if state expanded
Medicaid) or may be eligible for QHP
subsidies
May be eligible for subsidies with QHPs
No subsidies
{Recruit a mix of participants who qualify for financial subsidies (Medicaid or QHPs).
Only include one person per state who is not eligible for subsidies }
GO TO QUESTION 19
8
-----------------------------------4 PEOPLE IN HOUSEHOLD-----------------------/
16.
Would you say…
YES
READ
Less than $32,913
If yes, then stop and go to
Question 19
Less than $94,200
If yes, then stop and go to
Question 19
More than $94,200
If yes, then stop and go to
Question 19
Not sure/don’t know/doesn’t
want to answer
If yes, then stop and go to
Question 19
DO NOT READ
Medicaid-eligible (if state expanded
Medicaid) or may be eligible for QHP
subsidies
May be eligible for subsidies with QHPs
No subsidies
{Recruit a mix of participants who qualify for financial subsidies (Medicaid or QHPs).
Only include one person per state who is not eligible for subsidies }
GO TO QUESTION 19
17.
/-----------------------------------5 PEOPLE IN HOUSEHOLD-----------------------/
Would you say…
YES
READ
Less than $38,516
If yes, then stop and go to
Question 19
Less than $110,280
If yes, then stop and go to
Question 19
More than $110,280
If yes, then stop and go to
Question 19
Not sure/don’t know/doesn’t
want to answer
If yes, then stop and go to
Question 19
DO NOT READ
Medicaid-eligible (if state expanded
Medicaid) or may be eligible for QHP
subsidies
May be eligible for subsidies with QHPs
No subsidies
{Recruit a mix of participants who qualify for financial subsidies (Medicaid or QHPs).
Only include one person per state who is not eligible for subsidies }
GO TO QUESTION 19
9
/-----------------------------------6 PEOPLE IN HOUSEHOLD-----------------------/
18.
Would you say…
YES
READ
Less than $44,119
If yes, then stop and go to
Question 19
Less than $126,360
If yes, then stop and go to
Question 19
More than $126,360
If yes, then stop and go to
Question 19
Not sure/don’t know/doesn’t
want to answer
If yes, then stop and go to
Question 19
DO NOT READ
Medicaid-eligible (if state expanded
Medicaid) or may be eligible for QHP
subsidies
May be eligible for subsidies with QHPs
No subsidies
{Recruit a mix of participants who qualify for financial subsidies (Medicaid or QHPs).
Only include one person per state who is not eligible for subsidies }
/-----------------------------------END SUBSIDY SCREENER QUESTIONS------------------/
19. Did you enroll in insurance through the Health Insurance Marketplace in [INSERT
RESPONDENT’S STATE NAME HERE] 2014?
YES
NO
{Recruit a mix of enrollment history}
20. Did you have insurance coverage between October 1, 2013 and September 30, 2014?
YES
NO
PART OF YEAR
10
21. Has the doctor ever told you have: {ask each}
Diabetes
YES
NO
Asthma
YES
NO
Chronic Obstructive
Pulmonary Disease 4
YES
NO
Heart Disease
YES
NO
Cancer
YES
NO
Arthritis
YES
NO
Kidney Disease
YES
NO
Anxiety
YES
NO
Depression
YES
NO
Other chronic condition
YES
NO
Specify Other:
USE QUESTION 21 TO DETERMINE IF RESPONDENT HAS A CHRONIC
CONDITION
NO Diagnosed Chronic Condition (Group 1)
YES Diagnosed Chronic Condition (Group 2)
{Recruit a mix of those with and without chronic conditions using Question 21. }
INVITATION IF INDIVIDUAL MEETS RECRUITING CRITERIA:
Thank you for answering all of my questions. It looks like you’re eligible to participate in the
session to use the website.
We are having the session to use the website on XXXXXX. This will take 90 minutes and
you will be given $75 as a thank you for participating.
Would you be able to come participate in the session to use the website and tell us about your
experience at that date and time?
IF YES, CONFIRM APPOINTMENT
4
COPD is a term used to describe some lung diseases including emphysema and chronic bronchitis
11
So, let me confirm, that you are agreeing to use the Health Insurance Marketplace website and
tell us about your experiences while using it on XXXX.
IF NO:
{GO TO END SCRIPT A}
CONFIRM RESPONDENT’S INFORMATION
Now, let me just verify the spelling of your name and your address, so we can send you a
confirmation letter with directions. We will not use this information for any purpose outside
of this study.
Name:______________________________________Telephone:_______________________
Address:____________________________________________________________________
City,
State:____________________________________________Zip:____________________
Also, please let me confirm that I have all your correct phone numbers so that we may call
and confirm with you.
Home phone: __________________________ Other Phone: __________________________
If you have any questions or find that you can’t attend, please call us right away at {insert
phone number} so that we can find a replacement. Thank you for your time and for agreeing
to help.
END SCRIPT A (GENERIC THANK AND END):
I appreciate your taking the time to speak with me and I hope you have a good day.
END SCRIPT B (FOR PEOPLE WHO DO NOT MEET SELECTION CRITERIA):
Thank you very much for answering my questions. As I said earlier, we’re trying to recruit
people with specific characteristics and a mix of backgrounds. Unfortunately {you do not
meet our selection criteria OR we already have enough people in our study with
backgrounds that are similar to yours}. I appreciate your taking the time to speak with me
and I hope you have a good day.
12
Cognitive Testing Guide Outline:
Marketplace Enrollee and Qualified Health Plan
Survey Reports
Audiences: Consumers, small business employers, health plan quality managers,
and Marketplace regulators
Fall 2014
Testing materials
In-person interview materials
Interview guide
Laptops for note-taking
Interviewer clock
Pens and notepads in each testing room
Informed consent forms (1 copy for participant to sign, plus 1 copy for participant to
keep)
Packet of testing materials (1 per interviewee) that contains hard copy materials to
review.
Receipt form
Incentive
Telephone interview materials
Interview guide
Laptops for note-taking
Interviewer clock
Informed consent forms (1 copy for participant to sign, plus 1 copy for participant to
keep)
Packet of testing materials (1 per interviewee) that contains hard copy materials to
review.
Interviewee contact information
Procedures for obtaining informed consent
In-person interviews
As interviewees arrive, greeter should have them read and sign the informed consent form (if not
enough time, interviewer should do this prior to starting). Give each person an unsigned copy of
the form to keep.
Telephone interviews
Before we begin I need to obtain verbal consent from you.
o Review the consent form with the participants.
o Obtain verbal consent.
o Resend the consent form to the participants electronically for their records if
requested.
1
Do you understand the described project and interview and agree to be a
participant as part of this project?
Yes
No
Do you agree to have the interview recorded?
Yes
No
Do you understand that your name will not be associated with reports or
documents related to this project?
Yes
No
Do you understand that you can withdraw your consent at any time and stop
participating in the interview without any prejudice to you?
Yes
No
Testing goals
One-on-one testing will assess the usability and understanding of measure labels and potential
reporting displays for the Marketplace Survey results and QHP Enrollee Experience Survey
results.
Through one-on-one testing, interviewers will use think-aloud techniques—asking the participant
to process what they are seeing aloud with interviewers—and direct probing to evaluate
participants’:
o Understanding of the measures and composites: What information do users expect
here? Do users want less, more, or the same amount of data? Can users understand
and correctly interpret the information here? Do the measures that make up the
composites make sense to users? What text changes may be needed?
o Understanding of potential reporting displays: Can users understand and correctly
interpret and make a decision using the data displays as shown? What text changes
may be needed?
o For Marketplace regulators the focus of testing will be on results that
summarize information from the QHP and Marketplace surveys.
o For consumers, small business employers, and health plans, testing of QHP
Enrollee Survey results will be done in alignment with consumer testing
feedback for the public reporting of the QRS and other potential state-based
reporting efforts.
2
Timing
Topic
Approximate time
Consumers
Employers
Health Plan;
Regulators
Introduction
8
8
6
Overview
15
12
12
Label and language testing
65
38
40
Closing
2
2
2
Key interview questions and probes
Think aloud reminders:
• Remember to tell me your thoughts and reactions as you’re looking at the handout.
• Can you tell me what you’re thinking about now?
Track where and what participants are looking at:
• What do you see first?
• Can you show me which part you were looking at when you got that reaction?
• Where are your eyes going? What are you looking at?
• What are you looking at now? What are your thoughts?
To elicit further information:
• And you say that because…
• How so?
• In what way?
• Tell me more about that
• Remember, there aren’t any right or wrong answers. I just want to know your honest opinion.
That’s what will help in making improvements to this handout.
{Probe on significant non-verbal communication, smiles, eye-rolling, etc. Don’t over-probe nonverbal communication—i.e., don’t probe to the point that it makes participant self-conscious}
3
Introduction
•
Thank you for agreeing to do this interview. My name is [NAME] and I’ll be talking with
you today. [INTRODUCE NOTE-TAKER]
•
I work for a company called the American Institutes for Research, which is an independent
non-profit research organization.
•
Our discussion today is part of a project sponsored by the Centers for Medicare and
Medicaid Service, a government agency working to provide better care, access to coverage,
and improved health for all Americans.
•
Today we’ll be talking about some materials that are being developed to help:
o [Marketplace survey] improve consumer experience to compare and make decisions
in the Health Insurance Marketplace.
o [QHP survey] compare, make decisions, and improve quality of health plans
•
I’ll be asking you to look at materials on paper. Based on the feedback we get from you and
others, we will make recommendations about how to improve the materials. Eventually,
these materials will be available to the public.
•
Any questions?
Ground rules
•
We’ll be here about [1.5 hours/an hour today]. We won’t be taking any formal breaks, but
feel free to let me know if you need a break.
•
Everything you tell me will be confidential. To protect your privacy, we won’t connect your
name with anything that you say.
•
Please give us your honest feedback. Remember that we’re not testing you—we’re testing
the materials. My job is just to collect comments about them. If something is unclear or
confusing to you, it’s bound to be confusing to other people too.
•
I would like to record our discussion today so that I can make sure I capture all of your
feedback. As I mentioned before, your name or identity will not be associated with your
comments. Because we’re recording, please try to speak in a voice at least as loud as the
one I’m using now so that we can make sure the tape is picking up our voices.
•
We have a lot to talk about today, so there may be times when I need to move the discussion
along. Please understand that when I ask that we move to a new topic, I don’t mean to be
rude.
•
Before we begin, do you have any questions?
4
Overview
1. [Consumers] What information would you want to see to help you and (your family) pick a
health plan?
2. [Professionals] How often do you view comparative or quality improvement reports online?
In paper?
o Where do you find the definitions of the measures?
As I mentioned, today we’ll be talking about some materials that are being developed to help
people [compare and make informed decisions when selecting a health plan/review how well the
Marketplace provided services to people]. A survey was given to people to take and the materials
we are about to show you summarize the results of those surveys. The information here is
realistic, but not actually real data.
Before materials like these are made available to the public, we like to have a small group of
people look at them first so we can find out about any potential problems and fix them. The way
we do this is by having you look at the materials and tell us what you are thinking as you look at
them. In other words, we want you to talk out loud about any thoughts, feelings, reactions or
questions you have.
I am going to give you a few minutes to look over this page, and then we will talk about it
together. Go ahead and talk out loud and tell me what you’re looking at, thinking, and clicking
on. You may click on any link that you would like. If you click on a link we will be discussing
later, I will let you know.
3. What do you think about this page? What were your first impressions of this page? When
you first saw it what was your reaction?
4. How well does this match what you expected to find when told you would see a summary of
the results of surveys on [Marketplace experiences/qualified health plans]?
5. Which columns did you notice first? Which have the most important information to you?
6. In your own words, what is the table on this page about? Probe if needed: What is this table
showing you?
7. What do you think these {point to word icons} mean?
8. [Consumers, Health Plan, Employer] If you had to choose a plan based on the information on
this page, which plan would you choose? For what reasons did you select this plan?
9. Look at {Sample plan/Marketplace}. What can you say about this [Plan/Marketplace]?
10. Please tell me about why there is no [word icon] for this [plan/Marketplace].
o How would you explain why [plan/marketplace] has no information for this topic?
5
11. What are the differences between the [plans/Marketplaces[on this page? In your own words,
how would you describe the difference between {Sample plan A/Marketplace A} and
{Sample plan B/Marketplace B}?
12. In your own words, what does [Summary Title TBD for each Survey] mean to you?
o Now that we’ve discussed this information, what other word(s) would you suggest we
use instead of “[Summary Title TBD for each Survey]?”
13. Are there any terms or phrases that seem unfamiliar to you or may be confusing to someone
else? How would you rewrite the information here to make it clearer?
14. How would you use the information in this graph?
15. What did you think about the amount of information on the page? Was there just the right
amount of information, too much information, or not enough information? How so?
6
Label and Language Testing
There are 17 composites or individual items for testing that will be reviewed. Below are the technical
names by survey. They will be made into consumer-friendly labels and tested using the template
provided below.
Marketplace Survey composites and individual items
a.
b.
c.
Application Process
Premium Tax Credit Eligibility
Seeking Information on the Marketplace Website (4
items)
d. Seeking Information over the Phone
e. Seeking Information In-Person
f. Health Plan Enrollment Process
g. Specialized Services
h. Cultural Competence
i. Reasons why respondent did NOT give information
about the people in family who wanted health
insurance
j. Reasons why respondent did NOT give household
income information
k. Reasons why respondent did NOT get the
information you needed from
l. What kind of information on the {MARKETPLACE
NAME}’s website was NOT easy to understand
m. Were any of the following a reason why you did
NOT get the information or help you needed when
you called the {MARKETPLACE NAME}’s customer
service Help Line
n. What kind of information was NOT easy to
understand when you called the {MARKETPLACE
NAME}’s Help Line
o. Were any of the following a reason why you did
NOT get the information or help you needed when
you met in person with anyone about getting
health insurance from the {INSERT MARKETPLACE
NAME}?
p. What kind of information was NOT easy to
understand when you met in person with anyone
about getting health insurance from the
{MARKETPLACE NAME}?
q. Recommend marketplace to friends and family
Qualified Health Plan Survey composites and
individual items
a. Access to Care: Getting Needed Care
b. Access to Care: Getting Care Quickly
c. Health Promotion
d. Shared Decision Making
e. Specialized Services
f. How Well Doctors Communicate
g. Care Coordination
h. Access to Information
i. Plan Administration
j. Cost
k. Access to Care
l. Preventive Services
m. Rating of all health care
n. Rating of personal doctor
o. Rating of specialist
p. Rating of health plan
q. Would you recommend this health plan to your
friends and family?
The feedback you are providing is very helpful so far. I’d like to get your thoughts on some of
the language being used.
16. Take a look at the topic that says “{Composite/Measure Title}” What do you think it means
for a [health plan/Marketplace] to “{Composite/Measure Title}”?
7
o What types of things would a [health plan/Marketplace] do to “{Composite/Measure
Title}”?
o Take a look at Mid-Atlantic Health Plan. What does it mean that Mid-Atlantic Health
Plan is rated as average at “helping members prevent and manage health issues”?
o What do you think a health plan would have to do to get a “better” rating at this
topic? What do they do better or differently than other plans?
17. How important is this topic?
18. Is there anything that you think other people would find confusing or have a hard time
understanding? How would you rewrite the information here to make it clearer?
19. Where does the information for this page come from?
20. What do you think of the title for this column “{Composite/Measure Title}”?
21. (As needed/available) We are considering some other titles and would like to get your
opinion on them.{Hand participant hard copy of the page and other title options.} Which title
do you prefer? Tell me about the reasons you prefer that one.
Option 2
Option 3
Option 4
8
Sponsorship and Getting Answers
22. Who do you think is sponsoring this report of information?
a. How did you figure that out?
b. Do you feel like the information on this page can be trusted? Tell me more about
that.
23. What questions or concerns do you have about this information?
24. How would you find out more information or get help if you saw this information on a Web
page?
25. How would you prefer to get help? (website, phone, in-person). Please explain.
Closing
Before we end, I’d like to give you a chance to share any additional thoughts or comments about
the items we talked about today. Was there anything that you didn’t have a chance to say during
our discussion today or something that we didn’t talk about that you wish we had?
Thank you very much for participating in this discussion today. We appreciate your time.
9
| File Type | application/pdf |
| Author | Borsky, Amanda |
| File Modified | 2014-08-18 |
| File Created | 2014-08-18 |