CMS-10398 #49 Testing Protocol

Participant #:

Date:
Site:

First name:

New or past Medicaid applicant:

Last grade completed:

Age:

CMS Model Online Eligibility Results Page

Consumer Testing Interview Protocol

OMB control number: 0938-1148

CMS ID number: CMS-10398 #49

July 2016

Research Aims

• To determine if consumers can read and understand key messages and instructions in the model results page.

• To determine if consumers would take action based on key messages and instructions in the model results page.

• To assess consumers’ ability to navigate the model results page.

• To identify challenges or barriers that consumers face while using the model results page.

• To understand consumers’ overall impressions of the model results page.

Materials Tested

• A model online eligibility results page that appears at the end of an online application and conveys key information to consumers

• Key messages within the model online eligibility results page that describe how consumers who are ineligible for Medicaid can expect to interact with the Federally-Facilitated Marketplace (FFM)

Interview Questions and Purpose

Question	Purpose
Scenario 5
Let’s start with this page. We’re going to pretend you have come to this website because you just finished applying for Medicaid. When you’re ready, I’d like you to tell me: What is your first impression? [Probe if necessary:] Does it look easy or hard to read and understand? Why?	“Ice breaker” to establish that the researcher is interested in the user’s thoughts and opinion and the questions won’t be too difficult. Understand users’ first impressions.
What do you think is the main thing this page will tell you?	Do users understand the purpose of the page?
What do they mean by “eligibility”? [Point to “Eligibility Results.”]	Do users understand the term “eligibility”?
Thank you. And what is this number? [Point to tracking number.]	Can users identify the application tracking number?
Why are they asking you to keep it? [Probe if necessary:] What can you do with it?	Do users understand that they can use the application tracking number to return to their application later?
Would you be likely to print or save your application? Why or why not?	Would users take advantage of the opportunity to download and save or print their application for reference?
Now let’s take a look at the next steps. [Point to “Take these next steps.”] What do they want you to do first?	Do users understand the instruction to review eligibility results first?
And how would you do that?	Do users know where to find their eligibility results?
What is this little bar with the circles and numbers here for? [Point to tracking bar.]	Do users understand that they will use this bar to track their progress going through the steps?
Don’t do anything yet, but tell me, what would you do next? [Don’t let user click “Go to Step 2” yet.]	Do users understand the next action they need to take?
Thank you. Before we do that, let’s first take a closer look at Step 1. I’d like you to read it and when you are ready, tell me, what does Anna qualify for?	Do users understand that Anna is eligible for Medicaid?
Starting when?	Do users see and understand the start date information?
And what is this number? [Point to Medicaid ID number.] [Probe if necessary:] What is it for?	Can users identify the Medicaid ID number and its purpose?
Thank you. And what else are they telling Anna about her Medicaid? [Probe if necessary:] Is her Medicaid eligibility set? What happens if she doesn’t send the information they ask for?	Do users understand that eligibility is temporary pending further verification?
What information do they need?	Can users identify the verification needed?
OK. And now that you’ve read everything on this page, what would you do next? [Pause for answer.] Let’s go to Step 2.	Do users understand the next action they need to take?
Which do you like to do better, clicking to a new page or scrolling down on the same page to see the next step? Why?	Do users prefer clicking or scrolling?
I’d like you to read Step 2 next and, when you are ready, tell me in your own words, what does it say?	Do users understand that they must send proof of citizenship to keep coverage and download their eligibility letter to go to Step 3?
What would you do next? [If participant tries to go to Step 3 without downloading the eligibility letter and gets the error message, mark incorrect and probe:] And now what would you do?	Do users follow the instruction to download the eligibility letter?
And why do they want you to download the eligibility letter?	Do users understand they need to download the letter to read the information and go to Step 3?
How can you submit your proof here?	Do users understand they can submit verification online?
And if you didn’t want to send your proof online?	Do users see the eligibility letter will have more information about submitting verification by fax, mail or in person?
Now let’s go to Step 3. Take a minute to read this step, and when you’re ready, tell me in your own words what it’s telling you.	Do users understand that they can find a provider online, and that they can start using coverage right away but must send proof of citizenship to keep their coverage?
Scenario 8a
Thank you very much. Your answers are really helpful. Now I’m going to ask some questions about another page. Let’s go right to Step 1 on this one and take a look at what happened for this family. After you read Step 1, I’d like you to tell me, who qualifies for what here? [Probe if necessary:] Do they all qualify for the same thing? What do Anna, Peter and Tom qualify for? What does Susan qualify for?	Do users understand that three people qualify for Medicaid and one does not qualify for Medicaid?
I’d like you to read this part about Susan [point to “You still might be able to get health coverage…”] and tell me, what is happening next for her? [Probe if necessary:] What will they check for?	Do users understand that Susan’s application has been sent to the Marketplace for review of eligibility for assistance?
And what is the Marketplace?	Do users know what the Marketplace is?
Let’s look at Step 2 next. What would you do at this step?	Do users understand that they must download their eligibility letter next in order to read it?
We have only a few more questions to go. Let’s look at Step 3 next. What do Anna, Peter and Tom need to do? [Probe if necessary:] Is there a deadline?	Do users understand the three people who qualify for Medicaid must choose a health plan by the deadline?
And how do they do that?	Do users understand they can use the button to review and choose plan options?
Does Susan do that too? [Probe:] Why or why not?	Do users understand that this person does not choose a Medicaid plan because she did not qualify for Medicaid?
So how does Susan finish enrolling? [Probe if necessary to get both:] Is that the only way she can apply?	Are the choices clear to users? Do they understand that Susan can let the Marketplace use the information they received to begin an application or she can use the link to start a new application on the Marketplace?
If you were Susan, would you rather start a new application or have the Marketplace start one for you? Why?	Which method for applying through the Marketplace do users prefer?
Would either of these options help you get coverage sooner?	Do users understand that waiting for the Marketplace to start an application could result in a delay in coverage?
And if starting a new application meant you would have to re-enter your information, but may get coverage sooner, which option would you choose? Why?	Which method for applying through the Marketplace do users prefer, knowing that starting a new application means having to re-enter information but possibly getting coverage sooner?
Now take a minute to read this last paragraph. When you’re ready, I’d like for you to tell me in your own words what it says. [Probe if necessary:] What is the difference between an Open Enrollment Period and Special Enrollment Period?	Do users understand that the Marketplace will tell them whether they can get coverage now or have to wait? Do users understand the terms Open Enrollment Period and Special Enrollment Period?
And finally, is there anything else you would like to say about these pages?	Understand users’ final impressions.

Interview Script: Introduction

Thank you for participating in today’s testing. My name is . Today we’re going to review some materials together. We need your help to find out if they are easy or hard to read and use so we can make them the best they can be for people who will use them. To find out, I’ll show you the materials and ask you to read parts of them and answer questions.

Don’t worry about how you read or whether you read fast or slow. And don’t worry about answering my questions correctly. This is a test of the materials—not a test of how well you read or whether you answer the questions correctly. Your answers—whatever they are—will help us.

It will really help if you think out loud as we go and give me your honest opinions. I’ll be taking notes so I can remember what you say. I won’t use your full name in my notes; just your first name. And we won’t use your name at all in our report of what we find out today.

We’ll spend about an hour together. If you have any questions as we go along, just ask them. I may not be able to answer them right away, since we’re interested in whether people understand the materials without help, but if you still have questions after we are finished, I’ll try to answer. And if you need to take a break, just let me know.

Do you have any questions before we start? [Pause for questions.] OK, let’s begin!

Interview Script: Questions

Scenario 5

1. Let’s start with this page. We’re going to pretend you have come to this website because you just finished applying for Medicaid. When you’re ready, I’d like you to tell me: What is your first impression? [Probe if necessary:] Does it look easy or hard to read and understand? Why?

• Easy

• Hard

Comments [Why?]:

2. What do you think is the main thing this page will tell you?

• Correct [Eligibility results]

• Participant does not know / thinks something else

Comments:

3. What do they mean by “eligibility”? [Point to “Eligibility Results.”]

• Correct [Whether you do or do not qualify for Medicaid]

• Participant does not know / thinks something else

Comments:

4. Thank you. And what is this number? [Point to tracking number.]

• Correct [Application tracking number]

• Participant does not know / thinks something else

Comments:

5. Why are they asking you to keep it? [Probe if necessary:] What can you do with it?

• Correct [Keep it so you can return to your application later]

• Participant does not know / thinks something else

Comments:

6. Would you be likely to print or save your application? Why or why not?

• Yes

• No

Comments [Why or why not?]:

7. Now let’s take a look at the next steps. [Point to “Take these next steps.”] What do they want you to do first?

• Correct [Review eligibility results]

• Participant does not know / thinks something else

Comments:

8. And how would you do that?

• Correct [Look below at Step 1]

• Participant does not know / thinks something else

Comments:

9. What is this little bar with the circles and numbers here for? [Point to tracking bar.]

• Correct [It tells you which step you are on]

• Participant does not know / thinks something else

Comments:

10. Don’t do anything yet, but tell me, what would you do next? [Don’t let user click “Go to Step 2” yet.]

• Correct [Go to Step 2]

• Participant does not know / thinks something else

Comments:

11. Thank you. Before we do that, let’s first take a closer look at Step 1. I’d like you to read it and when you are ready, tell me, what does Anna qualify for?

• Correct [Medicaid]

• Participant does not know / thinks something else

Comments:

12. Starting when?

• Correct [January 1, 2016]

• Participant does not know / thinks something else

Comments:

13. And what is this number? [Point to Medicaid ID number.] [Probe if necessary:] What is it for?

• Correct [Medicaid ID number to use when getting services]

• Participant does not know / thinks something else

Comments:

14. Thank you. And what else are they telling Anna about her Medicaid? [Probe if necessary:] Is her Medicaid eligibility set? What happens if she doesn’t send the information they ask for?

• Correct [She must send more information in order to keep her coverage]

• Participant does not know / thinks something else

Comments:

15. What information do they need?

• Correct [Proof of citizenship]

• Participant does not know / thinks something else

Comments:

16. OK. And now that you’ve read everything on this page, what would you do next? [Pause for answer.] Let’s go to Step 2.

• Correct [Go to Step 2]

• Participant does not know / thinks something else

Comments:

17. Which do you like to do better, clicking to a new page or scrolling down on the same page to see the next step? Why?

• Clicking

• Scrolling

Comments [Why?]:

18. I’d like you to read Step 2 next and, when you are ready, tell me in your own words, what does it say?

• Correct [You need to send proof of citizenship by April 19, 2016 and read your eligibility letter]

• Partial [Note in comments part participant omits or does not understand]

• Participant does not know / thinks something else

Comments [Note part participant omits or does not understand]:

19. What would you do next? [If participant tries to go to Step 3 without downloading the eligibility letter and gets the error message, mark incorrect and probe:] And now what would you do?

• Correct [Download the eligibility letter]

• Participant does not know / thinks something else

Comments:

20. And why do they want you to download the eligibility letter?

• Correct [The letter tells you what proof to send and how to send it. It also gives more information about your determination. You must download it in order to get to Step 3.]

• Partial [Note in comments part participant omits or does not understand]

• Participant does not know / thinks something else

Comments [Note part participant omits or does not understand]:

21. How can you submit your proof here?

• Correct [Use link to send proof online]

• Participant does not know / thinks something else

Comments:

22. And if you didn’t want to send your proof online?

• Correct [The eligibility letter will tell you how to fax, mail or submit in person]

• Participant does not know / thinks something else

Comments:

23. Now let’s go to Step 3. Take a minute to read this step, and when you’re ready, tell me in your own words what it’s telling you.

• Correct [You can start using coverage right away, but it will end on April 19, 2016 if you don’t send proof of citizenship. You can find a provider online.]

• Partial [Note in comments part participant omits or does not understand]

• Participant does not know / thinks something else

Comments [Note part participant omits or does not understand]:

Scenario 8a

24. Thank you very much. Your answers are really helpful. Now I’m going to ask some questions about another page. Let’s go right to Step 1 on this one and take a look at what happened for this family. After you read Step 1, I’d like you to tell me, who qualifies for what here? [Probe if necessary:] Do they all qualify for the same thing? What do Anna, Peter and Tom qualify for? What does Susan qualify for?

• Correct [Anna, Peter and Tom qualify for Medicaid and Susan does not]

• Partial [Note in comments part participant omits or does not understand]

• Participant does not know / thinks something else

Comments [Note part participant omits or does not understand]:

25. I’d like you to read this part about Susan [point to “You still might be able to get health coverage…”] and tell me, what is happening next for her? [Probe if necessary:] What will they check for?

• Correct [Her application goes to the Marketplace to see if she qualifies for assistance paying for coverage]

• Participant does not know / thinks something else

Comments:

26. And what is the Marketplace?

• Correct [The online site where you apply for health insurance coverage]

• Participant does not know / thinks something else

Comments:

27. Let’s look at Step 2 next. What would you do at this step?

• Correct [Download eligibility letter and read it]

• Participant does not know / thinks something else

Comments:

28. We have only a few more questions to go. Let’s look at Step 3 next. What do Anna, Peter and Tom need to do? [Probe if necessary:] Is there a deadline?

• Correct [Choose a Medicaid plan by the deadline (February 15, 2016)]

• Participant does not know / thinks something else

Comments:

29. And how do they do that?

• Correct [Use the “Choose a Medicaid plan” button]

• Participant does not know / thinks something else

Comments:

30. Does Susan do that too? [Probe:] Why or why not?

• Correct [No, because she does not qualify for Medicaid]

• Participant does not know / thinks something else

Comments:

31. So how does Susan finish enrolling? [Probe if necessary to get both:] Is that the only way she can apply?

• Correct [She can let the Marketplace start an application using the information submitted or she can begin a new application]

• Partial [Note in comments part participant omits or does not understand]

• Participant does not know / thinks something else

Comments [Note part participant omits or does not understand]:

32. If you were Susan, would you rather start a new application or have the Marketplace start one for you? Why?

• Prefer to start new application

• Prefer to have Marketplace start application

Comments [Why?]:

33. Would either of these options help you get coverage sooner?

• Correct [Yes, starting a new application yourself may help you get coverage sooner than if you wait for the Marketplace to start an application for you]

• Participant does not know / thinks something else

Comments:

34. And if starting a new application meant you would have to re-enter your information, but may get coverage sooner, which option would you choose? Why?

• Start new application

• Have Marketplace start application

Comments [Why?]:

35. Now take a minute to read this last paragraph. When you’re ready, I’d like for you to tell me in your own words what it says. [Probe if necessary:] What is the difference between an Open Enrollment Period and a Special Enrollment Period?

• Correct [After you finish your application, the Marketplace will tell you whether you have to get coverage now or have to wait. Most people can only enroll during an OEP, but some people can enroll during a SEP if they have a qualifying life event.]

• Partial [Note in comments part participant omits or does not understand]

• Participant does not know / thinks something else

Comments [Note part participant omits or does not understand]:

36. And finally, is there anything else you would like to say about these pages?

Comments:

Interview Script: Closing

Thank you very much for your time. You have been very helpful!

June 10, 2016 CMS Model Online Eligibility Results Page Consumer Testing Interview Protocol 19

PRA Disclosure Statement: According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number.  The valid OMB control number for this information collection is 0938-1148. The time required to complete this information collection is estimated to average 1.0 hour per response, including the time to review instructions, search existing data resources, gather the data needed, and complete and review the information collection.  If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850