Form CMS-10437.GenIC#11 CMS-10437.GenIC#11.WebToolsResearch_Instrument_FINAL

CMS Consumer Research on Websites and Tools

Interview Guide

Website Navigation

The actions that participants will be asked to complete will depend on the website being tested. The example below is based on asking participants to find out how to sign up to receive an electronic version of the Medicare & You Handbook.

Introduction

Our discussion today is part of a project sponsored by the Department of Health and Human Services (HHS). We are helping them update a part of their website that will allow people with Medicare to sign up to receive an electronic version of the Medicare & You Handbook.

Today, we’ll start at the beginning and see how far you get. I’m interested in learning how easy or hard you think the website is to use.

Before you get started, I’d like to go over just a few more items with you:

• First, please know that you are using your own information to log in to MyMedicare.gov. You will not be required to sign up to receive an electronic version of the Handbook to complete this study.

• So, please type in your own name, birthdate, contact information, etc., to register, or your log in information if you have already registered on MyMedicare.gov.

• The information you type in the application will not be shared with anyone outside of the project team.

• Second, as you are going through this process, it would be helpful if you would speak any thoughts that come into your mind out loud. For example, as you are going through the application, you might say, “I have no idea what that means” or “I would have expected to click here but there’s no button” or “I’m not sure what this question is asking.” Be sure to say out loud what you are thinking, why you are clicking on a particular item, and what questions you have.

• It would be great if you speak non-stop throughout this whole exercise – the more you say the more we learn. Since we’ll be able to hear you, we’ll take notes on what you say, and then we can discuss your comments when we review each page.

• Third, in addition to hearing the questions you have as you complete the process, I want to know if you come to a place on the website where you can’t move forward without assistance. If that happens, please say “I’m stuck,” and I’ll help you move forward.

• If you have a question but are still able to proceed, just say your question out loud and continue with the process. We’ll write your question down, and we can talk about it when you’re done.

• Lastly, let me know if at any time you reach a point where you would quit if you were completing this process independent of this interview.

Those are all of the instructions I have for you. Do you have any questions at this point?

Please open up an internet browser and go to www.MyMedicare.gov.

Navigation

US1. Using the MyMedicare.gov website, please try to find out how to sign up to receive an electronic version of the Medicare & You Handbook.

US1A. Tell me what you are doing as you go through the process.

On each page, the participants will be asked the following questions, as applicable.

US1B. How easy/difficult was it to find?

US2B. Do you feel that you were successful?

US2C. What was the easiest/most difficult part of this task?

US2D. What could have made that task easier?

PR7. When you first saw this webpage what caught your attention and why?

PR7. Did you read the page in detail?

K1D. What is the most important information on this page?

K1C. What, if anything, is confusing or unclear?

US1B. How easy or hard was it to figure out the next steps? [IF HARD] How would you change the page to make it clearer?

SOI4. When you were on this webpage, I noticed that you clicked on [INSERT PARTICIPANT’S CHOICE].

PR1B. How did you decide on what to click on?

SOI4. What did you expect to see when you clicked on [INSERT PARTICIPANT’S CHOICE]?

PR1B. Was there anything else that you would have been interested in clicking? Why or why not?

PR2. What do you like most/least about this webpage?

PE2A. What makes you feel that way about it?

Discussion

PR4. What is your impression of the process of signing up for electronic delivery of the Handbook on MyMedicare.gov?

PR4A. What gives you that impression?

PR5A. How could this be changed to improve your impression?

US2. How likely would you be to use MyMedicare.gov to sign up for electronic delivery of the Handbook?

US2E. How likely is it that you would use MyMedicare.gov if you had to do it just like you did now?

US2F. How (if at all) have you ever tried to sign up for electronic delivery of the Handbook in the past?

US2G. Was doing this task harder or easier to do with this website than how you did it before?

US2H. What made this task easier/harder today?

Web-Based Tool Navigation

The questions for which participants will be asked to find answers will be tailored to the web-based tool tested. The example below is based on asking participants to use the application tool on HealthCare.gov.

Introduction

Our discussion today is part of a project sponsored by the Department of Health and Human Services (HHS). We are helping them update a part of their website that will allow people to apply for and get health insurance online.

Now let me tell you a little more about the Web site we’ll be reviewing today; it is divided into two main parts:

• In the first part, people complete an application that will determine whether they can buy health insurance through the new program. It also determines how much help people can get paying for their health insurance.

• The second part allows people to see the health insurance plans that are available in their area, conduct a side-by-side comparison of a select number of plans, and enroll in a plan of their choosing.

Today, we’ll start at the beginning and see how far you get. I’m interested in learning how easy or hard you think the website is to use.

• Who do you plan on getting health insurance for today? What makes you say that?

Before you get started, I’d like to go over just a few more items with you:

• First, please know that you are using your own information in the application, and you will submit your application as a part of this study. You will not be required to purchase a plan as a part of this study.

• So, please type in your own name, birthdate, contact information, etc., for yourself and anyone else you include on the application. We will stop you before you enter Social Security Numbers for yourself or your family so that you can turn your screen sharing function off to ensure the confidentiality of your private information.

• The information you type in the application will not be shared with anyone outside of the project team.

• Second, as you are filling out this application, it would be helpful if you would speak any thoughts that come into your mind out loud. For example, as you are going through the application, you might say, “I have no idea what that means” or “I would have expected to click here but there’s no button” or “I’m not sure what this question is asking.” Be sure to say out loud what you are thinking, why you are clicking on a particular item, and what questions you have.

• It would be great if you speak non-stop throughout this whole exercise – the more you say the more we learn. Since we’ll be able to hear you, we’ll take notes on what you say, and then we can discuss your comments when we review each page.

• Third, in addition to hearing the questions you have as you complete the application, I want to know if you come to a place in the application where you can’t move forward without assistance. If that happens, please say “I’m stuck,” and I’ll help you move forward.

• If you have a question but are still able to proceed, just say your question out loud and continue with the application. We’ll write your question down, and we can talk about it when you’re done.

• Lastly, let me know if at any time you reach a point where you would quit if you were filling out the application independent of this interview.

Those are all of the instructions I have for you. Do you have any questions at this point?

Please open up an internet browser and go to www.HealthCare.gov.

Navigation

HealthCare.gov Landing Page

New enrollee page view Re-enrollee page view

Before you get started (Screener Questions)

Start Your Application (cover page)

On each page, the participants will be asked the following questions, as applicable.

US1. Using this website, please try to figure out how to look for a health insurance plan.

US1A. Tell me what you are doing as you go through the process.

US1B. How easy/difficult was it to find?

US2B. Do you feel that you were successful?

US2C. What was the easiest/most difficult part of this task?

US2D. What could have made that task easier?

PR7. When you first saw this webpage what caught your attention and why?

PR7. Did you read the page in detail?

K1D. What is the most important information on this page?

K1C. What, if anything, is confusing or unclear?

US1B. How easy or hard was it to figure out the next steps? [IF HARD] How would you change the page to make it clearer?

SOI4. When you were on this webpage, I noticed that you clicked on [INSERT PARTICIPANT’S CHOICE].

PR1B. How did you decide on what to click on?

SOI4. What did you expect to see when you clicked on [INSERT PARTICIPANT’S CHOICE]?

PR1B. Was there anything else that you would have been interested in clicking? Why or why not?

PR2. What do you like most/least about this webpage?

PE2A. What makes you feel that way about it?

Discussion

PR4. What is your impression of this website?

PR4A. What gives you that impression?

PR5A. How could this be changed to improve your impression?

K3. What do you think is the purpose/main idea of HealthCare.gov?

K3A. How well do you think the purpose/main idea comes across?

K3B. What could make it clearer?

K3C. What do you think this website wants to motivate people to do?

K3D. How easy or difficult was the process of finding a health insurance plan on HealthCare.gov to understand?

US2. How likely would you be to use this website to find a health insurance plan?

US2E. How likely is it that you would use this website if you had to do it just like you did now?

US2F. How (if at all) have you ever looked for health insurance plans in the past?

US2G. Was doing this task harder or easier to do with this website than how you did it before?

US2H. What made this task easier/harder today?

Closing

Before we end, I’d like to give you chance to share any additional thoughts or comments about the Web site we talked about today. Is there anything else you would like to say that you didn’t have a chance to say during our discussion today? Is there something that we didn’t talk about that you wish we had?

Thank you very much for participating in this discussion today. We appreciate your time.