CMS-10710.GenIC#12 - DACA - Instrument

Status: Original and derived artifacts are available for this document.

Download: docx | pdf | html

Usability and Cognitive Testing: HC.gov Application Enhancements for DACA Recipients 

Dates: TBD upon OMB Clearance, Usability and Cognitive Testing Guide
Protocol (60 Minutes)
Agenda
    • Welcome and Overview (4 minutes)
    • Participant Background (5 minutes)
    • Independent Navigation (20 minutes) [“think aloud”]
    • Section-by-Section Debriefing (20 minutes)
    • Debrief on Overall Experience (8 minutes)
    • Wrap-up and Closing (3 minutes)
Welcome and Overview Introduction (4 Minutes)

Thank you for joining me today. My name is [Moderator], and I work for a company called [company] in the Washington DC area. Our discussion today is part of a project sponsored by the Department of Health and Human Services (HHS), the agency that runs the Federal Health Insurance Marketplace. We are helping them to gather information to improve the section of the website for the Health Insurance Marketplace for people who need to provide information on their immigration status when they apply for coverage. Before they share this information on their website, they want to be sure that it makes sense and is easy to use.  We will use what you tell us today to improve the materials.
In a few moments, I am going to share a prototype of this portion of the website with you and get your reactions and thoughts about the materials.  Before we start our discussion, there are a few things I’d like you to keep in mind. 
    1. Your participation today is completely voluntary. You can decide not to answer any question that we ask, and you can stop your participation at any time. Your participation or decision to stop participating will have no impact on your Marketplace benefits.
    2. The information you share will be combined with information from other people, your full name will not be connected with anything you say, and your privacy will be protected.
    3. I did not create the materials we are testing, so you won’t hurt my feelings with any feedback. Also, there are no right or wrong answers. Please be open and honest with your reactions and your answers to my questions. 
Informed Consent: Before joining this session, you reviewed the consent form. Do you have any questions? Do we have your consent to participate? 
Some housekeeping items:
    1. We will be here for 75 minutes today. If you need to step away at any point, just let me know.
    2. Remember that we are not testing you – even though it might seem like it at times, because I’ll be asking a lot of questions.  We are testing the materials. The more you tell us what you are thinking, the more we learn.  If something is unclear or confusing to you, it’s bound to be confusing to other people too. 
    3. Our discussion will be recorded so that we can make sure that we don’t miss anything that you tell us. The recording is only used for research purposes and will not be shared with anyone outside of our team. In addition, some people working on this project are observing this session live on a secured site so they can hear what you have to say.  
[Begin recording]
Participant Background (5 Minutes)

Before we start our discussion, I’d like to learn more about your health coverage and a couple of other things.  
    1. Do you currently have health insurance? 
        a. If YES – Please tell me a little bit about your coverage. Probe for details if necessary – source; duration of coverage; how did you sign up; details regarding anyone who may have helped. 
        b. If NO – Tell me about the last time you had insurance. Probe for details if necessary – source, how long ago was that; what made you end or stop your coverage. Review of the Notice 
    2. How do you feel about being [insured/uninsured]? Probe for elaboration if necessary – Are you able to get the health care that you need; medications, relation to current health status; workarounds. 
    3. Can you describe what, if anything you have heard about the Health Insurance Marketplace or the HealthCare.gov web site?  [If needed – Also known as the Affordable Care Act or ACA insurance – used to be called Obamacare].  
    4. How important is it to you to have health insurance?  Rate 1= very unimportant to 5= very important. Can you please elaborate on that?

Thank you; your answers have given me a better understanding of your situation. 
Independent Navigation (20 Minutes)

Today we’ll be reviewing a prototype of a part of the HealthCare.gov website that the Department is in the process of improving that allows people to provide information about their immigration status to see if they may be able to sign up for health insurance through the Marketplace.  This is part of the process for applying for coverage through the Health Insurance Marketplace.  
What do you think the application portion of this website will be like? Tell me more about that.
Before we get started, I’d like to go over just a few more items with you:
You will be reviewing a draft version of the website – a prototype - which is not completely functional. You will not be able to enter your own information. In fact, when you click on certain boxes, information will automatically appear. And sometimes a button or link may not work. 
So, as you go through the website, I’d like you to tell me how you would answer each question before you actually click on anything. It will also be helpful if when you’re trying to click on a button or link that is not working, if you let me know what you are trying to do, why you are interested in clicking on it, and what you expect to see next.
As you review and use the draft website, I want you to “think aloud.” Let me know what you are thinking, what you are clicking on and why, any questions you have, anything that is confusing or unclear, and anything you really like. It would be great if you talked non-stop throughout this whole exercise – The more that you talk, the more we will learn from you... I’ll also take notes on what you say, and then we can discuss your comments when we review it together.
Even though I’m asking you to tell me what you would do before actually clicking on anything and to “think aloud” as you are using the prototype, I want you to use the draft website like you would if you were using it outside of this session. That is, if you’re someone who usually skims the text on the page or skips it altogether, please don’t feel like you need to read it in-depth just because I’m watching you. On the other hand, if you’re someone who reads every word, please don’t feel like you need to rush through anything.
Do you have any questions before we move on to this next part?
[MODERATOR WILL SWITCH TO “SCREENSHARE” LAYOUT AND MAKE SURE THEY CAN SEE THE PARTICIPANT’S SCREEN BEFORE SENDING THE PROTOTYPE LINK TO THE PARTICIPANT] 
Now I’d like you to use this prototype as you would if you were interested in applying for coverage through the HealthCare.gov website, except, as I’ve mentioned, I want you to “think aloud.” When you reach the point where you would stop using this part of the website, please say “I’m done.”
At this point the participant will begin to independently navigate through the prototype, while thinking aloud.  This will be done with minimal guidance from the moderator. There are 9 distinct pages in this flow.  These include:
    • Your Marketplace Application
    • Section 1 – Application Set up
    • Tell us about yourself
    • Section 2 – Personal & household information
    • Citizenship & Immigration Status
    • [Applicant’s name] immigration status
    • Section 3 – Current coverage & life changes
    • Life Changes
    • Tell us about the recent change to eligible immigration status
Key observations that will be of interest include feedback and responses on the following application pages during independent navigation --
    • Existing page, without any changes: Citizen & Immigration status question (DACA consumers would answer “no” to this page). Are they answering appropriately? Should this be incorporated into the research (estimate for DACA recipients in FFM states is about 150K).
    • Immigration status page: Changes—just in time content—information panel and information from help drawer has been pulled into main page. Do DACA consumers and non-DACA consumers understand this page and answer appropriately?
    • Reporting a life change screen—One of the options is “gained eligible immigration status.” Does informational context help DACA user select the correct life change? Is the content understandable?
    • Recent changes to eligible immigration status page: Do DACA recipients know which date to enter? i.e., 11/1/24 vs. date they gained their DACA documentation status (due to a policy change rather than some individual reason)
Screenshots illustrating the flow of these pages are submitted separately as part of this package.   
Section-by-Section Debriefing (20 Minutes)

Thank you for talking me through your thinking as you worked through this part of the application process.  Now, let’s retrace together what you did on each of the pages.  
Moderator will return to the first page and probe on any relevant observed reactions to explore deeper underlying cognition.  If links to help content were not opened during independent navigation the moderator will probe on why that occurred [e.g., did participant think they did not need that information vs. participant did not notice the link].  All participants will view the associated help text.  This will be discussed to assess salience, clarity, and understanding of the information provided. Probes will be introduced to clarify the nature of any ambiguity and to explore how that might be resolved. If participant seemed frustrated at any point in the process the moderator will probe on the emotional reaction that is or was observed, clarify the root cause of the frustration from the perspective of the consumer, and discuss how the situation might be resolved. 
Key issues that will be explored for each section include:
        ◦ What terms or phrases were confusing, unclear, or misunderstood? Please elaborate. 
        ◦ How easy was it to understand the language used to explain the application process and the new eligibility status that is being accorded to DACA recipients? Please describe in your own words what has changed for DACA recipients according to the material that you have just reviewed?
        ◦ Do you have any suggestions for ways that the application process could be improved? 
        ◦ Was there information that you thought was missing that would have facilitated the application process?  Please elaborate. 
This process will be repeated for each page of the prototype.  
Thank you for discussing this with me.  Next, I’d like to get your general reactions to the prototype as a whole. 
Debrief on Overall Experience (8 Minutes)

    1. Overall, what was your reaction to the prototype that we reviewed today? 
    2. How helpful was the prototype in achieving the goals that a person with DACA or another type of immigration status would have in applying for coverage?  Scale 1 = very unhelpful to 5 = very helpful. 
    3. If you were going through this application process independently of this interview, would you have stopped and abandoned the process at any point?  If YES – Please tell me more about that. Probe if necessary to determine where decision to abandon would have occurred and the reasons for that decision.  Aim to uncover inner cognition as opposed to superficial rationale.  What could be changed to encourage you to continue with the application process? 
    4. What improvements could be made to make the application process easier for someone to complete?   
    5. Are there any features that you think should be added to the prototype to make it easier for someone like you to complete the process? 
    6. What additional resources or types of communication would you find helpful to better understand how to use the website and navigate the application process?

Wrap Up and Closing (3 Minutes)

We are just about finished with our discussion. I’d like to check with my colleagues who are behind the one-way mirror to see if there are any additional questions they would like me to ask you.
[Questions from observers]
Those are all of the questions I have for you.
Do you have any final thoughts you would like to add?
Is there anything we didn’t talk about, but you wish we had?
Thank you very much for participating in this discussion today. We appreciate your time, and your insights have been very helpful!

PRA Disclosure Statement
According to the Paperwork Reduction Act of 1995, no persons are required to respond to a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0938-1382 GenIC #12. This information collection is being conducted to help improve informational notices for people with Medicare and the results will be used to ensure that people with Medicare understand important information about their plans. The time required to complete this information collection is estimated to average less than 60 minutes per response, including the time to review instructions, search existing data resources, gather the data needed, to review and complete the information collection. This information collection is voluntary, and CMS will keep the information private to the extent provided by law. If you have comments concerning the accuracy of the time estimate(s) or suggestions for improving this form, please write to: CMS, 7500 Security Boulevard, Attn: PRA Reports Clearance Officer, Mail Stop C4-26-05, Baltimore, Maryland 21244-1850.