Usability Testing for the Health Insurance Marketplace

Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number: 0938-1185)

T ITLE OF INFORMATION COLLECTION:

Usability Testing for the Health Insurance Marketplace Websites: Health Insurance Marketplace Consumer Experience Surveys Project

PURPOSE:

Sections 1313 and 1321(a) of the Affordable Care Act provide the Secretary with general authority to establish standards and regulations related to Exchanges, Qualified Health Plans (QHPs), and other components of title I of the Affordable Care Act. In §155.1200(b)(3), State-based Marketplaces are directed to submit performance monitoring data including information on consumer satisfaction on an annual basis.

In response to this legislation, the Centers for Medicare & Medicaid Services is developing and conducting a survey for Marketplace users. OMB approval for the psychometric test component of the Marketplace survey was obtained on February 12, 2014 under OMB Control Number 0938-1221.

This fast track request to conduct usability testing with consumers for the State-based Marketplace (SBM) websites is part of the Health Insurance Marketplace Consumer Experience Surveys project. The website usability testing will provide qualitative feedback to inform the interpretation of the Marketplace Survey results, help the Marketplaces improve their survey scores in future years, and ultimately help the Marketplaces improve the consumer’s experience. This project will have additional activities that use similar qualitative cognitive testing methods; additional fast track clearance applications will be submitted in the future.

Usability testing uses the qualitative data collection methods of observation and cognitive interviewing to assess how a respondent understands or interprets information. The proposed usability testing will be used to gather information from consumers about whether the SBM websites provide information about 1) what consumers want to know; 2) if consumers are able to readily find the information they want (navigation); and 3) if consumers interpret the language used and information as intended (comprehension).^1,2 Consumers frequently interpret information very differently than experts expect and in ways that cannot be anticipated. This is the fundamental premise of all cognitive testing, including testing of survey items and educational materials.

Usability testing will be done during the open enrollment period for the 2015 coverage year and will be used to help improve consumer’s experiences with the Marketplace websites. During this testing, we will watch consumers using the website. We will ask each participant to perform a series of activities on the SBM website and then ask questions as he/she is performing the activities. For example, we will present the participant with the SBM web page to compare and choose plans. Interviewers would then ask questions such as: what do you notice on the page? (See Attachment A for the usability testing guide.)

The information collected through the usability testing is qualitative in nature and will be analyzed using qualitative methods to identify key themes; this data collection does not involve any statistical analyses.

CMS plans to share the qualitative results with each SBM for its own performance improvement efforts. SBMs will only receive results that are specific to its state (e.g., Massachusetts will receive the key themes from Massachusetts, but not the key themes for Connecticut). CMS will also share a de-identified global summary report that summarizes key themes from all SBMs, but does not call out or identify any specific SBM. For example, the global summary report would say something like “consumers in several SBMs experienced the following problem during the usability assessments;” but, it would not identify the names of the SBMs where consumers had these problems during the assessments.

DESCRIPTION OF RESPONDENTS:

Selection of the target population of respondents for the usability testing has been informed by two sources: (1) the CMS Multidimensional Insurance Data Analytics System (MIDAS) data on online applicants and enrollees from the 36 states that used healthcare.gov, and (2) key audience segments for the Marketplaces based on a national survey to inform Enroll America’s campaign (Undem and Perry, 2013³).

The respondents for the usability testing must be eligible for health insurance through the Marketplace.⁴ However, the testing will not be done with people who have applied or enrolled in health insurance through the SBM’s website being tested during the current 2015 coverage year, since their previous experiences with the website would bias their perceptions of it. It is important that we control their exposure to the website so that we can standardize the external environment, which might otherwise confound our findings. Consumers could have used the website during the prior 2014 coverage year since we know many of the websites have been updated or changed during the past year; thus, minimizing potential bias. Our complete eligibility criteria will include the following (see Attachment B for our respondent recruitment screener):

1. Must be eligible for health insurance through the Marketplace⁵

2. Must have used the internet in the past and have access to the internet either at home or through a public venue such as the library

3. Must not have applied or enrolled via the SBM Website being tested during the fall 2014/winter 2015 open enrollment period for 2015 coverage

4. Must be responsible for selecting health insurance for themselves and family, if applicable

5. Must be comfortable reviewing a website in English and answering questions in English (all testing will be conducted in English)

6. Must be between the ages of 18-64⁶

7. Mix of geographic representation (e.g., people living in the city and neighboring towns)

8. Mix of genders

9. Mix of race/ethnicity (White/non-White)

10. Mix of education (high school degree and some college)

11. Mix of family size (single and families)

12. Mix of eligibility for subsidies (eligible for public and private premium subsidies) and no subsidy⁷ (see criteria in Figure 1 for number of people in household and income)

13. Mix of enrolled and not enrolled for the 2014 coverage year

Screening criteria may be revised as needed based on recruitment experiences to ensure respondents are comparable across sites. After meeting the initial eligibility criteria, we will recruit individuals who fit two target populations that reflect the major groups of previously uninsured and underinsured persons who are likely to use the Marketplaces. The target populations will include:

1. Healthy (no chronic conditions)

2. Individuals with at least one chronic condition

We will conduct a minimum of 6 interviews per SBM (3 interviews per target population per SBM) for a total maximum of 90 interviews (assuming 15 SBMs). To economize on time and travel costs, testing will take place in four locations: one location on the west coast, one in the northeast, one in the Midwest, and one in the mid-Atlantic/southern region. The website of each SBM will be tested at the location in its region. For instance, if we choose San Francisco as the west coast location, we will recruit members of each persona from the San Francisco area to test the California and Washington websites.

CMS will invite all SBMs operating their own website to participate by disseminating an issue brief (see Attachment C) and through informal discussions with the SBMs; SBMs will voluntarily indicate their interest.

SBMs will not be responsible for recruiting eligible participants. A recruiting firm will be used to identify and recruit eligible respondents according to a recruitment screener (see Attachment B).

Respondent contact information will only be obtained for purposes of recruitment.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey

[X] Usability Testing (e.g., website or Software [ ] Small Discussion Group

[ ] Focus Group [ ] Other: ______________________

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name: Nidhi Singh-Shah and Kathleen Jack

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [X] Yes [ ] No

2. If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [X] No

3. If Yes, has an up-to-date System of Records Notice (SORN) been published? [] Yes [ X ] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [X] Yes [ ] No

The Agency plans to provide stipends of $75 for each respondent who participates in the usability testing. Renumeration is needed to attract the full range of respondents needed for this project. Inadequate respondent recruitment will limit the results from the usability testing. The amount of the stipend is needed for several reasons:

• The study population is very specific and may be difficult to recruit. Respondents will be recruited for specific characteristics that are related to the subject matter (e.g., questions would only be relevant to those eligible to receive health insurance through the Marketplaces who have not applied or enrolled via the SBM Website being tested during the current coverage year). Since this is a very specific population and subject matter, it may be difficult to recruit eligible respondents and a higher remuneration helps to attract a greater number of potential respondents.

• The usability testing will require participants to travel to a centrally located site and the stipend is intended to help compensate participants for their travel costs (e.g., train, bus, mileage, and parking).

• Usability testing uses the qualitative methods of observation and cognitive interviewing to understand consumer’s experiences using the website. It also requires consumers to perform specific activities on the website while being observed. This requires an unusual level of mental effort, as respondents are asked to perform tasks that may be difficult and to explain their mental processes as they hear conduct each task and describe why they did what they did on each task.

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time (hours)	Burden (hours)
Individuals or Households	90	1.5	135
Totals	90	1.5	135

FEDERAL COST: The estimated annual cost to the Federal government is $350,451.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [ ] Yes [X] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them? See Attachment B – Script to recruit respondents.

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[ ] Web-based or other forms of Social Media

[ ] Telephone

[X] In-person

[ ] Mail

[ ] Other, Explain

2. Will interviewers or facilitators be used? [X] Yes [ ] No

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Attachment A – Usability Testing Guide

Attachment B – Script to recruit respondents

Attachment C – Script to recruit SBMs

Instructions for completing Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback”

TITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g., Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for this collection of information. These groups must have experience with the program.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions. Note: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period of time that is necessary to achieve a specific objective.

Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households;(2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of respondents.

Participation Time: Provide an estimate of the amount of time required for a respondent to participate (e.g. fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of responses and the participation time and divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents. Please provide a description of how you plan to identify your potential group of respondents and how you will select them. If the answer is yes, to the first question, you may provide the sampling plan in an attachment.

Administration of the Instrument: Identify how the information will be collected. More than one box may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for focus groups) used.

Submit all instruments, instructions, and scripts are submitted with the request.

1 Nielsen, J., & Loranger, H. (2006). Prioritizing Web usability. Berkeley, CA: Nielsen Norman Group.

2 Robert Wood Johnson Foundation and American Institutes for Research. How to get consumer feedback and input into websites. Retrieved from http://forces4quality.org/node/2550

3 https://s3.amazonaws.com/assets.enrollamerica.org/wp-content/uploads/2013/11/Informing-Enroll-America-Campaign.pdf

4 To be eligible for health coverage through the Marketplace, respondents: must live in the United States, must be a U.S. citizen or national (or be lawfully present). Learn about eligible immigration statuses, and can't be currently incarcerated.

5 To be eligible for health coverage through the Marketplace, respondents: must live in the United States, must be a U.S. citizen or national (or be lawfully present). Learn about eligible immigration statuses, and can't be currently incarcerated.

6 According to the MIDAS data on online applicants and enrollees from the 36 states that used healthcare.gov, the median age for enrollees is 45. One person per state that is under 26 will be accepted and one per state that is over 50 will be accepted.

7 One person per state eligible to purchase through the Marketplace, but not eligible for subsidies will be accepted.

6