CMS-10726 Descripton of Focus Groups and Cognitive Testing Protoco

Attachment C - QHP Enrollee Survey Focus Group and Cognitive Testing Protocols

PROTOCOLS FOR FOCUS GROUPS AND COGNITIVE TESTING INTERVIEWS

CONSUMER FOCUS GROUPS

Description of Participants:

CMS will recruit approximately six to eight consumer representatives for each focus group (12-16 individuals total); CMS will conduct one focus group with adults with low income and a second with higher-income adults and seek to recruit a diverse mix of participants based upon gender, race, and ethnicity. Additionally, CMS will seek to have variation in subsidy eligibility by stratifying focus groups based upon income. We will strive to achieve the most diverse mix possible while still adhering to our timeline for data collection and analysis.

CMS will screen potential focus group participants to meet one of the two following criteria:

1. They must have purchased a health insurance plan through the Exchange during the 2018 or 2019 open enrollment period. Because we are particularly interested in consumers who have recent experience with the process of selecting health plans in this new environment, preference will be given to those who purchased coverage during the 2019 (for plan year 2020) open enrollment period due to full public display of plan quality and member experience ratings. Or,

2. Consumers who are uninsured, but eligible for health insurance through the Exchange, as a way of including potential consumers who might purchase coverage on an Exchange in the future. 

Additionally, participants will be stratified by income level and screened into one of the three health status categories in the tables below.

Table 1: Focus Group #1 Participant Target Population - Adults with Low Income

Target Population	Characteristics
Adults with low income	All participants must have an income below 250% of the federal poverty level, but not enrolled in Medicaid (However, Medicaid recipients using an 1115 waiver to purchase coverage on an Exchange can be included.)
Two Participants from Each of the Following Target Populations
Young and healthy	a. Must be between the age of 18 and 34 b. Must not have been diagnosed (now or ever) with any chronic condition
Healthy and age 35+	a. Must be between the ages of 35 and 64 b. Must not have been diagnosed (now or ever) with any chronic health condition
Adults with chronic conditions or disabilities	Must be currently be diagnosed with a chronic condition, such as asthma, diabetes, or hypertension, mental health condition (e.g., chronic depression or anxiety) and/or Must be a consumer who is deaf, blind, or has other physical or cognitive impairment

Table 2: Focus Group #2 Participant Target Population - Adults Higher Income

Target Population	Characteristics
Adults with higher income	All participants must have an income above 250% of the federal poverty level.
Two Participants from Each of the Following Target Populations
Young and healthy	a. Must be between the age of 18 and 34 b. Must not have been diagnosed (now or ever) with any chronic condition
Healthy and age 35+	a. Must be between the ages of 35 and 64 b. Must not have been diagnosed (now or ever) with any chronic health condition
Adults with chronic conditions or disabilities	Must be currently be diagnosed with a chronic condition, such as asthma, diabetes, or hypertension, mental health condition (e.g., chronic depression or anxiety) and/or Must be a consumer who is deaf, blind, or has other physical or cognitive impairment

Description of Consumer Focus Groups Protocol:

CMS will conduct two in-person consumer focus groups, with approximately six participants each, stratified by income group to encourage increased participation.¹ Both consumer focus groups will be conducted in person. At least six participants must reside in or near Washington, DC, or Denver, Colorado. Depending upon recruitment success CMS may conduct one focus group in each city or both focus groups in the same city. In either city, CMS will use private conference room space at a contractor office near accessible public transportation to hold the sessions. Focus groups may last two, but no more than two and a half hours. In addition to the facilitator, we will have a note taker and audio recording of the sessions. All participants will review and sign consent forms and provide recorded oral consent at the beginning of the session. Recordings and notes will have participant names and PII redacted for file storage and analysis. Recordings will not be transcribed; they serve as a backup to the formal notes taken during the focus group to help reduce recall bias of the facilitators.²

Formal notes will be recorded using a standardized Excel template. Each participant will be assigned a number and their input will be recorded on the corresponding row in the template. Demographic information will be included for each participant in their corresponding row. Upon completion of each focus group the key linking participant names to their corresponding numbers will be destroyed. CMS and its contractors will not keep any form of PII or PHI. Focus group recordings will be edited, where necessary, to remove all PII.

Description of Consumer Focus Group Questions

Consumer Focus Group Questions are detailed in Table 3 below.

Table 3: Proposed Consumer Focus Group Questions

#	Consumer Focus Group Questions	Notes and Rationale
1	Many people find health care coverage confusing. What kinds of information did you use when you purchased a plan or looked into purchasing a plan? What kind of information did you wish you had more of when you purchased a plan on the Marketplace?	We seek to understand what information consumers used (and whether they used information on the Exchange) and what information they would have liked to see on websites like the Exchange websites or those provided by QHP issuers.3
2	When you think about health care coverage and costs, what are the most important cost considerations?	Cost is the most important consideration for the majority of consumers. We seek to understand what the most important cost considerations are for consumers. The QHP Enrollee Survey has limited influence over price structure. Therefore, we hope to focus on topics of importance to consumers where the survey could provide issuers with actionable information.
3	Other than cost, what is the most important thing you look for in health insurance coverage? What is the least important?	When controlling for cost, often the most important consideration, we seek to understand what the other important considerations are for consumers and consumers.
4	What kinds of interactions do you have with your health plan over the course of the year?	The QHP Enrollee Survey asks about getting needed information and other interactions with QHP issuers. We seek to understand what types of interactions consumers are having and whether or not the survey measures these.
5	How do you get needed information to find a primary care provider or a specialist, or to understand your benefits?	The QHP Enrollee Survey contains questions about where and how consumers get information about their health plan. Currently, these questions provide little actionable information for QHP issuers. Depending upon consumer information seeking behavior, the survey could be refined to provide more actionable information. (For instance, if consumers report difficulty navigating plan websites, issuers could use survey data to measure improvement as they redevelop websites.)
6	How much control over your providers, like your physician, health clinic, and hospital, do you think your health insurer has?	The QHP Enrollee Survey measures enrollee experience with providers, but QHP issuers have stated they have little to no influence over these interactions. We seek to understand what the consumer perception of this influence is.
7	What can a health insurance plan do to help support your overall well-being? - What can a health insurance plan do to support your overall health and quality of life?	The QHP Enrollee Survey does not measure overall well-being. We seek to understand how consumers perceive the influence of health insurance coverage on their overall well-being.
8	What does “quality” mean to you when it comes to health insurance coverage?	The QHP Enrollee Survey is used to improve health plan quality. It is important to understand how consumers think about quality and whether or not the survey operationalizes and measures quality in alignment with consumer perception.4

Description of Consumer Focus Groups Analysis

Preliminary analysis will take place with the facilitator and note taker immediately after the focus group sessions, wherein the note-taker and facilitator will review notes, filling in gaps, adding impressions as needed, and removing short-hand (to ensure comprehension at later dates). Secondary analysis will be conducted after both consumer focus groups have taken place. This will involve developing a coding scheme of major themes, coding focus group notes, and developing coding trees as patterns emerge that relate major and minor themes. When applicable, illustrative quotes will be included in the relevant thematic categories; focus group recordings will be used to ensure exact participant language. These data will be used to identify places where consumer preferences align with the current QHP Enrollee Survey measures and where there may be gaps that indicate additional topics areas for inclusion.

QHP ISSUER REPRESENTATIVES FOCUS GROUP

Description of Participants

CMS also intends to hold one issuer focus group with approximately six to eight participants recruited through the following channels: email and telephone outreach using Healthcare Organization Questionnaire (HOQ) contact information, stakeholder groups like AHIP, and CMS’ issuer newsletters. To be eligible to participate in the issuer focus group, the participant must: 

1. Work for a QHP currently operating on the Exchange;

2. Have job responsibilities in QI or management, direct customer service, or appeals management for the past year;

3. Be interested in providing input on health care quality measures.

Description of QHP Issuer Representative Focus Group Protocol

Issuer representatives will not be offered a financial incentive for participating in a focus group. CMS will conduct the issuer focus group via WebEx^® video conference. Participants will receive focus group questions, a copy of the 2020 QHP Enrollee Survey English Questionnaire, the 2019 National Level QHP Enrollee Survey QI Report, and an informed consent form at least 72 hours before the focus group. Participants will be asked to return a signed consent form via email. At the start of the session, the team will record oral consent provided by all participants.

The focus group may last two, but no more than two and a half hours. In addition to the facilitator, we will have a note taker and audio recording of the sessions; the session will be recorded with either WebEx^® technology or a digital audio recorder. All participants will review and provide recorded oral consent at the beginning of the session. Recordings and notes will have participant names and personally identifiable information (PII) redacted for file storage and analysis. Recordings will not be transcribed; they serve as a backup to the formal notes taken during the focus group to help reduce recall bias of the facilitators.

Formal notes will be recorded using a standardized Excel template. Each participant will be assigned a number and their input will be recorded on the corresponding row in the template. Upon completion of the focus group the key linking participant names to their corresponding numbers will be destroyed. CMS or its contractor will not keep any o PII or PHI. Focus group recordings will be edited, where necessary, to remove all PII.

Description of Issuer Focus Group Questions

In the issuer focus group, we will ask questions shown in Table 4 to generate discussion among issuer representatives on key topics.

Table 4: Proposed Issuer Focus Group Questions

#	Focus Group Question	Notes and Rationale
1	Have you reviewed the QHP Enrollee Survey QI Report for your reporting unit? If not, did you know that this tool exists? If so, how do you use those data to improve plan quality?	We seek to understand how issuers are currently using the survey data.
2	Overall, how useful do you find the QHP Enrollee Survey for quality improvement? Which questions provide the most value for quality improvement? Which questions provide the least value for quality improvement?	We seek to understand how useful and valuable QHP issuers find specific questions.
3	Actionable information refers to meaningful data that are useful for decision making or problem solving. Is there specific missing actionable information that you would like to see included in the survey either by adding one or more questions or revising the existing questions?	QHP Issuers have reported a lack of actionable data collected by the QHP Enrollee Survey. CMS is seeking to understand what types of actionable information issuers need in order to make and monitor quality improvement.
4	What are the three top questions that Exchange enrollees most frequently ask your help desk?	Understanding why consumers contact their QHP issuer may help CMS develop and implement more relevant survey measures.
5	What are points of comparison plans use to evaluate performance? For example, issuer comments to the QRS call letter process have mentioned alignment of CAHPS surveys across product lines would allow for better comparison.	We seek to understand how Exchange plans are compared against other types of plans.
6	What data does the QHP Enrollee Survey gather that is not collected elsewhere?	We seek to understand whether the QHP Enrollee Survey gathers novel data issuers do not have access to from any other means.
7	What questions are not particularly helpful because your organization uses data from other sources? What are the other sources you routinely use to monitor quality?	Similarly, to question six, we seek to understand what QHP Enrollee Survey questions bring little to no value to the improvement of healthcare because similar or better data can be found elsewhere.
8	Are there other sources you use to assess member experience?	We seek to understand how else QHP issuers access member experience.
9	Are there any sources you use to assess member satisfaction with their healthcare coverage?	CAHPS surveys measure experience, not satisfaction. Is enrollee satisfaction with their healthcare measured? If so, how?

Description of Issuer Focus Group Analysis

Preliminary analysis will take place with the facilitator and note taker immediately after the focus group, wherein the note-taker and facilitator will review notes, filling in gaps, adding impressions as needed, and removing short-hand (to ensure comprehension at later dates). Secondary analysis will be conducted soon thereafter. This will involve developing a coding scheme of major themes, coding focus group notes, and developing coding trees as patterns emerge that relate major and minor themes. When applicable, illustrative quotes will be included in the relevant thematic categories; focus group recordings will be used to ensure exact participant language. These data will be used to identify how issuers use the QHP data, how useful they perceive the information to be, and whether there may be places for improvement from the issuers’ perspective.

CONSUMER COGNITIVE TESTING

Description of Participants

CMS will recruit participants for two rounds of cognitive testing. Cognitive testing interviews will be one-on-one consisting of one consumer participant and one research interviewer. Each round of cognitive testing will include 10 -12 participants, (for a total of 20-24 participants).

We will offer potential participants an incentive of $100. Consumers who complete a cognitive testing interview will receive the incentive.

Similar to the Consumer Focus Groups, CMS will screen potential cognitive testing participants to meet one of the two following criteria:

1. They must have purchased a health insurance plan through the Exchange during the 2018 or 2019 open enrollment period. Because we are particularly interested in consumers who have recent experience with the process of selecting health plans in this new environment, preference will be given to those who purchased coverage during the 2019 (for plan year 2020) open enrollment period due to full public display of plan quality and member experience ratings.

2. Additional eligible participants include consumers who are uninsured, but eligible for health insurance through the Exchange, as a way of including potential consumers who might purchase coverage on an Exchange in the future. 

Consumers will be screened further to include a diverse mix of participants from each of four target populations, as summarized in Table 5. The screening tool asks key demographic questions to assist in recruiting a diverse population based upon health status, income, plan/metal level, age, gender, and race. We will strive to get the most diverse mix possible while still adhering to our timeline for data collection and analysis.

Table 5: Cognitive Testing Participant Target Populations

Target Population	Characteristics
Young and healthy	a. Must be between the age of 18 and 34 b. Must not have been diagnosed (now or ever) with any chronic condition
Healthy and age 35+	a. Must be between the ages of 35 and 64 b. Must not have been diagnosed (now or ever) with any chronic health condition
Adults with chronic conditions or disabilities	Must be currently be diagnosed with a chronic condition, such as asthma, diabetes, or hypertension, chronic mental health condition(s) and/or Must be a consumer who is deaf, blind, or has other physical or developmental disabilities
Adults with low income	Must have an income below 250% of the federal poverty level, but not enrolled in Medicaid (However, Medicaid recipients using an 1115 waiver to purchase coverage on an Exchange can be included.)

Description of Consumer Cognitive Testing Protocol

CMS will conduct cognitive testing with consumers via telephone or using WebEx video conference, depending upon the consumer’s technical capabilities and preferences. Consumers will provide oral informed consent and oral consent to audio record the testing session. Testing will last at least one hour, but no more than two hours. Cognitive testing interviews will be recorded digitally and with participant consent.

Cognitive testing interviews will focus on the items described in Table 6 below. Additionally, testing will include review of the current questions in the QHP Enrollee Survey questionnaire. Given the limited time allotted for each interview, and the length of the current questionnaire we will be unable to review each question with every cognitive testing participant.

Table 6: Cognitive Testing Items

Change Type	Items affected	Cognitive Testing Items	Notes and Rationale
“Personal Doctor” wording changed to “Primary Care Provider”	26-38	“Your Personal Doctor” section explanatory text: “A personal doctor is the one you would see if you need a checkup, want advice about a health problem, or get sick or hurt.”  Q26. In the last 6 months, how many times did you visit your personal doctor to get care for yourself?   Testing Item: The “Your Personal Doctor” section would be renamed to “Your Primary Care Provider” followed by explanatory text: “A primary care provider is the one you would see if you need a check-up, want advice about a health problem, or get sick or hurt. A primary care provider may be a nurse practitioner, a physician’s assistant, or a doctor.” Q - In the last 6 months, how many times did you visit your primary care provider to get care for yourself?	“Primary care provider” is more current terminology than “personal doctor” and encompasses the broad range of providers (doctor, nurse practitioner, physician’s assistant) a consumer would use for primary care, rather than referring to only medical doctors. This term is also used on the healthcare.gov website.5
Test response Item order effect for former screen-in questions. Test wording change from “not applicable” to “this does not apply to me”.	3, 4, 5, 6, 9, 12, 20, 21, 24, 26, 32, 34, 39, 41	Cognitive Testing – Response Option 1 Q3, In the last 6 months, how often did written materials or the Internet provide the information you needed about how your health plan works? Never  Sometimes  Usually  Always  Not Applicable; did not need care right away  Cognitive Testing – Response Option 2 Q3, In the last 6 months, how often did written materials or the Internet provide the information you needed about how your health plan works? Never  Sometimes  Usually  Always  This does not apply to me, I did not need care right away  Cognitive Testing – Response Option 3 Q3, In the last 6 months, how often did written materials or the Internet provide the information you needed about how your health plan works? This does not apply to me, I did not need care right away  Never  Sometimes  Usually  Always	This change reduced the number of items on the survey; however, it may have inadvertently caused more respondents to select “never” when the more appropriate response choice was “not applicable.”   “This does not apply to me” is plain language that may clarify the distinction between never having used a service (not applicable) and not being able to get services when needed (never). In addition to testing alternate response options, it is necessary to test the item order effect (e.g., Response Option 2 vs. Response Option 3).
Test prescription medicine item to include birth control if used for a medical condition	64	Current Item:  Do you now need or take medicine prescribed by a doctor? Do not include birth control.    Testing Item: Do you now need or take medicine prescribed by a doctor? Do not include birth control prescribed for pregnancy prevention.	This change allows birth control to be counted as a prescription if it is used as a medication for a condition other than pregnancy prevention. This is a more accurate reflection of consumers’ medication use for medical conditions.
Test questions with inverted response options	8, 13, 14, 15, 16	Current Item: Q8. In the last 6 months, how often did the time that you waited to talk to your health plan’s customer service staff take longer than you expected? Never Sometimes Usually Always Testing Item, Q8: In the last 6 months, how often did you have to wait longer than you expected to talk to your health plan’s customer service staff? Q13. In the last 6 months, how often did your health plan not pay for care that your doctor said you needed? Never Sometimes Usually Always Q14. In the last 6 months, how often did you have to pay out of your own pocket for care that you thought your health plan would pay for? Never Sometimes Usually Always Q15. In the last 6 months, how often did you delay visiting or not visit a doctor because you were worried about the cost? Do not include dental care. Never Sometimes Usually Always Q 16. In the last 6 months, how often did you delay filling or not fill a prescription because you were worried about the cost? Never Sometimes Usually Always	The following questions were developed for the QHP Enrollee Survey: 13, 14, 15, 16. The first round of cognitive testing will test the questions and response options as they are currently drafted. CATI monitoring during data collection revealed that respondents were frequently confused by Q13 because of the awkwardly worded question and potential for a double negative (e.g., never not pay). This question would be must easier to understand and answer if the word “not” was removed. Based upon findings from the first round of testing for Q14, CMS will either draft alternate language to be tested or test a relevant cost composite that has been previously scored and tested for use in other surveys. For Q15 and Q16, we are likely to break each into two questions – one addressing a delay in visits or filled prescriptions, and the other addressing the act of not seeing a doctor or filling a prescription because of cost concern. In addition to testing current items in the cost composite, we may learn from focus groups that rather than these, “surprise” costs consumers may be more concerned with, or not understand, co-pays, co-insurance, deductibles, etc. We know that cost is the number one concern when choosing a health plan while issuers may not have flexibility in terms of consumer costs, they could take action in terms of transparency and explanations. Therefore, the QHP Enrollee Survey could provide more insight into actionable items for issuers.
Finding needed plan information on the internet	3	In the last 6 months, how often did written materials or the Internet provide the information you needed about how your health plan works? Never Sometimes Usually Always Not Applicable; did not look for any information about my health plan	Current question does not provide actionable information to QHP issuers because “the internet” is not a specific place an issuer can make specific changes to based on responses to this question. Based on findings from both issuer and consumer focus groups we may find it is necessary to ask more than one question regarding finding needed information. Such as, “How often were you able to find an in-network provider that suited your needs on your health plan’s website?” and “How often were you able to find understandable benefit information on your health plan’s website?” Conduct targeted literature review for validated questions on consumer use of websites to find needed information and include items in cognitive testing.
Test participant understanding of e-cigarette and/or vaping inclusion in current measure	46, 47, 48, 49	Q46. Do you now smoke cigarettes or use tobacco every day, some days, or not at all? Every day Some days Not at all If Not at all, go to #50 Don’t know If Don’t know, go to #50 Q47. In the last 6 months, how often were you advised to quit smoking or using tobacco by a doctor or other health provider in your plan? Never Sometimes Usually Always Q48. In the last 6 months, how often was medication recommended or discussed by a doctor or health provider to assist you with quitting smoking or using tobacco? Examples of medication are nicotine gum, patch, nasal spray, inhaler, or prescription medication. Never Sometimes Usually Always Q49. In the last 6 months, how often did your doctor or health provider discuss or provide methods and strategies other than medication to assist you with quitting smoking or using tobacco? Examples of methods and strategies are telephone helpline, individual or group counseling, or cessation program. Never Sometimes Usually Always	Test questions as currently drafted, probing participants to see if they believe electronic cigarettes/vaping is part of “tobacco and smoking” use. The Medical Assistance with Smoking and Tobacco Use Cessation (MSC) measure is designed to assess the different facets of providing medical assistance with smoking and tobacco use cessation.6 The MSC webpage notes that this matters because, “tobacco use causes disease in nearly every organ in the body.”7 E-cigarettes/vaping has been marketed as a tool to quit smoking; however, studies have indicated that e-cigarettes are linked with increased incidence of myocardial infarction8, respiratory illness9, stroke10, et al. In addition, between 2012 and 2017 the percentage of Americans who perceive e-cigarettes to be as harmful as cigarettes increased 25%.11 If participants do not understand e-cigarettes are included in these questions as currently written, we may want to consider adding e-cigarettes to the questions.
Testing Potential Addition of Narrative Response Questions		What are the most important things that you look for in a healthcare provider and the staff in his or her office? What are the most important things that you look for in a healthcare provider and the staff in his or her office? Now we’d like to focus on anything that has gone well in your experiences in the last 6 months with your provider and the staff in his or her office. Please explain what happened, how it happened, and how it felt to you. Next, we’d like to focus on any experiences in the last 6 months with your provider and the staff in his or her office that you wish had gone differently. Please explain what happened, how it happened, and how it felt to you. Please describe how you and your provider relate to and interact with each other.	Adding the CAHPS Patient Narrative questions is one-way CMS could “hear” the patient/consumer voice. These five items are part of the supplemental items for the CAHPS Clinician & Groups Adult Survey.12 Access to preventative health care that is culturally competent is known to help improve health and well-being. Understanding that the mandate of the QHP Enrollee Survey is to help understand enrollee satisfaction with the healthcare received under a plan purchased on the Exchange and to provide information to issuers for quality improvement these narrative response questions may provide more insight than that IHI measure.
Refine content of survey recruitment and outreach materials	Email and Letters	Refine and test content of the following: Email subject lines Email Survey Notification Email Survey Reminder Prenotification Letter Cover Letter 1 Cover Letter 2 Reminder Letter	Wording and messaging in survey invitation materials can positively and negatively impact response rates. New research conducted by AHRQ suggests revisions that will positively impact response rates on CAHPS surveys. We plan to revise QHP materials according to this new research and test it with the consumer population.13

1 Best practices for HCD and qualitative research require empathetic facilitation and a forum where all participants feel comfortable sharing experiences and opinions, therefore focus groups should share similar demographic characteristics (e.g., education level).

2 Recall bias is a systematic error caused by inaccurate recollections.

3 Hibbard JH, Pawlson LG. Why Not Give Consumers a Framework for Understanding Quality? Joint Commission Journal on Quality Improvement. 2004. 30(6); 347-351

4 Hibbard J. Engaging Consumers in Quality Issues: While the road to engaging consumers is steep, it is fairly well marked. Washington, DC: National Institute for Health Care Management Foundation; October 2005. Available at http://www.nihcm.org/pdf/ExpertV9.pdf

5 https://www.healthcare.gov/blog/finding-a-provider/

6 https://www.ncqa.org/hedis/measures/medical-assistance-with-smoking-and-tobacco-use-cessation/

7 U.S. Department of Health and Human Services (HHS). 2014. The health consequences of smoking—50 years of progress: a report of the Surgeon General.” Atlanta, GA. http://www.surgeongeneral.gov/library/reports/50-years-of-progress/full-report.pdf

8 Alzahrani T, Pena I, Temesgen N, Glantz SA. Association Between Electronic Cigarette Use and Myocardial Infarction [published correction appears in Am J Prev Med. 2019 Oct;57(4):579-584]. Am J Prev Med. 2018;55(4):455–461. doi:10.1016/j.amepre.2018.05.004 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC6208321/

9 Perez M, Atuegwu N, Mead C, Oncken E, Mortensen E. E-Cigarette Use is Associated with Emphysema, Chronic Bronchitis and COPD. Am J Respir Crit Care Med. 2018; 197: A6245. https://www.atsjournals.org/doi/pdf/10.1164/ajrccm-conference.2018.197.1_MeetingAbstracts.A6245

10 Ndunda P, Muutu T. Electronic Cigarette Use is Associated with a Higher Risk of Stroke. Stroke. 2019; 50: A9 https://doi.org/10.1161/str.50.suppl_1.9

11 Huang J, Feng B, Weaver SR, Pechacek TF, Slovic P, Eriksen MP. Changing Perceptions of Harm of e-Cigarette vs Cigarette Use Among Adults in 2 US National Surveys From 2012 to 2017. JAMA Netw Open. 2019;2(3): e191047. doi: https://doi.org/10.1001/jamanetworkopen.2019.1047

12 https://www.ahrq.gov/cahps/surveys-guidance/item-sets/cg/suppl-narrative-items-cg-survey30-adult.html

13 Developing Invitation Messages That Increase CAHPS Survey Response Rates. Content last reviewed November 2019. Agency for Healthcare Research and Quality, Rockville, MD. https://www.ahrq.gov/cahps/news-and-events/events/webinar-091019.html

5