Response to 60-day comments

Date Received	Topic	Comment	Relates to CAHPS Q?	Relates to QRS Q?	Supporting statement A or B?	Source	Team Decision
6/29/2015	Additional custom questions	Finally, we advocate for the opportunity to add a small number of custom questions to assist in cultural and linguistic reporting.	No	No	NA	Anthem/ AHIP/ BCBS Michigan	CMS will consider these questions and other revsions in the future; however because of the rounds of psychometric and beta testing already done we will not include additional questions at this time to ensure that we are able to implement the 2016 QHP Enrollee Survey in appropriate timeframes.
6/29/2015	Analysis	Include metal level as a variable of analysis	No	No	NA	Anthem	We are using metal level in our subgroup and disparities analysis.
6/29/2015	Cognitive testing	Ensure reliable cognitive testing of questions, especially those that are new to CAHPS.	No	No	NA	Anthem/AHIP/ BCBS Michigan	We have conducted three rounds of cognitive testing of all new items and all items were also included in the psychometric testing after the field test.
6/29/2015	Cognitive testing	Publicly share results of the cognitive testing for all newly added questions to the QHP survey in future public comment periods	No	No	NA	Anthem/ AHIP/ BCBS Michigan	CMS will take this into consideration.
6/11/2015	Cover Letter	Suggested Change: The middle sentence of the first paragraph should be changed to read: This survey is part of a national ongoing effort to understand the experiences enrollees have with their health plan. Rationale: The plan is interested in member experience, and this wording reflects the intent of the survey.	No	No	NA	Tufts	This wording is from the standard CAHPS Health plan cover letter. However, we will make the change to the letter to better reflect the intent of the survey.
6/30/2015	Cover Letter	As it concerns the QHP Survey Cover Letter, we believe that there may be some confusion with the intent and use of the term "health plan." The second paragraph reads, "If you are enrolled in a different health plan for 2015, please answer the questions in the survey thinking about your experiences in your previous health plan from July through December 2014." While some consumer may switch insurers, others may change metal tiers within the same issuer. Clarification with respect to issuer and metal tier will help survey respondents more accurately identify how they would like to respond.	No	No	NA	AHIP/ BCBS Michigan	We do not believe that adding the terms metal tier and issuer will clarify this for survey respondents. We have revised the wording to read: "If you changed your health plan for 2016 please answer the questions in the survey thinking about the health plan you had from July through December 2015."
6/29/2015	Psychometric testing	Ensure accuracy of the survey results through validity and reliability testing of the survey tool and results prior to implementation.	No	No	NA	Anthem/ AHIP/ BCBS Michigan	We have tested the validity and reliability in the 2014 psychometric field test and again in the 2015 beta test.
6/29/2015	Q12 & Q13	QHP Survey questions 12 and 13 focus on culturally and linguistically appropriate care and specifically whether enrollees needed an interpreter to speak with anyone at their doctor’s office, and how often an enrollee received an interpreter at their doctor’s office. Incorporating questions about interpretive services into the English version of the survey may lead to a low response rate for these questions because we expect very few English speaking respondents to request such services. These two questions are more appropriate for the Spanish or Mandarin versions of the survey. Additionally, questions 12 and 13 may cause confusion for the enrollee. The regulatory requirement to supply interpreter services in a provider’s office is directed at the physician and we believe the intent of the questions is to ask about interpreter services provided by the doctor’s office or clinic. Many times a patient relies on a family member to be the interpreter and as a result, an enrollee who indicates in question 12 that they need an interpreter, may respond that they received one if a family member or friend interpreted for them. This would undermine the value of the questions and CMS should engage in further cognitive testing of these questions and revise them as needed before incorporating them as part of the survey. CMS also needs to consider whether these questions should be revised to evaluate health plan customer service relative to culturally and linguistically appropriate care rather than focusing on care in the physician office, as some health plans currently provide members with interpreter services to help them communicate with their health care providers. These revised questions could be used to augment existing survey questions such as question 49 regarding availability of health plan forms in the language enrollees prefer. Additionally, we recommend that these questions be tailored to align with the accessibility standards [45 CFR 155.205(c)] to ensure access for individuals with limited English proficiency and individuals with disabilities. The revised questions should also be subject to reliability and validity testing so that a low response rate does not skew the survey results.	No	Yes	NA	Anthem/ AHIP/ BCBS Michigan	Most issuers were not able to provide a language preference indicator in the sample frame by which the survey vendor could send Spanish or Chinese surveys. As a result, these questions need to stay in the English version of the survey because that is the version most respondents will get. These are CAHPS items that have been tested. These items are part of the Quality Rating System measures so cannot be dropped at this time. CMS will consider this feedback for future administrations.
6/30/2015	Q14-31	Comments about: Personal Doctor • The first question in the “Your Personal Doctor” section defines a “personal doctor” as “the one you would see if you need a check-up, want advice about a health problem, or get sick or hurt.” This definition does not necessarily differentiate between doctors seen in any care facility (urgent care clinic, ER, etc.)? If there needs to be a distinction, we suggest including the term “primary care physician” or making reference to the regularity or frequency of contact with that provider; i.e. “the one you would normally (or typically) see if you need a check-up…”? • In the question “In the last 6 months, did you take any prescription medicine?” we recommend including a brief definition of prescription (as opposed to OTC) medicine.	Yes	Yes	NA	BCBS Michigan	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations.
6/11/2015	Q42	Suggested Change to Question Wording: In the last 6 months, did you seek information or help from your health plan’s customer service? Rationale: “Seek” implies looking for information and taking the action to contact customer service. “Get” implies that information was passively provided.	Yes	Yes	NA	Tufts	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations.
6/29/2015	Q42	Question 42 asks whether enrollees have received information or help from their health plan customer service. We recommend revising this question to include those members who needed or unsuccessfully tried to get information from customer service. In its current form, the question limits the subsequent questions to those respondents who received information or help from customer service and may therefore lead to an artificially high “Always” answer rate for question 43, which asks how often did the health plan customer service provide enrollees with the information or help they needed.	Yes	Yes	NA	Anthem/ AHIP/ BCBS Michigan	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations.
6/30/2015	Q42	We recommend changing the wording of “In the last 6 months, did you get care from more than one kind of health care provider or use more than one kind of health care service?” to “In the last 6 months, did you get care from more than one health care provider or use more than one health care service?” due to the ambiguity of what a “kind” of service or provider might refer to.	Yes	Yes	NA	AHIP/ BCBS Michigan	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations.
6/29/2015	Q44 & 45	Question 44 (customer service staff) and question 45 (wait time) both imply that the member successfully contacted customer service by phone, ignoring other possible modes of contact (e.g., the plan’s website or email) or outcomes (e.g., could not get through or get a live representative). This is an additional justification to revise the screener question (question 42) to explicitly ask about the number of member attempts to call customer service.	Q44- Yes Q45-No	Q44- Yes Q45-Yes	NA	Anthem/ AHIP/ BCBS Michigan	While we agree that there are additional modes of contact, phone is a constant mode among all plans and other modes vary considerably among plans.
6/30/2015	Q45	Additionally for Question 45, there is concern about the term “longer than you expected.” This will tend to be Usually or Always because a specific time period isn't specified. We suggest changing to 'longer than 10 minutes' or whatever time period reflects the generally acceptable standard wait time for service.	No	Yes	NA	BCBS Michigan	People's perceptions of the wait time can vary and their perception of whether it was longer than expected or not is what matters more than the actual wait time. This question was cognitively tested and included in the psychometric field test without issue.
6/29/2015	Q46	Question 46 asks how often did health plans provide enrollees with forms to fill out. This question implies that forms can only be given by the plan, and enrollees who go online to download claim forms may feel that the question does not apply to them. We recommend revising the question so that the language is broader. For example, the question could be asked, “In the last 6 months, did you have to fill out any forms from your health plan?”	Yes	Yes	NA	Anthem/ AHIP/ BCBS Michigan	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations.
6/30/2015	Q46 & 47	Because Question 46 and the follow-up Question 47 don’t identify specific forms, the results would not be useful to health plans or members because no corrective action plans could be targeted.	Yes	Yes	NA	BCBS Michigan	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations.
6/29/2015	Q52	Question 52 asks enrollees to rate their health plan from 0 to 10. Health plans have had difficulties with interpreting CAHPS responses to all rating questions and particularly, question 52. It is difficult for health plans to ascertain what factors such as enrollee experience with claims, customer service, providers or the coverage the plan provides, out of pocket expenses to the member, or public perception of the plan, affect an enrollee’s rating.	Yes	Yes	NA	Anthem/ AHIP/ BCBS Michigan	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations.
6/11/2015	Q53	Suggested Changes to Question Wording and Scale Used: The question should be changed to: How likely is it you would recommend your health plan to a friend or family member? The scale should be changed to 0 “Not At All Likely” to 10 “Extremely Likely.” Rationale: This wording and scale allows for the calculation of a Net Promoter Score.	No	No	NA	Tufts	Multiple stakeholders recommended changing the question and response scale to allow for calculation of a Net Promoter Score. We have made this change. The goal of the Net Promoter Score is to know the number of people that promote the plan and the number that do not. We believe this will be the first CAHPS Survey to include this.
6/29/2015	Q53	Question 53 asks about recommending the health plan to friends and family. We suggest this question be rated on a scale of 1 to 10 to allow for NPS (net promoter score) alignment if the health plan so chooses.	No	No	NA	Anthem/ AHIP/ BCBS Michigan	Multiple stakeholders recommended changing the question and response scale to allow for calculation of a Net Promoter Score. We have made this change. The goal of the Net Promoter Score is to know the number of people that promote the plan and the number that do not. We believe this will be the first CAHPS Survey to include this.
6/30/2015	Q53, 54, 55, 56, 57, 77, 81, 82, and 83	Test all new survey questions. As previously stated, ACAP strongly disagrees with inclusion of untested survey questions in publicly reported quality scores, and feels strongly that all new survey questions should be rigorously tested. Our concerns apply to all nine new survey questions included in the 2016 Qualified Health Plan Enrollee Experience Survey: 53, 54, 55, 56, 57, 77, 81, 82, and 83.	No	No	NA	ACAP	We have conducted three rounds of cognitive testing of all new items and all items were also included in the psychometric testing after the field test.
6/11/2015	Q54	Suggested Change: In the last 6 months, how often did your health plan not pay for care that your doctor said you needed? Rationale: Changing “service” to “care” makes the question consistent with Q55.	No	No	NA	Tufts	We have made this change.
6/30/2015	Q54	Eliminate or modify question #54 to avoid bias. ACAP recommends either eliminating Question #54 or modifying it to exclude the doctor reference, so that it reads “In the last 6 months, how often did your health plan not pay for a service that you thought the health plan would pay for?” The current draft’s wording of the new survey question #54 follows: “In the last 6 months, how often did your health plan not pay for a service that your doctor said you needed?” We believe that this language contains embedded bias by including a reference to a doctor, since it is possible not all doctors fully understand the health plan’s benefit coverage details. Furthermore, since there is a chance a doctor may not submit sufficient medical necessity documentation when requesting a prior authorization for a service, the responsibility for some resulting denials may rest with the provider. The modified wording we suggest above focuses instead on the member’s understanding of the health plan’s benefits, rather than on the doctor’s understanding of the member’s benefit coverage.	No	No	NA	ACAP	We recognize that payment is often denied for legitimate reasons and these questions capture both illegitimate denials and legitimate denials that the beneficiary may not understand. We are not trying to distinguish those because the beneficiary may not be able to make that distinction. We are trying to find out how often beneficiaries expectations aren't met and it's up to the plan to decide whether that is a coverage issue or a messaging issue. This information will not be publicly reported by CMS at this time and the items are included in this national testing in order to test whether the information is useful. Whether it's legitimate or illegitimate it represents a misunderstanding on the part of the consumer.
6/29/2015	Q54 & 55	Question 54 asks about services not paid by the health plan and Question 55 asks about services they had to pay for. Those new to coverage may have unrealistic expectations of coverage. In addition, those changing plans may have had broader coverage from their prior plan. We believe these questions are too subjective and should be eliminated or re-worded.	No	No	NA	Anthem	We recognize that payment is often denied for legitimate reasons and these questions capture both illegitimate denials and legitimate denials that the beneficiary may not understand. We are not trying to distinguish those because the beneficiary may not be able to make that distinction. We are trying to find out how often beneficiaries expectations aren't met and it's up to the plan to decide whether that is a coverage issue or a messaging issue. This information will not be publicly reported by CMS at this time and the items are included in this national testing in order to test whether the information is useful. Whether it's legitimate or illegitimate it represents a misunderstanding on the part of the consumer.
6/29/2015	Q54. 55. 56. & 57	QHP questions 54-57 pertain to information relating to affordability, such as the enrollee’s cost of services, and any unexpected incurred costs and appear to have been re-introduced to the QHP survey for 2016. These questions raise several concerns. First, questions 54-57 are vaguely written and do not address affordability relative to a QHP, as an enrollee’s answers are dependent upon benefit packages. We recommend these questions be redrafted. Question 54 asks whether a health plan has refused to pay for a service the enrollee’s doctor said they needed. We are concerned that the language “not pay” will be misinterpreted by someone who is new to coverage and will also negatively bias responses. For example, services need to be part of the benefit package in order to be reimbursed and in addition enrollee dissatisfaction when a deductible is imposed may appear to be non-coverage to a new enrollee. We recommend question 54 be reworded to avoid bias and clarified as to the intended purpose. Second, questions 55-57 seek to determine whether an enrollee experienced unexpected costs associated with care, and also whether the enrollee delayed or did not visit the doctor or fill a prescription due to cost. This language is vague and an enrollee’s response may be confounded due to the clinical course of treatment and unexpected complications. CMS should revise these questions to improve clarity.	No	No	NA	Anthem/ AHIP/ BCBS Michigan	These items have been cognitively tested and were tested in the field test. We are interested in what the consumer expectation was. Regardless of the reason, we want to understand whether there was consumer misunderstanding.
6/11/2015	Q56	Suggested Change: In the last 6 months, how often did you delay visiting or not visit a doctor because you were worried about the cost? Do not include dental care. Rationale: Adding the word “visiting” makes the sentence grammatically correct and easier to understand.	No	No	NA	Tufts	We agree and have made this revision.
6/29/2015	Q56 & 57	Questions 56 and 57 asks if an enrollee is ‘worried’ about cost of services. Anthem questions the inclusion of these specific questions as the term ‘worried’ is a broad term. CMS should consider adding follow-up questions to both Question 56 and Question 57 such as ‘did you investigate the costs by going on-line’ or ‘did you talk with member services to confirm.’	No	No	NA	Anthem	This is a common phrasing of cost questions and have been cognitively tested and included in the field test.
6/11/2015	Q57	Suggested Change: In the last 6 months, how often did you delay filling or not fill a prescription because you were worried about the cost? Rationale: Adding the word “filling” makes the sentence grammatically correct and easier to understand.	No	No	NA	Tufts	We agree and have made this revision.
6/30/2015	Q61-65	For Questions 61 and 65, “don’t know” should be removed as an option as people would be able to affirm their answers to these questions. Also, if Q65 is meant to assess aspirin use for cardiovascular benefits, the question should specify this as the reason for use. For example, people with other conditions such as migraines, frequent pain, etc. may take aspirin regularly. “Do you now smoke cigarettes or use tobacco every day, some days, or not at all?” Responses: Yes or No “Do you take aspirin daily or every other day to maintain or improve your heart health (excluding other reasons you may take aspirin)?” Responses: Yes or No	No	Yes	NA	BCBS Michigan	These questions are HEDIS items and have been previously tested. These are also included in the QRS measures and cannot be dropped.
6/30/2015	Q66-69	In the About You section, Q66-69, there are questions regarding daily aspirin use and health conditions the respondent may have or had had (high cholesterol, heart attack, etc.). These questions may seem too invasive for some respondents (fearing that responses will be shared with the health plan). Some responses may not be truthful.	No	Yes	NA	BCBS Michigan	These questions are HEDIS items and have been previously tested. These are also included in the QRS measures and cannot be dropped.
6/30/2015	Q68 & 69	Additionally, Questions 68 and 69 about health conditions ask respondents to “Mark one or more” (implying that at least one condition must be indicated). We suggest changing this to “Mark all that apply” and adding “None” as a response option.	No	Yes	NA	BCBS Michigan	We agree that this question should be modified. These questions are HEDIS items and cannot be changed at this time but we will give the feedback to NCQA for future revisions.
6/30/2015	Q70	Comments about: About You • In Question 70, respondents are asked if they have received health care “3 or more times for the same condition or problem” followed by “Is this a condition or problem that has lasted for at least 3 months? Do not include pregnancy or menopause.” Should the pregnancy/menopause disclaimer be moved or added to the first question, or is the intention to capture pregnancy and menopause as conditions for the first question?	Yes	No	NA	BCBS Michigan	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations.
6/30/2015	Q82	• In Question 82, “How confident are you that you understand health insurance terms,” it may be necessary to offer examples of “terms” – such as co-pay, co-insurance, referral, etc. to help focus responses on specific items, thereby making results more useful.	No	No	NA	BCBS Michigan	This item is part of the Health Insurance Literacy Measure that has been validated through cognitive testing and a national field test. We will pass along the feedback to the measure team.
6/11/2015	Q77	Suggested Changes to Answer Options: The “Unemployed” answer option should be expanded to include “Unemployed and actively seeking employment” and “Unemployed and NOT actively seeking employment.” For the “Other” option, the respondent should be able to specify their “custom” answer. Rationale: This allows for more specific information that can be used in data analysis.	No	No	NA	Tufts	This item is from the National HIV Behavioral Surveillance System Survey and the item was tested and validated in that survey. It's a one question approach to addressing this and doesn't increase burden. We do not think we will have the sample size to differentiate between reasons for unemployment and do not believe the information is necessary. The purpose is to identify people that do not have insurance because they are unemployed.
6/29/2015	Q77	Question 77 asks about employment status. We don’t believe that including student and home maker as distinct categories of employment status is appropriate. Those should be included in the ‘Other’ option.	No	No	NA	Anthem	This item is from the National HIV Behavioral Surveillance System Survey and the item was tested and validated in that survey. It's a one question approach to addressing this and doesn't increase burden.
6/11/2015	Q80	Suggested Change: There should be an “Other” option, and the respondent should be able to specify their “custom” answer. Rationale: This allows for more specific information on how the member self-identifies regarding race and does not force the member to choose one of the options provided if none of them are accurate.	Yes	No	NA	Tufts	This question is a data collection standard specified by the office of minority health.
6/29/2015	Q81, 82, & 83	With the addition of the new health literacy related questions to the “About You” section (question 81, 82 and 83), this section now accounts for just shy of one-third of all questions in the survey. We believe this extensive set of questions distracts from the purpose of evaluating plan performance.	No	No	NA	Anthem	More than half of these questions are included because they are required for public reporting in QRS measures. Many of the other questions are case mix adjusters.
6/11/2015	Q82	Suggested Changes to Question Wording and Answer Options: The question should be changed to: How well do you understand your health insurance terms? Response Options: Not At all, Not Very Well, Somewhat Well, Very Well. Rationale: Confidence is not relevant to the question. Also, adding the word “your” helps the plan better understand if the member understands the plan’s terms versus the terms of health insurance in general.	No	No	NA	Tufts	This question wording is consistent with multiple validated health literacy surveys. Further, this particular item comes from the Health Insurance Literacy Measure which is a validated instrument. We will give the feedback to the measure team.
6/30/2015	Q82	Provide examples of terms in question #82. ACAP supports the inclusion of question #82 and believes that surveying QHP members on health literacy will yield critical and useful results. However, we believe that the question will be more useful – and respondents will be better able to respond – if CMS includes specific insurance terms in the question. For example, the question could be altered in the following way: “How confident are you that you understand health insurance terms, such as “premium,” “deductible,” “copay,” “coinsurance,” and “maximum out-of-pocket limit”?	No	No	NA	ACAP	This item comes from the Health Insurance Literacy Measure which is a validated instrument. We will give the feedback to the measure team.
6/11/2015	Q83	Suggested Changes to Question Wording and Answer Options: The question should be changed to: How well do you understand most of the things you need to know about using health your health insurance? Response Options: Not At All, Not Very Well, Somewhat Well, Very Well. Rationale: Confidence is not relevant to the question. Also, adding the word “your” helps the plan better understand if the member understands how to use the specific plan versus how to use health insurance in general.	No	No	NA	Tufts	This question wording is consistent with multiple validated health literacy surveys. Further, this particular item comes from the Health Insurance Literacy Measure which is a validated instrument. We will give the feedback to the measure team.
6/29/2015	Q 14-31	QHP Survey questions 14-31 ask about an enrollee’s personal doctor and focus on provider communication and care coordination. To be truly reflective of a QHP’s performance, we support the replacement of these measures with questions that capture information about the quality of the plan’s provider network and that are applicable to areas that a health plan can directly influence. A health plan’s ability to influence clinicians can vary by type of provider. For example, the ACA requires health plans to include specific providers in their network who may not have previously contracted with private health insurers and thus may not have been part of performance reporting and consumer reviews (e.g., essential community providers). Additionally, those providers may not initially have the capacity to undertake quality improvement efforts needed to promote quality and patient satisfaction.	Yes	Yes	NA	Anthem/ AHIP/ BCBS Michigan	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations. Further, many of these items are included in measures for the QRS and cannot be changed at this time.
6/30/2015	Q 20-25	If Questions 20 and 25 are kept, a “don’t know” response option should be included: o “When you visited your personal doctor for a scheduled appointment in the last 6 months, how often did he or she have your medical records or other information about your care?” o “In the last 6 months, how often did your personal doctor seem informed and up-to-date about the care you got from specialists?”	Yes	Yes	NA	BCBS Michigan	There were multiple recommendations for revisions to item wording for core CAHPS questions that have been previously validated in cognitive testing and field testing. We are not making changes to items from the CAHPS Health Plan core questionnaire in order to allow CMS, policy makers, and health plans to make direct comparisons between their QHP and Medicaid populations. Further, many of these items are included in measures for the QRS and cannot be changed at this time.
6/29/2015	Q 36- 37	Questions 36 and 37 ask about written materials or Internet information about health plans. It is unclear whether CMS’ intent is to measure plans’ performance in providing information about the plan to consumers. Because these questions do not specify written materials from your health plan or your health plan’s website, respondents are likely to reference other sources of written materials or websites. For QHP members this is likely to include the Marketplace website and written materials. As such, these questions will not be a good measure of plans’ performance. We recommend changing the language of these questions to refer to “written material from your health plan” and “your health plan’s website” Alternatively, CMS should assess the availability of information on the Marketplace website through the Marketplace survey, rather than including questions in the QHP survey.	Q36- No Q37- No	Q36- Yes Q37- No	NA	Anthem/ AHIP/ BCBS Michigan	These questions are HEDIS items and have been previously tested. We will give the feedback to NCQA for future revisions.
6/30/2015	Q 36-37	Additionally, the first question asks if the respondent looked for information on the Internet about their health plan, but in the follow-up question, it asks how often they found information about how their health plan works. We recommend changing the wording of the second question to “In the last 6 months, how often did the written materials or the Internet provide the information you needed about how your health plan works?”	No	Yes	NA	BCBS Michigan	The question is currently worded this way.
6/30/2015	Q 3-9	Comments about: Your Health Care in the Last 6 Months • With regards to the survey section entitled “Your Health Care in the Last 6 Months,” questions 3-9 are asked first about care received “in a clinic, emergency room, or doctor’s office,” or about care received “at a doctor’s office or clinic.” The definition of “clinic” may be unclear to respondents. The term “urgent care” should be used in questions 3 and 9 if the intention is to capture visits in urgent care facilities, as well as emergency rooms, doctor’s offices, and other clinics.	Yes	Yes	NA	BCBS Michigan	"Clinic" is the term CAHPS survey uses so we will not change at this time.
6/29/2015	Sampling methodology	Clarify the survey methodology and indicate whether (and, if so, how) the survey sample will be selected to ensure the same person does not get both the Marketplace and the QHP survey	no	no	B	Anthem	There is no way to ensure that because the samples are drawn by different organizations.
6/29/2015	Sampling methodology	Is CMS reducing the number of surveys to 800 or does the health plan have to meet a threshold of 800 eligible members and send 1,000 surveys depending on the eligible membership? If the intent is to reduce the sample from 1,000 surveys to 800, we are not clear how decreasing the sample size will increase response rates. We recommend CMS increase its sample size to increase the number of completed surveys to improve the validity of the survey results.	No	No	B	Anthem	We are increasing the sample size to 1300 in 2016 and have made this correction in Supporting Statement B.
6/29/2015	Survey Length	Survey Length - We are concerned with the length of the survey as it contains 85 questions up from 76 in last year's survey. We recommend CMS taper down the number of survey questions to improve survey participation rate. CMS should use the results of the pilot test to limit the number of questions to ensure that the survey is a reasonable length which will improve survey completion rates.	no	no	NA	Anthem	We recognize the length of the survey, but there are many factors that affect response rates. Previous research has shown that the effect of additional questions up to even 95 questions had minimal effect on response rates. Additionally, many of the items included in the “About You” section are required for the aspirin use, tobacco cessation, and flu shot measures that are included in the Quality Rating System. Furthermore, other CAHPS surveys are of similar length. For example, the ACO CAHPS questionnaire is currently 80 questions.
6/29/2015	consistency of questions	Help ensure stability of the measures reported by maintaining consistency of the survey questions	No	No	NA	Anthem	All items on the 2016 survey were included in the 2014 psychometric field test.
6/29/2015	Analysis	Recommend that CMS assess whether satisfaction differences exist across those who have not previously had insurance, and determine if the surveys should account for these differences.	No	No	NA	Anthem/ AHIP/ BCBS Michigan	We will be doing that. Question 81 on the survey asks whether the enrollee had insurance the year before.
6/29/2015	Analysis	Given the uncertainty of reporting enrollee satisfaction at the QHP or metal level, we recommend CMS further study the survey sampling methodology and satisfaction differences across the metals levels to best account for the potential differences in enrollee satisfaction across the four metal tiers and catastrophic QHP plans. For example, an enrollee who selects a bronze plan with a lower actuarial value and higher out-of-pocket limits may be less satisfied with their QHP, resulting in a lower plan rating than an enrollee who selects a platinum plan. In the alternative, CMS could report scores at the different metal levels to account for any potential satisfaction differences across the metal tiers.	No	No	NA	Anthem/ AHIP/ BCBS Michigan	We do have this information and will include it in appropriate analyses.
6/29/2015	Analysis	We believe it would also be useful to ask if an enrollee has received an Advance Premium Tax Credit. This will assist in identifying whether a QHP population consists of low-income enrollees and the potential impact of the tax credit.	No	No	NA	Anthem/ AHIP/ BCBS Michigan	We expect to have access to this information through administrative data.
6/29/2015	Psychometric testing	Also for transparency purposes, the validity and reliability testing results of newly developed questions should be shared during a future public comment period.	No	No	NA	Anthem/ AHIP/ BCBS Michigan	CMS will take this into consideration.
6/29/2015	Case-mix adjusters	We also request clarification on case-mix adjustments for plans that enroll significantly large numbers of members who are enrolled for periods of less than three or six months. It is likely that plans experiencing churn with Medicaid and CHIP are more likely to enroll individuals with shorter enrollment spans and this may impact survey results.	No	No	NA	Anthem/ AHIP/ BCBS Michigan	These individuals would not be eligible for inclusion in the survey as members had to be enrolled in a plan for the last 5 of 6 months. We are case mixing for a number of variables.
6/30/2015	Cognitive testing	Make the survey understandable by average plan enrollee. Numerous Marketplace regulations related to enrollee-facing documents establish that such documents must be presented in a manner that is easily understandable by the average plan enrollee. ACAP supports these requirements. ACAP members have substantial experience providing coverage to people with low literacy skills. In order to ensure that critical materials are understandable, many state Medicaid programs require documents to be produced at a sixth grade reading level or lower. What is known currently about Marketplace enrollees suggests that the population is in many ways similar to the Medicaid population. Approximately 87 percent of Marketplace enrollees, for example, have low to moderate incomes, which enable them to receive premium tax credits to purchase coverage. Given this, we harbor general concerns that many Marketplace enrollees may not find the QHP enrollee survey understandable. ACAP urges CMS to examine the draft survey for understandability for the average Marketplace consumer, and, if necessary, revise the survey to ensure that it is accessible for health care consumers.	No	No	NA	ACAP	We have conducted three rounds of cognitive testing of all new items and all items were also included in the psychometric testing after the field test. Additionally, all core CAHPS questions have been previously validated in cognitive testing and field testing.
6/30/2015	Internet option	As mentioned previously, Issuers have reported early results from the 2015 beta test that indicated a much lower than expected response rate. As CMS considers ways to increase survey response rates, a better understanding of transient populations may help target likely respondents. Furthermore, assessing the feasibility of internet and or mobile based survey and subsequent implementation of a "paperless" survey option may be needed to increase response rates, understanding the need to maintain electronic security of personal information.	No	No	NA	AHIP/ BCBS Michigan	We included an internet survey option in the 2014 psychometric field test and 2015 beta test. We plan to continue offering this option and will expand internet surveys to include in English and Spanish in the 2016 Implementation. CMS is committed to a secure vendor. The most effective way of improving response rate is getting a sample with a higher proportion of cell phone numbers. Many of the issuers could not provide numbers for a substantial part of their sample.
6/30/2015	Disenrollee survey	While CMS continues to refine the QHP Enrollee Experience Survey, we recommend that CMS not proceed at this time with the initial assessment and development of a QHP Disenrollment Survey. Furthermore, we discourage CMS from using Medicare Advantage Disenrollment questions for the QHP population since these populations differ. Screening questions for this population will have to be very precise which will be challenging to develop since the assessment of populations is premature due to continued changes in eligibility and plan choice.	No	No	NA	AHIP/ BCBS Michigan	CMS will take this into consideration.
6/30/2015	Response scale	General Comments on the survey • We have a general concern with the Never/Sometimes/Usually/Always scale used in many of the questions. Considering that the surveyed time period is only regarding the last 6 months, many of the experiences asked about may have only occurred once or twice, if at all. For that reason, we recommend changing questions where applicable to a Yes/No scale. For example: o “In the last 6 months, did you and your personal doctor talk about all the prescription medicines you were taking?” o “In the last 6 months, were you advised to quit smoking or using tobacco by a doctor or other health provider in your plan?”	No	No	NA	BCBS Michigan	If there is a possibility that an experience could have happened more than once then the Never/Sometimes/Usually/Always scale should be used. If someone only had one experience then a never or always response would be appropriate. This is the standard CAHPS response option and has been tested multiple times.