Cognitive Testing for the Marketplace Dental Survey

Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback” (OMB Control Number: XXXX-YYYY)

T ITLE OF INFORMATION COLLECTION:

Cognitive Testing for the Marketplace Dental Survey

PURPOSE:

The Centers for Medicare & Medicaid Services (CMS) is responsible per section 1311(c)(4) of the Affordable Care Act (ACA) for assessing the experience of consumers with regard to the Health Insurance Marketplaces (HIMs). The Health Insurance Marketplace Consumer Experience Survey (HIM CES) project is designed to assist CMS by creating surveys that provide consumer assessments of the services provided by the Marketplaces, as well as evaluating the Qualified Health Plans (QHPs) that they purchased through the Marketplaces. To date, two major surveys have been created through this effort: the Qualified Health Plan Enrollee Experience Survey (QHP Enrollee Survey) and the Health Insurance Marketplace Survey (Marketplace Survey). At present, neither the QHP Enrollee Survey nor the Marketplace Survey examines the consumer experience with dental insurance purchased through the Marketplaces. To address this need, the current project develops a patient experience survey for individuals purchasing dental insurance through the Marketplaces, called the Marketplace Dental Survey (MDS). The draft survey instrument is presented in Attachment A.

This fast track request to conduct cognitive testing of a draft version of the Marketplace Dental Survey with consumers who purchased dental insurance through a Marketplace is part of the HIM CES project. We will conduct cognitive interviews with Marketplace dental plan consumers to test the validity of the draft Marketplace Dental Survey (MDS). Our overall purpose will be to evaluate cognitive interview participants’ understanding of the survey items. The data that we collect from cognitive interviews will help ensure the items are reliable, valid, address key domains, and minimize respondent burden. The goals for cognitive interviewing are to:

1. Determine whether items are understood as intended (making sure questions are clear and aligned with the goal of the question).

2. Determine if there are differences in how participants understand survey items based on whether:

   1. They have purchased a stand-alone versus an embedded dental plan

   2. They have purchased a dental plan from a state-based marketplace versus through HealthCare.gov

3. Ensure that the English and Spanish-language survey questions are conceptually equivalent -- that the English-language content has been translated effectively into Spanish.

4. Determine whether participants are having trouble recalling experiences of their dental care which occurred in the past six months.

5. Determine whether there are motivational or sensitivity problems that were not previously apparent to item developers.

6. Explore the impact of changing the reference period from 12-months for Dental CAHPS to 6-months, to be congruent with the other CAHPS Surveys and the Qualified Health Plan Consumer Experience Survey.

7. Identify whether response options are appropriate.

8. Suggest any revisions necessary to the survey items.

We will perform two rounds of cognitive testing in both English and Spanish.

The current Marketplace Dental Survey draft instrument was developed by a team of experts, using the formative research findings to inform survey content, style, domains and structure. Similar to the QHP Enrollee Survey, the MDS Draft takes a Consumer Assessment of Healthcare Providers and Systems (CAHPS^®) survey as its starting point. Specifically, the Marketplace Dental Survey was built upon the comprehensive work done to create the Adult Dental Plan CAHPS survey (D-CAHPS Survey)¹. In addition, the American Institutes for Research (AIR) has completed a literature review and environmental scan, stakeholder interviews, consumer focus groups, and the Technical Expert Panel (TEP) review to inform the development of the Marketplace Dental Survey.

DESCRIPTION OF RESPONDENTS:

We plan to conduct cognitive testing of the draft Marketplace Dental Survey in two locations to account for regional variations: Austin, Texas and Washington, DC. These locations provide variation in Marketplace-type, as Texas participates in the Federally-Facilitated Marketplace (FFM) and Washington, DC runs its own State-Based Marketplace (SBM). Each location has a large population of Spanish-speakers from a wide variety of countries of origin. Moreover, both of these locations have ethnic and economic diversity.

We will work with a recruitment firm to recruit cognitive testing participants. Eligible individuals will be identified using a screening questionnaire. We will recruit consumers who purchased stand-alone dental plans and consumers who purchased dental coverage included within their general Qualified Health Plan (QHP). We will over-select consumers with low health insurance literacy, as measured by the QHP Enrollee Survey item on how well the consumers comprehend insurance terms. Our eligibility criteria for cognitive testing participants will include the following (see Attachment B for our participant recruitment screener):

• Purchased a dental plan through a Marketplace,

• Received dental care within the past 6 months, and

• Between ages 18 and 64

• English speaker OR is a person with Limited English Proficiency who speaks Spanish as his or her primary language

We will perform two rounds of cognitive testing, with twelve participants per language per round. Thus, we will test the survey with a total of 48 participants.

Respondent contact information will only be obtained for purposes of recruitment.

TYPE OF COLLECTION: (Check one)

[ ] Customer Comment Card/Complaint Form [ ] Customer Satisfaction Survey

[ ] Usability Testing (e.g., Website or Software) [ ] Small Discussion Group

[ ] Focus Group [X] Other: Cognitive Testing (e.g., review of paper-based materials and language)

CERTIFICATION:

I certify the following to be true:

1. The collection is voluntary.

2. The collection is low-burden for respondents and low-cost for the Federal Government.

3. The collection is non-controversial and does not raise issues of concern to other federal agencies.

4. The results are not intended to be disseminated to the public.

5. Information gathered will not be used for the purpose of substantially informing influential policy decisions.

6. The collection is targeted to the solicitation of opinions from respondents who have experience with the program or may have experience with the program in the future.

Name: ________________________________________________

To assist review, please provide answers to the following question:

Personally Identifiable Information:

1. Is personally identifiable information (PII) collected? [X] Yes [ ] No

2. If Yes, will any information that is collected be included in records that are subject to the Privacy Act of 1974? [ ] Yes [X] No

3. If Yes, has an up-to-date System of Records Notice (SORN) been published? [ ] Yes [X] No

Gifts or Payments:

Is an incentive (e.g., money or reimbursement of expenses, token of appreciation) provided to participants? [X] Yes [ ] No

Cognitive testing participants will receive $100 for their participation. This amount will help to recruit the desired range of respondents described above, by addressing the following challenges:

• Participants from the diverse socioeconomic backgrounds we are seeking may not otherwise be able to afford forgoing income during the time required to both participate in, and travel to and from, the cognitive testing interview.

• The cognitive testing will require participants to travel to a centrally located site and the stipend is intended to help compensate participants for their travel costs (e.g., train, bus, mileage, and parking).

• This may be a difficult-to-recruit study population, and the stipend may make it easier to attract hard-to-reach participants who face unanticipated barriers to participation (e.g., child care)

BURDEN HOURS

Category of Respondent	No. of Respondents	Participation Time (min.)	Burden (hr.)
Individuals (consumers)	48	60	48
Totals	48	60	48

FEDERAL COST:

The estimated annual cost to the Federal government is $113,128

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents

1. Do you have a customer list or something similar that defines the universe of potential respondents and do you have a sampling plan for selecting from this universe? [ ] Yes [X] No

If the answer is yes, please provide a description of both below (or attach the sampling plan)? If the answer is no, please provide a description of how you plan to identify your potential group of respondents and how you will select them?

See Attachment B, “Cognitive Testing Eligibility Screener.”

Administration of the Instrument

1. How will you collect the information? (Check all that apply)

[ ] Web-based or other forms of Social Media

[ ] Telephone

[X] In-person

[ ] Mail

[ ] Other, Explain

2. Will interviewers or facilitators be used? [X] Yes [ ] No

Please make sure that all instruments, instructions, and scripts are submitted with the request.

Attachment A –Draft Survey Instrument

Attachment B – Marketplace Dental Survey Cognitive Testing Eligibility Screener

Attachment C – Cognitive Testing Interview Information and Consent Form

Attachment D – Cognitive Interview Guide

Instructions for completing Request for Approval under the “Generic Clearance for the Collection of Routine Customer Feedback”

T ITLE OF INFORMATION COLLECTION: Provide the name of the collection that is the subject of the request. (e.g. Comment card for soliciting feedback on xxxx)

PURPOSE: Provide a brief description of the purpose of this collection and how it will be used. If this is part of a larger study or effort, please include this in your explanation.

DESCRIPTION OF RESPONDENTS: Provide a brief description of the targeted group or groups for this collection of information. These groups must have experience with the program.

TYPE OF COLLECTION: Check one box. If you are requesting approval of other instruments under the generic, you must complete a form for each instrument.

CERTIFICATION: Please read the certification carefully. If you incorrectly certify, the collection will be returned as improperly submitted or it will be disapproved.

Personally Identifiable Information: Provide answers to the questions. Note: Agencies should only collect PII to the extent necessary, and they should only retain PII for the period of time that is necessary to achieve a specific objective.

Gifts or Payments: If you answer yes to the question, please describe the incentive and provide a justification for the amount.

BURDEN HOURS:

Category of Respondents: Identify who you expect the respondents to be in terms of the following categories: (1) Individuals or Households; (2) Private Sector; (3) State, local, or tribal governments; or (4) Federal Government. Only one type of respondent can be selected per row.

No. of Respondents: Provide an estimate of the Number of Respondents.

Participation Time: Provide an estimate of the amount of time (in minutes) required for a respondent to participate (e.g. fill out a survey or participate in a focus group)

Burden: Provide the Annual burden hours: Multiply the Number of Respondents and the Participation Time then divide by 60.

FEDERAL COST: Provide an estimate of the annual cost to the Federal government.

If you are conducting a focus group, survey, or plan to employ statistical methods, please provide answers to the following questions:

The selection of your targeted respondents. Please provide a description of how you plan to identify your potential group of respondents and how you will select them. If the answer is yes, to the first question, you may provide the sampling plan in an attachment.

Administration of the Instrument: Identify how the information will be collected. More than one box may be checked. Indicate whether there will be interviewers (e.g. for surveys) or facilitators (e.g., for focus groups) used.

Submit all instruments, instructions, and scripts are submitted with the request.

1 Keller, S., Col Martin, G. C., Evensen, C. T., & Capt Mitton, R. H. (2009). The development and testing of a survey instrument for benchmarking dental plan performance: using insured patients’ experiences as a gauge of dental care quality. Journal of the American Dental Association (1939), 140(2), 229–237.

6