SS Part B -- Use of Open-Ended Responses to Explore Disparities in Patient Experience

SUPPORTING STATEMENT

Part B

Use of Open-Ended Responses to Explore Disparities in Patient Experience

Version: September 13, 2023

Agency of Healthcare Research and Quality (AHRQ)

Table of contents

B. Collections of Information Employing Statistical Methods 3

1. Respondent universe and sampling methods 3

2. Information Collection Procedures 4

3. Methods to Maximize Response Rates 5

4. Tests of Procedures 6

5. Statistical Consultants 6

B. Collections of Information Employing Statistical Methods

1. Respondent universe and sampling methods

The respondent universe for this pilot test will be members of the Ipsos KnowledgePanel (formerly Knowledge Networks) of online consumers. KnowledgePanel panel consists of over 50,000 adult panel members who are recruited by random-digit dialing (RDD) or by using address-based sampling.  Typical panel members receive 3-4 invitations per month to participate in research projects.  

KnowledgePanel is constructed to include those who do not otherwise have Internet access by providing them with free access in return for their participation on the panel and computer equipment, if not otherwise available. KnowledgePanel provides sampling weights, which when applied provide estimates representative of the adult U.S. population. Ipsos will randomly select 4,998 participants from its panel for the current study, with the constraints that they have had a doctor visit in the last 12 months (approximately 83% of U.S. adults; https://www.cdc.gov/nchs/fastats/physician-visits.htm), and that one sixth of the sample will be in each of the following groups: non-Hispanic Asian American, Native Hawaiian, or Other Pacific Islander; non-Hispanic Black; Spanish-speaking Hispanic; English-speaking Hispanic; non-Hispanic Multiracial; and non-Hispanic White. We will strive to recruit an equal mix of men and women within each of these six groups. The incidence rate for eligibility (having a recent physician visit) is estimated at about 75%. Response rates are based on Ipsos’ past experience with similar surveys.

Strata	Number of panelists expected to meet eligibility requirements	To be sampled	Anticipated response rate	Expected completes
Non-Hispanic Asian American/Pacific Islander	1,388*	1,388	60%	833
Non-Hispanic Black	4,682	1,666	50%	833
Spanish-speaking Hispanic	2,730	1,666	50%	833
English-speaking Hispanic	5,398	1,666	50%	833
Non-Hispanic Multiracial	1,388*	1,388	60%	833
Non-Hispanic White	34,984	1,388	60%	833
Total	50,570	9,162	55%	4,998

* Includes a small number of people (<150) to be drawn from Ipsos’s “opt-in” sample, which is separate from the standing KnowledgePanel. If the response rate is less than expected for these groups, then more “opt-in” participants will be sought as needed.

2. Information Collection Procedures

Subsamples will be drawn randomly from Ipsos’ KnowlegePanel, stratified as described in Section 1. The available KnowledgePanel sample for two strata (Non-Hispanic Asian American, Native Hawaiian, or Other Pacific Islander and Non-Hispanic Multiracial) may require additional sampling from Ipsos’ opt-in sample. This option will be used only if recruitment within the KnowledgePanel proves insufficient. This is a one-time data collection, so considerations for periodic data collection are not applicable.

Procedurally, participants will receive the study invitation presented in Attachment A. Participants will complete the CAHPS Clinician & Group (CG-CAHPS) 3.1 Survey, CG-CAHPS Narrative Item Set (NIS), and two supplemental questions regarding unfair or insensitive healthcare treatment (Attachment B) through a secure online connection. The survey will be fielded in English and Spanish based on respondent-preferred language. Attachment C presents the informed consent language all respondents will receive.

To achieve the goals of the experiment, the following analyses will be performed:

1. Qualitative coding of the narratives (open-ended responses) will be based on coding schemes developed in our prior research. The narratives will be coded for presence of key elements of patient experience (e.g., doctor communication, care coordination, access to needed care), as well as positive/negative valence. New codes will be developed to capture perceived unfair or insensitive healthcare treatment.

2. Between-group t-tests and Pearson chi-square tests of independence will be used to test for significant differences in the presence/valence of key content elements across the six demographic subsamples, as well as by gender, disability status, and language preference.

3. Within-group t-tests and Pearson chi-square tests of independence will be used to test for significant differences between NIS-derived content and closed-ended CG-CAHPS-derived content, within demographic subsamples and by gender, disability status, and language preference.

4. Automated coding of the narratives will be based on off-the-shelf Natural Language Processing (NLP) algorithms, implemented to predict a subset of human codes and validated against a separate subset of human codes.

5. Human-generated and machine-generated codes will be compared across a range of algorithmic equity criteria, such as equality of false positive rates or equality of positive predictive values across race/ethnicity, gender, disability status, or preferred language.

The coding procedures and analytic methods are based on past experience developing and testing similar procedures (Grob et al., 2016; Ish et al., 2020; Schlesinger et al, 2018).

Data imputation will not be used in this study. In a prior study (Schlesinger et al., 2018) using the same Ipsos survey panel, observed rate of missing data was less than 2% per question. Furthermore, imputation is inadvisable (and likely infeasible) for narrative data, and given that use of the closed-ended CAHPS survey responses is in comparison to narrative data, imputation could risk introduction of biases to the results.

For codes with an overall prevalence of 30% or greater, a sample size of 833 per group provides 80% power to detect a 6.5-to-6.85 percentage point difference in two-sided tests at p = 0.05.

3. Methods to Maximize Response Rates

The overall response rate is estimated to be between 50% and 60% based on results obtained from the past projects conducted by Ipsos. Procedures for maximizing response rates include:

• Field period of 4 to 8 weeks (if needed, the field period may be extended to ensure sufficient participation)

• Use of the Federal agency name in the email invitation

• Email reminders

4. Tests of Procedures

This study will use validated and standardized instruments, including the CG-CAHPS 3.1 Adult English Survey (https://www.ahrq.gov/cahps/surveys-guidance/cg/index.html) and the CG-CAHPS NIS supplemental item set (Schlesinger et al., 2020). The closed-ended question on unfair and insensitive treatment was developed by CMS for potential inclusion on the Medicare CAHPS survey. As such, these instruments do not require further pretesting.

Prior to fielding the main survey, the programmed online survey instrument may undergo pilot testing by 9 or fewer respondents to ensure programming and data output are functioning correctly.

5. Statistical Consultants

The survey, sampling approach, and data collection procedures were designed by the RAND Corporation under the leadership of:

Ron D Hays, PhD

Senior Policy Researcher

RAND Corporation

1776 Main Street

Santa Monica, CA 90407

Marc Elliot, PhD

Senior Principal Researcher

RAND Corporation

1776 Main Street

Santa Monica, CA 90407

Claude Setodji

Senior Statistician

RAND Corporation

4570 Fifth Avenue, Suite 600

Pittsburgh, PA 15213

References

Schlesinger, M., Grob, R., Shaller, D., Martino, S.C., Parker, A.M., Rybowski, L., Finucane, M.L., & Cerully, J.L. (2020). A rigorous approach to large-scale elicitation and analysis of patient narratives. Medical Care Research and Review, 77(5), 416-427.