Evaluating Coverage to Care in Communities
Supporting Statement Part B for OMB Reinstatement
(CMS-10632; OMB 0938-1342)
B.1. Respondent Universe and Sampling Methods
This section provides a description of the respondent universes for the consumer and partner survey studies, how we plan to identify study participant samples from these universes, and the data collection procedures for these samples. No specialized sampling procedures will be required to obtain these samples.
B.1.1. Overview of Respondent Universe, Study Population, and Expected Response Rates
The two proposed datasets will be collected through cross-sectional, online surveys. The universe of participants for these data collection efforts are (1) organizations whose mission is (at least in part) connecting individuals with health care services in the greater metropolitan area of three to four cities (combined statistical areas of Chicago, Houston/Katy, Washington DC, and possibly Phoenix) and (2) health care consumers enrolled in Medicaid, Medicare, or a Marketplace health insurance plan. These cities were selected due to these locations having both (a) a high saturation of C2C materials, based on requests, and (b) an adequate sample size of individuals in the Ipsos KnowledgePanel who were known to be enrolled in Medicare, Medicaid, or a Marketplace plan. All survey efforts would first proceed in all areas except Phoenix, and we will field these surveys in Phoenix if needed to achieve the proposed quotas.
Exhibit 1. |
Respondent Universe, Study Population, and Expected Response Rate for Consumer and Partner Surveys.
|
Consumer Survey |
Partner Survey |
Respondent Universe |
Adult KnowledgePanel members in 3 geographic areas who receive CMS services (Medicare, Medicaid, or Marketplace plan)
|
Key Informants of Partner Organizations (one per organization) in 3 geographic areas placing C2C product orders between 9/1/2019 and study launch, and similar organization key informants nominated by those placing orders
|
Study Population |
Random selection of 667 individuals invited to obtain a target stratified sample of 400 individuals (200 exposed and 200 not exposed to C2C)
|
Respondent universe, which is estimated to be 118 (number of C2C partner organizations in geographic areas x 2) |
Anticipated Response Rate |
60% |
15–20% |
The respondent universe for the partner survey has two groups: (1) partners ordering C2C materials and (2) similar nominated organizations not ordering C2C materials. The partner survey will first be conducted with organizations who have placed C2C product orders between the federal fiscal years of 2020 and 2021. At the end of the survey with these organizations, participants will be asked for nominations of similar organizations in their city. The purpose of this is to obtain a sample of similar organizations who have not been exposed to C2C, as this can be useful for informing efforts to expand the reach of the initiative.
CMS product order data will be used to identify those ordering C2C materials in the four selected cities. The person ordering these materials will be the initial point of contact for conducting the survey. Using data available to date, 59 distinct organizations have placed orders across these four cities, where an average of 330 (SD=820) materials have been requested per organization. We anticipate collecting data from an equal number of nominated (or snowball sample) organizations that have not requested C2C materials. These organizations will be contacted by telephone or email to identify the appropriate point of contact.
The survey has been limited to 20 minutes (average), an estimate that includes five minutes for review of the study purpose and identification of the most appropriate person within the organization to complete the survey. Participants responding to the survey invitation will be asked to indicate if they are “knowledgeable about resources that aim to help people with new health care coverage understand their benefits and connect to primary care.” If they respond that they are not, they will be asked to provide the name and email address of a person in their organization who meets these criteria. These alternate contacts will then be emailed an invitation. Based on similar online cross-sectional surveys of organizations, we anticipate that the response rate for this survey will be between 15% and 20%. All individuals willing to participate will be interviewed.
Ipsos will administer the consumer survey using the KnowledgePanel survey panel. The KnowledgePanel is a probability-based sample of survey participants maintained by Ipsos, which can be enlisted to produce sample-based estimates of the U.S., large geographies in the U.S., and specific demographic subgroups (Ipsos, 2019). The panel contains 55,000 members ages 13 and older, selected through random-digit dialing or address-based sampling. Panel members participate in surveys to receive points that can be exchanged for products or cash. They participate in two surveys per month on average and are restricted to a maximum of one survey per week. Of note, the KnowledgePanel has a sample of individuals that would not typically have internet access, where these individuals are provided with a tablet and a data plan. Panel members complete regular topical surveys on a rolling basis, so data have already been collected from this panel on topics relevant to the present study, including health, health care utilization, and demographics. These existing data will be leveraged to (a) reduce survey burden and (b) obtain a sample known to be enrolled in a Medicare, Medicaid, or a Marketplace plan. The survey is anticipated to take 20 minutes to complete.
We wish to obtain a final sample of 200 adult individuals who have been exposed to C2C materials and 200 adult individuals who have not been exposed to C2C materials. We anticipate that 60% of individuals invited will participate in the survey, so 667 will initially be invited to participate in the survey to obtain the desired sample size of 400.
B.1.2. Statistical Methodology for Stratification and Sample Selection and Degree of Accuracy Needed
Statistical methodology for stratification and sample selection
Unfortunately, there is insufficient information to establish a sampling frame of partner organizations for two reasons. First, the type or size of the organization requesting information is not recorded in the product order data, as it is not relevant for the purpose of fulfilling requests for materials. These organizations are expected to be variable in their organization type (e.g., health providers, navigator programs, faith-based organizations, community-based non-profits) and size, which likely have an impact on the success of outreach efforts with C2C materials. More importantly for this study, the sampling frame should contain all organizations engaged in connecting individuals with health care services, even if they have not requested C2C materials, which is difficult to determine at best. There is not a clear way to identify the missions of all organizations in a city, which would be necessary to find all organizations not using C2C materials for which we have no information. Thus, a sampling frame based on what is currently known would be insufficient and could possibly be problematic (e.g., inaccurate estimates of the proportion of organizations falling into each type). While we realize that not being able to fully enumerate the sampling frame introduces potential biases in the generalizability of our findings, we feel this limitation is outweighed by the benefit of understanding more about the types and size of organizations that are responsible for distributing C2C materials in high saturation areas, as well as the characteristics of similar organizations that are not distributing C2C materials in high saturation areas.
Degree of accuracy needed
The partner study is designed to be descriptive in nature and largely concerned with why (or why not) organizations used C2C materials, what organizations did with the materials, and what materials were the most helpful in educating consumers about health care. Due to the small sample size anticipated for this study (n=30 for both organizations exposed and not exposed to C2C), we lack the statistical power to identify differences or relationships in this sample. More specifically, descriptive statistics and confidence intervals will be used to provide a sense of the differences in subgroups; however, we cannot get a sense of whether differences and relationships observed are different than what we would expect by chance alone. Nonetheless, we will attempt to obtain the universe of organizations requesting C2C materials in a city, so estimates should be representative of cities with a high saturation of C2C materials, assuming non-response bias is minimal. Those not exposed to C2C are being collected as a counterfactual group and are interesting inasmuch as they can inform why some organizations use C2C materials, relative to those who do not.
The proposed approach has the strengths of allowing us to examine the needs of partner organizations and whether C2C materials are meeting those needs. The proposed methods will also highlight why some organizations that could benefit from C2C materials are not familiar with or not using C2C materials. The primary weakness of the proposed approach is that small sample sizes make it difficult to (1) examine whether differences are larger than what we would expect by chance (or statistically significant) and (2) whether these differences are generalizable to organizations across the U.S. These potential weaknesses are mitigated by the approaches used and the study context: (a) confidence intervals and non-parametric statistical tests will be used to examine whether the data are suggestive of relationships and difference; (b) there are a relatively small number of organizations in any geography that could be selected, reducing the sampling frame; and (c) by selecting more than one geography, we reduce the likelihood that findings are idiosyncratic to selected geographies.
Statistical methodology for stratification and sample selection
The primary goal of this study is to (a) examine relationship among those who use C2C services and (b) examine differences between those who do and do not use C2C services. Thus, the sample size is guided more by the sample size needed for statistical significance, as opposed to the sample size needed for parameter estimation. While the KnowledgePanel is a well-designed probability-based sample of the U.S., we are not using the KnowledgePanel for this reason. Using the KnowledgePanel serves as an efficient method to have a pre-screened sample of those enrolled in Medicaid, Medicare, or a Marketplace plan. We will also equally allocate the sample across those exposed and those not exposed, which will yield more statistical power.
Exhibit 2. |
Estimation of Sample Sized Needed to Detect Meaningful Relationships and Differences.
Analysis Type |
Correlational Analyses |
Comparative Analyses |
Effect Size Measure |
r |
d |
Effect Size |
.20 |
.35 |
Type 1 Error Rate |
5% |
5% |
Type 2 Error Rate |
20% |
20% |
Per Arm N Needed |
198 |
129 |
Total N Needed (both arms) |
396 |
258 |
Degree of Accuracy Needed
The inferential analyses to be performed on the consumer data fall into two primary categories:
correlational analyses performed separately for consumers exposed and not exposed to C2C materials and comparative analyses comparing those exposed to those not exposed. We calculated the sample sizes necessary for a zero-order correlation and for an independent groups t-test, when assuming a low type 1 error rate of concluding a difference/relationship exists when it does not (5%, two-tailed) and a conventional type 2 error rate (20%, or 80% power) for missing a difference/relationship when one exists in reality. As we lacked suitable parameter estimates for the magnitude of relationships we would likely find, we assumed small-to-medium magnitude relationships for the correlational analysis (r=.20) and small-to-medium magnitude differences for the comparative analysis (d=.35 or a difference of .35 standard deviation units; Cohen, 1988). While these are somewhat arbitrary estimates of magnitude, they do likely represent relationships that are substantively meaningful, in additional to the statistical significance decisions being considered in our calculations. Given these parameters, a sample of 198 individuals would be needed in each of the exposed and not exposed groups (or 198 x 2 = 396 total) for the correlational analysis and 129 individuals would be needed per arm in each of the exposed and not exposed groups (or 258 total) for the comparative analysis. As such, we are proposing to obtain a final sample (after non-response) of 400 individuals (200 exposed and 200 not exposed) from three to four cities with a high saturation of C2C materials for our sample.
The proposed approach has the strengths of allowing us to examine the attitudinal and cognitive outcomes
(e.g., health care literacy) and behavioral outcomes (e.g., preventive health care, chronic disease management) of those exposed to C2C messaging relative to those who are not exposed. It also allows for us to examine which C2C materials have the biggest impact on outcomes and differences in direct exposure to C2C messaging relative to exposure through health care partners. While the proposed sample is not designed to produce an estimate of the proportion of those exposed to C2C in the U.S. as a whole, the consequences of this weakness are potentially mitigated by (a) dashboard data already being readily available that speak to the number of C2C materials circulating in the U.S. and (b) the data we are proposing to collect being representative of areas with high C2C saturation; information from the prior RAND survey could perhaps be used to augment the information collected in this survey to provide an approximation of actual C2C reach in the U.S.
This section describes the proposed data collection procedures for the partner and consumer online surveys. Relevant estimation procedures are also discussed. As this is a one-time data collection effort, we have not considered periodic data collection cycles.
B.2.1. Data Collection Procedure
Partner surveys will primarily be collected using an online survey programmed in Qualtrics (see Appendix A for survey). Participants will primarily be recruited for participation in the survey through email. The email will indicate the rationale for the study and an invitation to participate. Depending on the origin of the sample (i.e., organizations ordering C2C materials or other similar organizations nominated), participants will be told that they are being contacted due to a recent order of C2C materials or because their organization helps connect consumers with health care services. A letter of support from CMS, signed by the Director of the CMS Office of Minority Health, will be included in the email to encourage participation. Individuals who do not respond to the invitation or who do not recommend an alternate contact will be sent a follow-up e-mail reminder at one and two weeks following the initial invitation. A third reminder will be given through a telephone call three weeks following the invitation. Participants will be given the option to complete the survey over the telephone if desired. Final email reminders will be sent at 6 and 8 weeks after the initial invitation. This survey will be fielded for a period of three months.
Ipsos will use their standard survey protocols for implementing the online survey of consumers (see Appendix B for survey) with the KnowledgePanel. Ipsos will determine eligible participants and make a random selection of 667 potential participants, who will be invited to participate in the survey. When invited, potential participants will be notified by email with a password-protected link to the survey. Participants will not be asked the demographic, health, and health care questions in Appendix B, as these data have already been collection previously. Participants can complete the survey at their leisure, and they can refuse participation. Non-responding participants will be sent a reminder email three days after the initial contact. Automated phone calls reminders can also be used if there is a low level of survey response. As soon as the sample size quotas are reached (i.e., 200 C2C exposed and 200 C2C not exposed), the survey will be marked as a complete and no further responses will be accepted. Many KnowledgePanel surveys are fielded within a week’s time, but the survey will remain open as long as it takes to reach the desired sample size.
B.2.2. Estimation Procedure
Section A.16 provides a detailed discussion of the quantitative methods we will be using to analyze the data. As discussed in the foregoing, the present project is primarily concerned with being able to detect reliable differences and relationships, as opposed to providing point estimates. Survey non-response, especially differential survey non-response as a function of C2C exposure, poses one alternative explanation for putative study findings. As discussed in section A.16, a Heckman selectivity model will be used to (a) identify potential sources of bias in self-selection and (b) statistically adjust comparisons and relationships as necessary.
Several strategies will be used to help improve response rates for the consumer and partner surveys. First, participants will be given incentives to participate in the study. More specifically, participants in the partner survey will be offered a $35 gift card for completing the survey, and participants in the consumer survey will be offered approximately $5 in points for completing the survey. That is, KnowledgePanel participants earn points for completing surveys that they can exchange for cash, merchandise, gift cards, or game entries. Second, participants in both surveys will be using web survey platforms that allow them to easily complete the survey on different devices (e.g., tablets, smart phones, computers), which should improve response rates. Third, as noted above, the online survey systems used will send out follow-up reminders to participants who have not yet completed the survey. Fourth, we will use survey materials that are easy to read and clearly communicate the purpose and need for the survey. Moreover, for the partner survey, we will be including a letter from the Director of the CMS Office of Minority Health, which underscores the importance and legitimacy of the survey. While we are taking these steps to ensure a low level of survey non-response, both online survey systems allow us to track the number of individuals invited to participate and the number of participants who do participate through unique identifiers. All reporting of the data collected will provide responses rates as the number of individuals participating divided by the number of eligible participants.
Direct pre-testing was not conducted for either the partner or consumer surveys for several reasons. The primary reason is that most of the proposed survey measures have been used in RAND’s previously approved study under OMB 0938-1342. Thus, these measures have been fielded successfully in the past. There are several secondary reasons for not having direct pre-testing of the surveys. The consumer questionnaire mostly consists of validated survey instruments from the literature, where the response burden can be assessed directly from extant data. Both survey delivery systems (Qualtrics and Ipsos KnowledgePanel) have a proven record of reliably collecting data over the internet. Qualtrics has a proven track record of delivering high quality surveys for close to twenty years and acknowledges that they have been contracted by approximately 11,000 companies as their survey solution. Similarly, Ipsos, the parent company acquiring the KnowledgePanel, has been fielding research studies for nearly a half of a century and has offices in 89 countries. Several institutional review boards have approved the procedures used by the KnowledgePanel (e.g. Cornell University, Harvard University, Princeton University, Rutgers University, and University of Maryland, among others), as well as previous studies receiving U.S. Office of Management and Budget approval. Finally, all surveys to be fielded have been reviewed by our research team with close to 100 years combined experience in fielding surveys.
The data for this study is being collected by Ketchum, Inc. and Pacific Institute for Research and
Evaluation (PIRE) on behalf of the Centers for Medicare & Medicaid Services, Office of Minority Health. With CMS oversight, Ketchum/PIRE is responsible for the study design, data collection, analysis, and report preparation. Key input to the statistical aspects of the design was received from the following individuals:
Suzanne Niemeyer, Project Director;
William Scarbrough, PIRE Subcontract Manager and Senior Research Scientist;
Steve Shamblen, Research Scientist;
Andrew Gluck, Program Director;
Kathy Atwood, Senior Research Scientist;
April Schweinhart, Research Scientist;
Aree Sangpukdee, Associate Program Evaluator.
CMS OMH Staff, including Director LaShawn McIver, Deputy Director Wanda Finch, Subject Matter Expert Ashley Peddicord-Austin, and COR Scott Yeager, have overseen the design process.
Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum.
Ipsos (2019). Panel book: KnowledgePanel® brings you a probability-based online sample. Downloaded 11/11/2020 from https://www.ipsos.com/sites/default/files/june-2020_kp_book-1.pdf.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Evaluating Coverage to Care in Communities Supporting Statement Part B for OMB Reinstatement |
| Subject | Evaluating Coverage to Care in Communities Supporting Statement Part B for OMB Reinstatement |
| Author | HHS/CMS |
| File Modified | 0000-00-00 |
| File Created | 2021-05-08 |