SUPPORTING STATEMENT
Part B
Pretest of Proposed CAHPS® for Cancer Care Questions and Methodology
Submitted under AHRQ’s Generic Pretesting Clearance 0935-0124
Version: September 23, 2011
Agency of Healthcare Research and Quality (AHRQ)
Table of contents
B. Collections of Information Employing Statistical Methods 3
1. Respondent universe and sampling methods 3
2. Information Collection Procedures 7
Cancer care is delivered through many kinds of providers (e.g., medical oncologists, radiation oncologists, surgeons, therapists, nurses) and provider organizations (hospitals, hospital-owned outpatient clinics and ambulatory practices, independent ambulatory practices; regional and nationwide chains, and cancer centers, which are combinations of the previously listed organizations). The entities to be assessed by the CAHPS Cancer Care survey will be cancer centers. NCI estimates that about 85 percent of cancer care in the U.S. is delivered through community cancer centers. The best source of consistent and generally accepted information about cancer centers is the Commission on Cancer (CoC), which estimates that about 75 percent of cancer care is delivered through its 1,500 accredited cancer centers nationwide. For accreditation purposes, the CoC classifies cancer centers into the 12 categories in Exhibit 1.
Exhibit 1. Categories of CoC Accreditation
Source: http://www.facs.org/cancer/coc/categories.html#pcpc
The respondent universe will be eligible patients who received medical oncology, radiation oncology, or surgical oncology services at cancer centers from the four shaded cells in Exhibit 1. We will exclude the eight unshaded cells: VA cancer centers, pediatric cancer centers, centers that provide only one or two therapeutic modalities, and centers that treat fewer than 2,000 newly diagnosed cancer patients per year as determined by their tumor registries. The four shaded cells account for 89 percent of CoC’s accredited cancer centers1. Patients at selected centers will be eligible individually if they began medical, radiation, or surgical oncology services at the participating center within the three months preceding construction of the sampling frame. Patients who meet this criterion for more than one modality will be randomly assigned to one modality only. The intent of the three month period is to capture the patient’s most recent cancer treatment episode and allow enough experience with that episode for the patient to form opinions about his or her experience.
The statistical analysis has three main purposes: (1) to produce reliable quality assessments of cancer centers, (2) to compare the performance of the three main cancer treatment modalities to each other, and (3) to conduct a mode effects study. The cancer centers will be selected purposively. The Mayo Clinic will supply two of the 6 cancer centers to be recruited: Mayo Clinic Rochester and Mayo Clinic Jacksonville. The other four centers will be recruited from among the centers accredited by the CoC in the shaded categories in Exhibit 1. One will be located in California, because we have received supplemental funding from the California HealthCare Foundation.
Reliable Scores at the Cancer Center Level. The target number of completed questionnaires per center will be 300. Field testing of previous CAHPS surveys has demonstrated that the optimal number of responses per unit assessed is 300 if the unit is an organization, such as a health plan or hospital, in which the patient is likely to be served by a wide variety of individual physicians and other health professionals; for instance, the required sample size for the CAHPS Hospital Survey, the one most relevant to a cancer center survey, is 300.2 When the range of professionals working at the entity is narrow, the required sample size is smaller, because the inherent variance of patient reports within the unit is smaller; for instance, the required sample for the CAHPS Clinician and Group Survey is 45 per ambulatory care practice. Analysis of data from the field tests and subsequent implementations of the CAHPS Health Plan Survey and Hospital Survey has supported the initial recommendation of 300 per unit for health plans and hospitals. Because we are assessing cancer centers, which offer a wide variety of services delivered by various types of providers in both inpatient and ambulatory settings, and because cancer centers tend to be organized around a hospital, we assume that we will require approximately 300 completions per center to produce reliable center-level scores.
Comparison of Treatment Modalities. Discussions with stakeholders indicated that the main cancer treatment modalities are surgical, radiation, and medical (i.e., drug-based) oncology services and that each modality is delivered within a cancer center by separate teams. Thus, the survey will be most valuable to stakeholders if it produces scores for each modality, as well as for the center as a whole. Modality-specific scores will be valuable for use by providers to identify elements of the care process that will benefit from quality improvement interventions. Scores for the entire center are needed by patients to help them choose which cancer center they want to obtain care from. Thus, we will sample to obtain approximately one-third of the completed questionnaires (i.e., 100) from each of the three treatment modalities at each cancer center. This yields 600 observations per modality and 1,800 observations overall, given 6 cancer centers.
Mode Effects Study. We plan to conduct a mode experiment to compare telephone administration to mail with telephone prompting for non-respondents. The samples in each stratum at each center will be randomly distributed between the two modes to provide samples of about 1,600 observations in the mail mode and 200 observations in the telephone mode. We chose to allocate 200 observations to telephone because it is more expensive than mail and so we want to minimize the number of telephone cases while assuring that we have enough to detect a meaningful difference between the modes, with some margin of safety.
Exhibit 2 lists the minimum required sample size to obtain 80% power to detect large, medium, and small differences between the two modes. The power calculations, conducted using Optimal Design Software3, account for the multi-site clustering effect among the 6 cancer centers and are based on the following assumptions:
The outcome measure is a continuous variable (e.g., the composite score for each participant);
The standardized score difference between modes (e.g., mail vs. telephone) is either 0.8, 0.5 or 0.2, which represents large, medium, or small effect sizes, respectively;
The variance of effect size across the 6 cancer centers is 0.05; and
The proportion of variance explained by the cancer centers (the effect of clustering patients within centers) is 0.30.
Exhibit 2. Minimum Sample Size per Condition Required for 80% Statistical Power
Effect Size |
Sample Size per Condition |
|
|
With Multi-Site Clustering Effect |
Without Multi-Site Clustering Effect |
Large |
36 |
26 |
Medium |
120 |
64 |
Small |
Infinity |
390 |
If the clustering effect is taken into account, we will need at least 120 persons from each of the two modes to detect a medium sized difference between the telephone mode and mail mode. Similarly, to detect a medium sized difference between any two of the three treatment modalities (medical, radiation, and surgical), we will need at least 120 persons from each of the three types of services.
In order to achieve the primary purpose of the study—estimating quality scores for cancer centers—the study requires 300 observations for each of the 6 participating centers or a total of 1,800 observations. For the administrative mode effect study, we want to assign the minimum number of observations to the more expensive mode to control cost. According to the power calculation above, the minimum needed for the smaller mode to detect a meaningful difference between modes (i.e., a moderate effect size) is 120, but to be safe, we will allocate 200 to telephone in case our target response rates are not achieved. This allocation leaves the 1,600 residual observations in the mail mode.
This sample size also supports comparisons among the treatment modalities. The power analysis in Exhibit 2 indicates that 390 observations will be sufficient to detect a small difference between treatment modalities if clustering at the center level is negligible. Because we require 300 observations per center to achieve the main purpose of the study (reliable center-level scores), we will necessarily have 100 observations per treatment modality at each center and a total of 600 observations per treatment modality available to achieve the second purpose of the study—comparison of treatment modalities. Thus, 600 observations per treatment modality will enable us to detect moderate differences and possibly small differences if clustering is small. For pair-wise comparisons between cancer centers, clustering at the cancer center level is irrelevant, so the 300 persons per plan provides more than sufficient power to detect small-to-medium sized effects between centers.
Sampling Procedures. We assume that we will obtain the typical response rate for CAHPS surveys—40 percent4—so we will sample a total of 750 persons from each center (250 per treatment modality). At centers that have at least 250 eligible patients per modality at the time the sampling frame is established, we will draw a systematic random sample of 750 patients, ordered by type of cancer and gender within each modality. This will assure equal numbers of patients selected in each modality and proportional allocation across type of cancer and gender within each modality. Patients who are eligible for more than one modality will be randomly assigned to only one. At centers that do not have enough cases to fill out the entire sample at the start of the survey, we will use the systematic random sampling approach described above for however many patients are available when sampling begins and fill out the goal of 250 per modality on a flow basis during the 3-month data collection period. This approach is designed to yield an initial total sample of 4,500 patients resulting in 1,800 completions.
Although we have projected sample size and response rate conservatively, we have reason to believe that the response rate for the CAHPS Cancer Care Survey might exceed the typical CAHPS response rate. We are surveying patients in current or recent cancer care about the topic that is likely to be the most salient in their lives. Salience is positively associated with response. Several recent surveys of cancer patients on similar topics have obtained response rates that significantly exceed 40 percent. Hawley et al. had a 72.4 percent response rate for a survey about shared decision making.5 Chen at al. had a 64 percent response rate for a 90 minute telephone survey about treatment discussions between physicians and patients.6 Both studies used a sampling frame with patients who had Stage III cancer or less and we are including all stages of cancer. As a result, we are keeping the expected response rate at 40 percent as it is likely that sicker patients will be less likely to participate. Exhibit 3 describes the sampling plan.
Exhibit 3. Initial and final sample sizes
Data Collection Mode |
Completions per Treatment Center |
# Treatment Centers |
Total Completions |
Projected Completion Rate |
Initial Sample |
267 |
6 |
1,600 |
.40 |
4,000 |
|
Telephone |
33 |
6 |
200 |
.40 |
500 |
Total Survey |
300 |
6 |
1,800 |
.40 |
4,500 |
Data will be collected by the Mayo Clinic Survey Research Center. We will use two different modes: Mail with non-respondent telephone prompt and telephone, which will include these steps:
Mail arm
1st Mailing: A cover letter, survey, and return envelope will be mailed to potential respondents.
2nd Mailing: A reminder letter, survey & return envelope will be sent to non-responders 2 weeks after initial mailing.
Telephone follow-up reminders: We will make up to three reminder calls and send a new survey as needed to non-respondents.
Telephone arm
Send cover letter. We will send a letter to potential respondents alerting them to the survey.
Two weeks after the cover letter is sent, we will make up to 6 calls to contact the person and complete the survey on the telephone. Calls will be made twice during daytime on weekdays, twice during nighttime on weekdays, and twice during the day on weekends. Per the CAHPS guidelines, we will spread these 6 calls over different weeks.
At 40 percent, the target response rate is low. However, 40 percent is the typical rate obtained in CAHPS surveys and has been approved by OMB many times. The small body of research on nonresponse in CAHPS surveys has found little nonresponse bias in CAHPS scores when comparing one unit of assessment to another, which is the intended purpose of the CAHPS Cancer Care Survey.7
For incorrect addresses or telephone numbers, we will use available patient data, including the last four digits of the date of birth, and name to track with Accurint®. We will also determine the vital status of non-respondents using the Social Security Death Index.
The purpose of the mode effects experiment is to determine if some modes produce higher response rates than others, so that we can recommend the most effective modes in the final survey design kit.
We conducted two sets of cognitive interviews with cancer patients to evaluate the comprehension and value of the items to potential respondents. The first set included 9 interviews and covered the entire questionnaire. The second set included 9 interviews and covered the changes made as a result of the first set.
In addition, we plan to conduct the following studies with the field test data.
Psychometric analyses of individual items, based on classical test theory, that will assess missing data, item distributions, and the reliability and validity of the items included in the analyses.
The domain structure of the survey will be evaluated by examining item-scale correlations and factor analyses. Items that cluster together with high reliabilities (i.e., Cronbach’s alpha > .70) will be considered as potential composites for reporting survey results.
Case-mix adjustment analyses will be performed to determine if composites need to be adjusted for patient characteristics, data collection mode, type of cancer, stage of cancer, and type of treatment. Because evaluating care for a specific condition is a new area for CAHPS® developmental research, it is not clear what case mix adjusters will be important. One of the purposes of the field test will be to explore the usefulness of these variables and possibly other variables for the final survey.
We will assess the effectiveness of survey operations using reports from the Mayo Clinic SRC staff about problems and solutions encountered during the survey.
The following people have been consulted on statistical aspects of the design:
Steven A. Garfinkel, American Institutes for Research (AIR). (919) 918-2319.
Roger E. Levine, AIR, (650) 843-8160
San Keller, AIR (919) 918-2309
Manshu Yang, AIR, (919) 918-2312
Timothy Beebe, Mayo Clinic, (507) 538-4606
Kathleen Yost, Mayo Clinic, (507) 538-3894
Jeff Sloan, Mayo Clinic, (507) 261-4268
1 American College of Surgeons. (2010). 2007 percentage distribution of cancer programs by category. Retrieved on August 30, 2011 from http://www.facs.org/cancer/coc/approvalsdemographics.pdf
2 Elliot, MN, WG Lehrman, E Goldstein, K. Hambarsoomian, MK Beckett, LA Giordano. Do Hospitals Rank Differently on HCAHPS for Different Patient Subgroups? Med Care Res Rev February 2010 67: 56-73.
3 Liu, X., Spybrook, J., Congdon, R., Martinez, A., & Raudenbush, S.W., (2009). Optimal
Design for Longitudinal and Multilevel Research V.2.0
4 Gallagher, Patricia M. and Floyd J. Fowler, Jr. “Size Doesn’t Matter: Response Rates of Medicaid Enrollees to Questionnaires of Various Lengths.” Presentation delivered at the 4th National CAHPS User Group Meeting, Baltimore, MD, 1998.
5 Hawley, S. T., Griggs, J. J., Hamilton, A. S., Graff, J. J., Janz, N. K, Morrow, M., Jagsi, R., Salem, B., Katz, S., J. (2009).Decision involvement and receipt of mastectomy among racially and ethnically diverse breast cancer patients. Journal of the National Cancer Institute; 101, 1337–1347.
6 Chen, J. Y., Tao, M.L., Tisnado, D, Malin, J., Ko, C., Timmer, M., Adams, J. L., Ganz, P. A., Kahn, K. L.
(2008). Medical Care, Impact of Physician-Patient Discussions on Patient Satisfaction 46 (11):1157-1162
7 Elliott MN, Edwards C, Angeles J, Hambarsoomians K, Hays RD. (2005). Patterns of unit and item nonresponse in the CAHPS Hospital Survey. Health Services Research; 40 (6 Pt 2): 2096-119.
-
| File Type | application/msword |
| File Title | SUPPORTING STATEMENT B |
| Subject | OMB Application |
| Author | AIR & Mayo |
| Last Modified By | DHHS |
| File Modified | 2011-09-27 |
| File Created | 2011-09-27 |