PCMH Items Demonstration_OMB Statement B_05152017_FINAL

Supporting Statement

Part B

CAHPS PCMH Items Demonstration Study

Version January 11May 15, 2017

Agency for Healthcare Research and Quality (AHRQ)

TABLE OF ContentS

SUPPORTING STATEMENT: CAHPS PCMH ITEMS DEMONSTRATION STUDY 1

B. Collection of Information Employing Statistical Methods 1

B1. Respondent Universe, Sampling and Respondent Selection 1

B2. Data Collection Procedures 3

B3. Response Rates and Non-Response 4

B4. Tests of Procedures or Methods 4

B5. Statistical and Data Collection Consultants 5

SUPPORTING STATEMENT:
CAHPS PCMH ITEMS DEMONSTRATION STUDY

B. Collection of Information Employing Statistical Methods

B1. Respondent Universe, Sampling and Respondent Selection

Data collection will begin in 2017.

Selecting and Recruiting Practices

The study will sample from the universe of primary care practices that have submitted and received NCQA PCMH Recognition (currently and in the past). The study design will have three arms: current CAHPS Patient Experience Distinction (e.g., current user of CAHPS PCMH items), past CAHPS Patient Experience Distinction (e.g., past user of CAHPS PCMH items), and PCMH Recognition only (control) practices (e.g., are not administering the CAHPS PCMH items as part of their patient experience survey). All practices in the study across all 3 arms have NCQA PCMH Recognition. For each study arm, the sample design will stratify by three factors: location, PMCH history, and number of physicians, to allow for comparison of survey responses across these characteristics. The sample frame consists of a universe of more than 13,000 primary care practices. We will pull a sample of 320 practices from this universe with the expectation of achieving a final set of 150 practices. This assumes a response rate of 55% for the first contact (e.g. interview) and an 85% response rate for the second contact (e.g. PCMH assessment), based on prior work with recruiting physicians/clinical leaders (Blendon et al., 2004; Weissman et al., 2005) and based on the pretest experience of these same procedures (including the honorarium for instrument completion) and instruments that achieved a 38% response rate (e.g. 42% (10/24) response rate for the interviews and 90% (9/10) response rate for the PCMH assessments).

There are 13,191 practices in the NCQA PCMH Directory of practices ever submitting and receiving NCQA PCMH Recognition from 2008 through October 2016. We will select a sample of 320 practices with the expectation of achieving approximately 150 included practices (with both a completed interview and assessment). Of the practices contained in the NCQA directory data set, we will exclude a total of 3,449 practices that fit into at least one of the following categories: did not have a current PCMH status (1,930), were pediatric only practices (1,603), are located in Puerto Rico, or provided care to the Armed forces (58). Of the adult primary care practices with a current PCMH recognition, 300 had a current NCQA CAHPS Patient Experience Distinction and 242 had a past NCQA CAHPS Patient Experience Distinction.  We will sample a total of 320 practices (128 current PCMH recognition, 86 past PCMH recognition, and 106 control practices) proportionate to size from 5 strata defined by location (Northeast NCQA Initiative States (NY/VT), other Northeast states, South, West, and Midwest) as depicted in Table 1, to achieve an anticipated sample size of approximately 150 practices: 60 with current CAHPS Patient Experience Distinction, 40 with past CAHPS Patient Experience Distinction and 50 control practices (e.g. NCQA PCMH Recognition only). Our sample size targets are designed to yield enough completes to support the comparisons of interest.

Table 1: Sample Allocation for 320 Adult Primary Care Practices by Study Arm for Participation in CAHPS PCMH Items Demonstration Study

Sampling Queues	Current CAHPS Distinction		Past CAHPS Distinction		No CAHPS Distinction/ NCQA Recognition only (Control)
	Sample (128 of 300)	Targets (60)	Sample (86 of 242)	Targets (40)	Sample (106 of 9,200)	Targets (50)
Midwest	6	3	19	9	21	10
Initiative States (NY/VT) in Northeast	40	19	37	17	13	6
Other Northeast States	67	31	13	6	19	9
South	9	4	11	5	38	18
West	6	3	6	3	15	7

Note: Each cell is based on a 47% response rate except for the reallocation of two cases to account for the effect of rounding on the overall totals.

To increase variation in practice PCMH history and increase representation for practices with more physicians, sampling will be disproportionate for the control practices within each of the five geographic regions. In particular, practices with 4 or more, 3, or 2 physicians will be sampled at 4, 3, and 2 times the rates, respectively, of those with 1 physician. Practices in the lowest (Level 1 and 2) and highest (Level 3 for >5 years) categories of PCMH history will be sampled at twice the rate of practices with intermediate PCMH history (Level 3 for no more than 5 years). Table 2 shows how these two factors were combined to produce sampling weights. For example, a practice with PCMH Level 3: >5 years and 4 or more physicians has 8 times the chance of being selected as a practice with one physician and PCMH Level 3: for no more than 5 years. The design weights will be adjusted later for differential non-response rates.

Table 2: Weights for 106 Adult Primary Care Practices with a current NCQA PCMH Recognition only by Strata for Participation in CAHPS PCMH Items Demonstration Study

Weights Strata	PCMH Level 1 or 2	PCMH Level 3: less than 3 years	PCMH Level 3: 3-5 years	PCMH Level 3: >5 years
1 Physician listed in NCQA PCMH Directory	2	1	1	2
2 Physicians listed in NCQA PCMH Directory	4	2	2	4
3 Physicians listed in NCQA PCMH Directory	6	3	3	6
4 or more Physicians listed in NCQA PCMH Directory	8	4	4	8

Primary analyses will consist of descriptive statistics and comparisons of CAHPS patient experience and CAHPS PCMH measures across the three study arms of current, past, or no CAHPS Patient Experience distinction groups. We will have 80% power (with 2-tailed tests and alpha=. 05) to detect medium-to-large effect sizes, with an estimated design effect for weighting of 1.44 applicable to planned comparisons. Specifically, for comparisons between any two CAHPS groups (e.g., current distinction vs. past distinction, past distinction vs. no distinction), we will have 80% power to detect differences of 0.65-0.71 standard deviations (Cohen’s d; medium effect size = 0.5 SD, large effect size 0.8 SD) for continuous variables. For dichotomous variables, independent sample differences of 19.2 to 35.7 percentage points can be detected if prevalences range from 10-90%. We are making between-subjects comparisons of independent groups, using the corresponding tests. We will use continuous measures, where possible. Some analyses with dichotomous variables will be exploratory. For descriptive statistics, 95% margins of error will be +/- 0.28 to 0.34 standard deviations for continuous measures and +/- 9.1% to 18.6% for dichotomous measures.

B2. Data Collection Procedures

Data collection staff will contact each sampled practice using the contact information provided in the NCQA PCMH directory to confirm the practice remains active, and to confirm name, mailing address, fax, and telephone number of the listed physicians (or identified PCMH leader) using a standardized protocol. This will allow us to personalize the cover letter for the participation package. Confirmation of the contact information to send out the first wave of participation packages will last 3-4 weeks, after which data collection will ensue. Potential interviewees will receive a maximum of three invitations to participate in the study. These invitations will be sent via fax and Fed Ex and will contain sufficient information for informed consent as well as how to schedule an interview. We anticipate closing the interview field period after 12 weeks of data collection. Physicians/PCMH leaders who complete an interview will receive a post-paid $75 honorarium. All interviewees who return a PCMH assessment form will receive an additional $75 via post-paid incentive. Once interviews are completed, we will work with each participating practice to obtain a completed PCMH-A assessment tool and their CAHPS data files, if relevant. We anticipate closing the data file submission period 8-12 weeks after closing the interview fielding period.

Throughout data collection, we will track response and cooperation within each sample stratum and employ additional efforts or sample to achieve sufficient response in each stratum.

We anticipate the procedures outlined above will yield a minimum of 150 participating practices out of 320 sampled practices (47% response rate).

B3. Response Rates and Non-Response

Published surveys of physicians and practice leaders conducted in the past 10 years report response rates as low as 20% and as high as 63% (Blendon et al., 2004; Weissman et al., 2005). This study includes two main overlapping data collection requests. Due to the multiple requests, we estimate a minimum response rate of 47%, using the response rate from the literature combined for the initial contact and a 85% response rate for the second contact (e.g. 55% X 85% = 47%). In addition, the collection of assessments of organizational change have experienced a trend of overall decline in response rates similar to surveys of general populations (Cycota and Harrison, 2006; Baruch and Holtam, 2008). Thus, we conservatively estimate a minimum response rate of 47% and about 150 participating practices for which we will have obtained a physician/PCMH leader interview and an assessment tool. Our pilot test data of these processes yielded a 38% response rate (42% for first contact (e.g. interviews) and 90% for second contact (e.g. assessment)).

As described in Section B2 above, we plan to maximize response rates for each practice through:

• Careful identification of the appropriate respondent during recruitment (physician or PCMH leader),

• Use of personalization,

• Multiple contact attempts,

• Multiple modes of PCMH assessment data collection,

• Alternative modes for non-response contacts, and

• Use of a post-paid incentive ($75 for physician/PCMH leader interview and $75 for PCMH assessment)

To assess the possibility of physician response bias, we will analyze the demographic characteristics of the practices with responding and non-responding interviewees. This analysis will use data collected from the office manager interviews during the recruitment and scheduling process. We will analyze whether responding and non-responding interviewee practices differ based on practice characteristics. We will test whether there are significant differences between responding and non-responding practices by type (adult only vs. adult and child), practice size (based on number of primary care physicians), specialties offered at the practice, presence of an in-house pharmacy, access to a clinical pharmacist, extended office hours, presence of urgent care, and ownership of the practice.

B4. Tests of Procedures or Methods

As part of the pilot testing, a range of adult primary care practice types (location, PCMH history, and CAHPS Patient Experience Distinction type) were selected to capture variation in the expected range of responses. Testing informed the content of the survey, respondent identification procedures, and provided a basis for estimating administration burden. We contacted 24 practices in the pilot test and completed data collection of all instruments (office manager interview, physician/PCMH leader interview, and PCMH assessment) for 9 practices. Relevant testing of another study of the same population of respondents informed decisions about use of an incentive to enhance response.

B5. Statistical and Data Collection Consultants

The survey, sampling approach and data collection procedures were designed by the RAND Corporation under the leadership of:

Marc Elliot, PhD

RAND Corporation

1776 Main Street

Santa Monica, CA 90407

Denise D. Quigley, PhD

RAND Corporation

1776 Main Street

Santa Monica, CA 90407

Megan Mathews, M.A.

RAND Corporation

1776 Main Street

Santa Monica, CA 90407

References

Blendon, R.J., Schoen, C., DesRoches, C.M., Osborn, R., Zapert, K., & Raleigh, E. (2004).  Confronting Competing Demands to Improve Quality:  A Five-Country Hospital Survey.  Health Affairs, 23(3), 119-135.

 

Cycota, C.S. & Harrison, D.A. (2006).  What (Not) to Expect When Surveying Executives:  A Meta-Analysis of Top Manager Response Rates and Techniques Over Time.  Organizational Research Methods, 9, 133-160.

 

Baruch, Y. & Holton, B.C. (2008).  Survey Response Rate Levels and Trends in Organizational Research.  Human Relations, 61, 1139-1160.

 

Weissman, J. S., Annas, C. L., Epstein, A. M., Schneider, E. C., Clarridge, B., Kirle, L., Gatsonis, C., Feibelmann, S. & Ridley, N. (2005). Error reporting and disclosure systems: views from hospital leaders. JAMA, 293(11), 1359-1366.

2