Field Test Report 5/3/2011

Development of and Field-Test Results for the CAHPS PCMH Survey
The National Committee for Quality Assurance (NCQA) field tested a new version of the Consumer
Assessment of Healthcare Providers and Services Clinician & Group Survey (CG-CAHPS)
specifically designed to evaluate Patient-Centered Medical Homes (PCMH). This version of the
Clinician & Group Survey incorporates new items to address domains of care identified through a
multi-stakeholder input as critical for evaluating the functioning of PCMH practices. This report
summarizes the rationale for using patient experiences in the evaluation of medical homes, efforts
to prioritize domains for incorporation in the survey, and the decision to build on the existing
CAHPS survey. In addition, we summarize results of the field test and recommendations from the
CAHPS survey team for the PCMH Clinician & Group Survey.
BACKGROUND AND SIGNIFICANCE
The Patient-Centered Medical Home (PCMH) continues to gain momentum as a model for
improving delivery of primary care. In many states, practices are eligible for financial incentives for
adopting the medical home model, and the federal government has joined multi-payor
demonstrations and is supporting efforts to deploy the model in community health centers. NCQA’s
Patient-Centered Medical Home (PCMH) recognition program is the most widely used method for
qualifying practices for rewards in multi-payor demonstrations. Over 2,000 practices representing
more than 10,000 physicians have achieved recognition. NCQA recently released updated
standards (PCMH 2011).
Giving more prominence to patient engagement was a key focus in the development of PCMH
2011. Several commentators argued that NCQA’s original program did not have sufficient
emphasis on the “patient’s voice,” and early results questioned whether implementing systematic
processes associated with medical home would support improved patient experiences. A number
of stakeholders recommended that NCQA consider measures of patient experiences results as
part of the PCMH evaluation.
With funding from The Commonwealth Fund, NCQA undertook efforts to incorporate patient and
family feedback into requirements for practices seeking PCMH recognition. Our overall goal was to
identify feasible and sustainable approaches for evaluating practices on the results of patient
experiences, including:
A core set of survey items, representing a full range of the functions of medical homes and
the patient experiences;
A defined sampling process, which is auditable and captures the types of patients for whom
medical home services are most critical;
A specified data collection process, including mode(s) and timing of data collection; and
A fair scoring approach, that creates valid and meaningful benchmarks.
We convened a technical expert panel, reviewed evidence on effective care practices and existing
surveys, elicited broad public input on priorities for patient experiences surveys, and summarized
existing efforts to assess patient experiences of care at the physician and practice level. In
addition, we worked with the CAHPS consortium sponsored by the federal Agency for Healthcare
Research and Quality (AHRQ) to develop a new version of the CAHPS Clinician & Group Survey to
address specific processes of care relevant to patient-centered medical homes.

NCQA Document – Do Not Copy, Distribute, or Disclose

Page 1 of 11

DEVELOPMENT OF THE PCMH SURVEY
It is important to note that NCQA’s effort to identify domains and items included a broad review of
survey tools in addition to including the CAHPS survey. This section describes NCQA’s efforts to
identify potential content and the eventual decision to collaborate with the CAHPS Consortium on a
PCMH version of the Clinician & Group Survey.
With assistance from a technical panel of experts in survey research, we reviewed 22 surveys
identified from existing literature on patient-centered care or,used in PCMH evaluations and in
cross-national comparisons. We identified 616 items that focus on patient experiences. With advice
from the technical panel and the PCMH Advisory Committee, we identified 19 potential topics for
inclusion in a patient survey.
In February - March 2010, NCQA sought public comment on priorities for topics that should be
included in patient experiences surveys for evaluating practices that want to serve as medical
homes. Using a web-based survey tool, we invited public feedback to to rank five topics as high
priority and five as low priority. We disseminated the request for public comment through NCQA’s
outreach list as well as through the Patient-Centered Primary Care Collaborative and other
relevant groups. We received feedback from 1840 respondents; respondents reflected a variety of
stakeholder perspectives including patient/consumer/caregiver/advocates (44%), clinicians (49%),
and others (Employer/Purchaser, Researcher). Because all topics in patient-centeredness could be
considered important, we used a specific sorting approach that required respondents to nominate
high and low priority topics. The survey tool required respondents to name a maximum of five high
priority and five low priority measures. This approach allowed us to observe the topics receiving the
most “high priority” votes.
The topics receiving the most votes for “high priority” are listed in order below.
1. Listens and answers your questions
2. Involves you in decisions about your care
3. Explains care to you
4. Is aware of care you get from other doctors or places
5. Follows up on your test results
6. Helps you manage your health
7. Seeing the same doctor or nurse
Patients and clinicians comments agreed in the top six ranked topics. There were no differences by
gender, but “Is aware of your medications” received the third highest number of “high priority” votes
among respondents aged 65 and older. Interestingly, “seeing the same doctor or nurse” received
about the same number of high priority and low priority votes.
After reviewing these results, our technical experts and the PCMH Advisory Committee
recommended we consider stakeholder priorities, the evidence-base about factors that affect
quality, and key domains proposed for the PCMH 2011 standards in identifying content for the
survey. The domains recommended by the panels included:
Communication
Access
Coordination
Shared decision-making
Self management support
Whole person orientation/continuity
Comprehensiveness
In determining whether to select an existing survey tool or develop a new tool, we used the
following criteria: 1) does the tool address high-priority domains, 2) is the tool in widespread use or
does it have national endorsement; 3) is the tool applicable to diverse populations (including
NCQA Document – Do Not Copy, Distribute, or Disclose

Page 2 of 11

children and adults, available in different languages), and 4) how long is the survey. We
considered six existing survey tools: the CAHPS Clinician & Group Survey (including core and
supplemental items), the Ambulatory Care Experiences Survey (Safran), the Components of
Primary Care Index (Flocke), Primary Care Assessment Tool (Starfield), Patient Experience
Assessment Tool (TransforMed), and the How’s Your Health survey (Wasson). With input from our
panels, we decided to build on the CAHPS Clinician & Group Survey rather than using other
existing surveys or creating a new stand-alone survey. The CAHPS Clinician & Group Survey was
chosen because it is already widely used throughout the country, represented the content well, is
the only nationally endorsed patient experiences survey, and has multiple versions
(https://www.cahps.ahrq.gov/default.asp). Furthermore, the CAHPS research team had separate
efforts underway to seek public input on items for a PCMH version of the CAHPS Clinician & Group
Survey. Thus, we were able to merge our efforts with those of the CAHPS Consortiumto develop
and test a new survey for evaluating the PCMH.
NCQA worked with the CAHPS Consortiumon a Medical Home version of the CAHPS Clinician &
Group Survey (CAHPS PCMH Survey) to incorporate the technical expert panel and committees’
input on survey content and proposed domains. We identified a list of topics and potential items,
often drawn from existing surveys. These items were included in the PCMH 2011 call for public
comment. We also worked with the CAHPS Consortium to develop new items. The CAHPS
Consortium conducted focus groups and cognitive testing in English and Spanish with adult
patients and parents of pediatric patients and included both patients in medical home practices and
those in primary care practices that are not categorized as medical homes. The PCMH Survey
underwent cognitive testing last fall followed by a field test of draft versions of both the adult and
child CAHPS PCMH Survey conducted by NCQA. The field test version of the PCMH survey
included 115 items; in several cases we included items from existing surveys as well as new items
addressing the same content using CAHPS principles to maintain a level of standardization of
survey questions.
FIELD TEST METHODS
NCQA contracted with the Massachusetts Health Quality Partners (MHQP) to oversee a field test.
MHQP is a not-for-profit coalition of physicians, hospitals, health plans, purchasers, consumers,
academics, and government agencies that collects and publishes quality information to support
quality improvement among clinicians and practices and informed decision-making among
consumers. MHQP was responsible for executing data use agreements with all participating
practices,developing the survey protocol, sampling plan and all survey materials. MHQP contracted
with a survey vendor to administer the survey as a two-wave mail survey.
Study population
Forty-three (10 Adult and 33 Child) practices in the State of Massachusetts participated in the field
test. Twenty practices, including 10 sites serving adults and 10 serving children were recruited for
the field test. An additional 23 pediatric sites participated through funding of the Pediatrics
Physicians' Organization (PPOC) at Children's for practices within its network affiliated with
Children’s Hospital of Boston.
The study focused on adult and pediatric patients who receive care at primary care practices in
Massachusetts. The study group included any patient with a visit at the practice in the prior year
(from July 16, 2009 to July 15, 2010). A parent or guardian was asked to complete the survey for
eligible children.

NCQA Document – Do Not Copy, Distribute, or Disclose

Page 3 of 11

Approximately 800 patients were surveyed from each practice to yield a completed survey sample
of 200 per practice. To reduce patient burden, the sample was de-duplicated so that only one adult
member per household was included.
Procedures
Practice Recruitment
MHQP recruited practices from local health networks including a collaborative of safety net
providers. Participating practices provided patient administrative data to MHQP’s survey vendor to
identify sufficient number of sampled patients (approximately 800) to yield a minimum 200
completed surveys for each practice.
Survey Protocol
A two-wave mail survey protocol was used.
1st wave: Monday, November 15, 2010
Response data drawn for 2nd wave Monday, December 20, 2010 (5 weeks after wave 1)
2nd wave: Monday, January 3, 2011
End of fielding: Friday, February 4, 2011 (5 weeks after wave 2)
Providers Included
All primary care providers who are impaneled at participating practices were included. Primary care
specialties included were: Internal Medicine, Family Medicine, General Medicine, and Pediatrics.
Nurse practitioners with patient panels were also included. Note that OB/GYN practitioners and
residents were not included as primary care providers.
Eligible Population
We included patients of all payor types, including commercial, Medicare and Medicaid beneficiaries
as well as uninsured patients, which represented about 8 percent of the sample mostly from the
community health centers. The pediatric sample included children age 0 to 17 and the adult
sample included patients age 18 and over. The sampling frame was created with data provided to
the survey vendor by participating practices and health centers. Patients were assigned to sites
using visit, enrollment and site data provided by sites: All patients with an eligible visit to a site
were equally likely to be sampled for the site regardless of the number of visits, type of visit, or
number of providers seen. Patients were randomly selected at the site level sample for each adult
or pediatric site to be included in the survey ensuring that no more than one patient per address is
sampled across the entire project. Patients were selected in sufficient numbers to obtain 200
completed surveys per site. Sampling started at sites with the smallest available sample frame in
order to minimize the potential impact of cross-site de-duplication on achieving the desired sample
sizes. Final site sample sizes were determined according to payer mix as follows:
Practices having a majority of patients covered by Medicaid or other noncommercial payer (excluding Medicare) were sampled approximately 833 patients
per site
Practices having a majority of patients covered by commercial insurers or Medicare
were sampled approximately 667 patients per site.
Sites included in the survey through PPOC participation and not included as field
test sites were sampled according to a variable sampling scheme based on the
number of physicians at the practice.

NCQA Document – Do Not Copy, Distribute, or Disclose

Page 4 of 11

The PCMH CAHPS pilot survey instruments are focused on the patient experience of care for a a
provider and the practice site. At the start of the survey patients were asked to confirm that they
received care from a named provider. Once the site level sample was drawn, sample patients were
assigned to the provider with whom they had the most recent visit and this provider was the
provider named on the survey. The visit timeframe across sites was used to determine most recent
visit and provider.
A running list of all households selected for the sample for any physician was maintained as the
household participation list. One patient per household was surveyed across survey types (adult
and pediatric).
Analysis
The survey vendor conducted the analysis of response rates. All other analyses were conducted
by the Yale-Harvard CAHPS team. Initial analyses considered the rate of missing and valid
responses and the performance level for each item. Other analyses included a principal
components factor analysis for all items, item-to-total correlations and internal consistency for
proposed composites, and correlations among composites.
FIELD TEST RESULTS
Response rates
The overall response rate for the field test was about 25% yielding 4,875 surveys for analysis (25.4
for adults in 10 practices and 25.6 for children in 33 practices. The response rates were higher in
the adult group and in pediatric sites affiliated with Boston Children’s hospital. These responses
rates are lower than seen in other surveys; the timing of the survey fielding (during the winter
holidays) and the length of time between the selection of the sample and the second survey wave
probably contributed to lower response rates. Analysis of non-respondents showed that older
patients and patients with more visits and chronic conditions were more likely to respond to the
adult survey. Type of insurance was the key factor affecting response in the child survey; limited
resources and time available for field test prevented the use of telephone follow-up which has
typically boosted response rates in sites serving large numbers of Medicaid patients.
Domains
See Attachments 1 and 2 for the field test versions of the Adult and Child surveys. Table 3
summarizes the CAHPS Consortium’srecommendations for domains of the PCMH CAHPS, along
with the item counts for each domain. The adult survey includes 58 items and the pediatric version
includes 67 items. This is a 50% reduction in length from the PCMH survey prepared for the field
test. This content represents the CG-CAHPS core plus new items.
Attachments 3 and 4 include the full analysis of item-level responses for the adult and child
surveys respectively, including the percentages of valid responses and the performance rates.
Attachment 5 includes a summary of results by domain. Items that were not recommended for
inclusion in the CAHPS PCMH survey are listed at the end of this document.
Access
The C&G CAHPS core survey includes 5 items on Access. Two additional items were
recommended for inclusion in the PCMH CAHPS because of their salience for the PCMH: one item
addressed after hours care and the second addressed days to urgent appointments. Items related
to e-mail access had low performance in this field test primarily due to the small number of
respondents who said they had sought advice by email. The team recommended the 3-item
composite from the CAHPS Health IT survey be considered as a supplemental set of items where
access to and use of e-mail for advice is more prevalent.
NCQA Document – Do Not Copy, Distribute, or Disclose

Page 5 of 11

Information
Three items relevant to the practice’s efforts to provide information to patients were recommended.
These items originally grouped with other conceptual domains but did not group well with those
original domains; nor do they make up a cohesive composite. Still the content is particularly
germane to the PCMH and these items performed well in the field test.
Communication
The C&G CAHPS core survey includes 5 items on communication in the adult survey and 8 items
in the child survey. We tested a number of different items related to communication; all of these
were strongly correlated with the existing core items. The reliability of the core items in this field
test is much lower than the CAHPS Consortium has observed in other settings; the field test of the
Clinician & Group Survey showed a reliability of 0.71, compared to 0.62 in this report. New items
developed to address “whole person orientation” correlated strongly with the communication items.
Because of the high correlation among the communication and whole person orientation items, the
CAHPS Consortium recommended keeping the original core and adding an additional item
addressing the whole person aspect.
Care Coordination
The C&G CAHPS core includes one item in the care coordination domain – follow up of testing. Of
the additional items tested, the team recommends including an item on specialty care and another
on medication use. The item on specialty care did not perform well in this field test; cognitive
interviews showed that respondents were confused about the referent. This item performs well in
other settings (e.g., MHQP and health plan CAHPS). The CAHPS Consortium will explore ways to
alter the instructions or placements to improve performance but recommends keeping this item at
this time.
We explored whether these items form a composite. The team recommends including these as
individual items.
Comprehensiveness
The field test included items that captured several aspects related to comprehensiveness. The
CAHPS Consortium recommends a series related to behavioral health needs for adults and a
series from the existing pediatric C&G CAHPS for children. The 3-item behavioral health composite
has good reliability and internal consistency and addresses an important but often overlooked
domain of care. For children items related to development and prevention performed better than
the behavioral health domains. A new item related to screen time works well with the existing
content.
Self-Management Support
In factor analyses, items related to self-management support presented the strongest factor after
communication/whole person orientation. In designing the field test, we developed items to
address self-management support for general health needs as well as for chronic conditions. In
the field test, all respondents completed these items although an explicit goal of the study was to
determine whether some items would work better with a targeted sample of patients with chronic
conditions.
We tested different approaches to identifying patients with chronic conditions; practices provided
data on patients’ diagnoses from billing records, and the survey asked patients to report on
whether they had ever been told they had a chronic condition (using a list of conditions from the
Medical Expenditure Panel Survey frequently used to assess presence of chronic conditions) and
included “the presence of a chronic condition” screener question developed by the CAHPS
NCQA Document – Do Not Copy, Distribute, or Disclose

Page 6 of 11

Consortium. The results suggest that the CAHPS C&G approach (get care for a condition or
problem that has lasted for at least 3 months or used medicine to treat a condition or problem that
has lasted for at least 3 months) and the PCMH approach (specific chronic conditions) are
sensitive but not specific to the billing records. This is reasonable because the billing records are
from a single provider and may not represent care provided in other settings; in addition,
identification strategies that rely on diagnoses have been know to miss patients who should qualify
as having chronic conditions because of coding errors, misdiagnoses and lack of access to care. .
About three-quarters of adults self-identified in the survey as having a chronic condition using
CAHPS screener; only 7% of children were identified as having a chronic condition based on
parent report using CAHPS screener questions.
We compared performance on self-management support items for patients with and without a
chronic condition (based on the CAHPS chronic conditions screener). For the adult survey, the
self-management support items performed better with patients with self-reported chronic
conditions, and some items seem particularly less suited to a non-chronic condition population.
One option would be to keep five items related to self-management support (70, 72, 73, 74, and
76; this composite had a reliability of 0.87 and alpha=0.86) but this would require adding some
screener items and limiting the analysis of the composite to people with chronic conditions. A
second option, which the CAHPS Consortium recommends, is making slight changes in the
wording of items 70 and 71 and using these items for all respondents. The revised items would be:
Work with you to set specific goals for your health
Ask you if there are things that make it hard for you to take care of your health.
Shared Decision-Making
The field test includes a set of three items about decision-making on two different kinds of
decisions: stopping or starting a medication and having a surgery or procedure. Just under half of
adult respondents answered the series of questions about medications; about one quarter of
respondents answered the questions about surgery or procedure. Neither set of items met the
reliability standard of 0.70. However, because this topic is of critical importance to consumers, the
CAHPS Consortium recommended including this series in the PCMH CAHPS.
Because of the large number of items for prevention and development for children, and the small
proportion of children who identify with chronic conditions, the CAHPS Consortium recommends
using the self-management support composite for adults only.
Office Staff
While this domain was not prioritized by our advisory groups, this composite was included in the
field test as part of the CAHPS C&G core. The composite has good reliability.
Reliability
Table 4 summarizes the reliability of the composites and items by domain for the adult and child
surveys. Based on these data we will ask your advice on the number of completed surveys that
should be required for each participating practice.

NCQA Document – Do Not Copy, Distribute, or Disclose

Page 7 of 11

Table 1. Results of Public Comment on Survey Topics for Evaluating Practices that Seek
Medical Home Qualification (n=1840)

Topic Area
Listens and answers your questions
Involves you in decisions about your care
Explains care to you
Is aware of care you get from other doctors or places
Follows up on your test results
Helps you manage your health
Seeing the same doctor or nurse
Getting routine care
Respects you as a person
Is aware of your medications
Knows you well
Access to help without making a visit
Getting all of your primary care at one location
Access to your medical records
Wait time
Getting care after hours
Has confidence in your ability to manage your health
Office staff
Asks you about the quality of care

NCQA Document – Do Not Copy, Distribute, or Disclose

Number of Respondents
High Priority
Low Priority
1,185
53
1,037
92
802
83
680
248
639
119
638
248
541
573
539
409
474
196
471
145
365
448
302
791
291
928
265
623
259
881
249
908
249
540
128
914
86
1,001

Page 8 of 11

Table 2. Field Test Response Rates
Identified
as Ineligible
(Deceased,
Disabled, or
Language
Barrier) (c)

Requested
Removal

Survey
Type

# of
Practices

Outgoing
Sample (a)

Returned
as
Undeliver
-able (b)

Adult

10

7,335

335

0

6

1,781

24.3%

25.4%

Child
Child
PPOC

10

7,069

438

1

4

1,461

20.7%

22.0%

23

5,669

72

0

4

1,633

28.8%

29.2%

Total

43

20,073

845

1

14

4,875

24.60%

25.53%

NCQA Document – Do Not Copy, Distribute, or Disclose

Returned
(d)

Raw
Response
Rate (d/a)

Adjusted
Response
Rate*
(d/(a-b-c))

Page 9 of 11

Table 3. Comparison of the Domains and Number Items in the CAHPS Clinician & Group
Core Survey versus the Proposed PCMH Items/Composites

Domain
ADULT
Access
Information
Communication
Coordination of Care
Comprehensiveness: behavioral/whole
person
Self Management Support
Shared Decision Making
Office Staff
Rating
Eligibility

CAHPS C & G
Core Item
Count

PCMH Item
Count

CAHPS C & G
Core with
PCMH Item
Count

9
0
7
2

3
3
1
4

12
3
8
6

0
0
0
2
1
4

3
2
4
0
0
0

3
2
4
2
1
4

12
37

1
21

13
58

9
0
14
2

3
3
0
4

12
3
14
6

5
5
0
2
1
4
12
54

0
1
2
0
0
0
0
13

5
6
2
2
1
4
12
67

Demographics
Total Item Count
CHILD
Access
Information
Communication
Coordination of Care
Comprehensiveness: pediatric
development
Comprehensiveness: pediatric prevention
Self Management Support
Office Staff
Rating
Eligibility
Demographics
Total Item Count

NCQA Document – Do Not Copy, Distribute, or Disclose

Page 10 of 11

Table 4. Summary of Reliability Results for the CAHPS Clinician & Group Core Survey
versus the Proposed New PCMH Items/Composites
CAHPS C&G Core

Domain

ADULT
Access
Information (individual items)
Communication
Coordination of Care (individual items)
Comprehensiveness: behavioral/whole
person
Self Management Support
Shared Decision Making
Office Staff
CHILD
Access
Information
Communication
Coordination of Care
Comprehensiveness: pediatric
development
Comprehensiveness: pediatric prevention
Self Management Support
Office Staff

PCMH

Reliability

Number
needed to
achieve
reliability of
0.70

Reliability

Number
needed to
achieve
reliability of
0.70

0.87
-0.62
0.87

60
-239
53

0.89
0.75-0.96
0.66
0.09-0.71

49
13-124
205
140-2482

---0.91

---40

0.89
0.83
0.61
--

48
76
127
--

0.92
-0.78
0.54

17
-60
61

0.92
0.75-0.91
-0.52-0.55

18
19-65
-70-89

0.86
0.83
-0.91

33
41
-20

-0.87
0.69
--

-31
90
--

NCQA Document – Do Not Copy, Distribute, or Disclose

Page 11 of 11