ADA Trends - Development and Testing of a Survey Instrument...

T

R

E

N

D

Galley Proof

S

copyrighted material - not for distribution

The development and testing of a survey
instrument for benchmarking dental plan
performance
Using insured patients’ experiences as a gauge of dental
care quality
San Keller, PhD; Col Gary C. Martin, USAF, DC; Christian T. Evensen, MS; CAPT Robert H. Mitton,
DC, USN

he goal of dental care is to
improve the health and
meet the functional needs
of patients. However,
there is no standard, nonproprietary method for providing
national benchmarks of dental care
quality based on patient reports,
and dentistry has little systematic
information about delivery system
outcomes.1,2 However, it is difficult
to create a survey that provides
actionable results and covers all
topics important to various stakeholders while being short enough
for practical use. The purpose of
this research was to develop such a
tool.
Although investigators can use
clinical and administrative data to
obtain some performance indicators,
some aspects of dental care can be
captured only by surveying
patients.3 On the basis of a literature review of patient-reported outcomes in dental care, we determined
that the topics studied most frequently were in one of three areas:
communication and interaction with
the dental care provider4-9; patient
anxiety, fear in anticipation of pain
and comfort during treatment3,10-12;
and technical aspects of care, such
as comfort, functionality and
esthetics of dental work.13-21 Instruments that do focus on patients’
experiences tend to use satisfactiontype items to measure their experiences.8,9,19,22 Such reports may tell

T

ABSTRACT
Background. There is no standard, nonproprietary method for providing national benchmarks of dental care quality as described by
patients. The purpose of this research was to develop such a tool following
guidelines of the Consumer Assessment of Healthcare Providers and Systems (CAHPS) initiative.
Methods. The authors identified domains of dental care quality via
qualitative methods, including a literature review, stakeholder interviews
and focus groups with beneficiaries, and they cognitively tested draft questions with patients to yield a pilot survey. Psychometric analyses of pilot
data (n = 3,264) identified summary indexes and guided survey revisions.
The authors used two waves of subsequent data collection (n = 4,221) to
test the validity of the revised survey.
Results. The mean response rate across three rounds of data collection
was 51 percent. Statistical analysis indicated that 17 questions could be
reliably collapsed into three indexes: “Care From Dentist and Staff” (reliability = 0.89, scaling success = 100 percent); “Access to Dental Care” (reliability = 0.78, scaling success = 100 percent); and “Dental Plan
Coverage/Service” (reliability = 0.84, scaling success = 100 percent).
Conclusions. The validity of the survey was supported in mail and
Internet modes for the American English language, and the instrument
was approved by the CAHPS consortium for distribution as the CAHPS
Dental Plan Survey.
Practice Implications. A tool is available now for assessing dental
care quality by measuring adult patients’ experiences with their dental
care and coverage. The authors tested this instrument only in a population with third-party coverage, however, which is a potential limitation
that should be considered.
Key Words. Dental care quality; dental plan quality; CAHPS; patient
surveys; patient satisfaction; quality benchmarking.
JADA 2009;140(2):XX-XX.
Dr. Keller is a principal scientist, American Institutes for Research, 101 Chapel Hill, N.C.
At the time this study was conducted, Col Martin was the director, Dental Care Division, TRICARE Management Activity, Falls Church, Va. He now is an assistant professor, Uniformed Services University of
the Health Sciences, Bethesda, Md. Address reprint requests to Col Martin, Tri-Service Center for Oral
Health Studies, Building 141, Room 221, USUHS, 4301 Jones Bridge Rd., Bethesda, Md. 20814-6975,
e-mail “[email protected]”.
Mr. Evensen is a senior research analyst, American Institutes for Research, Chapel Hill, N.C.
CAPT Mitton is the chief, Dental Care Branch, TRICARE Management Activity, Falls Church, Va.

JADA, Vol. 140

http://jada.ada.org

February 2009

1

Galley Proof

T R E N D S

copyrighted material - not for distribution

Use the Best Scientific Evidence Available

Literature review; stakeholder interviews;
pilot test and peer review; wave 1 and 2 test
and peer review

delivery of care. To that
end, we followed a program
of research that addressed
the CAHPS survey design
principles shown in Table
1.

Measure Only Those Things for Which the
Respondent Is the Best or Only Source of
Information

Patient focus groups; patient cognitive testing

METHODS

Base the Assessment on Respondents’
Experiences With Specific Provider
Behaviors

Patient focus groups; cognitive testing

TABLE 1

CAHPS Dental Plan Survey developmental steps.
DESIGN PRINCIPLES

RESEARCH TASKS

We developed a conceptual
framework to develop the
survey content by using
multiple qualitative
Incorporate Stakeholder Input Throughout
Key informant interview; patient focus groups;
methods, including a literthe Development Process
patient cognitive testing; stakeholder review
of draft questionnaires
ature review, stakeholder
Design the Survey so That the Results Are
Patient focus groups; patient cognitive testing
interviews and focus
Communicated Easily to Consumer
groups with patients. The
Audiences
figure shows how the
Place Products in the Public Domain
CAHPS consortium submission
various steps in this
process fit together.
The institutional review
Provide Technical Assistance to Users
CAHPS consortium submission
board of American Institutes for Research (AIR),
* CAHPS: Consumer Assessment of Healthcare Providers and Systems.
Washington, reviewed and
approved all data collection tools (such as interview guides used in the
researchers about patients’ experiences, but they
focus groups, the various versions of the survey),
also can be subject to the emotional state of the
consent forms, privacy statements and protocols.
respondent and provide little in the way of actionWe obtained signed consent from all participants
able information. Our goal was to develop a
in the stakeholder interviews, focus groups and
survey based on design principles that would procognitive interviews. A privacy statement
vide scientifically sound, actionable results.
appeared on the cover page of the survey (or the
The design and testing of this tool were
introduction section of the online version of the
informed by the Consumer Assessment of Healthsurvey), and completion of the survey was
care Providers and Systems (CAHPS) initiative.
accepted as consent.
CAHPS is a public-private initiative begun in
Literature review. We conducted a search of
1994 and continuing through 2012 to develop a
the MEDLINE and PsycINFO (American Psychostandard set of surveys of health care quality as
experienced and reported by patients.23,24 Widelogical Association, Washington) databases for
spread adoption of these surveys by providers
articles published from 1966 to 2007 by using
and/or systems is facilitated by the quality of the
these key words: dental patient experiences, satmethods used to develop, test and disseminate
isfaction with care, domains, measurement and
them. These methods include rigorous scientific
surveys. Two of us (S.K., C.T.E.) sorted the
peer review of results, the involvement of key
survey items that we extracted from the reviewed
stakeholders in the design and testing of the surpublications into domains. We then evaluated
veys, and the distribution of surveys and supinformation regarding the reliability and validity
porting material free of charge at the Agency for
testing of the surveys. We supplemented the pool
Healthcare Research and Quality (AHRQ) Web
site.25
ABBREVIATION KEY: AHRQ: Agency for Healthcare
The objective of this project was to develop a
Research and Quality. AIR: American Institutes for
dental plan quality survey that, as part of the
Research. CAHPS: Consumer Assessment of HealthCAHPS family of surveys distributed and supcare Providers and Systems. CFA: Confirmatory factor
ported by AHRQ, could be used to provide
analysis. CFI: Comparative fit index. EFA:
national benchmarks for dental insurance plan
Exploratory factor analyses. NNFI: Nonnormed fit
performance, especially with regard to the
index. RMSEA: Root mean square residual.
2

JADA, Vol. 140

http://jada.ada.org

February 2009

T R E N D S

Galley Proof
copyrighted material - not for distribution

Literature
Review

Expert
Interviews

Dental Plan
Quality
Conceptual
Framework

Patient
Focus
Groups

Draft
Questions

Design
Survey

Survey
Cognitive
Testing

Stakeholder
Review

Pilot Test Version
CAHPS Dental Plan
Survey

Figure. Process of developing the Consumer Assessment of Healthcare Providers and Systems dental plan pilot test survey.

of survey items with several unpublished surveys
designed specifically to target the dental plan. We
incorporated this information into the protocols
that we designed for the stakeholder and focusgroup interviews.
Stakeholder interviews. The objectives of
the interviews (conducted by S.K., C.T.E. and
other employees of AIR) with stakeholders were
as follows:
didentify domains of dental care quality of
greatest interest;
ddetermine preferred survey operations;
ddetermine preferred data reporting formats;
dobtain advice about ensuring the relevancy of
the CAHPS Dental Plan Survey to a variety of
stakeholders.
The 12 participants included an expert in
dental care policy, an expert in dental services
research, dental care insurance plan purchasers
and dental care insurance plan providers.
Patient focus groups. A total of 72 dental
plan enrollees (recruited by a professional
recruiting firm from lists provided by patients’
insurance companies) participated in 12 focus
groups conducted on the east and west coasts
(North Carolina and California). The objectives of
the focus groups were to identify domains of
dental care quality that were important to dental
plan enrollees but were not covered by the literature review or key informant interviews; determine which domains of dental care quality were
of greatest interest to participants; and determine
participants’ preferred survey mode (that is, mail,
telephone, Internet).

We performed qualitative analyses of the literature, the audiotaped interviews with stakeholders and the audiotaped focus groups with
dental patients. We drafted questions to address
each of 117 unique features of dental care, which
we then organized into 20 topic areas. To
decrease the burden on respondents, we created a
shorter version of this question list by choosing
the subsets of questions addressing topic areas
that both dental care experts and dental patients
identified as the most critical aspects of care.
Cognitive testing. We evaluated the comprehensibility of the survey items as well as participants’ ability to navigate the survey by conducting cognitive interviews with 16 dental
patients who varied in age, education and health.
During each two-hour, one-on-one interview (with
an employee of AIR), participants verbalized their
thoughts as they responded to survey questions.
A trained cognitive interviewer asked scripted,
probing, follow-up questions to gain additional
information about the clarity of the questions and
the ease of completing the survey. We rewrote or
eliminated questions according to the results of
the cognitive testing.
Pilot survey. This process resulted in a 50item pilot survey that included 17 items to
describe the characteristics of respondents and 33
questions about patients’ interactions with dentists and staff members, ease of finding a dentist
and obtaining appointments, office waiting times,
and quality of the dental plan including coverage
of services and perceived value. The objective of
the pilot test was to determine how the responses
JADA, Vol. 140

http://jada.ada.org

February 2009

3

Galley Proof

T R E N D S

copyrighted material - not for distribution

visit in the six-month
period before the pilot test.
Description of data sets.
These patients were memCHARACTERISTIC
PILOT TEST
WAVE 1
WAVE 2
bers of three dental insur(N = 6,488)
(N = 4,255)
(N = 4,155)
ance plans, the memberSample Characteristics
ship of which currently
No. of ineligible respondents
88
52
7
represents approximately
No. of completed Internet surveys
316
231
136
2.9 million covered lives.
We drew a stratified
Total No. of completed surveys
3,264
2,201
2,020
random sample—in which
Response rate (%)
51
52
49
each plan represented its
Self-Reported Respondent Characteristics (%) *
own stratum—of 6,488
Female
68
71
69
members from the samAge (Years)
pling frame, with the goal
of obtaining 2,100 com18-44
53
52
50
pleted surveys (700 for
45-64
33
32
35
each of the three plans).
≥ 65
14
16
15
We employed Synovate,
Education
a certified CAHPS vendor,
High school/GED† or less
16
20
18
to collect the survey data.
The vendor mailed survey
Some college or two-year degree
41
41
41
packages (cover letter,
Bachelor’s degree or more
43
39
41
copy of the survey and a
Race/ethnicity
return envelope) to the
White
79
76
77
sampled plan members in
March 2006. We gave
African-American
7
8
8
respondents the option of
Hispanic
7
8
8
completing the survey
Other
7
9
7
online. The vendor mailed
Dental health
a reminder postcard one
Excellent
17
21
20
week later, followed by a
second mailing of the
Very good
43
42
43
survey package to nonreGood
31
28
29
spondents approximately
Fair
8
8
7
three weeks after that.
Poor
1
1
1
One week later, the vendor
* Percentages are based on the nonmissing data for each variable.
sent a final reminder post† GED: General Educational Development.
card. Collection of pilot
test data ended on May 31,
to the survey could be summarized into a smaller
2006. We followed the same administrative proset of indexes (that is, composite measures); to
cedures for the two subsequent data collections
evaluate the measurement properties of items,
that took place in the first (wave 1) and third
composite scores and overall ratings of dental
(wave 2) quarters of 2007. Table 2 characterizes
care; and to identify modifications that should be
the three samples.
made to the pilot test instrument on the basis of
Data analysis. Although the pilot survey conthese evaluations. The objective of the two subsetained 30 questions about dental care and dental
quent data collections was to field the revised
plan quality, 11 of these could not be summarized
survey and evaluate its reliability and validity.
into composite measures, either because they
Survey administration. The sampling frame
asked about the totality of the patient’s care expefor the pilot test consisted of 436,180 patients
rience or they were “screener” items designed so
residing in the 48 contiguous states who had been
that respondents skipped inapplicable questions.
enrolled in their dental plan for at least 12 con(For example, one question asked respondents if
secutive months and had had at least one dental
they had a regular dentist. If they responded “no,”

TABLE 2

4

JADA, Vol. 140

http://jada.ada.org

February 2009

T R E N D S

Galley Proof
copyrighted material - not for distribution

the survey instructed
them to skip questions
pertaining to patients’
experiences with their regular dentist.) Table 3
shows the paraphrased
content of the questions
we evaluated for inclusion
in composite measures
and the questions concerning the totality of the
patient’s care experience.
We used standard psychometric analyses to
summarize the 23 items
into composite measures,
as detailed in the technical
note in the box.26-30
Reliability. We computed Cronbach α31 as our
measure of internal consistency reliability.
Internal consistency reliability refers to the amount
of systematic variance in
scale scores. Scales with
reliability coefficients
above 0.70 are recommended to provide precision for use in statistical
analyses of group-level
comparisons.32
Validity. We evaluated
the validity of the questions as indicators of a
specific composite by
examining the Pearson
product moment correlations of each question with
each composite score (corrected for overlap)33 to
determine if those correlations exceeded 0.40 and
were higher than the correlation of the question
with the two alternative
composites (see Results
section). We assessed the
validity of the composite
scores by examining the
Pearson product moment
correlations of the composite scores with overall

TABLE 3

Paraphrased questions in pilot and final CAHPS*
dental plan surveys.
PARAPHRASED QUESTION

SURVEY TYPE

Questions About Dental Staff and Clinic

Pilot

Final

How often did your regular dentist explain things in a way that was easy to
understand?

✔

✔

How often did your regular dentist listen carefully to you?

✔

✔

How often did your regular dentist treat you with courtesy and respect?

✔

✔

How often did your regular dentist spend enough time with you?

✔

✔

How often did the dentist or staff tell you how much of the cost (of your dental
work) you would have to pay?

✔

How often did the dentist or staff do everything they could to help you feel as
comfortable as possible during treatment?

✔

✔

How often did the dentist or staff explain what they were doing while treating you?

✔

✔

How often did the dentist or staff tell you how to prevent future problems with
your teeth and gums?

✔

As a result of your treatment, were your dental problems fixed?

✔

How often was the dentist’s office or clinic very clean?

✔

Questions About Getting Care and Getting Care Quickly
How often were your dental appointments as soon as you wanted?

✔

✔

If you needed to see a dentist right away, did you get to see a dentist as soon as
you wanted?

✔

✔

How often did you get an appointment with a dental specialist as soon as you
wanted?

✔

✔

How often did you spend more than 15 minutes in the waiting room before you
saw someone for your appointment?

✔

✔

How often did someone tell you why there was a delay or how long the delay
would be?

✔

✔

✔

✔

Questions About the Dental Plan
How often did your dental plan cover all of the services you thought were covered?

✔

Did your dental plan cover what you and your family needed to get done?
Do you know how much the dental plan costs you in premiums and out-ofpocket costs?

✔

How often did the dental plan’s 800 number, written materials or Web site
provide the information you wanted?

✔

✔

Did this information from your dental plan help you find a dentist you were
happy with?

✔

✔

How often did your dental plan’s customer service give you the information or
help you needed?

✔

How often did your dental plan’s customer service staff treat you with courtesy
and respect?

✔

Overall Ratings of Different Aspects of Care
✔

What number would you use to rate your regular dentist?†
What number would you use to rate all of the dental care you personally
received?†

✔

✔

What number would you use to rate how easy it was for you to find a dentist?†

✔

✔

plan?†

✔

✔

Would you say that your dental plan is worth the cost?

✔

✔

Would you recommend this dental plan to people who want to join?

✔

✔

What number would you use to rate your dental

* CAHPS: Consumer Assessment of Healthcare Providers and Systems.
† On a scale from 0 to 10.

JADA, Vol. 140

http://jada.ada.org

February 2009

5

T R E N D S

Galley Proof
copyrighted material - not for distribution

BOX

Technical note detailing analyses conducted
to identify composite measures.

wave 1 and wave 2 to evaluate the modified version of the survey for comparability to the original (the pilot test).
RESULTS

Confirmatory factor analysis (CFA) of
the pilot test data indicated that the 15
The authors then performed confirmatory factor analysis (CFA) to determine
questions proposed to measure three
whether the pilot data were consistent with the composite structure around
aspects of dental care (“Care from Denwhich the survey was designed. They conducted a CFA based on structural
equation modeling, as implemented by PROC CALIS (SAS Institute, Cary, N.C.).
tist and Staff,” “Access to Dental Care”
They evaluated the goodness-of-fit of the model to the data by using χ2, the
and “Dental Plan”) demonstrated excelcomparative fit index (CFI), the nonnormed fit index (NNFI) and the average
root mean square residual (RMSEA). Common current practice with regard to
lent fit to the data in both the mail and
these indications of model fit is to
Internet collections (mail respondents:
2
dreport χ P values but not to reject models for which the P value is > .05 in
data sets with more than 250 observations;
χ279 = 350; CFI = .97; NNFI = .96;
0.95;
drequire CFI and NNFI to be greater than
RMSEA = .05; Internet respondents: χ279
drequire RMSEA to be less than 0.06.28-30
= 116; CFI = .96; NNFI = .95; RMSEA =
The CFA of the pilot questionnaire design revealed that the observed data did
.05).
not fit this model, so the authors conducted exploratory factor analyses (EFA)
to identify the pattern of relationships among questionnaire items by using
Reliability. With one exception (the
standard CAHPS factor analytic methods. They conducted the EFA on the corre“Access
to Dental Care” composite in
lation matrix by using the principle factor method with squared multiple correlations as initial communality estimates and oblique rotation (promax) with
the pilot data), these aspects of dental
Kaiser normalization. The authors determined the number of factors via the
care demonstrated high internal consiseigenvalues and the interpretability of the rotated factor pattern matrix.
tency reliability, with Cronbach α coeffiThe authors submitted the structure identified via EFA to a CFA to evaluate the
cients greater than 0.75.27,32
fit of the data to the new structure. They conducted this sequence of analyses
on a randomly selected half of the data so that they could test for the generalValidity. The median Pearson
izability of the findings in the other half (called the “hold-out” sample).
product moment correlations in Table 4
* CAHPS: Consumer Assessment of Healthcare Providers and Systems.
summarize the validity of the survey
questions as measures of their respecratings of quality.
tive composites (that is, scaling success). A comVariability. We evaluated the variability in
parison of the magnitude of the convergent and
the data by examining the distribution of scores
discriminant validity supports the overall validity
for each question and composite, particularly
of the items as indicators of their respective comnoting the percentage of respondents who gave
posite scales. Each composite includes items more
the highest (that is, the ceiling effect) and lowest
highly correlated with their own composite than
(that is, the floor effect) possible responses for the
they are with the two competing composites (100
composite. Ceiling effects indicate the percentage
percent scaling success).
of people for whom it would be impossible to
The second row of Table 4 (“Convergent
assess improvement over time or to distinguish
Validity”) shows that median correlations of items
among. Floor effects indicate the percentage of
with their own composite far exceed, for the most
people for whom it would be impossible to assess
part, the criterion of greater than 0.40, which
decrements over time or to distinguish among.
supports the validity of the survey questions as
Stability of measurement properties. We
indicators of the respective composite score.27,32
evaluated the validity and reliability of the
The only observed correlation that was lower
survey data collected through the Internet to
than 0.40 was for the question in the pilot data
determine whether the measurement properties
set regarding whether someone explained to the
of the survey were comparable across data collecpatient why there was a delay in the appointment
tion modes. Moreover, at the conclusion of the
(data not reported but available on request). The
pilot test, stakeholder representatives from the
influence of this item is reflected in the median
dental plans requested that additional questions
correlation of 0.44 for the “Access to Dental Care”
be tested for relevance to the dental plan comcomposite in the pilot data set, which is lower
posite. As a result, we added four questions to the
than the rest of the median correlations in that
field test survey targeted toward aspects of the
row.
dental plan. We conducted psychometric analyses,
The third row of Table 4 shows that the disas described above, on data collected from both
criminant validity of the three composites is good.
To make use of all available data, the authors imputed missing values by using
a procedure used in previous CAHPS* studies.26,27

6

JADA, Vol. 140

http://jada.ada.org

February 2009

T R E N D S

Galley Proof
copyrighted material - not for distribution

TABLE 4

CAHPS* dental plan composite measurement properties in three data sets.
MEASUREMENT PROPERTY

Reliability †27,32
Convergent

Validity ‡27,32

CRITERION

CARE FROM DENTIST
AND STAFF

ACCESS TO DENTAL
CARE

DENTAL PLAN
COVERAGE/SERVICE

Pilot

Wave 1

Wave 2

Pilot

Wave 1

Wave 2

Pilot

Wave 1

Wave 2

> 0.70

.88

.90

.88

.67

.78

.78

.76

.84

.84

> 0.40

.71

.74

.70

.44

.62

.61

.60

.66

.69

Discriminant Validity §27,32

< 0.40

.32

.23

.32

.41

.32

.35

.20

.18

.23

Overall Rating of Dental Care ¶34

> 0.40

.73

.70

.72

.56

.53

.54

.28

.18

.25

Plan #34

> 0.40

.19

.18

.23

.16

.20

.22

.76

.66

.64

< 10

54

62

60

9

16

8

9

8

10

< 10

0

0

0

0

0

0

0

0

0

Overall Rating of Dental
Percentage at Ceiling **
Percentage at

Floor ††

* CAHPS: Consumer Assessment of Healthcare Providers and Systems.
† Internal consistency reliability indicated by Cronbach α coefficient.
‡ Median Pearson product moment (PPM) correlation of the question score with the total composite score (with that question removed from the
composite score).
§ Median PPM correlation of the question score with the total score for the other two composites.
¶ PPM correlation of the composite score with the overall rating of dental care.
# PPM correlation of the composite score with the overall rating of the dental plan.
** Percentage of respondents who had the highest possible score on this composite.
†† Percentage of respondents who had the lowest possible score on this composite.

The median correlations of items with the competing composites are, with one exception, lower
than 0.40.27,32 The “Access to Dental Care” composite in the pilot data set is the one exception.
The correlations in rows 4 (“Relationship With
Dental Care”) and 5 (“Relationship With Dental
Plan” (Table 4) provide further evidence of the
validity of the composite measures by demonstrating how the composite scores are related to
respondents’ overall ratings of quality.34 Across
all three data sets, the “Overall Rating of Dental
Care” is highly correlated with the “Care From
Dentist and Staff” composite. Similarly, the
“Overall Rating of Dental Plan” is highly correlated with the “Dental Plan Coverage/Service”
composite. Finally, the “Access to Dental Care”
composite is correlated more highly with the
overall ratings of dental care than it is to the
overall ratings of the dental plan.
Variability. None of the three scales exhibited
floor effects and two of the three scales had
ceiling effects that were less than 10 percent in
two of the three data collections. We observed the
greatest problem with the lack of variability for
the “Care From Dentist and Staff” composite in
all three data sets; more than 50 percent of
respondents reported the highest possible score
on this composite.
Stability of measurement properties. The
measurement properties of the mail data were
the same as those of the Internet data (data not
reported but available on request) and improved,

for the most part, in the wave 1 and wave 2 data
sets. The ceiling effect for the “Care From Dentist
and Staff” composite did not improve from that
which we observed in the pilot sample.
We presented the results of the pilot test for
peer review by the consortium of CAHPS scientists. In November 2006, the consortium approved
the CAHPS Dental Plan Survey for inclusion into
the CAHPS family of instruments. The consortium evaluated subsequent modifications to the
survey in the analysis of waves 1 and 2 data
reported above. This modified version of the
CAHPS Dental Plan Survey is available to users
free of charge.35
DISCUSSION

Summary of results. We can reliably summarize responses to 17 questions on the survey into
three composite measures to indicate “Care From
Dentists and Staff,” “Access to Dental Care” and
“Dental Plan Coverage/Service.” The observed
internal consistency reliabilities of the composite
scores ranged from 0.67 to 0.90 and were comparable to or better than those of established
CAHPS measures. Although the reliability of the
access composite in the pilot data was slightly
lower than the recommended value, it compared
favorably with internal consistency reliabilities
reported for other CAHPS instruments.36,37 Moreover, the reliability estimates for the scores from
this composite passed acceptable levels in the two
subsequent data collections (waves 1 and 2). The
JADA, Vol. 140

http://jada.ada.org

February 2009

7

T R E N D S

Galley Proof
copyrighted material - not for distribution

“Access to Dental Care” and “Dental Plan Coverage/Service” composites demonstrated good
variability in their scores, but we observed substantial ceiling effects for the “Care From Dentist
and Staff” composite. The size of this ceiling
effect, however, is not unusual for questions that
ask respondents to evaluate their direct care
providers.38,39
Although we do not present the data in this
article, we conducted patient-mix analyses to
identify characteristics of patients that might
affect the way they responded to the survey but
that are not a consequence of their experiences
with their care or with their dental plan. We
assessed an item asking how many dental visits
the patient had in the previous 12 months, an
item asking about the respondent’s overall dental
health and several other items (for example, age,
education, sex, race, overall physical health,) as
possible patient-mix adjusters. The final version
of the instrument approved by the CAHPS consortium contains several questions that can be
used as patient-mix adjusters. We have recommended that age, education and overall dental
health be used as patient-mix adjusters, but users
of the survey may find it useful to test their own
potential adjusters.
Implications of results. The results of this
research suggest that users of this CAHPS Dental
Plan survey can be confident about the quality of
the data provided by the survey. The content
validity of the survey questions was supported by
a review of the literature, key informant interviews and focus groups with patients. The construct validity of the CAHPS Dental Plan Survey
composite scores was supported by the results of
CFA and by the relationship of the composite
scores to patients’ overall ratings of their dental
care and dental plan. In addition, we found the
measurement properties of the mail and Internet
survey to be comparable and stable when
assessed across three data sets. These findings
support administration of the survey in either or
both of two modes: mail or Internet.
Possible study limitations. Potential users
of the survey should note that the measurement
properties of the CAHPS Dental Plan survey have
been studied when administered via mail or
Internet. No data address the survey reliability
and validity when administered via telephone or
interactive voice response. While the measurement properties of the American English–language version of the survey are supported by the
8

JADA, Vol. 140

http://jada.ada.org

February 2009

study findings, the validity of the survey when
translated into other languages has not been
studied. For example, it is not known whether a
Spanish translation of the survey would have
comparable measurement properties. Finally, all
of the data reported here come from patients who
participated in one of three dental plans. A thorough evaluation of the measurement properties of
the survey awaits the implementation of the
survey by other dental plans or purchasers of
dental insurance. This instrument was not
designed to measure patient satisfaction or to
assess the experiences of patients who do not
have dental insurance.
CONCLUSION

The primary objective of the CAHPS Dental Plan
survey was to produce information that will
enable the comparison of plan performance, as
evaluated by dental patients. In terms of practice
implications, the instrument was not designed to
be used by individual dental practices, and many
of the items on the survey would be inapplicable
to patients in such a context. The instrument
would, however, be useful to various purchasers
of dental care plans, whether they be employers
or other organizations responsible for assessing
the performance of their dental plans. The rigorous testing of the reliability and validity of the
instrument demonstrates that this is a highquality survey, and that users of the survey and
readers of the results can have confidence that
the data collected via this instrument are scientifically sound. ■
Disclosure. The authors did not report any disclosures.
The development of this dental care quality benchmarking tool was
made possible by a contract from the TRICARE Management Activity,
Falls Church, Va., to the American Institutes for Research.
Data for this study were collected by the survey research firm Synovate, a vendor certified to collect data for the Consumer Assessments of
Healthcare Providers and Systems surveys.
1. Bader JD, Ismail AI. A primer on outcomes in dentistry. J Public
Health Dent 1999;59(3):131-135.
2. Birch S, Ismail AI. Patient preferences and the measurement of
utilities in the evaluation of dental technologies. J Dent Res 2002;81(7):
446-450.
3. Bader JD, Shugars DA, White BA, Rindal DB. Development of
effectiveness of care and use of services measures for dental care plans.
J Public Health Dent 1999;59(3):142-149.
4. Chisick MC. Satisfaction of active duty soldiers with family dental
care. Mil Med 1997;162(2):105-108.
5. Chapple H, Shah S, Caress AL, Kay EJ. Exploring dental patients’
preferred roles in treatment decision-making: a novel approach. Br
Dent J 2003;194(6):321-327; discussion 317.
6. Newton J, Brenneman D. Communication in Dental Settings Scale
(CDSS): preliminary development of a measure to assess communication in dental settings. Br J Health Psychol 1999;4(3):277-284.
7. Rankin JA, Harris MB. Patients’ preferences for dentists’

T R E N D S

Galley Proof
copyrighted material - not for distribution

behaviors. JADA 1985;110(3):323-327.
8. Street RL. Patients’ satisfaction with dentists’ communicative
style. Health Communications 1989;1(3):137-154. “www.informaworld.
com/smpp/content~content=a784771982~db=all”. Accessed Dec. 31,
2008.
9. Zimmerman RS. The dental appointment and patient behavior: differences in patient and practitioner preferences, patient satisfaction,
and adherence. Med Care 1988;26(4):403-414.
10. Bader JD, Shugars DA, White BA, Rindal DB. Development of
effectiveness of care and use of services measures for dental care plans.
J Public Health Dent 1999;59(3):142-149.
11. Pau AK, Croucher R, Marcenes W. Perceived inability to cope and
care-seeking in patients with toothache: a qualitative study. Br Dent J
2000;189(9):503-506.
12. Riley JL 3rd, Myers CD, Robinson ME, Bulcourf B, Gremillion
HA. Factors predicting orofacial pain patient satisfaction with improvement. J Orofac Pain 2001;15(1):29-35.
13. Skaret E, Berg E, Raadal M, Kvale G. Factors related to satisfaction with dental care among 23-year olds in Norway. Community Dent
Oral Epidemiol 2005;33(2):150-157.
14. Allen PF, McMillan AS, Locker D. An assessment of sensitivity to
change of the Oral Health Impact Profile in a clinical trial. Community
Dent Oral Epidemiol 2001;29(3):175-182.
15. de Bruyn H, Collaert B, Linden U, Bjorn AL. Patient’s opinion
and treatment outcome of fixed rehabilitation on Brånemark implants:
a 3-year follow-up study in private dental practices. Clin Oral Implants
Res 1997;8(4):265-271.
16. Hakestam U, Karlsson T, Soderfeldt B, Ryden O, Glantz PO. Does
the quality of advanced prosthetic dentistry determine patient satisfaction? Acta Odontol Scand 1997;55(6):365-371.
17. Hegarty AM, McGrath C, Hodgson TA, Porter SR. Patient-centred
outcome measures in oral medicine: are they valid and reliable? Int J
Oral Maxillofac Surg 2002;31(6):670-674.
18. Samorodnitzky-Naveh GR, Geiger SB, Levin L. Patients’ satisfaction with dental esthetics. JADA 2007;138(6):805-808.
19. Schropp L, Isidor F, Kostopoulos L, Wenzel A. Patient experience
of, and satisfaction with, delayed-immediate vs. delayed single-tooth
implant placement. Clin Oral Implants Res 2004;15(4):498-503.
20. Sloan JA, Tolman DE, Anderson JD, Sugar AW, Wolfaardt JF,
Novotny P. Patients with reconstruction of craniofacial or intraoral
defects: development of instruments to measure quality of life. Int J
Oral Maxillofac Implants 2001;16(2):225-245.
21. Stahlnacke K, Soderfeldt B, Unell L, Halling A, Axtelius B. Perceived oral health: changes over 5 years in one Swedish age-cohort.
Community Dent Oral Epidemiol 2003;31(4):292-299.
22. Reifel NM, Rana H, Marcus M. Consumer satisfaction. Adv Dent
Res 1997;11(2):281-290.

23. Crofton C, Lubalin JS, Darby C. Consumer Assessment of Health
Plans Study (CAHPS): forward. Med Care 1999;37(3 suppl):MS1-MS9.
24. Goldstein E, Farquhar M, Crofton C, Darby C, Garfinkel S. Measuring hospital care from the patients’ perspective: an overview of the
CAHPS Hospital Survey development process. Health Serv Res
2005;40(6 pt 2):1977-1995.
25. U.S. Department of Health and Human Services, Agency for
Healthcare Research and Quality. CAHPS surveys and tools to advance
patient-centered care. “www.cahps.ahrq.gov/default.asp”. Accessed
Nov. 13, 2008.
26. Rubin DB. Multiple Imputation for Nonresponse in Surveys. New
York City: John Wiley & Sons; 1987.
27. Keller S, O’Malley AJ, Hays RD, et al. Methods used to streamline the CAHPS Hospital Survey. Health Serv Res 2005;40(6 pt 2):
2057-2077.
28. Hu L, Bentler PM. Cutoff criteria for fit indexes in covariance
structure analysis: conventional criteria versus new alternatives.
Structural Equation Modeling 1999;6(1):1-55.
29. Kenny DA. Measuring model fit. “http://davidakenny.net/cm/
fit.htm”. Accessed Dec. 22, 2008.
30. Suhr DD. Exploratory or confirmatory factor analysis?
“www2.sas.com/proceedings/sugi31/200-31.pdf”. Accessed Dec. 12, 2008.
31. Cronbach LJ. Coefficient alpha and the internal structure of tests.
Psychometrika 1951;16(3):297-334.
32. Nunnally JC. Psychometric Theory. 2nd ed. New York City:
McGraw-Hill; 1978.
33. Howard KI, Forehand GG. A method for correcting item-total correlations for the effect of relevant item inclusion. Educ Psychol Meas
1962;22(4):731-735.
34. Cohen J. Statistical Power Analysis for the Behavioral Sciences.
2nd ed. Hillsdale, N.J.: Lawrence Erlbaum Associates; 1988.
35. U.S. Department of Health and Human Services, Agency for
Healthcare Research and Quality. CAHPS Dental Plan Survey.
“www.cahps.ahrq.gov/content/products/Dental/PROD_Dental_Intro.asp
?p=1021&s=214”. Accessed Jan. 3, 2009.
36. Hays RD, Shaul, JA, Williams VS, et al. Psychometric properties
of the CAHPS 1.0 survey measures: Consumer Assessment of Health
Plans Study. Med Care 1999;37(3 suppl):MS22-MS31.
37. Hargraves JL, Hays RD, Cleary PD. Psychometric properties of
the Consumer Assessment of Health Plans Study (CAHPS) 2.0 adult
core survey. Health Serv Res 2003;38(6 pt 1):1509-1527.
38. Castle N. Family satisfaction with nursing facility care. Int J
Qual Health Care 2004;16(6):483-489.
39. Gasquet I, Dehe S, Gaudebout P, Falissard B. Regular visitors are
not good substitutes for assessment of elderly patient satisfaction with
nursing home care and services. J Gerontol A Biol Sci Med Sci 2003;
58(11):1036-1041.

JADA, Vol. 140

http://jada.ada.org

February 2009

9