Final Cognitive Report CSR

CAHPS: Health Information Technology
Report of Cognitive Interview Findings
Deanne Dworski-Riggs, BA
Patricia Gallagher, PhD

March 10, 2008

The following is a summary of the main findings from the cognitive testing of items about health
information technology to be included as supplemental items in CAHPS surveys.

Methods
Cognitive interview subjects were drawn from the Center for Survey Research (CSR) volunteer
database as well as from a list of volunteers from the Beth Israel Deaconess Medical Center
(BIDMC). The volunteers were contacted to ascertain if they were appropriate subjects. Then
those who had screened in were called by specially-trained cognitive interviewers who explained
the purposes of the activity and arranged to conduct an in-person or phone interview at the
person’s home, at the Center for Survey Research (CSR) or in a public location.

Interviewers followed a semi-structured protocol. The interview schedule included the items
from the survey instrument and a set of structured probes designed to reveal how respondents
understood questions and went about answering them. Respondents were given one sheet of
paper during the course of the interview with the alternative question about after visit summaries.
Participants were asked to read the item aloud and mark down their answer. After the subject
completed the self-administered question, the interviewer administered the cognitive probes.

1

The remainder of the items were interviewer administered: the interviewers read the questions to
the subjects, obtained answers to each question, then proceeded to the cognitive probes.

With respondents’ written permission, all interviews were audio tape recorded. Interviewers
used the tape recordings to later review the interviews and record their observations. All subjects
received a $60 cash incentive to participate. The interviews were conducted in February, 2008,
by Deanne Dworski-Riggs and Marsha Schofield.

Findings
Our goal was to complete three cognitive interviews, including at least one person who used an
online medical system other than BIDMC. Two respondents used the BICMC system and one
used the Marino Health Center’s online services.

ID

Table 1. Participant Characteristics
Gender

Race

Age

Education

Health Center

1

F

Black

37

High School
Graduate

Beth Israel

2

F

White

36

College Graduate

Beth Israel

3

M

Hispanic

55

More than 4 year
College Graduate

Marino Health Center

The next section presents the item wording for each question tested, followed by a descriptive
account of the problems or issues found to be associated with each item and recommendations for
future action to address any identified issues.

2

Email
1. In the last 12 months, did you email this doctor with a medical question?
G
Yes

G

No --> If No, Go to next section

None of the subjects had any difficulty with this item. Respondents 1 and 3 answered yes and
respondent 2 answered no. Of those who answered yes one person has emailed “once or
twice” and the other had emailed about seven times. Everyone understood the term “medical
question.” Definitions included “knowledge about my situation, my medical history...” and
“any particular question about my health or some symptoms I was experiencing or
medication I was taking.” The subject who gave the latter definition was not talking about
asking for refills. Rather, she had asked her doctor about whether she should continue a
specific medication in light of her symptoms.
Recommendation:

Retain item as worded.

2. In the last 12 months, when you emailed this doctor, how often did you get an
answer to your medical question as soon as you needed?
G
Never

G
G
G

Sometimes
Usually
Always

There were no problems with this item. Respondent 1 answered always, respondent 3
selected usually, and respondent 2 skipped this question. Respondents definitions of “as soon
as you needed it” ranged from 3-4 hours to 3-4 days.
Recommendation:

Retain item as worded.

3

NEW. In the last 12 months, when you emailed this doctor, how often were all of
the questions in your e-mail answered?
G Never
G Sometimes
G Usually
G Always
There were no problems with this item. Respondent 3 answered always, respondent 1
selected usually, and respondent 2 skipped this question. Respondents understood that “all of
the questions in your email” was asking about the doctors “response and depth of
information.” Respondent 1 choose usually because her doctor sometimes only answers one
of her questions when she sends multiple questions in an email.
Recommendation:

Retain item as worded.

Online lab/test results
1. Does this doctor’s office put your laboratory or other test results on a web site
for you to see?
G Yes
G No --> If No, Go to next section
G Don't Know --> If Don't Know, Go to next section
All three respondents said yes to this question. Respondent 3 did not focus solely on his
healthcare center’s website. During probes he referred to other HIT systems, saying that his
doctor “can share results with other doctors online.” He also talked about using other sites
like WebMD to do internet research to educate himself about his test results. This problem
(including other web site when answering) recurred throughout the interview with this
participant.
Also, respondents had slightly different definitions of what should be included as lab and test
results. Respondents agreed that tests like blood testing and urine analyses count; however
they were conflicted whether to include x-rays and MRIs. Respondents 1 and 3 included xrays, but respondent 2 said she would not.
Recommendation: Use the same method for identifying “this doctor” to clarify which
web site patients should be reporting on. At the beginning of the survey instrument,
have a label with the name of the patient’s web site and explain that “for the rest of
the questions ‘this web site’ refers to the web site shown above.” Refer to “this web
site” in all relevant items.

4

2. In the last 12 months did you look for your lab or other test results on the web
site?
G
Yes
G
No --> If No, Go to next section

There were no problems with this item. Respondents 1 and 3 selected Yes and respondent 2
answered No.
Recommendation: Retain Item as worded.

3. In the last 12 months, was it easy to find these lab or other test results on the
web site?
G
Yes, Always

G
G

Yes, Sometimes
No

This was another question where Respondent 3 did not limit himself to answering for his
patient web site. Although the one time he went on his patient web site the results were easy
to find, he answered “yes, sometimes” because “it depends on...whether it requires more
research on my own part.”
There were no other problems. Respondent 1 answered yes and Respondent 3 skipped this
question.
Recommendation: Retain Item as worded.

4. In the last 12 months, how often were these lab or other test results put on the
web site as soon as you needed them?
G
Never

G
G
G

Sometimes
Usually
Always

5

Respondents had different ways of interpreting “as soon as you needed them.” Respondent 1
answered with an exact time of 48 hours. Respondent 2 said that the results should be up
within a few days of when the doctor tells her to expect them by. For example, if the doctor
tells her the results will be ready in two weeks, that is an acceptable wait time for her. Finally,
Respondent 3 interpreted the question to mean “prompt enough to be able to seek second
opinions, make decisions, etc.” If this question is intended to measure patient satisfaction with
wait time, then these differences in interpretation are not a problem. However, if the goal of
the question is to measure how long patients have to wait for lab results, then the question
should be reworded.
Recommendation: If it is a problem that patient responses are subjective, define “as
soon as you needed them.”

5. In the last 12 months, how often were these lab or other test results presented
in a way that they were easy to understand?
G
Never

G
G
G

Sometimes
Usually
Always

For this question, we found that patients who were provided similar information about test
results (e.g. test names, their scores, and the normal ranges) had very different answers (one
answered Always and the other answered Never) based on their expectations. For the
respondent who answered Always, having to do additional research to understand the lab
results was part of being a good patient; however for the respondent who answered Never
having to go “to Ask Jeeves and Web MD to figure out what the test was and if it was
important” meant that the results were not easy to understand. How respondents evaluated
whether the labs were “easy to understand” appeared to depend on their willingness to do
additional research.
Also, the respondents did not think about the response options for item as a frequency. Rather,
they seemed to be reporting how easily they understood the results. They did not use the
sometimes or usually categories at all.
Recommendation: If it is not a problem that patient responses are subjective, retain
item as worded.

6

Online Medication List
1. Does this doctor’s office put a list of the prescription medicines you take on a
web site for you to see?
G
Yes
G
No --> If No, Go to next section
G
Don't Know --> If Don't Know, Go to next section
All three respondents answered Yes to this question. This item created a problem in
subsequent items when respondent 2, who did not take any prescription medications,
answered Yes, because she knows “there is a section” on the web site for a listing of
prescription medications.
This is another question where respondent 3 when formulating his response thought about
other web sites in addition to the patient web site his doctor uses.
Recommendation: Add the following item at the beginning of this series as an
additional screener to allow respondents who do not have any prescription
medication to skip the items about the list of prescription medicines:
Do you now take any medicine that this doctor prescribed for you? If No, skip to the
next section. [Cognitive testing would be in order.]

2. In the last 12 months, did you look at this list of your prescription medicines
on the web site?
G
Yes

G

No --> If No, Go to next section

All three respondents answered yes to this question. This item was problematic for respondent
2. Once again she answered yes, because she had looked at this section of the website even
though she did not take any prescription medication.
Recommendation: Adding a screener will allow people who do not take prescription
medicine to skip this question (cf. the recommendation for Q1).

7

3. In the last 12 months, how often was the list of prescription medicines up to
date?
G
Never

G
G
G

Sometimes
Usually
Always

Respondent 2 (who does not have any prescription medications) answered always to this
question, because the fact that there were not prescription medicines listed meant the list was
up to date.
Respondent 1 answered never, because although she uses this web service frequently the list
was only up to date once.
Respondent 3 did not have a problem with this item. He answered always.
The respondents had different opinions of what it meant for the list to be “up to date.”
Respondent 3 define “up to date” as including “any changes or updates of medications
prescribed.” Similarly, respondent 2 thought the list should contain her prescription history,
including medicines that were prescribed, renewed, and discontinued. However, respondent 1
thought the list should only contain medicines the patient is currently taking, rather than the
entire prescription history.
Recommendation: Adding a screener will allow people who do not take prescription
medicine to skip this question.
Consider changing the question wording to: How often did this list of prescription
medicines include all the medicines you are taking? NB: needs cognitive testing.

After Visit Summary (AVS)
1. Some doctors supply patients with notes describing what was discussed
during the visit. Does this doctor supply you with visit notes?
G
Yes

G
G

No --> If No, Go to next section
Don't Know --> If Don't Know, Go to next section

8

There were no problems with this item. Respondent 3 answered Yes, but the other two
answered no. They all defined visit notes as a “summary of the points that were brought up at
the appointment or the next steps,” “if there was a diagnosis,” and any
“recommendations...for me and for follow up.”
Recommendation: Remove item in favor of the alternative (see below).

2. Are these notes put on a web site for you to see?
G
Yes

G
G

No --> If No, Go to next section
Don't Know --> If Don't Know, Go to next section

There were no problems with this item. Respondent 3 answered yes, the other two
respondents skipped.
Recommendation: Remove item in favor of the alternative (see below).

ALTERNATIVE VERSION FOR 1 & 2.
Does this doctor put notes describing what was discussed during the visit on a
web site for you to see?
G
Yes

G
G

No --> If No, Go to next section
Don't Know --> If Don't Know, Go to next section

There were no problems with this item. The answers to this question were consistent with the
answers to questions 1 and 2. Respondent 3 answered yes to both questions 1 and 2 as well as
this alternative question. Respondents 1 and 2 answered no to question 1 as well as this
alternative question.
Recommendation: In order to reduce the number of items and thereby the burden
on respondents, replace questions 2 and 3 with this alternative
question.
3. In the last 12 months did you look on the web site at notes about any of your
visits to this doctor?

G
G

Yes
No --> If No, Go to next section

9

There were no problems with this item. Only respondent 3 answered yes. The other two
respondents skipped this question.
Recommendation: Retain Item as worded

4. In the last 12 months, how often was it easy to find these notes on the web
site?
G
Never

G
G
G

Sometimes
Usually
Always

This is another question where respondent 3 thought about other web sites instead of
answering only for the patient site his doctor uses. Respondent 3 answered usually and the
other respondents skipped.
Recommendation: Retain item as worded.

5. In the last 12 months, how often were these notes easy to understand?
G
Never

G
G
G

Sometimes
Usually
Always

There were no problems with this item. Respondent 3 answered always. He defined easy to
understand as “available for a lay person to begin to comprehend.” The other two respondents
skipped this question.
Recommendation: Retain item as worded.

10

Online Appointments

1. Can you make appointments at this doctor’s office by email or on a website?
G
Yes

G
G

No --> If No, Go to next section
Don't Know --> If Don't Know, Go to next section

Respondents understood this item and answered appropriately. All three answered yes. Two
had used the system and one had not, but knew it exists because she had seen it on the web
site.
Recommendation: Retain item as worded.

2. In the last 12 months, did you use email or a web site to make an appointment
at this doctor’s office?
G
Yes

G
G

No --> If No, Go to next section
Don't Know --> If Don't Know, Go to next section

The respondents seem to understand this item. Respondents 1 and 3 answered yes and
Respondent 2 answered no.
However, it is unclear whether the cognitive goal that respondents understand that “E-mail
refers to either a special e-mail system or the general use of e-mail” was met. When asked the
probe: “are they asking about regular e-mail or a special kind?” all the respondents talked
about “a special kind” of email. They described it as “on a secure server” and “within the
system itself.” The fact that no one mentioned regular email may be because the probe was
phrased as an either-or question. (This phrasing may have deterred respondents from
answering both.) Also, the web sites respondents used all relied on “special email” to make
appointments. Therefore, this data does not allow us to confirm that respondents understood
regular email is included in this item, however it is possible that respondents would correctly
include personal email if that is how they make appointments.
Recommendation: Add the word either with visual emphasis to signal to the reader
that both modes should be included. The question would read:
...did you use either email or a web site to make an
appointment...

11

3. In the last 12 months, how often was it easy to make an appointment at this
doctor’s office by email or on a web site?
G
Never

G
G
G

Sometimes
Usually
Always

Neither of the two respondents who answered this question interpreted it correctly.
Respondent 1 answered usually. She answered usually because scheduling an appointment
online is slower than over the phone, but requires less effort. She has been making
appointments for over five years and described the system as getting “friendlier” and “easier
to navigate.” She did not mention any difficulties with the two appointments she made this
year. Therefore, she did not seemed to be using the scale as a frequency.
Respondent 3 answered always even though he said it took him a few days to figure out how
to make appointments. This answer is surprising given that he described himself as very
comfortable using the internet.
Recommendation: Change this question from a frequency to a yes/no item:
In the last 12 months, how often was it easy to make an appointment at this doctor's
office by email or on a web site?
G
Yes

G

No

Alternatively, the question wording could be changed to:
In the last 12 months, was it easy to use this system to make an appointment by
email or on a web site at this doctor's office?
G
Never

G
G
G

Sometimes
Usually
Always

Either of these options would require cognitive testing.

12

Shared Decision Making
1. Choices for your treatment or health care can include choices about medicine,
surgery, or other treatment. In the last 12 months, did this doctor tell you there
was more than one choice for your treatment or health care?
1
2

o Yes
o Noè If No, Go to End
Respondent 1was unsure whether “this doctor” still referred to her primary care
physician. She wanted to report on her orthopaedist because she had discussed treatment
options with him.
One respondent did not include medicine when answering.
Recommendation:

Switch the order of the question to “surgery, medicine or
other treatment” so that respondents are more likely to
take medicine into account.

2. In the last 12 months, did this doctor talk with you about the good things and
bad things about each choice for your treatment or health care?
1
2

o Yes
o No

All the respondents seemed to understand the item; however one respondent did not like
the item wording. She felt that “it’s too general, I’m not sure what it means. Also a little
judgmental, like there is a value in there, good and bad is a value statement. If talking about
making choices, there shouldn’t be a value attached to it, but more here are the facts, if you
don’t do treatment here’s what could happen to you.”
Recommendation:

Change item wording from “good things and bad things”
to “reasons for and reasons against.”

13

3. In the last 12 months, when there was more than one choice for your
treatment or health care, did this doctor ask which choice was best for you?

o Yes
2
o No
1

There were not problems with this item.
Recommendation: Retain item as worded.