AIR Cog Test Results - 9 Feb 08

Health Information Technology (HIT)
CAHPS Items
Cognitive Testing Findings

February 2008
Victoria Gonzalez, Ali Campot, and Roger Levine
American Institutes for Research
9 February 2008

1

BACKGROUND
After draft Health Information Technology items were developed and reviewed
internally, a set of items was identified for cognitive testing. A protocol for cognitive testing
these items (Appendix A) was developed by the A.I.R., RAND, and Yale/Harvard CAHPS II
instrument team leads.
This protocol and the associated subject recruitment, screening, and consent
procedures, were reviewed and approved by A.I.R.'s Institutional Review Board (IRB).
METHODS
Participants were recruited through use of an "all staff" broadcast e-mail. This e-mail
sought to identify friends and family members of staff with the following characteristics:
- Must be 18 years of age or older.
- Must be able to speak and read English.
- Cannot be employed as a medical caregiver (i.e., doctor, nurse, therapist) or in the field of
health services.
- Must have a doctor whom they usually see if they need a check-up, want advice about a
health problem, or get sick or hurt.
- Must have seen this doctor in the last 12 months.
- Must have done one of the following:
(a) E-mailed this doctor in the last 12 months
(b) Checked medical test results on-line in the last 12 months
(c) Check their medical record on-line in the last 12 months
(d) Made a doctor's appointment on-line or through e-mail in the last 12 months
Three individuals meeting these criteria were identified and agreed to participate. They were
cognitively interviewed in AIR's Palo Alto Cognitive Survey Laboratory on January 3, 10, and 11,
2008. These interviews were videotaped, with the signed consent of the participants.
SUBJECTS
Participants in this study included a female, Asian, between 20 - 29 years old; a white, female,
over 60 years old, and a white male, over 60 years old. In the course of these interviews, we were
able to determine that one subject was a Kaiser member. The other two received their health care
though the Palo Alto Medical Foundation.
GENERAL FINDINGS
Certain items could not be administered because they were appropriate only for
individuals with specific types of experiences. For example, none of our participants went to a
doctor who provided "online after visit summaries." Accordingly, the follow-up questions about
this HIT feature could not be administered. Similarly, only one participant had used email or a
web site to make an appointment, resulting in a follow-up item only being administered to one
respondent.

2

With a maximum of only three participants, it is difficult drawing firm conclusions from
our data. Nonetheless, we felt that individual participant's comments frequently reflected
concerns and issues that would be typical of other respondents.
Our general findings, subject to the above caveats follow.
"You" vs. "Anyone". Certain items were developed to determine if specific HIT features
were available through a provider's system. For example, one item asked if the doctor's office
put lab results "on a web site for you to see." Although "you" was intended to mean "anyone,"
one respondent interpreted this to mean "you, personally." Accordingly, even though this
feature was available, she answered negatively because she didn't sign up for it -- so the
doctor's office didn't put laboratory or test results on a website for HER to see.
The items asking about the presence of a feature can be reworked to clarify their
intention. Alternatively, if these items are only going to be used as screeners, the issue
becomes moot.
E-mail usage. Respondents used e-mail for reasons other than getting questions
answered (the focus of the HIT e-mail questions). One respondent wanted to report about
these e-mail experiences -- and did include these experiences -- in synthesizing her responses
to items asking about the speed with which their questions were answered and how often all of
these questions were answered.
Although this was only one respondent, we believe that the use of e-mail in ways other
than requesting answers to questions (to request completion of different forms; to request
medications) will be a common occurrence. Accordingly, we recommend that the focus of
these e-mail items be broadened, either through the development of additional items or
rewordings to clarify our intentions.
Use of never-always scale. For several items, the never-always scale did not seem
appropriate. For example, this scale was used to ask, "How often was it easy to find these lab
or other test results on the web site?" One respondent explicitly noted that the ease of finding
these items really didn't change over 12 months. The comprehensibility of test results is
another type of item that, most likely, will not change over a 12-month period, suggesting that
behavioral frequency response scales might not be the best way to try to measure these
constructs.
We also tested several "shared decision" items because of concerns about the
appropriateness of the never-always scale. For one of the items, a dichotomous yes/no scale
appeared to be preferable to a graduated yes/no scale. This item, which asked about whether
this doctor asked which choice was best for you, seems to be demanding a dichotomous
response. Accordingly, we recommend that serious consideration be given to changing the
ACAHPS items on shared decision from behavioral frequency items to either dichotomous or
graduated yes/no items.
SPECIFIC FINDINGS
Write-up Key
R# Respondent ID.
NP Response indicates no problem with the given question.
PP Response indicates a possible problem with the given question.
DP Response indicates a definite problem with the given question.
3

Is there a doctor you usually see if you need a check-up, want advice about a health problem,
or get sick or hurt?

 Yes
 No --> If No, please answer the following questions about the last doctor you visited
R1 PP Yes. However, there should be a box for clarification, because the doctor
who I correspond with on-line is not my primary physician. I don’t like my
primary physician, so I always have to get a second opinion.
R2 NP Yes
R3 NP Yes.
RECOMMENDATION: Since this item was created solely for survey administrative purposes (to
identify a focal doctor and better imitate actual survey administration procedures). It seemed to be
effective in enabling identification of a focal doctor, without having to ask the respondent to "think of
their personal doctor and answer the survey items about this doctor." So, we feel this approach can
be used in future cognitive testing.
Email [Communication Domain]
1. In the last 12 months, did you email this doctor with a medical question?
□ Yes
□ No --> If No, Go to next section
R1 DP Yes. I emailed her personal email address, but I didn’t hear back from her, so
then I emailed the email address that was posted on the website. After I got
in touch with the doctor, she informed me that she no longer uses the old
address I sent the first email to. Before you could send things to your doctor’s
personal email, but they have formalized things through the system online.
R2 NP No. I haven’t intended to, I did receive an email from him
R3 NP No. I haven’t emailed the doctor because you have to pay for an account with
those special features.
PROBLEM: R1 e-mailed the doctor, using an old e-mail address. This was a partial reason for her
"Yes" response.
RECOMMENDATION: Try to determine whether this will be a rare event or a common event. If
common, corrective action needs to be taken.

4

2. In the last 12 months, when you emailed this doctor, how often did you get an answer to
your medical question as soon as you needed?
□ Never
□ Sometimes
□ Usually
□ Always
R1 DP Sometimes. [The respondent had a difficult time defining “medical question”.
She did request a medical letter for her employer, but the respondent did not
consider this a question. A PP is that the respondent didn’t know whether it
took a long time to get a response was due to the email mix-up, and she had
to follow- up several times before she received a response. Another PP is the
term “as soon as you need it.” She received the information by the time she
needed it, but didn’t get it as soon as she would have wanted. So, she
changed her response to "Sometimes" as opposed to usually or always.]
R2 S
Skipped appropriately.
R3 S
Skipped appropriately.
PROBLEM: R1 included a response to a request for form completion as an example of a response to
a medical question.
RECOMMENDATION: Consider rewording item (to broaden construct) or include items about other
uses of e-mail.

3. In the last 12 months, when you emailed this doctor, how often were all of the questions in
your e-mail answered?
□ Never
□ Sometimes
□ Usually
□ Always
R1 PP Usually. [If R1 never had to follow-up the answer would have been “always”.
If R1 hadn’t had followed up so many times the answer could have been
“never.” The respondent interpreted “often” to mean “thoroughly.”
R2 S
Skipped appropriately.
R3 S
Skipped appropriately.
PROBLEM: R1 included a response to a request for form completion as an example of a response to
a medical question.
RECOMMENDATION: Consider rewording item (to broaden construct) or include items about other
uses of e-mail.
5

Online lab/test results [Access to Information & Communication Domains]
1. Does this doctor’s office put your laboratory or other test results on a web site for you to
see?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
R1 DP No. [R1 took this question to be personal, her doctor doesn’t post her
information on the website, but R1 knew if you signed up for a certain
program, that this option was available.] You and your should be changed to
“one’s”
R2 NP Yes. [She uses this system to view her laboratory and other test results. She
considers lab and other test results to mean any results you get when you get
tested for something, ex. Mammogram. Yes she would include X-rays.]
R3 NP Yes. [He uses this system and when asked what he considers “laboratory and
other test results” he responded:] I suppose that would include any fluids
tests, like blood or urine. I suppose it would also included x-rays.
PROBLEM: This was discussed as a general problem: A confusion as to whether "you" means "you"
in the singular sense (you, yourself) or "you" in the plural sense (anyone).
RECOMMENDATION: If this is not an idiosyncratic issue, the item should be reworded.

2. In the last 12 months did you look for your lab or other test results on the web site?
□ Yes
□ No --> If No, Go to next section
R1 S
Skipped appropriately,
R2 NP Yes. [She looked just once when she received a notice.]
R3 NP Yes. I just did it one time.
PROBLEM: No problems with this item were observed.

6

3. In the last 12 months, how often was it easy to find these lab or other test results on the
web site?
□ Never
□ Sometimes
□ Usually
□ Always
R1 S
Skipped appropriately.
R2 PP Always. The respondent had a problem with the working of “how often” was it
easy. R2 doesn’t think using the computer is easy and doesn’t use it often,
but once you get to the web-site she responds that it’s easy to use. She
though the question was asking for her to ‘rate the ease’ of finding the lab
results.
R3 NP Always. I would interpret this to mean the general ease of use of the website.
PROBLEM: R2 did not feel this was a behavioral frequency since ease of finding something on a web
site will not really change over time.
RECOMMENDATION: Consider changing this to a graduated "Yes/No" scale.
4. In the last 12 months, how often were these lab or other test results presented in a way that
they were easy to understand?
□ Never
□ Sometimes
□ Usually
□ Always
R1 S
Skipped appropriately.
R2 PP Usually. Respondent only used this once, she would have said Always if the
content of the results were easier for her to understand and if they didn’t use
the medical terms.
R3 NP Always. The results were well presented. The only confusion I had was the
medical terminology where a negative result is a good thing. But that didn’t
have to do with the presentation of the results. [Did you consider this when
responding?] No, because that doesn’t have to do with the presentation.
PROBLEM: Respondent considered the use of medical terms in the presentation of results in forming
a judgment about comprehensibility. If this is a factor we want considered, there is no problem.
RECOMMENDATION: None.

7

Online Medication List [Access to Information Domain]
1. Does this doctor’s office put a list of the prescription medicines you take on a web site for
you to see?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
R1 DP No. [Again the respondent took this question personally] they don’t put my
medicines up, but I know it’s an option if you sign up for the on-line account,
this is an option for all patients, but I chose not to.
R2 NP Don’t know. I think she does, but I don’t use it. I must have seen an
advertisement on the web-site that showed this capability, but I’m not sure.
R3 NP Don’t know. I don’t know because I don’t have a use for the service. I don’t
have a lot of regular medications for them to put on there.
PROBLEM: This was discussed as a general problem: A confusion as to whether "you" means "you"
in the singular sense (you, yourself) or "you" in the plural sense (anyone).
RECOMMENDATION: If this is not an idiosyncratic issue, the item should be reworded.

2. In the last 12 months, did you look at this list of your prescription medicines on the web
site?
□ Yes
□ No --> If No, Go to next section
R1 S
R2 S
R3 S

Skipped appropriately.
Skipped appropriately.
Skipped appropriately.

PROBLEM: No one answered this item. If this is a rarely used feature, there may not be a sufficient
number of respondents to permit the production of stable estimates.
RECOMMENDATION: If this is a "rare" feature, the item should be made into a supplemental item.

8

3. In the last 12 months, how often was the list of prescription medicines up to date?
□ Never
□ Sometimes
□ Usually
□ Always
R1 S
R2 S
R3 S

Skipped appropriately.
Skipped appropriately.
Skipped appropriately.

PROBLEM: No one answered this item. If this is a rarely used feature, there may not be a sufficient
number of respondents to permit the production of stable estimates.
RECOMMENDATION: If this is a "rare" feature, the item should be made into a supplemental item.

Online after visit summary (AVS) [Communication]
1. Some doctors supply patients with notes describing what was discussed during the visit.
Does this doctor supply you with visit notes?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
R1 NP No, but that would be great.
R2 NP No. [Respondent had an acceptable understanding of what “visit notes”
meant.]
R3 DP Don’t know. [“Visit notes”?] I would think those would be any diagnostic
discussion, treatment instructions, anything that was discussed during the
visit. That sort of thing. [Would you consider the handout you mentioned being
given by the dermatologist?] No, because when I answered this I was only
considering online materials, because of the heading and because previous
questions were asking about electronics. I would include that handout in
off-line visit notes, but I wouldn’t include it here because I was only
thinking about online notes.
PROBLEM: R3 was confused by the inconsistency between the section heading and the question.
RECOMMENDATION: Change the section heading.

9

2. Are these notes put on a web site for you to see?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
R1 S
R2 S
R3 S

Skipped appropriately.
Skipped appropriately.
Skipped appropriately.

PROBLEM: No one answered this item. If this is a rarely used feature, there may not be a sufficient
number of respondents to permit the production of stable estimates.
RECOMMENDATION: If this is a "rare" feature, the item should be made into a supplemental item.
ALTERNATIVE VERSION FOR ITEMS 1 and 2. Does this doctor put notes describing what was
discussed during the visit on a web site for you to see?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
R1 NP No. There is no difference in these two questions.
R2 NP No. I think these two questions are asking the same thing.
R3 S
Skipped appropriately.
PROBLEM: No problems were noted with this alternative.
RECOMMENDATION: Consider using this version in place of items 1 and 2 in this section,
dependent on further cognitive test findings.
3. In the last 12 months did you look on the web site at notes about any of your visits to this
doctor?
□ Yes
□ No --> If No, Go to next section
R1 S
R2 S
R3 S

Skipped appropriately.
Skipped appropriately.
Skipped appropriately.

PROBLEM: No one answered this item. If this is a rarely used feature, there may not be a sufficient
number of respondents to permit the production of stable estimates.
RECOMMENDATION: If this is a "rare" feature, the item should be made into a supplemental item.

10

4. In the last 12 months, how often was it easy to find these notes on the web site?
□ Never
□ Sometimes
□ Usually
□ Always
R1 S
R2 S
R3 S

Skipped appropriately.
Skipped appropriately.
Skipped appropriately.

PROBLEM: No one answered this item. If this is a rarely used feature, there may not be a sufficient
number of respondents to permit the production of stable estimates.
RECOMMENDATION: If this is a "rare" feature, the item should be made into a supplemental item.
5. In the last 12 months, how often were these notes easy to understand?
□ Never
□ Sometimes
□ Usually
□ Always
R1 S
R2 S
R3 S

Skipped appropriately.
Skipped appropriately.
Skipped appropriately.

PROBLEM: No one answered this item. If this is a rarely used feature, there may not be a sufficient
number of respondents to permit the production of stable estimates.
RECOMMENDATION: If this is a "rare" feature, the item should be made into a supplemental item.

11

Online Appointments [Getting Needed Care]
1. Can you make appointments at this doctor’s office by email or on a website?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
R1 PP Yes. If you register on-line you can get this service, I’m 99% sure.
[Respondent is inconsistent in her answer, the “Can you” instead of “Does
you doctor’s office” translated better for this respondent. She looked into this
option, but didn’t want this information to be posted online for security
reasons.]
R2 NP Yes. I know you can do this, but I have never done it.
R3 NP Yes. It’s possible but I personally haven’t done it.
PROBLEM: Although this did not appear to be a problem, this is an example of an item for which
"you" may be interpreted as "you, personally" rather than "anyone."
RECOMMENDATION: If this is not an idiosyncratic issue, the item should be reworded.

2. In the last 12 months, did you use email or a web site to make an appointment at this
doctor’s office?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
R1 NP No.
R2 S
Skipped appropriately.
R3 NP I haven’t for three reasons. 1) I’m not sure if it’s part of the service you have
to pay for; 2) I just forget to try that; 3)I believe the probably save some
appointment options for in person calls, that aren’t available through their
online service.
PROBLEM: No problems were noted.
RECOMMENDATION: Continue cognitive testing.

12

3. In the last 12 months, how often was it easy to make an appointment at this doctor’s office
by email or on a web site?
□ Never
□ Sometimes
□ Usually
□ Always
R1 S
R2 S
R3 S

Skipped appropriately.
Skipped appropriately.
I can’t answer this question, because I haven’t used the service. So I guess I
would just skip it. [He confused the skip pattern here, but ended up skipping
the question].

PROBLEM: No one answered this item. If this is a rarely used feature, there may not be a sufficient
number of respondents to permit the production of stable estimates.
RECOMMENDATION: If this is a "rare" feature, the item should be made into a supplemental item.
General Item
1. In the last 12 months, did this doctor use a computer to help explain things?

 Yes
 No

1
2

R1 DP No. The doctor did print something out for me once, but the printer wasn’t
working. I think using the computer to explain things would be if she showed
me something on the computer screen and explained it to me. {The
respondent answered no because the print out she would have received was
“useless information.” Although the print-out would have explained
something, the respondent didn’t value the information, so she answered no.]
R2 PP Yes. He sends lab results and comments on the lab results but it doesn’t
explain the problem. The comments on a high-blood pressure exam might
say “eat less salt”. Two weeks ago he sent an email with lab results and
comments that said “labs look good.” [The doctor has never printed
something out for R2, but considers print-outs to printed information to help
explain things.]
R3 DP Yes. [He did not have a problem answering this question as his doctor often
used a computer monitor to explain things to him. The problem enters, when
he says he would not include printed material here.] [If you have only
received print outs from your doctor would you answer this question the same
way?] No. I would not include those here.
PROBLEM: Respondents are inconsistent with whether or not materials printed from the computer
should be considered in answering this item.
RECOMMENDATION: In discussing this item, the HIT Team decided to delete it.
13

Shared Decision Making
1. Choices for your treatment or health care can include choices about medicine, surgery, or
other treatment. In the last 12 months, did this doctor tell you there was more than one
choice for your treatment or health care?
1
2

 Yes
 No If No, Go to End

R1 NP Yes.
R2 NP Yes. The doctor referred me to a knee doctor, but it’s been over 12 months.
R3 NP No.
NOTE: This is a standard CAHPS item. It was not cognitively tested.
2. In the last 12 months, did this doctor talk with you about the pros and cons of each choice
for your treatment or health care?

 Definitely yes
 Mostly yes
3
 Mostly no
4
 Definitely no
1
2

R1 PP Mostly yes. I have a problem with the wording “pros and cons.” The advice
was more based on do you want the treatment yes or no. She didn’t explain
the pros and cons, so that’s why I put mostly yes.
R2 NP Definitely yes. They have improved the medical system over the years.
R3 S
Skipped appropriately.
PROBLEM: It is not clear if "pros and cons" is being interpreted as intended.
RECOMMENDATION: The HIT Team, in a conference called, suggested this phrase be changed to
"good things and bad things."

14

3. In the last 12 months, when there was more than one choice for your treatment or health
care, did this doctor ask which choice was best for you?

 Definitely yes
 Somewhat yes
3
 Somewhat no
4
 Definitely no
1

2

R1 PP Definitely no. It’s not the style of this HMO, they assume the patient should
be asking this, it’s implied that I’ll choose the best option for myself. [R1 saw
the responses as completely dichotomous either yes or no, she couldn’t
explain why she might answer it “ Somewhat no” as opposed to “Definitely
no.”
R2 NP Definitely yes. [R2 thinks the doctor should decide what’s best for the
patient.] the doctor does ask what’s best for me and gives me the pros and
cons.
R3 S
Skipped appropriately.
PROBLEM: Respondents perceived this to be a dichotomous item -- and did not answer "Somewhat."
RECOMMENDATION: Change scale to "Yes/No."

15

APPENDIX A. HIT Cognitive Interview Protocol
12/24/2007
{This document contains the HIT CAHPS draft items (26 October 07)}

Email [Communication Domain]
1. In the last 12 months, did you email this doctor with a medical question?
□ Yes
□ No --> If No, Go to next section
Issues for Cognitive Testing:
How do respondents determine they are emailing the doctor vs. posting a message on a dedicated
site?
Where did the respondent get the email address? Is the address generic to the practice or for a
specific doctor?
IF YES, PROBE:
About how many times in the last 12 months did you email your doctor with a medical question?

What kind of question did you have – tell me as much as you feel comfortable sharing? (if more than
one, probe about most recent)

Where did you get the email address you used? How did you know it was the right email address for
your doctor? (Does anyone else in your doctor’s office use that email address?)

IF NO, PROBE: Have you ever emailed your doctor with a medical question? (What kind of question
did you have – tell me as much as you feel comfortable sharing . . .)

16

2. In the last 12 months, when you emailed this doctor, how often did you get an answer to
your medical question as soon as you needed?
□ Never
□ Sometimes
□ Usually
□ Always
Issues for Cognitive Testing:
How do respondents interpret “as soon as you needed”?
What are acceptable and unacceptable wait times for a response to an email?
How do respondents who emailed only once use the response scale?
How often are respondents emailing questions to their doctor (should this be a yes/no item rather
than a frequency?)
PROBE, ALL:
What do you think they mean by “as soon as you needed? (How long did you have to wait for a
response?)

IF NEVER/SOMETIMES/USUALLY, PROBE: What would have to be different for you to have
answered (ANSWER +1)?

PROBE, ALL: How long should it take for a doctor to answer your email? (How long is too long?)

17

NEW. In the last 12 months, when you emailed this doctor, how often were all of the questions
in your e-mail answered?
□ Never
□ Sometimes
□ Usually
□ Always
Issues for Cognitive Testing:
How do respondents interpret “all of the questions in your email”?
Do respondents send multiple queries in a single email?
Does this item capture whether the doctor’s response was thorough?
How do respondents who emailed only once use the response scale?
Should this be a yes/no item rather than a frequency?
PROBE:
In your own words, what do you think this question is asking?

Did you have more than one question in your email? (Did your doctor answer all of your questions?)

IF NEVER/SOMETIMES/USUALLY, PROBE: What would have to be different for you to have
answered (ANSWER +1)?

18

Online lab/test results [Access to Information & Communication Domains]
1. Does this doctor’s office put your laboratory or other test results on a web site for you to
see?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
Issues for Cognitive Testing:
Does this item have consistent interpretation across respondents (what are laboratory and test
results)?

PROBE, ALL: How did you decide on your answer to this question (what did you think about?)

What do you think of as laboratory and other test results?

IF NOT MENTIONED: Do "laboratory and other test results" include x-rays or not? Why (not)?

19

2. In the last 12 months did you look for your lab or other test results on the web site?
□ Yes
□ No --> If No, Go to next section
IF YES, PROBE:
About how many times in the last 12 months did you look for lab or other test results on the web site?

IF NO, PROBE: Why didn’t you look for your lab or other test results on the web site?

3. In the last 12 months, how often was it easy to find these lab or other test results on the
web site?
□ Never
□ Sometimes
□ Usually
□ Always
Issues for Cognitive Testing:
How do respondents interpret “easy to find these lab or other test results on the web site”?
How do respondents who had only one test or lab result use the response scale?
Should this be a yes/no item rather than a frequency?
Did anyone look for results and not find them?
PROBE, ALL:
In your own words, what do you think this question is asking?

IF NEVER/SOMETIMES/USUALLY, PROBE: What would have to be different for you to have
answered (ANSWER +1)?

20

4. In the last 12 months, how often were these lab or other test results presented in a way that
they were easy to understand?
□ Never
□ Sometimes
□ Usually
□ Always
Issues for Cognitive Testing:
How do respondents interpret “presented in a way that they were easy to understand”?
How do respondents who had only one test or lab result use the response scale?
Should this be a yes/no item rather than a frequency?
PROBE, ALL:
In your own words, what do you think this question is asking?

Have you ever seen lab or other test results that were not easy to understand? (Tell me about it)

IF NEVER/SOMETIMES/USUALLY, PROBE: What would have to be different for you to have
answered (ANSWER +1)?

21

Online Medication List [Access to Information Domain]
1. Does this doctor’s office put a list of the prescription medicines you take on a web site for
you to see?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
Issues for Cognitive Testing:
How do respondents interpret the posting of a medications list on a web site -- do they understand
that other patients won't be able to see their list?
IF YES: Please tell me a little bit about how that works.

IF NO: And you say that because....?

2. In the last 12 months, did you look at this list of your prescription medicines on the web
site?
□ Yes
□ No --> If No, Go to next section
IF YES: How many times in the last 12 months (did you look at the list of prescription medicines on
the website)?

IF NO: Have you ever looked at this list of your prescription medicines?
Yes

--> Tell me about it. Why did you look? What did you find out?

No

--> Why not?

22

3. In the last 12 months, how often was the list of prescription medicines up to date?
□ Never
□ Sometimes
□ Usually
□ Always
Issues for Cognitive Testing:
Do respondents interpret "up to date" as being the same as accurate?
What do you think they mean by a list of prescription medicines you are taking being “up to date"?

IF NEVER, SOMETIMES, OR USUALLY: What, if anything, did you do to have the listed updated?

23

Online after visit summary (AVS) [Communication]
1. Some doctors supply patients with notes describing what was discussed during the visit.
Does this doctor supply you with visit notes?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
Issues for Cognitive Testing:
How do respondents interpret “visit notes”?
In your own words, please tell me what you think they mean by ‘visit notes’.

2. Are these notes put on a web site for you to see?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
Issues for Cognitive Testing:
How do respondents interpret the posting of visit notes on a web site -- do they understand that other
patients won't be able to see their notes?
IF YES or NO: And you say that because ....?

Why would they put notes on a web site?

24

ALTERNATIVE VERSION FOR 1 & 2. Sometimes, we try out different ways of asking questions.
So, I'd like you to answer the following question. HAND RESPONDENT PAGE WITH THE
FOLLOWING QUESTION.
Does this doctor put notes describing what was discussed during the visit on a web site for
you to see?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
Issues for Cognitive Testing:
Do respondents interpret this question literally -- that is, as the doctor personally putting the notes on
the web site?
Do respondents interpret these notes as being the same as the "visit notes" in the previously
administered items?
What do you think they mean by "notes describing what was discussed during the visit?"

Would these notes be written for you to read -- or would they just be the doctor's notes that
make up your medical record? Or would they be something else?

How do you think the doctor would put the notes on a web site?

IF ANSWER IS DIFFERENT: How did you come up with your answer? PROBE TO UNDERSTAND
WHY ANSWERS ARE DIFFERENT.

25

3. In the last 12 months did you look on the web site at notes about any of your visits to this
doctor?
□ Yes
□ No --> If No, Go to next section
Issues for Cognitive Testing:
Confirm respondent's understanding of visit notes.
IF YES: How many times?
IF NO: Did you ever look at web site notes about your doctor visits?
Yes

--> Tell me about it. Why did you look? What did you find out?

No

--> Why not?
If you've never done this, how do you know that they put notes on a web site?

4. In the last 12 months, how often was it easy to find these notes on the web site?
□ Never
□ Sometimes
□ Usually
□ Always
Issues for Cognitive Testing:
Do respondents interpret this question as asking about the usability of the web site for themselves or
for people in general?
Do respondents interpret these notes as being the same as the "visit notes" in the previously
administered items?
Please tell me why you answered that it was [RESPONSE TO Q4] easy to find these notes on the
web site.

26

5. In the last 12 months, how often were these notes easy to understand?
□ Never
□ Sometimes
□ Usually
□ Always
Issues for Cognitive Testing:
What factors do respondents consider in deciding whether the notes were easy to understand?
What does it mean to have notes that are easy to understand?

IF NECESSARY: Is it necessary that you are familiar with all of the words the doctor used in the
notes for the notes to be easy to understand?

How did you use the information in the visit notes?

27

Online Appointments [Getting Needed Care]
1. Can you make appointments at this doctor’s office by email or on a website?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
Issues for Cognitive Testing:
This item is asking about the availability of a system to schedule appointments by email or on a
website rather than the respondent's capability of using this system. Did the respondent understand
this?
IF YES: How do you know this?

IF NO: How did you figure this out? That is, how do you know that someone can't make
appointments this way?

IF DON'T KNOW: How could you figure this out? That is, how could you figure out if someone could
make appointments this way?

2. In the last 12 months, did you use email or a web site to make an appointment at this
doctor’s office?
□ Yes
□ No --> If No, Go to next section
□ Don't Know --> If Don't Know, Go to next section
Issues for Cognitive Testing:
"E-mail" refers to either a special e-mail system or the general use of e-mail. Was this understood by
the respondent?
IF YES: Tell me about it. How did you make the appointment?

IF NO/DON'T KNOW: Why didn't you make appointments this way?

- What do you think they mean by e-mail? That is, are they asking about regular e-mail or a
special kind of e-mail? Why?
28

3. In the last 12 months, how often was it easy to make an appointment at this doctor’s office
by email or on a web site?
□ Never
□ Sometimes
□ Usually
□ Always
Issues for Cognitive Testing:
"Easy" refers to ease of use of the system for scheduling appointments. It does not refer to the ease
of getting an appointment at the desired time. Determine if respondent interprets item correctly.
Also, determine if behavioral frequency scale is appropriate for this item.
How many appointments did you make?

About how long did it take you to figure out how to make appointments?

And, how come you picked [RESPONSE] instead of [RESPONSE+1]?

General Item
1. In the last 12 months, did this doctor use a computer to help explain things?

 Yes
 No

1
2

Issues for Cognitive Testing:
Determine if respondent is only considering information that he or she sees on the computer screen
or if materials accessed through the computer and printed out are also considered.
IF YES: How did the doctor use a computer to help explain things? Any other ways?

IF NO: Did the doctor ever use a computer to help explain anything to you -- or to print out any thing
for you or not?
IF YES: Tell me about it. What happened? When was this?

If this doctor printed out materials from his computer and gave them to you, do you think this would
be an example of "using a computer to help explain things" or not? Why?

29

Shared Decision Making
1. Choices for your treatment or health care can include choices about medicine, surgery, or
other treatment. In the last 12 months, did this doctor tell you there was more than one
choice for your treatment or health care?
1
2

 Yes
 No If No, Go to End

Issues for Cognitive Testing:
Since this is a "standard" CAHPS item, do not probe.
2. In the last 12 months, did this doctor talk with you about the pros and cons of each choice
for your treatment or health care?

 Definitely yes
 Mostly yes
3
 Mostly no
4
 Definitely no
1
2

Issues for Cognitive Testing:
Do respondents distinguish a difference between "Mostly yes" and "Mostly no"?
IF RESPONDENT SAYS 'MOSTLY YES' OR 'MOSTLY NO,' CHOOSE THE NON-SELECTED
RESPONSE IN PROBE: How would a conversation with this doctor that was rated [Mostly
yes/Mostly no] be different from the conversation that you had? Or, would there really be a
difference? Please explain.

3. In the last 12 months, when there was more than one choice for your treatment or health
care, did this doctor ask which choice was best for you?

 Definitely yes
 Somewhat yes
3
 Somewhat no
4
 Definitely no
1

2

Issues for Cognitive Testing:
Do respondents distinguish a difference between "Mostly yes" and "Mostly no"?
Why did you say [RESPONSE] instead of [RESPONSE+1]?

30