-
TO: CAHPS HIT Team
FROM: Julie Brown & Trish Gallagher
DATE: December 19, 2008
SUBJECT: RAND & Yale Summary of findings from HIT cognitive testing
For ease of review, this document incorporates the findings from the Yale Team’s HIT cognitive interviews into RAND’s earlier summary of findings.
RAND Methods: During November 10-15, 2008 RAND conducted six cognitive interviews in support of HIT item development. Four interviews were interviewer administered and two interviews used a self-administered version of the survey. Four different versions of the survey were used to allow for differences in item order for alternative versions of Q47 and Q48.
Participants were recruited by a vendor familiar with the needs and goals of the CAHPS project. The vendor identified individuals with experience using at least two of the following forms of HIT:
Online scheduling of appointments,
Email communication with doctor,
Online review of results from lab or other tests,
Online listing of prescribed medicines, and
After visit summary notes
Several participants used HIT via Kaiser Permanente’s “My Health” portal, one participant used HIT via his health plan website, and another participant used HIT via a website maintained by a local testing laboratory.
Yale Methods: The Center for Survey Research (CSR), University of Massachusetts Boston, a member of the Yale team, conducted five cognitive interviews between November 19 – December 11, 2008 to gather feedback about candidate Health Information Technology items for use with CAHPS surveys. The subjects were recruited through CSR’s volunteer database, a posting on the Beth Israel Deaconess Hospital patient web portal, through advertisements on Craig’s List [on-line] and in the Metro, a free newspaper distributed to subway passengers. All participants were provided an $80 cash incentive to participate in the two-hour session and those who parked in the UMass parking lots were given parking passes. The interviews were audio taped and all participants signed a form indicating their informed consent to be video-taped.
Participant Characteristics: Table 1 summarizes the demographic characteristics of all interview participants as well as the types of HIT they endorsed in the cognitive interview.
Table 1. Participant Characteristics
Characteristics |
# of Participants (RAND/Yale) |
Gender |
Male=3/3 |
Age |
18-24=0/0 |
Education |
8th grade or less=0/0
Some college/2-year degree=3/1 Post graduate degree =0/1 |
Hispanic ethnicity |
Yes=1/1 |
Race* |
White=5/3 |
HIT Experience* |
Online medical history form=0/3 Online medication
request=4/2 |
*More than one response possible.
ITEM SPECIFIC FINDINGS
This section of the memo lists findings (problems, instrument design issues, or confirmation of item comprehension) from testing that are specific to the named items. If an item is not listed in this section, then no problems or design issues associated with that item were identified in our interview sessions.
*9. In the last 12 months, did you ever fill out a medical history form on a website before a visit to this doctor?
1 Yes
2 No
RAND Finding: No participants answered “yes” to Q9. Responses to probes indicated that interpretation of the item was consistent with our design goals (e.g., capturing a history form filled out in advance of a visit).
Yale Finding: Four subjects affirmed that they had filled out a medical history form on their doctor’s website. While three participants who answered ‘yes’ and the one who answered ‘no’ all appeared to understand the intent of the question, one of those who affirmed this item did so incorrectly.
As a high school graduate, this woman had the lowest level of education of this group of participants. She was requested to read this and all other survey items aloud during the interview. She read very quickly, albeit reading every word for this item. She seemed to pick out what she thought was the most salient term and answer the question based on that. In this case, she answered ‘yes’ but on probing it became clear that she had never completed a medical history form on the web. For other items she did not read the entire question, but answered as before finishing reading the entire question. This is an issue with self-administration that survey researchers have yet to find a solution to.
*13. In the last 12 months, how often was it easy to make an appointment at this doctor’s office?
1 Never
2 Sometimes
3 Usually
4 Always
RAND Finding: One participant interpreted the question as referring only to ease of making appointments via e-mail or a web site, rather than appointment making across all modes (phone, e-mail, web).
Yale Finding: As with RAND’s experience, one of the five subjects only responded thinking about making appointments via the website and didn’t consider appointment making by phone or email.
*18. In the last 12 months, did you e-mail this doctor with a medical question?
1 Yes
2 No If No, Go to Question 21 on Next Page
RAND Finding: Four participants answered “yes” to Q18. Three sent e-mail directly to the doctor’s e-mail address and one participant e-mailed a practice address (response came from doctor using same address).
Yale Finding: All five participants affirmed that they had emailed their doctors with a medical question. Two had used the email link in their doctors’ websites and the other three had emailed their doctors directly. One noted that sometimes her query is answered by a nurse or resident calling back; the other participants all received responses directly from their doctors.
*21. In the last 12 months, did you ever use e-mail to let this doctor know about how you are doing?
1 Yes
2 No
RAND Finding: 4 participants answered “yes” to Q21. One participant interpreted the question as asking if she had continued or responded to the e-mail communication referenced in Q18. One participant interpreted the question as asking if he had sent an e-mail “thank you” to the doctor after a visit. Other participants interpreted the question as asking if the patient had sent e-mail about how medication was working, about whether treatment was working, or to report on a chronic condition. We may want to move Q21 to come before Q18 or to come much later in the questionnaire (perhaps near the end of the communication section).
Yale Finding: It is not clear to subjects exactly what this question is asking. Two talked about emailing to let the doctor know about any side effects of new medications (although one of these Rs who responded ‘yes’ was referring to a telephone call and hadn’t had any experience emailing the doctor to let her know how she was doing (note that this is a college graduate)), Another participant asked if this question was asking about ‘how my general health is doing.’ She said that she would call him if treatment was not working and answered Q21 ‘no’ appropriately. Another subject thought that the question was asking if you “keep your doctor aware of your conditions and what is going on with you.” Consider either rewording or omitting this item.
*22. In the last 12 months, did this doctor ever send you an e-mail to remind you about tests or treatment you needed?
1 Yes
2 No
RAND Finding: Participants interpreted the question as referring to reminders about preventive care, as follow-up to a missed test or appointment, as a reminder to schedule a routine test for a chronic health condition, a list of scheduled appointments, reminders about which blood or other tests were needed before treatment or a hospital stay. Three participants answered “yes” to Q22 and all indicated that the “from” e-mail address or sender name matched their doctors name. One participant who answered “no” indicated that at each visit, her doctor verbally reminds her to check “online” for reminders to schedule appointments.
Yale Finding: Only one participant mentioned preventive care when responding to the probes for this item, and in his first example he talked about a reminder about an appointment that came from a specialist, not ‘this doctor’ and only talked about yearly appointments and allergy medicine refills on further probing. For two of the other subjects, this question was interpreted to mean is the office ‘proactive’ about things like letting patients know that they “have to be retested because they had protein in their urine” (note that this participant did not mention any reminders about preventive care) and about the results of current testing. One participant couldn’t imagine her doctor emailing reminders “unless it was something preoperative or really critical.”
*31. Choices for your treatment or health care can include things like medicine, surgery, or other treatment. In the last 12 months, did this doctor tell you there was more than one choice for your treatment or health care?
1 Yes
2 No If No, Go to Question 34
RAND Finding: Findings here are similar to findings from other CAHPS testing of the shared decision-making items in that participants broadly interpret “choices”. Examples of choices identified during the testing sessions include:
Whether or not to have surgery
Treatment options for their specific chronic condition (whether peritoneal dialysis was an option, alternatives to thyroid medication, how to manage blood pressure with life style changes rather than increasing medication)
Deciding which specialist to see
Alternatives to a colonoscopy
Whether or not to take antibiotics for a specific illness.
Yale Finding: The examples of treatment choices provided by this sample of patients follows:
One queried, “What kind of choices?” but went on to talk about a conversation with the cancer specialist, not ‘this doctor’, about what kind of chemotherapy to take. When asked directly about choices offered by his PCP, he talked about this doctor recommending postponing treatment for Hepatitis C for a year until chemotherapy for cancer is complete.
Another talked about a discussion about whether or not to have optional surgery.
Choices in where to go for diabetic treatment and what form to take insulin.
Taking medicines or discussions about lifestyle changes, e.g., diet and smoking cessation.
All subjects were asked to read this item aloud to see if they had any difficulty reading ‘pros and cons’: none did. They correctly defined pros and cons as meaning ‘the goods and the bads’; ‘good or adverse effects’; ‘benefits versus side effects,’ and ‘positives and negatives.’
*34. In the last 12 months, did this doctor ever give you educational materials by e-mail or on a website?
1 Yes
2 No If No, Go to Question 36
RAND Finding: One participant attended more to the delivery of materials rather than the mode of delivery. Upon probing it became clear that the participant was referring to paper materials (brochures or pamphlets) rather than material sent through electronic media. Two participants included the doctor’s verbal referral or reminder to view topic specific educational material on Kaiser’s web site when answering “yes.”
Yale Finding: Two of the Yale participants responded ‘yes’ but were referring to pamphlets rather than information
*35. In the last 12 months, how often were these educational materials useful to you?
1 Never
2 Sometimes
3 Usually
4 Always
RAND Finding: One participant could not answer this question and described educational material he receives at each visit about smoking cessation and reports that he has no desire to quit smoking.
Yale Finding: The subjects had no difficulty answering this question, albeit, some were not answering about electronic information.
37. In the last 12 months, when this doctor ordered a blood test, x-ray or other test for you, how often did someone from this doctor’s office follow up to give you those results?
1 Never
2 Sometimes
3 Usually
4 Always
RAND Finding: This item is problematic when online posting of results is the only mode of receiving test results offered by the practice or selected by the patient. As one patient put it, the doctor made it clear he was “off the hook” for following up by directing her to seek results on the web site. As a team, do we think that the doctor (or office) should follow up regardless of how the results delivered to make sure the patient saw the results and understood the results?
Yale Finding: One subject answered ‘never’ saying that she ‘doesn’t hear from them unless something is wrong.’ On further probing it was clear that she had been informed about worrisome findings; she should have responded ‘sometimes.’
*40. In the last 12 months, did you ever use e-mail or a web site to ask this doctor to refill or renew one of your prescriptions?
1 Yes
2 No
RAND Finding: Responses to probes indicated that interpretation of “refill or renew” was consistent with our design goals (e.g., capturing authorized refills to existing prescriptions or requests to renew a prescription without an existing refill). One participant ignored or didn’t hear the reference to “this doctor” in this item, and answered “yes” based on his experience logging into his health plan’s site to refill an existing prescription at the authorized online pharmacy. Another participant interpreted “ask this doctor” to include having the pharmacy ask the doctor.
Yale Finding: All participants understood what it meant to refill or renew a prescription. Those who affirmed the question gave appropriate examples of making these requests.
*41. In the last 12 months, did you ever use e-mail or a web site to ask this doctor to prescribe a new medicine for you?
1 Yes
2 No
RAND Finding: No findings noted.
Yale Finding: Two people, who answered appropriately, didn’t think that a doctor would prescribe a new medication without seeing the patient in person. One of these patients also wouldn’t want to talk about a new medicine in an ‘impersonal medium’; they wanted to talk face-to-face or at least on the telephone.
Another issue is whether or not a change in dosage counts as a ‘new medicine’. One subject answered ‘yes’ because she had requested an increased dosage.
*42. In the last 12 months, how often was it easy to fill prescriptions from this doctor?
1 Never
2 Sometimes
3 Usually
4 Always
RAND Finding: This item captures pharmacy experience (that is, slow or problematic delivery by the pharmacy), as well as problems communicating with the doctor’s office staff about a prescription (such as calls to the office, asking them to call the pharmacy).
Yale Finding: All participants responded thinking only about their interactions with their doctors around filling prescriptions. They included making requests by email, website, and in person.
*44. During your visits in the last 12 months, did this doctor ever use a computer or hand held device to look up test results or other information about you?
1 Yes
2 No
3 Don’t know
*45. During your visits in the last 12 months, did this doctor ever use a computer or hand held device to show you information?
1 Yes
2 No
RAND Finding: Findings here are consistent with our design goals. That is, participants perceived Q44 and Q45 as asking about two different ways the doctor might use a computer. Hence, both items are potentially useful.
Yale Finding: As with RAND’s findings, there were no problems with these items. Examples of a ‘computer or hand held device’ included: iPod, Blackberry, PDA, and Palm pilot. Only one person asked what a hand held device was and when asked what came to her mind, responded ‘some kind of hand held computer or something? – the only thing that comes to my mind is a Star Trek tri-coder.’
*46. In the last 12 months, did this doctor ever use a computer or a hand held device to order your prescription medicines?
1 Yes
2 No
3 Don’t know
RAND Finding: Three participants answered “yes” to Q46. Two “yes” responses indicate participants who watched the doctor transmit the prescription via computer during a visit, 1 “yes” response indicates a participant who e-mailed a doctor and believed the doctor then e-mailed the pharmacy. One participant answered “don’t know” as “I am not there when he orders prescriptions.”
Yale Finding: Two participants affirmed Q46 when the physicians typed their prescriptions into a computer and handed them the printouts to take to the pharmacy. The other three answered thinking about prescriptions that are delivered to the pharmacy electronically.
*47. During your visits in the last 12 months, was this doctor’s use of a computer or hand held device helpful to you?
1 Yes, a lot
2 Yes, a little
3 No, not at all
*A47 Using any number from 0 to 10, where 0 is not helpful to you and 10 is extremely helpful to you, what number would you use to rate this doctor’s use of computers during your visits in the last 12 months?
RAND Finding: Participants identified that these questions ask about the same topic. Four preferred Q47 to A47, describing it as shorter and easier to read and answer. One participant felt Q47 should be a yes/no question (“it’s either helpful or not”). One participant preferred A47 to Q47 as he felt it was easier to understand.
Yale Finding: All but one person thought that the two questions were asking about the same thing. The outlier responded to the 0-10 version as if it were a summary rating for her doctor and ignored the qualifier about this doctor’s use of computers. Another person attended to the different time frames in the items: last 12 months versus the nonspecific present that he called ‘overall during your last several visits.’ Most preferred the Q47 to A47.
*A48. In the last 12 months, did this doctor’s use of a computer make it harder or easier for you to talk with him or her?
1 Harder
2 Not harder or easier
3 Easier
*48. During your visits in the last 12 months, how often was it harder for you to talk with this doctor because he or she was using a computer?
1 Never
2 Sometimes
3 Usually
4 Always
RAND Finding: One participant identified these questions as asking about different things. To this participant, Q48 referred to use of a computer during a visit (due to specific reference to visits) and A48 referred to whether or you can communicate with the doctor e-mail (since visits weren’t specifically mentioned). The remaining 4 participants identified that these two questions ask about the same thing. Two preferred Q48 and two had no preference. One participant was confused during the probing about the “direction” of 48. She answered “always” but on probing realized she should have answered “never.”
Yale Finding: All of the participants understood the intent of these questions, with one characterizing it as: ‘is the computer a walkway or a wall?’ One person thought that Q48 was ‘leading’ because it only asks about computers making it harder to communicate with physicians. There was no clear preference for one version over the other.
*51. Does this doctor’s office put your laboratory or other test results on a web site for you to see?
1 Yes
2 No If No, Go to Question 56
3 Don’t know If Don’t know, Go to Question 56
RAND Finding: “Laboratory tests” and “lab tests” are perceived as identical. When asked to identify what counted as “lab tests” participants identified blood tests, urine tests, EKGs, stress tests, MRIs, and “all other tests.” No participants identified the item as including X-rays.
Yale Finding: The responses of these participants nearly mirrored those of RAND’s. They didn’t see any difference between ‘lab tests’ and ‘laboratory tests.’ One subject, however, included X-rays. Other test results not mentioned by RAND participants, but noted by Yale’s included CT scans, TB tests, biopsies, throat cultures, Hep C tests, pregnancy tests, and allergy testing.
*53. In the last 12 months, how often was it easy to find these lab or other test results on the web site?
1 Never
2 Sometimes
3 Usually
4 Always
RAND Finding: One participant couldn’t answer this question as it is easy to find the test results once they are posted, but sometimes she is not sure when to start looking for the results (which makes it difficult). We may want to consider wordsmithing Q53 to make explicit reference to “once tests are posted.”
Yale Finding: One respondent ignored the qualifier about the web and responded ‘always’ thinking about emails from her doctor.
*56. Does this doctor’s office put a list of your prescription medicines on a web site for you to see?
1 Yes
2 No If No, Go to Question 59
3 Don’t know If Don’t know, Go to Question 59
RAND Finding: One participant answered “don’t know” as she is not currently taking any prescription medicines and wasn’t sure if her doctor’s office could post prescription medicines. We may want to consider screening for prescription medicine use before asking Q56.
Yale Finding: One subject thought this question was asking if the patient brings a list of the medications s/he is taking to the doctor. Another responded ‘no’ because he does not ‘want any kind of personal information put on a website.’ The remaining three participants answered appropriately.
*58. In the last 12 months, how often was the list of prescription medicines up to date?
1 Never
2 Sometimes
3 Usually
4 Always
RAND Finding: No reported difficulties.
Yale Finding: Two participants considered the medication list to be ‘always’ up to date even if it included prescriptions that the patients are no longer taking. The key criterion appears to be if current medicines are on the list.
*59. Some doctors give patients their notes about what was discussed during the visit. Does this doctor give you visit notes?
1 Yes
2 No If No, Go to Question 64
3 Don’t know If Don’t know, Go to Question 64
RAND Finding: Three participants who reported receiving online after visit notes when screened to participate in the interview answered “no” to this item. Those participants interpreted the item as asking only about written, paper notes. Two patients who receive online summaries that follow a standard template indicating reason for visit, doctor’s recommendations and other similar info didn’t answer “yes” – in one case it was due to the use of the term “notes”. We may want to revise this item to ask about information summarizing the visit.
Yale Finding: Three participants missed the meaning of this item. One thought only about a note that the doctor would write for her if she were “applying for a disability.” Another thought the question was asking about how ‘forthcoming’ the doctor is. The third thought that these would be notes written ‘outside of what is inputted into the computer and the doctor has never shared those notes with me.’
*64. Do you ever use the Internet or World Wide Web, or send and receive email?
1 Yes
2 No
RAND Finding: Two participants attend more to the “send and receive email” portion of the question and identified the item as asking whether or not they use email rather than asking about use of email and all use of the Internet. We may want to simplify item wording (Do you ever use the Internet or email?”) or split Q64 into two questions, one on use of email and one on (other) use of the Internet.
Yale Finding: The reference to the World Wide Web was confusing for one participant. When asked to rephrase the question in her own words, she declined saying, ‘That’s going beyond my comprehension … World Wide – h have hear of that, but I don’t know what it is.” She does receive email, but is not sure how that is related to the WWW.
| File Type | application/msword |
| File Title | MEMORANDUM |
| Author | Julie Brown |
| Last Modified By | vhabedmcinnk |
| File Modified | 2009-09-01 |
| File Created | 2009-09-01 |