Form #3 Form #3 Usability Testing Interview Guide

A

OMB No. XXXX-XXXX
Exp. Date XX/XX/20XX

ttachment D: Interview Guide – User Testing

AHRQ Health IT Tool Evaluation

Usability Test Interview Guide

(for Site Visitors and Non-Visitors)

Initiate WebEx session and open another window with URL for AHRQ main site;

minimize it while doing the introduction

Introduction

Hi, and thanks for agreeing to participate in this session today. My name is [moderator’s name] and this is [Observer’s name, if applicable] and we work for Westat. Today we’d like to get your reactions to a tool that offers information to support evaluation of health IT efforts.

The Agency for Healthcare Research and Quality (AHRQ) is conducting an evaluation of tools available on its National Resource Center for Health IT Web site. We are interested in identifying areas for improvement, so we are seeking feedback from people who [might use it / have used it]. Your responses will be kept confidential to the extent permitted by law, including AHRQ’s confidentiality statute, 42 USC 299c-3(c). The results will be reported in aggregate, so your name will not be associated with anything you say.

Today’s session is completely voluntary and is scheduled to last 90 minutes. If you would like to stop at any time, please tell me and we will stop immediately.

Explain feature to record, turn it on, and then obtain consent to record session

If respondent does not agree, explain that it is for record-keeping only so that you do not miss any of the information; if the respondent still does not agree, continue and take detailed notes.

Public reporting burden for this collection of information is estimated to average 90 minutes per response, the estimated time required to complete the survey. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: AHRQ Reports Clearance Officer Attention: PRA, Paperwork Reduction Project (XXXX-XXXX) AHRQ, 540 Gaither Road, Room # 5036, Rockville, MD 20850.

I am going to ask you to do some tasks using a Web-based tool called the AHRQ Health IT Survey Compendium. There are no right or wrong answers for these tasks. I am simply interested in how you go about completing each task and how easy or difficult you find the site to use. We are not testing you, we are evaluating the Health IT Survey Compendium and identifying possible ways to improve it. I did not design the site, so you won’t hurt my feelings if you criticize it. All of your positive and negative feedback is welcome. We will save some time at the end of the session to cover your general impressions about the site.

For the first part of the session, we will do what is called a “think-aloud.” As you work on the tasks, please talk out loud to describe what you are thinking as you look at information on your screen and make decisions at each step. For example, you might say, “I am looking for a way to locate a tool related to hospitals. Maybe it’s this option to select ‘Setting’…” I think I’ll try that.”

Do you have any questions before we begin?

Probes for Use While Participants Work on Tasks

General probes; follow-up probes to use as needed are bulleted

What do you expect to happen at this point?

Is this what you expected to see, or not? Why?

Probe when participant first encounters results list

What is your reaction to the way the results are presented?

What do you think about the [4] categories you have to work with?

Probe if participant encounters no results for his/her search:

What do you think about this outcome?

Probe if participant is confused:

• Why do you think that happened?

• Tell me more about that…

• What would you do next?

Tasks

Maximize AHRQ National Resource Center for Health IT homepage and pass mouse control to participant.

Have you ever used the AHRQ Health IT Survey Compendium?

If yes, participant has used survey compendium, start with task A (user self-directed task).

If no, participant has not used survey compendium, go to numbered tasks.

Select a task from 1 – 8 below and ask participant to start with that task; randomize from then on until all 8 are completed.

A. [USER TASK] What have you used the Health IT Survey Compendium for? Starting at the AHRQ National Resource Center for Health IT homepage, could you please show me what you did when you visited the Health IT Survey Compendium?

If participant completed task a, skip to task 2

1. A co-worker mentioned finding a tool sponsored by the Agency for Healthcare Research and Quality (AHRQ) that provides a variety of surveys for evaluating various aspects of health information technology systems. Starting from the AHRQ National Resource Center for Health IT homepage, find a page where you can access these publicly available surveys.

Title of page: ___________________________

Moderator, record participant’s final answer

If participant does not get to survey compendium page on own, take mouse control and set to that page for continuing on to next task while on Health IT Survey Compendium homepage.

• In your opinion, what is the purpose of this page?

• What do you think you can do on this page?

2. Suppose you want to conduct a study to find out what nurses in a hospital think about the electronic medical record system that their hospital is using. Where would you look on this tool for examples of how you could collect that type of information?

One example: _________________

Moderator, record participant’s final answer

3. You are a researcher and plan to conduct a survey of patients to evaluate their satisfaction with the usability of a personal health record (PHR) system. Find a survey that you could use or adapt for this purpose.

One survey: _________________

Moderator, record participant’s final answer

4. You are planning to conduct a series of one-on-one discussions with clinicians about the frequency of alerting in clinical decision support systems. To help you structure your questions, you would like to see some examples of guides that others have developed and used. Find an example that you might use as a model for structuring your own guide.

One example: _________________

Moderator, record participant’s final answer

5. Your colleagues have developed a questionnaire to assess patients’ views about health information exchange – i.e. clinicians sharing their medical information through electronic networks. They would like to get other practitioners to use the questionnaire to collect data on a broader scale. What information does this tool provide that your colleagues could use to get AHRQ to consider adding their questionnaire to the collection?

Information that you could provide to colleagues: _______________________

Moderator, record participant’s final answer

6. You are a health care administrator who recently implemented a new health IT system [system of your own choosing]. Initial feedback indicates that the system users are unhappy that no needs analysis was conducted prior to implementation and are dissatisfied with the new system. Find a survey you could use to collect information about how to improve the new system.

One survey: _______________________

Moderator, record participant’s final answer

7. Participant directed task – to learn how participants understand and use the survey compendium.

Please think of a health IT project you are currently involved with, or have been involved with in the past. Use the Health IT Survey Compendium to find a tool to help you solicit evaluative feedback about some aspect of that project.

Prompts to use if participant has difficulty thinking of related project

• As someone who implements health IT projects, would you ever solicit feedback from the end-users of one of your project to evaluate how they like it?

• Could you give me an example of something you worked on for which you would like to get follow-up information? What kind of system was it? Who were the user groups? Where would you look on this tool for ideas for how to go about getting feedback on [project participant describes]?

One tool: _______________________

Moderator, record participant’s final answer

Additional Usability Topics

8. Please review the options in the “Survey Type” dropdown menu. What would you do if you didn’t know what one of these terms meant?

9. Let’s look now at those options provided for “Survey Type.” What do each of these terms mean to you?

1. Attitude assessment

2. Benefit

3. Current state assessment

4. Functionality

5. Interview guide

6. Needs assessment

7. Other

8. Perception

9. Questionnaire

10. Satisfaction

11. Usability

12. Vendor RFI

10. Please review the options in the “Care Setting” dropdown menu. What do each of these terms mean to you?

a. Across the health care system

b. Ambulatory

c. Home

d. Inpatient

e. Not specified

f. Pharmacy

g. Telehealth

What would you do if you didn’t know what one of these terms means?

11. Please review the options in the “Technology” dropdown menu. What do each of these terms mean to you?

1. Bar coding

2. Clinical decision support

3. Clinical documentation

4. Computerized provider order entry

5. Disease registry

6. Electronic health records

7. Electronic medication administration

8. Electronic prescribing

9. Enterprise systems

10. General computing

11. Health information exchange

12. Not specified

13. Personal digital assistants

14. Personal health records

15. Pharmacy information systems

16. Picture archiving and communication system

17. Practice management systems

18. Secure messaging

19. Telehealth

What would you do if you didn’t know what one of these terms means?

12. Please review the options in the “Respondent Type” dropdown menu. What do each of these terms mean to you?

1. Administrators

2. All clinical staff

3. Everyone

4. IT staff

5. Implementers

6. Not specified

7. Nurse

8. Office staff

9. Patient

10. Pharmacist

11. Physician

12. Researchers

13. Vendors

What would you do if you didn’t know what one of these terms means?

When finished with tasks, move on to debriefing questions

We have finished with the think-aloud part of the session. Now I’d like to ask you some questions about your experiences using the Health IT Survey Compendium.

Debriefing

Overall Impressions of Survey Compendium and Satisfaction

1. What are your general impressions of the Health IT Survey Compendium?

• What one or two things did you like most about it? Why?

• What one or two things did you like least about it? Why?

2. Overall, how satisfied would you say you are with the experience of using the Health IT Survey Compendium? Would you say…[READ SLOWLY] Very dissatisfied, Dissatisfied, Neither dissatisfied nor satisfied, Satisfied, or Very satisfied?

Audience Assessment

1. In your opinion, what types of people was this Health IT Survey Compendium developed for?

2. What types of people would find this Health IT Survey Compendium useful?

Survey Compendium Usefulness

1. What types of surveys are you most interested in finding on this site? Why?

• How would you apply the results of your searches?

2. Which types of surveys provided on this site are most relevant for your needs?

• Why are those most relevant?

• How would you apply them in your work?

3. Which types of surveys provided on this site are not relevant for your needs?

• Why not?

4. How will the surveys in this Health IT Survey Compendium help to address your needs for evaluating health information technology?

• How do you think you might apply these in your work?

5. What types of surveys are missing from the Health IT Survey Compendium?

• What types of surveys did you expect to see, but are not included?

6. What other sources of surveys have you used?

• How do those sources compare to the Health IT Survey Compendium?

7. How confident would you be in using surveys that you find in the Health IT Survey Compendium?

• What do you base that judgment on?

Survey Compendium Usability

Narrowing a Search

1. What do you think about the steps that you took to locate a survey?

• If positive: Why do you like [whatever participant mentions]?

• If negative: What would you suggest changing?

2. What do you think about the categories provided to work with here?

• What do you like about these 4 categories for narrowing your results?

• What do you dislike about these 4 categories for narrowing your results?

• What would you suggest instead?

• What would be more helpful for you?

3. What are your thoughts about the drop-down boxes for narrowing results?

If negative response:

• What other action would you like to be able to take here?

• What, if any, help do you need to work through this process to find results?

4. What other categories, if any, would help you to locate surveys of interest to you?

5. What would make the narrowing process more efficient for you?

Results

1. What do you think about how the results are displayed?

• The number of results?

• The order of results?

2. What additional information, if any, do you about surveys in order to decide which one(s) you want to look at?

3. What, if anything, would you suggest changing about the way the results are displayed?

Presentation and Layout

1. What do you think about the way that [each of these] is presented on the screen?

• Goals and Methods information/section

• Information about Survey Types

2. What do you think about the font size?

• Is it too big, too small, or just right? Why?

3. Do you have any comments about the layout of the pages?

Thank you very much for your time and assistance today. Your feedback has been very valuable.

Turn off recording feature. Then verify address for sending incentive and end the WebEx session.

HANDOUT

Participant Tasks

1. A co-worker mentioned finding a tool sponsored by the Agency for Healthcare Research and Quality (AHRQ) that provides a variety of surveys for evaluating various aspects of health information technology systems. Starting from the AHRQ National Resource Center for Health IT homepage, find a page where you can access these publicly available surveys.

Title of page: ___________________________

2. Suppose you want to conduct a study to find out what nurses in a hospital think about the electronic medical record system that their hospital is using. Where would you look on this tool for examples of how you could collect that type of information?

One example: _________________

3. You are a researcher and plan to conduct a survey of patients to evaluate their satisfaction with the usability of a personal health record (PHR) system. Find a survey that you could use or adapt for this purpose.

One survey: _________________

4. You are planning to conduct a series of one-on-one discussions with clinicians about the frequency of alerting in clinical decision support systems. To help you structure your questions, you would like to see some examples of guides that others have developed and used. Find an example that you might use as a model for structuring your own guide.

One example: _________________

5. Your colleagues have developed a questionnaire to assess patients’ views about health information exchange – i.e. clinicians sharing their medical information through electronic networks. They would like to get other practitioners to use the questionnaire to collect data on a broader scale. What information does this tool provide that your colleagues could use to get AHRQ to consider adding their questionnaire to the collection?

Information that you could provide to colleagues: _______________________

6. You are a health care administrator who recently implemented a new health IT system [system of your own choosing]. Initial feedback indicates that the system users are unhappy that no needs analysis was conducted prior to implementation and are dissatisfied with the new system. Find a survey you could use to collect information about how to improve the new system.

One example: _________________

7. Please think of a health IT project you are currently involved with, or have been involved with in the past. Use the Health IT Survey Compendium to find a tool to help you solicit evaluative feedback about some aspect of that project.

One tool: _________________

19