2 CPCC #20 Facilitator's Guide

U ser-Centered Design, Inc. 20548 Deerwatch Place Ashburn, VA 20147 (703) 729-0998

Attachment 5. Facilitator’s Guide for

Usability Testing Cancer Prevalence and Cost of Care Projections Website

November, 2012

Submitted by:

User-Centered Design, Inc.

20548 Deerwatch Place

Ashburn, VA 20147

(703) 729-2370

www.user-centereddesign.com

OMB No.: 0925-0642 Expiration Date: 9/30/2014

Collection of this information is authorized by The Public Health Service Act, Section 411 (42 USC 285a). Rights of study participants are protected by The Privacy Act of 1974. Participation is voluntary, and there are no penalties for not participating or withdrawing from the study at any time. Refusal to participate will not affect your benefits in any way. The information collected in this study will be kept private under the Privacy Act. Names and other identifiers will not appear in any report of the study. Information provided will be combined for all study participants and reported as summaries. You are being contacted by phone or in person to complete this instrument so that we can help improve the website.

Public reporting burden for this collection of information is estimated to average 60 minutes (for the usability test) per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0642). Do not return the completed form to this address.

[Note: The purpose of this document is to guide the moderator. The questions and tasks contained herein may not be asked as written. The facilitator often draws on participant comments and the natural flow of the testing process to determine the flow of the session. While the facilitator will try to follow the order of the guide, many times tasks will come up ahead of time or in different order. The facilitator may allow the order of the tasks to change in order to let the process flow naturally.]

Pre-Test

[Administer the informed consent]

Introduction

Thank you for agreeing to participate in this study.

The NCI is aiming to improve their Cancer Prevalence and Cost of Care Projections website, so today we are conducting a usability evaluation of the design. As a participant, you will explore the website and be given a set of tasks to do.

As we go through the tasks, feel free to offer any comments or suggestions that occur to you. We are looking for things about the design that are working well as well as things that could be improved. There are two important things you should keep in mind while you work with the website:

• First, I did not create the design so you can’t hurt my feelings with any criticisms you might have. If there are problems with the design I would like to discuss them with you to see if we can find a way to improve the design.

• Second, we are evaluating the website and not you, so you cannot make any mistakes. The site is supposed to be intuitive and easy to use. If it isn’t, that’s a problem with the website – not with you.

I’d like you to know that there are some observers with us today helping me by taking notes, but don’t worry about them. You and I will work on this together and they’ll just watch and listen.

Do you have any questions for me before we get started?

Background

Before we actually get started with the website, I’d like to know a little bit about your background.

• Where do you work?

• What sort of work do you do?

• In your work do you search for cancer prevalence data? Cost of care projections for cancer? Tell me how that fits into your work.

Site Usage

• [Open the website http://costprojections.cancer.gov so users can see it so they will better remember what they’ve done. Tell them not to use it yet.] Have you ever used the Cancer Prevalence and Cost of Care Projections website? [If yes:] When was the last time you used the site? On average, how frequently do you use the site? [If no, skip to tasks.]
  
  

• How long have you been using the website?

  • How did you initially find the website?

  • Did anything confuse you when you first started using this website?

  • Was it easy for you to determine how it operates?

  • What happened during your initial visits?

  • Have you used the site enough that you’ve changed your purpose in using the site?

  • What have your experiences taught you about using the website?

• Let’s see if you can tell me some specifics about how you usually use the website.

  • When you look at projected cancer information, do you usually look at specific cancers or just “cancer,” per se?

  • When you look at projected cancer information, do you look at the phase of cancer care (i.e., initial, continuing, and last) or do you look at overall care?

• Tell me about the data that you obtain from the website. How do you use the data?

Tasks

Now I’m going to ask you to do some specific tasks. Feel free to ask any questions that come to mind as you work, but I might not answer right away because I’d like to see if the design lets you find the answer on your own.

1. Explain to me the content on the homepage. [Probe on whether they use the information on the homepage directly.] Show me what you [would] usually do on the homepage. [Do you usually access more than one of the sections? If not described previously, ask “Do you have a routine in the way you use various links on the website?”]

1. [Previous users only:] What areas on the site have given you problems in the past? [Are there places where you get “stuck”?] Tell me about them. If you remember something later as we explore the site, feel free to mention it then.

2. Where would you look for costs of cancer?

   1. Find and compare information on the national projected expenditures in 2020 that lists breast and prostate cancers. [Note if P goes to the National Expenditures for Cancer Care table.]

   2. How would you use the information in this table? [Probe on terms such as base values, cost values, trend incidence, etc. Note P’s understanding of how the site defines the terms.]

   3. (Do they understand all parts of the graph? Base 2010/2020, trend columns, 2% increase section, etc.)

3. Where would you look for cancer prevalence data? [Note if the P goes to the Cancer Prevalence table.]

   1. Take a look at the data for colorectal cancer. There are four different columns of prevalence data. Which one(s) would you use in your work? [Probe on whether P realizes the labels (e.g., base and trend) have the same definitions as they do in the National Expenditures for Cancer Care table.]

   2. How would you use that data?

4. Where would you look for annual costs of care data for cancer? [Note if P goes to the Average Annual Costs of Care table.]

   1. What do you think the site means by “costs of care?” How does your definition of “costs of care” compare to the site’s?

   2. How do you think the site divides initial, continuing, and final years costs? Is that division helpful to you?

   3. Using the data, can you tell which cancer has the greatest average annual cost of care for the last year of life? [Probe on if the P understand the difference in data between death from cancer and other causes of death.]

   4. The data on this chart is adjusted for patient deductibles and coinsurance expenses. What does this mean to you?

5. Let’s go into the area where you can create your own graph. Create a graph that would show information that would be interesting or meaningful to you. [Probe that the P understands each selection and how it influences the output.]

   1. Then let’s see the graph. [Once the graph has been rendered] I’d like you to tell me the meaning of the graph you’ve created.

   2. Is there anything you would change about how the graph looks? [Probe on formatting (e.g., preference for bar graph), labels, etc.]

Post Test Interview Questions

Great, now I’d like to ask you a few questions.

1. This site displays data in graphs and tables. How would you use these? [Probe on whether they would put it into a report or briefing, or just use the data itself.]

2. Is there value in having access to the raw data used to display these graphs and tables? If so, to which data would you want to have access? [Probe to see if the P realized the site provides this capability for some material.]

3. Most websites have a learning curve associated with them. You [have used| haven’t used] the present website very much, right? How hard or easy was it for you to learn this website? Do you think that others with your background will have about the same experience on the website?

4. Would you reflect on the things you liked and didn’t like about the website? What three things did you like the most? What three things did you like the least?

5. [If not answered earlier:] Do you think that individuals in your line of work need additional resources within the site? Where or what sort of additional information would you need?
   
   

6. Are there other information resources that you depend on to meet the information needs that you get answered here? How does this website compare to those resources?

Wrap up

OK, we’re done. Are there any questions you would like to ask me about the website or about today’s experience that I did not ask you about?

Thanks again for your participation.

0