7 Attachment 5 Usability Testing

Attachment #5

Participant #________________

Sample Moderator’s Guide for NCI Open Call Usability Testing

Note: The questions specific to the communication/website material (such as navigation, functionality, graphics, page layout, etc.) may change a bit for each round of Open Call Usability testing, depending on the specific material being tested. The following questions would have to be customized to the material to be tested at that time.

Study Information

Date
Time
Moderator
User group
Additional info

5 min	Introductions: Interview start
5 min	Initial Interview
15 min	Navigation Task
15 min	Functionality Task
15 min	Design & Graphics
5 min	Thank you: Interview end

Checklist

______ Participant number assigned

______ Consent form signed

______ Conference line setup and ready

______ Go To Meeting setup and ready

Introduction

Thank you for participating in this study today. My name is _____, and I work with the Communications Group within the National Cancer Institute. We help to improve NCI’s websites and other materials. I am going to ask you some questions about different NCI materials and provide you with a few tasks that we’d like you to try to complete. The questions and tasks help us to better understand how people find and use cancer information so that we can provide suggestions on how to make the websites and materials better. The entire session will last about an hour.

Here are some things to know before we get started:

• As we create new sites, we meet with many different people, asking them to try them out and help us improve them. This means that there are no right or wrong answers. We are here to learn from you today, so please share any of your reactions with us. Keep in mind that difficulties you may run into reflect the design of the website and materials, not your skills or abilities.

• In order for us to get a better idea on what to suggest improving, we would you like you to think out loud when responding to the questions and working on the tasks. We’d really like to hear if you like or dislike something, what you are expecting to see on a page, where you are expecting a link to take you, and why you are going to a page or section for information.

• You and I will be working together for one hour, and _____ will be taking notes. There are also some people observing from the other room. We did not create the materials we are testing, so please do not feel like you have to hold back on your thoughts to be polite. Please share both your positive and negative reactions to the site.

• If you do not want to answer any of my questions, you do not have to. Please just say so, and we will move on. And of course, you can stop at any time.

• We previously mentioned that this session will be recorded. The recording will be of your voice and the activities we are doing on the computer and mobile device. Your face will not be recorded. The recording is primarily to help us take notes, but we might show anonymous clips to people working on these products. Is that all right with you?

• On parts of today’s session we will be tracking where you are looking on the screen to help us understand how you read the information. To do this, I will need to set up the eye tracker.

• Do you have any questions?

Initial Interview for Lay Audience: Patients, Caregivers, Friends, & Family

Web and health information use

Before we get started, I would like to learn a little bit about you. Let’s start with how you look for health information.

I understand you have had a recent experience with cancer? Can you tell me a little bit about that?

Did you use the web to find information that would help you understand this disease better and make decisions?

Have you ever heard of NCI? [ ] Yes [ ] Not sure [ ] No
If you had to describe NCI to someone, what would you say it is?

Have you ever been to the NCI web site, cancer.gov? [ ] Yes [ ] Not sure [ ] No
(if yes) Tell me about the last time you were there.

Do you use social media sites, like Facebook or Twitter? Do you use them to find health information? Tell me a little bit about that.

What kind of computer do you use to look for health information (home computer, work, computer, mobile device, phone, tablet, etc.)?

Are there sites that you found particularly useful? What did you like about them?

Do you go back to [those sites] as a starting point for new information? (Or just search)

Initial Interview for Healthcare Professionals

Web and health information use

Before we start, I would like to learn a little bit about you. Tell me a little bit about the work you do.

How about examples of topics that might end up on your radar in as they relate to your work?

What types of information do you use (web, social media, news media, government sources, PubMed, academic sources)?

What kind of computer do you typically use for this work (home computer, work, computer, mobile device, phone, tablet, etc.)?

If you were looking for information specifically about healthcare, cancer, or health research, are there particular sources you find most useful? What did you like about them?

Do you go back to [those sites] as a starting point for new information? (Or do you just search again)

Have you ever heard of NCI? [ ] Yes [ ] Not sure [ ] No

If you had to describe NCI to someone, what would you say it is?

Have you ever been to the NCI web site? [ ] Yes [ ] Not sure [ ] No

(If yes) Tell me about the last time you were there.

Navigation Task

Test goal: Receive feedback about the navigation and ease of use of the site.

Ok, great, we are ready to get started.

Open http://cancer.gov

1. What is your first impression of this page? (Allow 30 seconds).

2. Which content would you explore first? Is there content on this page that particularly interests you?

Thank you. Now I’m going to give you a few tasks to complete. The menus and some of the links will not work, but I want you to do what you normally would do. You can scroll and click as if this were a working website.

Task 3 Purpose: Assess ease of navigating to information about progress in cancer research.

3a. Let’s say that you are looking for information about recent cancer research discoveries. For this task, please find information about recent scientific discoveries on pancreatic cancer. (Allow 2 minute) Alternative: Let’s say that you are looking for information about recent cancer scientific discoveries. For this task, please find information about recent scientific cancer discoveries. (Research is the primary labels so having the word research in the task makes this a search task).

Correct progression: Research  Progress  Cancer Story

Time to find ____:__________ Number of steps _____

Accurate? Yes or No | Participant Gave Up? Yes or No | Time Expired? Yes or No

Please rate your experience with completing the task:

Not Difficult At All	Slightly Difficult	Moderately Difficult	Very Difficult	Extremely Difficult
1	2	3	4	5

What made the task __________ (insert participant’s rating)?

3b. Is there other content on the Research page that draws your attention?

Task 4 Purpose: Assess ease of navigating to information about training opportunities.

4. For healthcare professionals. Please return to the home page. Let’s say that you have heard that the website has information about training opportunities available through NCI. Where would you look to find this content?

Time to find ____:__________ Number of steps _____

Accurate? Yes or No | Participant Gave Up? Yes or No | Time Expired? Yes or No

Please rate your experience with completing the task:

Not Difficult At All	Slightly Difficult	Moderately Difficult	Very Difficult	Extremely Difficult
1	2	3	4	5

What made the task __________ (insert participant’s rating)?

Debriefing

1. What do you think about the Cancer.gov site? Is this something you would use?

2. Please return to the Research page. What do you think about the research page? Would this page be useful for you? What kind of content would you like to see on this page?

3. Is this what you expected when you clicked on ‘Progress.’ What do you think about how the navigation is organized on the left hand side. What would you expect to find under ‘Annual Plan’ and ‘Budget.’

4. Please return to the homepage. What do you think about the home page? What do you think about the layout of the home page? What do you think about the labels used on the home page? Which labels are not clear to you? What do you think about the sections on the home page? Are there any sections that you think are missing or not necessary?

5. What do you think about the Funding & Training page? Would you visit this page for any reason? What do you think about the labels used on this page? Which labels are not clear to you? What do you think about the sections on the page? Are there any sections that you think are missing or not necessary?

6. What do you think is the best thing about the Cancer.gov prototype?

7. What do you think is the worst thing about the Cancer.gov prototype?

8. If you could make one change to the Cancer.gov site or prototype, what would it be?

Functionality Task

Test goal: Assess the functionality of the site by observing how participants complete goals that are common among site visitors.

Cancer Research

1. Using this website, please locate basic information about cancer treatment. When you think you have found it, please tell me that you’re finished.

2. Now, please find information about side effects for radiation and chemotherapy. When you think you have found it, please tell me that you’re finished.

Time to find ____:__________ Number of steps _____

Accurate? Yes or No | Participant Gave Up? Yes or No | Time Expired? Yes or No

Please rate your experience with completing the task:

Not Difficult At All	Slightly Difficult	Moderately Difficult	Very Difficult	Extremely Difficult
1	2	3	4	5

What made the task __________ (insert participant’s rating)?

Cancer Types Pages

3. What did you think about the layout of this page?

4. Did you notice the Patient and Health Professional button at the top here? What would you expect to happen if you clicked either one of those buttons?

1. (Click on it, and probe further.) What do you think? What is missing?

5. How would you go about printing or sharing content on this page?

6. Did you notice these boxes at the bottom? What do you think about them?

7. Do you see any content on this site that seems relevant to you? Where would you click first on this page?

Debriefing

1. What do you think of the content and the layout of this page?

2. How do you feel about the options on this menu [point to Mega Menu] under the About Cancer heading?

3. Do you think you would use the content on this page?

Design and Graphics

Test goal: Assess ease of understanding and graphic illustration comprehension between prototype A and B.

1. Let’s say that you are looking for information about recent scientific discoveries about the causes of cancer. For this task, please find information about recent scientific discoveries on pancreatic cancer. (Allow 2 minutes)

2. Please explain the images that you see on this page. What is the difference between the malignant and non-malignant tumor cells?

Time to find ____:__________ Number of steps _____

Accurate? Yes or No | Participant Gave Up? Yes or No | Time Expired? Yes or No

Please rate your experience with completing the task:

Not Difficult At All	Slightly Difficult	Moderately Difficult	Very Difficult	Extremely Difficult
1	2	3	4	5

What made the task __________ (insert participant’s rating)?

Remember to counter balance prototype A and B.

For prototype A or B

3. Now I am going to show you a different version of this same page. Please find information about recent scientific discoveries on pancreatic cancer. (Allow 2 minutes)

4. Please explain the images that you see on this page. What is the difference between the malignant and non-malignant tumor cells?

Correct progression: Research  What is cancer?

Time to find ____:__________ Number of steps _____

Accurate? Yes or No | Participant Gave Up? Yes or No | Time Expired? Yes or No

Please rate your experience with completing the task:

Not Difficult At All	Slightly Difficult	Moderately Difficult	Very Difficult	Extremely Difficult
1	2	3	4	5

What made the task __________ (insert participant’s rating)?

Remember to counter balance prototype A and B.

5. For healthcare professionals. Please return to the home page. Let’s say that you have heard that the website has up to date information about grants that are available through NCI. How many grants were awarded in the current year and the previous year?

Correct progression: Grant Awards by year

1. Now I am going to show you a different version of this same page. Please find information about grants that are available through NCI. How many grants were awarded in the current year and the previous year?

Correct progression: Grant Awards by year

Time to find ____:__________ Number of steps _____

Accurate? Yes or No | Participant Gave Up? Yes or No | Time Expired? Yes or No

Please rate your experience with completing the task:

Not Difficult At All	Slightly Difficult	Moderately Difficult	Very Difficult	Extremely Difficult
1	2	3	4	5

What made the task __________ (insert participant’s rating)?

Debriefing

1. Please return to the fist pancreatic cancer page we visited. What do you think about the images and graphics of the first pancreatic cancer page?

2. Please return to the second pancreatic cancer page we visited. What do you think about the images and graphics of the second pancreatic cancer page?

3. Do you have a preference in terms of the images and graphics between the two pages?

4. Were any of the images in the pages offensive?

5. What did you think about the colors used on each page?

6. Which page images were easier to understand?

7. How would you use the images and graphic illustrations?

For healthcare professionals.

8. Please return to the fist grants page we visited. What do you think about the images and graphics on the first grants page?

9. Please return to the second grants page we visited. What do you think about the images and graphics of the grants page?

10. Do you have a preference in terms of the images and graphics between the two pages?

11. Do you have a preference in terms of the images and graphics between the two pages?

12. Were any of the images in the pages offensive?

13. What did you think about the colors used on each page?

14. Which page images were easier to understand?

15. How would you use the images and graphic illustrations?

Wrap-up Session End

Thank you for participating in this study. Your feedback has been very helpful. Do you have anything you want to mention about the website and materials you have seen today that we have not already talked about?

Ok, well thank you again.

Sample Moderator’s Guide in English—Open Call 11 | Page