Form 3 #8_Attach 8A, 8E, 8F, & 8G_Email, Guide & Qx_NExT

#8 Sub-study - NExT Attachments:

8A. Recruiting Email/Screener

8E. Facilitator’s Guide for NExT Website Evaluation

8F. System Usability Scale (SUS)

8G. Modified Cooper Harper Difficulty Rating Scale

Attachment 8A: Recruiting Email

Dear Colleague

At National Cancer Institute (NCI) those of us who are responsible for the NCI Experimental Therapeutics Program (NExT) website are working to improve the website. We would appreciate your help. We are conducting a series of evaluations to determine appropriateness and usability and would like you to spend an hour participating in a usability test of the website. This would require no preparation on your part and you would be given $ 60 as a token of appreciation for your time if you participate.

If you agree to participate, our contractor, User-Centered Design, Inc. (UCD) will arrange a mutually convenient date and time in Month, Year to arrange a remote testing session. You will participate from you own location/computer using a web meeting software, GoToMeeting, and use VOIP or your phone for teleconferencing during the test (giving opinions, etc.). To accomplish this, you will need to install a small utility (directions will be provided).  The utility will enable UCD and our personnel to watch your experience with the website.

Your session feedback will be combined with the input from others and incorporated into tangible improvements in our website and communications efforts, but your individual comments will be kept secure to the extent permitted by law.

To assess your background, we are asking you to complete a short questionnaire that will take less than 10 minutes of your time.

OMB No.: 0925-0589-08

Expiry Date: 5/31/2011

Public reporting burden for this collection of information is estimated to average 10 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0589-08). Do not return the completed form to this address.

Please answer these four questions:

1. How did you become familiar with NExT? (check all that are appropriate)

Not familiar with NExT

Attended the NExT launch in 2009

Have e-mailed DCTD or the NExT program seeking information

Involved in anti-cancer drug discovery and development

DCTD grantee

DCTD contractor

Other, please describe _____________________________________________

2. What is your relationship with NExT?

Engaged with the NExT Program

Aware of but not engaged with the program

Don’t know about the NExT program

3. Which of the following describes your employment?

Academic, (what position and what institution?)

Government, (what position and what department?)

Pharmaceutical industry (what position?)

Other Industry (what position and what industry?)

4. Please describe job or position by providing the information noted above in parentheses.____________________________________________________________________

5. Although we have your name and email address, for purposes of contacting you for testing and sending you reimbursement afterwards, we will need the following information. If you prefer, we can collect this information after we have scheduled you to participate.

• Name _________________________________________________________

• Mailing Address ________________________________________________

• Phone Number __________________________________________________

Attachment 8E. Facilitator’s Guide for NExT Website Evaluation

[Note: The purpose of this document is to guide the moderator.  The questions and tasks contained herein may not be asked as written.  The facilitator often draws on participant comments and the natural flow of the testing process to determine the flow of the session.  While the facilitator will try to follow the order of the guide, many times tasks will come up ahead of time or in different order.  The facilitator may allow the order of the tasks to change in order to let the process flow naturally.]

OMB No.: 0925-0589-08

Expiry Date: 5/31/2011

Public reporting burden for this collection of information is estimated to average 50 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0589-08). Do not return the completed form to this address.

Pre-Test

[Administer the informed consent and video release form.]

Introduction

Thank you for agreeing to participant in this study. Do you have any questions for me before we get started?

You signed a video release so you know I’ll be recording everything that shows up on the computer monitor. We’re going to be looking at a website today that was created for the National Cancer Institute. I’m going to ask you to try to explore the website and then to try to do some tasks with the site. Then I’d like to get your opinion about it. You and I will work on this together for about an hour. Additional observers in another room may watch your monitor screen to assist me in taking notes.

We have invited individuals who have and have not experienced the NExT website. So we can understand the point of view that you are bringing to the testing, I’d like for you to tell me a little bit about your previous experience with or knowledge about the NExT website.

• Where do you work? (What have you worked previously?)

• What type of work do you do? (What have you done previously?)

• Have you ever visited and used the NExT website? How much have you used it in the last month? In the last year?

[Participant talks….]

There are two important things you should keep in mind while you work with thiswebsite:

• First, I did not design it so you can’t hurt my feelings. If there are problems with the design I would like to discuss them with you to see if we can find a way to make the website better.

• Second, we are evaluating the website and not you, so you cannot make any mistakes. The website is supposed to be intuitive and easy to use. If it isn’t, that’s a problem with the website – not with you.

Any comments you have, either positive or negative, will help make the product better so feel free to tell me whatever is on your mind. After you’re done working, I’ll ask you a few questions and then give you some time to ask me anything you’d like. Do you have any questions for me before we get started?

Before we get started, I’d like to let you know that as you’re using the site, if you have questions as we go along, feel free to ask. I may not answer your questions right away in order to see how well you can figure out things for yourself, but it is valuable to know if you do have questions about the site.

[Facilitator should ask about specific pages and sections as they are encountered. If users explore areas listed as a task below, the facilitator should get the feedback during their exploration. Areas of the site that seem to be of interest and then are dropped should be asked about.]

Tasks

All Participants

1. [Open http://cancer.gov] How would you get to the NExT program site from the NCI website?

2. [Open http://dctd.cancer.gov/] How would you get to it from the DCTD website?

[If participant can’t find the site, take them to http://next.cancer.gov]

[Ask participant if they have used the site before]

IF YES: Previous users of the site[As participants are looking at the site]

1. Could you please explain to me your understanding of the purpose of the NExT program?.

   1. Did you learn this from using the website or from some other source?

   2. Which parts of the site helped contribute to your understanding of what NExT is?

[Follow-up: Are there sections that need further elaboration regarding how this program works? Is the role of the PI (participant) in NExT projects that are approved clear? If not, what types of elaboration are needed and where?]

2. When was the last time you visited the site, and what did you use it for? [Ask user to take a look at it now]

   1. Which sections of the site did you use?

   2. What was helpful to you in accomplishing what you wanted to successfully?

   3. When you are on the site, do you typically notice the date of the latest update?

   4. Did you run into any problems? If so, what?

IF NO: New users of the site

1. First, I’m going to let you explore the website by yourself. I want to see what you can find that might be helpful or interesting to you. I’d like you to tell me about what you see and whether it does or does not relate to finding information useful to you. Let me know when you’ve finished looking at whatever interests you.

   1. [Observe the participants and note what they find on their own. Keep track of any relevant comments.]

   2. [If not offered, prompt for participant’s impression of the main page.]

   3. [If not already covered, ask what prompted them to explore the sections they visited.]

2. [If not already discussed] Could you please explain to me your understanding of the purpose of the NExT program?

3. Find out what NExT is, what they offer, and whether or not the program would be useful for you specifically.

All participants

1. You’ve had a chance to look at the site now and have a feel for what NExT is. How would you find out what specific services they have to offer?

2. Let’s say you have decided to apply to the program. What information do you need to submit?

   1. Follow-up question: How would you submit that information? [Observe how easy it is for participant to find the submission portal. If they have difficulties finding it, probe further about why that might be.]

3. If you have applied to the program and your application was rejected, what further options do you have?

4. If your project is accepted, who will be involved in deciding what resources are available to you?

5. If your project is accepted, what milestones are you expected to meet as your project progresses?

6. [For sections the participant has not explored]. Let’s take a look at the [section] section and tell me what you think about it.

Post Test Interview Questions

Now, I’d like to collect a bit of information from you using some standard tools before we discuss your experience.

[Administer the System Usability Scale (SUS) and Modified Cooper Harper questionnaires.]

Next, there are a few specific questions about your experience with the website.

1. What is your overall impression of the site?

2. What are the three things you liked most about the way the site looks and works?

3. What three things did you like least about the way the site looks and works?

4. Now that you’ve explored the site, do you think that it makes an appropriate “first impression” on visitors? If not, what would you change on the home page to make it more accurate in representing the site?

5. [For those who are new to the site] As you were looking through the site, how confident were you that content was correct and up to date? What helped you make that decision?

6. Please comment on how the site allowed you to determine if the program fits your needs? Probe: --Was there enough detail? --Was there too much detail? --Was the language easy to understand?

7. Please describe what made it easy or difficult to find the application process and details.

8. Do you feel that the site is logically designed so as to allow you to get details without overwhelming you? Probe: What could make it easier to use.

9. What about the site surprised you the most?

10. Is there any information or other content you would have expected to find, but didn’t?

11. Did you notice any redundancies in the information in the various sections?

12. What other suggestions do you have for improving the site?

Wrap up

OK, we’re done. Are there any questions you would like to ask me about the site or about today’s experience that I did not ask you about?

Thanks again for your participation.

Attachment 8F. System Usability Scale (SUS)

OMB No.: 0925-0589-08

Expiry Date: 5/31/2011

Public reporting burden for this collection of information is estimated to average 5 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0589-08). Do not return the completed form to this address.

© Digital Equipment Corporation, 1986.

Strongly Strongly

disagree agree

1. I think that I would like to

use this system frequently

2. I found the system unnecessarily

complex

3. I thought the system was easy

to use

4. I think that I would need the

support of a DCTD program expert to

be able to use this system

5. I found the various functions in

this system were well integrated

6. I thought there was too much

inconsistency in this system

7. I would imagine that most people

would learn to use this system

very quickly

8. I found the system very

cumbersome to use

9. I felt very confident using the

system

10. I needed to learn additional information about NExT before I could get going

with this system.

Attachment 8G. Modified Cooper Harper Difficulty Rating Scale

OMB No.: 0925-0589-08

Expiry Date: 5/31/2011

Public reporting burden for this collection of information is estimated to average 5 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, ATTN: PRA (0925-0589-08). Do not return the completed form to this address.

NExT #0925-0589-08, Page 3