2 #4 Attachments B2, B5, B6 & B7

Evaluation of the SEER Training Website
0MB # 0925-0589-04

Attachment B2: Screener for SEER Training Website Evaluation
Attachment B5: Facilitator’s Guide for SEER Training Website Evaluation
Attachment B6: System Usability Scale
Attachment B7: Cooper-Harper Difficulty Rating Scale

USER-CENTERED DESIGN
20548 Deerwatch Place, Ashburn, VA 20147

TEL: (703) 729-0998

Attachment B2:
Screener for SEER Training Website Evaluation
We need to recruit both novice and experienced users of the SEER Training Website. Each individual will
be tested in one-on-one usability testing. Participants on the SEER Training Website will be asked in a
popup:
0MB No: 0925-0589-04
Expiry Date: xx!xx!2Oxx
Public reporting burden for this collection of information is estimated to average 5 minutes total, including the time
for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and
completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is
not required to respond to, a collection of information unless it displays a currently valid 0MB control
number. Send comments regarding this burden estimate or any other aspect of this collection of information,
including suggestions for reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC
7974, Bethesda, MD 20892-7974, ATTN; PRA (0925-xxxx-xx). Do not return the completed form to this address.

Would you like to help improve this website? Yes I No
[For those who answer “yes,” they will be directed to the following screening questionnaire.
Screening questionnaire:
1.

Briefly describe your previous experience (how many visits, over what period of time, etc.) with
the SEER Training Website.

2.

What is your background or what is the rationale that brings you to the website? (e.g., cancer
registrar, high school student, college student grad student, professors, physician, P1, medical
technologist, etc.)

3.

Briefly describe how you plan to use the material you find on the website.

[After those questions are answered, tell them.]
We are recruiting users to spend an hour of their time to evaluate the website
• either from your own home or office where you would need to be online
• or in our lab if you are in the Washington DC area and would like to come in to our Rockville
office.
These evaluations are being scheduled for [timeframe]. You will be compensated for your time.
If you are interested, we need contact information from you (business-related only):
Email address (work)
Times of availability
Telephone number (work)
SEER #0925-0589-04, Page 1

USER-CENTERED DESIGN
20548 Deerwatch Place, Ashburrt, VA 20147

TEL: (703) 729-0998

Attachment B5:

Facilitator’s Guide for
SEER Training Website Evaluation
April 29, 2010

Submitted to:
National Cancer Institute

Submitted by:
User-Centered Design, Inc.
20548 Deerwatch Place
Ashburn, VA 20147
(703) 729-2370
www. user-centereddesiqn.com

SEER #0925-0589-04

Page 1

User-Centered Design, Inc. 20548 Deerwatch Place Ashburn, VA 20147 (703) 729-0998
[Note: The purpose of this document is to guide the moderator. The questions and tasks contained herein may not be
asked as written. The facilitator often draws on participant comments and the natural flow of the testing process to
determine the flow of the session. While the facilitator will try to follow the order of the guide, many times tasks will come
up ahead of time or in different order. The facilitator may allow the order of the tasks to change in order to let the
process flow naturally.]
Pre-Test
[Administer the informed consent and video release form]
0MB No: 0925-0589-03
Expiiy Date: xx/xx/2Oxx
Public reporting burden for this collection of information is estimated to average 50 minutes total, including the time for
reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and
reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond
to, a collection of information unless it displays a currently valid 0MB control number. Send comments regarding this
burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: NIH,
Project Clearance Branch, 6705 Rockledge Drive, MSC 7974, Bethesda, MD 20892-7974, A1TN; PRA (0925-xxxx-xx). Do
not return the completed form to this address.

Introduction

Thank you for agreeing to participate in this study. Do you have any questions for me before we get started?
You signed a video release so you know I’ll be recording everything that shows up on the computer monitor. I’d also like
you to know that there may be observers helping me by taking notes from another room, but don’t worry about that.
You and I will work on this together and they’ll just watch.
We’re going to be looking at a web site today that was created for the National Cancer Institute. I’m going to ask you to
explore the web site and then do some tasks with the site. Then I’d like to get your opinion about it.
We have invited Certified Tumor Registrars and other participants who have reasons to find information about cancer
registration, surveillance, and other related concepts and terminology, to browse this web site. So we can understand
the point of view that you are bringing to the testing, I’d like for you to tell me a little bit about your previous experience
with, or knowledge about finding any of the aforementioned information.
•
•

Where are you in your career/education/training? (What have you done previously? Where are you headed?)
Have you ever been to any NCI web sites? [If yes] Do you remember which ones or what you were attempting
to do/access?
• Why did you visit the SEER Training Website
•
How and/or where did you hear about us?
• Are you from a SEER region? (If Participant is a CTR)
[Participant talks....]
[For in-lab participants] Since we’ll be working on a computer today, I want to let you use this computer in a way that it
looks like the computer you regularly use. Tell me a web site you use frequently. Now does this look too big or too
small? [Facilitator adjusts the resolution until it matches for the participant’s normal experience.]

There are two important things you should keep in mind while you work with this web site:

SEER #0925-0589-04

Page 2

User-Centered Design, Inc. 20548 Deerwatch Place Ashburn, VA 20147 (703) 729-0998
•

First, I did not design it so you can’t hurt my feelings. If there are problems with the design I would like
to discuss them with you to see if we can find a way to make the web site better.

•

Second, we are evaluating the web site and not you, so you cannot make any mistakes. The web site
is supposed to be intuitive and easy to use. If it isn’t, that’s a problem with the web site not with you.
—

Any comments you have, either positive or negative, will help make the product better so feel free to tell me whatever is
on your mind. After you’re done working, I’ll ask you a few questions and then give you some time to ask me anything
you’d like. Do you have any questions for me before we get started?
Initial Impression
[Open the browser to http://training.seer.cancer.govlj
First, I’m going to let you explore this web site by yourself. I want to see what you can find that might be helpful or
interesting to you. I’d like you to tell me about what you see and whether it does or does not relate to finding information
useful to you. If you have questions as we go along, feel free to ask. I may not answer your questions right away in
order to see how well you can figure out things for yourself, but it is valuable to know if you do have questions about the
site.
[Observe the participants and note what information relevant to cancer registration and surveillance that they find on their
own. Keep track of any relevant comment(s). After a few minutes
If not offered, prompt for participant’s impression of the main page.
If not already covered, ask what prompted them to explore the sections they visited
Facilitator should ask about specific pages and sections as they are encountered. If users explore areas listed as a task
below, the facilitator should get the feedback during their exploration. Areas of the site that seem to be of interest and
then are dropped should be asked about.)
Tasks
[Explore each of these tasks or subtasks if the participant did not explore them on their own)
1.

I’d like you to find information in the training module about cancer registration or surveillance that would be
appropriate for you considering your interests. I’d like to hear about how what you’re finding or not finding
fits with your needs. [If participant does not have a specific area of interest, ask them to explore Cancer as a
Disease if they are inexperienced, or Cancer Registry Operations and Procedures if they are experienced].
If time permits, have them explore other relevant concepts and terminology that can be found in the other
training modules.

2.

How did you find the information you were interested in? Let’s go through that process.

SEER #0925-0589-04

Page 3

User-Centered Design, Inc. 20548 Deerwatch Place Ashburn, VA 20147 (703) 729-0998
Post Test Interview Questions
Now, I’d like to collect a bit of information from you using some standard tools before we discuss your experience.
[Administer the System Usability Scale (SUS) and Modified Cooper Harper questionnaires.]

I’d like to ask you a few specific questions.
1.

Most sites have some type of learning curve associated with them. Does this site have a learning curve?

2.

What is your evaluation of the material presented in the web site? If you were ranking it where 5 was very high
quality and 1 was very low quality, what value would you give this material?

3.

What is your evaluation of the timeliness of the material presented in the web site? If you were ranking it where
5 was very up-to-date and 1 was very out-of-date, what value would you give this material?

4.

What are the three things you liked most about the way the site looks and works?

5.

What three things did you like least about the way the site looks and works?

6.

What about the site surprised you the most?

7.

Is there any information or other content you would have expected to find, but didn’t?

Wrapup
OK, we’re done. Are there any questions you would like to ask me about the site or about today’s experience that I did
not ask you about?
Thanks again for your participation.
[Pay stipend and obtain signature]

SEER #0925-0589-04

Page 4

Attachment B6:

System Usability Scale
0MB No: 0925-0589-04
Expiry Date: xx$xx!2Oxx
Public reprting burden for this cellecton of infonnation is estimated to verege 5 minutes total, including the
time for
eng irstuctions, seeihing existing data soulces, gathering and mainteining the data needed, and
completing and reviewing the collection of information. Anagency may not conduct or sponsor, and a
persons riotrequhud to respond to, a cocikn of hdornuithn unss h disphys a cumnqy valid 0MB
control number. Send commente iegaxding this burden etimab or any other aspect of this collection of
infonnation, including suggestions for zeducirig this buxden, to: NIH, Pioject Clearance Brench. 6705
Rocidedge Drive, MSC 7974. Bethesda. MD 20892-7974, ATfl4; PRA (O925.xxxxxx). Do it return the
completed fonti to this address.

Strongly
Disagree
1.

I

think

that

would

like

I

to

use this system frequently

2.

found

the

system

1

1

the

various

functions

I
5

I

I

3

1

2

3

4

5

1

2

3

4

5

4

5

4

I

I

5

I

I
I

I

in

this system were well integrated

1

6. I thought there was too much
inconsistency in this system

I

7. I would imagine that most people
would learn to use this system
very quickly

I

8. I found the system very
cumbersome to use

I

9. I felt very confident using the

I

I
2

I
1

I

I
1

I
3

I
2

I
1

I
3

2

I

2

10. I needed to learn a lot of
things before I could get going

1

2

5

4

5

I
3

I
1

I
4

I
3

2

system

with this system
Total score

I
4

2

I

4. I think that I would need the
support of a technical person to
be able to use this system
found

I
3

I

3. I thought the system was easy
to use

I

I
2

unnecessarily

complex

5.

Strongly
Agree

I
4

I

I
5

I

I

3

4

5

3

4

5

I

=

20

SUS Score =

50

I

Attachment B7:
Cooper-Harper Difficulty Rating Scale

0MB No: 0925-xxxx-xx
Expiry Date: xxlxxl2Oxx
Public reporting burden for this collection of information is estimated to average 5 minutes total,
including the time for reviewing instructions, searching existing data sources, gathering and
maintaining the data needed, and completing and reviewing the collection of information. An agency
may not conduct or sponsor, and a person is not required to respond to, a collection of
information unless it displays a currently valid 0MB control number. Send comments regarding
this burden estimate or any other aspect of this collection of information, including suggestions for
reducing this burden, to: NIH, Project Clearance Branch, 6705 Rockledge Drive, MSC 7974,
Bethesda, MD 20892-7974, ATTN; PRA (0925-xxxx-xx). Do not return the completed form to this
address.

Start

Even
though problems
may be frequent,
instructed tasks be
accomplished
most of the
time?
Yes

Problems are
small and
inconsequential?

Yes

The mental
effort required
acceptable?

Yes

N0

No

No

System redesign
is required

System redesign
is recommended

Mental effort
is high and
should be reduced

‘Ii

I

Impossible

Major difficulty

Major difficulty

Major difficulty

Intense mental effort was required to accomplish tasks,
but frequent or numerous problems still occurred

Tasks cannot be accomplished reliably

[3]

[2]

[i]

Rating

I

Maximum mental effort was required to avoid large or
numerous errors

I
I

I

Moderately high mental effort was required to attain
adequate system performance

High mental effort was required to attain adequate
system performance

[7]

6]

[5]

[4]

Moderately objectionable
difficulty

Acceptable mental effort required was to attain
adequate system performance

8

Maximum mental effort was required to bring errors to
a moderate level

Minor but annoying
difficulty

An acceptable mental effort was required to attain
adequate system performance

9

I

Fair, Mild Difficulty

The mental effort was low and desired performance is
attainable

10

I

Easy, Desirable

The mental effort required was minimal and the desired
performance is easily attainable

Very objectionable
tolerable difficulty

Very Easy, Highly
Desirable