#4_Justif Memo & Attach B1, B3, B4 & B8_SEER_rev3

Date: June 10, 2010

To: Office of Management and Budget (OMB)

Through: Seleda Perryman, DHHS Report Clearance Officer

Mikia Currie, Program Analyst, NIH Project Clearance Branch, OPERA

Vivian Horovitch-Kelley, NCI OMB Project Clearance Liaison, OMAA

From: Jennifer Ruhl, Public Health Analyst

Division of Cancer Control and Population Sciences (DCCPS),

National Cancer Institute/NIH

Gordon Willis, Cognitive Psychologist, DCCPS, National Cancer Institute/NIH

Subject: Evaluation of the SEER Training Website

Generic Sub-study under “Questionnaire Cognitive Interviewing and

Pretesting”, OMB No. 0925-0589-04, Expiry Date 05/31/2011.

Background

The Surveillance, Epidemiology, and End Results (SEER) Program of the National Cancer Institute (NCI) is an authoritative source of information on cancer incidence and survival in the United States. SEER currently collects and publishes cancer incidence and survival data from population-based cancer registries covering approximately 26 percent of the US population. Certified Tumor Registrars (CTRs) play an instrumental role in facilitating the collection and publication of this high quality data. The SEER program has approximately 100 Certified Tumor Registrars, affiliated partners, and contractors who are involved in this ongoing research. To ensure that they receive proper training and education, the SEER Program launched the SEER Training Website in 2001. The website is designed to train and support both current and aspiring CTRs, and increase their knowledge in the fields of cancer registration and surveillance.

The SEER Training website serves as a cornerstone of the SEER Program’s efforts to facilitate the collection and recording of the most complete and accurate cancer data possible. Currently, the website is focused solely on the needs of users who have been developing their skills in cancer registration and surveillance as part of their training to become CTRs. However, the website is undergoing a major redesign. The updated website will provide CTRs with newer and more relevant information that reflects current trends in the field of cancer registration and surveillance, while also serving a wider audience of users who utilize the SEER Training Website to learn about more generalized concepts and terminology that can be acquired by browsing the various training modules. These impending changes represent a major upgrade for the SEER Training Website. It is necessary to ensure that this educational tool functions well for its users, because it is a primary resource for training and educating CTRs nationwide, and also an ideal source of information for many other participants.

This goal of this evaluation is to examine, evaluate, and fix, as appropriate, the SEER Training Website. The site will be examined in tests conducted by usability professionals in one-on-one sessions with users who are either Certified Tumor Registrars or other participants who access the website. They will be asked to explore the website in tasks that resemble typical usage of the site by those who are looking for information regarding cancer registration and surveillance. The consultants will observe their behavior and interact as appropriate to determine if any text areas/terminology/navigation are causing confusion or difficulty and why. This information will be reported back to the SEER program’s leadership and the webmaster in a briefing; they will focus on changes to labeling, visual treatment, positioning and navigation.

After the changes have been made, a second usability test will be conducted following the method of the first to determine the effectiveness of the modifications. A report will be prepared by the contractor to document both usability tests. It is anticipated that modifications resulting from these tests will greatly enhance SEER’S ability to communicate effectively and accurately with CTRs and other users who use the website.

The specific nature of any possible issues with the SEER Training Website has not been explored for many years. This project will enable SEER’s Training Website to serve as a more effective information source on cancer registration and surveillance for Certified Tumor Registrars, the general public, and the external research community.

The anticipated 30 week timeline is shown in Attachment B1.

Participants

To recruit CTRs, an invitation to participate in a usability test session for the SEER Training Website will be sent by e-mail to each of the CTR groups. To recruit novices, students (undergrad, graduate, and post-docs), and other people unfamiliar or relatively new to the website, invitations will be sent to newcomers in the form of a “pop-up” window when they access the website. Users will then have the option to give their business-related email address for participation in the evaluation, or ignore the invitation and opt out of the study. Respondents will be subjected to a screener which will gauge the user’s level of experience with the website, reason(s) for accessing the website, and how they plan on using the materials on the website (see Attachment B2). The screener will help us ensure that respondents are selected in such a way to equally balance the number of expert and novice users, as well as proportionally balance the representation across various population groups. For both experts and novices, iterative invitations will be sent out until the appropriate mix is obtained.

Participants will be given a consent form either in person or it will be sent to the online participants (Attachment B3). Additionally, on-line participants will be called, and read a verbal consent form, and asked if they agree to participate based on the form that they have received and the summary that they have heard (Attachment B4).

Methodology

The study will employ one-on-one usability testing in one hour sessions. Some tests will be conducted in person and some will be conducted using an online conferencing tool. All sessions will be carried out based on protocols that have been outlined in our Facilitator’s Guide (Attachment B5). The sessions will consist of the user being given 2-3 “tasks.” These tasks will be designed to answer the various key questions identified above, and will be tailored to the specific user’s area of interest. While the user undertakes the tasks, the user will speak aloud their response to the Web site’s usefulness for that task and their assumptions about how the Web site should handle the task. Concurrently, the usability consultant will take notes on the steps taken by the user, focusing on difficulties and successes. The sessions will also include video tape of the screens as the user is undertaking the task, to capture their steps. Collecting data from one-on-one usability test sessions will follow common practice in the usability field which is to review both the notes taken by the facilitator during the session and review the footage of the screen capture of the session for incidents in which the users’ experience difficulty. Incidents are graded according to how severely they impact the users’ ability to use the site (e.g., critical/show stopper versus low). All results will be reviewed to identify specific aspects of the web site that are successful or where changes are needed.

Each participant will contribute data by simply using the web site and by their verbal responses to it. Following the test, participants will answer a series of questions that will indicate their evaluation of the site’s efficacy, efficiency, and their satisfaction while using the site. The moderator will administer two evaluation tools: the System Usability Scale (“SUS”—see Attachment B6), and the Cooper-Harper Difficulty Rating Scale (see Attachment B7). While using these tools, the moderator will provide guidance and clarification to the participant, if any issues or confusion arises during the administration of the SUS and Cooper-Harper. Both tools will be filled out by participants, and are designed to gauge their opinions regarding the ease of navigating and using the website, and what problems (if any), were found during testing. The data collected from these evaluation tools will generally be qualitative rather than quantitative because only nine participants does not lend itself to quantitative processes. Each participant will be remunerated for their time, since volunteers feel that an hour of their time is valuable (all participants will be compensated $50). It is important that we offer remuneration sufficient to attract the full range of needed participant types for our project. Without such remuneration, it may be difficult to recruit desirable participants. Inadequate participant recruitment would limit the effectiveness of our evaluation.

Research Instrument

Participants will work in the usability test following an interrupted think-aloud protocol. In this protocol, participants are provided with specific tasks to perform. During task performance, participants are allowed to work naturally on the task. However, if there are visual or verbal signs of hesitation, confusion, or if they have questions, task performance is interrupted. Interaction between the test participant and the test facilitator will explore the event of interest to determine the possible cause and possible solutions. Once the possible cause and possible solutions are explored, participants are allowed to return to the performance of the task. Also as appropriate, the following data will also be collected: (1) success rates, (2) pages visited. (3) pathways participants took, (4) problems participants had, (5) participants reflections on their work, and (6) participants’ confidence in the quality and timeliness of information located. Samples and/or data will be anonymized and unlinked to any contact information needed to arrange the usability tests.

NIH’s Office of Human Subjects Research (OHSR) exemption has been granted and is Attachment B8.

The usability test should take each of the participants approximately 50 minutes to complete. Additionally, there is the screener, SUS questionnaire and Modified Cooper Harper Screening Scale which adds an additional 15 minutes of time. The hourly wage rate is calculated by averaging the wage rate of a CTR ($25/hour) and a student ($17/hour), which is $21/hour. The total respondent burden is estimated to be 31 hours. This effort will account for 2 percent of the total burden hours (1800) granted in DCCPS’s approval package.

Estimates of Hour Burden and Respondent Cost
Types of Respondents	Instrument	Number of Respondents	Frequency of Response	Average Time Per Response (minutes/ hours)	Total Hour Burden	Hourly Wage Rate	Total Respondent Cost
CTRS and Students	Screener (Attach B2)	50	1	5/60 (0.083)	4	$21.00	$84.00
	Interview and Usability Testing (Attach B5)	27	1	50/60 (0.83)	23	$21.00	$483.00
	System Usability Scale (Attach B6)	27	1	5/60 (0.083)	2	$21.00	$42.00
	Cooper Harper Scale (Attach B7)	27	1	5/60 (0.083)	2	$21.00	$42.00
Total		131			31		$651.00

List of Attachments (Attached Below)

B1: Anticipated Timeline (attached below)

B3. Informed Consent Form (attached below)

B4. Verbal Consent (attached below)

B8. OHSR Exemption (attached below)

List of Attachments (Attached In Separate File)

B2. Screener

B5. Facilitator’s Guide for SEER Training Website Evaluation

B6. System Usability Scale (SUS)

B7. Modified Cooper Harper Difficulty Rating Scale

Attachment B1: Project Timeline for Evaluation of SEER Training Website

Task	Timeline
Pre-Contract Award
Develop project statement of work	Weeks 1-2
Solicit RFP and review proposals	Weeks 3-7
Post-Contract Award
Develop project work plan – NCI meets with contractor; contractor reviews background material and resources; contractor prepares draft and final work plan	Weeks 1-3
Web analytics data collection and review	Weeks 3-4
Heuristic assessment and review of other web sites	Weeks 5-8
Round 1: Recruitment and development for in-depth interview and usability testing questions/codebook	Weeks 3-6
Round 1: Conduct in-depth interviews and usability testing	Weeks 7-11
Implement feasible recommended Internet site changes	Weeks 12-14
Round 2: Recruitment and development for in-depth interview and usability testing questions/codebook	Weeks 15-17
Round 2: Conduct in-depth interviews and usability testing	Weeks 18-20
Round 3: Recruitment and development for in-depth interview and usability testing questions/codebook	Weeks 21-23
Round 3: Conduct in-depth interviews and usability testing	Weeks 24-26
Final Report: Contractor drafts report; NCI staff reviews it; contractor prepares final report	Weeks 27-30

Attachment B3: Informed Consent Form

Identification of Project	Evaluation of the SEER Training Website
Statement of Age of Subject	I state that I am at least 18 years of age, in good physical health, and wish to participate in a program of research being conducted by Jennifer Ruhl in the DCCPS (Division of Cancer Control and Population Sciences) Surveillance Research Program of the National Cancer Institute, Rockville, MD 20852.
Purpose	The purpose of this research is to determine the effectiveness and usability of the SEER Training Website.
Procedures	Participants will be asked explore the SEER Training Website to ensure that they can find relevant information about cancer registration, surveillance, and other related concepts and terminology quickly and easily, and that they can navigate efficiently through the various training modules. They will also be asked to reflect on the appropriateness and usability of the web site’s content and navigation. The total time involved, including instructions will be no more than 60 minutes.
Confidentiality	All information collected in this study will be kept secure, to the extent permitted by law. I understand that the data I provide will be grouped with data others provide for the purpose of reporting and presentation and that my name will not be used. I understand that the monitor showing the usability test and my voice will be captured on videotape, but my image will not be captured.
Risks	I understand that the risks of my participation are expected to be minimal in nature.
Benefits, Freedom to Withdraw, & Ability to Ask Questions	I understand that this study is not designed to help me personally but that the investigators hope to update and redesign the SEER Training Website in order to make the experience of utilizing the site easier for users. I am free to ask questions or withdraw from participation at any time and without penalty.
Contact Information of Investigators	Name: Jennifer Ruhl Position: Public Health Analyst, Cancer Statistics Branch Telephone: 301-594-8494 Email: [email protected]

Printed Name of Research Participant _____________________________

Signature of Research Participant ________________________________

Date______________________

Attachment B4:

Verbal Informed Consent Form for the

Evaluation of the SEER Training Website

As part of this research, I need to ask you to formally agree to this usability test and interview. As part of a research project for the National Cancer Institute, we are seeking to evaluate the appropriateness and usability of the SEER Training Website for Certified Tumor Registrars (CTRs) and other interested individuals who are looking for information about cancer registration, surveillance, and other related concepts and terminology.

Individuals who we think would have helpful knowledge about this subject are being asked to participate in a usability test and then to reflect on the website. So we are asking you for about 1 hour of your time. Both the monitor used in the usability test and your voice will be videotaped, but your image will not be videotaped.

We won’t be asking anything personal and no identifying information will be shared. Any findings will be reported in aggregated form.

Your participation is voluntary, and if you choose not to participate it will not affect your relationship with the National Cancer Institute. You may ask questions at any time during the interview. You are also free to stop the interview at any time without penalty and without any questions being asked of you. Do you have any questions about the process of the interview?

If you agree to participate, you are saying that you understand what I’ve told you and that any questions you have were satisfactorily answered. You are also saying that you are at least 18 years old, and that you voluntarily agree to participate. Do you agree to participate in this usability test and interview?

Attachment B8: OHSR Exemption

SEER #0925-0589-04, Page 8