#8_Justif Memo, Attach 8B, 8C, 8D, 8H_NExT

Date: November 29, 2010

To: Office of Management and Budget (OMB)

Through: Seleda Perryman, DHHS Report Clearance Officer

Mikia Currie, Program Analyst, NIH Project Clearance Branch, OPERA

Vivian Horovitch-Kelley, NCI OMB Project Clearance Liaison, OMAA

From: Lynn Cave, Communications Manager

Division of Cancer Treatment and Diagnosis (DCTD)

Gordon Willis, Cognitive Psychologist, DCCPS,

National Cancer Institute/NIH

Subject: Evaluation of the NCI Experimental Therapeutics (NExT) Website

Generic Sub-study under “Questionnaire Cognitive Interviewing and

Pretesting”, OMB No. 0925-0589-08, Expiry Date 05/31/2011.

Background

The National Cancer Institute (NCI) launched the NCI Experimental Therapeutics (NExT) Program in 2009 to accelerate cancer treatments from the lab to the clinic. The program is collaboration between the NCI Division of Cancer Treatment and Diagnosis (DCTD) and the Center for Cancer Research. NExT consolidates NCI’s anticancer drug discovery and development resources into a robust, balanced, goal-driven therapeutics pipeline of high-priority projects. These resources are capable of supporting a discovery and development continuum from initial discovery through Phase II clinical trial evaluation.

The NExT website (http://next.cancer.gov/) serves to educate scientists in government, academia, and industry about the NExT program and how they may participate in it. NExT is neither a grant or contract mechanism, but provides support to researchers whose proposals are assessed for scientific merit by a review committee. Most importantly, the website also contains an electronic application to the NExT program.

Program Goal(s)

The NExT website serves as the focal point of information for audiences about the program and how to apply for it. Relevant goals for the website include:

1. Rapid discovery and access to the Internet website by information seekers.

2. Having an information architecture that supports regular and occasional visitors who may have varying levels of knowledge about NExT.

3. Providing a means of promoting and disseminating information to audiences to increase their knowledge and understanding about the key characteristics of NExT (e.g., descriptions, qualifications, application process, and contact information).

4. Assuring that the layout and content of the Internet website is designed such that information and products can be rapidly found and are easily comprehensible.

5. Facilitating applications for admission to NExT, including all necessary supplemental materials.

6. Implementing procedures to ensure that the design layout and content remain current, i.e., regular reviews and updates.

7. Evaluating whether the Internet website is fully compliant with section 508 requirements, HHS Web standards, and accepted best practices in website design.

All of the goals are relevant to the evaluation.

Purpose of the Evaluation

This evaluation is intended to determine the effectiveness and usability of the NExT website to ensure that potential participants can find the information they need to understand and apply for the NExT Program and to determine if current NExT participants can find the information they need to proceed with the steps of the NExT Program.

Rationale for the Evaluation

NExT is a new vehicle for expanding and prioritizing the NCI drug discovery and development pipeline. The NExT website is the primary source of information about the program, its policies and guidelines, and contains the application form that participants need to apply for assistance from NExT. It is imperative that the website functions well for its visitors so that the program goal of accelerating cancer therapies from the lab to the clinic is realized. The NExT website has never been evaluated.

Participants

We anticipate including both people already familiar with NExT (“experts”) and people who know a little about NExT (“novices”), and those who are totally unfamiliar with the program. Within these three basic categories, it is our intent to recruit individuals who are Principle Investigators (PI) who work as academics, members of industry, representatives of the pharmaceutical industry, and government researchers.

User identification and selection: “Experts” will be recruited from current participants of the NExT program (several NExT awards have been made through three application cycles); “Novices” will be recruited from people who are NCI grantee applicants who have been notified each time a new NExT deadline occurs. The experts, and novices will be asked to recommend others who are totally unfamiliar with the program. Individuals representing a these three audiences will be identified for recruitment.

An email will be sent to potential participants asking them to answer questions relevant to recruiting criteria (e.g., where they work) so that an appropriate mix of individuals will participate (Attachment 8A). For each of the three rounds of testing, we anticipate having nine (9) participants—three (3) each from the 1) expert, 2) novice, and 3) totally unfamiliar groups. This follows the recommendation in the CIF (Common Industry Format) which became ISO/IEC 25062-2006 (http://zing.ncsl.nist.gov/iusr/) that a minimum of three individuals be used to represent a group of individuals.

Potential for bias: Since the primary audience for the NExT website is scientists who have or intend to apply for NExT funds, recruiting and selecting from NExT awardees and from those who have made inquiries about the NExT program or attended the launch meeting is appropriate. The recruitment plan will include iterative invitations/requests to participate to ensure the desired balance of experts and newcomers.

Recruitment plan: To recruit experts, an invitation to participate in a usability test session for the NExT website will be sent by e-mail to selected participants of the NExT program. For both experts and novices, iterative invitations will be sent out until the appropriate mix is obtained.

Any participants who participate in person will be given a consent form. Online participants will be sent a consent form (Attachment 8B). Additionally, online participants will be called, and read a verbal consent form, and asked if they agree to participate based on the form that they have received and the summary that they have heard (Attachment 8C).

Methodology

The goal is to refine the website in a step-wise manner by iteratively testing with small pools of users, responding to issues, and then testing again. This allows subsequent test sessions to gather feedback on the effectiveness of the previous modifications. The overall timeline for this study is estimated to be 29 weeks (Attachment 8D). The study will employ one-on-one usability testing in one hour sessions. Some tests may be conducted in person but most will be conducted using an online conferencing tool. All sessions will be carried out based on protocols that have been outlined in our Facilitator’s Guide (Attachment 8E). The sessions will consist of the user being given 2-3 “tasks.” These tasks will be tailored to the specific user’s area of interest and will be designed to answer these key questions:

1. How well can visitors, unfamiliar with NExT, learn both how it operates and what it has to offer? [Goals 1 and 2]

2. How easily can repeat visitors quickly locate materials and information they need to submit a NExT application? [Goals 1-5]

3. How easily can visitors locate goals, policies, guidelines, and other pertinent information for the program? How well does the website communicate this information? [Goals 3-5]

4. When visitors locate materials how confident are they that the information is correct and up to date? [Goal 6]

5. How well does the website comply with section 508, best practice guidelines, and HHS web standards? [Goal 7]

While the user undertakes the tasks, the user will speak aloud their response to the website’s usefulness for that task and their assumptions about how the website should handle the task. Concurrently, the usability consultant will take notes on the steps taken by the user, focusing on difficulties and successes. The sessions will also include video tape of the screens as the user is undertaking the task, to capture their steps. Collecting data from one-on-one usability test sessions will follow common practice in the usability field which is to review both the notes taken by the facilitator during the session and review the footage of the screen capture of the session for incidents in which the users’ experience difficulty. Incidents are graded according to how severely they impact the users’ ability to use the website (e.g., critical/show stopper versus low). All results will be reviewed to identify specific aspects of the website that are successful or where changes are needed.

Each participant will contribute data by simply using the website and by their verbal responses to it. Following the test, participants will answer a series of questions that will indicate their evaluation of the website’s efficacy, efficiency, and their satisfaction while using the website. The moderator will administer two evaluation tools: the System Usability Scale (“SUS”, Attachment 8F), and the Cooper-Harper Difficulty Rating Scale (Attachment 8G). While using these tools, the moderator will provide guidance and clarification to the participant, if any issues or confusion arises during the administration of the SUS and Cooper-Harper. Both tools will be answered by participants, and are designed to gauge their opinions regarding the ease of navigating and using the website, and what problems (if any), were found during testing. The data collected from these evaluation tools will generally be qualitative rather than quantitative because only eight participants does not lend itself to quantitative processes.

Because of our concern for finding appropriate participants who are willing to volunteer to participate during the day, each participant will be given an incentive for their time in the amount of $60, since volunteers feel that an hour of their time is valuable. It is important that we offer an incentive sufficient to attract the full range of needed participant types for our project. Without such incentive, it may be difficult to recruit appropriate participants, and the number of potential participants—especially for experts—is a limited field from which to recruit. Inadequate participant recruitment would limit the effectiveness of our evaluation.

Research Instrument

Participants will work in the usability test following an interrupted think-aloud protocol. In this protocol, participants are provided with specific tasks to perform. During task performance, participants are allowed to work naturally on the task. However, if there are visual or verbal signs of hesitation, confusion, or if they have questions, task performance is interrupted. Interaction between the test participant and the test facilitator will explore the event of interest to determine the possible cause and possible solutions. Once the possible cause and possible solutions are explored, participants are allowed to return to the performance of the task. Also as appropriate, the following data will also be collected: (1) success rates, (2) pages visited. (3) pathways participants took, (4) problems participants had, (5) participants reflections on their work, and (6) participants’ confidence in the quality and timeliness of information located. Samples and/or data will be anonymized and unlinked to any contact information needed to arrange the usability tests.

NIH’s Office of Human Subjects Research (OHSR) exemption has been applied for and is Attachment 8H.

The usability test should take each of the participants approximately 50 minutes to complete. Additionally, there is the SUS questionnaire and Modified Cooper Harper Screening Scale which adds an additional 10 minutes of time. The total respondent burden is estimated to be 32 hours. This effort will account for less than 2 percent of the total burden hours (1800) granted in DCCPS’s approval package.

Estimates of Hour Burden and Respondent Cost
Types of Respondents	Instrument	Number of Respondents	Frequency of Response	Average Time Per Response (minutes/ hours)	Total Hour Burden
Research Scientists	Responding to email (Attach 8A)	27	1	10/60 (0.167)	5
	Interview and Usability Testing (Attach 8E)	27	1	50/60 (0.833)	23
	System Usability Scale (Attach 8F)	27	1	5/60 (0.083)	2
	Cooper Harper Scale (Attach 8G)	27	1	5/60 (0.083)	2
Totals		108			32

Attachments (attached below):

8B. Informed Consent Form

8C. Script for Verbal Consent

8D. Anticipated Timeline

8H. Application for OHSR Exemption

Attachments (attached separately):

8A. Recruiting Email/Screener

8E. Facilitator’s Guide for NExT Website Evaluation

8F. System Usability Scale (SUS)

8G. Modified Cooper Harper Difficulty Rating Scale

Attachment 8B: Informed Consent Form

Identification of Project	Evaluation of the NExT Website
Statement of Age of Subject	I state that I am at least 18 years of age, in good physical health, and wish to participate in a program of research being conducted by Lynn Cave, Communications Manager, Division of Cancer Treatment and Diagnosis of the National Cancer Institute, Bethesda, MD.
Purpose	The purpose of this research is to determine the effectiveness and usability of the NExT website.
Procedures	Participants will be asked explore the NExT website to ensure that potential participants can find the information they need to understand and apply for the NExT Program and to determine if current NExT participants can find the information they need to proceed with the steps of the NExT Program. They will also be asked to reflect on the appropriateness and usability of the website’s content and navigation. The total time involved, including instructions will be no more than 60 minutes for the usability test (plus the time you’ve already spend, about 10 minutes, responding to our email).
Confidentiality	All information collected in this study will be kept secure to the extent permitted by law. I understand that the data I provide will be grouped with data others provide for the purpose of reporting and presentation and that my name will not be used. I understand that the monitor showing the usability test and my voice will be captured on videotape, but my image will not be captured.
Risks	I understand that the risks of my participation are expected to be minimal in nature.
Benefits, Freedom to Withdraw, & Ability to Ask Questions	I understand that this study is not designed to help me personally but that the investigators hope to update and redesign the NExT website in order to make the experience of utilizing the website easier for users. I am free to ask questions or withdraw from participation at any time and without penalty.
Contact Information of Investigators	Name: Lynn Cave, Position: Communications Manager, Division of Cancer Treatment and Diagnosis, NCI Telephone: 301-402-0912 Email: [email protected]

Printed Name of Research Participant _____________________________

Signature of Research Participant ________________________________

Date______________________

Attachment 8C:

Script for Verbal Informed Consent Form for the

Evaluation of the NExT Website

As part of this research, I need to ask you to formally agree to this usability test and interview. As part of a research project for the National Cancer Institute, we are seeking to evaluate the appropriateness and usability of the NExT website for researchers—both those who are already familiar with NExT (“experts”) and people unfamiliar or relatively new to NExT (“novices”). Individuals who we think would have an appropriate background are being asked to participate in a usability test and then to reflect on the website. So we are asking you for about 60 minutes of your time today plus the time you’ve already spend, about 10 minutes, responding to our email. Both the monitor used in the usability test and your voice will be videotaped, but your image will not be videotaped.

We won’t be asking anything personal and identifying information is only collected so we can send you $60 as a token of our appreciation of your time. Identifying information will not be shared. Any findings will be reported in aggregated form.

Your participation is voluntary, and if you choose not to participate it will not affect your relationship with the National Cancer Institute. You may ask questions at any time during the interview. You are also free to stop the interview at any time without penalty and without any questions being asked of you. Do you have any questions about the process of the interview?

If you agree to participate, you are saying that you understand what I’ve told you and that any questions you have were satisfactorily answered. You are also saying that you are at least 18 years old, and that you voluntarily agree to participate. Do you agree to participate in this usability test and interview?

Attachment 8D: Project Timeline for Evaluation of NExT Website

Task	Timeline
	Post-Contract Award
Heuristic assessment and review of other websites	Weeks 1-2
Round 1: Recruitment and other preliminaries(screening, facilitator’s guide) to usability testing	Weeks 3-6
Round 1: Conduct usability testing and briefing	Weeks 7-9
Round 1: Design short term fixes	Weeks 10-11
Implement short term fixes	Week 12-13 (estimated)
Round 2: Conduct usability testing and briefing	Weeks 14-17
Round 2: Design short term fixes	Weeks 18-19
Implement short term fixes	Week 20-21 (estimated)
Round 3: Conduct usability testing and briefing	Weeks 22-25
Round 2: Design short term fixes	Weeks 26-27
Implement short term fixes	Week 28-29 (estimated)
Final Report: Contractor drafts report; NCI staff reviews it; contractor prepares final report	Weeks 28-29

Attachment 8H: Office of Human Subjects Research (OHSR) Application

REQUEST FOR REVIEW OF RESEARCH ACTIVITY INVOLVING HUMAN SUBJECTS

INSTRUCTIONS: Please type directly on this form. You can expand the document if you need more space. If your research involves a survey or questionnaire, please attach it to this completed form.

Completed forms (with all required signatures) may be sent to OHSR by FAX (301-402-3443), email to [email protected], or by mail (2C146). If you have any questions, call OHSR at (301) 402-3444.

Date: __________________

To: OFFICE OF HUMAN SUBJECTS RESEARCH, Building 10, Room 2C-146

From: _____________________________________________________________

(Signature)

Through: ____________________________________________________________

(Signature of appropriate Official for IC, e.g., Lab/Branch Chief)

Protocol Title: ___________________________________________________________

Name of NIH Principal Investigator(s): Lynn Cave, Communications Manager, Division of Cancer Treatment and Diagnosis, NCI, phone 301-402-0912, e-mail: [email protected]

Building & Room No.: 31 Center Drive, Room 3A44, Bethesda, MD

Phone: 301-402-0912, fax: 301-496-0826, e-mail: [email protected]

Is the Principal investigator an NIH employee? __X___ Yes _______No

If no, please explain:______________________________________________________

1. What is the proposed research activity that you intend to perform at NIH (please use lay terms):

The NExT website (http://next.cancer.gov/) serves to educate scientists in government, academia, and industry about the NExT program and how they may participate in it. NExT is neither a grant or contract mechanism, but provides support to researchers whose proposals are assessed for scientific merit by a review committee. Most importantly, the website also contains an electronic application to the NExT program.

A series of three usability tests of eight persons will be conducted on the website; changes will be made to the website following each test based on the results and recommendations from the test. These evaluations are intended to determine the effectiveness and usability of the NExT website to ensure that potential participants can find the information they need to understand and apply for the NExT Program and to determine if current NExT participants can find the information they need to proceed with the steps of the NExT Program. See Attached Facilitator’s Guide. As part of the participant’s website evaluation the System Usability Scale and the Modified Cooper Harper Evaluation will be used; they are also attached. The two consent forms to be used in the test are also attached.

2. If applicable, list your non-NIH Collaborating Investigator(s).

Name Institution Address Tel. # FAX #

________________________________________________________________________

________________________________________________________________________

________________________________________________________________________

3. Proposed start date of your research __18 October 2010_____

Proposed completion date __16 May 2011__________

4. Will you be _________these samples or data?

Collecting X Yes/No

Receiving Yes/No X

Sending Yes/No X

5. Do the samples or data:

(a) Already exist? ___Yes __X__No

(b) Or are they being collected for the express purpose of this study? __X_Yes ____No

If “yes,” please describe: Participants will participate in usability tests of the NExT website. They will interact with the website and then give their opinion of it.

(c) Or a combination of (a) and (b)? ____Yes _____No

6. What role will you have in this research project? (Check all that apply)

__X_ Analyze samples/data only.

___ Consultant/advisor to collaborator(s) listed above.

___ Author of the protocol that is being implemented by your collaborating investigator (identified in question #2).

_X__ Co-authorship on publication(s)/manuscript(s) pertaining to this research.

___ You or NIH hold an IND for this research.

_X__ Decisional authority over the design or implementation of the research at the IRB approved site? If so, please explain. NCI created this design and is responsible for the implementation of this research. ________________________________________________________________________

___Other (If necessary, use this space to describe your role in this research).

________________________________________________________________________

________________________________________________________________________

7. Where are the subjects of this research activity located?

Subjects for this research are researchers who have expressed an interest in the NExT program (whose names would be obtained from NCI or found by a recruiter) or individuals who are novices but would be appropriate to use the NExT capabilities (whose names will be found by a recruiter). They are located all over the United States; the tests will be done remotely using conferencing software and telephone connection.

8. If human subjects are located elsewhere (not at NIH), will you have direct contact or intervention with them? (Examples: as subject's physician; in obtaining samples directly from the subject; by interviewing the subject?) _X___ Yes ____No

9. What kind of human samples (e.g., tissue, blood) or data (e.g., private information, responses to questionnaires) will be involved in your research?

In the usability test, participants will interact with the website to determine if it is effective and usable by exploring the website on their own and attempting assigned “tasks.” Then they will be asked to evaluate the website. Their responses will be the data.

10. If the samples, data do not come from an IRB approved protocol, do they come from:

(a) Repository ____Yes _X__ No

(b) Pathological waste ___ Yes _X__ No

(c) Autopsy material ___ Yes _X__ No

(d) Publicly available source ___Yes _X__ No

(e) Other_ Participants’ ideas about grouping the Training Activities.

11. Please check the box(es) that apply(ies) to the samples/data that you will receive.

(a) __X__ Samples and/or data will be anonymized/unlinked. (The samples/data cannot be linked to individual subjects by you or your collaborators at other sites.)

(b) ___ Samples and/or data will be coded, however that code cannot be used by either the sender or the receiver to identify specific individuals.

(c) ___ Samples and/or data will be coded so that the provider of the samples/data can link them to specific individuals but the receiver will not be able to do so.

12. Will you send results back to the provider(s) (listed in question 2 of this form)?

__X__ No, I will not send results back to the provider(s).

___ Yes, I will send aggregate results to the provider(s).

___ Yes, I will send results to the provider(s) that are linked to identifiable individuals.

If yes, does the provider intend to link your data to identifiable individuals?

_____Yes _______No

13. Has the research activity that you are proposing in this form been approved by an Institutional Review Board (IRB) elsewhere?

_________ Yes, the NIH research activity has been reviewed by the following IRB (s) (Please provide the following information for each IRB):

_________________________ Name of institution that provided the review

_________________________ Address of reviewing institution

_________________________ Name of PI for the IRB approved protocol

___________________________ Title of IRB approved protocol and protocol #

___________________________ Federal Wide Assurance (FWA) number**

__X__ No IRB review of the research activity described in question #1 above has taken place

(**An FWA is a contract between the U.S. Department of Health and Human Services (DHHS) and an entity receiving DHHS funds to conduct clinical research that the latter will follow ethical guidelines and federal regulations for the protection of human subjects. For a list of domestic and international institutions go to http://ohrp.cit.nih.gov/search/asearch.asp#ASUR

14. Per NIH guidance***, have conflicts of interest by NIH employees, if any, been resolved?

__X__ Yes _____No

If your answer is no, please see your Clinical Director about this matter before proceeding with this research.

***The January 5, 2005 NIH Guide to Preventing Conflict of Interest applies to all research conducted at NIH, http://ohsr.od.nih.gov/New/mpafwa_docs.html

NExT #0925-0589-08, Page 6