#5 Justif Memo & Attach C1 & C6_EGRP Web Testing_rev2

Date: 6/14/2010

To: Office of Management and Budget (OMB)

Through: Seleda Perryman, DHHS Report Clearance Officer

Mikia Currie, Program Analyst, NIH Project Clearance Branch, OPERA

Vivian Horovitch-Kelley, NCI OMB Project Clearance Liaison, OMAA

From: Christie Kaefer, Communications Coordinator, Epidemiology and Genetics

Research Program, Division of Cancer Control and Population Sciences (DCCPS), NCI, NIH

Gordon Willis, Project Officer, Applied Research Program, DCCPS, NCI, NIH

Subject: Needs Assessment for http://epi.grants.cancer.gov

Generic Sub-study under “Questionnaire Cognitive Interviewing and

Pretesting,” OMB No. 0925-0589-05, Expiration Date 5/31/2011.

The purpose of this evaluation is to determine the effectiveness and usability of the Epidemiology and Genetics Research Program’s (EGRP) website, http://epi.grants.cancer.gov, a key communication channel with extramural cancer epidemiologists (its primary target audience), and other stakeholders, such as cancer advocates. EGRP provides opportunities for investigators to conduct research to increase understanding of cancer causes and prevention in human populations

Background

EGRP is the largest funder of cancer epidemiology grants nationally and worldwide. In FY 2009, the Program supported more than 300 grants totaling $194 million (excluding American Recovery and Reinvestment Act funds). It also funds several research resources available to investigators, each valued at more than $1 million, including the Cancer Family Registries.

This website provides guidance about EGRP’s research priorities, research resources, funding opportunities, grantsmanship issues, and relevant news.

EGRP’s goals for its website include providing:

• An organizational structure easily navigated by website visitors.

• Effective orientation information about EGRP for new website visitors.

• Information about resources that visitors may need to collaborate, including about funding and/or other support opportunities, emerging research priorities, datasets that can be requested for collaborative research, and upcoming events.

• A means of promoting and disseminating information about a wide range of research initiatives and accomplishments.

• Content that is designed and presented professionally and kept current in order to facilitate comprehension in an online environment.

• Compliance with Section 508, best practice guidelines, and HHS Web standards.

• Use of Web metrics to monitor whether the website is meeting EGRP goals.

Significant resources are dedicated to maintaining the website (which consists of more than 900 pages); however, only two sections of the Website have been evaluated from a user’s perspective, and both sections have changed substantially since they were evaluated 8 to 10 years ago. Numerous sections and pages have been created in the past few years based on the assumption that website visitors would find the information valuable, including pages related to grantsmanship; access to genomics datasets and biospecimens; and epigenetics, pharmacoepidemiology and pharmacogenomics. Other sections have been greatly expanded and/or revamped, including information about cancer epidemiology research consortia, health disparities research, and information about EGRP’s staff. EGRP staff monitors website metric available through its Web support contractor; however, many questions about the site’s usage remain unanswered. EGRP’s goal is to conduct a comprehensive website evaluation to ensure that it is delivering valuable information in an effective and efficient manner.

Key questions to be addressed through this evaluation include:

• What are the primary reasons members of various target audiences utilize the website? Do these reasons differ by audience and their professional level of experience?

• How easily can visitors who are unfamiliar with EGRP learn how the website is organized and what the Program’s mission and research priorities are?

• How easily can website visitors quickly locate needed materials and information related to EGRP research priorities, new initiatives, funding opportunities, grantsmanship, and research resources?

• When visitors locate materials, how confident are they that the information is correct and up to date?

• Would existing web-based technologies and tools, such as RSS feeds, be valuable as adjuncts to the EGRP website?

• How well does the website comply with Section 508, best practice guidelines, and HHS Web standards?

• To what extent is the website visitors’ information needs being met by the website? If their needs are not being addressed, how could the website be revised to better meet these needs?

 Are there metrics beyond what is currently being captured by EGRP’s Web support contractor that would help better measure and monitor program goals for the website?

Although other Programs and Divisions within the NCI have conducted website usability testing in the past, and when possible, “lessons learned” from these previous efforts have been (or are in the process of being) incorporated into EGRP’s current Website design and structure. This evaluation will avoid unnecessary duplication with past DCCPS evaluations by focusing its key questions on content and aspects of http://epi.grants.cancer.gov that have never been user tested.

EGRP’s Communications Coordinator will disseminate the evaluation findings to EGRP’s leadership, Program staff, and contractors who assist with web support and communications activities. These results will be used to develop, revise, and/or prioritize future web design and maintenance efforts and broader EGRP communication strategies and goals.

The estimated completion date is February 25, 2011 assuming the start date is August 1, 2010. NIH Evaluation Set Aside funds must be obligated by August 13, 2010, so this is the latest possible start date for this project. A detailed project timeline is provided in Attachment C1.

Participants

The potential respondent universe is estimated based on membership data from the American Association of Cancer Research (AACR), the oldest and largest scientific association focused on cancer research. Of the 8,500 AACR associate members (e.g., trainees, fellows, etc.), approximately 4%, or 340, are epidemiologists. There are roughly 16,000 active members (e.g., junior and senior faculty), 3% of which, or 480, are epidemiologists. A database maintained by NCI’s Office of Advocacy Relations indicates there are about 50 cancer advocates interested in topics related to EGRP’s mission who are also willing to participate in website testing.

EGRP will solicit volunteers through its listserv initially, and if there is a poor response, can send targeted messages to current and potential grantees and to advocates through the NCI’s Office of Advocacy Relations.

Methodology

The goal is to refine the EGRP website in a step-wise manner by iteratively conducting one-time testing with small pools of users, responding to issues, and then testing again. This allows subsequent test sessions to gather feedback on the effectiveness of previous modifications.

Experienced extramural cancer epidemiologists will be recruited from long-standing EGRP grantees, research collaborators, “Friends of EGRP” listserv subscribers, and others who frequently participate in EGRP-sponsored meetings and workshops (e.g., working groups/committees, consortia, cooperative agreements, etc.). Junior cancer epidemiologists will be recruited from lists of new or prospective grantees, research collaborators, “Friends of EGRP” listserv subscribers, and others with interests related to EGRP who do not have long-standing relationships with the Program. In academic institutions, experienced cancer epidemiologists would likely be an Associate Professor or Full Professor and junior cancer epidemiologists would likely be an Assistant Professor. Post-doctoral fellows will be recruited through both NCI-designated cancer centers and other cancer research centers and possibly professional associations for epidemiologists (e.g., American College of Epidemiology and Society for Epidemiologic Research). Advocates with an interest in epidemiology will be recruited through NCI’s Office of Advocacy Relations, “Friends of EGRP” listserv subscribers, and from lists of participants in EGRP-sponsored consortia, meetings, and workshops.

The recruitment plan will include iterative invitations/requests to participate to ensure the desired balance of experts and novices (across EGRP’s diverse grants portfolio and research collaborations) from a variety of target audiences. The majority of the usability test sessions will be run remotely using Web conferencing software (so the user can share their screen with the session facilitator) to allow flexibility in scheduling users so they do not need to be in the Washington, D.C. metro area. If possible, one round of testing will be conducted at the annual Cohort Consortium (http://epi.grants.cancer.gov/Consortia/cohort.html) meeting which will take place in November 2010 in Atlanta, GA.

The study will employ one-on-one usability test sessions. The sessions will consist of the user being given 2-3 “tasks.” These tasks will be designed to answer the key questions outlined above and will be tailored to the specific user’s area of interest/expertise. (Following the typical approach to conducting usability studies, a predefined data collection instrument will not be employed.) While the user undertakes the tasks, the user will speak aloud their response to the web site’s usefulness for that task and their assumptions about how the web site should handle the task. Concurrently, the usability consultant will take notes on the steps taken by the user, focusing on difficulties and successes. The notes will be carefully reviewed to identify patterns, categorizing the notes by task type and problem type associated with each task, user type, and frequency of occurrence. For quantitative data, we will calculate percentages of participants who succeeded or not at each task, average time to complete tasks, average number of pages visited in each task, and the frequency of specific problems. We expect that the quantitative and qualitative data analyses will provide an overall assessment of usability issues associated with EGRP’s website. The sessions will also include audio tape as the user is undertaking the task, to ensure their comments are accurately captured. Collecting data from one-on-one usability test sessions will follow common practice in the usability field which is to review both the notes taken by the facilitator during the session and review the footage of the screen capture of the session for incidents in which the users’ experience difficulty.

Incidents are graded according to how severely they impact the users’ ability to use the site (e.g., critical/show stopper versus low). All results will be reviewed to identify specific aspects of the web site that are successful or where changes are needed. Some of the indicators that we anticipate using are (1) success rates, (2) time to complete tasks, (3) pages visited, (4) pathways participants took, (5) problems participants encountered, (6) what participants said as they worked through tasks, (7) participants’ confidence in the quality and timeliness of the information they located, and (8) modified Cooper-Harper Difficulty Rating Scales and System Usability Scales (SUS). We will obtain both quantitative and qualitative data as part of the usability testing. For quantitative data, we will calculate percentages of participants who succeeded or not at each task, average time to complete tasks, average number of pages visited in each task, and the frequency of specific problems. The use of the Cooper-Harper and SUS scales together indicate if the scales are showing some construct validity.

For qualitative data, the notes will be carefully reviewed to identify patterns, categorizing the notes by task type and problem type associated with each task, user type, and frequency of occurrence. We expect that the quantitative and qualitative data analysis will provide an overall assessment of usability issues associated with EGRP’s website.

All respondents in the pilot testing groups will receive modest remuneration at a flat rate of $30. The amount of remuneration represents the minimal amount of funds necessary to make participation attractive to participants and to provide nominal compensation for their time and effort. This remuneration amount is also consistent with the approved full generic clearance OMB No. 0925-0589.

Research Instrument

The Interview Guide, Facilitator’s Guide, modified Cooper-Harper Difficulty Rating Scale, and System Usability Scale (SUS) are provided in Attachments C2, C3, C4, and C5.

Individual participants will not be identified and participation will be strictly voluntary. Personal identifying data will not be collected. Participants will be assured that neither their participation/non-participation nor any responses to the questions will have any effect on their eligibility for, or receipt of, NCI grants, contracts, and/or other services. A copy of the informed consent form to be provided to all participants is available in Attachment C6. Additionally, this protocol has been submitted to the Office of Human Subjects Research (OHSR) for review.

All data will be collected by the contractor. All information provided by respondents will be kept in a secure manner to the extent permitted by law. NCI and the contractor’s Institutional Review Board (IRB) Research Integrity Officers will review the research instruments and ensure that all necessary human subject protection procedures are in place. A letter of confidentiality will be supplied once the contract is finalized with the contractors.

The interview, SUS and Modified Cooper-Harper should take each of the participants approximately 60 minutes to complete. The hourly wage rate is calculated by taking the average of an advocate ($17/hour), a post-doctorate fellow ($25/hour), a junior epidemiologist ($50/hour) and an experienced epidemiologist ($100/hour). The total respondent burden for this effort is 36 hours. This effort will account for 2 percent of the total annual burden hours (1800) granted in our approval package.

	Estimates of Hour Burden and Respondent Cost
Types of Respondents		Instrument	Number of Respondents	Frequency of Response	Average Time Per Response (Hours)	Total Hour Burden	Hourly Wage Rate	Total Respondent Cost
Junior and Experienced Epidemiologists, Post-doctoral Fellows, and Advocates		Interview and Usability Testing (Attach C2 & C3)	36	1	50/60 minutes (1 hour)	30	$48	$1,440.00
		System Usability Scale (Attach C4)	36	1	5/60 minutes (1 hour)	3	$48	$144.00
		Modified Cooper-Harper Questionnaire (Attach C5)	36	1	5/60 minutes (1 hour)	3	$48	$144.00
TOTAL			108			36		$1,728.00

List of Attachments (Attached Below)

Attachment C1: Project Timeline

Attachment C6: Informed Consent Form

List of Attachments (Attached in Separate File)

Attachment C2: Interviewer’s Guide

Attachment C3: Facilitator’s Guide

Attachment C4: System Usability Scale (SUS)

Attachment C5: Modified Cooper Harper questionnaires.

Attachment C1. Project Timeline

The estimated completion dates in the table below assume an August 1, 2010, start date. NIH Evaluation Set Aside funds must be obligated in by August 14, 2010, so this is the latest possible start date for this project.

Activity	Who	Expected Timeline	Approximate Completion Dates
Identify potential participants, extend invitations electronically, and process consents.	EGRP Communications Coordinator and consultant with evaluation expertise	4 weeks	August 27
Heuristic Review and Assessment of Other Relevant Websites and Tools	Consultant with evaluation expertise	8-12 weeks	September 24
Usability testing: Round 1 Sessions	Consultant with evaluation expertise	5 weeks (starting August 30)	October 1
Review of results and recommendations to determine which can be done in short-term and which will require long-term planning	Consultant, EGRP Communications Coordinator, IMS Web Support Contractor	+ 1 week	October 8
Implementation of short-term fixes	EGRP Communications Coordinator, IMS Web Support Contractor	+3 weeks	October 29
Usability testing: Round 2 sessions	Consultant with evaluation expertise	+1 week (in conjunction with Cohort Consortium Annual Meeting on or around 11/4/10)	November 5
Review of results and recommendations to determine which can be done in short-term and which will require long-term planning	Consultant, EGRP Communications Coordinator, IMS Web Support Contractor	+ 1 week	November 12
Implementation of short-term fixes	EGRP Communications Coordinator, IMS Web Support Contractor	+ 3 weeks	December 3
Usability testing: Round 3 sessions	Consultant with evaluation expertise	+6-8 weeks	January 28
Review of results and recommendations to determine which can be done in short-term and which will require long-term planning	Consultant, EGRP Communications Coordinator, IMS Web Support Contractor	+ 1 week	February 4
Implementation of short-term fixes	EGRP Communications Coordinator, IMS Web Support Contractor	+2-3 weeks	February 25

Attachment C6. Informed Consent

Informed Consent Form

Identification of Project	Needs Assessment for http://epi.grants.cancer.gov
Statement of Age of Subject Purpose	I state that I am at least 18 years of age, in good physical health, and wish to participate in a program of research being conducted by Christie Kaefer, M.B.A., in the Epidemiology and Genetics Research Program of the National Cancer Institute, Bethesda, MD 20892. The National Cancer Institute (NCI) proposes conducting an evaluation to determine the effectiveness and usability of the Epidemiology and Genetics Research Program’s (EGRP) website, a key communication channel with extramural cancer epidemiologists (its primary target audience), and other stakeholders, such as cancer advocates.
Procedures	NCI will employ one-on-one usability test sessions using web conferencing software and in-person individual sessions. The sessions will include collection of demographic data from the user and will require the user to complete 2-3 “tasks.” These tasks will be designed to answer the key questions outlined above and will be tailored to the specific user’s area of interest/expertise. (Following the typical approach to conducting usability studies, a predefined data collection instrument will not be employed.) While the user undertakes the tasks, the user will speak aloud their response to the website’s usefulness for that task and their assumptions about how the website should handle the task. Concurrently, the usability consultant will take notes on the steps taken by the user, focusing on difficulties and successes. Collecting data from one-on-one usability test sessions will follow common practice in the usability field, which is to review both the notes taken by the facilitator during the session and review the footage of the screen capture of the session for incidents in which the users’ experience difficulty. Incidents are graded according to how severely they impact the users’ ability to use the site (e.g., critical/show stopper versus low). All results will be reviewed to identify specific aspects of the website that are successful or where changes are needed. Some of the indicators that we anticipate using are: (1) success rates, (2) time to complete tasks, (3) pages visited, (4) pathways participants took, (5) problems participants encountered, (6) what participants said as they worked through tasks, and (7) participants’ confidence in the quality and timeliness of the information they located.
Confidentiality	All information collected in this study will be kept secure, to the extent permitted by law. I understand that the data I provide will be grouped with data others provide for the purpose of reporting and presentation and that my name will not be used. I understand that the discussion will be audiotaped, but my voice will not be played to others besides the research team without my written permission.
Risks	I understand that the risks of my participation are expected to be minimal in nature.
Benefits, Freedom to Withdraw, & Ability to Ask Questions	You may gain knowledge of EGRP that will aid you in your future research. There are no other direct benefits to you from participating in this study. The study results may help NCI’s Epidemiology and Genetics Research Program identify ways to improve its website. I am free to ask questions or withdraw from participation at any time and without penalty. In appreciation for your time, you will receive a $30 gift certificate. You will receive the gift certificate after the interview is completed.
Contact Information of Investigators	Name: Christie Kaefer Position: Communications Coordinator, EGRP, DCCPS, NCI Telephone: 301-435-4906 Email: [email protected]

I have asked and have had answered my questions about this study and consent to participate. I will receive a copy of this consent form.

Printed Name of Research Participant _____________________________

Signature of Research Participant ________________________________

Date______________________

EGRP #0925-0589-05, Page 9