#2_Justifi Memo Attach 2A 2B_GEM_FINAL

Date: May 20, 2011

To: Office of Management and Budget (OMB)

Through: Mary Forbes, Report Clearance Officer, HHS

Seleda Perryman, Program Officer, Project Clearance Branch, OPERA, NIH

Vivian Horovitch-Kelley, PRA OMB Project Clearance Liaison, OMAA, NCI

From: Richard Moser, Research Psychologist

Division of Cancer Control and Population Sciences (DCCPS),

National Cancer Institute (NCI)/NIH

Subject: Generic Sub-study, Evaluation of the Grid Enabled Measures (GEM) Database Website under “Questionnaire Cognitive Interviewing and Pretesting,”

(OMB No. 0925-0589-02, Expiry Date 4/30/2014).

Need and Use of Information

The Behavioral Research Program (BRP), located in the National Cancer Institute’s (NCI’s) Division of Cancer Control and Population Sciences (DCCPS), initiates, supports, and evaluates both basic behavioral research efforts and research that supports cancer interventions. Its mission is to increase the breadth, depth, and quality of cancer prevention and control in behavioral research.

To address the need for better quality measures in behavioral research, specifically in the area of health science, BRP developed the Grid Enabled Measures (GEM) Database Website (https://www.gem-beta.org/). Through the promotion of quality research measures, the GEM Database Website fully supports the goals of BRP as outlined in their mission statement by: (1) Increasing the breadth of research measures available to health science researchers, by way of a collaborative online system that encourages users to download from and contribute to an ever growing online database of research measures; (2) providing users with multiple associated measures for key constructs in health science (e.g., Anxiety, Numeracy, Stress); and (3) implementing a user driven interactive rating system of database measures, insuring the highest quality measures are highlighted.

Utilizing collaborative web technology to create a virtual community of health scientists, the GEM Database Website has two over-arching goals: 1) Provide an interactive online system that enables users to search for, share, download, and rate validated behavioral research measures that are tied to theoretically-based constructs; and 2) facilitate the sharing of harmonized data, resulting from the use of shared measures, through features that allow users to search, preview, upload, download, and provide meta-data (e.g., data collection mode, collection period, availability) on data sets.

The site will therefore aid in the initiation of more powerful studies and ‘prospective meta-analyses’ through the promotion of shared measures and harmonized data. These aims fit well with the Department of Health and Human Services (HHS) goals of the Open Government initiative (see for more information: http://www.hhs.gov/open/) to make data more freely available for use by others and to drive consensus on health indicators that can be shared and used by researchers, practitioners, and local communities.

We are proposing three iterative rounds of usability testing for the GEM Database Website to insure that users are able to easily navigate the site, utilize site features, and interact with other users. Numerous texts have indicated that usability testing can lead to improved user satisfaction, streamlined Website design, more intuitive system feedback mechanisms, and improved Website learnability ^1,2,3;outcomes that support the goals of the GEM Database Website. In particular, usability testing of the GEM Database Website would provide elucidation on the following questions:

1. What functions of the Website are beneficial to users (e.g., data sharing and downloading, database of constructs and measures, ability to share and download research measures, detailed meta-data about constructs, measures, and datasets)?

2. Does the Website successfully communicate the purpose and goals of the GEM Database?

3. Do users understand how to access and use interactive functions on the Website, (e.g., downloading, submitting, providing feedback)?

4. How easily and quickly were users able to navigate to a desired page?

5. Do users understand how to use the Website to add or edit meta-data relating to constructs, measures, and datasets?

6. Does the Website clearly communicate how users are able to share harmonized data sets and the rationale for sharing datasets?

7. To what extent are user’s research and information needs being met by the Website? If its needs are not being met, how could the Website be revised to better meet these needs?

8. Is the information on the Website written in a way that meets the needs of all users (e.g., users who both scan and systematically read information)?

9. How easily and quickly are users able to download resources from the Website (e.g., data sets, measures)?

Due to its community participation focus, a number of stakeholders within the health science community (e.g., researchers, practitioners, policymakers, the extramural community) would be benefited by, and have the potential to shape the creation and sustainability of, the GEM Database Website. So it is of interest to the health science community, and individual health science researchers, that shared measures and harmonized data sets on the Website are easy to find, upload, download, and rate.

There have been no previous efforts to conduct usability testing for the GEM Database Website, so there is a need to assess the ways in which the beta version of the site could be amended to meet the needs of users. Quantitative and qualitative indicators of usability issues, collected during testing, will be used to create written and verbal reports that outline site re-design options. The GEM Database Website Team will use these reports to re-design the site throughout and following the usability testing process.

Participants

Each round of usability testing will involve 20-25 participants carefully sampled to represent a typical member of potential user groups (health science researchers, practitioners, and policymakers). Potential participants will be recruited through health science professional listservs (e.g., American Public Health Association, Society of Behavioral Medicine) and among current NCI/NIH grantees and fellows. The universe of potential participants constitutes roughly 7,500 persons. Overall, the selection process for the three usability testing panels will consider both a participant’s research experience (expert vs. novice) and their familiarity using the GEM Database Website (have used vs. have not used). The participant pool would ideally be balanced between research “experts” who have used the site, research “experts” who have not used the site, research “novices” who have used the site, and research “novices” who have not used the site. In effect, this would create a 2x2 design for user selection. Potential participants will be recruited through the use of email or via snail mail, using a letter that explains the goals of the study (Attachment 2A).

Following this initial recruitment, all potential users will be provided an informed consent form (Attachment 2B) that details potential risks and benefits from engaging in the usability testing. If a participant consents to participate, they will be provided a short survey (Attachment 2C) that ascertains basic participant background information (e.g., area of work, field of study), site familiarity (e.g., uploaded or downloaded resources to and from the site, used the meta-data functions on the site) and level of research experience. This information will be used to divide participants into the four user groups discussed previously.

Methodology and Research Instrument

The study will employ three iterative rounds of one-on-one usability test sessions. The sessions will consist of the user being asked to perform a number of typical site “tasks.” These tasks will be designed so the user will interact with key site functions/features. The facilitator will lead users through “tasks” using an interview guide, which will contain both explicit “task” directions and follow up questions that assess possible usability issues (Attachment 2D). While the users navigate through the site to complete the “tasks”, they will narrate their thought processes and answer facilitator questions when prompted. In particular, users will be asked to comment on the usefulness of site functions/features and describe their thoughts on how the Website should handle a particular “task”. Concurrently, the facilitator will take notes on the steps taken by the user to fulfill a “task”, focusing on difficulties and successes, as well as noting any unprompted user comments. At the end of each session, a brief interview will be conducted with the user to probe for any additional information and/or tool needs they wish the Website could have provided. All sessions will be completed remotely, using a combination of a web meeting software (e.g., GoToMeeting) and VOIP/ phone teleconferencing. All sessions will include screen capture recordings, as the facilitator will “share” the screen with a participant during the session to note the process taken by a user to complete a particular “task.”

The data collected from one-on-one usability test sessions will follow common practice in the usability field. This is to review both the notes taken by the facilitator during the session and the footage of the screen capture. Incidents are graded according to how severely they impact users’ ability to use the site. All results will be reviewed to identify specific aspects of the Website that are successful or where changes are needed. Some of the indicators that we anticipate using are: (1) success rates, (2) time to complete tasks (3) pages visited (3) completion pathways (4) problems, (5) what participants said as they worked, and (6) participants’ confidence in the quality and timeliness of information located.

Both quantitative and qualitative data from each round of usability testing will be examined. For quantitative data, we will calculate percentages of participants who succeeded at each task, average time to complete tasks, average number of pages visited in each task, and the frequency of specific problems. For qualitative data, we will read through the notes carefully looking for patterns, and categorizing them by task type, problem type, user types, and frequency of occurrence. We will also analyze the interview guide data to identify any salient needs that the GEM Database Website failed to address. We expect that the quantitative and qualitative data analysis will provide an overall assessment of the design, functionality, and content of the GEM Database Website, and identify practical areas for improvement in these aspects of the site.

Prior to usability testing, a usability consultant will complete a heuristic evaluation and review other relevant Websites and tools. These processes involve a consultant independently examining the GEM Database Website and judging its compliance with a standard set of usability principles (http://www.usability.gov/methods/test_refine/heuristic.html), then examining specific Websites for comparison. These sites will be selected for their relevance to the design, functionality, and content of the GEM Database Website. The following aspects of other relevant Websites and tools will be reviewed: 1. page layout, 2. navigation, 3. links, 4. text appearance, 5. graphic design, 6. accessibility, and 7. search functions. Following these reviews, the consultant will create a qualitative report that indicates Website re-design suggestions. The GEM Database Website Team will consider these suggestions and re-design the site prior to usability testing. Participants will not receive incentive for their participation in usability testing. The completion date for this evaluation is estimated to be December 1, 2011, with a June 2011 start date. The heuristic review and review of relevant Websites will happen concurrently, with both processes anticipated to take 3 to 4 weeks to complete. The iterative usability testing is anticipated to take 5 months, with each round anticipated to take 6-8 weeks to complete.

Other Considerations

A request for Office of Human Subjects Research exemption was submitted and we are awaiting approval.

Information from this survey will be kept private under the Privacy Act. Personally identified information (PII), such as the participant’s name, email address, telephone number and availability, will be collected for the sole purpose of pre-testing contact (e.g., scheduling usability testing sessions) and to conduct remote usability testing. This information will be secured by password in a database, accessible only to those Westat staff members scheduling and conducting the usability sessions.  De-identified data from the usability sessions will be presented to NCI.  Once the study is complete, the PII will be destroyed. The NIH Privacy Act Officer has been consulted and a memo stating that the data is covered by the Privacy Act will be obtained.

Burden

The survey should take each of the participants approximately 5 minutes (.083 hours) to complete. The usability testing should take each of the participants approximately 60 minutes (1 hour) to complete. Therefore we expect the total respondent burden for this proposed effort to be 1.083 hours. This effort will account for less than 2.25 percent of the total burden hours granted in the full generic OMB clearance package. To date, a total of 0 burden hours have been used of the 3600 hours that were requested. Estimated cost to the Federal Government is $50,000.

Estimates of Burden Hours
Types of Respondents	Instrument	Number of Respondents	Frequency of Response	Average Time Per Response (Hours)	Total Hour Burden
Researchers, practitioners, and policymakers working in health science (Federal Government, State and Local Government, Private Sector, Individuals)	Participant Survey (Attachment 2C)	75	1	5/60 (0.083 )	6
	Facilitor Interview Guide (Attachment 2D)	75	1	60/60 (1)	75
Total		150	1		81

Attachments (below)

2A: Recruitment Letter/Email

2B: Informed Consent Form

Attachments (in separate file)

2C: Participant Survey

2D: Facilitator Interview Guide

Attachment 2A: Recruitment Letter/Email

Dear Colleague,

The National Cancer Institute recently released the GEM Database Website (https://www.gem-beta.org/), a resource that promotes the use of shared behavioral and social science measures for individuals working in health science. In an effort to better understand the usefulness and usability of this site for individuals such as yourself, over the next ___ months we will be conducting a series of usability testing sessions. We would greatly appreciate your participation in one of these sessions, as your input will help us improve this valuable research tool. Participation would require no preparation on your part and sessions will last approximately 60 minutes.

If you agree to participate, our contractor, __________ will arrange a mutually convenient date and time in (Month and Year) to arrange a remote testing session (you will complete the usability session from your own location, using your own computer). Using a combination of web meeting software (e.g., GoToMeeting) and VOIP/ phone teleconferencing, you will interact with a session facilitator and be asked to explore and comment upon features of our site. To utilize the web meeting software, you will need to install a small utility (directions will be provided by facilitator) which will enable the facilitator to view your computer screen as you work through the website.

The feedback you provide during your session will be combined with input from other sessions and used to re-design and improve our website. However, your individual comments will be kept secure to the extent permitted by law.

To assess your familiarity with the site and previous research experience, we will ask you to complete a short questionnaire prior to the session that will take less than 5 minutes of your time.

If you would like to participate, please contact __________ at __________ and he/she will work with you to schedule a date and time for the session and provide information on next steps.

Thank you for considering this request.

Regards,

Richard Moser, Ph.D.

Research Psychologist,

[email protected]

301-496-0273

Division of Cancer Control and Population Sciences,

National Cancer Institute, National Institutes of Health

Attachment 2B: Informed Consent Form

Identification of Project	Grid-Enabled Measures (GEM) Web Site Evaluation
Statement of Age of Subject	I state that I am at least 18 years of age, in good physical health, and wish to participate in a program of research being conducted by XXXXXX (session moderator) for the National Cancer Institute, Bethesda, MD 20742.
Purpose	The purpose of this research is to reach a better understanding of how people use and interact with the GEM website and to improve the way information is presented on the GEM web site
Procedures	Participants will be asked to use the GEM web site and comment on the experience and discuss their opinions and needs around measure and data sharing. The total time involved, including instructions will be no more than 65 minutes.
Confidentiality	All information collected in this study will be kept private under the Privacy Act. I understand that the data I provide will be grouped with data others provide for the purpose of reporting and presentation and that my name will not be used. I understand that screen activity (desktop, open windows, mouse clicks) and the voices of myself and the moderator will be recorded during the research session. My face will not be recorded, and the full recording will not be shown to others besides the research team. Short (< 3 minutes) clips of the recording may be used as part of academic presentations to researchers investigating how the presentation of statistical data affects how people understand the information.
Risks	I understand that the risks of my participation are expected to be minimal in nature.
Benefits, Freedom to Withdraw, & Ability to Ask Questions	I understand that this study is not designed to help me personally but that the investigators hope to update and redesign the GEM web site in order to make information on it easier to find for the general public. I am free to ask questions or withdraw from participation at any time and without penalty.
Contact Information of Investigators	Name: Richard Moser, Ph.D. Position: Research Psychologist, National Cancer Institute Telephone: 301-496-0273 Email: [email protected]

Printed Name of Research Participant _____________________________

Signature of Research Participant ______________________________

Date ______________________

1 Michael O. Leavitt and Ben Shneiderman, Research-Based Web Design & Usability Guidelines (Washington, DC: HHS, 2007)

2 Anetta Hinchliffe and W. Kerry Mummery, Applying usability testing techniques to improve a health promotion website, Health Promotion Journal of Australia 19.1 (2008): 29-35

3 Jeffrey Rubin and Dana Chisnell, Handbook of Usability Testing (Indianapolis: Wiley Publishing, Inc., 2008)

8