#3_Justif Memo & Attach A2 & A3_TS Toolkit_rev3

Date: June 10, 2010

To: Office of Management and Budget (OMB)

Through: Seleda Perryman, DHHS Report Clearance Officer

Mikia Currie, Program Analyst, NIH Project Clearance Branch, OPERA

Vivian Horovitch-Kelly, NCI OMB Project Clearance Liaison, OMA

From: Kara Hall Health Scientist, Behavioral Research Program,

Division of Cancer Control and Population Science (DCCPS)

National Cancer Institute/NIH

Gordon Wills, Project Officer, DCCPS, National Cancer Institute/NIH

Subject: Needs Assessment and Usability Testing of the Science of Team Science Toolkit

Website

Generic Sub-study under “Questionnaire Cognitive Interviewing and

Pretesting”, OMB No. 0925-0589-03, Expiry Date 05/31/2011.

The National Cancer Institute (NCI) proposes conducting a needs assessment and usability testing of the Science of Team Science (TS) Toolkit Website through web-based testing and follow-up interviews. The goal of this needs assessment and usability testing is to improve the functionality and content coverage of the TS Toolkit Website, which was developed as an on-line interactive resource center for users to learn about, engage in, and conduct research related to team science. The usability testing and interviews will be conducted on stakeholders from the team science community as well as those who are new to the team science field. The data collected will be analyzed and incorporated into the refinement of the website.

Background

Over the past few decades there has been an increase in funding of large initiatives designed to solve complex public health problems. These large initiatives typically require collaborative work across disciplines which have contributed to a shift from research endeavors primarily led by an individual scientist toward more team-based approaches. As a result, there has been an increased effort to better understand the processes and outcomes associated with conducting team based science. This has led to the emergence of a new field called the Science of Team Science (STS).

The STS field has expanded dramatically, in part due to the 2006 Science of Team Science (STS) Conference, Bethesda, Maryland, and the publication, in 2008, of a supplement about the science of team science in the American Journal of Preventive Medicine. Both of these projects were led by the Science of Team Science researchers in the Division of Cancer Control and Population Sciences (DCCPS), at the National Cancer Institute (see details at http://cancercontrol.cancer.gov/brp/scienceteam/). As an emerging leader of the STS field, the Science of Team Science team at DCCPS recognized the need to enable and support the further development of this rapidly emerging field within and outside the NIH community and in response has developed the Team Science (TS) Toolkit Website, an interactive website that serves as an on-line repository for learning, engaging, and studying team science. The TS Toolkit Website is designed to be an interactive website that includes: 1) a publicly accessible database, where users can upload and download resources to study and facilitate TS; and 2) new media including expert blogs, user groups, and a listserv, designed to create a forum for the continuing development of the team science community.

We launched the Beta version of the TS Toolkit Website on April 23, 2010 at the First International Science of Team Science (STS) Conference in Chicago, and the presentation of the website features were well received by the conference participants. We plan to release the fully functional version of the site to the public in July 2010 incorporating initial feedback received at the conference. The TS Toolkit Website serves a unique role in the rapidly emerging STS field, and it has not been evaluated before. Given that the publicly accessible version of the website seeks to serve a variety of stakeholders, it is critical that we solicit information through needs assessment and usability testing to ensure that the website functions well and its content and organization meets the needs of a broad range of audiences.

Participants

Participants in this usability testing and needs assessment will be scientists who have been involved in team science practices such as those funded through P (center grants) or U (cooperative agreement) mechanism by NIH ICs; researchers studying team science (e.g., evaluators, communication researchers, organization and management experts, etc.), program directors at NIH ICs, and those who are interested in TS but haven’t necessarily had experience in conducting team science oriented research (e.g., R01 investigators). Participants’ background/role information will also be collected through a brief questionnaire (see Attachment A1 for Participant Profile Questionnaire)

Participants will be invited to participate by email. We expect 9 users to complete testing and a follow up interview each round of testing for a total of 27 participants. The Participant Profile Questionnaire, usability test and follow-up interview should take each of the participants approximately 95 minutes to complete.

Methodology

The DCCPS team proposes conducting needs assessment and usability testing of the website through on-line website navigation/usability testing and a follow-up one-on-one interview with identified users.

User identification and selection: Three rounds of usability testing will be conducted. Each round of usability testing will involve at least 9 individuals, representing the range of users who are studying, practicing, or interested in learning about TS. The overall number of “experts” and “novices” in TS and STS will be balanced. However, there will be a major emphasis on “expert” users during the initial round of testing, and increasing recruitment of more “novice” users in later testing. “Expert” users will be recruited from participants of the April 2010 conference on the Science of Team Science and through professional relationships of the members of the project team. Among these experts, we will include individuals who have already uploaded or downloaded resources to and from the site, and interacted with the social media components of the site. Novice users will include NCI/NIH grantees who have never been funded under TS grant programs, and program directors who manage predominantly single-investigator grants. A consent form will be administered to each user prior to their participation in the study (Attachment A2).

Data collection: The study will employ one-on-one usability test sessions and follow-up interviews. The sessions will consist of the user being given 2-3 “tasks.” These tasks will be designed to answer the key research questions, and will be tailored to the specific user’s area of interest/expertise. Users will be asked to comment on the site’s usefulness for that task and their assumptions about how the website should handle the task. Concurrently, the usability session facilitator will take notes on the steps taken by the user, focusing on difficulties and successes. All results will be reviewed to identify specific aspects of the website that are successful or where changes are needed. Some of the indicators that we anticipate using are: (1) success rates, (2) time to complete tasks, (3) pages visited, (3) pathways, (4) problems, (5) what participants said as they worked, and (5) participants’ confidence in the quality and timeliness of information located.

Following the navigation of the website, a one-on-one 30-minute interview will be conducted with each user. The interview questions will focus on the following areas: general exploration of the site, general impressions and appeal of the website, content and organization of the website, and additional user needs and areas of improvement.

An invitation letter will be emailed to potential participants of the study. The Science of Team Science team will work together with the STS conference organizer to distribute the letter (Attachment A3). The invitation letter will explain the purpose of the study, the formative nature of the needs assessment and usability testing and how the results will be used by NCI to better support the development of the TS Toolkit Website to serve a wider range of users.

The data collected will be stored on a secure site and will be 508 compliant. The needs assessment and usability testing will be conducted by a contractor, who will work with the NCI project officer and the OCE Usability Lab, who will assist with the usability testing via their on-site facilities, when applicable.

After each round of the usability testing is completed, the contractor shall provide the NCI project team with recommendations for improvement and work with the Web design team to incorporate those recommendations into the website for the subsequent rounds of usability testing.

Users’ responses to the usability interview questions will be secure, to the extent permitted by law. After data collection and review, the contractor will synthesize the results by the role of the respondent, and names will not be recorded in the report of results.

Research Instrument

The needs assessment and usability testing will be conducted through web-based testing and individual interviews (See Attachment A4 for the Interview Guides). The goal of this needs assessment and usability testing is to improve the TS Toolkit Website in terms of content coverage and functionality of the website (e.g., ease of use and navigation, usefulness and completeness of interactive databases). The usability testing and needs assessment will address the following questions:

1. How well did users to the TS Toolkit Website learn how it operates and what it has to offer?

2. How useful are the resources contained in the website to TS and STS researchers and practitioners?

3. How easily and quickly did potential users download resources from the database that are specific to their needs?

4. How easily and quickly did potential users upload resources to the database?

5. How well have the Expert Blog, listserv, and discussion boards stimulated growth of the community? How have these social media tools been used? What other outcomes have occurred from the use of these social media?

6. What are the additional needs of users that can be accommodated through this website to achieve the ultimate goal of the website (i.e., foster and support the growth of the STS and TS community?)

7. What changes are needed to ensure that the website complies with Section 508, HHS web standards, and best practices of website design?

The total respondent burden for this effort is estimated at 43 hours. This effort will account for 2 percent of the total annual burden hours (1800) granted in our approval package.

Estimates of Hour Burden and Respondent Cost
Types of Respondents	Instrument	Number of Respondents	Frequency of Response	Average Time Per Response (Hours)	Annual Hour Burden	Hourly Wage Rate	Annual Respondent Cost
Stakeholders	Participant Profile Questionnaire (Attachment A1)	27	1	5/60 (0.083)	2	$75	$150.00
	Interview and Usability Testing (Attachment A4)	27	1	90/60 (1.5)	41	$75	$3,075.00
Total		54			43		$3,225.00

Please feel free to contact Kara Hall at 301-435-7789 if you have any questions.

List of Attachments (Attached Below)

A2: Informed Consent Form (attached below)

A3: Invitation Letter to Participant (attached below)

List of Attachments (Attached In Separate File)

A1: Participant Profile Questionnaire

A4: Interview Guides

Attachment A2: Informed Consent Form

Identification of Project	Needs Assessment and Usability Testing of the Science of Team Science Toolkit Website
Purpose	The National Cancer Institute (NCI) proposes conducting a needs assessment and usability testing of the Science of Team Science Toolkit Website (hereafter refer as TS Toolkit Website), an interactive on-line resource for the Team Science (TS) community to share tools, measures, instruments, and bibliographies related to team science as well as exchanging ideas and knowledge through the various social media outlets supported by the website (e.g., expert blog, listserv, announcement). The goal of this needs assessment and usability testing is to improve the functionality and content of the TS Toolkit Website so that the site will serve the needs of a broader range of users representing both experts and novice users.
Procedures	NCI proposes web-based testing using pre-identified scenarios and a follow-up interview with participants. Participant will be asked about his/her experience with using the website while completing a few tasks with the usability testing consultant (about an hour). This will be followed by an interview for suggestions on possible improvements to the website (0.5 hour).
Confidentiality	All information collected in this study will be kept secure, to the extent permitted by law. I understand that the data I provide will be grouped with data others provide for the purpose of reporting and presentation and that my name will not be used. I understand that the interview will be audiotaped, but my voice will not be played to others besides the research team without my written permission.
Risks	I understand that the risks of my participation are expected to be minimal in nature.
Benefits, Freedom to Withdraw, & Ability to Ask Questions	I understand that this study is not designed to help me personally but that the investigators aim to solicit information for improving the website functionality and content and addressing the needs of the TS community. I am free to ask questions or withdraw from participation at any time and without penalty. My participation is voluntary. I will receive a $25 gift card as a token of thanks.
Contact Information of Investigators	If you have questions about being a research participant or additional questions about this research, contact: Kara Hall Position: Director of the Science of Team Science Team, DCCPS/NCI Telephone: 301-594-9056 Email: [email protected]

I have asked and have had answered my questions about this study and consent to participate. I will receive a copy of this consent form.

Printed Name of Research Participant _____________________________

Signature of Research Participant ________________________________

Date______________________

Attachment A3: Invitation Letter to Participant

May 13, 2010

Dear Colleague,

The National Cancer Institute (NCI) is conducting a needs assessment and usability testing of the Team Science (TS) Toolkit Website, a newly developed on-line resource center devoted to the practice and study of team science. The website is designed to be an interactive website that includes: 1) a publicly accessible database, where users can upload and download resources to study and facilitate team science; and 2) new media including expert commentary, user groups, and a listserv, designed to create a forum for the continuing development of the team science community.

The goal of this needs assessment and usability testing is to improve the functionality and content coverage of the TS Toolkit Website so that it can serve as an effective tool to enable and support the further development of the rapidly emerging field of team science.

We cordially invite your participation in our efforts to improve this website through the usability testing. You will use a computer, remotely or on-site at NCI, to navigate the TS Toolkit Website while completing several tasks with the usability testing consultant. This portion of the usability testing will last approximately one hour. You will then complete a half hour interview with the usability testing consultant about your opinion of the site. We welcome feedback from users with previous experience and knowledge about the website in addition to novice users. You will receive a $25 gift card for participating in this usability testing as a token of thanks.

The testing will take place during Fall 2010 and Spring 2011, and it can take place at your home or work at a time that is convenient for you. If you would like to schedule a session, please e-mail _________(contractor’s name, email) or call ____(contractor’s name) at xxx-xxx-xxxx.

Thanks in advance for your support.

Sincerely,

Kara L. Hall, Ph.D. 

Director, Science of Team Science (SciTS) Team

Co-Director, Theories Project

Health Scientist, Office of the Associate Director

Division of Cancer Control and Population Sciences

National Cancer Institute

 

Science of Team Science #0925-0589-03, Page 6