#1 Justif Memo Attach 1B 1D_5-18-2011

Date: May 18, 2011

To: Office of Management and Budget (OMB)

Through: Mary Forbes, Report Clearance Officer, HHS

Seleda Perryman, Program Officer, NIH Project Clearance Branch, OPERA, NIH

Vivian Horovitch-Kelly, PRA OMB Project Clearance Liaison, OMAA, NCI

From: Amy Subar, Nutritional Epidemiologist, Applied Research Program and

Gordon Wills, Project Officer

Division of Cancer Control and Population Science (DCCPS)

National Cancer Institute (NCI)/NIH

Subject: Usability Testing of the ASA24 Respondent Site

Generic Sub-study under “Questionnaire Cognitive Interviewing and

Pretesting”, OMB No. 0925-0589-01, Expiry Date 4/30/2014

The National Cancer Institute (NCI) proposes conducting usability testing of the web-based Automated Self-Administered 24-Hour Dietary Recall (ASA24) Respondent system. The ASA24 is under development by NCI and its contractors, and is designed to allow a cost-effective manner for subjects in dietary assessment studies to report the foods they have eaten over the preceding 24-hour period.

Background and need for information. A major aspect of NCI's mission in cancer prevention and control is monitoring the influence of individual and societal risk factors and health behaviors that mediate cancer incidence, morbidity, mortality, and survival, both directly and indirectly. Dietary intake is a risk factor of substantial importance in this area of cancer control research. To fulfill this mission, the Applied Research Program (ARP), Risk Factor Monitoring and Methods Branch (RFMMB) (http://riskfactor.cancer.gov/), within the Division of Cancer Control and Population Sciences (DCCPS), supports the development and evaluation of high quality, affordable dietary assessment instruments for use by investigators and clinicians in their research. Over the past five years, ARP has supported the development of the Automated Self-Administered 24-hour Dietary Recall (ASA24) website system. The system to be evaluated currently is the Respondent site, which collects self-report dietary intake information for the previous day from study participants. Specifically, we will test a new Respondent interface that is being developed to overcome limitations of a beta version currently in use by the research community, and to integrate new features.

The ASA24 website system is designed to allow Researchers to collect self-administered 24-hour dietary recall data for their participants at the individual-level for epidemiologic, clinical, and behavioral studies that focus on the relationship of diet to the development of several chronic diseases, including cancer, heart disease, and Type-II Diabetes. In the past, 24-hour recalls, among the most trusted of dietary assessment methods, were collected by trained interviewers (as is currently done in the National Health and Nutrition Examination Survey OMB No. 0920-0237; expiration 11/30/2012). As such, this intensive, high-quality method was almost never used in large scale, population-based research because of the need to collect data at several points to estimate usual intake; the great expense of interviewers; and limitations with respect to feasibility. Development of an automated self-administered system that enables the collection of recall data for multiple days among large cohorts will greatly facilitate research in the dietary assessment area.

More specifically, the ASA24 Respondent website software is intended to provide user-friendly, easy-to-navigate software for self-administered 24-hour recalls among adults. The software is web-based and intended to be accessible to those of low-literacy. Further, both English and Spanish versions are supported. Researchers within the nutritional epidemiology community and the general public are regarded as stakeholders or potentially interested parties. Nutritionists and epidemiologists have expressed considerable interest in the use of the ASA24 system, now available for researchers (over 200 requests for access have been received from outside researchers – i.e., those not sponsored by NCI -- and the initial version is being actively used in numerous studies). In addition to its use in research studies, the ASA24 is being used by clinicians and educators in undergraduate and graduate nutrition courses. Further, respondents using the ASA24 System will of course be interested in its usability; once committed to participating in research, they will want a system that is both user-friendly and understandable.

Prior evaluation of the initial ASA24 Respondent website system has consisted of very preliminary usability testing during development by Westat, the prime contractor for system development. Five development/usability tests of 9 or fewer participants were conducted: 1) a computer-based test to determine whether respondents preferred reporting foods using a meal-based vs. unstructured method (Subar AF, Thompson FE, Potischman N, et al., J Am Diet Assn. 2007;107:1002-1007); 2) a sorting task and focus group to aid in development of a food grouping system for use in browsing; 3) a paper-based focus group testing different screen formats to be used for food reporting; 4) a paper-based focus group testing that varied screen formats to be used for asking detailed questions about food preparation and portion size; and 5) a study to ascertain the optimal method for displaying food images to aid in portion size estimation (Subar AF, Crafts J, Zimmerman TP, et al. J Am Dietet Assn 2010;110:55-64).

However, based on this testing, it was apparent that a revised version of the ASA-24, exhibiting improved usability, is needed. As such, NCI has enlisted Westat to develop an updated ASA-24 system that is markedly different from the original, and which is now nearing completion. This system is in need of systematic usability testing at this time. As such, the overall purpose of the proposed project is to ascertain the usability of the revised ASA24 Respondent system in its entirety. The goal is to evaluate the data collection strategies within the current system, and to determine the extent to which the ASA24 dietary collection system is operating as intended, in terms of its acceptability and clarity of use. Recently, we conducted a small (n = 9) usability test using members of the general adult population as participants. However, at this time we are especially concerned with ASA24 usability for elderly/low-literacy populations. Further, we propose testing a Spanish-language version of the system.

Key question(s) to be addressed through usability testing of elderly/low-literacy and Spanish-language versions:

1. How easy or difficult is the current ASA24 Respondent website system for users – especially those of low literacy?

2. How well does the Spanish version of the ASA24 Respondent website function?

3. What features of the Respondent website likely lead to user error, insufficient motivation, break-offs, difficulties, etc.?

Data Collection Method. NCI staff will direct the staff of User Centered Design, a software usability testing contractors, to conduct two usability tests of the ASA24 Respondent website, anticipated to be conducted during summer, 2011, as follows:

1) 12 older or low-literate adults

2) 12 Spanish-speaking adults (using the Spanish version of ASA24)

Subjects will be recruited by an outside recruiting firm that will locate appropriate individuals for testing. A recruitment screener will administered to potential individuals to confirm they fit criteria to be part of the study (Attachment 1A). The recruitment firm will identify individuals contained within an existing participant database and invite them to complete the usability test. Usability testing will be conducted by staff of User Centered Design and a consent form will be obtained from the participants prior to testing (Attachment 1B).

The participants will be emailed a confirmation letter & directions, and will also be called the day before the testing to confirm their attendance. Each usability test will be conducted at NCI facilities at 6116 Executive Blvd, Rockville, MD, or if necessary, will be conducted over the telephone. The usability test will be approximately one hour in length. Usability data shall be collected by trained facilitators who administer scripted questions, in particular, those testing respondents’ understanding of system usage (Attachment 1C). The usability testing shall also allow respondents to provide suggestions for improvement in an open-ended format. Based on the results, the contractor will make recommendations for changes to improve the design to maximize usability. Screen shots of the new Respondent interface will also be tested (Attachment 1D).

Participants will receive $50 for their participation when testing is conducted remotely, over the telephone; and $100 when conducted in the NCI usability lab (for which participants need to travel to that location). These amounts are the usual levels provided to individuals contained in the database maintained by the recruitment contractor (tentatively, the firm Engage in Depth), and that have been found to be the amounts required to effectively recruit individuals who may have relatively low incomes, difficulty arranging travel, language barriers (for low literacy or Spanish-speaking individuals), or who otherwise have been found to be challenging to incentivize for usability tests.

Personally identified information (PII) such as the participant’s name, email address, telephone number and availability will be collected for the sole purpose of pre-testing contact. Additionally, for the participants who will have remote testing, a mailing address will be obtained to mail the stipend. This information will be secured by password in a database, accessible only to User Center Design staff. De-identified data will be presented to NCI. Once the study is complete, the PII will be destroyed.

An OHSR Exemption from IRB review has been obtained.

The total respondent burden for this effort is estimated at 26 hours. This effort will account for less than 1 percent of the total burden hours (3600) granted in our approval package.

Burden Hours
Type of respondent	Instrument	Number of Respondents	Frequency of Response	Estimated Time Per Response (Hours)	Annual Burden Hours
Older or low-literacy	Screener (Attachment 1A)	12	1	5/60 (0.08)	1
	Testing Guide (Attachment 1C)	12	1	60/60 (1)	12
Spanish-speaking	Screener (Attachment 1A)	12	1	5/60 (0.08)	1
	Testing Guide (Attachment 1C)	12	1	60/60 (1)	12
TOTAL		48			26

Please feel free to contact Amy Subar (301)-594-0831

Attachments (below)

Attachment 1B. Informed consent form

Attachment 1D. ASA24 Respondent system screenshots

Attachments (in a separate file)

Attachment 1A. Screener for potential ASA24 participants

Attachment 1C. Usability testing guide

Attachment 1B: Informed Consent Form

Identification of Project	ASA24 Usability Test
Statement of Age of Subject	I state that I am at least 18 years of age, in good physical health, and wish to participate in a program of research being conducted by Amy F. Subar, PhD in the Risk Factors Methods and Monitoring Branch of the National Cancer Institute, Rockville, MD 20852.
Purpose	The purpose of this research is to test the design of a computer website (the ASA-24) to make sure it is easy to use.
Procedures	Participants will be asked to work with the ASA-24, perform certain tasks, and answer questions about the experience. The total time involved, including instructions will be no more than 60 minutes. A total of about 24 individuals such as yourself are being asked to participate in a test of the ASA24. As compensation for your efforts you will receive [$100 if in-lab; $50 if remote interview done by phone] as a thank you for your time.
Confidentiality	All information collected in this study will be kept secure to the extent permitted by law. I understand that the data I provide will be grouped with data others provide for the purpose of reporting and presentation and that my name will not be used.
Risks	I understand that the risks of my participation are expected to be minimal in nature.
Benefits, Freedom to Withdraw, & Ability to Ask Questions	I understand that this study is not designed to help me personally but that the investigators hope to check the new design of the ASA24 diet recall application in order to make the experience future participants in studies easier. I am free to ask questions or withdraw from participation at any time and without penalty.
Contact Information of Investigators	Name: Amy F. Subar Position: Nutritionist Telephone: 301-594-0831 Email: [email protected]

Printed Name of Research Participant _____________________________

Signature of Research Participant ________________________________

Date______________________

Attachment 1D. ASA-24 Respondent System Screenshots

Figure 1: Log in screen

Figure 2: Welcome screen

Figure 3: Instruction screen

Figure 4: Meal details screen

Figure 5: Actions, Browse/Search, and My Foods and Drinks Quick List screens

Figure 6: Search results

Figure 7: Meal gap review pop up

Figure 8: Food portion size selector

Figure 9: Details question

Figure 10: Beverage portion selector (container type)

Figure 11: Beverage portion selector (amount drank)

Figure 12: Done with details pop up

Figure 13: Final review

Figure 14: Frequently forgotten foods pop up

Figure 15: Last chance pop up

Figure 16: Supplements pop up

Figure 17: Duplicate check (also functional for foods)

Figure 18: Actions, Browse/Search, and My Supplements Quick List screens

Figure 19: Supplements portion size question

Figure 20: Supplements final review

Figure 21: Thank you pop up

26