Usability SS Part A

FDA COMMUNICATIONS USABILITY TESTING

REQUEST FOR OMB REVIEW

AND SUPPORTING STATEMENT

Food and Drug Administration

Office of Planning

DEPARTMENT OF HEALTH AND HUMAN SERVICES Public Health Service

Food and Drug Administration

_____________________________________________________________________________________________

Memorandum

Date	February 3, 2011
From	PRA Specialist, Paperwork Reduction and Records Management Staff Office of Information Management
Subject	Request for Generic Clearance of FDA Usability Testing
To	Human Resources and Housing Branch Office of Information and Regulatory Affairs, OMB
Through	HHS Reports Clearance Officer_______

The Food and Drug Administration (FDA) is seeking approval from the Office of Management and Budget (OMB) for the generic clearance, FDA Usability Testing. The purpose of the information collection is to provide tools to test FDA communications on specific topics and to assist in the development and modification of communication messages to promote public health and compliance with regulations. FDA is requesting approval for collecting information through a variety of research methods for developing and testing communications about products that are regulated by FDA.

FDA expects that developing and testing communications will require using a variety of research methods. Therefore, FDA is requesting approval of a generic clearance to approve the range of complementary research methods needed to comprehensively and quickly conduct formative testing of communication messages on FDA-regulated products. The requested generic clearance for testing formative communications seeks approval for up to 125 studies annually using such methods as individual in-depth interviews, mall-intercept interviews, focus groups, self-administered surveys, gatekeeper reviews, omnibus telephone surveys, and online surveys. The content, timing, and number of respondents to be included in each project will vary, depending on the nature of the message/material being tested, the methodology selected, and the target audiences. Total annual respondent burden is calculated at 41,041 hours.

FDA plans to use the data collected under this generic clearance to inform its communications campaigns. The data will not be used for the purposes of making policy or regulatory decisions.

Thank you in advance for your consideration.

INTRODUCTION

This proposal requests clearance for usability surveys on communications by the Food and Drug Administration (FDA). Usability surveys are surveys that ask potential or current consumers of FDA-regulated products to perform various tasks to determine how well FDA communications are performing. Observation and data collection on how users interact with the communications are critical in ensuring that users can find information, that they understand the messages, and that the messages meet the needs of specific audiences. This package requests clearance for two types of surveys: remote or in person. Remote surveys will collect data about how participants interact with FDA communications electronically. Users will take the survey at their home or work computers. In person surveys will have participants take the survey in a central location where their data can be captured electronically, as with the remote survey, but also the participants can be directly observed. The direct observation of in person surveys allows for enhanced collection of information such as observation of facial expressions and listening to verbal feedback. This package provides a list of generic tasks and questions for the surveys that can be used to develop a survey for a specific FDA communication. A list of screening questions (comprised of demographic and introductory questions) is also included in the package, and a subset of these screening questions will be used to create the proper sample for each usability survey. Participants in a usability survey are reflective of a product’s target audience.

A. JUSTIFICATION

A-1. Circumstances Making the Collection of Information Necessary

Executive Order 12862 (Attachment 1) directs Federal agencies that provide significant services directly to the public to survey customers to determine the kind and quality of services they need and their level of satisfaction with existing services. The FDA seeks to obtain approval to conduct usability surveys on communications of the FDA on an ongoing basis.

As part of the FDA Risk Communications Strategic Plan, this proposal was developed to help ensure that health information, interventions, and programs at FDA are based on sound science, objectivity, and continuous customer input. We need to be sure that FDA communications are designed to be easy to understand, easy to access, and effective providers of health information and resources to our target audiences.

FDA is requesting a 3-year generic clearance in order to carry out its mission. Generic clearance is needed to ensure that FDA can improve our communications continuously through regular surveys developed from these pre-defined questions.

Surveying FDA communications on a regular, ongoing basis will help ensure that the public has an effective, efficient, and satisfying experience viewing our communications, maximizing the health impact of the information and resulting in optimum benefit for public health. The surveys will ensure that this communication channel meets customer and partner priorities, builds FDA’s brand, and contributes to FDA public health outcome goals.

FDA employees, fellows, full-time contractors, or contract vendors will collect the data for these surveys. The data collected will include background participant information that does not identify individuals. The data will be collected electronically and stored digitally on a secure site or computer.

This survey is authorized under the Public Health Service Act (42 USC 241) Section 301. A copy of the legislation is included (Attachment 2).

A-2. Purpose and Use of the Information Collection

FDA has pledged in its Strategic Plan (currently in draft) to build a stronger, more effective agency. One of the means through which the Agency has vowed to do so is to promote public participation by increasing opportunities for input and feedback, so that the FDA can harness the best ideas from both inside and outside the agency.

By collecting communications usability information, FDA will be able to serve and respond to the ever-changing demands of consumers of FDA-regulated products. Additionally, we will be able to determine the best way to present messages. FDA communications are FDA’s primary channel to raise awareness of important issues for its target audiences.

These users include individuals (such as patients, educators, students, etc.), interested communities, partners, healthcare providers, industry, and businesses. Survey information will augment current content and delivery methods, so as to improve FDA’s ability to inform the public and improve the public health.

The purpose of usability surveys is to judge the content and presentation though which FDA communicates scientific health information to its target audiences to help ensure that health impact is maximized through the delivery of useful, efficient, and effective messages.

Primary objectives are to determine whether FDA communications:

(1) Meet the wants, preferences, and needs of its target audiences.

(2) Are effective vehicles for sending messages to target audiences.

(3) Provide users with the kind and quality of information they need.

(4) Deliver existing information at a satisfying level of quality.

Findings will help to:

(1) Understand how to serve the public better.

(2) Identify aspects of communications that require improvement in either content or delivery.

(3) Determine how to align communications with identified user need(s).

(4) Determine the kind and quality of information our target audiences need.

(5) Explore new or refined methods for offering, presenting and delivering information most effectively, to enable us to present messages as well as serve the needs of people who are seeking particular information or to learn about a particular topic.

The data collected from this effort will allow us to answer critical usability questions, including:

• What are the needs and preferences for our target audiences?

• How often and for what purposes (there can be several simultaneously) do our target audiences typically use the FDA website?

• How satisfied are they with their experience on the FDA website?

• What difficulties do they experience when trying to complete typical tasks on the FDA website?

• In what ways can we improve their speed and ability to find the information they want, expect or need on the FDA website?

• Were messages presented in such a way that they are noticeable, easy to understand, easy to remember, and have an impact on the viewer’s decisions?

• How does their awareness of, knowledge of, and opinions on a health topic change after viewing an FDA communication?

• Did they find information/messages about health issues they weren’t initially looking for when viewing information on the FDA website? Did the message have an effect, that is, did they change a health-related decision?

• Are they satisfied with the services offered through the FDA website?

• What improvements would the user like to see made to the existing services on the FDA website?

• What other services do they need for FDA to provide on its website?

• What changes do we need to make to increase the comprehensibility of our communications?

The survey will help ensure that FDA communications meet agency needs, build FDA’s brand, and contribute to FDA Strategic Plan public health outcome goals. Feedback from the public is necessary to fully judge the performance of FDA communications. All data collected through the survey will be used to determine whether FDA should revise content, labels, structure, or layout of its communications. If indicated, revisions would be intended to increase the success rate of FDA communications.

A-3. Use of Improved Information Technology and Burden Reduction

All data will be collected electronically to reduce the burden to the respondent.

For most questions in the survey, the respondent or interviewer will click on a “radio button” or checkbox that corresponds to the respondent’s response. For open-ended questions in usability surveys, the respondents either would be told to enter their answer in the provided text box or an interviewer will code the response. We have attempted to keep the format of the survey simple with short questions and clearly labeled and scaled answer choice-sets. There may also be up to 10 specific tasks where we ask the respondent to find an answer to a specific question (Attachment 8). For interviews conducted online, FDA will be able to determine how respondents found the answer by using click-stream technology which will lessen the time requirement to complete the survey, as they will not have to self-record their movements through the website. Additionally, users will be able to copy and paste from the Website to reduce the time required to answer open-ended questions.

Each survey will contain some questions from the following question sets: at least one option from Consent Forms (Attachment 4), up to 4 questions from Demographic Questions (Attachment 5), up to 6 questions from Introductory Questions (Attachment 6), up to 15 questions from Core Questions (Attachment 7), up to 10 questions from Activity/Task Questions (Attachment 8), and up to 15 questions from Follow up Questions (Attachment 9). There are no “standard questions” which will be asked in every survey because surveys will vary in scope. In the interest of the participant’s time and reducing burden to the participant, each survey will ask only those questions which are absolutely necessary to improve the specific program’s communication. Studies will allow for cross-topic or cross-survey comparison of results, which will provide insight into how to manage the portfolio of FDA communications.

The set of survey questions included in this package were gathered from (1) previous usability surveys conducted at the Centers for Disease Prevention and Control (CDC) or (2) recommended usability questions used by other usability professionals at FDA and considered best practices. In determining which questions to include in the package, usability professionals across FDA were consulted, and questions that had poor performance in the past or were not considered best practices were discarded. Because FDA is requesting a 3-year generic clearance for a wide variety of possible usability surveys on FDA communications, the list of questions is large enough that this package can cover all potential survey scenarios needed. However, as stated above, each survey is limited to a specific number of questions, and FDA staff cannot incorporate every question in the survey.

The remote online surveys will allow FDA to collect data from a variety of audiences across the nation; results are easily captured electronically by recording participants’ clicks on the website and their responses to the survey questions. In person surveys allow for more direct observation of the public reviewing FDA communications and are needed in some surveys to gain even greater insight into decisionmaking. Observation of the surveys is critical in getting changes implemented in communication design and resolving differences of opinion among staff, senior management, and other involved parties. Use of audio and video tapes is standard protocol by usability professionals, and in some cases may be recommended.

During initial usability surveys conducted by the CDC, the importance of in-person performance to add this enhanced feedback was noted. Remote online surveys are typically performed as follow up surveys of communications or on low-trafficked websites.

A-4. Efforts to Identify Duplication and Use of Similar Information

No similar information exists, although some usability surveys have been conducted on the CDC website. Such previously approved and conducted surveys were performed under OMB No. 0920-0572 for health message testing. Approval of this package will allow the FDA to gain the benefits of communication improvements that CDC pioneered.

A-5. Impact on Small Businesses or Other Small Entities

There is no burden on small businesses or small entities.

A-6. Consequences of Collecting the Information Less Frequently

There are a number of potential negative consequences if these data are not collected. In addition, if data collection is not conducted on an ongoing basis, FDA will not have valuable information needed to routinely revise messages and reorganize online health information in a way that is most easily understood and accessed by members of the public. Specifically, without this data there would be:

• No performance measures by which to determine effectiveness of the FDA website or other FDA communications as tools for the public and as message channels. This would result in lowered user satisfaction of the FDA website, fewer return visits, and decreased information dissemination.

• No user data to include in website design decision-making to ensure that user experience on our site is efficient, effective, and enjoyable. This results in an unfocused approach to Web design in which we are unable to determine whether our site is useful or not.

• For other communications, a rupture in interactions with the public reduces FDA’s ability to insure that communications are efficient, effective, enjoyable, and understandable.

• No two-way communication between FDA website visitors and website coordinators. Two-way communication and user feedback is essential to the proper production and dissemination of health information and it is widely used in the field of public health for non-Web products; we need to implement such a process for our Web-based products, as well.

• Vital feedback regarding customer and/or partner satisfaction with various aspects of the FDA’s services will be unavailable.

Usability surveys will only be conducted at intervals considered appropriate to measure the impact of FDA communications and to monitor the level of performance. In most cases each communications staff of the FDA will likely conduct usability surveys annually or biannually after the establishment of a baseline. Collection on a less frequent basis than annually or biannually likely will reduce the practical utility of the information and inhibit FDA’s ability to monitor changes.

We are only expecting one-time responses from respondents. Therefore, it is not possible to ask participants to fill out the survey less frequently. There are no legal obstacles to reduce the burden.

A-7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5

There are no special circumstances with this information collection package. This request fully complies with the guidelines of 5 CFR 1320.5.

A-8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency

In accordance with 5 CFR 1320.8(d), FDA published a 60 day notice for public comment in the FEDERAL REGISTER of 76 FR 34083 (Attachment 3). No comments were provided on this notice.

A-9. Explanation of Any Payments or Gift to Respondents

FDA will not directly provide remuneration to respondents. However, some respondents may receive remuneration through recruitment companies contracted to obtain participants. FDA may use these recruitment companies to find participants for larger surveys or when it is difficult to find specific types of audiences willing to participate, e.g. clinicians. It is typical for recruitment companies to provide remuneration to users as part of their practices. The amount of remuneration is based on pay scales these companies follow. FDA will pay a fixed price to a recruitment company for their services and not specifically for any set remuneration. The recruitment company would have full names and addresses of participants but this information would never be supplied to FDA or stored with any survey data or results.

A-10. Assurance of Confidentiality Provided to Respondents

The FDA Privacy Officers have reviewed this submission and determined that the Privacy Act does not apply to data collections conducted according to procedures described in this application. All questions for the surveys to be conducted under this OMB approval are included within this Information Collection Request.

Respondents will be advised of the nature of the activity, the length of time it will require, and that participation is purely voluntary. Respondents will be assured that they will not incur penalties if they wish to avoid answering specific questions or refuse to participate in the information collection as a whole. These procedures conform to ethical practices for collecting data from human participants. All information provided by respondents will be treated in a secure manner, unless otherwise compelled by law.

No names or other information that could identify the respondent will be recorded. A code number will be assigned to an individual’s responses. For remote online surveys and click-stream technology, it is not possible for FDA to link this information to the IP (Internet Protocol) addresses of participants, and the survey software will not do so. Thus, there is no way to identify respondents.

For remote online surveys, where FDA has e-mailed, phoned or written to request participation, the names collected for the purposes of contacting potential participants will not be recorded or kept with the survey responses. We will only know that we asked the participant to take the survey, not whether they did, and we will not be able to match results with participants.

For in-person surveys, some FDA staff may have the name of the participant should access to FDA facilities be required; however, this information will not be recorded with survey results. Only a code number will be utilized.

All participants will be informed at the beginning of the survey (prior to participation) that their responses will be treated in a secure manner, that all data will be safeguarded closely, and that no individual identifiers are planned to be used in survey reports.

All data will be stored in secured electronic files for at least three years and no longer than 10 years. After 10 years, any tapes and video tapes will be destroyed. Prior to 10 years these tapes will be stored in a locked file cabinet in the Office of Planning offices. Tapes will be marked only with code number.

This project is exempt from IRB requirements.

A-11. Justification for Sensitive Questions

Questions concerning Race and Ethnicity may be considered sensitive by a portion of respondents. Race and Ethnicity questions are included in the set of Demographic questions that may be asked of respondents. Where relevant to the evaluation of communications usability, Race and Ethnicity data will be collected consistent with HHS policy and standard OMB classifications.

A-12. Estimates of Annualized Burden Hours and Costs

There will be two lengths of surveys conducted, depending on whether the survey is in- person or remote and online. An in-person survey will last an average of 60 minutes and take place at an FDA computer or at a nongovernmental location; a remote survey will last approximately 30 minutes and take place at the participant’s computer. These estimates were determined through analysis of times from previous usability surveys using similar questions, survey of usability professionals to ascertain average times for users to perform tasks, and a pilot survey of 10 internal users comprised of CDC staff and CDC contractors Some remote surveys will take much less time. The majority of usability surveys conducted at CDC were done remotely, thus FDA estimates that in the future more surveys will be done remotely rather than in-person.

Estimate of survey respondents was based on an estimate of the ideal number of usability surveys that FDA would conduct over a 3-year period. Factored in were initial surveys and subsequent follow-up surveys utilizing a satisfactory level of participants. Because FDA has not conducted these types of surveys at the level needed previously, it is anticipated that most of FDA’s communications will require some sort of usability survey. Additionally, FDA anticipates conducting a number of important baseline surveys for its home webpage and other highly trafficked sub-sites in order to redesign these pages as part of FDA’s priority to more effectively utilize its website.

Estimates of Annual Burden

Survey Type	Number of Respondents	Frequency of Response per Respondent	Avg. Burden Per Response (hrs.)	Burden Hours
In-Person Surveys	7,500	1	1	7,500
Remote Online Surveys	67,000	1	30/60	33,500
Screener Only*	500		5/60	41
Total	75,000			41,041

*These participants take the Screener (which will be comprised of Demographic and/or Introductory Questions, Attachments 5 and 6) but are not selected for the full survey.

An average hourly salary of approximately $22.36 is assumed for all respondents, including clinicians and scientific users, based on the Department of Labor (DOL) National Compensation Survey.¹ Because of the scope of this generic clearance and the variety of the types of participants, the average salary was utilized rather than attempting to estimate salaries for groups of audiences. With a maximum annual respondent burden of 41,041 hours, the overall annual cost of respondents’ time for the proposed interviews is estimated to be a maximum $917,676.80 (41041 hrs x $22.36). There will be no direct costs to the respondents other than their time to participate in each survey.

Total Respondent Hours	Hourly pay rate	Total Respondent Burden
41,041	$22.36	$917,676.80

A-13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers

There are no additional costs to the respondents. There is no burden to record keepers.

A-14. Annualized Cost to the Federal Government

Usability surveys will be prepared by contractors or FDA staff (FTE). An FTE manager will review all surveys. Usability teams will vary across FDA web teams but typically an FTE and contractor will work together on survey preparations, coding the surveys electronically, conducting the surveys, and analyzing of data. Additionally, a senior level FTE will typically review and approve the activities. The amount of time staff and contractors spend on surveys will vary depending on the number of participants for each survey, the number of questions, and the site being surveyed. An average number of 125 surveys a year was assumed for estimation purposes. Overall time spent by FDA staff and contractors will be reduced as this package provides tasks and questions to be used in the survey; thus, reducing time staff normally would have spent developing these questions.

Staff or Contractor	Average Hours per Study	Average Hourly Rate	Average Cost
Contractor instrument preparation, conduction, analysis (GS-12/GS-13 FTE)	20/survey	$45.31	$906/yr
FTE survey preparation, conduction, analysis (GS-13)	20/survey	$49.22	$985/yr
FTE manager survey review (GS-14)	5/survey	$58.17	$291/survey
Average Costs per survey
Average 1 Year Cost (based on 125 surveys per year)			$272,693

A-15. Explanation for Program Changes or Adjustments

This is a new data collection for FDA.

A-16. Plans for Tabulation and Publication and Project Time Schedule

Activity	Time Schedule
1. Determine which communication will be surveyed. 2. Determine survey method and survey questions. 3. Determine target participation, quotas. 4. Determine method of recruitment.	Within 14 days after approval of this package.
5. Recruit participants for survey. (See B-1) 6. Invitation link posted on website and active respondent recruitment begins.	Within 28 days after approval of this package.
7. Completion of surveys.	Up to 60 days after OMB approval.
8. Analysis of surveys.	Up to 21 days after survey completion.
9. Adjustment of website based on results of the survey.	Up to 60 days after survey analysis.

A-17. Reason(s) Display of OMB Expiration Date is Inappropriate

Exemption is not being sought. The OMB expiration date will be displayed.

A-18. Exceptions to Certification for Paperwork Reduction Act Submission

There are no exceptions to certification.

1 Department of Labor, National Compensation Survey, December 2008-January 2010