Supporting Statement A, B, and Att_0920-06AK 11-12-2006
Supporting Statement A, B, and Att_0920-06AK 11-12-2006.doc
CDC Website Usability Evaluation
OMB: 0920-0735
CDC Website Usability Evaluation
National Center for Health Marketing (NCHM)
Centers for Disease Control and Prevention
Project Manager(s)
Carol Crawford, Acting Branch Chief, Internet Communications Branch
Catherine Jamal, Usability Specialist & Web Communications Strategist
National Center for Health Marketing (NCHM)
Centers for Disease Control and Prevention
MS D-42, 1600 Clifton Road.
Atlanta, Georgia 30333
404-639-7886
November 12, 2006
INTRODUCTION
This proposal requests clearance for usability surveys on the Centers for Disease Control and Prevention (CDC) website. Usability surveys are surveys that ask potential or current users of the CDC website to perform various tasks on the website to determine how well CDC’s website is performing. Observation and data collection on how users interact with the website is critical in ensuring that users can find information, that the site is easy to use, and that the site is designed to meet the needs of specific audiences. This package requests clearance for two types of surveys: remote or in person. Remote surveys will collect data about how participants interact with CDC’s websites electronically. Users will take the survey at their home or work computers. In person surveys will have participants take the survey in a central location where their data can be captured electronically, as with the remote survey, but also the participants can be directly observed. The direct observation of in person surveys allows for enhanced collection of information such as observation of facial expressions and listening to verbal feedback. This package provides a list of generic tasks and questions for the surveys that can be used to develop a survey for a specific CDC website. A list of screening questions (comprised of demographic and introductory questions) is also included in the package, and a subset of these screening questions will be used to create the proper sample for each usability survey. Participants in a usability survey are reflective of a website’s target audience.
A. JUSTIFICATION
A-1. Circumstances Making the Collection of Information Necessary
Executive Order 12862 (Attachment 1) directs Federal agencies that provide significant services directly to the public to survey customers to determine the kind and quality of services they need and their level of satisfaction with existing services. The Centers for Disease Control and Prevention (CDC) seeks to obtain approval to conduct usability surveys on pages of the CDC websites on an ongoing basis.
As part of the CDC Futures Initiative, the National Center for Health Marketing (NCHM) was established to help ensure that health information, interventions, and programs at CDC are based on sound science, objectivity, and continuous customer input. We need to be sure the CDC website, its web pages and web applications are designed to be easy to use, easy to access, and effective providers of health information and resources to our target audiences.
CDC is requesting a 3-year generic clearance in order to carry out its mission. Generic clearance is needed to ensure that CDC can continuously improve its websites though regular surveys developed from these pre-defined questions.
Surveying the CDC website on a regular, ongoing basis will help ensure that users have an effective, efficient, and satisfying experience on any of our websites, maximizing the health impact of the information and resulting in optimum benefit for public health. The surveys will ensure that this communication channel meets customer and partner priorities, builds CDC’s brand, and contributes to CDC health impact goals.
CDC employees, fellows, full-time contractors, or contract vendors will collect the data for these surveys. The data collected will include background participant information that does not identify individuals, time and routes taken through the website to complete certain online tasks, and overall satisfaction measures. The data will be collected electronically and stored digitally on a secure site or computer.
This survey is authorized under the Public Health Service Act (42 USC 241) Section 301. A copy of the legislation is included (Attachment 2).
A-2. Purpose and Use of the Information Collection
The entire CDC website gets over 150 million hits a month and contains over 200,000 pages of information, products, guidelines, and training focused on diseases, health conditions, and public health. The CDC website is comprised of multiple smaller sites, and most usability surveys will focus on evaluating just one portion of www.cdc.gov. The CDC website is accessible to everyone on the World Wide Web and has many different audiences, including public health professionals, physicians, media, policy makers, and the general public.
By collecting website usability information, CDC will be able to serve and respond to the ever-changing demands of its website users. Additionally, we will be able to determine the best way to present messages on the CDC website. The CDC site is one of CDC’s primary channels to raise awareness of important issues for its target audiences.
These users include individuals (such as patients, educators, students, etc.), interested communities, partners, healthcare providers, and businesses. Survey information will augment current Web content, delivery, and design surveys which are used to understand the Web user, and more specifically, the CDC user community.
The purpose of such usability surveys is to judge the Web content and presentation though which CDC communicates scientific health information to its target audiences to help ensure that health impact is maximized through the delivery of a useful, efficient, and effective website.
Primary objectives are to determine whether the CDC website:
(1) Meets the wants, preferences, and needs of its target audiences.
(2) Is an effective vehicle for sending messages to target audiences.
(3) Provides users with the kind and quality of services they need.
(4) Delivers existing services at a satisfying level of quality.
Findings will help to:
(1) Understand the user community and how to better serve CDC website users.
(2) Identify areas of the website requiring improvement in either content or delivery.
(3) Determine how to align Web offerings with identified user need(s).
(4) Determine the kind and quality of services our target audiences need.
(5) Explore new or refined methods for offering, presenting and delivering information most effectively, to enable us to present messages as well as serve the needs of people who are already coming to seek particular information or to learn about a particular topic.
The data collected from this effort will allow us to answer critical usability questions, including:
What are the Web needs and preferences for our target audiences?
How often and for what purposes (there can be several simultaneously) do our target audiences typically use the CDC website?
How satisfied are they with their experience on the CDC website?
What difficulties do they experience when trying to complete typical tasks on the CDC website?
In what ways can we improve their speed and ability to find the information they want, expect or need on the CDC website?
Were messages on the site presented in such a way that they are noticeable, easy to understand, easy to remember, and have an impact on the viewer’s behavior plans?
How does their awareness of, knowledge of, and opinions on a health topic change after viewing the CDC website?
Did they find information/messages about health issues they weren’t initially looking for when viewing information on the site? Did the message have an effect, e.g. change their behavior plans?
Are they satisfied with the services offered through the CDC website?
What improvements would the user like to see made to the existing services on the CDC website?
What other services do they need?
The survey will help ensure that the CDC website meets site visitor and agency needs, build CDC’s brand, and contributes to health impact goals. Feedback from the user base is necessary to fully judge the performance of CDC’s website. All data collected through the survey will be used to determine whether CDC should revise content, labels, structure, or layout of its Web pages. If indicated, revisions would be intended to increase the success rate of information–seeking website visitors.
A-3. Use of Improved Information Technology and Burden Reduction
All data will be collected electronically to reduce the burden to the respondent.
For most questions in the survey, the respondent will click on a “radio button” or checkbox that corresponds to their response. For open-ended questions in usability surveys, the respondent would be told to enter their answer in the provided text box. We have attempted to keep the format of the survey simple with short questions and clearly labeled and scaled answer choice-sets. There may also be up to 10 specific tasks where we ask the respondent to find an answer to a specific question (Attachment 8). We will be able to determine how they found the answer by using clickstream technology which will lessen their time requirement to complete the survey since they will not have to self-record their movements through the website. Additionally, users will be able to copy and paste from the Website to reduce the time required to answer open-ended questions.
Each survey will contain some questions from the following question sets: at least one option from Consent Forms (Attachment 4), up to 4 questions from Demographic Questions (Attachment 5), up to 6 questions from Introductory Questions (Attachment 6), up to 15 questions from Core Questions (Attachment 7), up to 10 questions from Activity/Task Questions (Attachment 8), and up to 15 questions from Follow up Questions (Attachment 9). There are no “standard questions” which will be asked in every survey because surveys will vary in scope. In the interest of the participant’s time and reducing burden to the participant, each survey will ask only those questions which are absolutely necessary to improve the specific program’s website. Studies will allow for cross-topic or cross-web page comparison of results, which will provide insight into how to manage the portfolio of pages and topics within the CDC website.
The set of survey questions included in this package were gathered from (1) previous usability surveys conducted at CDC or (2) recommended usability questions used by other usability professionals in other organizations and are considered best practices. In determining which questions to include in the package, usability professionals across CDC were consulted and questions that had poor performance in the past or were not considered best practices were discarded. Because we are requesting a 3-year generic clearance for a wide variety of possible usability surveys on CDC websites, the list of questions is large enough that this package can cover all potential survey scenarios needed. However, as stated above, each survey is limited to a specific number of questions and CDC staff cannot incorporate every question in the survey.
The remote surveys allow CDC to collect data from a variety of audiences across the nation; results are easily captured electronically by recording participants’ clicks on the website and their responses to the survey questions. In person surveys allow for more direct observation of users on CDC’s website and are needed in some surveys to gain even greater insight into users’ web behaviors. During an in person survey, the responses are tracked electronically as they are for the remote surveys; however, these in person surveys also allow CDC to observe where users place their mouse (before clicking), observe physical responses (shock, confused looks, etc.), and allow them to explain why they are clicking on various parts of the website as they are doing it. These responses are recorded using video and audio tapes and are analyzed along with the electronic responses. Observation of the surveys or viewing of video or audio tapes from the surveys is critical in getting web changes implemented by web developers and resolving differences of opinion among staff, senior management, and other involved parties. Use of audio and video tapes is standard protocol by usability professionals.
During initial surveys of CDC’s website(s), new innovations, new parts of CDC’s website, or highly trafficked or high priority areas of the website, it is important to perform in person surveys to add this enhanced feedback. Remote surveys are typically performed as follow up surveys or on low-trafficked sites.
A-4. Efforts to Identify Duplication and Use of Similar Information
No similar information exists, although some usability surveys have been conducted on a few, specific pages of the CDC website. Such previously approved and conducted surveys are under OMB No. 0920-0572 for health message testing. Approval of this package will greatly expand CDC’s ability to perform usability surveys on the CDC website.
A-5. Impact on Small Businesses or Other Small Entities
There is no burden on small businesses or small entities.
A-6. Consequences of Collecting the Information Less Frequently
There are a number of potential negative consequences if these data are not collected. In addition, if the collection is not conducted on an ongoing basis, we will not have valuable data needed to routinely revise messages and reorganize online health information in a way that is most easily understood and accessed by website visitors. Specifically, without this data there would be:
No performance measures by which to determine effectiveness of the CDC website as a tool for our visitors and a message channel. This results in lowered user satisfaction, fewer return visits, and decreased information dissemination.
No user data to include in website design decision-making to ensure that user experience on our site is efficient, effective, and enjoyable. This results in an unfocused approach to Web design in which we are unable to determine whether our site is useful or not.
No two-way communication between CDC website visitors and website coordinators. Two-way communication and user feedback is essential to the proper production and dissemination of health information and it is widely used in the field of public health for non-Web products; we need to implement such a process for our Web-based products, as well.
Vital feedback regarding customer and/or partner satisfaction with various aspects of the CDC’s services will be unavailable.
Usability surveys will only be conducted at intervals considered appropriate to measure the impact of CDC website changes and to monitor the level of performance. In most cases each section of the CDC website will likely conduct usability surveys annually or biannually after the establishment of a baseline. Collection on a less frequent basis than annually or biannually will likely reduce the practical utility of the information and inhibit CDC’s ability to monitor changes.
We are only expecting one-time responses from respondents. Therefore, it is not possible to ask participants to fill out the survey less frequently. There are no legal obstacles to reduce the burden.
A-7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
There
are no special circumstances with this information collection
package. This request fully complies with the guidelines of 5 CFR
1320.5.
A-8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
A. A 60-day Federal Register Notice was published in the Federal Register on Thursday, December
8, 2005, Vol. No. 70, No. 235, pp. 73007-73008 (Attachment 3). No public comments were received.
Although no outside consultation was used, extensive review and input was received from Usability Engineers and Web coordinators across the Agency, including:
Marianne Hartin, Information Specialist, National Center for Environmental Health/Agency for Toxic Substances and Disease Registry, 404.498.0255
Glen Doyle, Technical Information Specialist, National Institute for Occupational Safety and Health, 513.533.8386
Susan Leonard, Health Communication Specialist, Division of Nutrition and Physical Activity, 770.488.5233
Sharon McAleer, Webmaster, National Center for Human Immunodeficiency Virus, Sexually Transmitted Diseases, and Tuberculosis Prevention, 404.639.5089
Jeanie Barker, Usability Analyst, National Center for Chronic Disease Prevention and Health Promotion, 770.234.6556
Susan Schuffenhauer, Usability Analyst, National Center for Infectious Diseases, 404.639.2939
Natalie Ferguson, Health Communication Specialist, National Center for Infectious Diseases, 859.335.6780
Susan Robinson, Branch Chief, National Center for Environmental Health/Agency for Toxic Substances and Disease Registry, 404.498.0312
Sandra Bonzo, Technical Information Specialist, National Center for Injury Prevention and Control, 770.488.4228
Lisa Richman, Usability Analyst, National Center for Health Marketing, 404.498.6118
Cathy Hogan, Web Coordinator, National Immunization Program, 404.639.8914
Sara Thrift, Chief - Technical Information Services, National Center for Human Immunodeficiency Virus, Sexually Transmitted Diseases, and Tuberculosis Prevention, 404.639.6122
Nancy Bryant, Information Specialist, National Center for Human Immunodeficiency Virus, Sexually Transmitted Diseases, and Tuberculosis Prevention, 404.639.6008
Cari Wolfson, Usability Analyst, Current CDC.gov contract vendor, President of Focus on U!, 850.385.7300
Janice Nall, Health Communication Specialist, National Center for Health Marketing, 404.498.224
A-9. Explanation of Any Payments or Gift to Respondents
CDC will not directly provide remuneration to respondents. However, some respondents may receive remuneration through recruitment companies contracted to obtain participants. CDC may use these recruitment companies to find participants for larger surveys or when it is difficult to find specific types of audiences willing to participate, e.g. clinicians. It is typical for recruitment companies to provide remuneration to users as part of their practices. The amount of remuneration is based on pay scales these companies follow. CDC will pay a fixed price to a recruitment company for their services and not specifically for any set remuneration. The recruitment company would have full names and addresses of participants but this information would never be supplied to CDC or stored with any survey data or results.
A-10. Assurance of Confidentiality Provided to Respondents
The CDC Privacy Officer reviewed this submission and determined that the Privacy Act does not apply to data collections conducted according to procedures described in this application. All questions for the surveys to be conducted under this OMB approval are included within this Information Collection Request.
Respondents will be advised of the nature of the activity, the length of time it will require, and that participation is purely voluntary. Respondents will be assured that they will not incur penalties if they wish to not respond to the information collection as a whole or to any specific questions. These procedures conform to ethical practices for collecting data from human participants. All information provided by respondents will be treated in a secure manner, unless otherwise compelled by law.
No names or other information that could identify the respondent will be recorded. A code number will be assigned to an individual’s responses. For remote surveys and clickstream technology, it is not possible for CDC to link this information to the IP (Internet Protocol) addresses of participants, and the survey software will not do so. Thus, there is no way to identify respondents.
For remote surveys, where CDC has e-mailed, phoned or written to request participation, the names collected for the purposes of contacting potential participants will not be recorded or kept with the survey responses. We will only know that we asked the participant to take the survey, not whether they did, and we will not be able to match results with participants.
For in person surveys, some CDC staff will have the name of the participant in order to grant them access to CDC facilities; however, this information will not be recorded with survey results. Only a code number will be utilized.
All participants will be informed at the beginning of the activity (prior to participation) that their responses will be treated in a secure manner, that all data will be safeguarded closely, and that no individual identifiers are planned to be used in survey reports.
All data will be stored in secured electronic files for at least three years and no longer than 10 years. After 10 years, the tapes and video tapes will be destroyed. Prior to 10 years these tapes will be stored in a locked file cabinet in the National Center for Health Marketing (NCHM) offices. Tapes will be marked only with code number.
This project is exempt from IRB requirements.
A-11. Justification for Sensitive Questions
Questions concerning Race and Ethnicity may considered sensitive by a portion of respondents. Race and Ethnicity questions are included in the set of Demographic questions that may be asked of respondents. Where relevant to the evaluation of website usability, Race and Ethnicity data will be collected consistent with HHS policy and standard OMB classifications.
A-12. Estimates of Annualized Burden Hours and Costs
There will be two lengths of surveys conducted, depending on whether the survey is In Person or Remote. An In Person survey will last an average of 60 minutes and take place at a CDC computer; a remote survey will last approximately 30 minutes and take place at the participant’s computer. These estimates were determined through analysis of times from previous usability surveys using similar questions, survey of usability professionals to ascertain average times for users to perform tasks, and a pilot survey of 10 internal users comprised of CDC staff and CDC contractors Some remote surveys will take much less time. The majority of usability surveys conducted at CDC have been done remotely, thus we estimate that in the future more surveys will be done remotely rather than in person.
Estimate of survey respondents was based on an estimate of the ideal number of usability surveys that CDC would conduct over a 3 year period. Factored in was initial surveys and subsequent follow up surveys utilizing a satisfactory level of participants. Because CDC has not been able to conduct these types of surveys at the level needed previously, it is anticipated that most of CDC’s websites will require some sort of usability survey. Additionally, CDC anticipates conducting a number of important baseline surveys for its home page and other highly trafficked sub-sites in order to redesign these pages as part of CDC’s priority to more effectively utilize its website.
Estimates of Annual Burden
-
Survey Type
Number of Respondents
Frequency of Response per Respondent
Avg. Burden Per Response (hrs.)
Burden Hours
In Person Surveys
7,500
1
1
7,500
Remote Surveys
67,000
1
30/60
33,500
Screener Only*
500
5/60
41
Total
75,000
41,041
*These participants only took the Screener (which will be comprised of Demographic and/or Introductory Questions, Attachments 5 and 6) but were not selected for the full survey.
An average hourly salary of approximately $18.09 is assumed for all respondents, including clinicians and scientific users, based on the Department of Labor (DOL) National Compensation Survey. Because of the scope of this generic clearance and the variety of the types of participants, the average salary was utilized rather than attempting to estimate salaries for groups of audiences. With a maximum annual respondent burden of 41,041 hours, the overall annual cost of respondents’ time for the proposed interviews is estimated to be a maximum $742,431.69 (41041 hrs x $18.09). There will be no direct costs to the respondents other than their time to participate in each survey.
-
Total Respondent Hours
Hourly pay rate
Total Respondent Burden
41,041
$18.09
$742,431.69
A-13.
Estimates of Other Total Annual Cost Burden to Respondents or Record
Keepers
There
are no additional costs to the respondents. There is no burden to
record keepers.
A-14. Annualized Cost to the Federal Government
Usability surveys will be prepared by contractors or CDC staff (FTE). An FTE manager will review all surveys. Usability teams will vary across CDC web teams but typically an FTE and contractor will work together on survey preparations, coding the surveys electronically, conducting the surveys, and analyzing of data. Additionally, a senior level FTE will typically review and approve the activities. The amount of time staff and contractors spend on surveys will vary depending on the number of participants for each survey, the number of questions, and the site being surveyed. An average number of 125 surveys a year was assumed for estimation purposes. Overall time spent by CDC staff and contractors is lessened as this package provides tasks and questions to be used in the survey; thus, reducing time staff normally would have spent developing these questions.
-
Staff or Contractor
Average Hours per Study
Average Hourly Rate
Average Cost
Contractor instrument preparation, conduction, analysis (GS-12/GS-13 equivalent)
20/survey
$36.00
$720/yr
FTE survey preparation, conduction, analysis (GS-13)
20/survey
$39.00
$780/yr
FTE manager survey review (GS-14)
5/survey
$45.00
$225/survey
Average Costs per survey
$1725
Average 1 Year Cost (based on 125 surveys per year)
$215,625
A-15. Explanation for Program Changes or Adjustments
This
is a new data collection for CDC and essential to ensuring that
visitors to the CDC website are able to find information.
A-16. Plans for Tabulation and Publication and Project Time Schedule
-
Activity
Time Schedule
1. Determine which website will be surveyed.
2. Determine survey method and survey questions.
3. Determine target participation, quotas.
4. Determine method of recruitment.
Within 14 days after approval of this package.
5. Recruit participants for survey. (See B-1)
6. Invitation link posted on website and active respondent recruitment begins.
Within 28 days after approval of this package.
7. Completion of surveys.
Up to 60 days after OMB approval.
8. Analysis of surveys.
Up to 21 days after survey completion.
9. Adjustment of website based on results of the survey.
Up to 60 days after survey analysis.
A-17. Reason(s) Display of OMB Expiration Date is Inappropriate
Exemption is not being sought. The OMB expiration date will be displayed.
A-18. Exceptions to Certification for Paperwork Reduction Act Submission
There are no exceptions to certification.
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
B-1. Respondent Universe and Sampling Methods
The data collection will not use statistical methods to select respondents. The responses are intended to serve as feedback to help ensure the CDC website meets user needs. CDC would consider it unacceptable for Web information to be unclear or hard to find for more than 1 in 5 of its Web users. A convenience sample need not be representative to indicate a need for further message refinement.
Each survey will have a desired quota of respondents, based on the website being studied. All interested in participating will be asked a series of up to 4 Demographic Questions (Attachment 5) and up to 6 Introductory Questions (Attachment 6). Responses to the screening questions will determine when to stop the survey or to not administer the survey to that type of respondent. When the desired number of respondents is obtained, further surveys will not be administered and the survey will be removed from the site.
There are many possible respondent types, including the general public, public health professionals, and healthcare professionals. Participants will be invited to participate through one of three possible ways:
Invited through a message and link on a CDC Web page.
Recruited nationwide from a combination of pre-existing databases of volunteer.
Recruited from lists drawn from appropriate professional organizations and personal contacts made available through CDC contacts. The participants will be telephoned, sent an e-mail, or mailed a letter and invited to join.
We estimate that approximately 20% of the total number of participants recruited will participate in the survey. This figure is based on a review of literature regarding web-based surveys and the previous results of similar surveys distributed by customer survey professionals and web survey vendors.
The survey will be provided to all participants via one or more of the methods described in Section A-3 of this document, Use of Improved Information Technology and Burden Reduction. Respondents will fill out the survey by clicking on the appropriate response, entering text for each question, or attempting to find the answer to specific tasks. The answer will automatically be stored electronically, apart from any personal identifiers.
The data collected will be stratified in different ways during the analysis. Survey data will be examined by profession, geography, frequency of Web usage, information needed, usage of information, answer to task questions, and experience on the website while performing tasks. Demographic data will be used to ensure that participants are selected from a wide range of demographics or from a specific demographic, as needed to represent the target audience for a website. Profession or role (eg. public health professional, practicing physician, mother, care-giver, or student) is one of the primary ways survey data will be stratified since CDC’s website has multiple target audiences, each with different goals, wants, needs, and preferences. There will be additional ways of stratifying the data in an individual survey, based on the website’s specific target audiences and overall goals.
B-2.
Procedures for the Collection of Information
For both remote and in person surveys, responses will be collected automatically by the survey software. Handwritten notes, video, and/or audiotape will be used to supplement data collected during in person surveys. In person surveys will be conducted on government-owned computers or CDC contractor-owned computers and will be facilitated by CDC staff or contractors. Remote surveys will be conducted at the user’s computer and cannot be used to identify users as described in A-10.
B-3. Methods to Maximize Response Rates and Deal with Non-response
The usability surveys will be designed to minimize respondent burden. The survey will be logical, easy to read, and relatively quick to complete. The surveys are Web-based, which will allow for a visually appealing layout and remote surveying (in which participants take the survey from their own computer.)
The purpose of the web-based survey is to obtain feedback so each Web page or topic can be judged to determine how to improve information presentation and dissemination. Therefore, we do not always require a high response rate to obtain useful information. Sampling size will vary depending on the website in question and the scope of the usability survey. The lowest sampling size necessary to glean useful data will be used, and no survey will have more than 150 respondents.
B-4. Tests of Procedures or Methods to be Undertaken
We have already performed technical testing to ensure technical quality, ease of use, and proper collection of data for existing usability survey methods at CDC. Principles of user research were used in creating the survey questions. CDC web experts and programmers from all CIOs have reviewed the proposed survey questions. Ten comprised of CDC staff and CDC contractors took pilot surveys. Additionally, all survey questions in this package have been used in previous surveys at CDC or are standard usability questions utilized by web professionals
B-5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
We anticipate the data to be straightforward and not require complex analysis techniques. Each program will obtain input from in-house staff or vendors who are skilled in survey methodology and information systems technology. Data collection and analysis will be performed by CDC FTEs or contracted services. Such persons include: Catherine Jamal (Usability Engineer & Web Communications Strategist, NCHM), Carol Crawford (Internet Communications Branch Chief, NCHM), and Janice Nall (Acting Deputy Director, NCHM).
CDC Project Managers
Carol Crawford, Acting Branch Chief, Internet Communication Branch
and
Catherine Jamal, Usability Specialist & Web Communications Strategist
Centers for Disease Control and Prevention
National Center for Health Marketing (NCHM)
MS D-42, 1600 Clifton Road.
Atlanta, Georgia 30333
404-639-7886
List of Attachments
Executive Order 12862
Public Health Service Act (42 USC 241) Section 301
60-day Federal Register Notice
Consent Forms
Demographic Questions
Introductory questions
Core questions
Activity/Task questions
Follow up questions
Example Survey and Screen Shots
Attachment 1: Executive Order 12862
Attachment 2: Public Health Service Act (42 USC 241) Section 301
Attachment 3: 60-day Federal Register Notice
Attachment 4: Consent Forms
Consent Forms
*Note: Option 1 or Option 3 will be used. Option 2 will be used, if relevant.*
Form Approved
OMB No.: 0920-0000
Expiration Date:
Option 1: Online Welcome Message
Welcome! Thank you for agreeing to help The Centers for Disease Control and Prevention (CDC) evaluate their [insert health topic] website/page/application. Your feedback is extremely important. We anticipate that it will take approximately [insert estimate] minutes to complete this questionnaire.
The first few questions will be about your purpose for using the CDC [insert health topic] website/page/application. Then we will ask you to perform some tasks on the website/page/application. We are not testing your abilities in any way; we are only testing the website/page/application to see how well it works. Please use the website/page/application in whatever manner is comfortable and normal for you.
Your responses to all questions will be kept in a secure manner. No personal identifiers will be recorded. All information is used for evaluation purposes only and does not involve sales of any kind. CDC does not plan to share the data with anyone outside CDC.
Click “Continue” to begin. To proceed through the survey, select your answer for each question and click “Next.”
Public reporting burden of this collection of information is estimated to average 20 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/ATSDR Reports Clearance Officer; 1600 Clifton Road NE, MS D-74, Atlanta, Georgia 30333; ATTN: PRA (0920-XXXX)
Option 2: Audiotape and Videotape Consent Form
*Note: If used, this consent form will be present in conjunction with Option 1 or Option 3.*
Purpose: The purpose of this document is to obtain your consent to audiotape and videotape today’s usability testing session. We want record the session in order to analyze in depth at a later time the information collected today.
The tape will be used only internally within CDC and CDC does not plan to share the tape with anyone outside CDC. It will not be broadcast or used for any other purpose. The tape may be copied and used internally without further permission. No personal identifiers will be linked to the data and your signed consent form will be stored separately from the recording.
If you agree with this, please sign where indicated.
Print Name: _______________________________________
Signature: ________________________________________
Date: ____________________________________________
Option 3: Participation Consent Form
We are interested in your opinions!
We’ve revised The Centers for Disease Control and Prevention’s (CDC) [insert health topic] website/page/application and would like you to "try out" the new website. We want to know what works well for you and what doesn't, so that we can further improve the revised design.
During this session, we'll
1. Ask you about your background
2. Ask you to give us feedback on the new homepage
3. Ask you to perform a series of tasks to find information on the website
After you finish all of the tasks, we'll also ask you for your thoughts on the new design and suggestions for improvement. The whole exercise will take approximately one hour to complete.
During this session, we are keeping track of your interactions with the website, so that we can find ways to improve the website before it is launched.
The information that is captured will only be used to improve the website and for evaluation purposes. No personal identifiers will be linked to the data and your signed consent form will be stored separately from the recording.
To consent to the use and release of this information, please sign your name below.
Thank you for your time. Your opinion is very valuable to us!
Name _______________________
Date: _______________________
Attachment 5: Demographic Questions
Form Approved
OMB No.: 0920-0000
Exp Date:
Demographic Questions
*Note: Up to 4 questions will be used from this section. All questions will be either radio button, freeform, or rating scale.*
Are you male or female?
Male
Female
How old are you?
17 years old or younger
18-24 years old
25-34 years old
35-49 years old
50-64 years old
65 years old or older
How would you describe your ethnicity and race?
Hispanic or Latino
Not Hispanic or Latino
American Indian or Alaska Native
Asian
Black or African American
Native Hawaiian or Other Pacific Islander
White or Caucasian
Don’t Know/Prefer not to answer
What is your highest level of education?
GED
High school diploma
Associates degree
4-year degree
Bachelors
Masters
Graduate degree
Post-graduate degree
Doctoral degree
Other: _________
In what country were you born?
What is(are) your primary language(s)?
In what country, state, county, and city do you currently live?
Would you describe the area in which you live as rural or urban?
Rural
Urban
I don’t know
Attachment 6: Introductory questions
Form Approved
OMB No.: 0920-0000
Exp Date:
Introductory Questions
*Note: Up to 6 questions will be used from this section. All questions will be either radio button, freeform, or rating scale.*
What is your job title and role?
Mother
Father
Stay-at-home parent
Primary Caregiver
General public
Retired
Self Employed
Non-Health Care Related
Non-Health Related Business/Industry
Office and Administrative Support
Manager
Business Owner or Operator
International
Architecture or Engineering
Arts, Design, Entertainment, Sports, or Media
Attorney or other Legal Professional
Building and Grounds Cleaning and Maintenance
Computer and Mathematical
Construction and Extraction
Correctional Institution
Food Preparation and Serving Related
Installation, Maintenance, and Repair
Insurance Company
Life, Physical, and Social Science
Protective Service and Safety Professional (Firefighter, EMT)
Sales and Related
Transportation and Material Moving
Farming, Fishing, Forestry, Ranching, or Agriculture
Government
Law Enforcement/Criminal Justice
Policy maker or their staff
Military
Federal Agency (not military)
Centers for Disease Control and Prevention (CDC)
FTE
Full-time Contractor
Part-time Vendor
Fellow
Grantee
Partner
Program
Research Grantees
Local or State government agency besides health department
State/Local Health Department
Health
Nursing Home
Clinic
Healthcare provider (physician, nurse, physician’s assistant, nurse practitioner)
Healthcare Support
Hospital
Medical or Allied Health School or Library
Medical or Scientific researcher
Medicine
Staff at Hospital or Clinic
Nurse/Advanced Practice Nurse
Pharmacy, Physician Assistant
National Model [insert health topic] Center
Public Health
Epidemiologist
Health Communication
Health Education
Community/Patient Advocacy
Non-profit Organization
Community and Social Services
Community Based Organization (CBO)
Homeless Shelter
Education
Student
Student (Medicine and Allied Health Professions)
School District
Student (non-Medicine and non-Allied Health Professions)
Teacher/Educator
Librarian/Information Specialist
Other: __________________
How many children do you have and what are their ages?
Infants (less than 1 year)
Toddlers (1–4 years)
Youth (5–12 years)
Teens (13–18 years)
Adults (over 18 years)
How long have you worked in your current profession?
Less than one year
1 year
2 – 5 years
6 – 10 years
More than 10 years
What personal technology devices to you regularly use?
PDAs
iPod
Cell phone
Blackberry or Treo
Other: ______
On average, how much time do you spend using a computer each day?
More than 4 hours a day
1 – 4 hours a day
Less than 1 hour a day
Where do you use your computer most often?
At Home
At Work
At School
At Friend’s/Relative’s
At Library
Other: _______
Please tell us about the environment in which you typically use a computer/the Internet:
Noisy
Quiet
Indoor
Climate-controlled
Outdoor
[health topic specific]
[health topic specific]
[health topic specific]
Other:___________
How would you rate your computer experience?
Novice – I am new to computers and/or I only use one for a specific purpose
Low – I am somewhat new to computers and/or I am relatively comfortable with one or two software applications/programs (e.g. Microsoft Word, Excel, etc.).
Intermediate – I am comfortable with computers and/or I have learned and use between three and ten different software applications/programs.
Expert – I know my way around computers and/or I am comfortable troubleshooting most of the problems that arise with computers and/or I have used many different software applications and have some programming skills.
In general, what are the most frequent computer-related activities you do? (check all that apply)
Email
Internet/Intranet
Software programs/applications. Please describe:
Other: __________
What resolution is your monitor set to?
To determine your resolution on a PC: 1. Minimize or close all applications; 2. Right-click on the background of your desktop, select Properties from the menu, select the Settings tab; 3. look for the Screen Area box. Resolution will be shown as “# by # pixels”
To determine your resolution on a Mac: 1. Click the Apple menu, select Control Panels and choose Monitors & Sound; 2. Click the Monitor icon; 3. Resolution should be displayed as “# by # pixels”
What percentage of your Internet time do you use for the following activities?
Email: _________
Web Surfing: _________
Shopping: __________
Instant Messaging: _________
Other: ________
What type of Internet connectivity do you have?
Dial-up
Cable modem
T1
Wireless
DSL (Digital Subscriber Line)
ISDN (Integrated Services Digital Network)
I don’t know
Other:___________
How much time do you spend using the World Wide Web/Internet per week (including office and personal use)?
Less than 1 hour
1 – 5 hours
6 – 10 hours
11 – 15 hours
More than 15 hours
Which activities have you performed online?
Ordered a product/service from a business, government or educational entity by filling out a form on the Web
Made a purchase online for more than $100
Created a Web page
Customized a Web page for yourself (e.g. MyYahoo, CNN Custom News)
Changed your browser's "startup" or "home" page
Changed your "cookie" preferences
Participated in an online chat or discussion (not including email)
Listened to a radio broadcast online
Made a telephone call online
Used a nationwide online directory to find an address or telephone number
Taken a seminar or class about the Web or Internet
Bought a book to learn more about the Web or Internet
Used Online Help
Download audio / video files
Subscribe to email alerts when content changes
Participate in message boards / chats / blogs
Attend online training seminars
Personalize / customize web pages
Download podcasts
Subscribe to RSS feeds
Text message others
Other: ____________
How often do you usually connect to the Internet via a wireless connection?
Daily
Weekly
Monthly
Almost never
I don’t know
How would you rate your level of experience when it comes to finding information on the Internet?
Novice – I am new to finding information on the Internet and/or I only use the Internet to find one specific type of information
Low – I am somewhat new to finding information on the Internet and/or I am relatively comfortable with finding information on the Internet for one or two subjects
Intermediate – I am comfortable with finding information on the Internet and/or I have used the Internet to find between three and ten different types of Information
Expert – I know my way around finding information on the Internet and/or I am comfortable troubleshooting most of the problems that arise when finding information on the Internet and/or I regularly find information on the Internet on a variety of topics
Attachment 7: Core questions
Form Approved
OMB No.: 0920-0000
Expiration Date:
Core Questions
*Note: Up to 15 questions will be used from this section. All questions will be either radio button, freeform, or rating scale.*
What is your typical purpose when searching for [insert health topic] information on the Web? I am looking for…
Specific [insert health topic] information for someone else (loved one, family member, patient, client, etc.)
Vaccination information
Professional training/education materials
Patient education materials
Clinical information/resources
Hospitals, doctors offices, or clinics located near me
Symptoms
Treatment
Prevention
Data or statistics
Registries and Surveillance
Fact Sheets
Partners (e.g. Public Health collaborators, Research collaborators, etc)
Events
Training
Outbreak information
Publications
Health Disparities
Legislation
Diagnostic tools/aids
Information in other languages
Program information
Research activities (e.g. intramural, extramural)
News
Funding opportunities (e.g. grant, cooperative agreement)
Recommendations/Guidelines
Meeting/Conference information
Caregiver resources
Partner/Other Organization opportunities and resources
Best Practices/Success Stories
General Health Information including prevention and/or treatment
Other (please specify): ______________________________
Where do you typically acquire [insert health topic] information? Which do you use most? Least?
Internet/Web
Radio
T.V.
Journal articles
Textbooks
Library
Brochures, Pamphlets
Healthcare provider (Doctor, Nurse, etc.)
Friend
Family
Online library
Online journals
Listservs
Newsletters
Magazines
Newspapers
Don’t consult sources
Other: _______
Do you use search engines more often than the navigation links on a website?
How do you typically access [insert health topic] information on the Internet/Web?
Search engine (Examples: Google, Yahoo.)
Specific websites/pages/applications
Online Newsletters of Listservs
Other (please specify): ______________________________
Which website/page/applications do you find useful for acquiring health-related information?
HHS (Department of Health and Human Services)
CDC (Centers for Disease Control and Prevention)
NIH (National Institutes of Health)
WebMD
Yahoo! Health
[insert health topic] specific partner’s website
[insert health topic] specific partner’s website
[insert health topic] specific website
[insert health topic] specific website
Other: ________
How do you usually find and select websites - and specific pages on those websites - when researching [insert health topic] information? [Check all that apply.]
I choose specific websites that I am already familiar with
I choose websites that appear on search engines (e.g., Yahoo, Google)
I click banner ads or referring links from other websites
I use a health section on a portal website (e.g., MSN Health, Yahoo! Health)
I choose websites recommended by friends / colleagues / family members
I look at general health websites (e.g., WebMD, MayoClinic)
I look at specific drug or treatment websites (e.g., Advair.com, Flovent.com)
I look at disease-specific consumer advocacy websites (e.g., AAFA.org)
I look at disease-specific medical association websites (e.g., aaaai.org)
I look at government health websites (e.g., CDC, NIH)
I look at websites mentioned in TV, news, or radio reports
I look at websites mentioned in publications I read
Other: _________________
Have you ever visited the CDC website before?
Yes
No
I’m not sure
Who do you think is the main audience for the CDC website?
Is there any other way you have obtained information from CDC besides using their website?
Have you ever visited the CDC [insert health topic] website/page/application before?
Yes
No
I’m not sure
If yes, how often do you visit the CDC [insert health topic] website/page/application?
Yearly
Monthly
Weekly
Daily
What was the reason you visited the CDC [insert health topic] website/page/application today? How did you get here?
Specific [insert health topic] information for myself
Specific [insert health topic] information for someone else (loved one, family member, patient, client, etc.)
Vaccination information
Professional training/education materials
Patient education materials
Clinical information/resources
Hospitals, doctors offices, or clinics located near me
Symptoms
Treatment
Prevention
Data or statistics
Registries and Surveillance
Fact Sheets
Partners (e.g. Public Health collaboratos, Research collaborators, etc)
Events
Training
Outbreak information
Publications
Health Disparities
Legislation
Diagnostic tools/aids
Information in other languages
Program information
Research activities (e.g. intramural, extramural)
News
Funding opportunities (e.g. grant, cooperative agreement)
Recommendations/Guidelines
Meeting/Conference information
Caregiver resources
Partner/Other Organization opportunities and resources
Best Practices/Success Stories
Public Health program specific resources
General Health Information including prevention and/or treatment
Other (please specify): ______________________________
Were you able to find what you were looking for?
Yes
No
Partially
Still looking
Does the depth of information on this website provide:
Too much detail
Just the right amount of detail
Not enough detail
In your own words, what is the main purpose of the CDC [insert health topic] website?
Describe the CDC [insert health topic] website/page/application home page.
What [insert health topic] information do you think the CDC [insert health topic] website/page/application contains?
Specific [insert health topic] information for someone else (loved one, family member, patient, client, etc.)
Vaccination information
Professional training/education materials
Patient education materials
Clinical information/resources
Hospitals, doctors offices, or clinics located near me
Symptoms
Treatment
Prevention
Data or statistics
Registries and Surveillance
Fact Sheets
Partners (e.g. Public Health collaborators, Research collaborators, etc)
Events
Training
Outbreak information
Publications
Health Disparities
Legislation
Diagnostic tools/aids
Information in other languages
Program information
Research activities (intramural and extramural)
News
Funding opportunities
Meeting/Conference information
Caregiver resources
Partner/Other Organization opportunities and resources
Best Practices/Success Stories
Recommendations/Guidelines
Public Health program specific resources
General Health Information including prevention and/or treatment
A means to provide questions, feedback or comments
Press Releases
Forms
Other (please specify): ____________________
How do you generally use the information from the [insert health topic] website/page/application?
Read online
Print for reference
Print for hand-out
Email to others
Refer others to the website
Other ________________
I think the CDC [insert health topic] website/page/application is:
Credible
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Scientific
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Up-to-date
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Relevant to current events
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Research-based
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Easy to understand
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Useful for researching
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Useful for making health decisions
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Related to my profession
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Action-oriented
Strongly disagree
Disagree
Neither agree nor disagree
Agree
Strongly agree
Attachment 8: Activity/Task questions
Form Approved OMB No.: 0920-0000
Expiration Date:
Activity/Task Questions
*There will be no more than 10 activity questions relevant to the specific website/page/application. Respondents will be asked to click through the specific website/page/application to complete the activities. For each activity question, there will be a series of follow up questions to evaluate the participant’s ease or difficulty in finding the answer to the question. An example of a task question and follow up questions is shown below. All questions will be either radio button, freeform, or rating scale.*
Now we are going to ask you to perform some tasks on the website. For each task, the page presented will be reset to the CDC [insert health topic] website/page/application Home page. Please start your task from this page. Click “Next” to proceed to the first task.
[Example Activity 1 of 10] Use the CDC website to determine [e.g. the current phase of an influenza pandemic].
I was able to complete the task by using the website.
I was able to complete the task by recalling prior knowledge or using another website.
I was not able to complete the task.
I think I was able to complete the task but I’m not sure.
[Follow up question 1] Regardless of whether you think you completed the task correctly, what is the title of the last Web page you visited?
[insert health topic] specific Web page title
[insert health topic] specific Web page title
[insert health topic] specific Web page title
[insert health topic] specific Web page title
Other: _________
[Example Follow up question 2] What is [e.g. the current phase of an influenza pandemic]?
Phase 1
Phase 2
Phase 3
Other: __________
[Follow up question 3] Please rate the USEFULNESS of the information on this Web page scale of 1 to 10.
Strongly disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
[Follow up question 4] Please rate the EASE OF FINDING of the information on this Web page scale of 1 to 10.
Strongly disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
[Follow up question 5] What difficulties and frustrations, if any, did you encounter in completing this task?
Link labels were unclear
Information was unclear
I was lost on the website
The task took too long to complete
There were too many link choices
Links did not go where I expected them to go
The information was not where I thought it would be
I could not find the information
The search function was difficult to use
The information was available but not complete
Information on a topic was located in more than one place on the website
I had no difficulties or frustrations in completing this task
The link was broken
The Web page was outdated
Other (please specify): _______________________
[Follow up question 6] Which link labels were unclear? What alternate names would be better?
[Continue with up to 9 more activity questions and follow up questions.]
Attachment 9: Follow up questions
Form Approved OMB No.: 0920-0000 Expiration Date:
Follow up Questions
*Note: Up to 10 questions will be used from this section. All questions will be either radio button, freeform, or rating scale.*
What is the main purpose of the CDC [insert health topic] website?
Did you learn anything new from visiting the CDC [insert health topic] website/page/application?
After viewing this website/page/application, do you plan to change your practices regarding [insert health topic] screening? Why or why not?
Please describe any difficulties you encountered when trying to complete tasks on this website/page/application.
Overall what did you like MOST about the website?
Overall what did you like LEAST about the website?
How useful is the [insert health topic] website/page/application to you as
A repository of [insert health topic] information
A tool for doing your work for the [insert health topic] program
A tool for collaborating with other [insert health topic] grantees
A resource for your staff
A resource for your partners (e.g. Public Health collaborators, Research collaborators, etc)
A tool for school
A way to understand [insert health topic]
A resource for your family and friends
A resource for yourself
With which grantees do you regularly interact?
CDC grantees
NIH grantees
All Federal government grantees
[insert health topic specific]
[insert health topic specific]
Other: ______
Would you recommend the CDC [insert health topic website/page/application] to anyone? Who? Why?
Who do you think would find this website/page/application most useful?
Mother
Father
Stay-at-home parent
Primary Caregiver
General public
Retired
Self Employed
Non-Health Care Related
Non-Health Related Business/Industry
Office and Administrative Support
Manager
Business Owner or Operator
International
Architecture or Engineering
Arts, Design, Entertainment, Sports, or Media
Attorney or other Legal Professional
Building and Grounds Cleaning and Maintenance
Computer and Mathematical
Construction and Extraction
Correctional Institution
Food Preparation and Serving Related
Installation, Maintenance, and Repair
Insurance Company
Life, Physical, and Social Science
Protective Service and Safety Professional (Firefighter, EMT)
Sales and Related
Transportation and Material Moving
Farming, Fishing, Forestry, Ranching, or Agriculture
Government
Law Enforcement/Criminal Justice
Policy maker or their staff
Military
Federal Agency (not military)
Centers for Disease Control and Prevention (CDC)
FTE
Full-time Contractor
Part-time Vendor
Fellow
Grantee
Partner
Program
Research Grantees
Local or State government agency besides health department
State/Local Health Department
Health
Nursing Home
Clinic
Healthcare provider (physician, nurse, physician’s assistant, nurse practitioner)
Healthcare Support
Hospital
Medical or Allied Health School or Library
Medical or Scientific researcher
Medicine
Staff at Hospital or Clinic
Nurse/Advanced Practice Nurse
Pharmacy, Physician Assistant
National Model [insert health topic] Center
Public Health
Epidemiologist
Health Communication
Health Education
Community/Patient Advocacy
Non-profit Organization
Community and Social Services
Community Based Organization (CBO)
Homeless Shelter
Education
Student
Student (Medicine and Allied Health Professions)
School District
Student (non-Medicine and non-Allied Health Professions)
Teacher/Educator
Librarian/Information Specialist
Other: __________________
Is the information on this website/page/application up to date? Please explain your answer.
Yes
No
I don’t know
What information did you expect to find on the CDC [insert health topic] website/page/application but did not?
What information and/or functions would you like to see included or highlighted on this website/page/application?
Specific [insert health topic] information for myself
Specific [insert health topic] information for someone else (loved one, family member, patient, client, etc.)
Vaccination information
Professional training/education materials
Patient education materials
Clinical information/resources
Hospitals, doctors offices, or clinics located near me
Symptoms
Treatment
Prevention
Data or statistics
Registries and Surveillance
Fact Sheets
Partners (e.g. Public Health collaborators, Research collaborators, etc)
Events
Training
Outbreak information
Publications
Health Disparities
Legislation
Diagnostic tools/aids
Information in other languages
Program information
Research activities (intramural and extramural)
News
Funding opportunities
Meeting/Conference information
Caregiver resources
Partner/Other Organization opportunities and resources
Best Practices/Success Stories
Recommendations/Guidelines
Public Health program specific resources
General Health Information including prevention and/or treatment
Other (please specify): ____________
Did the link labels on this website seem intuitive and appropriate? Which ones did not? Please explain your answer.
Was the website organized in a way that made it easy for you to find information? Please explain your answer.
Please rate the following statements.
Overall, I found the website was easy to use
I found the link labels and terms used in the website clear.
The health information on this website/page/application was presented in a manner that I could understand.
It was easy to find the information I was looking for on this website.
This website was aesthetically pleasing.
The website contained useful information.
I would use this website as my primary source for [insert health topic] information.
The organization of information within the categories (e.g. Basic Information, Data & Statistics, etc.) were appropriate.
The browsing conditions during the test were optimal.
I trust the information I found on this website.
Which version of the [insert health topic] website/application did you prefer?
Version 1
Version 2
Version 3
Version 4
No Preference
Which version of the [insert health topic] website/application was easier to use?
Version 1
Version 2
Version 3
Version 4
No Preference
Which version of the [insert health topic] website/application was more visually appealing?
Version 1
Version 2
Version 3
Version 4
No Preference
Did using this website/application/pages require any new skills or challenges? If Yes, please explain.
Yes
No
Were you able to concentrate and avoid disruption while using this website/application/pages?
Yes
No
When completing the tasks did you feel in control of finding the information?
Yes
No
Describe your experience finding the information.
Would you describe finding the information as fast or slow?
Fast
Slow
Were there any factors that may have slowed down your progress?
Small font
Bad mouse
Slow browser
Confusing terminology
Other (please specify)
Did you find the information correct or incorrect?
Correct
Incorrect
Did the website/application provide adequate on-screen help?
Did you always know where you were within the website/application?
Yes
No
Did it take a long or short time to find the information?
Long
Short
How do you prefer to navigate this website?
Search
Menu structure
Text links
Combination
Other: ________
No preference
Did you encounter difficulties navigating the website, and if so, what was the primary issue?
I did not encounter any difficulties navigating the website
Could not determine the best link on most pages
Links did not take me where I expected
Difficulty finding related information
Technical difficulties (e.g., broken links, error messages, etc.)
Too many links or navigational choices
Links/icons/labels are difficult to understand
Navigated to the general area but could not find the specific content I needed
Other: ________
Did you encounter difficulties with the Search feature, and if so, what was the primary issue?
I did not encounter any difficulties with the search functionality
Too many results
I was not sure what words to use in my search
The descriptions/abstracts were not helpful
The results were not relevant to my search terms
The search returned no results
The results were not relevant to my needs
Search speed was too slow
Other: ________
What is the primary way you get information about [insert health topic] to use for your job? (Examples: textbook, curriculum, colleagues, searching online)
In general, what information or resources do you need to do an effective job? What are the primary tools and/or software you use to do your job?
Can you describe a time when you went to a website to acquire [insert health topic] resources? Specifically why did you go to the website? What were you hoping to find there? What did you like or not like about the experience?
What is the best way to inform other [user type] that this website/application exists?
In what aspect of your job do you find the [insert health topic] website/application assists you? Please explain.
I want to understand how information flows through your group in order for you to do your job. Who are the primary people you interact with and how/why?
Explain to me the steps involved in completing [a specific task]. What do you need before you begin? Where or from whom do you get that information?
Based on what you saw today, how would you describe the [insert health topic] website/page/application to someone else?
In what situations (if at all) would you find this website useful?
Attachment 10: Example Survey and Screen Shots
OMB No.: 0920-0000
Expiration Date:
Example Survey
Welcome Screen
Welcome! Thank you for agreeing to help The Centers for Disease Control and Prevention (CDC) evaluate their website. Your feedback is extremely important. We anticipate that it will take approximately 20 minutes to complete this questionnaire.
The first few questions will be about your purpose for using the CDC website. Then we will ask you to perform some tasks on the website. We are not testing your abilities in any way; we are only testing the CDC website to see how well it works. Please use the website/page/application in whatever manner is comfortable and normal for you.
Your responses to all questions will be kept in a secure manner. No personal identifiers will be recorded. All information is used for evaluation purposes only and does not involve sales of any kind. CDC does not plan to share the data with anyone outside CDC.
Click “Continue” to begin. To proceed through the survey, select your answer for each question and click “Next.”
Survey Questions
Demographic and Introductory Questions
We would like to start by asking you some introductory questions. After these questions you may be asked to do a few tasks on the CDC website.
We would like to get some demographic information about you. This information is for evaluation purposes only, and all responses will be kept in a secure manner. No personal identifiers will be stored with or linked to this information.
Are you male or female?
Male
Female
How old are you?
17 years old or younger
18-24 years old
25-34 years old
35-49 years old
50-64 years old
65 years old or older
What is your profession?
Mother
Father
Government
Policy maker or their staff
State/Local Health Department
Health
Nursing Home
Clinic
Healthcare provider (physician, nurse, physician’s assistant, nurse practitioner)
Hospital
Public Health
Epidemiologist
Health Communication
Health Education
Other
On average, how much time do you spend using a computer each day?
More than 4 hours a day
1 – 4 hours a day
Less than 1 hour a day
In general, what are the most frequent computer-related activities you do? (check all that apply)
Email
Internet/Intranet
Software programs/applications. Please describe:
Other: __________
Core Questions
What is your typical purpose when searching for health information on the Web? I am looking for…
Professional training/education materials
Patient education materials
Partners (e.g. Public Health collaborators, Research collaborators, etc)
Events
Training
Legislation
News
Meeting/Conference information
Partner/Other Organization opportunities and resources
Other (please specify): ______________________________
Which websites do you find useful for acquiring health-related information in the Internet?
HHS (Department of Health and Human Services)
CDC (Centers for Disease Control and Prevention)
NIH (National Institutes of Health)
WebMD
Yahoo! Health
[insert health topic] specific partner’s website
[insert health topic] specific partner’s website
[insert health topic] specific website
[insert health topic] specific website
Other: ________
Have you ever visited the CDC website before?
Yes
No
I’m not sure
Activity/Task Questions
Now we are going to ask you to perform three tasks on the website. For each task, the page presented will be reset to the CDC website Home page. Please start your task from this page. Click “Next” to proceed to the first task.
Activity 1: Use the CDC website to find information on Diabetes.
I was able to complete the task by using the website.
I was able to complete the task by recalling prior knowledge or using another website.
I was not able to complete the task.
I think I was able to complete the task but I’m not sure.
What are some things you can do to prevent diabetes?
Please rate the USEFULNESS of the information on this Web page scale of 1 to 10.
Not useful 1 2 3 4 5 6 7 8 9 10 Extremely Useful
Please rate the EASE OF FINDING of the information on this Web page scale of 1 to 10.
Not easy 1 2 3 4 5 6 7 8 9 10 Extremely Easy
What difficulties and frustrations, if any, did you encounter in completing this task?
Link labels were unclear
Information was unclear
I was lost on the website
The task took too long too complete
There were too many link choices
Links did not go where I expected them to go
The information was not where I thought it would be
I could not find the information
The search function was difficult to use
The information was available but not complete
Information on a topic was located in more than one place on the website
I had no difficulties or frustrations in completing this task
Other (please specify): _______________________
Activity 2: Use the CDC website to find out how many women in Maryland died from lung cancer in 2002.
I was able to complete the task by using the website.
I was able to complete the task by recalling prior knowledge or using another website.
I was not able to complete the task.
I think I was able to complete the task but I’m not sure.
How many women in Maryland died from lung cancer in 2002?
247
1090
1333
10,660
Please rate the USEFULNESS of the information on this Web page scale of 1 to 10.
Strongly disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
Please rate the EASE OF FINDING of the information on this Web page scale of 1 to 10.
Strongly disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
What difficulties and frustrations, if any, did you encounter in completing this task?
Link labels were unclear
Information was unclear
I was lost on the website
The task took too long too complete
There were too many link choices
Links did not go where I expected them to go
The information was not where I thought it would be
I could not find the information
The search function was difficult to use
The information was available but not complete
Information on a topic was located in more than one place on the website
I had no difficulties or frustrations in completing this task
Other (please specify): _______________________
Activity 3: Use the CDC website to determine whether you need a Yellow Fever vaccination if you are traveling to Chile.
I was able to complete the task by using the website.
I was able to complete the task by recalling prior knowledge or using another website.
I was not able to complete the task.
I think I was able to complete the task but I’m not sure.
Do you need a vaccination for Yellow Fever vaccination if you are traveling to Chile?
Yes
No
Please rate the USEFULNESS of the information on this Web page scale of 1 to 10.
Strongly disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
Please rate the EASE OF FINDING of the information on this Web page scale of 1 to 10.
Strongly disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
What difficulties and frustrations, if any, did you encounter in completing this task?
Link labels were unclear
Information was unclear
There were too many link choices
Links did not go where I expected them to go
The information was not where I thought it would be
Information on a topic was located in more than one place on the website
I had no difficulties or frustrations in completing this task
Other (please specify): _______________________
Follow up Questions
We'd like to conclude the survey by asking about the experience you just had on the website.
Did you learn anything new from visiting the CDC [insert health topic] website/page/application?
Overall what did you like MOST about the website?
Overall what did you like LEAST about the website?
What information and/or functions would you like to see included or highlighted on this website/page/application?
D
Please rate the following statements on a scale of 1 to 10.
Strongly disagree 1 2 3 4 5 6 7 8 9 10 Strongly Agree
The health information on this website/page/application was presented in a manner that I could understand. _____
It was easy to find the information I was looking for on this website. _____
This website was aesthetically pleasing. _____
The website contained useful information. _____
I would use this website as my primary source for [insert health topic] information. _____
I trust the information I found on this website. _____
Did you encounter difficulties with the Search feature, and if so, what was the primary issue?
I did not encounter any difficulties with the search functionality
Too many results
I was not sure what words to use in my search
The descriptions/abstracts were not helpful
The results were not relevant to my search terms
The search returned no results
The results were not relevant to my needs
Search speed was too slow
Other: ________
Please share any additional comments you have about this website/page/application.
Thank you Message
Thank you! You have completed the survey. We appreciate your participation in this evaluation. We value your input and look forward to using your feedback to help us in our ongoing effort to improve our website/page/application.
Example Screen Shot of a Welcome Screen
Example Screen Shot of a Survey Question
During the survey, participants read the task question on the left side of the screen, click through the live website on the right side of the screen, and then select their answer from the choices on the left side of the screen.
| File Type | application/msword |
| Author | PHPPO_User |
| Last Modified By | gzk8 |
| File Modified | 2006-11-21 |
| File Created | 2006-11-06 |
© 2024 OMB.report | Privacy Policy