Voluntary Product Satisfaction and Usability Assessment
National Center for Injury Prevention and Control (NCIPC)
Centers for Disease Control and Prevention (CDC)
November 2009
Project Manager:
Jennifer Middlebrooks, MSW, MPH
Health Education Specialist
Division of Violence Prevention
National Center for Injury Prevention and Control
Center for Disease Control and Prevention
4770 Buford Highway, NE
Mailstop F-64
Atlanta, GA 30341-3717
(770) 488-4223 Phone
(770) 488-4349 Fax
Voluntary Product Satisfaction and Usability Assessment
Table of Contents
Justification
Circumstances Making the Collection of Information Necessary
Background
Privacy Impact Assessment
Overview of Data Collection System
Items of Information to be Collected
Identification of Website(s) and Website Content Directed at Children Under 13 Years of Age
Purpose and Use of the Information Collection
Privacy Impact Assessment
Use of Improved Information Technology and Burden Reduction
Efforts to Identify Duplication and Use of Similar Information
Impact on Small Businesses or Other Small Entities
Consequences of Collecting the Information Less Frequently
Special Circumstances Relating to the Guidelines of 5-CFR 1320.5
Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
Explanation of Any Payments or Gifts to Respondents
Assurance of Confidentiality Provided to Respondents
Privacy Impact Assessment
Justification for Sensitive Questions
Estimates of Annualized Burden Hours and Costs
Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers
Annualized Cost to the Federal Government
Explanation for Program Changes or Adjustments
Plans for Tabulation and Publication and Project Time Schedule
Reason(s) Display of OMB Expiration Date is Inappropriate
Exceptions to Certification for Paperwork Reduction Act Submission
Collection of Information Employing Statistical Methods
Respondent Universe and Sampling Methods
Procedures for the Collection of Information
Methods to Maximize Response Rates and Deal with Non-Response
Tests of Procedures or Methods to be Undertaken
Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
List of Attachments
Executive Order 12862
Public Health Services Act (42 USC 24) section 301
Question Bank
Sample response card
Sample web-based assessment
Sample e-mail assessment (data collection portion same as attachment 5)
Federal Register Notice (July 2009)
Non-research determination
A. Justification
1. Circumstances Making the Collection of Information Necessary
Background
Information Collection Request (ICR) classification: New
The mission of the National Center for Injury Prevention and Control (NCIPC) at the Centers for Disease Control and Prevention (CDC) is to prevent injuries and violence, and to reduce their consequences. CDC/NCIPC accomplishes this by providing timely, accurate, and high quality scientific and programmatic information to its customers.
CDC/NCIPC is seeking approval via a 3-year generic clearance to evaluate the quality of its products through customer satisfaction assessments. This generic clearance would allow CDC/NCIPC to carry out its mission by better meeting customer needs.
CDC/NCIPC releases a number of new products each year to its customers, a diverse group that includes health care providers, researchers, public health practitioners, policy makers, and the general public. The term product is broadly defined to include publications, web pages, podcasts, e-cards, CD-ROMs, and videos. At present, there is no mechanism for evaluating whether these products are meeting customer needs.
The proposed evaluation activities will focus on obtaining customer feedback in a number of areas. These include, but are not limited to, the accessibility of the product to a wide audience; the usefulness of the product in public health practice; and the overall quality of the product (appropriate content, organization, and visual appeal).
Customer feedback obtained on a regular, on-going basis will help ensure that consumers find CDC products to be useful. Feedback will be used to continually assess and revise products so they better reflect the goals of CDC, the Department of Health and Human Services, and the United States Government. This type of evaluation will allow CDC to maximize the impact of its products which will ultimately benefit the public’s health.
Executive Order 12862 (Attachment 1) directs Federal agencies that provide services directly to the public to survey customers to determine the kind and quality of services they need and their level of satisfaction with existing services.
CDC/NCIPC employees, contractors, or fellows will collect the information during the course of these evaluation activities. The data collected will only include limited and non-sensitive background information on customers. It will not include personal identifying information such as name or address. When data are collected electronically through an on-line system, no information on the time and routes taken through the website will be collected. The information collected will be stored electronically on a secure site or CDC computer.
This assessment is authorized under the Public Health Service Act (42 USC 241) section 301. This is the general data collection authority for the Secretary of Health, Education, and Welfare (Attachment 2).
Privacy Impact Assessment
Overview of Data Collection System
The CDC/NCIPC website contains more than 4000 individual web pages that house over 200 products aimed at preventing injuries and violence. These products are available electronically or through an on-line ordering system. More than 1.5 million hits are registered on the CDC/NCIPC website each month. Approximately 1/3 of users are public health practitioners, 1/3 are researchers, and 1/3 are members of the general public.
At present, there is no process for evaluating the quality of CDC/NCIPC products and determining whether they meet customer needs. Collecting information on product satisfaction will enable CDC/NCIPC to better serve and respond to the changing needs of its customers. Evaluation findings will help ensure that customers find the products provided by CDC/NCIPC easy to access, clear, informative, and useful. Specifically, the evaluation will determine whether the products are presented in an appropriate format and whether they meet customer needs.
Primary objectives of this evaluation are to (1) identify customers and their needs related to injury and violence prevention; (2) determine gaps between existing products and customer needs; (3) assess the overall quality and usefulness of existing products; and (4) revise existing products based on customer feedback.
Three methods will be used to collect information on customer satisfaction:
(1) Response Cards
Hard copies of products ordered on-line via CDC-INFO and shipped from the CDC warehouse will include a one-page, paper-based evaluation (known as a response card). Evaluations will be comprised of open-and close-ended questions that will assess customer satisfaction (see Attachment 3 for proposed questions). The evaluation will take no longer than 10 minutes to complete. To reduce the burden on the customer, an addressed and stamped envelope will accompany the evaluation. The evaluation will not be coded and it will not solicit any identifying information from the customer including name or address. CDC-INFO is a public inquiry management system that is maintained by CDC’s National Center for Health Marketing. CDC/NCIPC staff members do not have access to the system or to any customer contact information. Evaluation information will be maintained for 3 years and then destroyed.
(2) Web-based Assessments
Customers that access electronic products via the Internet will be given the option of completing a web-based evaluation via Zoomerang. The link to the on-line evaluation will reside on the product web page. Evaluations will be comprised of open-and close-ended questions that will assess customer satisfaction (see Attachment 3 for proposed questions). The evaluation will take no longer than 10 minutes to complete. It will not be coded and will not solicit any identifying information from the customer including name or address. It is not possible for CDC/NCIPC staff members to link evaluation information to the Internet Protocol (IP) address of customers. Therefore, there is no way to identify customers who choose to participate. No information on time spent on the website or routes to/from the website will be collected. Evaluation information will be maintained for 3 years and then destroyed.
E-mail Assessments
Customers that receive e-mail announcements for new CDC/NCIPC products will be given the option of completing a web-based evaluation via Zoomerang. The link to the on-line evaluation will be included in the e-mail announcement. Evaluations will be comprised of open-and close-ended questions that will assess customer satisfaction (see Attachment 3 for proposed questions). The evaluation will take no longer than 10 minutes to complete. It will not be coded and will not solicit any identifying information from the customer including name or address. It is not possible for CDC/NCIPC staff members to link evaluation information to the Internet Protocol (IP) address of customers. Therefore, there is no way to identify customers who chose to participate. No information on time spent on the website or routes to/from the website will be collected. Evaluation information will be maintained for 3 years and then destroyed.
Items of Information to be Collected
The proposed evaluation activities do not include collecting Information in Identifiable Form (IFF). See Attachment 3 for a list of proposed questions.
Identification of Website(s) and Website Content Directed at Children Under 13 Years of Age
The information collection does not involve any websites or content that is directed at children under 13 years of age. The products and subsequent evaluation activities are intended for adults.
2. Purpose and Use of the Information Collection
CDC/NCIPC currently provides a variety of products on injury and violence prevention to customers throughout the United States. The products address a wide range of public health topics and contain scientific, programmatic, and general information. To better meet the needs of consumers, these products are provided in multiple formats (printed, electronic, and multi-media). Approximately 15-25 new products are released each year. In addition, older products are continuously revised and re-released.
At present, there is no mechanism for evaluating these products. We cannot answer basic questions about who is using our products, how they are using them, and whether they are making an impact. This lack of information is hampering our ability to advance the Agency’s goals around injury prevention and meet the needs of our customers.
Product evaluation will allow CDC/NCIPC to better respond to the ever-changing demands and needs of our customers. We will be able to identify gaps and develop new products accordingly. In addition, we will be able to assess the quality of existing materials and make the appropriate revisions before the product is re-released. Ultimately, we hope that the collection of evaluation information will allow CDC/NCIPC to develop products that are more useful, efficient, and cost-effective. Once OMB approval is received, data collection instruments using the proposed methodologies will be constructed using the library of questions (Attachment 3).
Primary
objectives of the evaluation include:
Identifying customers and their needs related
to injury and violence prevention information;
Determining gaps between existing products and
customer needs;
Assessing the overall quality and usefulness
of existing products; and
Revising existing products based on customer feedback.
The evaluation information collected will be used to improve CDC/NCIPC products on various topics related to injury and violence prevention. The information will not necessarily be generalizable to other areas of public health.
Privacy Impact Assessment Information
CDC/NCIPC releases a number of new products each year to its customers. The term product is broadly defined to include publications, web pages, podcasts, e-cards, CD-ROMs, and videos. At present, there is no mechanism for evaluating whether these products are meeting customer needs.
The proposed evaluation activities will focus on obtaining customer feedback in a number of areas. These include, but are not limited to, the accessibility of the product to a wide audience; the usefulness of the product in public health practice; and the overall quality of the product (appropriate content, organization, and visual appeal).
Customer feedback obtained on a regular, on-going basis will help ensure that consumers find CDC products to be useful. Feedback will be used to continually assess and revise products so they better reflect the goals of CDC, the Department of Health and Human Services, and the United States Government. This type of evaluation will allow CDC to maximize the impact of its products which will ultimately benefit the public’s health.
The proposed evaluation activities do not include collecting Information in Identifiable Form (IFF). See Attachment 3 for a list of proposed questions.
There is no sensitive information being collected during the course of these proposed evaluation activities. The evaluation will have little or no effect on the respondent’s privacy.
3. Use of Improved Information Technology and Burden Reduction
Each evaluation will contain 10-15 questions from the question bank (Attachment 3). The evaluation will be voluntary and require no more than 10 minutes of the customer’s time. There are no “standard questions” that will be asked in every evaluation because the information collected will vary in scope. This flexibility is needed because of the wide variety of CDC products (e.g., DVD’s, websites, printed publications, and podcasts). To reduce the burden on the customer, the assessment will only include those questions necessary to evaluate and improve the quality of the product.
The set of questions included in this package were gathered from previous CDC customer satisfaction surveys, educational product evaluations, and website usability assessments. In determining which questions to include in the package, professionals from CDC were consulted. Questions that performed poorly in the past or were not considered best practices were discarded. Because a 3-year generic clearance is being requested, the list of questions is large enough that this package can cover the wide variety of products that require evaluation. However, as stated above, each evaluation is limited to 10-15 questions.
Three methods will be used to collect information on customer satisfaction:
(1) Response Cards
Hard copies of products ordered on-line via CDC-INFO and shipped from the CDC warehouse will include a one-page, paper-based evaluation (known as a response card). To reduce the burden on the customer, an addressed and stamped envelope will accompany the evaluation. Response cards will comprise 9% of the customer satisfaction evaluations collected. See Attachment 4 for a sample response card.
(2) Web-based Assessments
Customers that access electronic products via the Internet will be given the option of completing a web-based evaluation via Zoomerang. The link (see http://www.zoomerang.com/Survey/?p=WEB229Q9NX3W3L) to the on-line evaluation will reside on the product web page. Web-based assessments will comprise 80% of the customer service evaluations collected. See Attachment 5 for a sample web assessment.
E-mail Assessments
Customers that receive e-mail announcements for new CDC/NCIPC products will be given the option of completing a web-based evaluation via Zoomerang. The link to the on-line evaluation will be included in the e-mail announcement. E-mail assessments will comprise 11% of the customer satisfaction evaluations collected. See Attachment 6 for a sample e-mail assessment text. For the data collection sample see attachment 5.
4. Efforts to Identify Duplication and Use of Similar Information
At present, there is no mechanism for evaluating whether CDC/NCIPC products are meeting customer needs. Approval of this package will allow CDC/NCIPC to assess and continually improve the quality of its products.
5. Impact on Small Businesses or Other Small Entities
There is no burden on small businesses or small entities. No small businesses will be involved in this data collection. This proposed evaluation will determine the satisfaction of individual customers.
6. Consequences of Collecting the Information Less Frequently
There are a number of potential negative consequences if CDC/NCIPC cannot evaluate its products. Specifically, there will be:
No information on who is requesting products.
Therefore, CDC/NCIPC will not be able to assess if the products are
getting to the intended target audience (e.g., health care
providers, public health officials, the general public, etc.)
No information on how the products are being used. As a result, CDC/NCIPC will be unable to determine if the product is increasing knowledge, changing behavior, or improving public health practice.
No feedback on customer satisfaction with
CDC/NCIPC products.
No information on what changes or revisions are necessary to improve the quality of CDC/NCIPC products.
The evaluation will be limited to customers who request and receive CDC/NCIPC products. Customer participation in the evaluation will be completely voluntary. It is expected that customers will only receive and respond to one evaluation for a particular product. Therefore, it is not possible to ask customers to fill out the evaluation less frequently. There are no legal obstacles to reduce the burden.
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
There are no special circumstances with this information collection package. This request fully complies with the guidelines of 5 CFR 1320.5.
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
Another 60-day Federal Register Notice was published in the Federal Register on July 28, 2009 in volume 74, number 143, page 37230. (Attachment 7). There were no public responses received.
Although no outside consultation was used, extensive review and input was received from a variety of Subject Matter Experts including Behavioral Scientists, Health Education Specialists, Epidemiologists, and Health Communication Specialists. CDC staff members involved in the development of the questions submitted in this package include:
Teri Barber, MA
Lead Health Education Specialist
(770) 488-4277
Marie Boyle, MA
Health Communication Specialist
(770) 488-2040
Pam Cox, PhD
Behavioral Scientist
(770) 488-1206
Lynn Jenkins, MD
Branch Chief, Etiology and Surveillance Branch
(770) 488-3931
Deb Karch, PhD
Epidemiologist
(770) 488-1307
Jennifer Middlebrooks, MSW, MPH
Health Education Specialist
(770) 488-4223
Rich Puddy, PhD
Behavioral Scientist
(770) 488-1369
Janet Saul, PhD
Branch Chief, Program Development and Evaluation Branch
(770) 488-4733
Terrie Sterling, PhD
Lead Behavioral Scientist
(770) 488-1557
Linda Anne Valle, PhD
Lead Behavioral Scientist
(770) 488-4297
9. Explanation of Any Payments or Gifts to Respondents
CDC/NCIPC will not provide any payments, gifts, or remuneration of any type to those participating in the product evaluation.
10. Assurance of Confidentiality Provided to Respondents
The CDC Privacy Officer reviewed this submission and determined that the Privacy Act does not apply to data collections conducted according to procedures described in this application. All evaluation questions to be used under this OMB approval are included within this Information Collection Request (see Attachment 3).
Customers will be advised of the nature of the activity, the length of time the evaluation will require, and that participation is completely voluntary. In addition, customers will be assured that they will not incur penalties if they wish not to participate in the evaluation as a whole or to any specific questions. These procedures conform to ethical practices for collecting data from human participants. All information provided by customers will be treated in a secure manner, unless otherwise compelled by law.
No identifying information, including names or addresses, will be recorded. Evaluation forms will not be coded. For information collected via on-line assessments, it is not possible for CDC/NCIPC to link this information to the Internet Protocol (IP) addresses of customers. Therefore, there is no way to identify customers who choose to participate.
Customers will be informed at the beginning of the evaluation (prior to participation) that their responses will be treated in a secure manner, that all information will be safeguarded closely, and that no individual identifiers will be used.
The information collected will be stored in secured electronic files.
This project is exempt from IRB requirements. See CDC/NCIPC determination form (Attachment 8) for official IRB exception.
Privacy Impact Assessment Information
A. This submission has been reviewed by ICRO, who determined that the Privacy ACT does not apply.
B. Paper-based evaluation forms will be stored in a secure location on CDC’s campus. The forms will be stored in a locked file cabinet in a secure file room. Only staff approved to work on the project will have access to the forms.
Electronic copies of evaluation forms will be stored on a secure CDC computer. Only staff approved to work on the project will have access to the forms.
All evaluation materials will be destroyed after 3 years.
C. Research is not being conducted as part of this project. Therefore, consent is not being obtained from customers. Participation in the evaluation is completely voluntary. Prior to completing the product evaluation, customers will be informed how the information provided will be used to improve the quality of CDC/NCIPC products.
D. Customers will be informed that their participation in the product evaluation is completely voluntary. They will still receive the products they request free-of-charge regardless of whether they complete the evaluation.
NO IIF is being collected.
11. Justification for Sensitive Questions
The product evaluation will not include sensitive questions. See Attachment 3 for a list of proposed questions.
12. Estimates of Annualized Burden Hours and Costs
Section A: Estimates of Annualized Burden Hours
Table 1 presents burden estimates for the collection of the customer satisfaction information. These estimates were determined through analysis of time from previous product and web surveys using similar questions and interviews with usability professionals to ascertain average times for users to perform tasks.
The
respondents will be limited to customers who request and receive CDC
products. Customer participation in the evaluation is completely
voluntary. Names of customers will not be collected. The only
personal information collected will relate to professional
discipline, job duties, and experience working with public health
topics. No sensitive data (e.g., age, race, or gender) will be
collected. The evaluation data will be collected using a combination
of methodologies including:
Response cards: Each product that is shipped by the CDC warehouse will include a one page, paper-based evaluation (known as a response card) along with a self-addressed and stamped envelope. Customers can then voluntarily choose whether to return the response card. Response time for completing the evaluation is estimated at 10 minutes or less. This time estimate is based on an internal pilot of nine respondents comprised of CDC staff and contractors. See Attachment 4 for sample response card.
Web-based assessments: Products are available on-line in an electronic format. Each product web page will include a link to a web-based evaluation. Customers can then voluntarily choose whether to complete the evaluation. Response time for completing the evaluation is estimated at 10 minutes or less. This time estimate is based on an internal pilot of nine respondents comprised of CDC staff and contractors. See Attachment 5 for sample web-based assessment.
E-mail assessments: Products are marketed to customers via an e-mail announcement that includes a link to the electronic version of the product plus a link to a web-based evaluation. Customers can then voluntarily choose whether to complete the evaluation. Response time for completing the evaluation is estimated at 10 minutes or less. This time estimate is based on an internal pilot of nine respondents comprised of CDC staff and contractors. See Attachment 6 for sample e-mail assessment.
There is no cost to the respondents other than their time. The total estimated annualized burden hours are 90,333.
The estimate of annual burden was based on approximately 20 new products being released by CDC/NCIPC each year. This number is consistent with the number of products released annually over the last 5 years. Approximately 2500 hard copies of each product are distributed to customers annually. Each product is disseminated electronically (via e-mail) to 3000 customers each year. Finally, product websites receive approximately 1800 hits a month or 21,600 hits a year.
Table 1: Estimate of Annual Burden
Types of Respondents |
Types of Form |
No. of Respondents |
No. of Responses per Respondent |
Average Burden per Response (in hours) |
Total Burden (in hours) |
||
Public |
Response cards |
50,000 |
1 |
10/60 |
8,333 |
||
E-mail Assessments |
60,000 |
1 |
10/60 |
10,000 |
|||
Web-Based Assessments |
432,000 |
1 |
10/60 |
72,000 |
|||
|
|
90,333 |
|||||
Section B: Estimated Annualized Burden Costs
An average hourly salary of approximately $18.09 is assumed for all customers participating in the evaluation, including clinicians and scientific users, based on the Department of Labor (DOL) National Compensation Survey. Because of the scope of this generic clearance and the variety of the types of participants, the average salary was utilized rather than attempting to estimate salaries for groups of audiences. With a maximum annual burden of 90,333 hours, the overall annual cost of customer time is estimated to be a maximum $1,634,123.97.
The information being collected will not impose a cost burden on the respondents beyond that associated with their time to provide the required data.
Table 2: Estimated Annualized Burden Costs
Type of Respondent |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
Public |
90,333 |
$18.09 |
$1,634,123.97 |
13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers
There are no additional costs to the customers participating in the evaluation. There is no burden to record keepers.
14. Annualized Cost to the Federal Government
The product evaluations will be prepared, administered, and analyzed by CDC staff (GS-13 level FTEs). The average hourly rate was obtained from the Office of Personnel Management’s website (http://www.opm.gov/oca/09tables/html/atl_h.asp). The hourly rate for a GS-13 in metro Atlanta is $40.11 per hour. The postage needed for the self-addressed and stamped envelopes was calculated using the current first class rate of $0.44. There is no cost for Zoomerang, the on-line assessment service, included. CDC/NCIPC has already purchased a license. The system can be used to assess customer satisfaction without further cost to the government.
Labor Costs |
Average Hours per Evaluation |
Average Hourly Rate |
Average Cost |
Evaluation preparation, conduction, and analysis (GS-13) |
20/evaluation |
$40.11 |
$802.20/year |
Average labor costs per evaluation |
|
|
$802.20/year |
Average 1 year labor costs (based on 20 products being evaluated) |
|
|
$16,044 |
Operational Costs |
Number of stamped envelopes needed |
Cost per Stamp |
Annual Operational Costs |
Postage |
50,000 |
$0.44 |
$22,000 |
Total Costs |
|
|
Total Annual Cost |
|
|
|
$38,044 |
15. Explanation for Program Changes or Adjustments
This is a new data collection for CDC/NCIPC and is essential to ensuring that customers are satisfied with the products they receive.
16. Plans for Tabulation and Publication and Project Time Schedule
Activity |
Time Schedule |
Determine which products will be evaluated. |
Within 30 days of OMB approval. |
Determine evaluation questions to be used. |
Within 60 days of OMB approval. |
Determine evaluation method. |
Within 60 days of OMB approval |
Administer evaluation. |
Evaluation activities will begin within 90 days of OMB approval. They will continue as new products are released throughout the project period. |
Analyze evaluation results. |
Analysis will begin within 120 days of OMB approval. It will continue as new products are released throughout the project period. |
Revision of products based on the results of the evaluation. |
Depends on the type of the product. Web sites can be revised quickly (within 150 days of OMB approval). Other products, like printed publications, will take longer (within 12-15 months of OMB approval). |
17. Reason(s) Display of OMB Expiration Date is Inappropriate
Exemption is not being sought. The OMB expiration date will be displayed.
18. Exceptions to Certification for Paperwork Reduction Act Submission
There are no exemptions to certification.
| File Type | application/msword |
| File Title | INTRODUCTION |
| Author | emi2-su |
| Last Modified By | tfs4 |
| File Modified | 2009-11-20 |
| File Created | 2009-11-20 |