Evaluation of the Centers for Disease Control and Prevention’s National
Contact Center (CDC-INFO)
Request for Revision of OMB No: 0920-0753
Division of Community Engagement
Office of the Associate Director for Communication
Centers for Disease Control and Prevention
CDC-INFO
Project Manger
Amy Burnett, Director
1600 Clifton Road
M/S E-06
Atlanta, GA 30333
Centers for Disease Control and Prevention
Ph: 404-498-2487
October 13, 2010
SUMMARY
The Centers for Disease Control and Prevention proposes to conduct continual national evaluation of the agency’s Consumer Response Services Center—CDC-INFO. CDC-INFO is a proactive, unified, and integrated approach to the delivery of public health information. CDC-INFO is designed to contribute to improving the health and safety of the public. Customers are defined as any individual or group seeking health or public health information from CDC. This includes the public, media, medical and healthcare professionals, public health professionals, partner groups, businesses, researchers, and others. Customer interactions occur through multiple channels, e.g., telephone calls, e-mails, and postal mail and may occur in Spanish or English.
Program/Study Objectives
(1) Proactively evaluate customer interactions and service effectiveness by employing assessment measures and data collection mechanisms to support performance management, gathering insights and understandings for improving service levels, and implementing effective measures to meet customer satisfaction goals;
(2) Develop an ongoing understanding of customer requirements and satisfaction trends to achieve “best practice” quality standards and to provide qualitative assessments, quantitative data, and cost factors to drive improvement and reinforce operational objectives; and
(3) Measure CDC-INFO contractor service performance to assist in determining whether performance incentives have been achieved.
(4) Contribute to overall learning in order to advance public sector consumer response service centers.
In February 2005, CDC began a four-year effort to combine its 40+ hotlines, interactive voice response services system, and clearinghouses into one integrated service--CDC-INFO. CDC-INFO has an exceptionally wide scope because of the range of health topics handled by CDC. In its first year the contact center alone handled 700,000 inquiries, including response to Hurricane Katrina, seasonal influenza, HIV Testing Day and Avian and Pandemic Flu.
Literature searches within CDC’s programs and with 30 other Federal agencies were conducted to identify studies with similar goals and expected outcomes. No similar study of consumer response services which integrate information and referral services for more than 40 health-related topics has been conducted. Even though the current evaluation is unique from others that have been conducted in the field, information from these literature searches has helped the evaluation design and enhanced the expected utility of study results. In addition, a great deal of published information is available about private sector call centers. An exhaustive review of this body of research informed the evaluation of the CDC-INFO currently approved.
For the past three years, the evaluation has provided volumes of data, reports, and presentations on the progression of CDC-INFO, an innovative, multi-million dollar, Federal public health contact center. The outcome of this feedback is tangible, with the average number of incoming calls to the contact center reaching new heights on an annual basis, and consumer satisfaction hovering around the best practice benchmark of 75 percent of callers participating in a satisfaction survey endorsing the highest level of satisfaction. In addition, self-reported behavior change is a key outcome of CDC-INFO’s evaluation.
CDC management benefits from a built-in feedback system for performance improvement and quality assurance monitoring. The consumer satisfaction surveys are administered in the same fashion as the original mode of inquiry, i.e. by phone and by e-mail. The surveys track the utility of this service to the public over-time, and complement measures of agent performance, by directing analysis of programs that are underperforming or receiving very high endorsements, relative to industry best practices.
CDC is seeking OMB approval of the revised version of the “Evaluation of the Centers for Disease Control and Prevention’s National Contact Center (CDC-INFO) Information Collection Request. The evaluation approach described in this study has been revised to accommodate budget cuts; many of the survey instruments in the previously approved clearance have been eliminated. For a summary of the changes, please see (Attachment A). The revised request seeks approval to collect data from CDC-INFO inquirers through two surveys: The Interactive Voice Response Survey (Attachment E – English) (Attachment F – Spanish), and the Web Survey for E-Mail Inquirers (Attachment G - English) (Attachment H – Spanish), which were previously approved and have undergone minimal changes in order to better align survey items requesting demographic data with the U.S. Census standards (see Attachment A: summary of Changes).
Customer Satisfaction
1. Interactive Voice Response Survey (Satisfaction Survey for Callers):
Each year, 100% of CDC-INFO callers are offered the opportunity to participate in a brief interactive, touchtone survey at the end of their call with the customer service representative. (See Attachment E for English version, Attachment F for Spanish version). Callers are routed electronically, without human intervention, to the brief automated survey. The average response rate has been 23.3% over the past three fiscal years. Based on an expected response rate of 23.3%, the sample size would number 92,000 annually.
2. Web Survey for E-Mail Inquirers (Satisfaction Survey for E-Mail Inquiries):
All e-mail inquirers (not including publication orders) are offered an opportunity to complete a web-based feedback survey by selecting the link shown at the bottom of the response e-mailed to them. (See Attachment I for English version, Attachment J for Spanish version.) CDC-INFO received an average of 73,000 e-mails over the past three fiscal years. Participation in the e-mail inquirer satisfaction survey is two percent, for a projected total sample size of 1,460.
The proposed evaluation of CDC-INFO is compliant with the procedures and principles outlined in the U.S.A. General Services Administration’s Citizen Service Levels Interagency Committee (CSLIC) report (Attachment D). The CSLIC report describes performance measures, practices and approaches for government-wide citizen contact activities.
This study is authorized under the Public Health Service Act (42 USC 241) Section 301.
A. JUSTIFICATION
1. Circumstances Making the Collection of Information Necessary
Background
The Centers for Disease Control and Prevention’s (CDC) mission is to promote health, safety, and quality of life by preventing and controlling disease, injury, and disability. As the Nation’s lead prevention agency, CDC works with States, local public health agencies, and partners throughout the Nation and the world to accomplish this mission.
Increasingly, the dissemination of health and safety information plays a critical role in the Nation’s efforts to prevent and control diseases, injury and disabilities. In recent years several events (e.g. war, natural disasters) have heightened public awareness and their need to know, the focus remains on CDC to educate the public, medical personnel, clinicians, public health workers, and the media on disease specific details, risk, and prevention information. Direct customer interaction is an important and visible part of the agency’s effort to carry out its mission. Therefore, CDC is faced with the major challenges associated with information delivery that has become more voluminous and complex.
(CDC-INFO is an integrated approach to the delivery of public health information designed to contribute to improving the health and safety of the public. Customers are defined as any individual or group seeking health or public health information from the Centers for Disease Control and Prevention (CDC). This includes the public, media, medical and healthcare professionals, public health professionals, partner groups, businesses, researchers, and others. Customer interactions occur through multiple channels, e.g., telephone calls, e-mails, and postal mail and may occur in Spanish or English.
The Centers for Disease Control and Prevention (CDC) seeks to obtain approval for a revision of the currently approved Information collection: Evaluation of the Centers for Disease Control and Prevention Consumer Response Service Center (CDC-INFO) 0920-0753, to conduct customer satisfaction surveys with the general public and health care professionals who call or email the CDC National contact center (CDC-INFO).
In September 2005, the Centers for Disease Control and Prevention launched CDC-INFO, a consolidated, comprehensive effort to respond to consumer, provider and partner inquiries on a broad spectrum of public health topics by telephone, e-mail, fax, or postal mail. More than 40 nationwide public health hotlines and warm lines were consolidated into one central phone number using a phased approach from 2005 to 2008. Management of CDC-INFO services is increasingly guided by a comprehensive evaluation that includes point-of-service and follow-up customer satisfaction surveys. These surveys provide the public with ongoing opportunity to express their level of satisfaction and report how they have used this information. All members of the public, health care providers and businesses can contact CDC-INFO by phone or e-mail to request health information or order CDC publications.
Since 2006, CDC-INFO has received more than 2,013,630 phone and email inquiries, in Spanish and English. In Fiscal Year 2010 alone, more than 600,000 inquiries for CDC health and safety information was received. Each of these interactions is an opportunity to collect customer satisfaction feedback and data that are critical for assessing the study’s performance and quality of interactions with the public. The information generated through this evaluation is used primarily by CDC in its efforts to assess consumer satisfaction and service effectiveness and to continually improve services. The data collected through this study are used by the CDC to assess the contact center’s contractual performance and identify areas of improvement and focus for customer service efforts on a biannual basis.
Information is shared throughout CDC on a biannual, quarterly, or monthly basis so that programs are aware of the contact center’s performance on inquiries related to a particular health topic or subject matter managed by CDC-INFO. The programs use these data to improve their efforts to reach select populations. Weekly customer satisfaction reports facilitate internal agent training, coaching, and quality assurance exercises within the contact center. The current study has been pared down to only include The Interactive Voice Response Survey (Attachment E - English) (Attachment F – Spanish) and the Web Survey for E-Mail Inquirers (Attachment G -English) (Attachment H – Spanish).
Executive Order 12862, authorized in 1993, directs Federal agencies that provide significant services directly to the public to survey customers to determine the kind and quality of services they need and their level of satisfaction with existing services. More recently, the Obama Administration has called for a transformation of customer service as part of the White House’s focus on modernizing government, saying, “The Federal Executive Order 12862 directs Federal agencies that provide significant services directly to the public to survey customers to determine the kind and quality of services they need and their level of satisfaction with existing services. Government must transform its customer services in ways similar to what has been done in the private sector, both in terms of improving customer satisfaction measuring and monitoring, and in terms of better delivery of citizen-facing services” (whitehouse.gov)1.
The customer satisfaction survey approach is based on the procedures and principles outlined in the U.S.A. General Services Administration’s Citizen Service Levels Interagency Committee (CSLIC) report released in 2005 (Attachment D) .
This study is authorized under the Public Health Service Act (42 USC 241) Section 301 (Attachment B).
Privacy Impact Assessment
Overview of the Data Collection System
CDC contract vendors will collect the data for the Interactive Voice Response Survey and the Web Survey for E-Mail Inquirers. Data are collected electronically, whenever possible, to reduce the burden to the respondent.
Interactive Voice Response Survey
At the end of each CDC-INFO call, the caller is presented with the option of completing the Interactive Voice Response Survey. If the caller chooses to participate, the caller is connected to the Interactive Voice Response Survey, an 11-item, automated survey that allows the caller to respond to the questions by entering selections on their touchtone phone.
The survey data are transmitted daily to the contact center vendor’s File Transfer Protocol (FTP) server. The next day, the contact center vendor transfers the data file to a secure FTP that can be accessed by the third-party evaluation contractor. The evaluation contractor logs into the secure site and downloads the customer satisfaction data to conduct analyses.
Call contact data that supplement the customer satisfaction analyses are captured by the contact center contractor’s customer relationship management (CRM) software are transmitted by the contact center contractor to the evaluation contractor on a weekly basis. These data are linked to the customer satisfaction data for analysis.
Web Survey for E-Mail Inquirers
When the CDC-INFO contact center vendor responds to an inquirer’s email, the email contains a link to a web survey hosted by the evaluation contractor. The email states “Please visit CDC-INFO’s Customer Satisfaction Survey and tell us about your recent experience with CDC-INFO. The survey results will help improve our services. The address for the survey is http://www.emtservices.org/cdcinfo/en/email_satisfaction_survey.htm. Thank you.” The inquirer can then choose to click on the link to complete the web survey. The customer satisfaction survey data are captured on the evaluation contractor’s server and are then routed to an evaluator’s work station for analysis in the contractor’s Quality Assurance Tracking System (QATS).
Email contact data that supplement the customer satisfaction analyses captured by the contact center contractor’s customer relationship management (CRM) software are transmitted by the contact center contractor to the evaluation contractor on a weekly basis. These data are linked to the customer satisfaction data for analysis.
Once the call and email customer satisfaction analyses are complete the contractor prepares reports that are sent to the contact center contractor and the CDC-INFO program office.
Items of Information to be Collected
Interactive Voice Response Survey
The Interactive Voice Response Satisfaction Survey for Callers is an 11 item questionnaire to assess customer satisfaction and to collect basic demographic information. The following data are collected from this survey. (See Attachment E for the English version, Attachment F for the Spanish version)
If this was the respondent’s first call to CDC-INFO
How well the information provided by CDC-INFO answered the question
If the information provided by CDC-INFO answered the question
How well the customer service representative understood the respondent’s information need
How quickly the customer service representative answered the respondent’s question
How satisfied the respondent was with the services provided by CDC-INFO
Gender
Age
The race/ethnic group with which the respondent most identifies
If the respondent identifies with any other race/ethnicity
The other race/ethnic group with which the respondent identifies
Web Survey for E-Mail Inquirers
The e-mail inquirer satisfaction survey is similar to the Interactive Voice Response Survey, in that it is an 11-item questionnaire that measures elements of satisfaction and collects demographic information. A text box at the bottom of the web form allows respondents to leave comments about their experience. (See Attachment G for the English version, Attachment H for the Spanish version)
The following data are collected from this survey:
If this was the respondent’s first email to CDC-INFO
How well the information provided by CDC-INFO answered the question
If the information provided by CDC-INFO answered the question
How well the customer service representative understood the respondent’s information need
How quickly the customer service representative answered the respondent’s question
How satisfied the respondent was with the services provided by CDC-INFO
If the respondent indicated “less than very satisfied,” provide the reason for the dissatisfaction
Gender
Age
The race/ethnic group with which the respondent most identifies
If the respondent identifies with any other race/ethnicity
The other race/ethnic group with which the respondent identifies
Identification of Website(s) and Website Content Directed at Children under 13 Years of Age
The Web Survey for E-Mail uses a web form for data collection. Any respondents who click on the link to the web survey will have access to the website. Cookies are not being used on this website. No websites have content directed at children less than 13 years of age.
2. Purpose and Use of Information Collection
The currently approved study and attendant data collection activities will help support CDC in achieving its information dissemination objective “to ensure consistent, timely, reliable health information is disseminated to meet the needs of a evolving diversity of consumers (public, health professionals, researchers, etc.) and to address variations in inquiry volumes related to public health emergencies, news events, and dynamic, shifting public health priorities” (CDC-INFO Request for Funding Announcement). Providing a full complement of information support for CDC-INFO, the new “one face to the public” will require a comprehensive, multi-phased system of quality assurance, performance monitoring and impact evaluation. This evaluation will provide CDC with information about the effectiveness of CDC-INFO.
The findings and conclusions from the study are a resource for CDC policy priorities, performance measurement, and in designing and promoting an optimally effective consumer response services center.
Information generated through this evaluation has been and will continue to be used primarily by CDC in its efforts to assess consumer satisfaction and service effectiveness and to continually improve services. CDC Programs will use evaluation information to refine efforts to reach elect populations. In addition, CDC will continue to examine health messages for effectiveness and customer satisfaction. Information gained from this evaluation and lessons learned will continue to be shared with other Federal organizations operating consumer response services centers.
In addition, information was made available to State and local program planners and the public through publications. States and localities launching 211 and 311 efforts find the evaluation findings informative as they design state-of-the-art consumer response services centers. 211 and 311 represent alternatives to 911, with 311 for county/borough/parish non-emergency services, and 211 for social, educational and public health services. Each state is presently developing plans for the implementation of 211 and 311. Information generated by the CDC-INFO evaluation will certainly be helpful in furthering these initiatives. While a great deal of information is available about private sector call centers, little evaluation information is available about public sector consumer response services centers. The evaluation of CDC-INFO provides this much-needed information. Policy makers in State and local governments have evidence on “best practice” with regard to information dissemination by consumer response services centers and strategies for maintaining high customer satisfaction with disparate populations. The evidence will be useful in setting policy priorities (e.g., topics and services to be covered through statewide and local 211 services).
Data collected since OMB approval was obtained have been used for assessment of contact center performance and customer satisfaction. Since 2007, data have been collected from more than 379,109 survey participants: 363,879 from the CDC-INFO Interactive Voice Response Survey (English and Spanish), 2351from the CDC-INFO Web Survey for E-Mail Inquirers (English and Spanish), 12,813 from CDC-INFO Live Phone Follow-up Survey, and 64 from a pilot of the Emergency Response Surveys (General Public and Professionals) launched during the H1N1 response in 2009.
Since data collection began in 2007 CDC-INFO has identified trends and areas of focus for performance management:
Overall customer satisfaction scores have varied from 78.5% in FY08 to 72% in FY10. The decline in scores presents further impetus to OMB for the need to continue collecting satisfaction, as FY 10 puts us below the best practice benchmark. (Requirement: 75% of respondents indicate the highest levels of customer satisfaction)
Significant differences in customer satisfaction between email respondents and phone respondents were found. For example, in FY09, 48.2% of email respondents reported the highest levels of customer satisfaction compared to 70.5% of phone respondents (EMT FY2009 Annual Report).
Significant differences in customer satisfaction were found based upon the CDC health and safety topic. For example, in FY09, inquirers were less satisfied with topics related to Influenza and Chemical Safety, and more satisfied with topics related to Sexually Transmitted Diseases and Chronic Diseases (EMT FY2009 Annual Report).
For the first time, CDC-INFO was able to identify predictors of customer satisfaction were identified through Structural Equation Model (SEM) analysis that examined customer service data compared with agent quality assurance assessments.
CDC-INFO is one of the few contact centers that has collected health impact data. Since FY 08, 56.3% of the respondents to the CDC-INFO Live Phone Follow-up Survey reported that they learned something new due to their interaction. Since 2007, 3814 survey participates reported that they changed their health behavior based on the information they received from the contact center. The most common types of behavior change reported were reducing risky sexual behaviors, exercising better hygiene, and getting immunized. (EMT FY 2010 Annual Report, FY2009 Annual Report, and FY 2008 Annual Report).
During the 2009-2010 H1N1 event, a pilot test of the Emergency Response Surveys (General Public and Professionals) was launched. Preliminary results showed:
Most respondents had taken precautions to prevent acquiring H1N1 infection (82.7%; n=43), such as being vaccinated or engaging in everyday precautions (e.g., frequent hand washing, hand sanitizer, staying away from sick people).
Approximately eight in ten respondents expressed that they would take stronger action to protect themselves if convinced that H1N1 is more dangerous (78.8%; n=41).
The TV/radio (51.9%; n=27) was the most popular resource for information about H1N1 for respondents from the general public, however the CDC Website was equivalently as popular as the TV or radio as a point-source when the person had questions about H1N1 (28.8%; n=15 would go on the CDC Website, whereas 21.2%; n=11 would refer to the television.)
The CDC Website was the most popular source for professional information about H1N1 among health professionals (70.0%; n=7).
One-third of respondents from the general public chose to call CDC-INFO because they trust the CDC’s information and reputation (32.7%; n=17) or that they thought of CDC first (11.5%; n=6)
One in five respondents reported reading something about H1N1 on social media websites like Facebook, MySpace, or Twitter (11.5%; n=6). In three cases, the respondent stated that they had seen a post about someone being infected with H1N1. One person described an ad for the H1N1 vaccine on Facebook. (EMT H1N1 Survey Summary, 2010)
Privacy Impact Assessment Information
The primary objectives of the customer satisfaction surveys are to evaluate:
How well the customer service representative understood the caller’s/emailer’s needs
How quickly the customer service representative responded to callers’/emailer’s questions
How well the information provided answered the caller’s/emailer’s questions
The relationship between survey respondents’ demographics and satisfaction to gain an understanding of how satisfaction levels differed between groups (age, race/ethnicity, etc.)
The survey will help ensure that the CDC-INFO meets customer needs and contributes to CDC’s public health goals and objectives. Feedback from customers is needed to assess CDC-INFO performance and identify areas in need of improvement.
For analysis purposes, the contractor analyzes the contact data and the customer satisfaction data collected through the surveys. Safeguards and procedures are in place to ensure that the contact information (name & email address) and the customer satisfaction survey data cannot be linked. The contact data are transmitted by the contact center contractor to the evaluation contractor on a weekly basis. Before the email contact data are transmitted to the third party contractor, the contact center contractor “strips” the email address from the data set. Therefore, the evaluation contractor is unable to link to any IIF when analyzing the survey data collected as part of this evaluation package. The proposed data collection will have no effect on the respondent’s privacy.
3. Use of Improved Information Technology and Burden Reduction
To the maximum extent possible, the evaluation has taken advantage of improved technology to reduce burden to consumers. All (100%) of responses to the Interactive Voice Response Survey and the Web Survey for E-Mail Inquirers involve the use of automated data collection techniques. This approach is consistent with best practices in contact center customer satisfaction data collection. (List of Technologies Used—Attachment I).
4. Efforts to Identify Duplication and Use of Similar Information
Staff at more than 30 Federal agencies were interviewed in an effort to identify duplication. None of these Federal organizations has conducted a national evaluation of a consumer response services center that is similar to the one being proposed for this study. The majority of those contacted offer specialized information specific to one health issue. These centers serve as referral resources for CDC-INFO (e.g., individuals who call contact center requesting specialized treatment information for advanced AIDS patients are referred to the National Institutes of Health’s AIDS INFO service).
A comprehensive literature search was conducted. To date there has been no published evaluation of consumer response services programs comparable scope to this study - a study spanning more than 40 health-related topics, or using the same measures to document satisfaction and impact. Those Federal agencies who conducted evaluation studies of their consumer response services centers were examined closely to take advantage of applicable methods and to identify any methodological problems that might detract from the validity, generalizability, or application of results.
5. Impact of Small Businesses or Other Small Entities
While small businesses or other small entities are not a target audience of CDC-INFO, if an employee of such a business chooses to contact CDC-INFO with a health-related question, he/she will be asked to participate in the Interactive Voice Response Survey or the Web Survey for E-Mail Inquirers. To minimize burden, the surveys are voluntary, automated, and include the minimum number of survey items to obtain the necessary customer satisfaction data.
6. Consequences of Collecting the Information Less Frequently
There are no legal obstacles to reduce the burden.
7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5
There are no special circumstances with this information collection request.
8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside the Agency
A. 60-day Federal Register Notice was published in the Federal Register on July 2, 2010, Vol. 75, No. 127, pages 38531-38532. (Attachment C) One public comment was received. (Attachment L)
B. Input was received from CDC employees and CDC-INFO contractors:
Elizabeth Harris, PhD
Vice President, EMT, Inc.
(818)990-8301
Paul Abamonte
Health Communications Specialist
National Center for Infectious and Respiratory Diseases, CDC
404-639-8584
Rasaan Jones
Health Communications Specialist
Office of the Associate Director for Communication
Division of Community Engagement, CDC-INFO Program
404-498-0202
Amy Burnett
Health Communications Specialist
Office of the Associate Director for Communication
Division of Community Engagement, CDC-INFO Program
404-498-2437
9. Explanation of Any Payment or Gift to Respondents
There will be no remuneration to respondents.
10. Assurance of Confidentiality Provided to Respondents
This submission has been reviewed by ICRO, who determined that the Privacy Act is not applicable. The information to be collected was reviewed by the national center Associate Director for Communication Science and It has been determined that this project is non-research, therefore, does not require a CDC IRB review (Attachment J). Evaluation data to be collected under this clearance are separate from the general operations of the Consumer Response Services Center. Evaluation surveys do not involve the collection of personal information from respondents, evaluation data will not be identifiable, and evaluation data will not be stored in a system of records as defined in the Privacy Act. EMT has provided an updated IRB exception letter from the previous OMB package (Attachment K)
CDC’s Consumer Response Service Center was established to respond to the public’s requests for public health information. Data relevant to the requestor’s identity (e.g., name, e-mail or postal address) are often required for CDC-INFO’s routine operational purposes. By “relevant to the request’s identity,” we mean that the data may suggest identity, without being absolute identifiers of an individual person (e.g., SSN).
B. In some cases, the evaluation data to be collected under this clearance can be distinguished from the general operations-related information that is needed to fulfill CDC-INFO’s core mission. Distinctions between evaluation data (which is the major subject of this OMB approval request) and operations-related information (which is generally outside the scope of this clearance request) are described in this section, where relevant to Privacy protections or Privacy Act applicability.
In the day-to-day operations of the CDC-INFO contact center IIF are used to facilitate the response to some inquiries. Specifically, email addresses and other non-IIF contact data (e.g. customer service request number, tracking numbers) are automatically captured by the contact center contractor’s customer relationship management (CRM) software when an inquirer chooses to email CDC-INFO. These data are stored in a secure environment by the contact center contractor, and the contractor has undergone a full IT Certification and Accreditation (C&A) FISMA assessment, in conjunction with the CDC Office of the Chief Information Security Officer, to ensure the security of the contractor’s systems.
For analysis purposes, the contractor analyzes the contact data and the customer satisfaction data collected through the surveys. Safeguards and procedures are in place to ensure that the contact information and the customer satisfaction survey data cannot be linked. The contact data are transmitted by the contact center contractor to the evaluation contractor on a weekly basis. Before the email contact data are transmitted to the third party contractor, the contact center contractor “strips” the email address from the data set. Therefore, the evaluation contractor is unable to link to any IIF when analyzing the survey data collected as part of this evaluation package.
Note: Only the area codes of incoming calls to CDC-INFO are captured; therefore, no other IIF are collected in the day-to-day operations of responding to CDC-INFO phone inquiries. Complete telephone numbers are not captured or recorded for routine calls to CDC-INFO. On a routine basis, the customer service representative only has access to the caller’s area code and exchange (first three digits of the number).
C. Responding to the phone or email surveys is voluntary. For the Interactive Voice Response survey (IVR), an automated message will ask callers for their consent to participate. Those who consent will be automatically asked the Brief Interactive Voice Response Survey questions after indicating consent. Active consent is required in order to participate in the Brief Interactive Voice Response Survey, in that the caller will be required to indicate affirmative consent by touch tone or voice. Callers who do not wish to participate can indicate so via touch tone or voice, or simply by hanging up. Callers who agree to participate are routed electronically to the IVR survey and there is no further contact with a Customer Service Representative. No personal information about the respondent, such as name or telephone number, is collected or stored with the responses to the customer satisfaction survey.
Requestors who contact CDC-INFO via email or the Internet are provided with a link to the appropriate web-based customer satisfaction survey. (Attachment G - English) (Attachment H – Spanish) The web-based survey is hosted by EMT, which will not ask for or capture any personal information from the respondent, or the respondent’s IP address. Like the brief IVR survey, this satisfaction survey of e-mail inquirers will also be completely automated. All e-mail inquirers will receive an auto-message asking if they would like to participate in a brief web satisfaction survey. The brief survey will be conducted using the web-based survey software Snap Surveys. E-mail inquirers will need to click on a link contained in the auto e-mail (thereby consenting to the survey) in order to participate in the web survey.
D. All survey tools provide the participant an opportunity to decline. All evaluation processes are voluntary and all available in English and Spanish.
11. Justification for Sensitive Questions
Type of Respondent |
Form Name |
No. of Respondents |
No. Responses per Respondent |
Average burden per Response (in hrs) |
Total Burden Hours |
General Callers |
Brief Interactive Voice Response Survey (English & Spanish) |
92,000 |
1 |
4/60 |
6,133 |
Email Inquirers |
Web Survey for E-mail Inquirers (English & Spanish) |
1,460 |
1 |
3/60 |
73 |
Total Burden Hours |
|
|
|
|
6,206 |
Questions concerning Race and Ethnicity may be considered sensitive by a portion of respondents. Race and Ethnicity questions are included in the set of demographic questions that may be asked of respondents. CDC-INFO only reports aggregate data and not individual data. Where relevant to the customer satisfaction evaluation of CDC-INFO, Race and Ethnicity data will be collected consistent with HHS policy and standard OMB classifications.
12. Estimate of Annualized Burden Hours and Costs
The estimated number of respondents represents the product of the average number of calls or emails handled by CDC-INFO and response rates to each respective survey are based on evaluation data obtained in the past 3 fiscal years. Two new additions to the original IVR survey increase the burden from three to four minutes. Additionally, as mentioned in the summary (Attachment A). The revised burden has decreased from 176,203 to 6,202 burden hours from. The number of respondents has decreased from 327,433 to 93,460. The potential annualized cost to respondents has decreased from a maximum of $8,676,627 to $44,993.50
Table 12A. Estimated Annualized Burden Hours
Below is a table outlining the average annual cost to respondents for participating in data collection activities.
Table 12B. Estimated Annualized Burden Costs
Type of Respondent |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
General Callers (English & Spanish) |
6,133 |
$7.25 |
$44,464.25 |
E-Mail Inquirers (English & Spanish) |
73 |
$7.25 |
$529.25 |
Total Respondent Cost 44,993.50 |
|||
The Federal Minimum Wage rate of $7.25/hr is included in this table to reflect what the average callers’ time would be worth when contacting CDC-INFO. The rate was taken from the United States Department of Labor’s website.
Approximately 92,000 surveys at 4 minutes per survey amounts to a yearly burden of 6,133 hours. Given the minimum wage of $7.25 per hour, the annual cost to respondents to participate in the Interactive Voice Response Survey is $44,466.
Below is a summary of the information contained in the table above
Interactive Voice Response Survey (Satisfaction Survey for Callers): The estimated number of persons to be surveyed per year is 92,000. Two new additions to the original IVR survey extend the length from three to four minutes. Approximately 92,000 surveys at 4 minutes per survey amounts to a yearly burden of 6,133 hours. Given the minimum wage of $7.25 per hour, the annual cost to respondents to participate in the Interactive Voice Response Survey is $44,466.
Web Survey for E-Mail Inquirers (Satisfaction Survey for E-Mail Inquiries): The estimated number of respondents to be surveyed each year is 1,460. The survey takes 3 minutes to complete for a total annual burden of 73 hours. Given the minimum wage of $7.25 per hour, the annual cost to respondents to participate in the Web Survey for E-Mail Inquirers is $529.
13. Estimates of Other Total Annual Cost Burden to Respondents or Recordkeepers
There are no additional costs to the respondents.
14. Annualized Cost to the Government
The estimated annualized cost to the government for the IVR Satisfaction Survey and the Web Survey for E-mail Inquirers is $489,936.
Invoice data from FY10 were analyzed to determine the direct labor costs for the evaluation contractor to collect the data; analyze the results; and produce reports and briefings. Further, the personnel costs of the two CDC-INFO FTEs involved in evaluation work and contract oversight were included: 10% of CDC-INFO program manager’s salary and 80% of the CDC-INFO evaluation contract Project Officer’s salary. CDC labor costs for oversight of the project is the salary for a GS 11 at $59,987 and GS 14 at $101,035 for a total of $161,022.
15. Explanation for Program Changes or Adjustments
The evaluation approach described in this study has been revised to accommodate budget cuts; many of the survey instruments in the previous OMB package have been eliminated (Attachment A). The revised request seeks approval to collect data from CDC-INFO inquirers through two surveys: The Interactive Voice Response Survey and the Web Survey for E-Mail Inquirers were previously approved under the existing clearance. Minimal changes have been made to both surveys in order to better align survey items requesting demographic data with the U.S. Census standards (see Attachment A: summary of Changes).
The revised burden of 6,206 is less than the currently approved burden of 176,283, based on (1) using the actual averages of call and e-mail volume and response rates over the past 3 fiscal years, and (2) additional questions included.
The average number of incoming calls over the past three fiscal years is less than projected in the original OMB application; the average annual call volume amounts to 80% of the 500,000 annual calls proposed initially. Conversely, the response rate to the Interactive Voice Response Survey is substantially higher than projected, with 25% of callers participating in this survey compared to the 5% estimated, and amounting to approximately 92,000 respondents annually.
16. Plans for Tabulation and Publication and Project Time Schedule
Data collection will take place continuously throughout the three years approved by OMB. Analyses will consist primarily of frequencies and descriptive statistics. The contact center data will be stratified by various factors: age, race and ethnicity. The data collected will be used by CDC staff for conferences, presentations, and publications as well as for evaluation purposes.
Project Time Schedule
Activity |
Time Schedule |
Data Collection |
Immediately after OMB approval and conducted continuously thereafter |
Monthly Report |
Monthly after OMB Approval |
Annual Report |
3 months after end of the calendar year |
Presentations at Conferences |
Ongoing |
17. Reason(s) Display of OMB Expiration Date is Inappropriate
Exemption is not being sought.
18. Exceptions to Certification for Paperwork Reduction Act Submissions
There are no exceptions to certification.
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL
METHODS
1. Respondent Universe and Sampling Methods
No statistical methods will be used to select respondents for the Interactive Voice Response Survey or the Web Survey for Email Inquirers because all persons who contact CDC-INFO via phone or email are encouraged to complete the surveys.
The potential respondent universe for the Interactive Voice Response Survey is similar to the average number of callers who have responded to the IVR survey over the past 3 fiscal years. One hundred percent (100%) of callers are asked to participate in the survey and the average annual call volume is 400,000. On average, 23% of callers respond to the Interactive Voice Response survey annually; therefore, an annual sample of 92,000 survey respondents is projected.
Similarly, the potential respondent universe for the Web Survey for Email Inquirers is similar to the average number of email inquirers who have responded to the survey over the past 3 fiscal years. One hundred percent (100%) of e-mailers are asked to participate in the web-based satisfaction survey. Based on data from the past three fiscal years, the total respondent universe is approximately 74,000, with 2% of e-mailers responding to the Web Survey for Email Inquirers annual; therefore, an annual sample of 1,480 survey respondents is projected.
2. Procedures for the Collection of Information
For the Interactive Voice Response Survey, an automated message will ask callers for their consent to participate. Those who consent will be automatically asked the survey questions after indicating consent. Active consent is required in order to participate in the Brief Interactive Voice Response Survey, in that the caller will be required to indicate affirmative consent by entering a touch tone response. Callers who do not wish to participate can indicate so via touch tone or simply by hanging up during the survey.
For the Web Survey for Email Inquirers, all email responses from CDC-INFO include a link to a web-based survey. E-mail inquirers will click on the link, thereby acknowledging consent to the survey, in order to participate in the web survey.
3. Methods to Maximize Response Rates and Deal with Nonresponse
All callers and email inquirers to CDC-INFO will be offered the opportunity to complete the surveys. The response rate for CDC-INFO’s Interactive Voice Response Survey is above normal response rates for contact center customer satisfaction surveys. While the response rate for the Web Survey for Email Inquirers is low, this is consistent with other contact center email surveys. Plans have been developed to improve the email response rate. A third party evaluator is currently developing a proposal to highlights steps that can increase the response rate for the Web Survey for Email Inquirers.
4. Tests of Procedures or Methods to be Undertaken
The data collection instruments included in the appendices were pilot tested on no more than nine adults per instrument. Pilot tests included administration of draft instruments and solicitation of comments on the instrument from respondents. The purpose of the pilot testing was to:
Identify instructions and/or questions within the instrument that were unclear or confusing
Obtain suggestions for improving questions or instructions
Identify data collection procedures that were unclear or confusing
Measure respondent burden based on the average time to complete instruments
Identify ways to minimize respondent burden and improve accuracy in the completion of instruments
Information gathered from pilot testing was critical and central to the instrument development and revision process. Pilot data were analyzed to identify the psychometric properties of the survey. A reliability) analysis was conducted on each measure and scale. In addition, distributions of each variable within each scale were examined in order to detect ceiling or floor effects.
A thorough review of qualitative responses to the surveys was also conducted. This resulted in a variety of changes to the instruments, including re-wording of introductory instructions, rewording of specific items and modifications/additions to response categories.
The data collection instruments for the Interactive Voice Response Survey and the Web Survey for Email Inquirers were used at the CDC-INFO contact center under the current OMB package 0920-0753.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
The third-party evaluation contractor, EMT Associates, Inc., developed the data collection tools to be used in the evaluation of CDC-INFO. The persons responsible for analyzing the data are:
J. Fred Springer, Ph.D. Project Director EMT Associates, Inc. 230 Franklin Road, Suite 12H Franklin, TN 37065 (615) 595-7658 |
|
The Project Manager and person responsible for overseeing receipt, processing and data analysis is:
Elizabeth Harris, Ph.D.
Project Manager
EMT Associates, Inc.
15720 Ventura Blvd., Penthouse
Encino, CA 91436
(818) 990-8301
Within the Centers for Disease Control and Prevention, the following individuals will have oversight responsibility for all contracts receiving and approving contract deliverables:
Rasaan Jones Project Officer CDC-INFO, Division of Community Engagement, Office of the Associate Director for Communication CDC Century Center Bldg 2500 Rm 4105 MS E 90 Atlanta, GA 30329 Ph: (404) 498-0202
|
Nancy Lindsey COTR CDC-INFO, Division of Community Engagement, Office of the Associate Director for Communication CDC Century Center Bldg 2500 Rm 4120 MS E 90 Atlanta, GA 30329 Ph: (404) 498 – 0039 |
|
|
List of Attachments
Summary of Revisions
Public Health Service Act (42 USC 241) Section 301
Federal Register Notice
Citizen Service Levels Interagency Committee (CSLIC) Report
Brief Interactive Voice Response Survey (English)
Brief Interactive Voice Response Survey (Spanish)
Web Survey for E-Mail Inquirers (English)
Web Survey for E-Mail Inquirers (Spanish)
List of Technologies Used
CDC IRB Exemption
IRB Approval letter
Public Comment on Federal Register Notice
1 Executive Order 12862. (2010, March 12). In Wikisource, The Free Library. Retrieved 17:03, October 12, 2010, from http://en.wikisource.org/w/index.php?title=Executive_Order_12862&oldid=1814311
| File Type | application/msword |
| Author | imn1 |
| Last Modified By | ziy6 |
| File Modified | 2010-10-14 |
| File Created | 2010-10-13 |