B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL
METHODS
1. Respondent Universe and Sampling Methods
A. Interactive Voice Response Survey (Customer Satisfaction Survey for Callers): We need to restrict the confidence intervals on our parameter estimates to a reasonable width. A width of 1% of the 5-point scale would be appropriate, or +/- 0.025 points, using a 90% confidence interval. If a standard deviation of 1.2 is expected for a 5-point scale, then the sample size would be:
N = (2*Z90 / W * S)2
= (2*1.645 / 0.025 * 1.2)2
= 24,938.73
≈ 25,000,
Where N is the sample size, Z90 is the standard normal deviate for the 90% confidence interval, W is the desired width of the confidence interval, and S is the standard deviation.
Response rate estimates taken from similar IVR surveys are about 5%, so we will ask the entire population to participate, in order to get the desired sample of 25,000 (i.e., 500,000 * .05 = 25,000).
B. Live Phone Follow-up Survey (Follow Up Survey): After the IVR survey, respondents will be asked if they will participate in a possible follow-up telephone survey within the next 30 days. Among those who agree to the IVR, we expect about 0.25 will agree to go on for the follow-up call, of whom we will successfully reach approximately 0.5 (totaling about 3,125 respondents).
This high expected agreement rate is based on the general psychological tendency to agree to larger requests, once smaller requests have been accepted.
C. Web Survey for E-Mail Inquirers (Customer Satisfaction Survey for E-Mail Inquiries): Because of extremely low expected response rates, everyone who e-mails CDC-INFO will be asked to participate in the web-based satisfaction survey. Based on the PEC’s prior work with web surveys of consumer response services programs, approximately 3% are anticipated to participate for a total of 330 respondents. The total respondent universe is approximately 11,000.
Postcard Survey for Single Publication Orders (Postcard Survey for Individual Publication Orders): The respondent universe for the survey is 210,000. Private sector fulfillment literature shows that response rates are very low for this type of survey, around 1%, for an expected sample size of 2,100.
Postcard Survey for Bulk Mailing: The respondent universe for the survey is 95,000. Private sector fulfillment literature shows that response rates are very low for this type of survey, around 1%, for an expected sample size of 950.
Web Survey for Internet Publication Orders: The respondent universe for the survey is 95,000. Private sector fulfillment literature shows that response rates are very low for this type of survey, around 1%, for an expected sample size of 950.
Web Survey for E-Mailed Publications: The respondent universe for the survey is 100,000. Private sector fulfillment literature shows that response rates are very low for this type of survey, around 1%, for an expected sample size of 1,000.
H. Customer Representative Survey (Key Informant Survey): Because those being surveyed are employees of the CDC-INFO contractor, we expect that all who are asked will participate in the Customer Representative Survey for a total of 100 respondents.
I. Special Event/Outreach Surveys (General Public): Because the Special Outreach Survey data is collected by the Customer Service Representative, with whom the caller has presumably had a positive interaction, we expect the response rate to be 80% for a total of 25,600 respondents. The respondent universe is estimated to be 32,000.
J. Special Event/Outreach Surveys (Professionals): Because the Special Outreach Survey data is collected by the Customer Service Representative, with whom the caller has presumably had a positive interaction, we expect the response rate to be 80% for a total of 10,400 respondents. The respondent universe is estimated to be 13,000.
K. Emergency Response Surveys (General Public): The CDC’s Emergency Epidemic Investigations (OMB Approval #0920-0008, expiration date 12/21/06, revision application currently under review at OMB) report response rates of between 90-95% when collecting emergency response or outbreak data. We anticipate 75%-95% response rate, depending on the size and scope of the emergency. However, CDC-INFO capacity to survey 90% of callers in the event of a Level 2-4 emergency is limited. Therefore, CDC-INFO will only survey 20% of callers in the event of a Level 2-4 emergency. The anticipated sample size ranges from 31,151 in the event of a Level 1 emergency to 645,630 respondents in the event of a Level 4 emergency. The number is an estimate, since it is impossible to determine in advance what emergencies will emerge during the next year, and the scope of emergencies which may emerge in any given year.
L. Emergency Response Surveys (Professionals): The CDC’s Emergency Epidemic Investigations (OMB Approval #0920-0008, expiration date 12/31/06, revision application currently under review at OMB) report response rates of between 90-95% when collecting emergency response or outbreak data. We anticipate 90-95% response rate as well. However, CDC-INFO capacity to survey 90% of callers in the event of a Level 2-4 emergency is limited. Therefore, CDC-INFO plans to survey between 5%-20% of callers in the event of a Level 2-4 emergency. The anticipated sample size is 7,459 respondents in the event of a Level 1 emergency and 596,504 respondents in the event of a Level 4 emergency. The number is an estimate, since it is impossible to determine in advance what emergencies will emerge during the next year, and the scope of emergencies which may emerge during any given year.
2. Procedures for the Collection of Information
The evaluation data will be collected using differing methodologies, depending upon the target audience and purpose:
Brief Interactive Voice Response Survey (Customer Satisfaction Survey for Callers). All callers to CDC-INFO who are not warm-transferred out to another agency will be automatically sent to an automated message at the conclusion of the call. A warm transfer involves the operator dialing the number of the agency for the caller, and then connecting the caller directly. For example, when 411 is dialed there is an option provided after the number is received to connect the caller directly with the agency whose number was sought. This process of “warm transfer” avoids the caller having to hang up and then dial the number themselves. Regarding the brief Interactive Voice Response feedback survey, the automated message will ask for their consent to participate in the brief Interactive Voice Response (IVR) feedback survey. The system will be completely automated—callers will hear an automated voice asking if they would like to participate in a brief satisfaction survey. Those who consent will be automatically asked the Brief Interactive Voice Response Survey questions after indicating consent. Active consent is required in order to participate in the Brief Interactive Voice Response Survey, in that the caller will be required to indicate affirmative consent by touch tone or voice. Callers who do not wish to participate can indicate so via touch tone or voice, or simply by hanging up.
Live Follow-Up Survey (Follow Up Survey): All callers who elect to participate in the brief IVR satisfaction survey will be offered the opportunity to participate in the follow-up telephone interview. Active consent is required, in that callers who wish to participate in the live follow-up survey will be required to input their telephone number via touch-tone or voice response. Callers who do not wish to participate can indicate by touch tone or voice response, or simply by hanging up. The consent question for the live follow-up survey is the last question in the Brief Interactive Voice Response Survey. Callers are informed it is the last question, so they feel free to hang up if they do not want to participate. Follow-up telephone interviews will be conducted with callers to more fully assess satisfaction and utility and use of information. The primary purpose will be to gain feedback on the utility of the information gained in the call and their perception of how well the call has met their need. Follow-up telephone interviews using a structured interview protocol will be conducted by the program evaluation center within 6 weeks of contact with CDC-INFO. Upon contact, potential respondents must give the code word “CDC-INFO” in order for the survey to progress. The phone call will immediately terminate if the code word is not given. No information at all about who is calling or the survey itself will be provided without being given the code word first. Following survey completion or drop, the telephone number will be stripped from the database and replaced with a randomly generated number.
Web Survey for E-Mail Inquirers (Customer Satisfaction Survey for E-Mail Inquiries). All e-mail inquirers will have the opportunity to complete a brief web-based feedback survey. Like the brief IVR survey, this satisfaction survey of e-mail inquirers will also be completely automated. All e-mail inquirers will receive an auto-message asking if they would like to participate in a brief web satisfaction survey. The brief survey will be conducted using the web-based survey software Snap Surveys. E-mail inquirers will need to click on a link contained in the auto e-mail (thereby consenting to the survey) in order to participate in the web survey.
Postcard Survey for Single Publication Orders (Postcard Survey for Individual Publication Orders): A brief paper and pencil survey (will fit on a postcard) of satisfaction, responsiveness to publication requests, and timeliness/state of publications received will be conducted to assess satisfaction with fulfillment center transactions. Fulfillment center surveys will be conducted through a self-addressed and stamped postcard included with each publication distributed. The potential respondent will choose whether s/he wants to fill out and send in the survey, or throw it away in the trash. There is no consequence for failure to complete the survey, since it will be included with the publication order.
Postcard Survey for Bulk Mailing: A brief paper and pencil survey (will fit on a postcard) of satisfaction, responsiveness to publication requests, and timeliness/state of publications received will be conducted to assess satisfaction with fulfillment center transactions. Fulfillment center surveys will be conducted through a self-addressed and stamped postcard included with each publication distributed. The potential respondent will choose whether s/he wants to fill out and send in the survey, or throw it away in the trash. There is no consequence for failure to complete the survey, since it will be included with the publication order.
Web Survey for Internet Publication Orders. All web inquirers submitting a request for an electronic publication, hard copy publication or web link, and all telephone inquirers requesting a PDF file or web link via e-mail will have the opportunity to complete a brief web-based feedback survey. Like the brief web satisfaction survey for individuals e-mailing questions to the CDC, this satisfaction survey of the Internet and telephone publication ordering process will also be completely automated. All individuals submitting a request for bulk publications, web links and/or single PDF publications will receive an auto-message confirming their order. At the bottom of the e-mail will be a message asking if they would like to participate in a brief web satisfaction survey. The brief survey will be conducted using the web-based survey protocol SNAP Surveys. Potential respondents will need to click on a link contained in the auto e-mail (thereby consenting to the survey) in order to participate in the web survey.
Web Surveys for E-Mailed Publications Orders. All web inquirers submitting a request for an electronic publication or web link and all telephone inquirers requesting a web link or PDF file to be sent via e-mail will have the opportunity to complete a brief web-based feedback survey. Like the brief web satisfaction survey for individuals e-mailing questions to the CDC, this satisfaction survey of the publication received will also be completely automated. When individuals receive their PDF file containing the publication or e-mail containing the web link, the accompanying e-mail will include (at the bottom) a message asking if they would like to participate in a brief web satisfaction survey. The brief survey will be conducted using the web-based survey protocol SNAP Surveys. Potential respondents will need to click on a link contained in the auto e-mail (thereby consenting to the survey) in order to participate in the web survey.
Web Survey of Customer Service Representatives (Key Informant Survey): Surveys of key informants will be conducted with all CDC-INFO customer service representatives. These surveys provide opportunity for further insight into CDC-INFO quality. Surveys of key informants will be conducted on a monthly basis via the internet.
Special Event/Outreach Survey (General Public): Special event surveys for the general public will assess consumer response to national CDC campaigns and media events. Special event surveys will be conducted via live CDC INFO agent at the conclusion of the call during CDC campaigns and media events. All callers contacting CDC-INFO regarding the special event/campaign will be asked to participate in the special event survey. Callers must provide verbal consent in order to participate in the survey.
Special Event/Outreach Survey (Professionals): Special event surveys for professionals will assess consumer response to national CDC campaigns and media events. Special event surveys will be conducted via live CDC INFO agent at the conclusion of the call during CDC campaigns and media events. All callers contacting CDC-INFO regarding the special event/campaign will be asked to participate in the special event survey. Callers must provide verbal consent in order to participate in the survey.
Emergency Response Surveys (General Public): Emergency response surveys will assess issues related to public health emergencies, such as the caller’s health or exposure to outbreak, as well as concerns and issues needing to be addressed. Emergency response surveys will be conducted via live CDC-INFO agent during the call, during public health emergencies. Demographic questions will only be asked at the calls’ conclusion, and only when appropriate. Between 20%-75% of callers contacting CDC-INFO regarding the public health emergency will be asked to participate in the emergency response surveys. Callers must provide verbal consent in order to participate in the survey.
Emergency Response Surveys (Professionals): Emergency response surveys will assess issues related to public health emergencies, such as the exposure of patients or constituents to outbreak, as well as concerns and issues needing to be addressed. Emergency response surveys will be conducted via live CDC-INFO agent during the call, during public health emergencies. Between 5% to 90% of callers contacting CDC-INFO regarding the public health emergency will be asked to participate in the emergency response surveys. Callers must provide verbal consent in order to participate in the survey.
Each data collection strategy is tailored to the method of CDC-INFO inquiry used by the potential respondent. Where electronic communication (e-mail and web) served as the method of inquiry, electronic data entry will serve as the method for data collection.
Web-based surveys were not deemed appropriate for respondents submitting inquiries by telephone because of the following factors:
Industry standards for measuring satisfaction with consumer response service centers involve matching the method of inquiry to the evaluation method.
Customer satisfaction should be measured as close as possible to the inquiry, in order to avoid recall problems. Immediate telephone surveying is a standard method for collecting customer satisfaction data at the conclusion of a call.
Response rates for web surveys of consumer response services centers are a fraction of the response rate for telephone surveys (Harris et al., 2004).
It cannot be assumed that telephone inquirers have regular, reliable Internet access.
3. Methods to Maximize Response Rates and Deal with Nonresponse
This section discusses methods to maximize response rates for live surveys ONLY.
The Program Evaluation Center’s Program Manager will coach data collectors conducting the Live Follow-Up Survey, Special Event and Emergency Response Surveys to use several approaches to reduce refusals by callers. Examples of these coaching approaches are as follows:
Data collectors will be coached to use the survey introduction exactly as written.
Data collectors will be coached to use an inviting tone of voice. Callers who believe the data collector is uninterested in completing the survey may by refuse to complete the survey. Data collectors who present the survey enthusiastically can encourage callers to complete the survey.
Respondents who refuse to participate (live follow-up survey only) will be asked if there is a better time that they can be contacted, in order to provide another opportunity for participation.
Data collectors will be provided with a script that describes the purpose of the Live Follow-Up Survey, which will describe for the potential respondent the value of their participation.
In addition to these examples, the Program Manager will adjust coaching to the individual data collector needs in order to improve response rates.
The data collectors will be trained to explain that the information will be used to help manage CDC-INFO more effectively and to provide the best service. In addition, the Program Evaluation Center’s Program Manager will receive weekly reports on individual survey response rates for all of their team members. The Program Manager will monitor these rates to make sure that all team members, as well as the entire team collectively, adhere to specified performance guidelines.
4. Tests of Procedures or Methods to be Undertaken
The data collection instruments included in the appendices were pilot tested on no more than nine adults (no more than nine respondents for each instrument). Pilot tests included administration of draft instruments and solicitation of comments on the instrument from respondents. The purpose of the pilot testing was to:
Identify instructions and/or questions within the instrument that were unclear or confusing
Obtain suggestions for improving questions or instructions
Identify data collection procedures that were unclear or confusing
Measure respondent burden based on the average time to complete instruments
Identify ways to minimize respondent burden and improve accuracy in the completion of instruments
Results from pilot testing suggested that some response categories and questions needed to be reworded to improve clarity and understandability, and, in a few instances, eliminated entirely. Information gathered from pilot testing was critical and central to the instrument development and revision process. Pilot data was analyzed to identify the psychometric properties of the survey. A reliability (using coefficient alpha) analysis was conducted on each measure and scale. In addition, distributions of each variable within each scale were examined in order to detect ceiling or floor effects.
A thorough review of qualitative responses to the surveys was also conducted. This resulted in a variety of changes to the instruments, including re-wording of introductory instructions, rewording of specific items and modifications/additions to response categories.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
EMT Associates, Inc., grantee for the Program Evaluation Center (PEC) developed the data collection tools to be used in the evaluation of CDC-INFO. The PEC will conduct the national evaluation of CDC-INFO. The Principal Investigators for the national evaluation, and persons responsible for the statistical design and data analysis are:
J. Fred Springer, Ph.D. Project Director EMT Associates, Inc. 230 Franklin Road, Suite 12H Franklin, TN 37065 (615) 595-7658 |
Elizabeth Harris, Ph.D. Project Manager EMT Associates, Inc. 15720 Ventura Blvd., Penthouse Encino, CA 91436 (818) 990-8301
|
Consulting on the statistical aspects of the study will be:
J. Fred Springer, Ph.D. Project Director EMT Associates, Inc. 230 Franklin Road, Suite 12H Franklin, TN 37065 (615) 595-7658 |
Preston Kiekel, Ph.D. Statistician EMT Associates, Inc. 15720 Ventura Blvd., Penthouse Encino, CA 91436 (818) 990-8301
|
Dr. Kiekel also provided the statistical analyses for this report.
Although PEC research assistants and Pearson Government Solutions Customer Service Representatives are responsible for conducting the planned data collection, the Project Manager and person responsible for overseeing receipt, processing and data analysis, is:
Elizabeth Harris, Ph.D.
Project Manager
EMT Associates, Inc.
15720 Ventura Blvd., Penthouse
Encino, CA 91436
(818) 990-8301
Within the Centers for Disease Control and Prevention, the following individual will have oversight responsibility for all contracts and grant activities, including the data analysis:
Dottie Knight
Program Manager
1600 Clifton Road
M/S E-06
Atlanta, GA 30333
(404) 498-3208
Dottie Knight also provided substantial feedback on all instruments to be used in the evaluation of CDC-INFO. Draft instruments were circulated to CDC Programs for their feedback as well.
List of Attachments
Attachment A: Federal Register Notice
Attachment B: CDC Program Transition Timeline
Attachment C: Public Health Service Act (42 USC 241) Section 301
Attachment D: Citizen Service Levels Interagency Committee (CSLIC) Report
Attachment E: Brief Interactive Voice Response Survey (English) (Customer Satisfaction Survey for Callers)
Attachment F: Brief Interactive Voice Response Survey (Spanish) (Customer Satisfaction Survey for Callers)
Attachment G: Live Follow-up Telephone Survey (English) (Follow Up Survey)
Attachment H: Live Follow-up Telephone Survey (Spanish) (Follow Up Survey)
Attachment I: Web Survey for E-Mail Inquirers (English) (Customer Satisfaction Survey for E-Mail Inquires)
Attachment J: Web Survey for E-Mail Inquirers (Spanish) (Customer Satisfaction Survey for E-Mail Inquiries)
Attachment K: Postcard Survey for Single Publication Orders (English) (Postcard Survey for Individual Publication Orders)
Attachment L: Postcard Survey for Single Publication Orders (Spanish) (Postcard Survey for Individual Publication Orders)
Attachment M: Postcard Survey for Bulk Mailing (English)
Attachment N: Postcard Survey for Bulk Mailing (Spanish)
Attachment O: Web Survey for Internet Publication Orders (English)
Attachment P: Web Survey for Internet Publication Orders (Spanish)
Attachment Q: Web Survey for E-Mailed Publications (English)
Attachment R: Web Survey for E-Mailed Publications (Spanish)
Attachment S: Customer Service Representative Survey (Key Informant Survey)
Attachment T: Special Event/Outreach Survey (General Public) (English)
Attachment U: Special Event/Outreach Survey (General Public) (Spanish)
Attachment V: Special Event/Outreach Survey (Professionals) (English)
Attachment W: Special Event/Outreach Survey (Professionals) (Spanish)
Attachment X: Emergency Response Survey (General Public) (English)
Attachment Y: Emergency Response Survey (General Public) (Spanish)
Attachment Z: Emergency Response Survey (Professionals) (English)
Attachment AA: Emergency Response Survey (Professionals) (Spanish)
Attachment BB: Historical Evolution of Consumer Response Services Centers
Attachment CC: List of Technologies Used
Attachment DD: “This call may be monitored for quality assurance purposes”
Attachment EE: List of Federal Contacts
References
| File Type | application/msword |
| File Title | DRAFT 8/04/05 |
| Author | Elizabeth Harris, Ph.D. |
| Last Modified By | ziy6 |
| File Modified | 2007-05-21 |
| File Created | 2007-05-21 |