Supporting Statement B

Data Collection for Evaluation of Education, Communication, and Training (ECT) Activities for the Division of Global Migration and Quarantine (0920-0932)

Evaluating the Effectiveness of Ebola CARE Plus Program

Generic Information Collection Request

April 2, 2015

Statement B

Contact:

Amy McMillen

Office of Policy and Planning

National Center for Emerging and Zoonotic Infectious Diseases

Centers for Disease Control and Prevention

1600 Clifton Road, N.E., MS D76

Atlanta, Georgia 30333

Phone: (404) 639-1045

Fax: (404) 248-4146

Email: [email protected]

TABLE OF CONTENTS

PART B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 3

B.1. Respondent Universe and Sampling Methods 3

2. Procedures for the Collection of Information 6

3. Methods to Maximize Response Rates and Deal with No Response 9

4. Test of Procedures or Methods to be Undertaken 9

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 10

Reference 12

PART B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

Five airports in the United States are currently designated for Ebola screening: Chicago O’Hare (ORD); Newark Liberty (EWR); Atlanta Hartsfield-Jackson (ATL); JFK International (JFK); and Dulles International (IAD). For this evaluation, we intend to interview a systematic convenience sample (Sudman, 1976) of travelers who undergo Ebola screening after arriving at JFK and IAD within our data collection window (as described below). These airports have been selected because enhanced CARE+ program activities are now being implemented at them and because they have the highest numbers of travelers. CDC would like to assure the usefulness of CARE+ program activities. This request also includes conducting interviews with persons from Ebola-affected countries to evaluate a new tool (Attachment E) designed to be used for assessing the cultural competency of the CARE Kit (Attachment F) These persons attended a West African Diaspora Ebola Summit that CDC staff participated in and indicated that they were willing to provide feedback to CDC on Ebola messages.

To better assess travelers’ experiences through the Ebola screening process, we propose to survey about 1200 travelers who speak English or French and are 18 years or older, coming into the United States (U.S.) at two of the five designated entry airports from countries with widespread Ebola outbreaks: John F. Kennedy (JFK) and Dulles International Airport (IAD). All interviews will be conducted over a four-month data collection period (April – July 2015), and the evaluation team proposes conducting:

1. 1200 in-person surveys in two airports (10 minutes)

2. 960 phone surveys with travelers near the beginning of each traveler’s interaction with a health department (within 3-5 days after the airport survey) (10 minutes)

3. 768 phone surveys with travelers near the end of each traveler’s interaction with a health department (2 days prior to their monitoring and reporting end date) (5 minutes)

Respondent Universe

CDC and RTI analyzed flight record data from the CDC Quarantine Activity Reporting System (QARS) to calculate the anticipated total number of travelers arriving into both JFK and IAD. We analyzed data from March 8 to March 21, 2015, to calculate a weekly average of travelers. QARS data indicates that there are 245 travelers per week at JFK and 140 per week at IAD. This includes all passengers, including children. When assuming that 90% are adults, (an assumption supported by CDC data) we calculate that there are 221 adult travelers entering JFK and 126 adult travelers entering IAD per week. Together, the expected number of adult travelers is 347 per week.

If the data collection began April 19, 2015, and continued through July 31, 2015, (approximately 15 weeks), we would expect up to 5,205 adult passengers (see “Total expected adult travelers during data collection period” in table B-1) in the total respondent universe, of whom 99% would speak either French or English for 5,152 adult passengers. (Feedback from staff at CDC Quarantine Stations suggests 99% of incoming passengers from Ebola-affected countries speak French and/or English).

Although the total universe for the full projected data collection window is 5,152 adult passengers, if data collection ends before July 31, 2015, due to us meeting sample quotas early then the total available universe will shrink. The actual potential respondent universe would be smaller in light of the number of weeks that data subsequently is not collected given that we will have met the required sample size. The longer timeframe was proposed because when this request was originally being prepared it looked like Liberia was going to be taken off the list of Ebola-affected countries. Since Liberia accounts for approximately 30% of travelers, it would have taken us longer to complete 1200 surveys. Liberia is still on the list since it recently had a confirmed case of Ebola.

In practice, we anticipate inviting 1,600 travelers to participate (see “Invited to participate” in table B-1). This is based upon first considering our desired sampling size (1,200) and then assuming 75% of approached travelers would agree to the study. 1,600 travelers would need to be invited to achieve 1,200 passengers for the airport intercepts. It is conceivable that we will need to invite more if we achieve a lower response rate. Regardless, we will be drawing from the 5,152 in the universe.

Anticipated Sampling

We expect 75% of the 1,600 invited travelers will participate in airport intercept interviews (n=1200), of these 80% will participate in first telephone survey (n=960), and of these 80% will participate in the second telephone survey (n=768). In other words, between the airport survey and each of the telephone surveys, we expect some travelers to decline participation, and some that consent to participate will not be reachable within the five attempts that we plan for surveying them. (See “Response rate assumptions” below for explanation on how participation rates were calculated.)

At the two airports and during peak arrival dates and times, we will ask every passenger who leaves the secondary screening area to participate in a brief interview. Participation is entirely voluntary. Logistically, this approach will be feasible due to the manner in which passengers are processed by U.S. Customs and Border Protection. Traveler surveys will be conducted in English or French by fluent interviewers depending on traveler preference.

Table B1 first describes the total number of adult passengers in our response universe assuming data that data collection would occur between April 19 and July 31, 2015 (column 2) followed by the number of these who would speak English or French (column 3).

Next, Table B1 describes the number of people we anticipate inviting to participate (column 4), and then the number expected in each of the three survey points (columns 5, 6 and 7).

Table B-1. Eligibility and participation rates determining sample size for traveler surveys

	Traveler respondent universe, assuming data collection through 7/31			Three Survey Points
Airport	Total expected adult travelers during data collection period (4/19/15-7/31-15)	English or French Speaking (4/19/15-7/31-15))*	Invited to Participate**	Anticipated to Participate in survey at airport	Anticipated to Participate in 1st phone follow-up survey	Anticipated to Participate in 2nd phone follow-up survey
JFK	3,315	3,281	896	672	538	431
IAD	1,890	1,871	704	528	422	337
Total	5,205	5,152	1,600	1,200	960	768

*Percent of travelers who speak English or French estimated to be 99%

** Survey interviewers will only be at airports during peak traveler days and times so not all travelers expected to arrive in the U.S. at these two airports will be invited to participate. This numbers is based on an analysis of how many travelers would be invited to participate during the peak traveler days and times identified. These numbers are estimates based on historical information of reported travelers from Ebola affected countries drawn from CDC Quarantine Activity Reporting System (QARS) between March 8th – March 21, 2015 at JFK and IAD airports.

Response Rate Assumptions

The proportion of respondents we expected to participant in the airport intercept and the telephone surveys is based on a previous CDC baseline study completed in December 2014 (OMB Gen IC No. 0920-0932) where they experienced over 80% response. Many participants in the baseline evaluation expressed appreciation for being invited to offer their insights and suggestions about the process.

Power Analysis

The evaluation team has centered the power analysis on traveler monitoring behavior, as measured by an additive index comprising 6 dichotomous items that will be asked during the first telephone follow-up. That measure will have a maximum range of 0 to 6. Assuming (conservatively) a standard deviation of .5 and a sample size of 960 at the first telephone follow-up, we would have the power described in Table B-2 available for two-group comparisons, e.g., mean comparison between those arriving during the first half of the data collection period and those arriving during the second half, and for predictive correlations, e.g., explaining behavior in terms of traveler perceptions of trust. Using G*Power software (Faul et al., 2007), the team calculated the power available to detect various effect sizes for different types of tests, as shown in Table B-2, with a two-tailed alpha set at the traditional level of 0.05 for each type of test. As shown, the anticipated sample size should be sufficient to detect even modest effect sizes in the first follow-up data. Assuming attrition and a sample size of 768 for the second follow-up, there will be slightly lower power for the second telephone follow-up but there will still be relatively high power to detect modest effect sizes at that stage as well.   

Table B-2. Sample size calculations.

Test type (total sample size)	Effect size (g)	Power (1-beta)
Independent-sample t-test (960)
	.4	.999
	.3	.996
	.2	.871
Pearson correlation (960)
	.2	.999
	.1	.929
Independent-sample t-test (768)
	.4	.999
	.3	.986
	.2	.797
Pearson correlation (768)
	.2	.999
	.1	.877

The information collected through this assessment will be used to help refine interventions that are designed to enhance the travelers’ experience of entry screening experience and increase travelers’ initial uptake and participation in active monitoring for the full 21-day period. Finally, this information will be used to develop presentations, reports, and manuscripts to document the program and lessons learned in order to inform future programs of this sort.

2. Procedures for the Collection of Information

This traveler data collection involves three assessments at three time points: a) An initial in-person survey at the airport using the In-Person Survey (Attachment A), b) A follow-up survey conducted 3-5 days later over the phone using the Telephone Survey within 3-5 days of the airport (Attachment B), and c) a follow-up survey conducted 2 days before the end date for the traveler’s active monitoring and reporting period (Attachment C). The first telephone survey asks travelers to tell the interviewer when their end date is for active monitoring (it is a date that the health department gives them that is accordance with the health department’s active monitoring policy). Once collected, the computer-automated telephone interview system calculates when the next call should take place to allow them to actually experience active monitoring for a while yet still interview them before they finish. The data collection instruments are designed to assess traveler knowledge of Ebola, awareness of active monitoring, intention to participate in active monitoring, and initiation and retention in active monitoring. In addition, the instruments will provide insight into traveler comprehension of Check and Report Ebola (CARE) Kit messages and traveler perceptions of utility of CARE Kit materials.

Travelers who complete the secondary screening process are released by U.S. Customs and Border Protections and can leave the screening area. At this point, the CARE Ambassador encounter occurs, which lasts between 4-7 minutes depending on the traveler’s schedule (e.g., if they have to get to a connecting flight). After the encounter with the CARE Ambassador, the interviewer will approach the traveler and obtain verbal consent. The survey will be conducted in English or French depending on traveler’s preference using the “In-person Survey of Traveler Intercepted at the Airport” (see Attachment A). Survey responses will be entered by the interviewer using an Android tablet with the survey pre-programmed in it. Some open-ended responses will be audio recorded with the voice capture feature of the tablet. The survey is expected to last no more than 10 minutes. At the conclusion, the interviewer will ask the respondent if they are interested in participating in a follow-up survey over the phone in 3-5 days. If they agree to be called, the interviewer will collect the individual’s name, CARE ID number, and phone number. This information will be recorded separately from the responses to the survey in an electronic log. The log will assign a unique project tracking number to the survey and to the separately stored contact information. The tracking number will include the airport where the survey was conducted, the date the survey was conducted, and the participant number for the day (e.g., JFK.05-01.001).

The second data collection will be a follow-up phone survey 3 to 5 days after the traveler has received the CARE+ educational encounter with a CARE Ambassador after the airport screening process and after the in-person survey at the airport. The survey will assess Ebola knowledge, traveler initiation and retention in active monitoring. Data will be collected using the “Telephone Survey within 3-5 days of the airport survey” (Attachment B). Computer-assisted telephone interviewing (CATI) systems will be used by the contractor to attempt to reach the traveler by phone to either conduct the survey “on the spot” or schedule a time in the future, according to the respondent’s preference. The contractor will make up to 5 attempts to reach the respondent. This follow-up survey will take approximately 10 minutes. If the passenger agrees to be contacted for the second and final phone survey, the interviewer will confirm the traveler’s name, CARE ID number, and phone number, which is stored in the secure CATI system.

The third data collection will be a follow-up phone survey 2 days before the traveler’s scheduled end date for monitoring and reporting. The survey will assess Ebola knowledge, traveler initiation and retention in active monitoring. Data will be collected using the “Telephone Survey 2 days before the traveler’s end date for monitoring and reporting” (Attachment C). Computer-assisted telephone interviewing (CATI) systems will be used by the contractor to attempt to reach the traveler by phone to either conduct the survey “on the spot” or schedule a time in the future, according to the respondent’s preference. The contractor will make up to 5 attempts to reach the respondent. This follow-up survey will take approximately 5 minutes.

Up to 12 interviewers will collect data in the airports over the four month period. The contractor’s survey research call center, with a team of up to 10 interviewers, will conduct all of the telephone surveys. All are experienced survey interviewers and will be trained on these specific data collection procedures as well as the project and expectations. They will follow procedures as outlined and will meet with the project staff on a regular basis by conference calls. Any needed adjustments or corrections will be discussed in person, on regular conference calls, or via email.

After all surveyed travelers have completed their monitoring and reporting period, the evaluation team will request information from health departments on a monthly basis about aspects of active monitoring that the CARE+ program was intended to influence including: (1) date of arrival to jurisdiction; (2) date and time of first contact with the traveler; (3) end date of active monitoring; (4) preferred phone number for contacting and reporting (e.g., was it the CARE phone issued to them?); (5) did traveler ever report a fever or Ebola symptoms; (6) was contact with traveler ever lost for more than 48 hours; (7) did traveler complete monitoring and reporting requirements to the end of their reporting period; and (8) reason traveler did not complete monitoring and reporting requirements (e.g., transferred to another jurisdiction, left U.S., etc.). Health departments will be asked to provide this information on a monthly basis over a four month period using the “Data request to health departments” form (Attachment D).

For the cultural competency assessment of the Ebola CARE kit, up to 30 consultants who are from Ebola-affected countries and who have indicated an interest in being consulted by CDC staff will be contacted to participate in the assessment of materials for cultural competency. On December 5, 2014, CDC staff participated in the West African Diaspora Ebola Summit where the Ebola outbreak and response efforts were discussed. At the gathering, CDC had an exhibit booth that had a sign-up sheet for receiving partnership e-newsletters. On that sign-up sheet there was a question with a check box that asked summit participants if they would like to be contacted by CDC to help on future partnership efforts. CDC contacted all individuals who said yes to see if they would like to be consulted from time to time on messaging to specific audiences. CDC’s Joint Information Command (JIC) outreach team maintains this list and has engaged this group several times over the course of the Ebola response via email, telephone conference calls, and interactive webinars. Individuals from the list will be invited to use it to review CDC’s Ebola CARE Kit (Attachment F) using the cultural competency assessment tool (Attachment E), which we estimate to take 30 minutes. The participant will have the option to conduct their review using one of three formats depending on their preference: (1) pen and paper survey; (2) on-line survey; or (3) phone interview. The first 26 questions use a four point Likert-scale for response items from strongly agree to strongly disagree followed by one open ended question at the end. Insights will be used to refine the instrument.

3. Methods to Maximize Response Rates and Deal with No Response

The following procedures will be used to maximize response rates and deal with no response:

• Informing respondents of what the project is asking, why it is being asked, who will see the results, and how the results will be used, as well as discussing how respondents will benefit from the results and how the findings will be put into action.

• Addressing data security with respondents. Although we cannot guarantee anonymity, we will assure travelers that significant efforts will be made to secure data and that their answers will not be linked to them in any way. Additionally, participants will be informed that the survey is voluntary, and they are free to skip questions they do not wish to answer, respond “I don’t know,” or end the interview at any time for any reason.

• Outgoing calls that result in no answer, a busy signal, or an answering machine will be automatically rescheduled for subsequent attempts (up to five additional attempts will be tried before determining that a traveler is lost to follow-up).

4. Test of Procedures or Methods to be Undertaken

The estimate for survey burden hours is based on the burden hours from interviews conducted in the assessment of the CARE program the CARE Program is the precursor of the Ebola CARE+ program which added CARE Ambassadors in airports and the distribution of cell phones) completed in December 2014 (OMB Gen IC No. 0920-0932.) as well as pilot-testing of the revised instruments. Revised instruments have more close-ended questions with response options derived from the qualitative analysis of traveler responses to open-ended questions asked in the baseline interviews. In addition, questions were added to measure knowledge, beliefs, and behaviors that that the CARE+ program was intended to influence. The contractor pilot-tested the three brief surveys of travelers without the use of Android tablets or the CATI system in order to assess whether the surveys could be administered within the proposed time frames. The times recorded for the three pilot tests of each survey instrument were as follows:

• In-airport surveys were completed in 10, 12, and 13 minutes

• Telephone surveys within 3-5 days of airport interview were completed in 10, 13, and 15 minutes.

• Telephone surveys 2 days before reporting end date were completed in 4, 6, and 7 minutes

Because the instruments were new to the pilot-tester and because none of the planned technologies were used in the pilot test, the contractor and CDC believe that current estimates for survey administration are sound.

The burden for the monthly information request of health departments has not been tested but is based on state experience implementing the 21-day active monitoring program and reporting their findings to CDC.

To summarize, the evaluation team believes the average length of time to complete each of the surveys is as follows:

• In-person Survey of Traveler Intercepted at the Airport – 10 minutes

• Telephone Survey within 3-5 days of the airport survey – 10 minutes

• Telephone Survey 2 days before the traveler’s end date for monitoring and reporting – 5 minutes

• Data request to health departments – 240 minutes

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following individuals, provided advice about the protocol design, sampling methods, and data collection tools in alphabetical order by last name after the Principal Investigators at CDC (Christine Prue) and at RTI International (Brian Southwell).

Name (Last, First)	Title	Degree	Roles/Responsibilities
Prue, Christine	Associate Director for Behavioral Science, NCEZID	PhD, MSPH	Principal Investigator Primary in assessment design, data analysis, and outputs. Oversight of data collection.
Southwell, Brian	Director, Science in the Public Sphere Program, RTI	PhD, MA	Primary in assessment design, data analysis, and outputs. Oversight of data collection.
Alexander, Jennifer	Research Associate, RTI	MPH/MSW	Primary in assessment design, data analysis, and outputs. Leads data collection.
Joseph, Heather	Behavioral Scientist, NCHHSTP	MPH	Assists in assessment design, data analysis, and outputs.
Lee, Amanda	Acting, Associate Director for Training, Education, and Communication, DGMQ	MPH	Assists in assessment design, data analysis, and outputs
Raber, Anjanette	Evaluation Fellow, DHQP	PhD, RN	Primary in assessment design, data collection, data analysis, and outputs. Support in data collection.
Ray, Sarah	Research Associate, RTI	MA	Primary in assessment design, data analysis, and outputs. Supports data collection.
Williams, Peyton	Research Associate, RTI	BA	Primary in assessment design, data analysis, and outputs. Leads data collection.
Winter, Kelly	Training Specialist, NCEZID	MPH, PhD Candidate	Assists in assessment design, data analysis, and outputs.
Wojno, Abbey	Senior Training Specialist, NCEZID	PhD	Assists in assessment design, data analysis, and outputs.
Zulkiewitz, Brittany	Research Associate, RTI	BS	Primary in assessment design, data analysis, and outputs. Supports data collection.

SPSS software will be used to conduct all quantitative analyses. Descriptive statistics will be calculated for the entire sample. Dependent samples t-tests, Chi-square, and binary logistic regression will be used to examine relationships between independent and dependent variables at baseline and follow-up and to measure change between time points. To determine statistical significance, alpha will be set at the traditional level of 0.05.

The contractor will transcribe audio recordings captured on the Android tables and CATI-trained phone interviewers will write verbatim responses of travelers during phone interviews. The text responses will be uploaded to NVivo, ATLAS.ti, or MAXQDA. The goal of inductive thematic analysis is to make sense of core consistencies and meanings as they emerge from text in a way that retains the spirit the traveler’s response. To ensure findings are credible and trustworthy, this assessment will use the Consolidated Criteria for Reporting Qualitative Research (COREQ), a 32-item checklist for interviews and focus groups. Items in the COREQ checklist are grouped into three domains, Evaluation Team and Reflexivity-checklist, Assessment Design-Checklist, Analysis and Findings-Checklist and help ensure explicit and comprehension throughout the assessment process.

Reference

Faul, F., Erdfelder, E., Lang, A., & Buchner, A. (2007). G*Power 3: A flexible statistical power analysis program for the social, behavioral and biomedical sciences. Behavior Research Methods, 39, 175-191.

Sudman, S. (1976). Applied sampling. New York: Academic Press.

7