B. Statistical Methods
Respondent Universe and Sampling Methods
The primary outcome study will consist of a baseline survey and follow-up cross-sectional surveys with an embedded longitudinal cohort in campaign and control cities beginning approximately 6 months after campaign launch. The outcome evaluation data collection will occur in 12 campaign-targeted cities and 12 similar (“comparison”) cities. The 12 campaign and 12 comparison cities were randomly selected from a list of 24 potential campaign cities identified by FDA and their media contractor. We drew a stratified random sample of 12 cities to serve as comparison cities, with the remaining 12 cities selected as campaign markets. All 24 markets were initially grouped into regions. The primary method for this data collection involves conducting intercept interviews at LGBT social venues (e.g., bars and nightclubs). Eligible respondents will be young adults who are 18 to 24 years old and who self-identify as LGBT. The sample will include young adults who self-identify as LGBT, as well as young adults who self-identify as being queer, transsexual, gender variant or pansexual (also referred to as omnisexual or trisexual). We will screen potential respondents at these venues. Eligible participants will be invited to complete the outcome survey online (via the participant’s personal computer or mobile device). We will complement this strategy by recruiting LGBT young adults via social media from the same cities. Each wave of the three waves of follow-up data collection will consist of a sample size of 3,150, with half of the sample (N=1,575) from 12 campaign-targeted cities and half (N=1,575) from comparison cities. Our goal is to recruit 75% of the sample via intercept interviews and 25% via social media at baseline. Baseline respondents will be invited to complete follow-up surveys at 6-month intervals. We are assuming a 50% retention rate at each wave and will recruit additional participants at each follow-up wave to keep the sample size constant at 3,150. Once again, 75% of the newly recruited participants will come from intercept interviews and 25% from social media.
Field Data Collection
For the intercept surveys we will seek input from local LGBT Outreach Coordinators at Health Departments, and Rescue Social Change Group, and conduct online searches to identify LGBT bars and nightclubs. We will select a convenience sample of bars and nightclubs and particular dates and times within each sampled bar to maximize likelihood of intercepting the targeted number of participants per city within a brief field period within each city. As needed to establish contact with bar owners, recruitment calls to bar owners and managers will be supplemented by in-person contacts by local field staff. As agreed upon with the bar owners, a small team of interviewers will visit the bar at an established time and will approach patrons who appear to be in the target age range. Participants who agree to participate will be asked to fill out a short screening instrument (Attachment 3) on a tablet.
Data Collection Via Social Media
To supplement the field data collection strategy above, we will recruit a convenience sample of additional LGBT young adults through social media platforms, such as Twitter and Facebook. We will post advertisements to social media and invite young adults 18 to 24 to complete a brief online screener to determine their eligibility (i.e., LGBT, 18 to 24, and living in one of the 20 selected cities). Sample selection for respondents recruited from social media will involve posting advertisements on Facebook and Twitter for viewing by members more likely to be 18 to 24 years old and self-identify as a LGBT young adult, according to their account settings. The advertisements will be geographically targeted in these social media platforms to people living in the 20 selected cities. Eligible respondents will be directed to complete the outcome survey online until the required number of surveys is obtained.
Power Analysis
Statistical power estimates provide guidance on reasonable expectations for observing statistically significant change in outcomes of interest. This process requires an understanding of the study design, planned analyses (i.e., statistical model), expectations about the minimum detectable effect (MDE), as well as characteristics of the population and measures involved.
For the purpose of estimating statistical power for the [LGBT Campaign], we assume data collection will reflect a repeated cross-sectional design among 24 cities, with 12 [LGBT Campaign] cities and 12 cities that will serve as a comparison group. Although the planned data collection includes an embedded longitudinal design, for the purposes of the analyses and power calculations, we are treating each wave of data collection as an independent cross-section.
The proposed impact analysis accounts for the repeated cross-sectional data collection using a generalized linear hierarchical regression model that assesses change in the proportion of young adults that agree with a belief statement related to smoking tobacco (e.g., perceived approval, perceived prevalence, and perceived popularity). The test statistic will involve a two-tailed hypothesis test with a Type I error rate of 0.05 and a Type II error rate of 0.20, yielding 80% statistical power. Our parameter estimates include an intraclass correlation coefficient (ICC) of 0.01 to account for the geographic clustering of respondents and a variance inflation factor of 1.25 to account for potential imbalance across conditions. To some extent, these factors are offset by parameters that will serve to reduce variation. Those parameters include over-time correlation corrections of 0.55 at the cluster levels that account for repeated measures in the same cities as well as a 0.25 variance reduction at the individual level for the inclusion of demographic and socio-economic covariates. These parameter estimates are available in the published literature and support by our experience conducting similar studies (Murray & Short 1997; Murray & Blitstein 2003; Janega, Murray et al. 2004; Farrelly, Davis et al. 2005).
The campaign evaluation’s goal is to be able to identify change of 10 percentage points or greater as statistically significant. There is little available data in the peer-reviewed literature on the level of agreement we can anticipate at baseline. Accordingly, we rely on the conservative assumption that 50% of young adults will agree with campaign messages at baseline.
Given the parameters and assumptions detailed above, the impact evaluation of the [LGBT Campaign] will require data from 1,575 LGBT young adults in total in the 12 campaign cities and 1,575 LGBT young adults total in the 12 comparison cities (N = 3,150) at each point in time. This sample size is predicated on the assumption that agreement with campaign messages is 50% at baseline and increases to 60% at the time of follow up data collection. If actual agreement at baseline is either higher or lower than this value, statistical power is improved and smaller program impacts can be detected with the same sample of respondents. This effect would result in an odds ratio of approximately 1.50, meaning that young adults exposed to the campaign would be 1.50 times more likely than young adults not exposed to the campaign to agree with campaign messages about the effects of smoking tobacco.
Procedures for the Collection of Information
2.1 Field Data Collection
This section describes the procedures for in person data collection. Data will be collected in 4 waves beginning in late 2015, prior to the launch of the campaign, and ending 18 months post campaign launch.
Recruitment
RTI staff will attempt to recruit bars for data collection following the Bar Recruitment Guide included as Attachment 7. As needed to establish contact with bar owners, these recruitment calls will be supplemented by in-person contacts by local field staff using the same Recruitment Guide. Calls will be made (and in person visits, when possible) to the list of bars identified with input from local LGBT Outreach Coordinators at Health Departments, Rescue Social Change Group and online searches. Bar owners will be first be given brief background information about data collection procedures, they will then be asked questions about when and how often LGBT young adults visit their establishments and be provided with information about participant cash incentives (as a benefit of allowing for recruitment in bars). RTI staff will then confirm recruitment times with bar owners who agree to allow recruitment in their establishment.
As agreed upon with bar owners, a small team of interviewers will visit the bar at the established time and will approach patrons who appear to be in the target age range using the talking points and FAQs shown in Attachment 8. RTI staff will introduce themselves and provide the following information to potential participants: incentive amount for completing screener survey ($10 cash), time needed to complete survey (5 minutes), and incentive amount for emailed web survey ($20-25 depending on how quickly they respond). In addition, RTI staff will provide the following information based on questions asked by potential participants: study purpose, reasons for participating, study sponsor information, background information on RTI and FDA CTP, information on privacy and confidentiality, and IRB contact information.
Screening
Once a participant agrees to complete the screener, the interviewer will open the case, select the appropriate city and bar, advance the screener to the informed consent screen (Attachment 4) and then pass the tablet to the respondent to complete. Within the screener, respondents will be asked about their zip code, age, sexual identity and sexual orientation (all necessary to establish eligibility including LGBT status, being within the appropriate age range and living in the recruitment city). Respondents will also be asked two questions about their tobacco use; these tobacco questions are included to allow comparison of eligible respondents who do and do not later participate in the full survey to facilitate nonresponse bias analysis.
Completed screeners will be automatically analyzed within the survey software to determine the young adult’s eligibility. Respondents who screen as eligible will be asked for their email address, cell phone number, and first name so we can confirm we haven’t previously enrolled this person and so we can send an invitation for the full web-survey to those who are eligible and new to the study. GPS data will be associated with each screened case to assist in detecting possible falsification by interviewers. Respondents will be paid $10 in cash and will be asked to initial an incentive receipt (Attachment 9) at the end of the screener. Respondents who screen as eligible will be provided with an information card (Attachment 10) advising them that they will receive an email or text message invitation for the full web survey within the next two days.
Incentives
Intercept respondents will be paid $10 in cash for completing the screener and will be asked to initial an incentive receipt (Attachment 9) at the end of the screener. Those who are eligible will receive an email invitation to complete the full survey. If they choose to complete the full survey they will receive an online gift card of $20.00, with a $5 bonus (total of $25.00) if they complete the full survey within two days of invitation.
Eligible participants who are recruited via social media (procedure described in section B.2.2 below) will complete the screener integrated with the web-survey and will be compensated $20 via an online gift card. Ineligible participants will not be compensated for completing the short screener. We will only be compensating screener participants who participate in LGBT recruitment venues because we need to provide an additional incentive for these participants to complete the survey using the link that they receive via email at a later time. Participants who screen in as eligible via social media ads will proceed immediately to the survey and thus will not need an additional incentive to complete the survey.
Data Security
All interview data recorded on tablets will be transmitted at least daily via secure encrypted data transmission to RTI’s offices, where the data will be subsequently processed and prepared for analysis, reporting, and data file delivery. Upon transmission to RTI, all survey item data will be automatically wiped from all data collection devices used in the field.
At follow-up, young adults who participated in prior waves of data collection will be re-contacted using a follow-up email (Attachment 11). All other data collection procedures relating to the embedded longitudinal cohort will be the same as those described above.
2.2 Recruitment Via Social Media
To supplement this sample, young adult respondents will also be recruited in 24 U.S. cities (12 campaign and 12 comparison cities) through social media advertisements on Facebook and Twitter targeted at LGBT 18 to 24-year-olds, living in the same 24 U.S. cities.
RTI will place ads on social media platforms Twitter and Facebook. Several examples of these ads are included in Attachment 12. As much as possible, these ads will be targeted toward potentially eligible respondents, who are thought to be 18 to 24 years old, live in the data collection cities, and potentially may self-identify as LGBT. When clicked, the ads will direct the potential participant to a web-based screener instrument (Attachment 3). Respondents who are deemed eligible following completion of the screener will then go on to provide consent (Attachment 6) and complete either the baseline or follow-up instrument (Attachments 1 and 2), which will be administered online. All respondents who complete this survey will receive a virtual gift card valued at $20. Participants recruited via social media will not be paid to complete the screener.
Methods to Maximize Response Rates and Deal with Nonresponse
The ability to obtain the cooperation of potential respondents in the baseline survey and maintain their participation across all survey waves will be important to the success of this study. In preparation for launching the baseline data collection, we will review procedures for enlisting respondent cooperation across a wide range of surveys, incorporate best practices from those surveys into the data collection procedures, and adapt the procedures through continuous improvement across the survey waves.
In addition to the $20 incentive (with $5 bonus for intercept participants who complete the survey within 2 days of receipt of the survey link) and $10 incentive for intercept participants to complete the screening survey, the study will use procedures designed to maximize respondent participation. The incentive procedures and amounts for new cross-sectional sample at follow-up waves will be identical to the baseline survey. For the longitudinal sample, participants will receive $20 for completing the follow-up survey and an additional $5 if they complete the survey within 48 hours of the invitation to participate. Prior intercept studies have demonstrated the importance of careful recruitment and training of field staff. Matching the characteristics of the interviewers to the study population has been found to be helpful (Spooner. et. al, 1997). To the extent possible, field interviewers will recruited from the LGBT community within each city. The ideal candidate will be familiar with the venues we will be using for data collection and knowledgeable about times in which 18 to 24 year olds are most likely to be present in high numbers. In addition, the ideal candidate will be reliable, articulate, outgoing, confident, and non-judgmental. Interviewer training will be participatory allowing for ample time to practice approaching respondents to introduce the survey, answer common questions, and overcoming objections. As noted in Section B.2, interviewers recruiting bar/nightclub intercept participants will use a document that includes talking points and FAQs (Attachment 8) to encourage participation.
When interviewers transmit their data from completed intercept screenings, the data will be summarized in daily reports posted to a Web-based case management system accessed by field supervisors and RTI’s data collection managers. On a daily basis, supervisors will use these reports to review response rates and production levels. This information will allow supervisors to determine progress toward production goals and adjust goals for the remaining venues within each city. Supervisors will discuss information and challenges with their interviewers each week.
Test of Procedures or Methods to be Undertaken
RTI will conduct rigorous internal testing of the online survey instrument prior to data collection. Evaluators will review the online test version of the instrument that we will use to verify that instrument skip patterns are functioning properly, delivery of campaign media materials is working properly, and that all survey questions are worded correctly and are in accordance with the instrument approved by OMB.
Individuals Consulted on Statistical Aspects and Individuals Collection and/or Analyzing Data
The following individuals inside the agency have been consulted on the design of the campaign evaluation plan, audience questionnaire development, or intra-agency coordination of information collection efforts:
Tesfa Alexander
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 301-796.9335
E-mail: [email protected]
Lucinda Miner
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 301-796-9181
E-mail: [email protected]
Gem Benoza
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-0088
E-mail: [email protected]
David Portnoy
Office of Science
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 301-796-9298
E-mail: [email protected]
Matthew Walker
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-3824
E-mail: [email protected]
Leah Hoffman
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-743-1777
E-mail: [email protected]
Janine Delahanty
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-402-9705
E-mail: [email protected]
The following individuals outside of the agency have been consulted on questionnaire development. Additionally, input has been solicited and received from FDA on the design of this study, including participation by FDA in meetings with OMB:
Matthew Farrelly
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-6852
E-mail: [email protected]
Jennifer Duke
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-2269
E-mail: [email protected]
Jamie Guillory
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-316-3725
E-mail: [email protected]
Kristine Fahrney Wiant
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-5531
E-mail: [email protected]
Jane Allen
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-597-5115
E-mail: [email protected]
Youn Lee
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-485-5536
E-mail: [email protected]
Amy Henes
RTI International
701 13th Street, NW
Washington, DC, 20005
Phone: 202-974-7821
E-mail: [email protected]
Patricia LeBaron
RTI International
230 W Monroe Suite 2100
Chicago, IL 60606
Phone: 312-777-5204
E-mail: [email protected]
Pamela Rao
Akira Technologies, Inc.
1747 Pennsylvania Ave NW Suite 600
Washington, DC 20002
Phone: (202) 517-7187
Email: [email protected]
Xiaoquan Zhao
Department of Communication
George Mason University
Robinson Hall A, Room 307B
4400 University Drive, 3D6
Fairfax, VA 22030
Phone: 703-993-4008
E-mail: [email protected]
Joseph Lee
East Carolina University
Belk Building
Greenville, NC
Phone:
E-mail: [email protected]
Jeff Jordan
Rescue Social Change Group
3436 Ray Street
San Diego, CA 92104
Phone: 619-231-7555 x 150
Email: [email protected]
Mayo Djakaria
Rescue Social Change Group
660 Pennsylvania Avenue SE, Suite 400
Washington, DC 20003
Phone: 619-231-7555 x 120
Email: [email protected]
Dana Wagner
Rescue Social Change Group
660 Pennsylvania Ave SE, Suite 400
Washington, DC 20003
Phone: 619-231-7555 x 331
Email: [email protected]
References
Farrelly, M.C., Davis, K.C., Haviland, M.L., Messeri, P., Healton, C.G. (2005). Evidence of a dose-response relationship between "truth" antismoking ads and youth smoking prevalence. American Journal of Public Health, 95(3), 425-431.
Janega, J.B., Murray, D.M., Varnell, S.P., Blitstein, J.L., Birnbaum, A.S., Lytle, L.A. (2004). Assessing the most powerful analysis method for schools intervention studies with alcohol, tobacco, and other drug outcomes. Addictive Behaviors, 29(3), 595-606.
Murray, D.M., Blitstein, J.L. (2003). Methods to reduce the impact of intraclass correlation in group- randomized trials. Evaluation Review, 27(1), 79-103.
Murray, D.M., Short, B.J. (1997). Intraclass correlation among measures related to tobacco- smoking by adolescents: Estimates, correlates, and applications in intervention studies. Addictive Behaviors, 22(1), 1-12.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Matthew Farrelly |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |