OMB Control No. 0910-0810
Exp. Date: 12/31/2024
U.S Food and Drug Administration
Center for Tobacco Products
Increasing Understanding of Digital Advertising:
Awareness, Receptivity, and Comprehension (ARC) Experiment
Supporting Statement: Part B
B. Statistical Methods
1. Respondent Universe and Sampling Methods
The respondent universe for this study is young adults aged 18-24 living in 9 randomized media markets across the United States and a control area where advertising is not aired.
The study will use convenience samples rather than probability samples. As this is an experimental study, we do not intend to generate nationally representative results or precise estimates of population parameters from the study. The screener survey will include questions assessing race, ethnicity, gender identity, and location.
Sampling Methods
The study sample will include young adults aged 18 to 24-years in 9 media markets in the United States and a control area where advertising will not air. The study involves a randomized field experiment where the dose of media (i.e., low (L), medium (M), or high (H) dose) and mix between social media and streaming media (i.e., 20%, 40% or 60% of campaign assets devoted to social media) will be randomly assigned to the 9 markets. The primary purpose of the study is to understand the dose-response relationship between digital media advertising and self-reported awareness. A secondary and related purpose is to understand the relative effectiveness of social media and streaming media in promoting audience awareness of hookah education video advertisements. This study is considered formative research for campaign development and planning; these methods are not intended to generate nationally representative samples or precise estimates of population parameters.
Participants will be recruited by an online survey vendor, Dynata, that manages a national online panel of adults and through targeted social media advertising (i.e., Facebook, Instagram). Approximately half of the sample will be recruited from Dynata’s panel and the remainder from social media. Participants will be directed to the online screener which will take approximately 3 minutes to complete. Screener instructions note that respondents may only complete the survey one time. The screener does not include questions that are especially sensitive or that we anticipate will be upsetting to those who read or complete it. The primary purpose of the online screener is to screen out participants that are not eligible to complete the online survey. If individuals do not complete the screener, they will not be invited to be in the study. Individuals may decline to participate in the online screener if they wish. The social media platforms will not share any other information about their account with RTI International (RTI), the organization conducting the survey on behalf of the U.S. Food and Drug Administration (FDA).
Sample Size
Based on previous similar recruitments RTI has conducted, we estimate that to obtain a final sample of 4,035 participants aged 18 to 24-years, we will need to screen approximately 12,105 potential participants. Burden calculation assumes up to 4,435 participants because it is possible that recruitment will not close the instant the target sample is obtained. Portions of the sample will be collected at 4 and 8 weeks after the launch of the experiment.
Justification for collecting sexual orientation and gender identity (SOGI) data
The FDA is committed to advancing health equity for LGBTQ+ youth. In accordance with the White House issued the Executive Order on Advancing Equality for Lesbian, Gay, Bisexual, Transgender, Queer, and Intersex Individuals, this study is one way in which FDA can meet its mission to protect youth, improve public health, and advance health equity for all Americans. Collecting data on the health and health behaviors (e.g., tobacco use) of LGBTQ+ youth is crucial to advancing health equity and reducing health disparities in these populations. Decades of research has shown significant disparities in tobacco use by gender identity (e.g., Johnson et al, 2019; Delahanty et al, 2019), and sexual orientation (e.g., Johnson et al, 2019; McCabe et al., 2018). To ensure that future tobacco public education campaigns effectively reach youth demographic groups experiencing tobacco use disparities (e.g., LGBTQ+ youth), it is necessary to collect data on these populations. These data will be used to inform tobacco public education campaigns that aim to reduce tobacco use disparities, including among LGBTQ+ populations.
Sexual orientation survey item. The sexual orientation item conforms to the items approved for the Monthly Monitoring Study (OMB control number 0910-0914) and the Hookah Beliefs Survey (OMB control number 0910-0810).
Gender identity survey item. The gender identity item conforms to sample gender identity items in the White House Office of the Chief Statistician guidance (Office of the Chief Statistician of the United States, 2023), which recommends including response options such as “non-binary” or “another gender identity” along with “male,” “female,” and “transgender.”
2.
Procedures for the Collection of Information
This section provides an overview of the study procedures, provides information on the degree of accuracy required for the study, and discusses the estimation procedures. There are no unusual problems requiring specialized sampling procedures.
2a. Study Procedures
Research Question
This study aims to answer the following research question: How does message dose, duration, and media mix (social and over-the-top [OTT]/connected television [CTV]) influence self-reported message awareness, comprehension, and receptivity?
Study Design and Protocol
To address the research question above, we will conduct a 3x3 experimental study to assess their individual and combined influence on self-reported message awareness, receptivity, and comprehension (assessed via surveys). We will assess the influence of duration by conducting repeated cross-sectional surveys at 4 and 8 weeks following the launch of the experiment to determine how awareness and other outcomes change as time progresses. This experiment will take place in 9 treatment markets and one control area. We will deploy digital advertising on OTT/CTV and social media advertising in these markets and conduct surveys at the campaign midpoint and end to measure ad awareness and receptivity, message comprehension, and related beliefs.
Table 1. ARC Experimental Design Framework for Testing Effects of Dose and Media Mix on Self-Reported Outcomes
Dose/Share for Social Media |
20% Social Media |
40% Social Media |
60% Social Media |
Low |
L20 |
L40 |
L60 |
Medium |
M20 |
M40 |
M60 |
High |
H20 |
H40 |
H60 |
Participants will be recruited through social media ads or through Dynata’s online panel. Young adults who click on the recruitment link via the social media ad or are sent the link via Dynata’s online panel will proceed to a Qualtrics survey that begins with a screener consent and screener questions. If eligible, they will proceed to an online consent form. Those eligible will be sent directly to the full online survey, which will take approximately 15 minutes to answer. Participants recruited via social media who complete the survey will receive a $5 token of appreciation at the email address they provide. RTI will process the incentives by sharing the provided email addresses (via a password-protected Excel document) with the vendor Creative Group Inc. Participants recruited via Dynata’s online panel will receive an approximately equivalent reward through the vendor. RTI will pass through a unique ID provided by Dynata when the participant enters the screener. Once the participant completes the survey in full, RTI will re-direct the participant back to Dynata passing only the unique ID to ensure that they receive their reward for completing the survey.
For the purposes of estimating statistical power, we assumed a sample size of 4,035 participants, with a sample of 315 in each of the 9 media markets and 1,200 in a control sample from non-adjacent media markets. This sample size should give us sufficient power to detect a 7-percentage point difference between doses (i.e., control (no dose), low, medium, and high) when we collapse across the social media conditions (creating N=945 for low, medium, and high doses). That will also give us sufficient power (80%) to detect differences of about 7 percentage points between the social media/streaming media mixes after collapsing the dose conditions (N=945 per media mix condition) (note that power calculations were made with self-reported awareness as the outcome, meaning that we expect a 7-percentage point difference in awareness).
Statistical analyses will be conducted to address the study’s primary research questions. We will conduct descriptive and multivariable analyses. First, we will conduct chi square tests of association for tests of differences in self-reported ad awareness by media dose using two awareness measures—dichotomized awareness of the ad and frequency of awareness (i.e., never, rarely, sometimes, often, very often). We will repeat this analysis, testing for differences in the two ad awareness outcomes by media mix. We will repeat these tests with multivariable analyses using a logistic regression for dichotomous awareness and multinomial logistic regression for frequency of awareness. We will test for significance of individual-level sociodemographic variables in the multivariable analyses to account for potential differences in the distribution of demographic characteristics across media markets. Following the recommendation of the FDA’s Center for Tobacco Products’ (CTP’s) Office of Science, analytic procedures will be based on nonparametric tests of statistical significance.
The ability to obtain the cooperation of potential respondents in the survey will be important to the success of this study.
RTI will minimize the non-response rate by employing the following measures:
Use targeted social media advertising and multi-sourcing panel recruitment to best reach the desired sample,
Provide a token of appreciation to participants who complete and submit the survey. Participants recruited through social media will receive a $5 digital gift card from Amazon and participants recruited through Dynata’s online panel will receive panel points.
Complete survey programming testing and a soft launch to monitor for possible technical difficulties with the survey to minimize potential dropout rates due to user issues.
We will use targeted advertising on social media (i.e., Facebook, Instagram). We estimate that the survey will take 15 minutes to complete, which we intentionally aimed for to keep the survey at a reasonable length to minimize non-completion. As a token of appreciation, eligible participants (determined by completing the screener) recruited through social media who complete and submit the survey will receive a $5 digital gift card from Amazon, and participants recruited through the online panel will receive panel points. As participants often have competing demands for their time, a token of appreciation for participation in research is warranted. The use of a token of appreciation treats participants justly and with respect by recognizing and acknowledging the effort participants expend to participate.
4. Tests of Procedures or Methods
RTI will conduct internal testing of the online survey instrument prior to fielding. Survey testers will review the online test version of the instrument that we will use to verify that instrument skip patterns are functioning properly. Based on skip pattern testing and instrument length, some items may be excluded. Survey testing will aid survey clarity (e.g., address typos and decrease confusing wording) and will ensure that all survey questions are in accordance with the instruments approved by OMB and IRB.
5. Individuals Involved in Statistical Consultation and Information Collection and/or Analyzing Data
The following individuals inside the agency have been consulted on the design of the study plan, questionnaire development, or intra-agency coordination of information collection efforts and plans for data analysis:
Elizabeth Petrun Sayers
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-750-5297
E-mail: [email protected]
Lindsay Pitzer
Office of Health Communication & Education
Center for Tobacco Products
Food and Drug Administration
10903 New Hampshire Avenue
Silver Spring, MD 20993
Phone: 240-620-9526
E-mail: [email protected]
The following individuals outside of the agency have been consulted on questionnaire development and/or will be collecting and/or analyzing data:
Jamie Guillory
RTI International
64 Dame Street
Dublin D02RTY72, Ireland
Phone: 919-695-9264
Email: [email protected]
Kim Hayes
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-1215
E-mail: [email protected]
Jessica Sobolewski
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-541-6657
Email: [email protected]
Vaughn Armbrister
RTI International
3040 Cornwallis Road
Research Triangle Park, NC 27709
Phone: 919-248-8521
Email: [email protected]
References
Delahanty, J., Ganz, O., Hoffman, L., Guillory, J., Crankshaw, E., & Farrelly, M. (2019). Tobacco use among lesbian, gay, bisexual and transgender young adults varies by sexual and gender identity. Drug and alcohol dependence, 201, 161-170.
Johnson, O’Brien, E. K., Coleman, B., Tessman, G. K., Hoffman, L., & Delahanty, J. (2019). Sexual and Gender Minority U.S. Youth Tobacco Use: Population Assessment of Tobacco and Health (PATH) Study Wave 3, 2015–2016. American Journal of Preventive Medicine, 57(2), 256–261. https://doi.org/10.1016/j.amepre.2019.03.021
McCabe, S. E., Matthews, A. K., Lee, J. G., Veliz, P., Hughes, T. L., & Boyd, C. J. (2018). Tobacco use and sexual orientation in a national cross-sectional study: age, race/ethnicity, and sexual identity–attraction differences. American journal of preventive medicine, 54(6), 736-745.
Office of the Chief Statistician of the United States. (2023). Recommendations on the best practices for the collection of sexual orientation and gender identity data on federal statistical surveys. https://www.whitehouse.gov/wp-content/uploads/2023/01/SOGI-Best-Practices.pdf
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Chapman, LeTonya |
| File Modified | 0000-00-00 |
| File Created | 2023-08-28 |