Ssb

SUPPORTING STATEMENT FOR THE

New

Emerging Tobacco Products Communication Initiative

Pre-testing of Ads

(OMB No. 0920-0910, Exp. Date 1/31/2024)

Part B: STATISTICAL METHODS

Submitted by:

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

Department of Health and Human Services

Refer questions to:

Program Official/Contact

Sarah Lewis, MPH, CHES

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

4770 Buford Highway, NE MS F-79

Atlanta, Georgia 30341

770-488-5582

FAX: 770-488-5939

Email: [email protected]

July 20, 2021

TABLE OF CONTENTS

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information

3. Methods to Maximize Response Rates and Deal with No response

4. Tests of Procedures or Methods to be Undertaken

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

LIST OF ATTACHMENTS

Attachment 1: Focus Group Recruitment Screener

Attachment 2: Limelight Privacy Policy

Attachment 3: Focus Group Consent Form

Attachment 4: Focus Group Advance Logistics Letter

Attachment 5: Focus Group Moderator Guide

Attachment 6: Survey Questionnaire

Attachment 7: Lucid Privacy Policy and Terms of Service

Attachment 8: Survey Questionnaire Email Invitation

Attachment 9: Survey Questionnaire Eligibility Screener

Attachment 10: Survey Questionnaire Consent Form

Attachment 11: Survey Questionnaire Sample Screenshots

Attachment 12: RTI International Institutional Review Board (IRB) Approval

Notes on Excluded Attachments. In this generic information collection request (under the existing generic clearance package #0920-0910), CDC outlines a plan to test ads with content that may be considered sensitive. The draft ads are not included because the ads have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA). To support adequate review of this GenIC by OMB, Centers for Disease Control and Prevention requests permission to provide OMB with a secure link to the draft ads.

Part B: Statistical Methods

B.1 Respondent Universe and Sampling Methods

This is a request for a mixed methods (qualitative and quantitative) data collection. In this GenIC, Centers for Disease Control and Prevention (CDC) requests OMB approval to collect information for testing of draft ads to inform the development of a future Emerging Tobacco Products Communication Initiative. The objective of this proposed information collection is to pre-test ads among educators of middle and high school students in the U.S. in a variety of geographical settings (e.g., urban, suburban, and rural regions). Participants’ roles will include: (1) teachers, (2) coaches, and (3) administrators (e.g., principals, assistant principals, guidance counselors). This testing will help identify the ads that best motivate educators to talk with youth about the importance of avoiding or quitting e-cigarettes and ensure the ads are credible, persuasive, and clear. RTI International will conduct the data collection with support from two recruitment firms—Limelight Insights and Qualtrics. RTI will lead the analysis for both the qualitative and quantitative portions of this proposed project.

Qualitative: Web-based Virtual Focus Groups. The goal is for six to eight participants each to participate in eight web-based virtual focus groups during the qualitative portion of the project. During each focus group, three creative themes (each including a written description, a mood board, and a sample ad in the form of a static storyboard) will be tested to inform the development of future ads using both moderated group discussion and online polls assessing creative theme preferences. Approximately 90 participants will need to be screened in order to recruit up to 64 people for the study (using the screener in Attachment 1), of which 48 to 64 will participate in the focus groups (depending on the number of cancellations and no-shows). Focus groups will be conducted remotely using the ZOOM™ web-based conferencing platform. Focus groups will be segmented by role and school type: 12-16 high school teachers and/or coaches, 12-16 middle school teachers and/or coaches, 12-16 high school administrators, and 12-16 middle school administrators. Focus groups will also aim to achieve the diversity quotas in Table 1 to the extent possible given the use of a convenience sample.

Table 1. Target Enrollment Quotas for Focus Groups

Priority	Characteristic	Target Enrollment Quota
1	Coaching Role	At least 2 participants per group who currently serve as a coach.
2	Professional Experience	At least 20% (10 participants) with less than 5 years of experience as an educator. At least 20% (10 participants) with 5-10 years of experience as an educator. At least 20% (10 participants) with more than 10 years of experience as an educator.
3	School Resource Availability	At least 40% of sample (20 participants) working at a school with limited economic resources, determined based on national or state ranking.1
4	Geography	At least 12 participants who work in a school that is in a rural area. At least 12 participants who work in a school that is in a suburban area. At least 12 participants who work in a school that is in an urban area.2
5	Race	At least 20% non-White (10 participants) in sample.3
6	Ethnicity	At least 20% Hispanic (10 participants) in sample.3
7	Gender Identity	At least 20 participants who identify as males. At least 20 participants who identify as females. Include those who identify as non-binary..4

¹Based on respondent’s answer to recruitment screener question about whether their school is a Title I school or a school with limited resources.

²Based on the US Census Bureau 2010 Census Urban and Rural Classification and Urban Area Criteria (source).

³Participants may be double counted for Race and Ethnicity quotas.

⁴Based on the Fenway Institute and Center for American Progress gender identity question design methodology (source).

Screening will continue until the target number of participants is recruited (64 people recruited with a minimum of six and maximum of eight participants for each of the eight focus groups). These 64 participants will be purposively sampled based on the following criteria, which are included in the focus group eligibility screener (Attachment 1):

• Employed as a teacher (includes teachers who also serve as coaches) or administrator (e.g., principal, vice principal, guidance counselor)

• Works in a middle school or high school

• 50% or more of professional time is spent in direct contact with students (or if less than 50%, participant oversees school operations, such as curriculum development or school policy)

• Before COVID-related school closures, works on-site at a school or facility with students

• Speaks and reads English fluently

• U.S. resident

The sampling procedures for qualitative activities are summarized in Figure 1 below.

Figure 1. Focus Group Sampling Procedures

Quantitative: Online Questionnaire. The sample of respondents for the quantitative data collection will be drawn from Lucid, an existing, online, convenience panel available to Qualtrics (see https://luc.id/legal/privacy-policy/ for more detail on this panel). Qualtrics employs a double opt-in process for individuals to participate in a survey: they must opt-in to become panelists, and they must also opt-in to each survey. The panelists will be screened using the online questionnaire eligibility screener (Attachment 9) to screen out persons who do not live in the U.S. and who do not work directly with middle or high school students as a teacher, coach, or administrator. Although the sample will be a convenience sample, the panel sample has a reasonable degree of diversity in key demographic characteristics, such as age, gender, race/ethnicity, and geography/region of residence. As the quantitative portion of the project is considered part of formative work for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. However, the design allows for high internal validity to provide information on the perceived effectiveness of the ads being tested.

The online survey will be used to pre-test four draft ads among educators, who will view the ads and react to questions on key measures, including perceived effectiveness (PE) believability, visual design, and relevance. A power analysis was run to determine the sample size needed to detect statistically significant differences on key measures (e.g., average perceived effectiveness scores for each ad). Given our analytical goals for the quantitative survey, we have planned for a sample size of 600 completed participants (n=200 teachers, n=200 administrators, and n=200 coaches). All 600 survey respondents will review all four ads, although the order in which the ads are presented to respondents will be randomized. This will provide us with sufficient statistical power (0.80) to detect small- to medium-sized effects in our primary and secondary analyses using a p-value threshold of 0.05.

Our primary analyses will examine (1) differences in ad ratings by type of ad and (2) differences in ad ratings by educator role (i.e., teacher vs. administrator vs. coaches), which is our primary stratification variable.

Ad Rating Differences by Type of Ad. For the first analysis, we will conduct one-way within-subjects (i.e., dependent sample) ANOVAs for each outcome variable followed by pairwise comparisons using dependent sample t-tests. This will provide us with 0.80 power to detect small effect sizes (f = 0.048 and d = 0.115).

Ad Rating Differences by Educator Role. For the second analysis, we will conduct one-way between-subjects (i.e., independent sample) ANOVAs for each outcome variable followed by pairwise comparisons using independent sample t-tests. This will provide us with 0.80 power to detect small effect sizes (f = 0.127 and d = 0.28).

Secondary Analyses. Our secondary analyses will examine differences in ad ratings by key demographic and professional variables, including school type (middle vs. high), school geography (urban vs. suburban vs. rural), gender (male vs. female), race (White vs. non-White), and ethnicity (Hispanic vs. non-Hispanic). For all of these analyses, we will conduct either (1) one-way between-subjects ANOVAs followed by pairwise comparisons using independent sample t-tests or (2) independent sample t-tests. For ANOVAs, we will have 0.80 power to detect small effect sizes (f = 0.147). For t-tests, we will have 0.80 power to detect small- to medium-sized effects (d = 0.229 to d = 0.363).

To achieve this sample size of 600, we conservatively anticipate screening 9,000 individuals; this estimate is based on two factors from prior experiences in the field. First, it is anticipated that roughly 84 percent of screener respondents (n=7,570) will be deemed ineligible for the proposed project because of not meeting inclusion criteria by not having direct contact with students or not meeting our segment criteria (e.g., working as a coach). In addition, another 800 individuals are likely to be excluded because, although eligible, they do not meet the necessary demographic or professional quotas (e.g., geographic location, race). Second, of those deemed eligible and invited to participate (n=630), an estimated five percent (n=30) will start but not complete the questionnaire. Thus, 9,000 respondents are needed to obtain 600 in the final sample. The sampling procedures for quantitative activities are summarized in Figure 2.

Figure 2. Survey Sampling Procedures

B.2 Procedures for the Collection of Information

The data collection contractor, RTI International, will be responsible for coordinating data collection activities, collecting, and summarizing information, and preparing topline and final reports. Information for this proposed project will be collected using qualitative and quantitative methods. The qualitative testing will provide information as to which creative theme (presented as a text description, a mood board, and a sample ad in the form of storyboard) is most clear, credible, and motivating. The quantitative testing will collect information about the respondents’ reactions to the ads, as well as basic demographics to assess whether responses to ads differ by subpopulation. The procedures are described below in the following sections.

Qualitative: Web-based Virtual Focus Groups

Recruitment and Screening

Potential focus group participants will be contacted and screened using the focus group eligibility screener (Attachment 1), and if eligible, invited to participate in the focus groups. RTI International (RTI) will be responsible for recruiting respondents for the focus groups using a combination of direct outreach to schools and postings to professional organization listservs, newsletters, and social media sites. For direct outreach, RTI will identify a subset of middle and high schools in urban, suburban, and rural areas within most states, will contact schools using publicly available contact information on school websites, and will ask them to disseminate study information to educators at the school. Educators who are interested in the study will be directed to an online survey hosted by Limelight Insights, who will review and invite eligible individuals using the eligibility screener (Attachment 1) provided by CDC.

The screener will collect information on age, gender, race/ethnicity, years of experience, job type/role, school type, and geography. As noted previously, focus groups will aim to meet specified diversity quotas (Table 1). Participants will be segmented according to their role at the school and their type of school (middle or high school). Each eligible participant will be scheduled to attend a focus group corresponding to their role and school type: two focus groups for middle school teachers and coaches; two focus groups for high school teachers and coaches; two groups for middle school administrators; and two groups for high school administrators.

Focus Group Participation

Prior to beginning the focus group, all participants will be required to submit a signed consent form (Attachment 3) to confirm their willingness to participate in the study. We will also provide an advance logistics letter (Attachment 4) with login credentials for joining the focus group. A trained moderator will lead the focus group discussion; discussions will follow the semi-structured moderator’s guide (Attachment 5). During the focus group, discussion questions will explore participants’ perceptions of relevance, comprehension, self-efficacy, motivation, and visual design for each creative theme, presented in the form of a text description, a mood board, and a sample ad. After providing feedback on each theme and discussing them as a group, participants will be asked to answer a poll assessing their favorite and least favorite themes, followed by a group discussion comparing them. This integrated ZOOM™ poll about ad preferences is necessary because it allows for rapid collection of personal preferences (versus spending time having each person verbally “vote”); in addition, the responses to the poll will help the moderator focus the conversation and probe on emergent themes (e.g., The group generally preferred Sample Ad A, why did they prefer that ad over Sample Ads B and C? What about that sample ad makes them more likely to click and learn more?). The poll does not include measures that will be statistically analyzed and reported across groups.

The focus group discussions, which are anticipated to last 90 minutes, will be moderated using ZOOM™, a high-definition video conferencing service. CDC and key staff from RTI (researchers), Golin (content developers), and Limelight Insights (recruiter) will be able to remotely observe the focus group discussions through a secure link to the online session. During the focus groups, the moderator will have the ability to receive messages from researchers based on the discussion, and researchers will be able to “listen in” to the conversation. RTI International will retain focus group recordings and de-identified transcripts on a password-protected database for six months, after which all video files will be deleted. Project staff from RTI International will have access to the focus group recordings and de-identified transcripts. CDC and their creative agency contractor, Golin, will have access only to the de-identified transcripts, which will be handled in accordance with the record control schedule (maintained at least six years, but no longer than ten years).

Quantitative: Online Questionnaire

Recruitment and Screening

The quantitative phase of data collection will include respondents recruited from one of Qualtrics’ survey panels, which uses an established, online panel system that includes more than 100 million people in the U.S. Qualtrics has profiled their panels regarding demographics and can target and identify respondents who are pre-identified as being potentially eligible for the survey. However, additional screening will be conducted to confirm that Qualtrics’ profiling information is current and to assess whether any information has changed (e.g., age, region of residence/geography). To protect potentially identifiable information (PII) of respondents, no comparison to the original individual profiling data will be made after assessment. The screener and online questionnaire (Attachments 9 and 6) will be hosted on Qualtrics’ server.

Individuals who are enrolled in the online panel will be sent an invitation to participate in the proposed project using an email invitation (Attachment 8) provided by Qualtrics. The invitation will include a link behind a “Start” button, with the link going to a web page that contains the screener (Attachment 9). In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded. If the potential respondent agrees to participate in the proposed project, he or she will click the “Start” button.

Approximately 9,000 potential respondents are anticipated to complete the screener, and 630 qualified respondents will then continue to the online questionnaire (Attachment 6). Criteria for being eligible for the questionnaire are:

• Employed as a teacher, administrator (e.g., principal, vice principal, guidance counselor), or coach

• Works in a middle school or high school

• 50% or more of professional time is spent in direct contact with students (or if less than 50%, participant oversees school operations, such as curriculum development or school policy)

• Before COVID-related school closures, works on-site at a school or facility with students

• Speaks and reads English fluently

• U.S. resident

If the respondent does not meet the eligibility criteria assessed during screening, he or she will be routed to a page that thanks the respondent but indicates that he or she does not fit the specific criteria needed for this proposed project. It is estimated that in total 7,570 respondents will be terminated after completing the screener. Criteria for termination are:

• Potential participants who are school staff but not educators (e.g., cafeteria workers, custodians) will be excluded.

• Does not work in a middle school or high school.

• Spends less than 50% of professional time in direct contact with students, and does not oversee school operations, such as curriculum development or school policy.

• Aside from COVID-related school closures, does not work on-site at a school or facility where there are students.

• Does not speak or read English fluently.

• Not a U.S. resident.

• Recruitment is complete for participant focus population (e.g., 200 administrators have already completed the survey) or demographic/professional quotas.

Survey Administration

Participants who meet the eligibility criteria will be routed to the online consent form and questionnaire (Attachments 10 and 6). A preamble to the questionnaire states the length of the survey and provides other information about the survey. The questionnaire will include questions regarding demographic characteristics, perceptions about youth e-cigarette use, and reactions to the ads (e.g., perceived effectiveness (PE), personal relevance, confusion, believability, effect on motivation to talk with youth, etc.). All participants will view all four ads, presented consecutively and in a random order each time. Participants will be allowed to view the ad for as long as they would like. Then, thumbnail pictures of the ad will be displayed for subsequent survey questions.

Approximately 30 participants are expected to discontinue the questionnaire before completing it. Given identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited or for a respondent to complete the survey more than once.

B.3 Methods to Maximize Response Rates and Deal with No Response

Four methods will be used to maximize survey response or completion rates in the quantitative portion of the proposed project: First, inviting only those who are likely to be eligible by comparing panelist profile data against the eligibility criteria listed (e.g., educator role, school type, percent time in contact with students, years of professional service, geographic location of school, race, ethnicity, and gender) in the online questionnaire eligibility screener (Attachment 9) to reduce the proportion of “incomplete” responses due to not meeting the inclusion criteria. Second, drafting the invitations in a manner that has been shown, through prior testing, to yield optimal results. For the email invitation, this includes attending to the following: subject line, topic description, survey details, a description of the token of appreciation provided upon survey completion, and format (html vs. text) that elicits the most favorable response rates. Third, survey responses during the field period will be closely monitored, and a slightly revised invitation email will be sent to potentially eligible participants who have not yet responded. Finally, to encourage participation and thus maximize the response rate, each participant who completes the questionnaire is offered a $5 Amazon e-gift card as a token of appreciation.

For the qualitative portion of the proposed project, each participant will receive a token of appreciation of $75 for participating in the focus group discussion. In the event that all of the scheduled participants (64) show up for the focus groups, the additional participants (beyond 48) will be permitted to participate.

B.4 Test of Procedures or Methods to be Undertaken

The proposed project involves the collection of qualitative and quantitative information. Similar procedures were used to test the concepts and ads developed for the 2021 Tips from Former Smokers® campaign under this generic clearance, specifically, Agency GenIC #18 titled “National Tobacco Education Campaign Creative Concept Testing (OMB No. 0920-0910).”

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Primary responsibility for methodological design, data collection, and data analysis will be performed by Doug Rupert from RTI International, whose information is listed below:

Doug Rupert

Senior Health Communication Scientist

RTI International

P.O. Box 12194

Research Triangle Park, NC 27709-2194

Phone: 919.541.6495

Email: [email protected]

Individuals consulted at CDC on the proposed project design are listed below.

Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Highway, NE MS F-79 Atlanta, GA 30341
Brian Armour	Associate Director for Science, Office of the Associate Director for Science	Phone: 404.498.3014 E-mail: [email protected]
Elizabeth Courtney-Long	Health Scientist, Office of the Associate Director for Science	Phone: 404.498.0264 E-mail: [email protected]
Jane Mitchko	Deputy Chief, Health Communications Branch	Phone: 770.488.5752 E-mail: [email protected]
Sarah Lewis	Health Communication Specialist, Health Communications Branch	Phone: 770.488.7424 E-mail:  [email protected]
Michelle O’Hegarty	Health Communication Specialist, Health Communications Branch	Phone: 770.488.5582 E-mail: [email protected]
Lauren Boyle-Estheimer	Sr. Health Communications Specialist, Health Communications Branch	Phone: 404.498.2283 E-mail: [email protected]

10 of 11