Supporting Statement B - 05MAY2017

SUPPORTING STATEMENT FOR THE

National Tobacco Education Campaign

Creative Concept Testing

(OMB No. 0920-0910, Exp. Date 03/31/2018)

PART B: STATISTICAL METHODS

April 19, 2017

Submitted by:

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

Department of Health and Human Services

Refer questions to:

Michelle O’Hegarty, PhD

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

4770 Buford Highway, NE MS F-79

Atlanta, Georgia 30341

770-488-5582

FAX: 770-488-5939

Email: [email protected]

TABLE OF CONTENTS

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information 3. Methods to Maximize Response Rates and Deal with No response

4. Tests of Procedures or Methods to be Undertaken

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

LIST OF ATTACHMENTS

Attachment 1: Online Questionnaire Email Invitation to Potential Respondents

Attachment 2: Online Questionnaire Recruitment Screener

Attachment 3: Online Questionnaire

Attachment 4: Focus Group Recruitment Screener - Exclusive Conventional Cigarette Smokers and Dual Users

Attachment 5: Pre-Focus Group Questionnaire and Informed Consent

Attachment 6: Focus Group Moderator’s Guide- Exclusive Conventional Cigarette Smokers

Attachment 7: Focus Group Moderator’s Guide – Dual Users

Attachment 8: Participant Feedback Questionnaire

Attachment 9: Toluna Terms and Conditions

Attachment 10: Battelle Institutional Review Board Approval

Attachment 11: Toluna Privacy Policy

Attachment 12: Schlesinger Privacy Policy

Notes on Excluded Attachments. In this information collection request, CDC outlines a plan to test creative concepts with content that may be considered sensitive. The draft materials are not included because the creative concepts have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA). To support adequate review of this Gen IC by OMB, the Centers for Disease Control and Prevention requests permission to provide OMB with a secure link to the draft materials.  

Part B: Statistical Methods

B.1 Respondent Universe and Sampling Methods

This is a request for a quantitative and qualitative data collection. In this GenIC, the Centers for Disease Control and Prevention (CDC) requests OMB approval to collect information for creative concept testing of four concepts developed for a future National Tobacco Education Campaign, in order to identify one concept that will motivate the largest number of smokers of conventional cigarettes and dual users to quit smoking conventional cigarettes completely. The proposed information collection will involve testing of concepts among adult smokers 18-54 years old, categorized as either exclusive smokers of conventional cigarettes, or dual users of conventional cigarettes and e-cigarettes. Exclusive conventional cigarette smoking is defined by having smoked > 100 cigarettes in their lifetime, smoked at least one cigarette in the past 30 days, and currently smoke conventional cigarettes every day or some days. Dual use of conventional cigarettes and e-cigarettes is defined as being a conventional cigarette smoker and currently using e-cigarettes every day or some days. Exclusive e-cigarette users will not be included in the study. The Plowshare Group, and subcontractors, Qualtrics, and Battelle, will conduct both the quantitative and the qualitative portions of this study.

Quantitative: Online Questionnaire. The sample of respondents for the quantitative data collection will be drawn from Toluna’s online panel (see http://www.toluna-group.com//choose-the-people#global-reach for more detail on this panel). Toluna employs a double opt-in process for individuals to participate in a survey – they must opt-in to become panelists, and they must also opt-in to each survey. For this study, Toluna will gather information for pre-screening as well as review both a privacy policy and a terms and conditions statement that outlines the parameters for their participation. The latter information is designed to screen out persons < 18 and > 54 years of age, for this study. Although the sample will be a convenience sample, the panel sample has a reasonable degree of diversity in key demographic characteristics such as age, gender, region of residence, race/ethnicity, education, and income. The findings from this study while having high internal validity, are not expected to be widely generalizable to the universe of US smokers.

Within the two major strata (i.e., exclusive conventional cigarette smokers and dual users of conventional cigarettes and e-cigarettes), efforts will be made to ensure sufficient numbers of the following groups of individuals for inclusion in the study because of their disproportionately high smoking prevalence. These groups are not being considered as strata, but rather as tabulation variables. To ensure that statistically precise estimates can be generated for the subpopulations identified below, efforts will be made to oversample individuals from these subgroups:

• Low-socioeconomic status (SES)¹

• Young adult (18-26 years old)

• English-speaking Hispanics

• African-Americans

• Individuals with self-reported anxiety/depression

• LGBT individuals

As this study is considered part of formative work for campaign development and planning, these methods are not intended to generate nationally-representative samples or precise estimates of population parameters. However, the design allows for high internal validity to provide information on the perceived effectiveness of creative concepts under test.

A power analysis was run to determine the sample size needed to detect statistically significant differences on key measures (e.g., average perceived effectiveness scores for each creative concept). The sample size calculation was done based on the following parameters: (1) power = 80%; (2) effect size, Cohen’s f = 0.10; (3) adjustment for multiple comparisons based on a Bonferroni adjusted α error rate of p= 0.008 for four “groups” (concepts). Based on these assumptions, and with an oversampling of the previously identified subgroups to allow for precise estimates, a minimum of 1,343 persons per concept per stratum are needed, or a total sample of 5,372 (1,343 x 4 concepts) per stratum. This yields a total sample size of 10,748 (5,372 participants x 2 strata) for the two strata combined. Hence, for any given concept, an anticipated 2,686 respondents in the overall quantitative sample will be allocated to view it, within the two strata combined. The overall sample size will allow us to assess meaningful and adequately-powered differences between the four concepts within each stratum. Sub-group differences within each concept arm will be assessed and interpreted qualitatively.

To achieve this sample size, we conservatively anticipate screening 15,828 individuals. Based on a prior round of testing for National Tobacco Education Campaign (NTEC), which used a roughly similar sample, 10,613 respondents started the quantitative instrument, and a total of 8,809 qualified adult respondents completed the online survey (a completion rate of 83%).

To identify the concept that is the most motivating to the target audience, respondents will be randomly assigned and routed to view only one of the four concepts within each stratum, each with two TV ad executions (see section B.2 for more information on how the concepts and executions are viewed).

Randomization of participants into the different concepts being tested ensures that there is a similar distribution of individuals of different characteristics (e.g., age, sex, sexual orientation) across the different concept arms, thus controlling for any confounding influences. A block design will be used to ensure that the number of persons in each arm is approximately even to ensure robust statistical comparisons.

4 concept arms per stratum

4 concept arms per stratum

2 strata in total

Figure 1. Assignment of Participants to Strata and Concept Arms for Quantitative Data Collection

The sampling procedures for quantitative activities are summarized in Figure 2 on the next page.

Figure 2. Flowchart of the Sampling of Participants for Quantitative Data Collection

Qualitative: In-Person Focus Groups. The goal is to have 10 participants in each in-person focus group of the qualitative study. Approximately 400 respondents will need to be recruited and screened (accounting for contacting 3 people for every one that qualifies), in order to enroll up to 144 people for the study, of which 120 will participate in the focus groups. Four focus groups will be conducted in each of three cities across the U.S.: Louisville, KY; Philadelphia, PA; and Phoenix, AZ. Focus groups will be segmented by tobacco use status (exclusive conventional cigarettes smokers and dual users) and age (young adults aged 18-26 and older adults ages 27-54). Focus groups will be at least 50% male, given higher rates of smoking among males compared to females (e.g., in 2014, the prevalence was higher among males at 19% than females 15%) (CDC, 2015), at least 50% low-SES, and will have a roughly equal mix of smoking frequency (i.e., < 20 cigarettes/less than a pack per day versus 20+ cigarettes/a pack or more per day). Recruitment will continue until the target number of participants is recruited (12 people recruited with a minimum of six and maximum of 10 participants included in each of the 12 focus groups). These 120 participants will be purposively sampled based on their smoking behavior profile and other specific eligibility criteria (see Table B.1 on the next page).

Table B.1. Qualitative Subpopulations1
Tobacco Use	Subpopulations	Louisville, KY	Philadelphia, PA	Phoenix, AZ
Exclusive Conventional Cigarette Smokers	Adult Exclusive Conventional Cigarette Smokers (27-54 years) – All-SES	1 group (n=10)	1 group (n=10)	1 group (n=10)
	Young Adult Exclusive Conventional Cigarette Smokers (18-26 years) – All-SES	1 group (n=10)	1 group (n=10)	1 group (n=10)
Dual Users of Conventional Cigarettes & E-Cigarettes	Adult Dual Users (27-54 years) – All-SES	1 group (n=10)	1 group (n=10)	1 group (n=10)
	Young Adult Dual Users (18-26 years) – All-SES	1 group (n=10)	1 group (n=10)	1 group (n=10)
Total 12 groups (N=120)
1 For each group, we recruit up to 12 participants, and we will hold the focus groups with up to 10 participants. Groups will be at least 50% male, at least 50% low SES, and will be roughly one-half smoking < 20 cigarettes/less than a pack per day and one-half smoking 20+ cigarettes/a pack or more per day.

The sampling procedures for qualitative activities are summarized in Figure 3 below.

Figure 3. Flowchart of the Sampling of Participants for Qualitative Data Collection

B.2 Procedures for the Collection of Information

The data collection subcontractor, Qualtrics, will be responsible for coordinating data collection activities, collecting and summarizing information, and preparation of topline reports. Battelle will prepare final reports, in collaboration with Qualtrics. Information for this study will be collecting using quantitative and qualitative methods. The quantitative and qualitative portions will occur simultaneously. The quantitative testing will collect information about the respondents’ reactions to the creative concepts, as well as basic demographic and cigarette and e-cigarette use information to assess whether responses to creative concepts differ by subpopulation or tobacco use status. The qualitative testing will result in a deeper understanding of which of the four concepts is most clear, credible, and effective in prompting changes in tobacco-related knowledge, attitudes, beliefs, and behavior, as well as why that is the case. Each of the four creative concepts will have two TV ad executions (variations of the concept). The procedures are described below in the following sections.

Quantitative: Online Questionnaire

The quantitative phase of data collection will include respondents recruited from the Toluna online panel, which is an established, online panel system that includes more than 1.7 million people in the U.S. Toluna has profiled their panels regarding smoking behavior and can target and identify respondents who are pre-identified smokers ages 18-54 for the survey. Toluna also collects demographics such as gender, ethnicity, and parenthood status. However, additional screening will be conducted to confirm that Toluna’s profiling information is current and to assess whether any information has changed (i.e., educational status, state of residence). To protect potentially identifiable information (PII) of respondents, no comparison to the original individual profiling data will be made after assessment. The screener and online questionnaire will be hosted on Qualtrics’ server farm.

Recruitment and Screening

Individuals who are enrolled in the online panel will be sent an invitation to participate in the study using an Email Invitation (Attachment 1) provided by Qualtrics’ sample management system. The invitation will include a link behind a “Start” button, with the link going to a web page that contains the Screener (Attachment 2). In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded. If the potential respondent agrees to participate in the study, he or she will click the “Start” button.

Approximately 15,828 potential respondents are anticipated to complete the screener, and 11,080 respondents in the target age range of 18-54 years will then continue to the questionnaire. Criteria for being eligible for the questionnaire are:

1. Adult exclusive conventional cigarette smoker criteria: persons between 18-54 years of age who reported smoking > 100 conventional cigarettes during their lifetime and who, at the time of the survey, reported smoking conventional cigarettes every day or some days.

2. Adult dual user criteria: current smokers between 18-54 years of age (i.e., those who meet the criteria in Option A) who reported using e-cigarettes, even one time and who, at the time of the survey, reported using e-cigarettes every day or some days.

Quantitative information collection will occur concurrently for both exclusive conventional cigarette smokers and dual users of exclusive conventional cigarettes and e-cigarettes. If the respondent does not meet the eligibility criteria assessed during screening, he or she will be routed to a page that thanks the respondent, but indicates that he or she does not fit the specific criteria needed for this study. The page that thanks the respondent is located in the respondent’s panel system outside of the survey. It is estimated that in total 4,749 respondents will discontinue their participation after completing the screener. Criteria for termination are:

1. Non-smokers (never smokers or non-current smokers).

2. Persons younger than 18 years of age or older than 54 years of age.
   
   

Persons meeting the eligibility criteria based on age and conventional cigarette and/or e-cigarette use will be sent a link to the Online Questionnaire (Attachment 3).

Survey Administration

A preamble to the questionnaire states the length of the survey and other information about the survey, if they qualify for and complete the survey. The questionnaire will include additional questions regarding demographic characteristics and smoking behavior.

Participants who meet basic eligibility criteria will be routed to the online questionnaire (Attachment 3) and then randomly assigned to view one of the four concepts. Each concept will include two executions, presented consecutively (in a random order each time) during that concept viewing, and will be described to participants using images, text, and audio file. Specifically, there will be an MP4 file for each execution (with four concepts, and two executions each, this means there will be eight MP4s). The execution MP4 file will contain storyboards, with each frame displayed. There will be no script on the board; the audio will play along with the storyboards. When the audio stops, the execution is over. Participants can click the “restart” button to view the concept as many times as they would like. Then, thumbnail pictures representing the concept will accompany the aided response section. Participants will provide responses to the concept, as a whole, and will not be responding based on impressions from a single execution.

Approximately 332 participants are expected to discontinue the questionnaire before completing it. Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited or for a respondent to complete the survey more than once.

Qualitative: In-Person Focus Groups

Recruitment and Screening

Potential focus group participants from the Schlesinger Associates respondent panel in each of the three cities will be contacted and screened via phone. Potential participants will be drawn from existing panels of focus group respondents that each focus group facility maintains. The facilities will call potential respondents to invite them to participate in the study, to screen them for eligibility based on conventional cigarette and e-cigarette use status and age using the Focus Group Screener (Attachment 4), and to identify participants who match the characteristics of priority groups that should have adequate representation in the groups (i.e., males, and low-SES individuals). The screener will collect information on age and conventional cigarette and e-cigarette use to determine eligibility for focus group participants. This screener will also include demographic questions, including gender, race/ethnicity, educational attainment, income, and employment status. As noted previously, focus groups will aim to include at least 50% males, at least 50% low-SES, and diversity in smoking frequency (< 20 cigarettes per day versus 20+ cigarettes per day). Participants will be segmented into four groups in each of the three cities, according to their age and use of conventional cigarettes and e-cigarettes:

• Adult Exclusive Conventional Cigarette Smokers (27-54 years)

• Young Adult Exclusive Conventional Cigarette Smokers (18-26 years)

• Adult Dual Users (27-54 years)

• Young Adult Dual Users (18-26 years)

Each eligible participant will be scheduled to attend a focus group corresponding to their age and cigarette and e-cigarette use.

Focus Group Participation. Prior to beginning the focus group, all participants will be required to submit a signed consent form and complete a brief, self-administered Pre-Focus Group Questionnaire (Attachment 5) to confirm in-person their eligibility to participate in the study. A trained moderator will lead the focus group discussion; discussions will follow one of the semi-structured Moderator’s Guide for Focus Groups, which are tailored to exclusive conventional cigarette smokers or dual users (Attachments 6 and 7, respectively). During the focus group, participants will complete the Participant Feedback Questionnaire (Attachment 8), which collects additional information on the concepts that they viewed. Specifically, this questionnaire asks about emotional responses to the concepts and about how they think the concepts portrayed smokers. This questionnaire contains close and open-ended questions to collect information about potentially sensitive reactions to the concepts and any opinions that participants did not feel comfortable sharing with the group. This instrument is necessary because it allows for information to be collected on target measures (e.g., shame and anger) in a way that mimics the quantitative survey, thus allowing for comparisons, and, more importantly, because it allows for participants to report on private emotions without fear of judgement from fellow focus group participants.

During the focus groups, all four concepts (each with two executions, presented consecutively and in a random order during each concept viewing) will be presented to participants using non-animated storyboards including relevant images accompanied by an audio track, which will be accompanied by a preamble explaining that these are concepts that will be developed into ads. Specifically, there will be an MP4 file for each execution (with four concepts, and two executions each, this means there will be eight MP4s). The execution MP4 file will contain storyboards, with each frame displayed. There will be no script on the board; the audio will play along with the storyboards. When the audio stops, the execution is over. The audio track will be played to participants once, but the storyboard in its full size will be displayed for the duration of the discussion.

Each of the four concepts (with both executions) will be discussed individually, and then four concepts will be compared to one another. The focus group discussions, which are anticipated to last 90 minutes, will be live-streamed to staff who cannot observe the focus groups in person using FocusVision (or similar services), which is a live video-streaming service. CDC and key staff from the contractors and subcontractors will be able to remotely supervise the focus group discussions through a secure link to the live video streams. During the focus groups, the moderator will have the ability to receive messages from researchers based on the discussion, and researchers will be able to “listen in” (in mute-only fashion) to the conversation. Qualtrics will retain focus group recordings and de-identified transcripts on a password-protected database for six months, after which all video files will be deleted. Project staff from Qualtrics, Plowshare, Battelle and Arnold Worldwide will have access to the focus group recordings and de-identified transcripts. CDC will only have access to the de-identified transcripts which will be handled in accordance with the record control schedule (maintained at least six years, but no longer than ten years).

B.3 Methods to Maximize Response Rates and Deal with No Response

Five methods will be used to maximize response or completion rates in this current study: (1) inviting only those who have been profiled as smokers to take the survey to reduce the proportion of “incomplete” responses due to not meeting the inclusion criteria, (2) identifying the CDC as the agency of record, since this agency is credible and serves the public good, (3) drafting the invitations in a manner that has been shown, through prior testing, to yield optimal results. For the email invitation, this includes attending to the following: types of subject lines, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates. Additionally, (4) survey responses during the field period will be closely monitored and the invitation email will be resent, with the same content as the original invite, to eligible participants who have not yet responded. Finally, (5) to encourage participation and, thus, maximize the response rate, the participants who complete the questionnaire will receive a point-based incentive, which can be redeemed for other items, such as Amazon gift cards. Each participant will be paid $75 cash for participating in the focus group. In the event that more than 10 participants show up for a focus group, the additional participants (beyond 10) will be thanked, paid $75, and sent home.

B.4 Test of Procedures or Methods to be Undertaken

The proposed project involves the collection of quantitative and qualitative information. Similar procedures were used to conduct rough-cut testing of the ads developed for the 2016 Tips™ campaign under this same generic clearance, specifically, GenIC #11 titled “National Tobacco Prevention and Control Public Education Campaign: Message Platform Testing for Development of Future Advertising.”

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Primary responsibility for methodological design, data collection, and data analysis will be performed by Carol Haney and David Vannette from Qualtrics, and Lisa John and Robert Alexander from Battelle, whose information is listed below.

Carol Sue Haney

Senior Research and Data Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (802) 258-0518

Email: [email protected]

David L. Vannette, PhD Candidate

Principal Research Scientist

Qualtrics

333 West River Park Drive

Provo, UT 84604

Phone: (616) 502-4828

Email: [email protected]

Lisa V. John, PhD, PMP

Project/Program Manager

Battelle

5712 Oakland Ave

St. Louis, MO 63110

Phone: (314) 880-3652

Email: [email protected]

Robert Alexander, PhD, MPH, CHES

Director, Health Communication, Policy, and Surveillance

Battelle

6115 Falls Rd, Suite 200

Baltimore, MD 21209

Phone: (404) 460-1462

Email: [email protected]

Individuals consulted at CDC on the study design are listed below.

Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Highway, N.E MS F-79 Atlanta, GA 30341
Israel Agaku	Office of the Associate Director for Science	Phone: 770.488.5138 E-mail:  [email protected]
Diane Beistle	Chief, Health Communication Branch	Phone: 770.488.5066 E-mail:  [email protected]
Michelle O’Hegarty	Health Communication Specialist	Phone: 770.488.5582 E-mail:  [email protected]
Lindsey McCarter	Team Lead, Health Communication Branch	Phone: 770.488.4239 E-mail:  [email protected]
Sarah Lewis	Health Communication Specialist	Phone: 770.488.7424 E-mail:  [email protected]

References

CDC. (2013). Vital signs: current cigarette smoking among adults aged≥ 18 years with mental illness-United States, 2009-2011. MMWR. Morbidity and mortality weekly report, 62(5), 81.

CDC. (2015). Current cigarette smoking among adults—United States, 2005–2014. MMWR Morb Mortal Wkly Rep, 64(44), 1233-1240.

Department of Health and Human Services (HHS). (2014). The Health Consequences of Smoking—50 Years of Progress: A Report of the Surgeon General. Atlanta, GA: US Department of Health and Human Services, Centers for Disease Control and Prevention, National Center for Chronic Disease Prevention and Health Promotion, Office on Smoking and Health.



1 Low SES is defined as satisfying at least two of the following three conditions: 1) having less than a high school education, or completed high school but without further schooling except for job-specific training; 2) having a household income in 2015 of less than $24,999; and 3) being unemployed, excluding retirees and the disabled.

12