Supporting Statement B

Information Collection Request

New

National Tobacco Education Campaign

Rough-Cut Testing of Television Advertisements

(OMB No. 0920-0910)

Supporting Statement: Part B

Program Official/Contact

Michelle O’Hegarty, PhD

Office on Smoking and Health

National Center for Chronic Disease Prevention and Health Promotion

Centers for Disease Control and Prevention

Atlanta, Georgia 30341

770-488-5582

FAX: 770-488-5939

Email: [email protected]

October 4, 2024

TABLE OF CONTENTS

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

2. Procedures for the Collection of Information

3. Methods to Maximize Response Rates and Deal with No response

4. Tests of Procedures or Methods to be Undertaken

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or

Analyzing Data

LIST OF ATTACHMENTS

Attachment 1: Online Questionnaire Email Invitation to Potential Respondents

Attachment 2: Online Questionnaire Recruitment Screener

Attachment 3: Online Questionnaire

Attachment 4: Online Questionnaire Screenshots

Attachment 5: Panel Terms and Conditions

Attachment 6: Battelle Institutional Review Board Approval

Attachment 7: Panel Privacy Policies

Notes on Excluded Attachments. In this information collection request (ICR), CDC outlines a plan to test rough-cut advertisements (ads) with content that may be considered sensitive. The draft materials are not included because the near final, “rough-cut” ads have not been approved for public distribution by HHS/Assistant Secretary for Public Affairs (ASPA). To support adequate review of this Gen IC by OMB, the Centers for Disease Control and Prevention requests permission to provide OMB with a secure link to the scripts used to develop the rough-cut ads.  

Part B: Statistical Methods

B.1 Respondent Universe and Sampling Methods

This is a request for a quantitative data collection. In this GenIC, the Centers for Disease Control and Prevention (CDC) requests OMB approval to collect information for rough-cut testing of 12 rough-cut television ads developed for a future Tips From Former Smokers^® (Tips^®)¹ campaign among adults who smoke 18 to 64 years of age. This testing will provide information that will inform whether changes must be made to the ads before finalizing, to help ensure that the ads are credible, persuasive, clear, and will motivate the largest number of people who smoke to quit smoking cigarettes completely. The proposed information collection will involve testing rough-cut ads among people who smoke, aged 18 to 64. The PlowShare Group, and subcontractor Battelle, will conduct the data collection and analysis for this proposed project. Battelle partners with Sago, a global research company, to recruit participants and collect data.

The sample of respondents for the quantitative data collection will be drawn from established online panels (such as Sago, Toluna, Prodege, Quest Mindshare, RFG, Torfac, Solugo, Savanta, and Mfour). These panels employ a double opt-in process for individuals to participate in a survey – they must opt-in to become panelists, and they must also opt-in to each survey. For this proposed project, panel providers will gather information for pre-screening as well as ask participants to review both a privacy policy and a terms and conditions statement that outlines the parameters for their participation. For this proposed project, the pre-screening is designed to screen out persons < 18 years of age or > 64 years of age, who live outside the U.S., or who don’t smoke. Although the sample will be a convenience sample, the panel sample has a reasonable degree of diversity in key demographic characteristics such as age, gender, region of residence, race/ethnicity, education, household size, and income. The findings from this proposed project will have high internal validity, but are not expected to be widely generalizable to the universe of people who smoke in the U.S. As this proposed project is part of formative work for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. However, the design allows for high internal validity to provide information on the perceived effectiveness of rough-cut ads under test.

Power analyses were run to determine the sample sizes needed to detect statistically significant differences on key measures (e.g., PE scores for each rough-cut ad) for people who smoke

based on the observed differences in previous rounds of rough-cut testing. The primary outcome of interest is perceived effectiveness of ads among people who smoke. To attain 80% power to detect a 0.31 standard deviation difference between any pair of ads, using a two-sided t-test, a total of 800 people who smoke are needed per ad. With a sample of this size, we will be able to reliably detect differences in perceived effectiveness within and between people with low-SES who smoke and people who are not of low-SES who smoke.

Another aim of this study is to examine ad perceptions among members of the racial/ethnic groups featured in the rough-cut ads. This allows us to test whether each ad resonates with members of that population and represents their culture appropriately. To address this aim, some rough-cut ads will be evaluated within supplemental samples of racial minority group members. These samples are expected to include:

• English-Speaking American Indians and Alaska Natives

• English-Speaking Asians

We will not use statistical tests or other means to compare the perceptions of participants from these samples to the perceptions of main sample participants. We will only examine key variables (e.g. perceived effectiveness, motivation to quit smoking, believability, and confusion) within these groups, with an emphasis on open-ended responses that could indicate information that is unclear to these populations or represents their culture inaccurately.

The maximum number of respondents for each ad is 1,200 (800 + 400) and the maximum number of respondents for all ads is 10,800 ([1,200 x 3 ads] + [800 x 9 ads]). Table B.1 provides additional information by intended audience about the ads to be tested and the desired number of respondents associated with each ad to be tested. To achieve this sample size, we conservatively anticipate screening 13,579 respondents; this estimate is based on two factors from prior experiences in the field. First, it is anticipated that roughly 18 percent of screener respondents (n=2,445) will be deemed ineligible for the proposed project because of not meeting inclusion criteria or because they belong to strata that have reached their recruitment goal. Second, of those deemed eligible (n=11,135), an estimated three percent (n=335) will start but not complete the questionnaire. Thus, 13,579 respondents are needed to obtain 10,800 in the final sample (Figure B1).

Table B.1 Desired Number of Respondents to View Each Ad
	Main Sample Randomly Assigned to Ads			Supplemental Samples Assigned to Ads by Race/Ethnicity
Rough-Cut Ad	People With Low-SES Who Smoke	People Who Are Not of Low-SES Who Smoke	Total Participants Randomly Assigned to View Each Ad	English-Speaking American Indians and Alaska Natives	English-Speaking Asians	Total Participants
Ad 1	400	400	800	400		1,200
Ad 2	400	400	800	400		1,200
Ad 3	400	400	800		400	1,200
Ad 4	400	400	800			800
Ad 5	400	400	800			800
Ad 6	400	400	800			800
Ad 7	400	400	800			800
Ad 8	400	400	800			800
Ad 9	400	400	800			800
Ad 10	400	400	800			800
Ad 11	400	400	800			800
Ad 12	400	400	800			800
All Ads	4,800	4,800		800	400	10,800
¹ People who smoke are defined as persons between 18-64 years of age who report smoking > 100 cigarettes during their lifetime and who, at the time of the survey, report smoking cigarettes every day or some days, and have smoked at least one cigarette in the past 30 days. 2 English-Speaking American Indians and Alaska Natives, and English-Speaking Asians who smoke will view the rough-cut ads featuring a member of their racial/ethnic group to determine if the ad resonates with members of that population and represents their culture appropriately. English-Speaking American Indians and Alaska Natives will be randomly assigned to view one of two ads featuring an American Indian. English-Speaking Asians will view an ad featuring a Native Hawaiian. 3 For a total final sample size of 10,800 participants, it is estimated that a starting sample of 13,579 respondents will be needed.

Figure B.1 Flowchart of the Sampling of Respondents for Quantitative Data Collection*

*Respondents who are screened out include respondents from strata that have filled, after which recruitment is closed.

B.2 Procedures for the Collection of Information

The data collection subcontractor, Battelle, will be responsible for coordinating data collection activities, collecting, and summarizing information, and preparing final reports. Battelle partners with Sago, a global research company, to recruit participants and collect data. Information for this proposed project will be collected using quantitative methods. Data collection will test respondents’ reactions to 12 rough-cut ads. The testing will also collect basic demographic and cigarette use information.

Recruitment and Screening

Potential participants will be recruited from existing, online, convenience panels such as those managed by Sago, Toluna, Prodege, Quest Mindshare, RFG, Torfac, Solugo, Savanta, and Mfour. Several panels are being used for this data collection due to their ability to recruit people who smoke with different demographic profiles. While the smallest panel being used has only 220,000 participants, the largest panel being used has over seven million participants. The panel providers have profiled their panels regarding smoking behavior and can focus on and identify respondents who are pre-identified as being aged 18 to 64 for the survey. They also collect demographics such as gender and ethnicity. However, additional screening will be conducted to confirm that profiling information is current and to assess whether any information has changed (i.e., educational status, state of residence). To protect potentially identifiable information (PII) of respondents, no comparison to the original individual profiling data will be made after assessment. The screener and online questionnaire (Attachments 2 and 3) will be hosted on Sago’s servers.

Individuals who are enrolled in one of the online panels will be sent an invitation to participate in the proposed project using an Email Invitation (Attachment 1) provided by the panels’ sample management system. The invitation will include a link behind a “Start” button, with the link going to a web page that contains the Screener (Attachment 2). In addition, the same-worded invitation will be sent at regular intervals after the original invitation is sent to those respondents who have not yet responded. If the potential respondent agrees to participate in the proposed project, he or she will click the “Start” button.

Approximately 13,579 potential respondents are anticipated to complete the screener, and 11,135 respondents aged 18 to 64 will then continue to the Online Questionnaire (Attachment 3). Respondents will be eligible for the questionnaire if they are aged 18 to 64, report smoking > 100 cigarettes during their lifetime, and, at the time of the survey, report smoking cigarettes every day or some days, and had smoked at least one cigarette in the past 30 days.

People who smoke will be classified as either low-SES or non-low-SES based on education, employment status, household size, and income. People who smoke will be classified as low-SES status if they meet criteria for at least two of the following three criteria:

1. Have a high school education or less, or completed high school without further schooling except for job-specific training;

2. Have a household income that is below the Federal Poverty Line for their household size;

3. Are currently unemployed (excluding retirees and the disabled).

People who smoke may be assigned to a supplemental sample based on their race/ethnicity. Supplemental samples for this data collection will include:

• English-Speaking American Indians and Alaska Natives

• English-Speaking Asians

If the respondent does not meet the eligibility criteria assessed during screening, he or she will be routed to a page that thanks the respondent but indicates that he or she does not fit the specific criteria needed for this proposed project. The page that thanks the respondent is located in the respondent’s panel system outside of the survey. It is estimated that in total 2,445 respondents will be terminated after completing the screener. Criteria for termination are:

1. Persons younger than 18 years of age or older than 64 years of age.

2. Does not reside in the U.S.

3. Does not smoke.

4. Recruitment is complete for participant audience (e.g., all needed for this project have already completed the survey).

Survey Administration

Participants who meet basic eligibility criteria will be routed to the Online Questionnaire (Attachment 3). A preamble to the questionnaire states the length of the survey and provides other information about the survey. The questionnaire will include questions regarding demographic characteristics and smoking behavior. A sample of people who smoke will be stratified by SES and randomly assigned to view one of 12 rough-cut ads. Randomization of participants to view the different rough-cut ads being tested ensures that there is a similar distribution of individuals of different characteristics (e.g., age, gender, etc.) across the different ads. Additionally, we will recruit samples of participants who will be assigned to view rough-cut ads based on their race/ethnicity. English-Speaking American Indians and Alaska Natives will be randomly assigned to view one of two ads featuring an American Indian. English-Speaking Asians will view an ad featuring a Native Hawaiian. Each rough-cut ad will be in a video file. Participants can click the “restart” button to view the video as many times as they would like.

Approximately 335 participants are expected to discontinue the questionnaire before completing it. Due to identity protection technology, it will not be possible for anyone to enter the survey who has not been recruited or for a respondent to complete the survey more than once.

B.3 Methods to Maximize Response Rates and Deal with No Response

Five methods will be used to maximize response or completion rates in this current proposed project: (1) inviting only those who have been identified as aged 18 to 64 to take the survey to reduce the proportion of “incomplete” responses due to not meeting the inclusion criteria, (2) identifying the CDC as the agency of record, since this agency is credible and serves the public good, and (3) drafting the invitations in a manner that has been shown, through prior testing, to yield optimal results. For the email invitation, this includes attending to the following: types of subject lines, topic description, survey details, incentive description, and format (html vs. text) that elicits the most favorable response rates. Additionally, (4) survey responses during the field period will be closely monitored and the invitation email will be resent, with the same content as the original invite, to eligible participants who have not yet responded. Finally, (5) to encourage participation and thus maximize the response rate, the participants who complete the questionnaire will receive a point-based incentive, which can be redeemed for other items, such as Amazon gift cards.

B.4 Test of Procedures or Methods to be Undertaken

The proposed project involves the collection of quantitative information. The same procedures were used to conduct rough-cut testing of the ads developed for the 2020 Tips campaign under this generic clearance, specifically, Agency GenIC # 21 titled “National Tobacco Education Campaign Rough-Cut Testing of Television Advertisements (0MB No. 0920-0910).”

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Primary responsibility for methodological design, data collection, and data analysis will be performed by Lisa John, Abigail Evans, and Brian Miller from Battelle, whose information is listed below.

Lisa V. John, PhD, PMP

Project/Program Manager

Battelle

505 King Ave

Columbus, OH 43201

Phone: (614) 456-5051

Email: [email protected]

Abigail Evans, PhD

Lead Social Scientist

Battelle

6115 Falls Road, Suite 200

Baltimore, MD 21209

Phone: (410) 372-2750

Email: [email protected]

Brian Miller, MS

Data Scientist III

Battelle

505 King Ave

Columbus, OH 43201

Phone: (614) 424-7645

Email: [email protected]

Individuals consulted at CDC on the proposed project design are listed below.

Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Highway, N.E MS F-79 Atlanta, GA 30341
Michelle O’Hegarty	Health Communication Specialist, Health Communications Branch	Phone: 770.488.5582 Email:  [email protected]
Elizabeth Courtney-Long	Health Scientist, Office of the Associate Director for Science	Phone: 404.498.0264 E-mail: [email protected]
Lauren Boyle-Estheimer	Sr. Health Communication Specialist	Phone: 404.498.2283 Email:  [email protected]
Rebecca Murphy	Research and Evaluation Team Lead, Health Communications Branch	Phone: 770.488.8964 Email:  [email protected]



1 Use of trade names is for identification only and does not imply endorsement by the U.S. Department of Health and Human Services.

3