Message Testing for Tobacco Communication Activities (MTTCA)
OMB No. 0920-0910 exp. 01/31/2024
Supporting Statement B
Program Official/Contact
Michelle O’Hegarty, PhD
Office on Smoking and Health
National Center for Chronic Disease Prevention and Health Promotion
Centers for Disease Control and Prevention
P: 770-488-5582
F: 770-488-5939
5/22/2023
TABLE OF CONTENTS
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 3
B1. Respondent Universe and Sampling Methods 3
B2. Procedures for the Collection of Information 4
B3. Methods to Maximize Response Rates and Deal with No Response 7
B4. Tests of Procedures or Methods to be Undertaken 8
B5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 8
Attachment 1: Public Health Service Act [42 U.S.C. 241]
Attachment 2: Federal Register Notice
Attachment 3: Overview of Planned Information Collections
Attachment 3a: Example Data Collection Instrument – Screenshots Rough-Cut Testing Survey Screener
Attachment 3b: Example Data Collection Instrument – Screenshots Rough-Cut Testing Survey
Attachment 3c: Example Data Collection Instrument – Creative Concept Testing Survey Screener
Attachment 3d: Example Data Collection Instrument – Creative Concept Testing Survey
Attachment 3e: Example Data Collection Instrument – Creative Concept Testing Focus Group Screener
Attachment 3f: Example Data Collection Instrument – Creative Concept Testing Focus Group Moderator’s Guide
Attachment 3g: Example Data Collection Instrument – Screenshots Message Platform Testing Screener
Attachment 3h: Example Data Collection Instrument – Screenshots Message Platform Testing Survey
Attachment 3i: Example Data Collection Instrument – Moderator’s Guide For Adults Who Smoke Cigarettes
Attachment 3j: Example Data Collection Instrument – Moderator’s Guide For Adults Who Smoke Cigarettes and Use E-cigarettes (Dual Use)
Attachment 3k: Example Data Collection Instrument – Emerging Tobacco Products: Focus Group Screener
Attachment 3l: Example Data Collection Instrument – Emerging Tobacco Products: Focus Group Moderator’s Guide
Attachment 3m: Example Data Collection Instrument – Emerging Tobacco Products: Survey Questionnaire Eligibility Screener
Attachment 3n: Example Data Collection Instrument – Emerging Tobacco Products: Survey Questionnaire
Attachment 4: Example Institutional Review Board Approval Letter
Attachment 5: Example Informed Consent Form
Attachment 6: Toluna Privacy Policy
Attachment 7: Example Reminder Invitations
B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS
The Centers for Disease Control and Prevention (CDC) requests OMB approval for a three-year period, as a revision, to the Message Testing for Tobacco Communication Activities (MTTCA, OMB No. 0920-0910, exp. 1/31/2024) clearance. The purpose of this generic clearance is to collect information among people who smoke in regard to their attitudes and perceptions, and to pretest draft messages and materials for clarity, salience, appeal, and persuasiveness. Information collected will be used to develop tobacco-related health messages, including messages related to the National Tobacco Education Campaign (NTEC).
The primary NTEC audience is adults who smoke aged 18 years or older. The existing clearance also includes youth ages 13-17 years old. The only proposed change to the populations of interest from the last extension request to MTTCA is to remove the upper age limit, previously 54 years of age, to include all adults aged 18 years or older. Respondents will not include members of the state or local government.
There is a need to continue to test prevention and cessation messages related to combustible and noncombustible tobacco products. As these information collections are considered part of formative work for campaign development and planning, these methods are not intended to generate nationally representative samples or precise estimates of population parameters. A variety of qualitative and quantitative information collection strategies are supported through this mechanism, including in-person and online in-depth interviews, in-person and online focus groups, and online surveys.
Qualitative: Qualitative methods are not intended to yield results that are statistically projectable or used to derive quantitative estimates. However, these methods allow tobacco communication messages and/or programs to be designed and marketed with specific audiences in mind (e.g., people who smoke aged 18 years or older). In qualitative studies, quota sampling is often used to select a convenience sample of individuals who meet certain qualifications that reflect characteristics typical of the intended audience. Response rate is not applicable to quota sampling because this type of sampling results in a nonprobability sample which is not representative of the population. In qualitative studies, respondents can be initially contacted by telephone, through the mail, or online; over-recruiting is done to compensate for non-respondents.
Quantitative: Where quantitative methods are used, information collection activities will focus on particular audiences with statistical sampling procedures employed to identify potential survey respondents. Online surveys for message testing will seek a convenience sample (e.g., through an online panel) that nonetheless has an acceptable degree of diversity in key demographic characteristics such as age, gender, education, and race/ethnicity. CDC does not intend to generate nationally representative results or precise estimates of population parameters using these surveys.
CDC will request Office of Management and Budget (OMB) approval for each data collection by submitting an Information Collection Request that describes project purpose, use, and methodology. Sampling methods will vary based on the intended audiences and methodology for each round of data collection.
The methodologies planned for use in this submission will follow standard state-of-the-art approaches adapted from marketing and communications research. Electronic data collection methods will be employed, minimizing COVID-19 and/or other exposure risks. Data will be stored on password protected computers that are only accessible to study personnel. Additionally, all PII will be stored separately from survey data and will not be shared with CDC. If interviews and focus groups are conducted in-person during periods of high COVID-19 and/or other exposure risk, additional precautions will be put into place to ensure staff, contractors, and participants are protected from COVID-19 and/or other exposures. Mitigation strategies could include a) enacting social distancing protocols (i.e., six feet of space between persons to eliminate close contact), b) requiring appropriate personal protective equipment (PPE) use by all parties (e.g., N95 or three-layer cotton masks), c) symptom screening prior to data collection, and d) disinfection of surfaces between sessions/interviews. The precautions may be revised to add or remove strategies as needed. Lastly, precautions may vary from site to site based on conditions on the ground and the timing of when activities are conducted. Project-specific plans will be described in each GENIC as applicable.
In this context, the term pretesting refers to testing messages, strategies, and communication materials before they are finalized and fielded. Questions in all pretesting methodologies include standard measures of communications that are designed to assess to what degree the message was successful in communicating information, including perceived effectiveness, main idea recall, comprehension, believability, personal relevance, motivation to quit smoking, and likes and dislikes. Additional questions may be added to address any specific concerns regarding a message or advertisement, such as how a respondent views a particular logo or caption related to the ad. The following describes examples of the types of methodologies to be used.
Screening: Brief screening questionnaires will be used to ensure that individuals who complete data collection activities meet approved eligibility criteria. The screening questions will confirm demographic and tobacco use or non-combustible product use, as appropriate to the specific data collection activity.
In-Depth Interviews: Individual in-depth interviews can be conducted in-person or online. These interviews are used to collect information from key informants to elicit attitudes and perceptions that offer insight into critical influences on individual’s belief structures or for pretesting message concepts, draft materials, and communication strategies. Individual in-depth interviews are ideal when the information in question requires in-depth probing or when individual, rather than group, responses are considered more appropriate. This methodology is appropriate for determining intended audience attitudes, beliefs, and feelings.
Focus Groups: Focus groups or group interviews can also be conducted in-person or online. Focus groups are used to obtain insights into intended audience perceptions, beliefs, and attitudes in the early stages of the communication process (i.e., in concept, strategy and materials development) and to understand how individuals discuss a message or advertisement with each other. Focus groups are usually composed of 8 - 12 people who have characteristics similar to the intended audience or subgroups of the intended audience. The groups are conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a structured discussion outline. Focus groups are valuable in exploring consumer reactions to message concepts before additional resources are put into their development.
Surveys: Surveys can be conducted using self-administered online, telephone, or paper-and-pencil questionnaires. A sample of consenting participants is recruited from the intended audience; respondents will be asked to respond to questions regarding their reactions to messaging or ads with respect to the main message, believability, comprehension, perceived effectiveness, and whether it would impact their behavioral intentions regarding tobacco use, non-combustible product use or secondhand smoke. This method of formative testing is not designed to generate nationally representative results or estimates of population parameters but rather is used to test whether messages or ads are credible, comprehensible and persuasive. Surveys can also be conducted for recruitment of real people to appear in advertisements used in future national tobacco education campaigns.
Online bulletin boards: Online bulletin boards may be used to aid in identifying messages and themes that resonate the most with audience members, as well as to identify gaps for the development of new creative materials. Bulletin board sessions typically run over a two- to three-day period, and respondents can respond at their convenience during the session, while a trained live moderator will monitor responses throughout the duration of the bulletin board.
For all methodologies, professionally recognized procedures will be followed in each information collection activity to ensure high quality data. Some examples of these procedures include the following:
Training sessions, supervision and monitoring will be conducted for all data collection efforts in which moderators/interviewers interact with respondents (e.g., focus groups).
Observers will monitor focus groups, focus group proceedings will be recorded, and online technical support will be made available, should the need arise.
An institutional review board approval will be requested, when needed.
Online survey procedures require that respondents answer or explicitly decline each presented question before moving forward. This maximizes the utility of data collected.
Data submitted through online surveys will be subjected to statistical validation techniques (such as disallowing out-of-range values).
All data collection and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.
These information collections are formative work for campaign development and planning and are not intended to generate nationally representative samples or precise estimates of population parameters. In the case of data collection activities that involve interviews or surveys conducted in-person, online, or on the telephone, several procedures can be used to increase responses. Below are a few examples.
Interviewers will participate in thorough training sessions. Training topics will include study objectives, question-by-question reviews of data collection instruments, strategies for engaging respondents, role playing, and techniques for fostering respondent cooperation and survey completion.
Experienced, highly trained staff will moderate all focus groups and bulletin board sessions in-person and online.
After an initial online qualifying screener, all respondents will be rescreened and confirmed via telephone. Prior rounds of data collection indicate that this is a very effective method for keeping high response and show rates for bulletin boards and focus groups.
For bulletin boards, if a respondent has not joined the discussion, they will typically be sent a follow-up email at the start of Day 2.
Potential respondents will be informed through a variety of methods, such as email messages, about the importance of these projects and encouraged to participate.
Content and layout of email invitations will be assessed for clarity to ensure robust response rates.
After the original invitation, respondents who have not completed the survey after 48 hours may receive a reminder invitation (see Attachment 7 for examples).
Creative and attractive graphics will be used to attract the attention of respondents, where relevant.
CDC, a credible agency that serves the public good, will be identified as the agency of record.
Participants will be given incentives, which can include cash, prepaid gift cards, or points that can be redeemed for other items, such as Amazon gift cards; they will be made aware of the type and amount of incentive prior to participating in the study.
This project involves the collection of quantitative and qualitative information. This request is a revision of a previously approved generic clearance package. Similar procedures have already been used to conduct testing through this clearance. Additionally, contractors or CDC staff may pretest or pilot test the instrument(s) and method of data collection, when time permits and if deemed necessary. Lessons from the pretest or pilot test will be identified, and changes, as necessary, will be incorporated into the instrument and method. Pretests and pilot tests will typically involve between 10 and 30 individuals unless OMB clearance is sought for a larger number.
Primary responsibility for data collection and analysis through this clearance will be conducted by Qualtrics and Battelle, whose information is listed below.
Qualtrics
400 Qualtrics Drive
Provo, UT 84604
Phone (800) 340-9194
Email: [email protected]
Battelle
505 King Avenue
Columbus, Ohio 43201
Phone: (800) 201-2011
Email: [email protected]
Individuals consulted at CDC on the study design are listed below.
Centers for Disease Control and Prevention Office on Smoking and Health 4770 Buford Highway, N.E MS F-79 Atlanta, GA 30341 |
||
Name |
Contact Info |
Role |
Brian Armour |
P: 404.498.3014 |
Associate Director for Science |
Elizabeth Courtney-Long |
P: 404.498.0264 |
Health Scientist |
Diane Beistle |
P: 770.488.5066 |
Chief, Health Communication Branch |
Lauren Boyle-Estheimer |
P: 404.498.2283 |
Health Communication Specialist |
Lindsey McCarter |
P: 770.488.4239 |
Health Communication Specialist |
Michelle O’Hegarty |
P: 770.488.5582 |
Health Communication Specialist |
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Supporting Statement B template |
| Subject | Supporting Statement B template |
| Author | Centers for Disease Control and Prevention |
| File Modified | 0000-00-00 |
| File Created | 2024-09-25 |