Message Testing for Tobacco Communication Activities (MTTCA)
Previous Title: “Testing and Evaluation of Tobacco Communication Activities”
New
Supporting Statement: Part B
Centers for Disease Control and Prevention
National Center for Chronic Disease Prevention and Health Promotion
Office of Smoking and Health
Health Communications Branch
December 30, 2011
Project Manager: Robert Alexander, Ph.D.
Health Communications Specialist
Phone: (770) 488-1212
Fax: (770) 488-5939
Email address: [email protected]
Centers for Disease Control and Prevention
4770 Buford Highway, Mailstop K-50
Atlanta, GA 30341
B. Collection of Information Employing Statistical Methods
B.1. Respondent Universe and Sampling Methods
B.2. Procedures for the Collection of Information
B.3. Methods to Maximize Response Rates and Deal with Non-response
B.4. Test of Procedures or Methods to be Undertaken
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS
B.1. Respondent Universe and Sampling Methods
Formative evaluation for health communication campaigns and programs includes various methods and approaches. Methodologies and sample sizes are based on information needed, a review of the relevant literature, consultation with experts in the field, and information gleaned from developing and pretesting tobacco-related campaign or program materials for various audiences.
In general, formative testing relies on qualitative methods and is not intended to yield results that are statistically projectable or used to derive quantitative estimates. However, these methods allow tobacco communication messages and/or programs to be designed and marketed with specific audiences in mind (e.g., 18-49 year old smoking parents, tobacco users who intend to quit, etc.). In qualitative studies, quota sampling is often used to select a convenience sample of individuals who meet certain qualifications that reflect characteristics typical of the target audience. Response rate is not applicable to quota sampling because this type of sampling results in a nonprobability sample which is not representative of the population. In qualitative studies, respondents can be initially contacted by telephone, through the mail or online; over-recruiting is done to compensate for not following up with non-respondents.
Where quantitative methods are used, information collection activities will target particular audiences with statistical sampling procedures employed to identify potential survey respondents. Online surveys for message testing will seek a convenience sample that nonetheless has an acceptable degree of diversity in key demographic characteristics such as age, gender, education, and race/ethnicity. CDC/HCB does not intend to generate nationally representative results or precise estimates of population parameters using these surveys.
Attachment 3 provides an overview of the types of information collections planned for 2012-2014, the methodologies to be employed, and estimates of burden per response and total annualized burden hours.
Specific methods to select respondents will be described in each information collection request submitted for approval under this generic clearance.
B.2. Procedures for the Collection of Information
The methodologies planned for use in this submission will follow standard state-of-the-art approaches adapted from marketing and communications research. In this context, the term pretesting refers to testing messages, strategies, and communication materials. Questions in all pretesting methodologies include standard measures of communications that are designed to assess to what degree the message was successful in communicating information, including main idea recall, comprehension, believability, personal relevance, and likes and dislikes. Additional questions may be added to address any specific concerns regarding a message or advertisement, such as how a respondent views a particular logo or caption related to the ad.
The following describes examples of the type of methodologies to be used.
Overview of Potential Formative Testing Procedures – Interviews, Focus Groups, Bulletin Boards and Surveys
Individual interviews – Individual in-depth interviews are generally conducted in-person or online at a designated Internet location. These interviews are used to collect information from key informants to elicit attitudes and perceptions that offer insight into critical influences on individual’s belief structures or for pretesting message concepts, draft materials, and communication strategies. Individual in-depth interviews are ideal when the information in question requires in-depth probing or when individual, rather than group, responses are considered more appropriate. This methodology is appropriate for determining target audience attitudes, beliefs, and feelings, particularly those addressing potentially sensitive or emotional topics.
Focus Groups – Focus groups or group interviews can also be conducted in-person or online. Focus groups are used to obtain insights into target audience perceptions, beliefs, and attitudes in the early stages of the communication process (i.e., in concept, strategy and materials development) and to understand how individuals discuss a message or advertisement with each other. Focus groups are usually composed of 8 - 12 people who have characteristics similar to the target audience or subgroups of the target audience. The groups are conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a loosely structured discussion outline. Focus groups are valuable in exploring consumer reactions to message concepts before additional resources are put into their development.
Bulletin Boards – Online Bulletin Boards are a form of qualitative data collection used in formative projects and in pretesting to aid in identifying messages and themes that resonate the most with audience members, as well as to identify gaps for the development of new creative materials. Online Bulletin Boards may be composed of different target audience members (e.g., a general population sample of 18-49 year olds from diverse media markets; community opinion leaders). Bulletin board sessions typically run over a two to three day period and respondents can respond at their own convenience during the session, while a trained live moderator will monitor responses throughout the duration of the bulletin board.
Surveys – Surveys can be conducted using self-administered online questionnaires with respondents from online panels to test rough drafts of television, radio ads and print ads. A sample of consenting participants is recruited from the target audience and is shown one or several advertisements. Respondents will respond to questions from an online questionnaire regarding their reactions to the ads with respect to the main message and believability of each ad, comprehensibility, perceived effectiveness and whether the ad would impact their behavioral intentions regarding tobacco use or secondhand smoke. This method of formative testing is not designed to generate nationally representative results or estimates of population parameters but rather is used to test whether advertisements are credible, comprehensible and communicate intended messages.
For all methodologies, professionally recognized procedures will be followed in each information collection activity to ensure high quality data. Some examples of these procedures include the following:
Training sessions, supervision and monitoring is conducted for data collection in which moderators interact with respondents (i.e., focus groups or online bulletin board).
Observers will monitor focus groups, and focus group proceedings will be recorded; and
Online technical support is made available to respondents during research, should the need arise.
Online survey procedures require that respondents must respond or explicitly decline each presented question before moving forward, which ensures that as much information pertinent to the research is provided.
Data submitted through on-line surveys will be subjected to statistical validation techniques (such as disallowing out-of-range values).
All data collection and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.
B.3. Methods to Maximize Response Rates and Deal with Non-response
In the case of data collection activities that involve interviews or telephone, online, and in-person surveys, several procedures can be used to responses. Here are a few examples.
Interviewers will participate in thorough training sessions. Training topics will include study objectives, question-by-question reviews of data collection instruments, strategies for engaging respondents, role playing, and techniques for fostering respondent cooperation and survey completion.
Experienced, highly-trained staff will moderate all focus groups or bulletin board sessions in-person or online.
After an initial online qualifying screener, all respondents are rescreened and confirmed via telephone. We have found that this is a very effective method for keeping high response and show rates for bulletin boards and focus groups.
For bulletin boards, if a respondent has not joined the discussion, we will typically send a follow-up email at the start of day 2.
Potential respondents will be informed about the importance of these projects and encouraged to participate through a variety of methods, which could, for example, include e-mail messages.
Content and layout of email invitations will undergo routine testing to develop best-in-class emails to support robust response rates.
After the original invitation, respondents who have not completed the survey after 48 hours may receive a reminder invitation.
Creative and attractive graphics will be used to attract the attention of respondents, where relevant.
B.4. Tests of Procedures or Methods to be Undertaken
Before each data collection activity is implemented, when time permits, an evaluation contractor or CDC evaluation staff will pre-test or pilot test the instrument(s) and method of data collection. Lessons from the pre-test or pilot test will be identified, and changes, as necessary, will be incorporated into the instrument and method. Pre-tests and pilot tests will typically involve no more than a minimal number of individuals unless OMB clearance is sought for a larger number.
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
OSH policy analysts and statistical contractors may be chosen to collect formative and outcome data and will be responsible for the design of sampling procedures and statistical analyses undertaken as part of these data collection activities.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | goodmann |
| File Modified | 0000-00-00 |
| File Created | 2021-01-31 |