Health Message Testing System (HMTS)
OMB No. 0920-0572
Expiration Date: October 31, 2024
Part B
Project Manager: Dawn B. Griffin
Health Communication Specialist
Office of Communication
Phone: (404) 498-0129
Email address: [email protected]
Centers for Disease Control and Prevention
1600 Clifton Road, MS H19-B
Atlanta GA 30345
September 30, 2024
Section
Page
B. Collection of Information Employing Statistical Methods 3
B.1. Respondent Universe and Sampling Methods 5
B.2. Procedures for the Collection of Information 6
B.3. Methods to Maximize Response Rates and Deal with No Response.. 6
B.4. Test of Procedures or Methods to be Undertaken 20
B.5. Statistical Consultants 20
B. Collection of Information Employing Statistical Methods
In general, the HMTS will not employ statistical methods to sample respondents. This introductory section justifies the decision not to use statistical sampling and analysis for the data collection methods that HTMS will employ. The same justifications apply to the focus group methodology.
The HMTS methods are:
Intercept interviews of up to 300 participants (a number that allows for the inclusion of members of several different sub-cultural groups).
Telephone surveys of 300 members of target audience segments, secondary audiences, or gatekeepers (analogous to consumer satisfaction interviews in length and purpose).
Individual in-depth (cognitive) interviews of 200 individuals using probing techniques to develop a deeper understanding of how respondents interpret a message.
Focus groups of nine or less respondents. Like the focus groups in all HMTS activities to date, the anticipated data collections will be small in scale because they are intended to inform an iterative process of developing health communication campaign messages, not to be generalized to a specified respondent universe.
Online surveys of 400 individuals, often administered to members of professional organizations.
These audience-specific techniques rely not on statistical power but on the theoretical premise that language is interpreted through shared cultural knowledge and frameworks.1, 2 To increase the likelihood that a message will be noticed, to avoid miscommunication, and to guard against insensitivity in specialized communication to sub-cultural groups, the proposed data gathering techniques provide “…a ’window’ on a particular worldview.” (Priest, p. 114)
By incorporating qualitative and quantitative elements in various mixtures, these techniques allow the flexibility for in-depth, individualized probing, can be feasible in time-sensitive situations, and have worked well in message development for commercial advertising. 3
Intercept Interviews
Trained staff will approach people in convenient, high volume areas (e.g., shopping malls, health clinics, etc.) and determine whether they match criteria specified in screening questions that identify members of a particular target audience. In health facilities, participants might be pre-screened through clinic records and sent invitational letters through their health care providers. Clinic staff may encourage clients to participate at their convenience, either before or after their medical appointments. Information tables with attractive displays are often set up to stir interest.
Interviewers will describe the study to eligible individuals as explained in Section A11, and offer them a small incentive to participate, as explained in Section A9. However, the default for the government is not to offer incentives for interviews and surveys.
Eligible, interested participants will be interviewed on the spot, or given instructions about how to get to a nearby location, such as a movie theater, where the survey will be administered. Whenever possible, participants will be guided through the brief interview process by touch-screens or other user-friendly strategies.
When gatekeeper interviews are conducted in person, they fall under the category of central location intercept interviews because respondents are identified through a convenient, central source, often a professional association or other social network. Because public and patient education materials are often distributed to their intended target audiences through health professionals or other intermediaries, their reactions to the concepts and formats of these materials can be critical to a program’s success.
Ordinarily, interviewers working in partnership with a professional group would travel to a location convenient to the gatekeepers such as a work site or professional meeting, explain the study, offer a small incentive for participation, and guide respondents through a very brief questionnaire that might include questions about physical samples of mocked-up communication materials.
Telephone Interviews
When members of a target audience can be identified and reached by telephone, phone interviews may be conducted to test messages. Gatekeepers of information, such as physicians or members of the press, and groups who have influence with a particular audience of interest, so-called “secondary” audiences, could also be interviewed by phone.
The total number of telephone phone interviews for most tests is likely to remain well under 50 per audience segment; normally more than one segment would be interviewed. The universe would be narrowed by means of marketing databases and organizational lists, and eligible individuals would be identified by screening questions. The purpose of the data gathering activity would be explained, but it is unlikely that an incentive would be offered.
Potential respondents can be identified by census clusters and random calling procedures would be followed to draw a small number of individuals from this pool. Individuals who agree to participate may be sent materials to examine before the interview is conducted. A call-back may be scheduled to conduct the survey at a time designated as convenient by the respondent.
Individual In-depth Interviews (Cognitive Interviews)
Cognitive interviewing or cognitive assessment is a technique used to probe a respondent to identify a deeper understanding of a particular response or answer to a survey question. Questions are usually open-ended and give the respondent an opportunity to fully express their thoughts without being influenced by the constraints of a close-ended question.
Two types of cognitive interviewing can be used; think aloud interviewing and verbal probing techniques. The think aloud technique asks a respondent to verbalize their thought process as he answers each survey question. An alternative to this method is the verbal probing technique which allows an interviewer to ask for more specific information relevant to the question or to the given response.
Cognitive interviews can be used successfully in a variety of settings with diverse populations. When studying under-represented populations, such as low socioeconomic status individuals or those with less education, cognitive interviews create space for survey questions to be better understood by the respondents. Based on the latest health literacy estimates, these individuals most likely will have trouble completing a written or online survey.
Also, the interaction inherent in cognitive interviewing can improve the interviewer’s understanding of the subject’s cognitive processes and the decision-making process behind a given response.
Focus Groups
Online Focus Group - Nine or fewer participants will be recruited. Focus groups will be conducted over the Internet, in the format of a moderated chat room. A moderator will guide a real-time discussion, but will also allow the direction of the discussion to take its own course. For online focus groups, participants will type in their comments, which can be seen by all other participants as soon as they are entered.
Traditional Focus Groups - Traditional focus groups are face-to-face versions of the procedure described above. Nine or fewer participants engage in a verbal discussion that is partly directed by a moderator’s prompts and partly by the comments of other participants. Notes or tapes are made, tapes are often transcribed, and themes are drawn from the notes or transcripts.
Online Surveys
Audiences that have ready access to the Internet could be interviewed at their place of employment or homes. Potential respondents would be contacted via traditional mail and/or Email, or invited to participate when they visit Web sites hosted by CDC or its partners. The initial contact would inform the participants about the study and provide a password and instructions for the online survey. Depending on the duration of data collection, multiple email reminders may be sent to respondents.
For all methods described, the message dimensions of interest are:
Comprehension – Is the message clearly understood? Is the audience able to identify and recount the intended main messages? Is the intended information presented in a manner that makes it effective and actionable for the intended audience?
Liking – How much does the audience like the presentation? What elements do they especially like? Which do they dislike?
Personal Relevance - Do respondents perceive the messages as relevant to themselves personally, as well as to their peers? In other words, does the message resonate with their personal perceptions and experience?
Believability – Is the message and/or its source perceived as credible? Does it portray the message realistically and convincingly?
Acceptability – Is there anything in the message that is perceived as offensive or unacceptable to either the primary or secondary audiences? In particular, do parents have significant concerns about ads intended for youth audiences?
Behavioral Intent – Do respondents think they will take action as a result of seeing/hearing the message?
Participation will be encouraged by:
Stressing the value of the interview as a means of helping to understand the characteristics of the health problem or prevention message.
Keeping the interviews short.
Providing the opportunity for respondents to call a toll-free line maintained by the contractor to receive assurance regarding the authenticity of the survey.
B.1. Respondent Universe and Sampling Methods
Any adult, non-incarcerated, non-institutionalized member of a target audience for any of CDC’s numerous health messages could be a respondent in some study falling within the purview of the HMTS.
B.2. Procedures for the Collection of Information
Many of the studies under HMTS will generate qualitative, top line reports and will not involve inferential statistics. For others, basic descriptive statistical analyses will be conducted with software for manipulation and tabulation of survey data. For example, for response categories including Strongly Agree, Agree Somewhat, Disagree Somewhat and Strongly Disagree, mean, standard deviation, and standard error of the mean might be calculated.
In trend-tracking polls and some other infrequent circumstances, larger samples will be needed so that significance tests can be conducted for differences in means and in proportions among the demographic subgroups. A discussion of power for these tests can be found in Cohen’s Statistical Power Analysis for the Behavioral Sciences.4
In the comparison of means using a two-sided two group t-test with alpha = .05, the number of cases required to detect a medium effect size with a power of .80 is 64 cases per group. Relative effect size is defined by Cohen as a difference between the two means that is equal to one half of the size of the pooled within-group standard deviation (Ch. 2, Section 2.2). Such a difference is the equivalent of a point-biserial correlation between group membership and dependent variable of r = .243, or a proportion of variance accounted for of 5.9%. Such a test of means could be used on responses considered to be continuous in nature.
In comparison of two independent proportions, using a two sided normal curve test applied to the arcsine transformation of the proportions with alpha = .05, the number of cases required to detect a “medium effect size” with a power of .80 is also 64 cases per group. This relative effect size is a difference between proportions of about .15 at the ends of the 0 - 1 interval and of about .25 in the mid-range. It also corresponds to a four-fold point correlation, or phi coefficient, between group and dichotomous dependent variable varying from r = .238 to r = .248, with a proportion of variance accounted for of approximately 5.9%.
The majority of the procedures to be used by HMTS are already well-established, and the online panel approach has been used with increasingly frequency. Studies using this method have passed scientific peer review,5 and this method of data collection is now used by major commercial research firms such as Harris Interactive (www.harrisinteractive.com), Knowledge Networks (www.intersurvey.com/ganp/), and MSInteractive (www.cinfo.com).
Qualitative and quantitative pilot studies of the online panel approach were requested by OMB as a condition of approval of the original version of this package, and they indicated that the online approach is appropriate for both qualitative and quantitative data collection.
B.5. Individual(s) Consulted on Statistical Aspects and Individual(s) Collecting and/or Analyzing Data
Elizabeth (Betsy) Mitchell, Associate Director for Communication Science, 404.498.0251, [email protected], from the Office of the Associate Director for Communications assisted by statisticians in the CDC units conducting the data collections and contract personnel, will consult on statistical issues when inferential statistics are used.
Most HMTS studies will not employ complicated inferential statistics.
1 Priest, S.H. (1996) Doing Media Research: An Introduction. Thousand Oaks California: Sage Press.
3 Sen, A.K. (1982) Bank uses mall-intercept interviews to test ad concepts, Marketing News, Chicago, 15(15), p. 20.
4 Cohen, J. (1988) Statistical Power Analysis for the Behavioral Sciences, 2nd ed., Hillsdale, NJ, Lawrence Erlbaum Associates
5 Schlenger, W.E., Caddell, J.M., Ebert, L., Jordan, K., Rourke, K.M., Wilson, D., Thalji, L., Dennis, M., Fairbank, J.A., & Kulka, R.A. (2002) Psychological reactions to terrorist attacks: Findings from the National Study of Americans’ reactions to September 11, JAMA, 288(5), 581-588,
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Center for Disease Control and Prevention and |
| Author | Angela Ryan |
| File Modified | 0000-00-00 |
| File Created | 2024-10-28 |