OMB clearance package Section B 11-19-07

Supporting Statement Section B

Table of Contents

B. Collection of Information Employing Statistical Methods 20

B. 1. Respondent Universe and Sampling Methods 23

B. 2. Procedures for the Collection of Information 24

B. 3. Methods to Maximize Response Rates and Deal with Non-Response 26

B. 4. Test of Procedures or Methods to Be Undertaken 27

B. 5. Individuals Consulted on Statistical Aspects and Individuals Collecting

and/or Analyzing Data 27

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

The methodologies planned for use in this pretesting submission represent the standard state-of-the-art approaches adapted from marketing and com­munica­tions research. In this context, the term pretesting refers to testing messages, strategies, and communication materials, and should not be confused with "pretesting of questionnaires" prior to their full-scale use.

Various qualitative and quantitative methods are used for pretesting: The methods chosen for use depend on the nature of the message, materials or other strategies tested, their intended audience(s), and the research questions. For each type of pretest proposed, recommended methodologies and sample sizes are based on a review of the relevant literature, consultation with experts in the field, and a baseline of data gathered over many years of pretesting concepts on cancer, HIV and other related health topics among professional, patient, and public audiences.

Each method planned for use under this submission is briefly described in the following sections.

Individual In-depth Interviews. Individual in-depth interviews are used for pretesting message concepts, draft materials, and communication strategies when the information in question requires in-depth probing or when individual rather than group responses are considered more appropri­ate. This methodology is appropriate for determin­ing target audience attitudes, beliefs, and feelings, particularly those addressing potentially sensitive or emotional topics. In-depth interviews are also cost-effective in elicit­ing comments on print materi­als. Respon­dents can be sent a copy of the material in advance, asked to read it, and told that someone will call to get their opinion. At the designated time, the recipient of the material is called and interviewed.

Respondents for in-depth interviews are recruited to reflect members of the target audience for the particular educational message, print material or program activity being tested. They are also recruited based on meeting other appropri­ate screen­ing criteria, including their willingness to be inter­viewed. Specific written instructions in the form of a screening question­naire are used during the recruitment process. The interviews themselves are conduct­ed by skilled inter­viewers who follow a prescribed discussion outline. A minimum of 20-25 respondents are queried in pretests using this method of data collec­tion; in a standard pretest, 50 inter­views are conducted. In-depth inter­views are generally 30 to 45 minutes in length.

Intercept Interviews. Intercept interviews involve positioning interviewers at a central point or location commonly used by individuals who comprise the desired target audi­ence, such as a shopping mall, a hospital, or a walk-in medical clinic. This method­ology is usually employed when pretest reactions are desired on a non-sensitive topic over a fairly short period of time. In an intercept interview, people are randomly asked to partici­pate in a message or strategy pretest about health. After several initial screen­ing ques­tions, participants are asked a series of questions, often in relation to a draft material they have been shown. Total interview length is up to 20 minutes, depending on the nature and complexity of the topic and material present­ed.

Self-Administered Questionnaires/Surveys Self-administered questionnaires or surveys can either be mailed to respondents along with the draft being pretested, accessed on-line at a designated Internet location, or distributed to respondents gathered at a central location. When the latter method is employed, people frequenting a central location representative of the target audience (for example, a conference exhibit hall) are randomly stopped and screened to determine whether they meet the pre-determined selection criteria. When this method of pretest­ing is used, at least 50 respondents are included; using approximately 100 respondents is desirable. Self-administered surveys can also be used to gain reactions to entire websites or messages contained on particular pages of websites.

Gatekeeper Review. Because NHVREI messages and materials will often be distributed to their intended audiences through health professionals or other intermediary organizations, the input of these groups to the concepts and materials is critical to the initiative's success. As a result, these interme­diaries, or gatekeepers, are often queried through self-administered question­naires as part of the pretesting process. Up to 50 gatekeep­ers may be queried as part of a materials pretest, though there may be as few as 10-15, depending on how the material will be distributed. The informa­tion included in the questionnaire is complementary to that request­ed of the intended audience members.

Telephone Survey. At times, telephone surveys are used to gather information on an audience’s existing knowledge, attitudes, beliefs, practices, awareness of health topics or specific messages, or preferences among specific messages. Telephone surveys typically involve approxi­mately 1000 adults who are representative of the U.S. popula­tion or specific subgroups of it. Computer-assisted tele­phone interviewing (CATI) is used to complete the interviews­.

Focus Groups. Focus groups discussions are used to obtain insights into target audience perceptions, beliefs, and attitudes in the early stages of the communication process (i.e., in concept, strategy and materials develop­ment.) Focus groups are usually composed of 8 - 9 people who have charac­teristics similar to the target audience, or subgroups of the target audience. The groups are conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a loosely structured discussion outline, which allows him/her to change direction as the discus­sion unfolds and new topics emerge. Focus groups are valuable in exploring consumer reac­tions to message concepts before additional resources are put into their develop­ment.

B.1. Respondent Universe and Sampling Methods

The respondent universe for each pretest will be the universe of individuals for whom the strategy is designed, which is individuals at risk of contracting HIV/AIDS in some or all of the following population groups: Blacks/African Americans, Hispanics/Latinos, and men who have sex with men (MSM) of all racial/ethnic groups, and key influencers of these populations. The universe for gatekeeper reviews will be the gatekeepers for a particular concept, such as health professionals for materials to be distributed in primary care settings or communication directors of community-based organizations or voluntary associations.

When pretests use qualitative methods, such as focus groups, telephone or in-person in-depth interviews, and gatekeeper reviews, the nature of the data collection is such that results would not be statistically projectable to the respondent universe regardless of sampling method. In qualitative studies, quota sampling is used to select a sample of individuals who meet certain qualifications that reflect characteristics typical of the target audi­ence. Response rate is not applicable to quota sampling because this type of sampling results in a nonprobability sample; every member of the universe does not have an equal chance of being sampled.

B.2. Procedures for the Collection of Information

For studies employing quota sampling, respondents are initially contacted by telephone, in person, or through postal or electronic mail and asked to complete an interviewer- or self-administered questionnaire to determine if they possess the target audience characteristics of interest. Those who do are informed of the purpose of the research and the date, time and location of data collection (or asked what date and time is convenient is for them) and asked if they wish to participate; over-recruiting is done to compensate for not following up with non-respondents. For data collection activities taking place in the future, respondents are sent a written reminder of the date, time and location via electronic mail, facsimile, or postal mail, along with procedures to follow if their participation needs to be canceled or rescheduled.

For studies using a probability sample, sampling techniques appropriate for the sampling frame and desired result will be employed. For example, to conduct a projectable survey of the U.S. population, the vendor fielding the survey would use a fully replicated, stratified, single-stage, random-digit-dialing sample of telephone households. Within each sample household, one adult respondent would be randomly selected using a computerized procedure based on the “Last Birthday” method of respondent selection. Following data collection, the results would be weighted to account for disproportionate probabilities of household selection due to the number of separate telephone lines and the probability associated with random selection of an individual household member. Data will then be stratified and balanced based on the national distribution of the relevant population along variables such as age, sex, race and ethnicity, educational attainment, and U.S. Census region to ensure reliable and accurate representation of persons in the target population. 

For all data collection efforts, data will be collected by trained interviewers (or focus group moderators for focus groups). Ques­tions in all pretest­ing methodologies include the following:

• Standard measures of communications that are designed to assess to what degree the message was successful in communicating informa­tion. These questions include measures of main idea recall, comprehension, believability, personal relevance, and likes and dislikes.

• Questions tailored for the specific print piece/public service message to assess audience comprehension of specif­ic information or to address any special concerns the producer of the message may have -- for example, are the illustrations appro­priate?

B.3. Methods to Maximize Response Rates and Deal with Non-Response

The intended use of pretesting data is to gather sufficient information to revise messages, materials or distribution strategies prior to final production, not to develop reasonably precise point estimates of the population.

Qualitative Pretests. Qualitative methods are often most appropriate for pretests when the objective is to explore reactions to particular message concepts or materials. Such methods are inherently not projectable to or representative of any population. Pretests using qualitative methods and central-location intercepts use quota samples without an enumerated sampling frame; it is not possible to calculate response rates for such nonprobability samples. For respondents recruited in advance of a pretest, such as those for focus groups or in-depth interviews, adequate sample size is maintained by recruiting more participants than the desired sample size.

Quantitative Pretests. Response rates for pretests using an enumerated sampling frame and probability sampling techniques will be calculated using American Association of Public Opinion Research (AAPOR) standard definitions of response rates in use when the survey is conducted.

Response rates that provide adequate accuracy and reliability for communication pretests are lower than those necessary for “influential information,” e.g., information that will “have a clear and substantial impact on important public policies or important private sector decisions” as described in OMB and agency Information Quality Guidelines. To keep data collection costs low enough to be an appropriate investment and to receive data in a timely fashion, communication pretests typically employ RDD samples and fewer contact attempts than surveys collecting influential information. It is industry standard to accept a response rate of 50-60 percent for communication pretests. The additional costs associated with obtaining higher response rates are not justified given how the resulting data are used.

B.4 Test of Procedures or Methods to Be Undertaken

Data collection instruments are typically constructed using standard items previously used in health message and materials pretests conducted by other NIH Institutes and Centers or other governmental or nongovernmental organizations. When new questions are developed, they are administered to a small number of individuals (typically less than 10) to assess the need for refinement.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Individuals consulted in the design of methodologies are listed in Attachment 1. In general, the contractor collecting and analyzing information for the NHVREI will be the Academy for Educational Development (AED); for particular pretests conducted under this clearance requiring resources unavailable within AED, competitive subcontracts will be awarded. Responsibility for collecting and analyzing information obtained through the methodologies described above will rest with AED. All data collec­tion and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.

19

Pretesting of NIAID’s HIV Vaccine Research Communications Messages