B-0925-0046

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

Pretesting includes various methods and approaches. The methods chosen for use depend on the nature of the message or materials tested, as well as their intended audience. Recommended methodologies and sample sizes are based on a review of the relevant literature, consultation with experts in the field, and a baseline of data gathered over many years of pretesting materials on cancer and other related health topics among professional, patient, and public audiences.

In general, pretesting relies on qualitative methods and is not intended to yield results that are statistically projectable. In qualitative studies, quota sampling is used to select a sample of individuals who meet certain qualifications that reflect characteristics typical of the target audi­ence. Response rate is not applicable to quota sampling because this type of sampling results in a nonprobability sample which is not represen­tative of the population. In qualitative studies, all respondents are initially contacted by telephone or through the mail; over-recruiting is done to compensate for not following up with non-respondents.

Ques­tions in all pretest­ing methodologies include the following:

• Standard measures of communications that are designed to assess to what degree the message was successful in communicating informa­tion. These questions include measures of main idea recall, comprehension, believability, personal relevance, and likes and dislikes.

• Questions tailored for the specific print piece/public service message to assess audience comprehension of specif­ic information or to address any special concerns the producer of the message may have -- for example, are the illustrations appro­priate?

The methodologies planned for use in this pretesting submission represent the standard state-of-the-art approaches adapted from marketing and com­munica­tions research. In this context, the term pretesting refers to testing messages, strategies, and communication materials, and should not be confused with "pretesting of questionnaires" prior to their full-scale use. The follow­ing methodologies will be used:

Individual In-depth Interviews. Individual in-depth interviews are used for pretesting message concepts, draft materials, and communication strategies when the information in question requires in-depth probing or when individual rather than group responses are considered more appropri­ate. This methodology is appropriate for determin­ing target audience attitudes, beliefs, and feelings, particularly those addressing potentially sensitive or emotional topics. In-depth interviews are also cost-effective in elicit­ing comments on print materi­als. Respon­dents can be sent a copy of the material in advance, asked to read it, and told that someone will call to get their opinion. At the designated time, the recipient of the material is called and interviewed.

Respondents for in-depth interviews are recruited to represent members of the target audience for the particular educational message or print material being tested. They are also recruited based on meeting other appropri­ate screen­ing criteria, including their willingness to be inter­viewed. Specific written instructions in the form of a screening question­naire are used during the recruitment process. The interviews themselves are conduct­ed by skilled inter­viewers who follow a prescribed discussion outline. A minimum of 20-25 respondents are queried in pretests using this method of data collec­tion; in a standard pretest, 50 inter­views are conducted. In-depth inter­views are generally 30 to 45 minutes in length.

Intercept Interviews. Intercept interviews involve positioning interviewers at a central point or location commonly used by individuals who comprise the desired target audi­ence. In an intercept interview, people are randomly asked to partici­pate in a message or strategy pretest about health. This method­ology is usually employed when pretest reactions are desired on a non-sensitive topic over a fairly short period of time. Intercept interviews may be done in person or by telephone: a) Central Location Intercepts. In the case of central location inter­cepts, the point of interception is someplace like a shopping mall, a hospital, or a walk-in medical clinic. After several initial screen­ing ques­tions, participants are asked a series of questions, often in relation to a draft material they have been shown. This is followed by an interview of up to 20 minutes in length, depending on the nature and complexity of the topic and material present­ed. b) Telephone Intercept Interviews. With telephone intercepts, the intercept point is usually the terminus of a call-in number (e.g., a 1-800 number). The Cancer Information Service (CIS) has a 1-800 number that receives over 300,000 calls a year on cancer topics. As an add-on to some of these calls, several brief questions about cancer messages will be asked, providing valu­able input on whether NCI cancer prevention and detection messages are under­stood and accepted. These telephone intercept interviews, conducted on a simple random sample basis, may range in length from less than a minute to several minutes.

Self-Administered Questionnaires/Surveys Self-administered questionnaires or surveys will be used to pretest drafts of NCI concepts and materi­als. They can either be mailed to respondents along with the draft being pretested, accessed on-line at a designated Internet location, or distributed to respondents gathered at a central location. When the latter method is employed, people frequenting a central location representative of the target audience (for example, a conference exhibit hall) are randomly stopped and screened to determine whether they meet the pre-determined selection criteria. When this method of pretest­ing is used, at least 50 respondents are included; using approximately 100 respondents is desirable.

Gatekeeper Review. Because public and patient education materials are often distributed to their intended audiences through health professionals or other intermediary organizations, the input of these groups to the concepts and materials is critical to the program's success. As a result, these interme­diaries, or gatekeepers, are often queried through self-administered question­naires as part of the pretesting process. Twenty-five to 50 gatekeep­ers are generally queried as part of a materials pretest. The informa­tion included in the questionnaire is complementary to that request­ed of target audience members.

Omnibus Survey. An omnibus survey is a telephone interview survey in which different organizations add questions to a single questionnaire, thereby sharing the cost. This technique uses random-digit-dialing (RDD) to speak to approxi­mately 1000 adults who are nation­ally representative of the U.S. popula­tion. Because these surveys are conducted on a weekly or bi-weekly basis, they are an efficient way to obtain pretest data from a larger number of consumers in a short period of time. To get such a quick and cost-effective turnaround, however, the vendor can make up to only four callbacks, resulting in a lower response rate than for custom surveys (where additional callbacks are made). It is industry standard to accept a response rate of 50-60 percent for surveys. Generally, for the most cost-effective ap­proach, computer-assisted tele­phone interviewing (CATI) is used to complete the interviews­.

Focus Groups. Focus groups, or group interviews, are used to obtain insights into target audience perceptions, beliefs, and attitudes in the early stages of the communication process (i.e., in concept, strategy and materials develop­ment.) Focus groups are usually composed of 8 - 9 people who have charac­teristics similar to the target audience, or subgroups of the target audience. The groups are conducted by a professional moderator who keeps the session on track while allowing respondents to talk openly and spontaneously. The moderator uses a loosely structured discussion outline, which allows him/her to change direction as the discus­sion unfolds and new topics emerge. Focus groups are valuable in exploring consumer reac­tions to message concepts before additional resources are put into their develop­ment.

Responsibility for collecting and analyzing information obtained through the methodologies described above will rest with OESI’s Senior Analyst, Nina Goodman. All data collec­tion and analysis will be performed in compliance with OMB, Privacy Act, and Protection of Human Subjects requirements.