Nccam B

National Center for Complementary and Alternative Medicine

Communications Program Planning and Evaluation

Request for Renewal

0925-0530

Section B

Introduction

For most data collection methodologies proposed for use under this clearance request, no formal statistical methods are employed. In some cases, particularly when adding questions to omnibus surveys, we sample a number of potential respondents, using stratified or random sampling methods.

With each new data collection, a description of the sampling plan, methodologies, procedures, and testing specific to that collection will be included in the submission package described in the Introduction to Section A.

B.1. Respondent Universe and Sampling Methods

As described above, for most data collection methodologies used under this clearance request, no formal statistical methods are employed. An overview of typical recruitment and sample selection methods for the proposed types of data collection follow; details about the respondent universe and sampling methods will be submitted to OMB in conjunction with specific data collection requests.

Individual In-depth Interviews. Individual in-depth interviews are used for pretesting or pilot testing public service announcement concepts, draft materials, and communication strategies when the information in question requires in-depth probing or when individual rather than group responses are considered more appropriate. Respondents for in-depth interviews are recruited to represent members of the target audience for the particular public service message or print material being tested. They are also recruited based on meeting other appropriate screening criteria, including their willingness to be interviewed. Specific written instructions in the form of a screening questionnaire are used during the recruitment process. Nine in-depth interview studies are anticipated over the 3-year clearance period (an average of three studies per year). The scale of the studies will depend on the nature of the data collection. However, it is estimated that each study will include an average of 10 respondents.

Focus Group Studies. Focus groups, or group interviews, are used to obtain insights into target audience perceptions, beliefs, and attitudes in the early stages of the communication process (i.e., in concept, strategy, and materials development), as well as during evaluation of specific program aspects. Focus groups are usually composed of 8 to 10 people who have characteristics similar to the target audience or subgroups of the target audience. Focus group recruitment firms will be responsible for recruitment of respondents based on specifications provided by NCCAM. Specifications will be tailored to the data collection and will include parameters such as:

• Required number of participants

• Demographic characteristics of the target audience

• Required starting sample size (based on estimated response/attrition rate for the population)

• Screening requirements (including articulation screening).

 The proposed data collection estimates a total of six focus group studies (two studies per year).

Intercept Interviews. Intercept interviews involve positioning interviewers at a central point or location commonly used by individuals who comprise the desired target audience. In an intercept interview, people are randomly asked to participate in a message or strategy pretest about health. The point of interception is a place like a shopping mall, a hospital, or a walk-in medical clinic. Participants are selected based on responses to several initial screening questions. A maximum of six intercept interview studies is projected for the proposed evaluation (two studies per year). The individual data collections will include an average of 50 respondents, based on the research topic and/or materials tested. 

Self-administered Questionnaires. Self-administered questionnaires or bounceback cards are mailed to respondents or distributed at a central location, depending on the nature of the data collection. This data collection method is used to pretest OCPL concepts and materials, either independently or in conjunction with intercept studies. Self-administered data collections are also used to posttest materials and messages during the outcome evaluation phase. For the mail methodology, a simple random sampling technique is used to select a representative sample of respondents. If materials are distributed at a central location, individuals who are potentially representative of the target audience are randomly selected and screened to determine if they meet the predetermined selection criteria. A maximum of six self-administered data collections will be conducted for the proposed evaluation (two per year). Some may be conducted in conjunction with the intercept studies. The individual data collections will include an average of 100 respondents, based on the nature of the data collection effort.

Gatekeeper Reviews. Because public and patient education materials are often distributed to their intended target audiences through health professionals or intermediary organizations, the input of these groups to the concepts and materials is critical to the program’s success. As a result, these intermediaries, or gatekeepers, may be queried through methods such as in-depth interviews or self-administered questionnaires as part of the pretesting or program outcome evaluation process. The information included in the questionnaire is complementary to that requested of target audience members. A sample of “gatekeepers” (e.g., health professionals or other intermediary organizations) will be randomly selected, particularly during pretests and posttests of selected OCPL concepts and materials. A simple random sampling technique will be used to select a sample of respondents who are representative of the respective target population. Three gatekeeper review studies are proposed for this data collection (one per year). A total of 150 individual gatekeeper reviews are estimated for the overall evaluation. It is anticipated that the majority of reviews (100) will be conducted during the needs assessment/product development and pretesting phase.

Omnibus Surveys. An omnibus survey is a telephone interview survey in which different organizations add questions to a single questionnaire, thereby sharing the cost. NCCAM’s proposed plan assumes the addition of two or three customized items to existing omnibus surveys. The quantitative omnibus surveys will employ the most rigorous sampling procedures of all of the proposed data collection methodologies. Omnibus surveys use random digit dialing to complete interviews with approximately 1,000 adults who are representative of the U.S. population.

 

B.2. Information Collection Procedures/Limitations of the Study

General information about data collection procedures for each of the proposed data collection types is included in the subsections that follow. Again, specific information will be submitted to OMB with each request for a specific data collection effort.

B.2.1. Statistical Methodology for Stratification and Sample Selection

No statistical methodologies or quantitative analyses are used in conjunction with most of the data collection types described here (i.e., individual in-depth interviews, focus group interviews, intercept interviews, or gatekeeper reviews).

Individual In-depth Interviews. Since respondents are purposefully selected, and qualitative methods are not intended to yield results that are statistically projectable, sampling stratification procedures are not applicable to these studies. Quota sampling is used to select a sample of individuals who meet certain qualifications that reflect characteristics typical of the desired target audience. Further, standard response rate calculations are not applicable because quota sampling results in a nonprobability sample that is not representative of the population.

Focus Group Studies. To allow for geographic dispersion and inherent differences in the target population, three separate focus group discussions are projected for each study. Consistent with standard focus group methodology, each focus group will include 8 to 10 respondents. Similar to the in-person interviews, respondents are purposefully selected. Therefore, sampling stratification procedures are not applicable. Again, quota sampling is used to select a sample of individuals who meet qualifications that reflect characteristics typical of the target audience of interest.

Intercept Interviews. Although participants are randomly selected for screening and interviewing, rigorous sampling procedures are not typically used in intercept studies.

Self-administered Questionnaires. A random number generator will be used for the mail methodology (e.g., a specific proportion of individuals who request via NCCAM's clearinghouse that  information be mailed to them about a specific topic will also receive a questionnaire). For the central location methodology, participants will be self-selecting (e.g., the questionnaire will be part of a selection of materials about NCCAM and CAM available at NCCAM's exhibit booth at a conference).

Gatekeeper Reviews. These reviews will generally take the form of in-depth interviews or self-administered questionnaires. (See above for information on these methods.)

Omnibus Surveys. Details about the rigorous sampling procedures to be used will be included with each request submitted to OMB for specific omnibus survey questions.

B.2.2. Estimation Procedure

Quantitative analyses will not be conducted for most information collected under this generic clearance as described earlier.  Due to the nature of the sampling and data collection procedures described in Sections B.1 and B.2, when quantitative analyses are performed, estimates will take the form of simple frequency tables of respondent counts for each survey item.

B.2.3. Degree of Accuracy Needed for the Purpose Described in the Justification

For most data collection efforts covered under this request; results will be analyzed qualitatively rather than quantitatively. For formative research and evaluation of  NCCAM's communication activities, the precision levels described earlier in Section B.2 provide sufficient accuracy.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

No unusual problems requiring specialized sampling procedures are anticipated.

B.2.5. Use of Periodic (Less Frequent Than Annual) Data Collection Cycles

Data will not be collected periodically from the same respondent. Each respondent to the data collections is expected to respond only once to one of the collections.

B.3. Methods for Maximizing the Response Rate and Addressing Issues of Nonresponse

Because the focus of most data collection efforts will be qualitative, followup of nonrespondents will not generally be required.

Individual In-depth Interviews.  Respondents are overrecruited by approximately 25 percent to allow for nonresponse.

Focus Group Studies. For these studies, respondents are overrecruited by 25 to 50 percent, based on the target audience.

Intercept Interviews. Individuals are interviewed as soon as they agree to take part in the data collection, so overrecruitment is not required.

Self-administered Questionnaires. These questionnaires will typically be provided to potential respondents in conjunction with specific written material(s) about a topic in which the potential respondents have expressed interest. They will use a simple format and be easy to fill out in order to place minimal burden on respondents.

Gatekeeper Reviews. Gatekeeper reviews will generally take the form of in-depth interviews or self-administered questionnaires. (See above for details on maximizing response rates for these types of data collection.)

Omnibus Surveys. To achieve a quick and cost-effective turnaround, only one callback is made, resulting in a lower response rate than for custom surveys (where multiple callbacks are made). It is the industry standard to accept a response rate of 50 to 60 percent for omnibus surveys. Therefore, a starting sample of at least 2,000 adults 18 years or older is required for each survey. Generally, to minimize respondent burden and employ the most cost-effective approach, computer-assisted telephone interviewing is used to complete the interviews.

B.4. Tests of Procedures or Methods

Each data collection instrument will be pretested with up to 9 contractor staff to estimate response time and refine the wording and design of the instrument as needed.

B.5. Names and Telephone Numbers of Individuals Consulted

Contract personnel at Aspen Systems / Lockheed Martin and at GYMR will implement sample design, conduct data collection, conduct data management and analysis, and develop statistical reports. NCCAM staff will provide direction and review contractor functions. Individuals consulted on statistical aspects of the design of this data collection include the following: 

Nancy Hassett
Senior Statistician
Aspen Systems / Lockheed Martin
Phone:  301-519-6372
E-mail: [email protected]

Marie Pogozelski
Manager, Survey Operations
Aspen Systems / Lockheed Martin
Phone: 301-519-6377
E-mail: [email protected]

Ellen Berman
Former Project Director, NCCAM Clearinghouse
Aspen Systems / Lockheed Martin
Phone:  301-519-5492
E-mail: [email protected]