Supporting
Statement B
for
Request for Generic Clearance
Cognitive
Testing of Instrumentation and Materials for the Population
Assessment of Tobacco and Health (PATH) Study (NIDA)
Kevin P. Conway, Ph.D.
Deputy Director
Division of Epidemiology, Services, and Prevention Research
National Institute on Drug Abuse
6001 Executive Blvd., Room 5185
Rockville, MD 20852
Phone: 301-443-8755
Email: [email protected]
Section Page
B. STATISTICAL METHODS 1
B.1 Respondent Universe and Sampling Methods 1
B.2 Procedures for the Collection of Information 1
B.3 Methods to Maximize Response Rates and Deal with Nonresponse 4
B.4 Tests of Procedures or Methods to be Undertaken 4
B.5 Individuals Consulted on Statistical Aspects and Individuals Analyzing Data 5
LIST OF ATTACHMENTS
(In order of appearance in Supporting Statement B)
Attachment 2. Description of Cognitive Interviewing/Testing 3
Attachment 5. Examples of Consent Materials 3
Attachment 7. List of Cognitive Testing Statistical Consultants 5
The purpose of cognitive interviewing, and of survey pretesting in general, is not to obtain statistically representative survey data, but rather to obtain information about the processes people use to answer survey questions, as well as to identify any potential problems in the questions. Data collection procedures for intensive forms of pretesting depart from those of standardized survey data collection. Although survey interviewers strictly adhere to scripted questionnaires, intensive pretest interviews treat draft survey questions as starting points, to initiate a more detailed consideration of the questions themselves: how participants interpret key concepts; their ability to recall the requested information; and the appropriateness of response categories. Because the interviews often generate open-ended verbal responses rather than statistical estimates, results are analyzed using qualitative methodologies. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of surveys.
Whereas survey research employs a quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, pretesting usually employs qualitative methodology and generally relies upon relatively small purposive samples. The objective of cognitive testing is to provide an in-depth exploration of specific concepts, processes and/or patterns of interpretation. Cognitive interviewing (and other pretesting) samples generally do not achieve full inclusion of all social and demographic groups. As a rule, sample selection is based upon the content of the survey, as well as on the purpose and objectives of the study. For the PATH study, efforts will be made to include current and former tobacco users as well as non-tobacco users, a diverse range of types of tobacco users (e.g., cigarette, pipe, dissolvable tobacco users), youth ages 12 to 17 years, and adults ages 18 years and older. The use of pretesting methods does not preclude basic tabulations of responses. However, such tabulations would only be used for purposes of understanding question performance; they would not, in themselves, constitute sources of survey data.
Recruitment. Recruitment for intensive interviewing activities, such as cognitive interviews, will normally be by means of advertisements (e.g., Craigslist, newspapers or flyer posted in a public location). Prior experience at federal and private cognitive laboratories has shown that these advertisements attract a large pool of potential participants. Such recruitment mechanisms are productive for obtaining a diverse group of participants to help determine potential sources of error in survey questions. To test questions targeted to specific subgroups, (i.e., by age, race/ethnicity, or other factors), the advertisement or flyer may be developed to specify appropriate participants. Direct contact with church groups, employers, and/or social or service organizations is occasionally used as a way to recruit potential participants. In these cases, a flyer will be provided to a contact person who then either posts it in a prominent location or distributes it to members of the organization. Each request submitted under this generic clearance will include attachments of the advertisements, introduction email, or scripts for fielding calls for potential participants pertaining to that request as well as details on its specific recruitment methods and sample sizes.
Screening and Scheduling Procedures. For intensive interviewing activities, the first contact with potential research participants occurs in response to flyers or advertisements. Interested persons will call a listed telephone number, and usually leave contact information (name and telephone number) on voicemail. The responsible contact person, labeled the Project Recruiter, will call the person back, give a brief description of the nature of the study (e.g., face-to-face, telephone, self-administered interviewing), where the interview will take place, video/audio taping procedures, and the incentive to be offered for completing an interview. First, the recruiter will determine through a brief series of questions whether the participant possesses the desired research characteristics (e.g., gender, age, and educational level); also, the recruiter will identify and exclude an individual who is enrolled in the main PATH study. If the participant does meet the selection criteria and agrees to complete a cognitive interview, he/she will be scheduled for an interview. Telephone numbers and minimal demographic information (e.g., gender, age, and race/ethnicity) are obtained for all scheduled participants. For those callers who are ineligible for the study (e.g., because they are under 12 years old or not members of a subgroup of interest), or no longer interested after hearing a description of the research, no identifying information will be retained. Recruitment data will be maintained in separate, encrypted files, with password protection and access limited to authorized personnel. Any hard-copy data will be stored in locked files at the contractor’s facilities.
Interviewing Methods for Intensive Interviews. After an interview is scheduled, the participant will usually travel to the local contractor facilities. On rare occasions, a participant may be unable to travel to the intended location (e.g., an individual may be housebound or have limited mobility). In such cases, the interview may be conducted in his/her home or at a location normally frequented by the participant, such as a senior center. To reduce the number of "no shows" for cognitive interviews, participants scheduled more than a week in advance will receive a reminder telephone call by the recruiter a day or two prior to the scheduled interview.
When the participant arrives, he/she will be greeted by a project coordinator/representative or the interviewer. The participant will then be brought to the interview room and asked to read a consent form. Examples of these forms are presented in Attachment 5. The consent forms and, as needed, parental consent and youth assent forms, will be included with each request submitted to OMB under this generic clearance. In the rare instance that the cognitive testing participant consents to the interview, but not to recording it, the session will be carried out but not recorded. If the participant grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will stop the recording equipment, but will ask the participant for consent to retain the interviewing materials and the portion already recorded. If the participant does not give oral consent, the entire recording will be destroyed immediately subsequent to the interview.
The interview will begin with the interviewer reading a more detailed explanation of the purpose and the procedures to be used. Interviewing procedures vary depending on the specific technique to be applied. The selection of technique is determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the concurrent think-aloud interview with interviewer probing (see Attachment 2). In these interviews, participants are presented draft survey questions and asked to think aloud about how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved.
If possible, the cognitive interview will be conducted in the mode intended for the instrument, either face-to-face; telephone; or self-administered via paper or computer-administered, such as audio computer-assisted self-interviewing (ACASI) or web-based interviewing. For an ACASI interview, for example, the participant would complete items on a computer with a headset and an interviewer would ask questions about his/her experience with the computer and about the content of the items they answered, and may observe the participant as they interact with the computer.
During the conduct of a cognitive interview, if the interviewer observes emotional stress in the participant (e.g., in response to questions that elicit thoughts related to experiences of personal trauma), he/she will stop the interview and ask if the participant would like to take a break, terminate the interview, or continue.
As described in section A.9, participants may receive an incentive as a thank you for their participation. They will be asked to sign a receipt form indicating receipt of the remuneration. Immediately following the interview, the data collection sheet will be separated from the consent form, so that no demographic information will be associated with the individual’s name. Signed consent forms will be stored separately from any data collected from the participant. Again, specific steps to be taken by each project will be specified in the request submitted to OMB under this generic clearance.
Experience with cognitive and other intensive interviewing techniques has shown that advertisements in local newspapers and flyers attract a large pool of potential participants. These recruitment mechanisms have been shown to be effective for obtaining a diverse group of participants for a variety of research studies such as will be conducted as part of this generic clearance. For those PATH study instruments that target specific subgroups, special recruitment procedures will be developed to identify potential participants. Direct contact to solicit support from church groups, employers, and/or social or service organizations will be explored as possible recruitment methods. Modest remuneration will be used to thank participants for their time and contributions to the cognitive testing interview.
After cognitive testing participants have been recruited, the probability of failure to show up at the appointed time will be minimized by making reminder phone calls. According to staff at the Questionnaire Design Research Laboratory (QDRL) of the National Center for Health Statistics (NCHS), the “show rate” for on-site scheduled interviews has generally been between 80 and 90 percent, across a wide range of studies.
This submission is a request for authorization to conduct tests of data collection procedures, materials, and instruments to be used to inform the population-based PATH study. The purpose of cognitive interviewing is to obtain information about the cognitive processes people use when answering questions and to identify any potential problems in the wording of particular questions. This work has been effective for enhancing the quality of data of Federal surveys cognitively tested by the NCHS QDRL, the Census Bureau, the Bureau of Labor Statistics, and by private research organizations over the past 20 years. The procedures and methodologies to be used are consistent with those of other Federal cognitive testing laboratories.
Consultants from the prime contractor’s instrument design, evaluation, and analysis group will provide statistical and technical expertise for the cognitive testing of PATH study materials and instruments (see Attachment 7).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Ilene Klein |
| File Modified | 0000-00-00 |
| File Created | 2021-01-30 |