Supporting Statement Part B
Request for Generic Clearance
QUESTIONNAIRE COGNITIVE
INTERVIEWING AND PRETESTING (ARP/DCCPS/NCI)
(OMB #0925-0589, Expiry 4/30/2014)
This is a revision of a currently approved submission.
Changes are indicated in yellow highlights.
Contact Information:
Gordon Willis, PhD.
Cognitive Psychologist
Applied Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
9609 Medical Center Drive, Rm 3E358,
Bethesda, MD 20892-9762
Phone: 240-276-6788
Fax: 240-276-7906
April 11, 2014
Table of Contents
B. Statistical Methods 1
B.1. Respondent Universe and Sampling Methods 1
B.2. Procedures for the Collection of Information 2
B.3. Methods to Maximize Response Rates and Deal with Nonresponse 6
B.4. Tests of Procedures or Methods to be Undertaken 7
B.5. Individuals Consulted on Statistical Aspects and Individuals
Analyzing Data 8
LIST OF ATTACHMENTS FOR SUPPORTING STATEMENTS A & B
ATTACHMENT 1: Background and History of Cognitive Interviewing
ATTACHMENT 2: List of Approved and Currently Pending Generic Sub-studies from 2011-2014
ATTACHMENT 3: Description of cognitive testing: Willis, G.B. (2005). Cognitive Interviewing. In S.J. Best & B. Radcliff, Polling America: An Encyclopedia of Public Opinion, pp. 92-98. Greenwood Press: Westport, CT.
ATTACHMENT 4: Consultants both Within and Outside of the National Cancer Institute
ATTACHMENT 5: Letter from NIH Privacy Act Officer
ATTACHMENT 6: Sub-study #1 - Reliability of Computer Adaptive Tests (CAT) Study for the NIH-SSA Collaboration to Improve Disability Determination
Sub-study #1_Mini-Supporting Statement A
Sub-study #1_Mini-Supporting Statement B
All Behavioral Health Items
All Physical Function Items
CAT Instrument Screenshot
IRB Review
VR12 Legacy Instrument
Consent
B. STATISTICAL METHODS
B.1. Respondent Universe and Sampling Methods
The purpose of cognitive interviewing, usability testing, focus group, behavior coding, and of survey pretesting in general, is not to obtain statistically representative survey data, but rather to obtain information about the processes people use to answer survey questions, as well as to identify any potential problems in the questions. Data collection procedures for intensive forms of pretesting in particular depart from those of standardized survey data collection. Although survey interviewers strictly adhere to scripted questionnaires, intensive pretest interviews treat draft survey questions as starting points, to initiate a more detailed consideration of the questions themselves: how participants interpret key concepts, their ability to recall the requested information, and the appropriateness of response categories. Because the interviews often generate open-ended verbal responses rather than statistical estimates, results are analyzed using qualitative methodologies. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of surveys.
Whereas survey research employs a quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, pretesting usually employs qualitative methodology and generally relies upon relatively small samples. The objective of cognitive testing in particular is to provide an in-depth exploration of particular concepts, processes and/or patterns of interpretation. Cognitive interviewing (and other pretesting) samples generally do not achieve full inclusion of all social and demographic groups. As a general rule, sample selection is based upon the content of the survey, as well as the purpose and objectives of the particular study. The use of pretesting methods does not preclude basic tabulations of responses. However, these tabulations are used only for purposes of understanding question performance, and do not constitute, in themselves, sources of survey data.
B.2. Procedures for the Collection of Information
Recruitment. Recruitment for intensive interviewing activities, such as cognitive interviews, will normally be by means of advertisements (e.g., newspaper or flyer posted in a public location). Prior experience at Federal and private cognitive laboratories has shown that advertisements in local newspapers and flyers attract a large pool of potential participants. These recruitment mechanisms are productive for obtaining a diverse group of participants to help determine potential sources of error in survey questions. To test questions that are targeted toward specific subgroups, defined by age, race/ethnicity, or other factor, the advertisement or flyer may be developed to specify appropriate participants. For example, if the questionnaire to be tested includes a majority of questions about non-traditional medical providers used by cancer survivors, then the recruitment will target cancer survivors. Specific attachments, such as the advertisement, introduction email, and the script for fielding calls for potential participants will be specified in the sub-study memo submitted to OMB. Direct contact to solicit support from church groups, employers, and/or social or service organizations is occasionally used as possible sources of volunteers. In these cases, a flyer is provided to a contact person who then either posts the flyer or distributes it to members of the organization.
Recruitment for Pilot household interviews will normally be done by identifying individuals who are eligible for the tested survey, normally through convenience sampling. For example, for the testing of a (hypothetical) telephone-based questionnaire on tobacco use in Hispanics, involving 50 completed pretest interviews, the researchers might rely on a sample of telephone numbers of individuals having Hispanic surnames, obtained from a list-sample contractor. For a Pilot test of a household, face-to-face survey questionnaire, interviewers may go door-to-door in several areas (e.g., defined by Census Bureau Tracts) in the Washington, DC, metropolitan area that are selected on the basis of Census data on income and age; the interviewer will simply screen for eligibility at the door, and attempt to conduct the interview, without attempting normal field procedures such as follow-back to refusals or those not at home. For example, for a questionnaire involving opinions concerning the usefulness of cancer screening tests, the interviewer would attempt to administer the questions to any adult (18 or older) currently at home. Potential participants in these activities will be informed that the procedures involve the testing of a survey questionnaire.
Screening and scheduling procedures. For intensive interviewing activities, the first contact with potential research participants occurs in response to flyers or advertisements. Interested persons call a listed telephone number, and usually leave contact information (name and telephone number) on an answering machine. The responsible contact person (usually a contractor to NCI), labeled the Project Recruiter, then calls the person back, gives a brief description of the nature of the study, i.e., one-on-one interview (face-to-face, telephone, self-administered) or focus group, where the interview will take place, video/audio taping procedures, and the remuneration to be offered. First, the Recruiter determines through a brief series of questions whether the volunteer possesses the desired research characteristics (e.g., gender, age, and educational level to avoid interviewing multiple people with very similar demographic characteristics). If the person does and would like to participate, he/she is scheduled for an interview/focus group. Telephone numbers and the minimal demographic information listed earlier are obtained for all scheduled volunteers. For those callers who are ineligible for the study, or no longer interested after hearing a description of the research, no identifying information will be retained.
For Pilot household interviews of household participants, the questionnaires to be tested are generally applicable to the general population, so no special population selection will be necessary. Or, if particular types of individuals are eligible (e.g., females 50 or older), the interview process will make use of the same type of screening questions to be employed subsequently in the fielded survey.
Note that details of recruitment methods for individual sub-studies will be specified in the relevant memos to be submitted for each, to OMB.
Interviewing methods for intensive interviews. After an interview is scheduled, the individual will usually travel to either NCI facilities in Rockville, Maryland, or to local contractor facilities (depending on the location of the contractor – e.g., at Westat offices in Rockville, Maryland). On rare occasions, a participant may be unable to travel to the intended location (for example, an individual may be housebound or have limited mobility). In such cases, the interview may be conducted in their home or at a location normally frequented by the participant, such as a senior center. To reduce the number of "no shows" for cognitive interviews, volunteers scheduled more than a week in advance receive a reminder telephone call by Recruiter a day or two prior to the scheduled interview.
When the participant arrives, he/she is greeted by a project coordinator/ representative, or the interviewer, is brought to the interview room, and asked to read a consent form (these forms will be submitted with each sub-study memo). In the rare instance that the participant consents to the interview, but not to recording it, the session is carried out but is not recorded. If the participant grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will stop the recording equipment, but will ask the participant for consent to retain the interviewing materials and the portion already recorded. If the participant does not give verbal consent, the entire recording will be destroyed immediately subsequent to the interview.
In some sub-studies, a screener and/or a participant data collection sheet will be completed by participants. The purpose of this sheet is to collect recruitment and demographic information on the participant. Participants are also asked to fill-out a receipt form to indicate that they received an incentive. Immediately following the interview, the data collection sheet will be separated from the consent form, so that no demographic information will be associated with the individual’s name. Signed consent forms will be stored separately from any data that are collected by the participant, and will be destroyed within one year after completion of the interview. Again, specific steps to be taken by each project will be specified in the sub-study memos submitted to OMB.
The interview begins with the interviewer reading a more detailed explanation of the purpose and the procedures to be used. Interviewing procedures vary depending on the specific technique to be applied. The selection of technique is in turn determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the concurrent think-aloud interview with interviewer probing (see Attachment 3). In these interviews, participants are presented draft survey questions and asked to think-aloud about how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved.
If possible, the cognitive interview will be conducted in the mode intended for the survey, either face-to-face, telephone, or self-administered (whether paper or web-based). For a telephone interview, the participant is sometimes called from one room by an interviewer located in a second room; and in-person debriefing follows administration of the questionnaire.
If, during the conduct of a cognitive interview, the interviewer notes any overt evidence of excess emotional reaction (e.g., in response to questions that elicit thoughts related to experiences of personal trauma), he/she will stop the interview, and ask if the participant would prefer to terminate the interview, or continue.
Interviewing methods for focus groups. If a participant is scheduled, he/she will travel to the location of the activity (whether NCI, contractor office, or another location). When participants arrive they are greeted by staff working on the project and directed to the interviewing location, where they are individually greeted by the focus group Moderator. Participants will be given a packet containing the consent form and additional forms related to the specific sub-study. Once the forms have been completed, they will be returned to the Focus Group Moderator.
Participants then each receive a separate packet containing a thank-you letter, remuneration, and a copy of the consent form. Participants are then led to the focus group room and are seated around a table. In the rare instance that consent is not granted and an individual cannot participate, he/she will be excused and will receive the agreed upon incentive.
B.3. Methods to Maximize Response Rates and Deal with Nonresponse
Experience with cognitive and other intensive interviewing techniques has shown that advertisements in local newspapers and flyers attract a large pool of potential participants (Willis, 2005, cited previously). These recruitment mechanisms have been productive in the past for obtaining a diverse group of participants to help to determine potential sources of error in survey questions. For those questionnaires that target specific subgroups, special recruitment procedures will be developed to identify participants. Direct contact to solicit support from church groups, employers, and/or social or service organizations will be explored as possible recruitment methods. Also, the offer of remuneration for the laboratory participant's time has been a proven incentive for volunteers to participate in the study.
After laboratory volunteers have been recruited, the probability of the participant failing to show is minimized by making reminder phone calls to volunteers. According to NCHS cognitive laboratory staff, the “show rate” for on-site scheduled interviews has generally been between 80 and 90%, across a wide range of studies.
The notion of response rate, as the term applies to sample surveys, does not similarly pertain to Pilot Household Interviews, as the major objective of that pretesting activity is simply to complete the targeted number of interviews over a wide range of individuals, within a relatively short (several-day) time period. Assessment of representativeness is determined more in terms of the breadth of individuals surveyed, rather than in a way that emphasizes strict representation in a manner that reflects, for example, adherence to Census demographic data. However, analyses of Pilot Household Interviews include basic tabulation of respondent demographic information such as gender and age.
B.4. Tests of Procedures or Methods to be Undertaken
This submission is in itself a request for authorization to conduct tests of procedures and methods – normally those to be used in population surveys. The purpose of cognitive and other forms of intensive interviewing, and of the pretesting methods in general covered by this request, is not to obtain survey data, but rather to obtain information about the processes people use to answer questions as well as to identify any potential problems in the questions. This work has been effective for enhancing the quality of data of Federal surveys cognitively tested by the NCHS Questionnaire Design Research Laboratory (QDRL), the Census Bureau, BLS, and by private research organizations over the past twenty years. The procedures and methodologies to be used by NCI and its contractors are consistent with those of other Federal cognitive testing laboratories.
B.5. Individuals Consulted on Statistical Aspects and Individuals Analyzing Data
The person with overall responsibility for the statistical and technical aspects of the described activities is:
Gordon Willis, Ph.D.
Cognitive Psychologist
Applied Research Program
Division of Cancer Control and Population Sciences
National Cancer Institute
Room 3E358
9609 Medical Center Drive, Rm 3E358, MSC 9762
Bethesda, MD 20892-9762
Phone: 240-276-6788
Fax: 240-276-7906
Page
| File Type | application/msword |
| Author | Vivian Horovitch-Kelley |
| Last Modified By | Vivian Horovitch-Kelley |
| File Modified | 2014-04-11 |
| File Created | 2013-10-24 |