Ssb_10-31-07

Supporting Statement Part B

Request for Generic Clearance

QUESTIONNAIRE COGNITIVE

INTERVIEWING AND PRETESTING (ARP/DCCPS/NCI)

Contact Information:

Gordon Willis, PhD.

Cognitive Psychologist

Applied Research Program

Division of Cancer Control and Population Sciences

National Cancer Institute

6130 Executive Blvd, MSC 7344

EPN 4005

Bethesda, MD 20892

301-594-6652

301-435-3710 (fax)

[email protected]

October 15, 2007

Table of Contents

B. Statistical Methods 1

B.1. Respondent Universe and Sampling Methods 1

B.2. Procedures for the Collection of Information 2

B.3. Methods to Maximize Response Rates and Deal with Nonresponse 7

B.4. Tests of Procedures or Methods to be Undertaken 8

B.5. Individuals Consulted on Statistical Aspects and Individuals

Analyzing Data 9

LIST OF ATTACHMENTS FOR SUPPORTING STATEMENTS A & B

ATTACHMENT A: Description of cognitive testing: Willis, G.B. (2005). Cognitive Interviewing. In S.J. Best & B. Radcliff, Polling America: An Encyclopedia of Public Opinion, pp. 92-98. Greenwood Press: Westport, CT.

ATTACHMENT B-1: Sample questionnaire instrument to be tested: Initial draft of 2010-2011 Tobacco Use Supplement to the Current Population Survey

ATTACHMENT B-2: Sample questionnaire instrument to be tested: Sample items on Cancer Quality of Life

ATTACHMENT C: Sample advanced letter to respondents, based on that used for the 2005 HINTS

ATTACHMENT D: Letter from NIH Privacy Act Officer

ATTACHMENT E-1: Assurance of Confidentiality and Informed Consent templates for cognitive interviews

ATTACHMENT E-2: Assurance of Confidentiality and Informed Consent templates for focus groups

ATTACHMENT F: Form for Consent for Expanded Use of Video and Audio Recordings

ATTACHMENT G: NCI Office of Human Subjects Research Exemption Approval

ATTACHMENT H: Sample advertisement (for newspaper or flyer)

ATTACHMENT I: Sample screening script for newspaper/flyer recruitment

ATTACHMENT J: Sample script for answering machine

ATTACHMENT K: Participant Data Collection Sheet

ATTACHMENT L: Detailed explanation of cognitive interviewing procedure read by interviewer to participant

B. STATISTICAL METHODS

B.1. Respondent Universe and Sampling Methods

The purpose of cognitive interviewing, of behavior coding, and of survey pretesting in general, is not to obtain statistically representative survey data, but rather to obtain information about the processes people use to answer survey questions, as well as to identify any potential problems in the questions. Data collection procedures for intensive forms of pretesting in particular depart from those of standardized survey data collection. Although survey interviewers strictly adhere to scripted questionnaires, intensive pretest interviews treat draft survey questions as starting points, to initiate a more detailed consideration of the questions themselves: how participants interpret key concepts, their ability to recall the requested information, and the appropriateness of response categories. Because the interviews often generate open-ended verbal responses rather than statistical estimates, results are analyzed using qualitative methodologies. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of surveys.

Whereas survey research employs a quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, pretesting usually employs qualitative methodology and generally relies upon relatively small samples. The objective of cognitive testing in particular is to provide an in-depth exploration of particular concepts, processes and/or patterns of interpretation. Cognitive interviewing (and other pretesting) samples generally do not achieve full inclusion of all social and demographic groups. As a general rule, sample selection is based upon the content of the survey, as well as the purpose and objectives of the particular study. The use of pretesting methods does not preclude basic tabulations of responses. However, these tabulations are used only for purposes of understanding question performance, and do not constitute, in themselves, sources of survey data.

B.2. Procedures for the Collection of Information

Recruitment. Recruitment for intensive interviewing activities, such as cognitive interviews, will be by means of advertisements (newspaper or flyer posted in a public location). Prior experience at Federal and private cognitive laboratories has shown that advertisements in local newspapers and flyers attract a large pool of potential participants. These recruitment mechanisms are productive for obtaining a diverse group of participants to help determine potential sources of error in survey questions. To test questions that are targeted toward specific subgroups, defined by age, race/ethnicity, or other factor, the advertisement or flyer may be developed to specify appropriate participants. For example, if the questionnaire to be tested includes a majority of questions about non-traditional medical providers used by cancer survivors, then the recruitment will target cancer survivors. A sample advertisement is enclosed as Attachment H; and a sample script for fielding calls for interested potential participants is in Attachment I. Direct contact to solicit support from church groups, employers, and/or social or service organizations is occasionally used as possible sources of volunteers. In these cases, a flyer is provided to a contact person who then either posts the flyer or distributes it to members of the organization.

Recruitment for Pilot household interviews will normally be done by identifying individuals who are eligible for the tested survey, normally through convenience sampling. For example, for the testing of a (hypothetical) telephone-based questionnaire on tobacco use in Hispanics, involving 50 completed pretest interviews, the researchers might rely on a sample of telephone numbers of individuals having Hispanic surnames, obtained from a list-sample contractor. For a Pilot test of a household, face-to-face survey questionnaire, interviewers may go door-to-door in several areas (e.g., defined by Census Bureau Tracts) in the Washington, DC, metropolitan area that are selected on the basis of Census data on income and age; the interviewer will simply screen for eligibility at the door, and attempt to conduct the interview, without attempting normal field procedures such as follow-back to refusals or those not at home. For example, for a questionnaire involving opinions concerning the usefulness of cancer screening tests, the interviewer would attempt to administer the questions to any adult (18 or older) currently at home. Potential participants in these activities will be informed that the procedures involve the testing of a survey questionnaire.

Screening and scheduling procedures. For intensive interviewing activities, the first contact with potential research participants occurs in response to flyers or advertisements. Interested persons call a listed telephone number, and usually leave contact information (name and telephone number) on an answering machine (Attachment J contains a sample script of the answering machine message). The responsible contact person (usually a contractor to NCI), labeled the Project Recruiter, then calls the person back, gives a brief description of the nature of the study, i.e., one-on-one interview (face-to-face, telephone, self-administered) or focus group, where the interview will take place, video/audio taping procedures, and the remuneration to be offered. First, the Recruiter determines through a brief series of questions (Attachment I) whether the volunteer possesses the desired research characteristics (e.g., gender, age, and educational level to avoid interviewing multiple people with very similar demographic characteristics). If the person does and would like to participate, he/she is scheduled for an interview/focus group. Telephone numbers and the minimal demographic information listed earlier are obtained for all scheduled volunteers. For those callers who are ineligible for the study, or no longer interested after hearing a description of the research, no identifying information will be retained.

For Pilot household interviews of household participants, the questionnaires to be tested are generally applicable to the general population, so no special population selection will be necessary. Or, if particular types of individuals are eligible (e.g., females 50 or older), the interview process will make use of the same type of screening questions to be employed subsequently in the fielded survey.

Interviewing methods for intensive interviews. After an interview is scheduled, the individual will usually travel to either NCI facilities in Rockville, Maryland, or to local contractor facilities (depending on the location of the contractor – e.g., at Westat offices in Rockville, Maryland). On rare occasions, a participant may be unable to travel to the intended location (for example, an individual may be housebound or have limited mobility). In such cases, the interview may be conducted in their home or at a location normally frequented by the participant, such as a senior center. To reduce the number of "no shows" for cognitive interviews, volunteers scheduled more than a week in advance receive a reminder telephone call by Recruiter a day or two prior to the scheduled interview.

When the participant arrives, he/she is greeted by a project coordinator/ representative, or the interviewer, is brought to the interview room, and asked to read the Assurance of Confidentiality and Informed Consent (Attachment E-1 or E-2). In the rare instance that the participant consents to the interview, but not to recording it, the session is carried out but is not recorded. If the participant grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will stop the recording equipment, but will ask the participant for consent to retain the interviewing materials and the portion already recorded. If the participant does not give verbal consent, the entire recording will be destroyed immediately subsequent to the interview.

Attached to the Assurance of Confidentiality and Informed Consent Form is Attachment K, the Participant Data Collection sheet that participants are asked to complete. The purpose of this sheet is to collect recruitment and demographic information on the participant. Participants are also asked to fill-out a receipt form to indicate that they have received payment. Immediately following the interview, the data collection sheet will be separated from the consent form, so that no demographic information will be associated with the individual’s name. Signed consent forms will be stored separately from any data that are collected by the participant, and will be destroyed within one year after completion of the interview.

The interviewer begins with the interviewer reading a more detailed explanation of the purpose and the procedures to be used (see Attachment L). Interviewing procedures vary depending on the specific technique to be applied. The selection of technique is in turn determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the concurrent think-aloud interview with interviewer probing (see Attachment A). In these interviews, participants are presented draft survey questions and asked to think-aloud about how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved.

If possible, the cognitive interview will be conducted in the mode intended for the survey, either face-to-face, telephone, or self-administered (whether paper or web-based). For a telephone interview, the participant is sometimes called from one room by an interviewer located in a second room; and in-person debriefing follows administration of the questionnaire.

In addition to providing consent for the interview to be taped at the beginning of the interview, the participant may be asked at the end of the interview, after receipt of remuneration, to sign Attachment F, the Consent for Expanded Use of Video and Audio Recordings. The purpose of this form is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique or especially interesting problem with the questionnaire, and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. The form is designed at an 8th grade reading level. If the participant signs this form, he/she will be given a copy.

If, during the conduct of a cognitive interview, the interviewer notes any overt evidence of excess emotional reaction (e.g., in response to questions that elicit thoughts related to experiences of personal trauma), he/she will stop the interview, and ask if the participant would prefer to terminate the interview, or continue. Interviewers will have access to the telephone number of a local (Montgomery County, Maryland) crisis response hotline, which can be given to the participant for use before he/she leaves the interview situation, or for use at a later time. The Crisis Center is available 24 hours a day, seven days a week to assist individuals as they encounter personal crises, and is staffed by a team of professionals including counselors, clinical social workers, psychologists and psychiatrists.

Interviewing methods for focus groups. If a participant is scheduled, he/she will travel to the location of the activity (whether NCI, contractor office, or another location). When participants arrive they are greeted by staff working on the project and directed to the interviewing location, where they are individually greeted by the focus group Moderator. Participants will be given a packet containing the Assurance of Confidentiality and Informed Consent, as fashioned for focus group interviews (Attachment E-2), and the Participant Data Collection Sheet (Attachment K). Once the forms have been completed, they will be returned to the Focus Group Moderator.

Participants then each receive a separate packet containing a thank-you letter, remuneration, and a copy of the Assurance of Confidentiality and Informed Consent. Participants are then led to the focus group room and are seated around a table. In the rare instance that consent is not granted and an individual cannot therefore participate, he/she will be excused, but will receive remuneration.

A project staff member will moderate the focus group. Before discussion begins, the Moderator will distribute name tags and will ask participants to choose a name to put on the name tag (participants are told that they do not have to use their real names). The Moderator will then describe the process of the focus group and ask if there are questions. After all questions are answered, the Moderator will then begin the focus group discussion following a Moderator guide designed for that particular study.

B.3. Methods to Maximize Response Rates and Deal with Nonresponse

Experience with cognitive and other intensive interviewing techniques has shown that advertisements in local newspapers and flyers attract a large pool of potential participants (Willis, 2005, cited previously). These recruitment mechanisms have been productive in the past for obtaining a diverse group of participants to help to determine potential sources of error in survey questions. For those questionnaires that target specif­ic subgroups, special recruitment procedures will be developed to identify participants. Direct contact to solicit support from church groups, employers, and/or social or service organizations will be explored as possible recruitment methods. Also, the offer of remuneration for the laboratory participant's time has been a proven incentive for volunteers to participate in the study.

After laboratory volunteers have been recruited, the probability of the participant failing to show is minimized by making reminder phone calls to volunteers. According to NCHS cognitive laboratory staff, the “show rate” for on-site scheduled interviews has generally been between 80 and 90%, across a wide range of studies.

The notion of response rate, as the term applies to sample surveys, does not similarly pertain to Pilot Household Interviews, as the major objective of that pretesting activity is simply to complete the targeted number of interviews over a wide range of individuals, within a relatively short (several-day) time period. Assessment of representativeness is determined more in terms of the breadth of individuals surveyed, rather than in a way that emphasizes strict representation in a manner that reflects, for example, adherence to Census demographic data. However, analyses of Pilot Household Interviews include basic tabulation of respondent demographic information such as gender and age.

B.4. Tests of Procedures or Methods to be Undertaken

This submission is in itself a request for authorization to conduct tests of procedures and methods – normally those to be used in population surveys. The purpose of cognitive and other forms of intensive interviewing, and of the pretesting methods in general covered by this request, is not to obtain survey data, but rather to obtain information about the processes people use to answer questions as well as to identify any potential problems in the questions. This work has been effective for enhancing the quality of data of Federal surveys cognitively tested by the NCHS Questionnaire Design Research Laboratory (QDRL), the Census Bureau, BLS, and by private research organizations over the past twenty years. The procedures and methodologies to be used by NCI and its contractors are consistent with those of other Federal cognitive testing laboratories.

B.5. Individuals Consulted on Statistical Aspects and Individuals Analyzing Data

The person with overall responsibility for the statistical and technical aspects of the described activities is:

Gordon Willis, Ph.D.

Cognitive Psychologist

Applied Research Program

Division of Cancer Control and Population Sciences

National Cancer Institute

6130 Executive Blvd, MSC 7344

EPN 4005

Bethesda, MD 20892-7344

[email protected]

Page 2