B. STATISTICAL METHODS
B.1. Respondent Universe and Sampling Methods
The purpose of cognitive interviewing is not to obtain survey data, but rather to obtain information about the processes people use to answer survey questions, as well as to identify any potential problems in the questions.
Data collection procedures for cognitive interviewing are quite different from survey interviewing. While survey interviewers strictly adhere to scripted questionnaires, cognitive interviewers use survey questions as starting points to begin a more detailed discussion of questions themselves such as: how participants interpret key concepts, their ability to recall the requested information, and the appropriateness of response categories. Because the interviews generate narrative responses rather than statistics, results are analyzed using qualitative methodologies. This type of in-depth analysis reveals problems in particular survey questions and, as a result, can help to improve the overall quality of surveys.
While survey research employs a deductive, quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, cognitive testing usually employs an inductive, qualitative methodology and generally relies upon a relatively small sample. Unlike survey research, the primary objective of cognitive testing is not to produce statistical data that can be generalized to an entire population. Rather, the objective of cognitive testing is to provide an in-depth exploration of particular concepts, processes and/or patterns of interpretation. Cognitive interviewing samples generally do not achieve full inclusivity of all social and demographic groups. As a general rule, sample definitions are based upon the content of the survey, as well as the purpose and objectives of the particular study.
Whenever possible and appropriate, laboratory interviews may also rely on quantitative analysis, e.g., whether participants were assigned to random groups to receive different versions of questionnaires, or engage in well-established laboratory tests such as “card sorts.” Cognitive interviewing also does not preclude basic tabulations of responses, especially when samples are relatively large.
For the Pilot Household Interviewing study, participants will be sampled from household units in unused Census Bureau area segments in local communities, or, if conducted by telephone, will be selected through random-digit dialing.
B.2. Procedures for the Collection of Information
Recruitment. Participants will usually be recruited by means of flyers and other advertisements posted in public places, newspaper advertisements, or word-of-mouth. Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential participants. These recruitment mechanisms have been productive in the past for obtaining a diverse group of participants to help us determine potential sources of error in survey questions. To test questions that are targeted toward specific subgroups, the advertisement or flyer may be developed to identify appropriate participants. For example, if the questionnaire to be tested includes a majority of questions about asthma-related health behaviors, then the recruitment may target asthma sufferers. A sample recruitment flyer and a sample newspaper advertisement are enclosed as Attachments L and M. Direct contact to solicit support from church groups, employers, and/or social or service organizations is occasionally used as possible sources of volunteers. In these cases, a flyer is provided to a contact who then either posts the flyer or distributes it to members of the organization.
For the pilot household interviewing studies, such as those done for NHIS, participants were sampled from household units in unused Census Bureau area segments in local communities. Census Bureau field interviewers followed their standard procedure by walking through neighborhoods, knocking on doors, and utilizing an advance letter (See Attachment C for copy of an Advance letter that is sent to actual survey respondents) to find persons willing to participate.
b) Screening and scheduling procedures. The first contact with potential laboratory research participants occurs in response to flyers or advertisements. Interested persons leave contact information (name and telephone number) on an answering machine. The Laboratory Manager then calls the person back, gives a brief description of the nature of the study, i.e., one-on-one interview (face-to-face, telephone, self-administered) or focus group, where the interview/focus group takes place, video/audio taping procedures, and the remuneration to be offered. First, the Laboratory Manager determines through a brief series of questions (Attachment K) whether the volunteer possesses the desired research characteristics (e.g., we ask for gender and age to avoid interviewing 8 people with very similar demographic characteristics). If the person does possess the desired research characteristics and would like to participate, he/she is scheduled for an interview/focus group. Otherwise, the volunteer is asked whether he/she would be interested in participating in future laboratory interviews. Telephone numbers and the minimal demographic information listed earlier are obtained for all scheduled volunteers and for those who would like to be contacted in the future. For those callers who are ineligible for the study and do not want to be contacted in the future, only demographic characteristics will be maintained for future analysis of successful recruitment efforts. Attachment N contains a sample script of the answering machine message. Attachment K contains a sample screening script.
For pilot interviews of household participants, the questionnaires to be tested are applicable to the general population, so no special population selection will be necessary. Any adult household participant is eligible for the interview.
QDRL Interview Methods.
i) One-on-one Interviews. If a laboratory interview is scheduled, the individual will usually travel to the QDRL. The QDRL is located on the 6th floor in the NCHS office building in Hyattsville, Maryland. The lab contains a waiting room, a control room, two private interview rooms, and a remote observation room for use by survey sponsors and other researchers to view an interview while in progress. On rare occasions, a participant may be unable to travel to the laboratory for the interview (for example, an individual may be housebound or have very limited mobility). In such cases, the interview may be conducted in their home or at a location normally frequented by the participant, such as a senior center. To reduce the number of "no shows" for laboratory interviews, volunteers scheduled more than a week in advance receive a reminder telephone call by the Laboratory Manager before the day of interview.
When the participant arrives at the QDRL, he/she is greeted by the Laboratory Manager and asked to read a brief description of the study, which includes assurances of confidentiality and the legislative authority for the research (Attachment E). The need for recording the interview (audio or video) is explained and the participant is asked to sign a consent form. The form is designed at an 8th grade reading level. In the rare instance that consent is not granted, the session is not recorded in audio or video, depending upon the individual’s concern area. If the participant grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will ask for verbal consent to retain the interviewing materials and the portion already taped. The interviewer will get verbal consent from the participant to do so prior to turning off the machine. If the participant does not give verbal consent the interviewer will mark the tape for erasure, and the interviewing materials will be destroyed. If offsite, the interviewer will give the tape to the Laboratory Manager for erasure upon their return to NCHS. Otherwise the tape will be erased and the interviewing materials destroyed within 24-hours of the interview.
Attached to the Assurance of Confidentiality and Informed Consent Form is Attachment F, the Participant Data Collection sheet that participants are asked to complete. The purpose of this sheet is to collect recruitment and sociodemographic information on the participant. Participants are also asked to fill-out the SF-1164 Payment form.
On occasion, sponsors requesting cognitive testing on sensitive topics (e.g., HIV testing behaviors, smoking behaviors in American Indians) require that we do not collect personal identifiers (name, address, telephone number, and social security number) which the QDRL routinely collects in order to 1) pay respondents through the SF-1164 form and 2) to acquire informed consent. It is the sponsor’s belief that collection of these identifiers would put the participant at risk of potential harm resulting from breach of confidentiality. In these cases, the QDRL has made arrangements with the CDC Budget Office not to use the SF-1164. In its place, a project specific tabular form is used to collect participant number, date, time, and location of the interview. Two interviewers sign-off certifying that payment in the advertised amount has been made to the participant.
In addition, the QDRL requests a waiver of signed informed consent from the NCHS ERB.
The interviewer then begins the interview by reading a more detailed explanation of the purpose of the interview and the procedures to be used (see Attachment O). Interviewing procedures vary depending on the specific laboratory technique to be applied. The selection of the laboratory technique is in turn determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the cognitive interview with concurrent probing. In these interviews, participants are presented draft survey questions and asked to explain how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved. The interviewer may also ask the participant to think aloud while answering.
If possible, the cognitive interview will be conducted in the mode intended for the survey, either face-to-face, Computer Assisted Personal Interviewing (CAPI), telephone, Computer Assisted Telephone Interviewing (CATI), self-administered or web-based. For a telephone interview, the participant is called from one lab room to another and in-person debriefing follows.
In addition to consenting for the interview to be taped at the beginning of the interview, the participant may be asked at the end of the interview, and after receipt of remuneration, to sign Attachment G, the Special Consent for Expanded Use of Video and Audio Recordings. The purpose of the special consent is to allow for the playing of tapes at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. If the participant does sign the special consent form, he/she will be given a copy which contains contact information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.
The participant may also be asked by the interviewer to sign Attachment I, Special Consent to Send Video and Audio Recordings to Off-site Researchers. The purpose of this form is to allow permission to send the recording via Federal Express to off-site researchers working on the project so they may view the recording at their location. Offsite-researchers requesting the recordings would sign a contract with NCHS stating how they will protect QDRL participant’s privacy and the recording until it is returned to NCHS. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. If the participant does sign the special consent to send video and audio recordings form, he/she will be given a copy which contains information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.
ii) Focus Group Methods. Participants generally need to travel to the focus group location, which could be at NCHS, another Federal agency, or a room at another institution.
When participants arrive they are greeted by staff working on the project and directed to the focus group room where they are individually greeted by the Laboratory Manager/Staff person. Participants are given a packet containing the Assurance of Confidentiality and Informed Consent for focus groups (Attachment E), the Participant Data Collection Sheet (Attachment F), the SF-1164 Payment form and instructed to fill them out. To maintain confidentiality, participants are seated at separate tables. Once the forms have been completed, they will be returned to the QDRL manager for completeness, verification and safekeeping.
Participants then each receive a separate packet containing a thank-you letter signed by the Director of NCHS, their remuneration, and a copy of the Assurance of Confidentiality and Informed Consent for focus groups (Attachment E). Participants are then ushered into the focus group room and are seated around a table. In the rare instance that consent is not granted, participants will still receive remuneration.
A QDRL staff member or person working on the project, as outlined in the Assurance of Confidentiality and Informed Consent, will moderate the focus group. Before discussion begins, the moderator will distribute name tags and will tell participants to pick a name to put on the name tag. Participants will be told that they do not have to use their real names. The moderator will then describe the process of the focus group and ask if there are any questions. After all questions are answered, the moderator will then begin the focus group discussion following the moderator guide designed for that particular study.
In addition to consenting for the discussion to be taped at the beginning of the focus group, participants may be asked, at the close of the discussion, to sign Attachment H, the Special Consent for Expanded Use of Video and Audio Recordings for Individual Participants of Discussion Groups. The purpose of the special consent is to allow for the playing of tapes at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); tapes of interviews with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. Participants will be given a copy of the form which contains contact information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer. If any one participant from the focus group does not grant special consent, the tape will not be used in this way.
Once the focus group has concluded, the QDRL staff member or person working on the project will usher participants to the elevator and take them to the lobby exit.
Pilot Household Interview Methods.
For each of the tested questionnaires, between 50 and 100 household interviews will be conducted. These interviews will be conducted at such a time that the questionnaire to be tested has been developed in the laboratory, but has not yet been field pre-tested. Questionnaires will be combined for this testing when they are at similar levels of development, or where it is logically dictated that they should be combined, such as for the appropriate NHIS Core and Topical Modules. Questionnaires will be either draft paper versions, or draft CAPI instruments contained on laptop computers.
For each pilot test to be conducted, approximately five professional field interviewers having experience administering the particular survey to be tested will be assigned on a temporary (one week) basis to NCHS for purposes of carrying out these interviews. Households for these interviews will be selected randomly within a prescribed interviewing area (defined according to Census tract location), or through random-digit dialing; these methods have been found to be successful in the past. For all household and telephone interviews administered in this manner, the field interviewer will follow approved informed consent and survey administration procedures specific to the survey being tested. However, these procedures may be slightly modified for QDRL testing to reflect “participation in the testing of survey questions” rather than “participation in the actual survey.” Individual project protocols submitted for NCHS ERB approval will specify whether participation is for an actual survey or the testing of survey questions. If, at the time of pilot household testing, the survey’s informed consent and administration procedures are in the process of being amended, previous approved procedures will be modified for use. Individual 10-day notices submitted for OMB approval will outline the informed consent procedures and survey administration procedures specifically modified for QDRL testing. If time allows, prescripted follow-up probes may be asked to clarify participants’ responses. An NCHS staff member or a staff member of the agency sponsoring the questionnaire will observe as many of the interviews as possible. The results of pilot household testing will be used to modify the questionnaire for the later field pretest.
B.3. Methods to Maximize Response Rates and Deal with Nonresponse
Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential participants. These recruitment mechanisms have been productive in the past for obtaining a diverse group of participants to help us determine potential sources of error in survey questions. For those questionnaires that target specific subgroups, special recruitment procedures will be developed to identify participants. Direct contact to solicit support from church groups, employers, and/or social or service organizations will be explored as possible recruitment methods. Also, the offer of remuneration for the laboratory participant's time has been a proven incentive for volunteers to participate in the study.
After laboratory volunteers have been recruited, the probability of the participant failing to show is minimized by making reminder phone calls to volunteers.
B.4. Tests of Procedures or Methods to be Undertaken
This submission is a request for authorization to conduct tests of procedures and methodologies typical in cognitive testing research. The purpose of cognitive testing is not to obtain survey data, but rather to obtain information about the processes people use to answer questions as well as to identify any potential problems in the questions. This work has been effective for enhancing the quality of data of CDC and other Federal surveys cognitively tested by the Questionnaire Design Research Laboratory (QDRL) over the past 20 years. The procedures and methodologies used by the QDRL are consistent with other cognitive testing laboratories such as the Census Bureau, and the Bureau of Labor Statistics.
B.5. Individual Consulted on Statistical Aspects and Individuals and/or Analyzing Data
The person with overall responsibility for the statistical and technical aspects of the described activities is:
Lawrence H. Cox, Ph.D.
Associate Director, Office of Research and Methodology
Room 3211
National Center for Health Statistics
3311 Toledo Road
Hyattsville, Maryland
(301) 458-4631
| File Type | application/msword |
| Author | kls6 |
| Last Modified By | kls6 |
| File Modified | 2006-11-08 |
| File Created | 2006-11-08 |