Supporting Statement B for Request for Clearance:
QUESTIONNAIRE DESIGN RESEARCH LABORATORY
2015-2018
OMB No. 0920-0222
Contact Information:
Karen R. Whitaker, B.S.
Laboratory Manager, Questionnaire Design Research Laboratory
Office of Research and Methodology
National Center for Health Statistics/CDC
3311 Toledo Road, Room 3215
Hyattsville, MD 20782
301-458-4569
301-458-4031 (fax)
May 28, 2015
Table of Contents
B. Collections of Information Employing Statistical Methods
B.1. Response Universe and Sampling Methods 3
B.2. Procedures for the Collection of Information 3
B.3. Methods to Maximize Response Rates and Deal with Nonresponse 9
B.4. Tests of Procedures or Methods to be Undertaken 9
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting
and/or Analyzing Data 9
B. STATISTICAL METHODS
1. Respondent Universe and Sampling Methods
Given the variety of individual collections that we anticipate will fall under this generic clearance, we will have a number of different respondent universes and employ different sampling methods, depending on the specific purpose of the collection. However, there are some generalities about the universes and how we will sample them based on whether the proposed collection uses qualitative or quantitative methodologies.
a) Qualitative Collections. While survey research employs a deductive, quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, methods such as cognitive interviewing employs an inductive, qualitative methodology and generally relies upon a relatively small sample. Unlike survey research, the primary objective of the qualitative methods the National Center for Health Statistics (NCHS) and the Questionnaire Design Research Laboratory (QDRL) employ is not to produce statistical data that can be generalized to an entire population. Rather, their objective is to provide an in-depth exploration of particular concepts, processes and/or patterns of interpretation. Samples used for qualitative research generally do not achieve full inclusivity of all social and demographic groups. As a general rule, sample definitions are based upon the content of the survey, as well as the purpose and objectives of the particular study.
b) Quantitative Collections. In the instances when NCHS/QDRL conducts field tests/pilot tests, non-purposive sampling methods will normally be used. Again, the universe for these studies will depend on the overall research purpose, but will tend to be geographically larger than those used in the qualitative studies. Sampling will also depend on the specific project. For instance, when conducting a field test for the National Health Interview Survey NHIS, an address-based sample might be used. Across the quantitative research we anticipate the QDRL to accomplish under this clearance, both probability (i.e. Address Based Sampling (ABS), Random Digit Dialing (RDD), etc.) and non-probability (i.e. web panels) samples may be drawn.
2. Procedures for the Collection of Information
Recruitment. Laboratory respondents will usually be recruited by means of flyers and other advertisements posted in public places, newspaper advertisements, or word-of-mouth. Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential respondents. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. To test questions that are targeted toward specific subgroups, the advertisement or flyer may be developed to identify appropriate respondents. For example, if the questionnaire to be tested includes a majority of questions about asthma-related health behaviors, then the recruitment may target asthma sufferers. A sample recruitment flyer and a sample newspaper advertisement are enclosed as Attachments J and K. Direct contact to solicit support from church groups, employers, and/or social or service organizations is occasionally used as possible sources of volunteers. In these cases, a flyer is provided to a contact that either posts the flyer or distributes it to members of the organization.
For some field tests/pilot interviewing, such as those done for NHIS, addresses will be pre-identified. Interviewers (such as those contracted through the Census Bureau, or through other outside contractors) will use the same procedures as used in the actual survey and acquire informed consent (via, for instance, the NHIS Advance Letter) using the HIS-100C, Manual for NHIS Field Representatives. See Attachment M for a copy of the advance letter. Other field tests will use other forms of sampling, as discussed above. For instance, some research may use an online platform and sample (and recruit) respondents through an existing email list, while other research may use RDD to recruit telephone respondents.
b) Screening and scheduling procedures. The first contact with potential laboratory research respondents occurs in response to flyers or advertisements. Interested persons leave contact information (name and telephone number) on the QDRL voice mail system. The QDRL Recruiter/QDRL Staff person then calls the person back, gives a brief description of the nature of the study, i.e., one-on-one interview (face-to-face, telephone, self-administered) or focus group, where the interview/focus group takes place, video/audio recording procedures, and the remuneration to be offered. First, the QDRL Recruiter/QDRL Staff person determines through a brief series of questions (Attachment D) whether the volunteer possesses the desired research characteristics (e.g., we ask for gender and age to avoid interviewing people with very similar demographic characteristics). If the person does possess the desired research characteristics and would like to participate, he/she is scheduled for an interview/focus group. Otherwise, the volunteer is asked whether he/she would be interested in participating in future laboratory interviews. Telephone numbers and the minimal demographic information listed earlier are obtained for all scheduled volunteers and for those who would like to be contacted in the future. For those callers who are ineligible for the study and do not want to be contacted in the future, only demographic characteristics will be maintained for future analysis of successful recruitment efforts. Attachment L contains a sample QDRL voice mail script . Attachment D contains a sample screening script.
For field test/pilot interviews, the questionnaires to be tested are generally applicable to the general population, so no special population selection will be necessary. Any adult household respondent is eligible for the interview. It is possible that the QDRL may be asked to conduct a field test with a specialized population (practicing doctors, for instance). In this case, a screening plan will be developed and included in that project’s 10-day letter.
c) QDRL Interview Methods.
i) One-on-one Interviews. If a laboratory interview is scheduled, the individual will usually travel to the QDRL. The QDRL is located in the NCHS office building in Hyattsville, Maryland. The lab contains a waiting room, a control room, two private interview rooms, and a remote observation room for use by survey sponsors and other researchers to view an interview while in progress. On rare occasions, a respondent may be unable to travel to the laboratory for the interview (for example, an individual may be housebound or have very limited mobility). In such cases, the interview may be conducted in their home or at a location normally frequented by the respondent, such as a senior center. To reduce the number of "no shows" for laboratory interviews, volunteers scheduled more than a week in advance receive a reminder telephone call by the QDRL Recruiter/QDRL Staff person before the day of interview.
When the respondent arrives at the QDRL, he/she is greeted by the QDRL Recruiter/QDRL Staff person and asked to read a brief description of the study, which includes assurances of confidentiality and the legislative authority for the research (Attachment E). The need for recording the interview (audio or video) is explained and the respondent is asked to sign a consent form. The form is designed at an 8th grade reading level. In the rare instance that consent is not granted, the session is not recorded in audio or video, depending upon the individual’s concern area. If the respondent grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will ask for verbal consent to retain the interviewing materials and the portion already recorded. The interviewer will get verbal consent from the respondent to do so prior to turning off the machine. If the respondent does not give consent for the QDRL to retain the recording it will be physically labeled for destruction. Upon return to the QDRL, the researcher will give the recording marked for destruction to the QDRL technician. The QDRL technician will use a degausser to erase the recording. The technician will physically check the recording to ensure erasing has occurred. A note will be place in the hardcopy file and the QDRL database indicating that particular recording (identified by the unique identification number assigned to the respondent) has been destroyed.
Attached to the Assurance of Confidentiality and Informed Consent Form is Attachment E, the Respondent Data Collection sheet that respondents are asked to complete. The purpose of this sheet is to collect recruitment and sociodemographic information on the respondent.
On occasion, sponsors requesting cognitive testing on sensitive topics (e.g., HIV testing behaviors, smoking behaviors in American Indians) require that we do not collect personal identifiers (name, and address) which the QDRL routinely collects in order to 1) pay respondents and 2) to acquire informed consent. It is the sponsor’s belief that collection of these identifiers would put the respondent at risk of potential harm resulting from breach of confidentiality.
In addition, the QDRL requests a waiver of signed informed consent from the NCHS ERB for these instances.
The interviewer then begins the interview by reading a more detailed explanation of the purpose of the interview and the procedures to be used (see Attachment O). Interviewing procedures vary depending on the specific laboratory technique to be applied. The selection of the laboratory technique is in turn determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the cognitive interview with concurrent probing. In these interviews, respondents are presented draft survey questions and asked to explain how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved. The interviewer may also ask the respondent to think aloud while answering.
If possible, cognitive interviews will be conducted in the mode intended for the survey, i.e., face-to-face; telephone, self-administered, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), Audio Computer-Assisted Self-Interview (ACASI), web-based, or video-over-internet conferencing software, such as Skype, GoToMeeting, Lync, or WebEx. For a telephone interview or video-over-internet interview, we will conduct the interview in our laboratory, but calling/contacting the respondent from another laboratory room with face-to-face debriefing following.
In addition to consenting for the interview to be recorded at the beginning of the interview, the respondent may be asked at the end of the interview, and after receipt of remuneration, to sign Attachment F, the Special Consent for Expanded Use of Video and Audio Recordings. The purpose of the special consent is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used in the case of interviews with minors (persons under the age of 18); recordings of interviews with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. If the respondent does sign the special consent form, he/she will be given a copy which contains contact information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.
The respondent may also be asked by the interviewer to sign Attachment G, Special Consent to Send Video and Audio Recordings to Off-site Researchers. The purpose of this form is to allow permission to send the recording via Federal Express to off-site researchers working on the project so they may view the recording at their location. Offsite-researchers requesting the recordings would sign a contract with NCHS stating how they will protect QDRL respondent’s privacy and the recording until it is returned to NCHS. This form is not used in the case of interviews with minors (persons under the age of 18); recordings of interviews with minors will never be shown to others not included in the study staff. If the respondent does sign the special consent to send video and audio recordings form, he/she will be given a copy which contains information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.
ii) Focus Group Methods. Respondents generally need to travel to the focus group location, which could be at NCHS, another Federal agency, or a room at another institution.
When respondents arrive they are greeted by staff working on the project and directed to the focus group room where they are individually greeted by the QDRL Recruiter/QDRL Staff person. Respondents are given a packet containing the Assurance of Confidentiality and Informed Consent for focus groups (Attachment E), the Respondent Data Collection Sheet (Attachment N) and instructed to fill them out. To maintain confidentiality, respondents are seated at separate tables. Once the forms have been completed, they will be returned to the QDRL manager for completeness, verification and safekeeping.
Respondents then each receive a separate packet containing a thank-you letter signed by the Director of NCHS, their remuneration, and a copy of the Assurance of Confidentiality and Informed Consent for focus groups (Attachment E). Respondents are then ushered into the focus group room and are seated around a table. In the rare instance that consent is not granted, respondents will still receive remuneration.
A QDRL staff member or person working on the project, as outlined in the Assurance of Confidentiality and Informed Consent, will moderate the focus group. Before discussion begins, the moderator will distribute name tags and will tell respondents to pick a name to put on the name tag. Respondents will be told that they do not have to use their real names. The moderator will then describe the process of the focus group and ask if there are any questions. After all questions are answered, the moderator will then begin the focus group discussion following the moderator guide designed for that particular study.
In addition to consenting for the discussion to be recorded at the beginning of the focus group, participants may be asked, at the close of the discussion, to sign Attachment H, the Special Consent for Expanded Use of Video and Audio Recordings for Individual Respondents of Discussion Groups. The purpose of the special consent is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from the focus group. Use of this form is at the discretion of the moderator and is typically warranted if (1) the focus group demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used in the case of focus groups with minors (persons under the age of 18); recordings of focus groups with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. Participants will be given a copy of the form which contains contact information for the QDRL Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer. If any one respondent from the focus group does not grant special consent, the recording will not be used in this way. If participants grant Special Consent, recordings are kept for as long as there is a justifiable, scientific use for the recordings as determined by the NCHS Research Ethics Review Board.
Once the focus group has concluded, the QDRL staff member or person working on the project will usher respondents to the elevator and take them to the lobby exit.
d) Field Tests/Pilot Interview Methods. For most field tests, professional field interviews (Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested) usually conduct approximately 200 pilot interviews (person/telephone). However, in 2012, a field test of almost 600 interviews, and a full scale dress rehearsal field test of 5,600 interviews was approved. Likewise, in the future larger sample sizes may be required for specific projects. These interviews will be conducted at such a time that the questionnaire to be tested has been developed and gone through cognitive testing. Questionnaires may be combined for this testing when they are at similar levels of development, or where it is logically dictated that they should be combined, such as for the appropriate NHIS Core and Topical Modules. Questionnaires will be either draft paper versions, or draft CAPI or ACASI instruments contained on laptop computers.
If the field test/pilot interviews are to be conducted in households that are part of the NHIS sample, the test questions will be incorporated into the NHIS questionnaire and administration will follow NHIS protocols.
Pilot interviews will be conducted by professional field interviewers (Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested). Households for these interviews will be pre-defined, or selected randomly through random-digit dialing; these methods have been found to be successful in the past. For all household and telephone interviews administered in this manner, the field interviewer will follow approved informed consent and survey administration procedures specific to the survey being tested. As time and resources allow, a subset of the behaviors of both interviewers and survey respondents may be observed by NCHS staff or staff of the agency sponsoring the questionnaire and observations manually recorded to allow for systematic analysis. In addition, NCHS staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata (e.g., respondent movement within ACASI, response times), interviewer observations, and respondent debriefing data. Subject matter staff are debriefed, and findings are used to modify the questionnaire for follow-up field tests/pilot interviewing or prior to the actual survey being conducted.
3. Methods to Maximize Response Rates and Deal with Nonresponse
Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential laboratory research respondents. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. For those questionnaires that target specific subgroups, special recruitment procedures will be developed to identify respondents. Direct contact to solicit support from church groups, employers, and/or social or service organizations will be explored as possible recruitment methods. Also, the offer of remuneration for the laboratory respondent's time has been a proven incentive for volunteers to participate in the study.
After laboratory volunteers have been recruited, the probability of the respondent failing to show is minimized by making reminder phone calls to volunteers.
4. Tests of Procedures or Methods to be Undertaken
This submission is a request for authorization to conduct tests of procedures and methodologies typical in cognitive testing research and to build on this research through field tests/pilot interviews. The purpose of questionnaire evaluation is not to obtain survey data, but rather to obtain information about the processes people use to answer questions as well as to identify any potential problems in the questions. This work has been effective for enhancing the quality of data of CDC, NCHS, and other Federal surveys cognitively tested by the QDRL over the past 29 years.
5. Individual Consulted on Statistical Aspects and Individuals and/or Analyzing Data
The person with overall responsibility for the methodological and technical aspects of the described activities is:
Kristen Miller, Ph.D.
Director, Questionnaire Design Research Laboratory
National Center for Health Statistics
3311 Toledo Road
Hyattsville, Maryland
(301) 458-4625
LIST OF ATTACHMENTS
ATTACHMENT A: Public Health Service Act
ATTACHMENT B: 60-day Federal Register Notice
ATTACHMENT C: Nondisclosure affidavit
Template 1: Federal Employee Version
Template 2: Contractor Version
ATTACHMENT D: Sample screening script
Template 1: Respondent recruited from newspaper
advertisement
Template 2: Respondent recruited from QDRL Database
ATTACHMENT E: Informed consent templates for QDRL interviews
Template 1: Adults
Template 2: Minors (and parents of minors)
Template 3: Focus Groups
Template 4: Offsite
ATTACHMENT F: Special Consent for Expanded Use of Video and Audio Recordings
ATTACHMENT G: Special Consent to Send Video and Audio Recordings to Off-site Researchers
for Individual Respondents of Discussion Groups
ATTACHMENT I: IRB Approval Documentation (NCHS Research Ethics Review Board)
ATTACHMENT J: Sample recruitment flyer
ATTACHMENT K: Sample newspaper advertisement
ATTACHMENT L: Sample QDRL Voice mail script
ATTACHMENT M: Sample of a 2014 advance letter sent to NHIS respondents
ATTACHMENT N: Respondent data collection sheet
ATTACHMENT O: Detailed explanation of cognitive interviewing procedure read by interviewer to respondent
| File Type | application/msword |
| File Title | Supporting Statement for Request for Clearance: |
| Author | Karen Whitaker |
| Last Modified By | Buie, Verita (CDC/OPHSS/NCHS) |
| File Modified | 2015-05-28 |
| File Created | 2015-05-28 |