Supp Statement B 070218

Supporting Statement B for Request for Clearance:

COLLABORATING CENTER FOR QUESTIONNAIRE DESIGN AND EVALUATION RESEARCH

(Currently approved under the title of “NCHS Questionnaire Design Research Laboratory)

OMB No. 0920-0222

Expiration Date: 07/31/2018

Contact Information:

Karen R. Whitaker, B.S.

Laboratory Manager, Collaborating Center for Questionnaire Design and Evaluation Research

Division of Research and Methodology

National Center for Health Statistics/CDC

3311 Toledo Road, Room 5448

Hyattsville, MD 20782

301-458-4569

[email protected]

July 2, 2018

Table of Contents

B. Collections of Information Employing Statistical Methods

B.1. Response Universe and Sampling Methods 3

B.2. Procedures for the Collection of Information 3

B.3. Methods to Maximize Response Rates and Deal with Nonresponse 9

B.4. Tests of Procedures or Methods to be Undertaken 9

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting

and/or Analyzing Data 9

LIST OF ATTACHMENTS

ATTACHMENT A: Public Health Service Act

ATTACHMENT B: 60-day Federal Register Notice

ATTACHMENT C: Nondisclosure affidavit

Template 1: Federal Employee Version

Template 2: Contractor Version

ATTACHMENT D: Sample screening script

Template 1: Respondent recruited from newspaper

advertisement

Template 2: Respondent recruited from CCQDER Database

ATTACHMENT E: Informed consent templates for CCQDER interviews

Template 1: Adults

Template 2: Minors (and parents of minors)

Template 3: Focus Groups

Template 4: Offsite

ATTACHMENT F: Special Consent for Expanded Use of Video and Audio Recordings

ATTACHMENT G: Special Consent for Expanded Use of Video and Audio Recordings

for Individual Respondents of Discussion Groups

ATTACHMENT H: IRB Approval Documentation (NCHS Research Ethics Review Board)

ATTACHMENT I: Sample recruitment flyer

ATTACHMENT J: Sample newspaper advertisement

ATTACHMENT K: Sample CCQDER Voice mail script

ATTACHMENT L: Sample of a 2017 advance letter sent to NHIS respondents

ATTACHMENT M: Sample Respondent data collection sheet

ATTACHMENT N: Detailed explanation of cognitive interviewing procedure read by interviewer to respondent

B. STATISTICAL METHODS

1. Respondent Universe and Sampling Methods

Given the variety of individual collections that we anticipate will fall under this generic clearance, we will have a number of different respondent universes and employ different sampling methods, depending on the specific purpose of the collection. However, there are some generalities about the universes and how we will sample them based on whether the proposed collection uses qualitative or quantitative methodologies.

a) Qualitative Collections. While survey research employs a deductive, quantitative methodology and relies on a relatively large population-based probability sample to support statistical inference and representativeness, methods such as cognitive interviewing employs an inductive, qualitative methodology and generally relies upon a relatively small sample. Unlike survey research, the primary objective of the qualitative methods the National Center for Health Statistics (NCHS) and the Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) employ is not to produce statistical data that can be generalized to an entire population. Rather, their objective is to provide an in-depth exploration of particular concepts, processes and/or patterns of interpretation. Samples used for qualitative research generally do not achieve full inclusivity of all social and demographic groups. As a general rule, sample definitions are based upon the content of the survey, as well as the purpose and objectives of the particular study.

b) Quantitative Collections. In the instances when NCHS/CCQDER conducts field tests/pilot tests, non-purposive sampling methods will normally be used. Again, the universe for these studies will depend on the overall research purpose, but will tend to be geographically larger than those used in the qualitative studies. Sampling will also depend on the specific project. For instance, when conducting a field test for the National Health Interview Survey (NHIS) (OMB No. 0920-0214, Exp. Date 12/31/2019), an address-based sample might be used. Across the quantitative research we anticipate the CCQDER to accomplish under this clearance, both probability (i.e. Address Based Sampling (ABS), Random Digit Dialing (RDD), etc.) and non-probability (i.e. web panels) samples may be drawn.

2. Procedures for the Collection of Information

1. Recruitment. Laboratory respondents will usually be recruited by means of flyers and other advertisements posted in public places, newspaper advertisements, or word-of-mouth. Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential respondents. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. To test questions that are targeted toward specific subgroups, the advertisement or flyer may be developed to identify appropriate respondents. For example, if the questionnaire to be tested includes a majority of questions about asthma-related health behaviors, then the recruitment may target asthma sufferers. A sample recruitment flyer and a sample newspaper advertisement are enclosed as Attachments I and J. Direct contact to solicit support from church groups, employers, and/or social or service organizations is occasionally used as possible sources of volunteers. In these cases, a flyer is provided to a contact who either posts the flyer or distributes it to members of the organization.

For some field tests/pilot interviewing, such as those done for NHIS, addresses will be pre-identified. Interviewers (such as those contracted through the Census Bureau, or through other outside contractors) will use the same procedures as used in the actual survey and acquire informed consent (via, for instance, the NHIS Advance Letter) using the HIS-100C, Manual for NHIS Field Representatives. See Attachment L for a copy of the advance letter. Other field tests will use other forms of sampling, as discussed above. For instance, some research may use an online platform and sample (and recruit) respondents through an existing email list, while other research may use RDD to recruit telephone respondents.

b) Screening and scheduling procedures. The first contact with potential laboratory research respondents occurs in response to flyers or advertisements. Interested persons leave contact information (name and telephone number) on the CCQDER voice mail system. The CCQDER Recruiter/CCQDER Staff person then calls the person back, gives a brief description of the nature of the study, i.e., one-on-one interview (face-to-face, telephone, self-administered) or focus group, where the interview/focus group takes place, video/audio recording procedures, and the incentive to be offered. First, the CCQDER Recruiter/CCQDER Staff person determines through a brief series of questions (Attachment D) whether the volunteer possesses the desired research characteristics (e.g., we ask for gender and age to avoid interviewing people with very similar demographic characteristics). If the person does possess the desired research characteristics and would like to participate, he/she is scheduled for an interview/focus group. Otherwise, the volunteer is asked whether he/she would be interested in participating in future laboratory interviews. Telephone numbers and the minimal demographic information listed earlier are obtained for all scheduled volunteers and for those who would like to be contacted in the future. For those callers who are ineligible for the study and do not want to be contacted in the future, only demographic characteristics will be maintained for future analysis of successful recruitment efforts. Attachment K contains a sample CCQDER voice mail script.

For field test/pilot interviews, the questionnaires to be tested are generally applicable to the general population, so no special population selection will be necessary. Any adult household respondent is eligible for the interview. It is possible that the CCQDER may be asked to conduct a field test with a specialized population (practicing doctors, for instance). In this case, a screening plan will be developed and included in that project’s 10-day letter.

c) Questionnaire Design Research Laboratory (QDRL) Interview Methods.

i) One-on-one Interviews. If a laboratory interview is scheduled, the individual will usually travel to the QDRL. The QDRL is located in the NCHS office building in Hyattsville, Maryland. The lab contains a waiting room, a control room, two private interview rooms, and a remote observation room for use by survey sponsors and other researchers to view an interview while in progress. On rare occasions, a respondent may be unable to travel to the laboratory for the interview (for example, an individual may be housebound or have very limited mobility). In such cases, the interview may be conducted in their home or at a location normally frequented by the respondent, such as a senior center. To reduce the number of "no shows" for laboratory interviews, volunteers scheduled more than a week in advance receive a reminder telephone call by the CCQDER Recruiter/CCQDER Staff person before the day of interview.

When the respondent arrives at the QDRL, he/she is greeted by the CCQDER Recruiter/CCQDER Staff person and asked to read a brief description of the study as captured in the consent form, which includes assurances of confi­dentiality and the legislative authority for the research (Attachment E). The need for recording the interview (audio or video) is explained and the respondent is asked to sign a consent form. The form is designed at an 8th grade reading level. In the rare instance that consent is not granted, the session is not recorded in audio or video, depending upon the individual’s concern area. If the respondent grants consent to record the interview but changes his/her mind while the session is being recorded, the interviewer will ask for verbal consent to retain the interviewing materials and the portion already recorded. The interviewer will get verbal consent from the respondent to do so prior to turning off the machine. If the respondent does not give consent for the CCQDER to retain the recording it will be physically labeled for destruction. Upon return to the QDRL, the researcher will give the recording marked for destruction to the CCQDER technician. The CCQDER technician will use a degausser to erase the recording. The technician will physically check the recording to ensure erasing has occurred. A note will be place in the hardcopy file and the CCQDER database indicating that particular recording (identified by the unique identification number assigned to the respondent) has been destroyed.

Attached to the Informed Consent Form is Attachment M, the Respondent Data Collection sheet that respondents are asked to complete. The purpose of this sheet is to collect recruitment and sociodemographic information on the respondent.

On occasion, sponsors requesting cognitive testing on sensitive topics (e.g., HIV testing behaviors, smoking behaviors in American Indians) require that we do not collect personal identifiers (name, and address) which the CCQDER routinely collects in order to 1) give incentives to respondents and 2) to acquire informed consent. It is the sponsor’s belief that collection of these identifiers would put the respondent at risk of potential harm resulting from breach of confidentiality. In these cases, the CCQDER requests a waiver of signed informed consent from the NCHS ERB. The consent form is modified to include a statement that the interviewer witnessed the respondent reading the informed consent document. The interviewer signs the statement. In addition, the IAA or MOU Statement of Work will contain the date that the testing sponsor wishes the recordings to be destroyed. Confirmation of destruction of the recordings will be sent to the testing sponsor/point of contact.

The interviewer then begins the interview by reading a more detailed explanation of the purpose of the interview and the procedures to be used (see Attachment N). Interviewing procedures vary depending on the specific laboratory technique to be applied. The selection of the laboratory technique is in turn determined by the nature of the project, or the stage of development of the questionnaire or set of questions under study. The most commonly used method is the cognitive interview with concurrent probing. In these interviews, respondents are presented draft survey questions and asked to explain how and why they answered as they did. The interviewer usually probes extensively to ascertain the degree of comprehension and the recall processes involved. The interviewer may also ask the respondent to think aloud while answering.

If possible, cognitive interviews will be conducted in the mode intended for the survey, i.e., face-to-face; telephone, self-administered, Computer Assisted Personal Interviewing (CAPI), Computer Assisted Telephone Interviewing (CATI), Audio Computer-Assisted Self-Interview (ACASI), web-based, or video-over-internet conferencing software, such as Skype, GoToMeeting, Lync, or WebEx. For a telephone interview or video-over-internet interview, we will conduct the interview in our laboratory, but calling/contacting the respondent from another laboratory room with face-to-face debriefing following.

In addition to consenting for the interview to be recorded at the beginning of the interview, the respondent may be asked at the end of the interview, and after receipt of the incentive, to sign Attachment F, the Special Consent for Expanded Use of Video and Audio Recordings. The purpose of the special consent is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from cognitive interviewing. Use of this form is at the discretion of the interviewer and is typically warranted if (1) the interview demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used when the topic of the cognitive interview is considered to be an illegal behavior (self-report or proxy report) or in the case of interviews with minors (persons under the age of 18); recordings of interviews with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. If the respondent does sign the special consent form, he/she will be given a copy which contains contact information for the CCQDER Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer.

ii) Focus Group Methods. Respondents generally need to travel to the focus group location, which could be at NCHS, another Federal agency, or a room at another institution.

When respondents arrive they are greeted by staff working on the project and directed to the focus group room where they are individually greeted by the CCQDER Recruiter/CCQDER Staff person. Respondents are given a packet containing the Assurance of Confidentiality and Informed Consent for focus groups (Attachment E, Template 3), the Respondent Data Collection Sheet (Attachment M) and instructed to fill them out. To maintain confidentiality, respondents are seated at separate tables. Once the forms have been completed, they will be returned to the CCQDER manager for completeness, verification and safekeeping.

Respondents then each receive a separate packet containing a thank-you letter signed by the Director of NCHS, their incentive, and a copy of the Assurance of Confidentiality and Informed Consent for focus groups (Attachment E, Template E). Respondents are then ushered into the focus group room and are seated around a table. In the rare instance that consent is not granted, respondents will still receive an incentive.

A CCQDER staff member or person working on the project, as outlined in the Assurance of Confidentiality and Informed Consent, will moderate the focus group. Before discussion begins, the moderator will distribute name tags and will tell respondents to pick a name to put on the name tag. Respondents will be told that they do not have to use their real names. The moderator will then describe the process of the focus group and ask if there are any questions. After all questions are answered, the moderator will then begin the focus group discussion following the moderator guide designed for that particular study.

In addition to consenting for the discussion to be recorded at the beginning of the focus group, participants may be asked, at the close of the discussion, to sign Attachment G, the Special Consent for Expanded Use of Video and Audio Recordings for Individual Respondents of Discussion Groups. The purpose of the special consent is to allow for the playing of recordings at conferences, meetings, or in the classroom to illustrate particular findings from the focus group. Use of this form is at the discretion of the moderator and is typically warranted if (1) the focus group demonstrated a unique question problem or research finding and (2) there is an anticipated need to demonstrate the research findings at a conference, meeting, or instructional session. This form is not used in the case of focus groups with minors (persons under the age of 18); recordings of focus groups with minors will never be shown to others not included in the study staff. The form is designed at an 8th grade reading level. Participants will be given a copy of the form which contains contact information for the CCQDER Laboratory Manager, the NCHS ERB Chair, and the NCHS Confidentiality Officer. If any one respondent from the focus group does not grant special consent, the recording will not be used in this way. If participants grant Special Consent, recordings are kept for as long as there is a justifiable, scientific use for the recordings as determined by the NCHS Research Ethics Review Board.

Once the focus group has concluded, the CCQDER staff member or person working on the project will usher respondents to the elevator and take them to the lobby exit.

d) Field Tests/Pilot Interview Methods. For most field tests, professional field interviewers (Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested) usually conduct approximately 200 pilot interviews (person/telephone). However, in 2012, a field test of almost 600 interviews, and a full scale dress rehearsal field test of 5,600 interviews was approved. Likewise, in the future larger sample sizes may be required for specific projects. These interviews will be conducted at such a time that the questionnaire to be tested has been developed and gone through cognitive testing. Questionnaires may be combined for this testing when they are at similar levels of development, or where it is logically dictated that they should be combined, such as for the appropriate NHIS Core and Topical Modules (OMB No. 0920-0214, Exp. Date 12/31/2019. Questionnaires will be either draft paper versions, or draft CAPI or ACASI instruments contained on laptop computers.

If the field test/pilot interviews are to be conducted in households that are part of the NHIS sample, the test questions will be incorporated into the NHIS questionnaire and administration will follow NHIS protocols.

Pilot interviews will be conducted by professional field interviewers (Census Bureau Field Representatives or other interviews who are contracted for the tested survey or have experience administering the particular survey to be tested). Households for these interviews will be pre-defined, or selected randomly through random-digit dialing; these methods have been found to be successful in the past. For all household and telephone interviews administered in this manner, the field interviewer will follow approved informed consent and survey administration procedures specific to the survey being tested. As time and resources allow, a subset of the behaviors of both interviewers and survey respondents may be observed by NCHS staff or staff of the agency sponsoring the questionnaire and observations manually recorded to allow for systematic analysis. In addition, NCHS staff may conduct analysis of outcome data such as response rates and response distributions to key items, paradata (e.g., respondent movement within ACASI, response times), interviewer observations, and respondent debriefing data. Subject matter staff are debriefed, and findings are used to modify the questionnaire for follow-up field tests/pilot interviewing or prior to the actual survey being conducted.

3. Methods to Maximize Response Rates and Deal with Nonresponse

Our experience has shown that advertisements in local newspapers and flyers attract a large pool of potential laboratory research respondents. These recruitment mechanisms have been productive in the past for obtaining a diverse group of respondents to help us determine potential sources of error in survey questions. For those questionnaires that target specif­ic subgroups, special recruitment procedures will be developed to identify respondents. Direct contact to solicit support from church groups, employers, and/or social or service organizations will be explored as possible recruitment methods. Also, the offer of incentives to help cover out of pocket expenses has been a proven motivation for volunteers to participate in the study.

After laboratory volunteers have been recruited, the probability of the respondent failing to show is minimized by making reminder phone calls to volunteers.

4. Tests of Procedures or Methods to be Undertaken

This submission is a request for authorization to conduct tests of procedures and methodologies typical in cognitive testing research and to build on this research through field tests/pilot interviews. The purpose of questionnaire evaluation is not to obtain survey data, but rather to obtain information about the processes people use to answer questions as well as to identify any potential problems in the questions. This work has been effective for enhancing the quality of data of CDC, NCHS, and other Federal surveys cognitively tested by the CCQDER over the past 29 years.

5. Individual Consulted on Statistical Aspects and Individuals and/or Analyzing Data

The person with overall responsibility for the methodological and technical aspects of the described activities is:

Kristen Miller, Ph.D.

Director, Collaborating Center for Questionnaire Design and Evaluation Research

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland

(301) 458-4625

[email protected]

8