10 day letter to Dr. Schwab

DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service

Centers for Disease Control and Prevention

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland 20782

April 13, 2011

Margo Schwab, Ph.D.

Office of Management and Budget

725 17th Street, N.W.

Washington, DC 20503

Dear Dr. Schwab:

The staff of the NCHS Questionnaire Design Research Laboratory (QDRL) (OMB No. 0920-0222, exp. 03/31/2013) plans to conduct research to 1) to conduct usability testing on the Audio Computer-Assisted Self Interview (A-CASI) software developed to administer a proposed measure on sexual identity question for the National Health Interview Survey (NHIS) and, 2) to cognitively test the proposed sexual identity measure, as well as other demographic questions that appear on the NHIS, including questions on race, ethnicity, marital status, employment status and family income. We will also test questions on alcohol use and record linkage questions that currently appear on the NHIS. The sexual identity question being tested in this project was developed from previous cognitive testing studies. The first draft of this question was tested in a recent cognitive interviewing project using an interviewer administered paper instrument described in a letter to you dated November 22, 2010.

We propose to start advertising for volunteer participants as soon as we receive clearance and to start testing as soon as possible after that.

Background Information about Cognitive Testing of Questionnaires

The methodological design of this proposed study is consistent with the design of typical cognitive testing research. As you know, the purpose of cognitive testing is to obtain information about the processes people use to answer survey questions as well as to identify any potential problems in the questions. The analysis will be qualitative.

Proposed project: National Health Interview Survey (NHIS) A-CASI

The QDRL has been tasked with developing a measure of sexual identity for NHIS and an A-CASI system on which the question will be administered. A primary difficulty in designing a sexual identity question for a national population survey is that the construct of sexual identity (as opposed to a more objective construct, such as behavior) is a complex concept; respondents’ conceptions of the construct itself can vary dramatically depending on demographic characteristics (e.g., race/ethnicity, gender, age), socio-cultural context and geographic location. QDRL has conducted multiple evaluations using cognitive interviewing methods on various sets of questions designed to capture information on sexual identity.

Analysis consistently shows that the questions produce measurement error due to problems with respondent comprehension of terminology as well as the use of labels that are not consistent with the way respondents identify themselves. Analyses of survey data from the National Health and Nutrition Examination Survey (NHANES) and the National Survey of Family Growth (NSFG) support these conclusions. Lack of understanding key terms and unsuitable use of labels results in high levels of responses in non-substantive categories such as 'something else', 'don't know', and refused. As a result, responses that are not analyzed have higher prevalence than the responses of interest. Additionally, depending on how questions are worded, estimates vary and do not always make theoretical sense. For example, estimates from NHANES show twice as many gay men than bisexual men, while NSFG estimates show an equal proportion. Also of concern is the fact that the percent of the population responding in the non-substantive categories varies across demographic and socioeconomic statuses, which further reduces the ability to interpret the data. It is believed that developing an A-CASI sexual identity question, with help screens to clarify concepts for respondents, will resolve many of these interpretive problems.

The sexual identity question being tested in this project was developed from previous cognitive testing studies. The first draft of this question was tested in a recent cognitive interviewing project using an interviewer administered paper instrument. Testing this question using an A-CASI system, will allow us to determine both the usability of the A-CASI system and whether the question itself is interpreted differently based on mode of administration. In addition, we will also test other demographic questions that appear on the NHIS, including questions on race, ethnicity, marital status, employment status and family income. We will also test questions on alcohol use and record linkage questions that currently appear on the NHIS. Please note, respondents are asked question 1 in Attachment 2 (names of all family members) in order to determine the size of the family living in their home. In order to administer the appropriate family income questions, we must know the family size.

A number of help screens are included in the questionnaire. For the sexual identity questions, the “help screens” include definitions of the terms used in the response categories. The definitions will appear on the same screen as the question. Respondents will not have to move to a different screen in order to access these definitions. This is being done for testing purposes only. This will ensure that all respondents have an opportunity to review the definitions while answering the questions, and so interviewers can then probe on their reactions to the definitions. In all other questions which contain help screens (i.e., Q5, Q23, Q24, Q25, Q26, Q27, Q28, Q29, Q30) respondents will only see the information provided on the help screens if they request help when answering the questions.

The introduction to the usability test/cognitive interview and usability interview guide appears in Attachment 1. The cognitive testing interview guide and A-CASI instrument appears in Attachment 2. Testing procedures will conform to the cognitive interviewing techniques that have been described in our generic OMB clearance package.

We propose to recruit as many as many as 100 adults (aged 18 years and older). We plan to recruit 50 respondents who identify as heterosexual with some demographic variety (particularly in terms of gender, education, race/ethnicity, and income), and 50 respondents who identify as gay, lesbian, bisexual or transgendered with some demographic variety (particularly in terms of gender, education, race/ethnicity, and income). We will begin testing in the QDRL and we may need to test the instrument in various locations across the U.S. in

order to ensure adequate coverage of respondent demographics. Respondents will be recruited through a newspaper advertisement (Attachment 3a), flyers distributed at community centers (Attachment 3a and 3b), by contacting respondents from past QDRL projects listed in the QDRL Respondent Database who have expressed interest in participating in upcoming projects, and through word-of-mouth.

General Interview Procedures

Cognitive interviews will be conducted by QDRL staff members, and interviews will last for 90 minutes each. Respondents will be informed of audio taping procedures during the screening process, and in the process of reviewing the consent forms prior to the start of the interview. Only individuals who agree to be audio taped will be eligible to participate in the study.

Due to the sensitive nature of the questions, the NCHS QDRL will conduct the study anonymously. QDRL Staff will collect minimal personal identifiers. A first name and a contact telephone number for scheduling and reminder calls, if available, may be used—however, full name, address and home telephone number, will not be collected. The QDRL routinely collects this information in order to 1) pay respondents through the use of the approved cash payment receipt form, and 2) to acquire informed consent. It is the QDRL’s belief that the collection of these identifiers would put the respondent at risk of potential harm resulting from a breach of confidentiality.

Respondents who decide to participate will be asked to read the waived signed Informed Consent form which allows for the audio taping of the interview. The interviewer will witness the reading of the waived signed Informed Consent form. In addition, prior to the start of the interview the interviewer will turn on the tape recorder and respondents will be asked to verbally acknowledge that they have agreed to participate in the research study and that they have agreed to be audio taped. If the respondent decides to turn off the audio tape anytime during the interview, they will be asked for consent to retain the interviewing materials and the portion already taped. The interviewer will get verbal consent from the respondent to do so prior to turning off the tape.

Interviews will begin with an overview about the purpose of the study. After a brief training session on using the A-CASI system, respondents will be asked to answer the questions using a computer with headphones. When respondents are done answering the questionnaire, interviewers will print off their responses. Interviewers will then conduct a face-to-face debriefing interview to assess respondents’ confidence in using the A-CASI, comfort level, and any problems them might have encountered. For purposes of the usability test, we will also collect paradata using the BLAISE software. Paradata include recording of response timing, keystrokes, access of definitions, access of help menus, use of audio, respondent’s general navigation through the instrument and changes to answers. Following the debriefing, we will conduct a cognitive test of the questionnaire. The QDRL interviewer will read the respondents the questions they were administered during the ACASI and remind them of their answers. Interviewers will ask emergent probes to better understand the question-response process. Examples of the sorts of probes that may be asked at the interviewer’s discretion include:

• Why did you answer that way?

• How did you arrive at your answer?

• Can you tell me more about that?

• Can you clarify what you mean?

Interviewers may use some or all of these probes, depending upon the content of the interview and how much information the respondents reveal without being prompted. In some interviews, probes will be administered throughout the interview, and for others they will be administered after completing the questionnaire.

We propose paying participants $50, which is $10 over our standard payment. Given the length of the cognitive interview (90 minutes vs. our traditional 60 minute interview), and given the sensitive nature of the questions we hope the extra $10 above our $40 standard payment will encourage respondents to participate in the study. In total, for this project, the maximum respondent burden will be 150 hours of interviewing in addition to travel time.

After the interview, respondents will be given the thank-you letter signed by the Director of NCHS, a copy of the informed consent document, and $50.

Because the interviews contain sensitive information and the respondent’s voice will be on the tape, the respondent will not be asked for Special Consent for Expanded Use of Video and Audio Recordings which would allow QDRL staff to play the tape at conferences as part of a presentation, for students, or for other people who write survey questions. Audio tapes will not be shared.

Extreme care will be taken with all tapes and paperwork from the interviews conducted off-site. Tapes and identifying paperwork will be stored in a secured travel case until returned to NCHS, at which point they will be transferred to the usual secured locked storage cabinets.

An updated burden table for this project is shown below:

Projects	Number of Participants	Number of Responses/ Participant	Average hours per response	Response burden
QDRL Interviews
2) NCHS Surveys	100	1	1.5	150

Attachments (3)

cc:

M. Moien

D. Holcomb

DHHS RCO

4