QDRL NHIS A-CASI 5600 letter to Dr

DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service

Centers for Disease Control and Prevention

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland 20782

June 20, 2012

Margo Schwab, Ph.D.

Office of Management and Budget

725 17th Street, N.W.

Washington, DC 20503

Dear Dr. Schwab:

The staff of the NCHS Questionnaire Design Research Laboratory (QDRL) (OMB No. 0920-0222, exp. 06/30/2015) plans to conduct a field test of 3,300 Audio Computer-Assisted Self-Interview (ACASI) and 2,300 Computer-Assisted Personal Interview (CAPI) interviews for the National Health Interview Survey (NHIS) beginning August 1. The 5,600-case test is a follow-up to the 575- and 50-case tests presented to OMB in letters dated October 6, 2011, and January 10, 2012, respectively. The 50-case test was conducted November 15-28, 2011, while the 575-case test was conducted in April of 2012. The purpose of these field tests is to test specific sexual identity questions and to test alternate approaches to collecting this information in a household interview setting.

Select Results from the National Health Interview Survey (NHIS) ACASI 50 and 575-Case Field Tests

The 50-case test focused on the programming of the ACASI portion of the questionnaire and the transition from orally administered to self-administered questions. The 575-case test used procedures consistent with the annual NHIS, providing a more realistic reflection of the field effort required to obtain completed ACASI interviews and respondent acceptability of ACASI. Interviewing was conducted in areas with high concentration of Spanish speaking persons in order to thoroughly test the translation. Preliminary results reveal no significant problems in acceptability or usability of the ACASI instrument. Of the 564 respondents who reached the ACASI part of the interview, 19 (3.4%) refused to participate. In debriefings, several interviewers noted that ACASI was a selling point for many respondents, providing a more interactive component of the interview process. Others noted that respondents found ACASI “fun” and/or “easy.” Nonetheless, interviewer debriefings revealed ways to streamline the transition from CAPI to ACASI, as well as strategies to encourage reluctant respondents to complete ACASI. Recommended strategies will be covered in training for the 5,600-case test.

Further evidence of the acceptability and ease of ACASI use can be found in respondents’ answers to the respondent debriefing questions included at the end of the ACASI interview. Only 3.2% of respondents reported difficulty using the keys to select their answers, and only 4.2% reported that interviewers did not adequately explain how to use the computer. Among those respondents who listened to all or some of the audio recordings, roughly 94% found the headphones or earbuds to be somewhat or very comfortable. Surprisingly, only 59% of respondents felt that it was important for the ACASI portion of the interview to be private. And finally, it took about 14 minutes (median = 12 minutes), on average, for respondents to complete the ACASI module.

Sample from the 575-case test was matched to a comparable NHIS sample (621 2011 NHIS interviews drawn from the same Census tracts used for the field test) to allow comparisons of responses to content currently included in the NHIS sample adult module but moved to the ACASI module for the field test (height, weight, HIV testing, mental health, sleep, alcohol questions). The primary finding from these comparisons was elevated levels of item nonresponse among the ACASI interviews. For example, while the item nonresponse rate (don’t know and refused) for HIV testing among a matched set of 2011 production NHIS interviews was 3.8%, the corresponding rate from the 575-case field test was 7.9%. Bivariate and limited multivariate analyses failed to explain away a persistent effect of ACASI administration across the items compared. We attribute these differences to the design of the ACASI screens where explicit don’t know and refused options were presented with each question, and respondents could press Enter to advance to the next question without selecting a response. After consultation with a noted expert on ACASI and web survey design, as well as a thorough literature review, we decided on a revised screen design for the 5,600-case test (see Attachment 1). Don’t know and refused response options will be removed from the screen for each question to be more consistent with oral administration of questions. Respondents can still press Enter to advance to the next question without answering, but doing so will skip them to a follow-up question:

You did not enter an answer for the question. That is because you:

Made a mistake and would like to have another chance

Are not sure how to answer

Do not want to answer

Selecting the first response option will return the respondent to the question. Selecting either the second (don’t know) or third (refused) response option advances the respondent to the next question. This design is consistent with: that used with the NHANES ACASI module, the screens used with the sexual identity items in the 50- and 575-case tests, and CAPI administration more generally.

In addition, comparisons of prevalence estimates and item nonresponse rates for numeric questions on height, weight, and alcohol consumption revealed significant differences between CAPI and ACASI. These items will be returned to the interviewer administered portion of the questionnaire.

Finally, preliminary analyses did not reveal significant measurement issues in the collection of sexual identity data. Overall, 0.4% of respondents were unsure how to answer the question, 1.5% refused to answer, and 1.3% had comprehension problems. Differences did emerge between English and Spanish speakers, with 7.6% of the latter either refusing to answer or expressing comprehension problems. However, when limiting the sample to adults aged 18-44, crude comparisons with the National Survey of Family Growth (NSFG OMB # 0920-0314) data reveal an overall reduction in item nonresponse in the field test. (Caution is warranted in drawing conclusions from a non-nationally representative dataset.) Therefore, no changes are recommended to the questions on sexual identity for the 5,600-case test.

Proposed Project: National Health Interview Survey (NHIS) ACASI 5,600-Case Field Test

Limited data are currently available on health disparities that involve the Lesbian, Gay, and Bisexual (LGB) populations.  Many Healthy People 2010 objectives went unmeasured because most general health studies do not include questions on sexual identity and those that do are usually targeted to a specific health-related issue or population subgroup. The introduction of questions on sexual identity to ongoing surveys requires methodological development and testing to assure that the data collected are valid across all population subgroups and can be used for policy and program development and evaluation.  The objective of asking a question on sexual identity in the NHIS is to fill the tremendous gap that exists regarding knowledge of general health behaviors, health status, and health care utilization of LGB persons.

The 5,600-case field test will entail a full scale dress rehearsal employing a nationally-representative sample with a goal of 5,600 completed interviews. As with the previous test, impacts of the new content/ACASI module on response rates, break-offs, and key NHIS indicators (critical for monitoring trends using existing NHIS measures) will be monitored, and a systematic assessment of response error in the sexual identity measure will be performed.

Test cases will be randomly assigned to receive the ACASI content, including the sexual identity items, in either CAPI (n=~2,300) or ACASI (n=~3,300).  The questionnaire batteries will be identical using both modes. Prevalence estimates for the sexual identity question will be compared by mode of administration. By doing so, we can determine if sexual identity questions can be included in the NHIS without the use of ACASI. Since a documented advantage of ACASI is the enhanced level of privacy it affords, we anticipate higher prevalence estimates of sexual minorities from this mode of administration.  Estimates for sensitive items (included in the ACASI module) on mental health, HIV testing, financial worries, and others will also be compared.

Direct and indirect indicators of data quality, such as item nonresponse and response times, can also be compared between the CAPI and ACASI administrations.  And since the field test will use a nationally-representative sample, content pulled from the core NHIS instrument and moved to the ACASI module can be compared between the field test NHIS and the ongoing, production NHIS. Along with mode effects, this will enable the exploration of possible context effects on prevalence estimates, item nonresponse rates, and other quality indicators, and provide much needed input on possible impacts on trends for key NHIS indicators.

Based on analyses of respondent and interviewer debriefing data from the 575-case test, a set of revised debriefing questions will be added for this test (respondent debriefing questions appear at the end of Attachment 1). And finally, audit trails capturing item times and respondent movement through the ACASI module will be available. Analyses of audit trails will continue to aid detection of respondent usability issues.

As many as 5,600 field interviews may be completed by Census Bureau Field Representatives (interviewers) who have experience conducting the NHIS. Sample for the Split Ballot Test was assembled from unused but listed NHIS sample from Quarter 1, Weeks 1 and 2 of years 2006-2010.  Sample was created through 2010 for the first two weeks of each year but unused because interviewers were trained on the new NHIS during those weeks.  Each week is nationally representative. This contributes almost 10,000 addresses.  An additional 1,300 addresses will be taken from non-MEPS sample during Weeks 5 and 6 of 2006, which is also a nationally representative sample.  Addresses for this test are distributed nationally to ease the burden on interviewers who must thoroughly work these cases in addition to their usual workload.   Many interviewers work on multiple Census surveys simultaneously.  Interviewers will receive a self-study training module on ACASI and field test procedures, which they must complete before attending a one-day classroom training. In the field, each interviewer will be assigned approximately 20-25 sample cases and asked to work those cases as they would a normal NHIS caseload. Each

assigned case will be worked for up to one month and receive a final case disposition (e.g., interview, nonresponse, out-of-scope).

Interviews will be conducted in English and Spanish. The 2011 NHIS Core and the proposed ACASI questions (including sexual identity questions) will be asked as part of the 5,600-case test. The administration of the 2011 NHIS Core and the ACASI questions will take approximately 60 minutes.

The 2011 NHIS Core is conducted using computer-assisted personal interviewing (CAPI). CAPI allows interviewers to read questions from the computer screen and enter the respondents’ responses directly into the computer during the interview. The 2011 NHIS Core is shown in Attachment 4. The proposed ACASI questions will be administered directly after the 2011 NHIS Core. ACASI allows respondents to listen to prerecorded questions from the laptop computer through headphones and record their responses using the laptop computer. Respondents may or may not simultaneously read the questions from the computer screen. They also have the option of turning off the sound. See Attachment 1 for the sexual identity questions and other proposed ACASI questions.

Census interviewers will use the same procedures as used in the production NHIS to explain the survey and acquire informed consent (via the NHIS Advance Letter) using the HIS-100C, January 2011, Manual for NHIS Field Representatives. See Attachment 2 for excerpted pages A-10 through A-14 explaining the survey. The entire NHIS Field Representatives Manual can be accessed on NHIS’ webpage at: ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/Survey_Questionnaires/NHIS/2011/frmanual.pdf. See Attachment 3 for the NHIS production advance letter to be used for the 5,600 case test.

Respondents will not be remunerated for participation. Interviews will not be taped.

In total, for this project, the maximum respondent burden will be 5,600 hours of interviewing. A burden table for this project is shown below:

Projects	Number of Participants	Number of Responses/ Participant	Average hours per response	Response burden
QDRL Interviews
3) Research on the effects of alternative questionnaire design	5,600	1	1	5,600

Attachment (4)

cc:

M. Moien

T. Richardson

DHHS RCO

3