QDRL CDME 10-day letter-07242013

DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service

Centers for Disease Control and Prevention

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland 20782

July 11, 2013

Margo Schwab, Ph.D.

Office of Management and Budget

725 17th Street, N.W.

Washington, DC 20503

Dear Dr. Schwab:

The staff of the NCHS Questionnaire Design Research Laboratory (QDRL) (OMB No. 0920-0222, exp. 06/30/2015) plans to conduct a comparative evaluation of two sets of disability questions as well as begin a rigorous study of mode effects in cognitive interviewing. This project is part of a collaborative effort among the University of Michigan’s Institute for Social Research and Statistics Norway. The purpose of this study is twofold. First to iteratively and comparatively evaluate two sets of disability questions that focus on motor functioning. Secondly, we will evaluate the feasibility and resultant data quality of performing cognitive interviews via video conferencing and telephone, as compared to more standard face-to-face cognitive interviews. Overall, the project will proceed over two phases, examine two separate core questionnaires (Attachments 1a and 1b) using three modes of cognitive interviewing. The two questionnaires largely differ only in the specific questions, with the topic areas and constructs remaining constant. The three modes of cognitive interviewing will be traditional face-to-face, telephone, and video conferencing. Analysis of the disability questions will occur across modes, though the focus of the comparative analysis will be on the data from the face-to-face interviews. The procedures for both of these two sub-studies are described in detail below.

We propose to start recruiting for volunteer participants as soon as we receive clearance and to start testing as soon as possible after that.

Background Information about Cognitive Testing of Questionnaires

The methodological design of this proposed study is consistent with the design of typical cognitive testing research. As you know, the purpose of cognitive testing is to obtain information about the processes people use to answer survey questions as well as to identify any potential problems in the questions. The analysis will be qualitative.

Proposed project: 2013 Comparative Disability Questions and Mode Effects Study

The staff of the NCHS Questionnaire Design Research Laboratory is requesting approval to conduct one hundred twenty total cognitive interviews. Table 1 shows how the data will be collected across the two phases. Phase 1 data collection will iteratively evaluate a questionnaire whose functional disability questions were designed by the World Health Organization (WHO) and the World Bank. After an initial round of testing, this questionnaire will be revised and re-evaluated in a second round. Interviews in this first round of Phase 1 will be conducted by staff members of both the QDRL and the University of Michigan’s Institute for Social Research (ISR), whereas interviews in the second round of Phase 1 will be conducted only by QDRL staff members. Phase 2 data collection will evaluate a questionnaire whose functional disability questions were designed by NCHS in association with the UN’s Washington Group on Disability Statistics across three modes of cognitive interviewing—in person face-to-face, telephone, and video-over-internet. All interviews in Phase 2 will be conducted by QDRL staff members.

In addition to the WHO and NCHS-designed functional disability components, both questionnaires will also evaluate a set of questions on health insurance exchanges for potential inclusion in the National Health Interview Survey (NHIS). The questionnaire in Phase 2 will additionally include a small set of previously-tested, non-disability focused questions, designed to contribute to the analysis of mode effects.

Table 1: Data Collection Plan

Phase	Round	Questionnaire	Interviewers	Location	Mode	N=
1	1	WHO (Original)	QDRL and ISR	QDRL	Face-to-Face	20
1	2	WHO (Revised)	QDRL	QDRL and Offsite	Face-to-Face	30
2		NCHS	QDRL	QDRL	Face-to-Face	30
2		NCHS	QDRL	QDRL	Telephone	20
2		NCHS	QDRL	QDRL	Video-over-Internet	20
TOTAL						120

Comparative Disability Sub-Study

The search for a set of standard, international questions to provide disability statistics continues. In the first proposed sub-study, the QDRL will test two different sets of questions, written by two different international organizations, which both attempt to measure functional disabilities. The first set, written by the WHO and the World Bank is part of a longer questionnaire designed to produce disability statistics across a number of countries. Parts of this questionnaire, including the section on functional disability planned for this study, are also being tested by Statistics Norway and the University of Michigan's ISR. This first set of disability questions is found in Attachment 1a. The second set, found in Attachment 1b, is an alternative version of the WHO questions designed and written by NCHS for the Washington Group on Disability Statistics, which operates under the aegis of the United Nations' Statistical Commission. This second set applies the findings from previous rounds of testing by the QDRL on disability questions proposed by the Washington Group.

Testing these two sets in the same project will allow for a comparative analysis across not only the two versions being tested by the QDRL, but also across the three organizations testing the first set. Along with colleagues from ISR, we will administer the first set (2a) to 20 respondents in face-to-face cognitive interviews, following the procedures set out in our generic OMB Protocol 0920-0222. All 20 of these interviews will take place in NCHS’ Questionnaire Design Research Laboratory. Following this round, the functional disability questions will be revised as appropriate. A second round of 30 face-to-face interviews will then be administered, again following the procedures set out in our generic OMB ICR 0920-0222. Depending on respondent availability and recruiting, these face-to-face interviews will either take place in the Questionnaire Design Research Laboratory at NCHS, or at a mutually-agreed upon outside location such as a library or office.

The second set of functional disability questions, written by NCHS (found in Attachment 1b) will be administered to 70 respondents across three modes, as detailed in the following section on the Mode Effects Sub-Study. The testing for this second set will also conform to the procedures set out in the generic OMB Protocol 0920-0222.

Mode Effects Sub-Study

The demand for cognitive interviews as a way of evaluating survey questions has steadily increased. At the same time there has been an increased need to interview a geographically diverse sample. This has traditionally been handled by having cognitive interviewers travel to conduct cognitive interviews; however, in light of current budget constraints and short timeframes provided to complete cognitive interview studies this is not always feasible. One possible solution to this is to utilize video-over-internet conferencing software, such as Skype, GoToMeeting, Lync, or WebEx to conduct cognitive interviews which allows interviewers to see a respondent’s facial expressions and body language, which can play an important role in a cognitive interview. This project is designed to investigate the feasibility of utilizing new technologies to conduct cognitive interviews, evaluate the quality of data obtained, and determine whether the mode of the interview impacts the interpretation of the survey questions.

Respondents participating in this sub-study will receive the questionnaire that appears in Attachment 1b. In addition to the Washington Group functional disability questions mentioned above, this questionnaire also includes a small set of previously-tested, non-disability health and general-interest questions obtained from the National Health Interview Survey (NHIS), National Health and Nutrition Examination Survey (NHANES), American Community Survey (ACS), and the Federal Statistical System Public Opinion Survey. In some cases the questions were slightly modified from their original form to better meet the structure of this research project. These questions will allow the examination of modal effects on non-disability topics.

We will recruit 70 respondents to participate in this sub-study. Of those 70, 30 will participate in face-to-face interviews, 20 will participate in telephone interviews, and 20 will participate in video-over-internet interviews (using Lync software). All 70 interviews in this sub-study will take place in the Questionnaire Design Research Laboratory at NCHS, not only to attempt to control for the surroundings, but also so that the interviews can all be videotaped. Additionally, by running all the video-over-internet interviews from within the CDC network, we will be able to use the secure Lync software, and reduce the likelihood of network issues that could impact respondent privacy.

When considering the effectiveness and data quality of the various modes of the cognitive interviews, measures such as the variance in interpretations and respondent engagement will be considered. In order to standardize this analysis, metadata such as overall interview time, response time, interviewer-speaking time, behavior codes, and number of cognitive issues might be collected. The collection of this metadata will not add additional burden to the respondents.

Recruiting and Informed Consent Procedures across both Sub-Studies

We propose to recruit the 120 adult respondents (Age 18 and older) through newspaper advertisements, flyers, special interests groups, word-of-mouth, QDRL Respondent Database, etc. The newspaper advertisement/flyer used to recruit respondents is shown in Attachments 2a, 2b, and 2c. The screener used to determine eligibility of individuals responding to the newspaper advertisement/flyer, etc. is shown in Attachment 3a. The screener used to determine eligibility of individuals from the QDRL Respondent Database is shown in Attachment 3b. Within these constraints, we hope to recruit participants with some demographic variety (particularly in terms of gender, education, race/ethnicity, and income.

Interviews will be conducted by QDRL and University of Michigan Institute for Social Research staff members in the QDRL with English speaking respondents for up to 90 minutes per interview as laid out in the data collection plan in Table 1 above. All interviews conducted in the Questionnaire Design Research Laboratory will be video and audio recorded to allow researchers to review the behaviors and body language of the respondents. Interviews conducted offsite will only be audio recorded. These recordings will allow researchers to conduct behavior coding in order to better evaluate the quality of the data captured in each mode. Video-over-internet interviews will be conducted using CDC issued laptops and Microsoft Lync software which has been provided to all CDC employees. Lync was selected for this project because it operates only on CDC servers versus other video conferencing software which would require that the video stream be transmitted to third party servers.

After respondents have been briefed on the purpose of the study and the procedures that QDRL routinely takes to protect human subjects, respondents will be asked to read and sign an Informed Consent document.

The interviewer will then ask the respondent to confirm that he/she understands the information in the Informed Consent, and then state that we would like to record the interview. The recorder will be turned on once it is clear that the procedures are understood and agreed upon. The interviewer will then orient the respondent to the cognitive interview with the following introduction:

Lauren may have told you that we will be working on some questions that may be on, or will eventually be added to, national and international health surveys. Before that happens, we would like to test them out on people like you. Most of the questions in this study are about your health and your behaviors. We are interested in your answers, but also how you go about making them. I may also ask you questions about the questions—whether they make sense, what you think about when you hear certain words, and so on.

I will read each question to you, and I’d like you to answer as best you can. Please try to tell me what you are thinking as you figure out how to answer. Also, please tell me if:

• there are words you don’t understand,

• the question doesn’t make sense to you,

• you could interpret it more than one way,

• it seems out of order,

• the answer you are looking for is not provided.

The more you can tell us, the more useful it will be to us as we try to develop better questions. Okay? Do you have any questions before we start?

For the mode effects sub-study, and in order to make comparisons between modes, all cognitive interview probes have been set for the non-disability, non-health insurance exchange questions in order to provide standardization across interviews. These structured probes are all attached to their corresponding questions in Attachment 1b. Only the non-disability (N1-N5, found in Attachment 1b on pages 9-10) questions are being assigned structured probes. The probing of the disability questions will follow the normal semi-structured procedures laid out in the QDRL’s generic package. Each interview in this sub-study will be followed by a semi-structured debriefing session with the respondent which will allow the interviewer to discuss any inconsistencies that arose as well as additional questions that may have arisen as part of the scripted portion of the interview.

After the interview, respondents will be given the thank-you letter signed by Charles J. Rothwell, Acting Director of NCHS, a copy of the informed consent document, and $40. Extreme care will be taken with all recordings and paperwork from the interviews conducted off-site. Recordings and identifying paperwork will be stored in a secured travel case until returned to NCHS, at which point they will be transferred to the usual secured locked storage cabinets.

We propose paying participants $40, which is our standard payment. In total, for this project, the maximum respondent burden will be 180 hours of interviewing. A burden table for this project is shown below:

Projects	Number of Participants	Number of Responses/ Participant	Average hours per response	Response burden
QDRL Interviews
2) Other Questionnaire Testing	120	1	1.5	180

Attachments (3)

cc:

M. Moien

T. Richardson

DHHS RCO

5