RANDS 10-Day Letter

DEPARTMENT OF HEALTH & HUMAN SERVICES Public Health Service

Centers for Disease Control and Prevention

National Center for Health Statistics

3311 Toledo Road

Hyattsville, Maryland 20782

January 24, 2018

Margo Schwab, Ph.D.

Office of Management and Budget

725 17th Street, N.W.

Washington, DC 20503

Dear Dr. Schwab:

The staff of the NCHS Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER), (OMB No. 0920-0222, exp. 07/31/2018) plans to continue a methodological study exploring if and how data from non-opt-in web panels may be linked to existing NCHS datasets and to quantify measurement error. Here, we are requesting approval for the first of two separate information collections: (1) the evaluation of a sub-set of National Health Interview Survey questions, both in their original interviewer-administered mode and a new self-administered mode, using cognitive interviewing methodology; and (2) the administration of a short web survey using a pre-existing, non-opt-in, commercial survey panel (in this case, GfK’s KnowledgePanel). We propose to start the first of these related projects—the cognitive evaluation of the Research and Development Survey (or RANDS) questionnaire—in January 2018. We will request approval for the second information collection—the web survey—in a separate generic IC. A description of the overall RANDS project is attached as Attachment 1.

We propose to start recruiting for volunteer participants as soon as we receive clearance and to start testing as soon as possible after that.

Background Information about Cognitive Testing of Questionnaires

The methodological design of this proposed study is consistent with the design of typical cognitive testing research. The purpose of cognitive testing is to obtain information about the processes people use to answer survey questions as well as to identify any potential problems in the questions. The analysis will be qualitative.

Proposed project: 2017-2018 Division of Research and Methodology Research and Development Survey (RANDS) Cognitive Interviewing Sub-Study

The staff of the NCHS Collaborating Center for Questionnaire Design and Evaluation Research (CCQDER) is requesting approval to conduct up to 80 total cognitive interviews, broken into at least two separate rounds. All interviews will take place at the CCQDER or at a mutually agreed-upon location (such as a library or an office), and all respondents will be sampled from the Washington DC and Baltimore areas. The testing procedure conforms to the cognitive interviewing techniques that have been described in CCQDER’s generic OMB clearance package (No. 0920-0222, exp. 07/31/2018).

The first round of cognitive testing will evaluate the proposed questionnaire for the first administration of the 2018 RANDS. This initial questionnaire will be comprised of current National Health Interview Survey (NHIS) (OMB No. 0920-0214, Exp. Date 12/31/2019) questions from the sample adult and sample adult functional disability questionnaires¹. This initial questionnaire can be seen in Attachment 2. The initial questionnaire we are proposing for the 2018 RANDS is similar to a previously submitted GenIC under this same generic that was approved on 9/18/2015. It covers a range of NHIS subjects, with a focus on chronic conditions and health behaviors. A few new topics are included in this year’s RANDS, including questions about sleep behavior and opioid use that the Division of Health Interview Statistics (DHIS) wants more information about prior to their planned redesign next year. While this GenIC is only asking for clearance to proceed with the cognitive interviewing portion of the RANDS project, the overall analysis plan for the eventual survey data is included as Attachment 9 to give you a sense of the purpose for including the items on the initial questionnaire that we did.

The round will include up to 20 interviews, and will evaluate the questions in their original, interviewer-administered format. The questionnaire shown in Attachment 2 includes questions that will be used in split-panel experiments in the eventual web survey. Cognitive interviewing respondents will only receive individual sets of the split sample questions (i.e. respondents will not receive both the Generalized Anxiety Disorder (GAD) and the Patient Health Questionnaire for Depression (PHQ) question sets). The number of respondents who receive each split set of the questionnaire will be dependent on the narrative data that emerges from the interviews—so while we will aim for an even split, the data itself will dictate the final distribution. Given the length of the questionnaire, not all questions will be probed in each interview. The interviewing team will administer probes to as many questions that they can during the allotted time, and will coordinate their interviews to ensure that sufficient data is collected on each question.

Following this first round of cognitive interviewing, the CCQDER will develop “structured probe questions” that will be added to the second administration of the RANDS/Knowledge Panel Sub-Project. These structured probes will be designed to elicit the specific patterns of interpretation that a respondent used when answering a proceeding NHIS question on the RANDS².

The second round will evaluate both the usability and validity of the self-administered version of the questions tested in the first round and the structured probes. The self-report versions of the questions will be designed following the first round of cognitive testing, and respondents will then take the survey on a laptop computer while a CCQDER interviewer observes and administers either concurrent or retrospective probes. This round will include up to 20 interviews. The questionnaire for this round of testing is not yet available, as the probe questions cannot be designed until after the first round of cognitive testing

A second set of two rounds of cognitive interviews may follow the first-round fielding of the 2018 RANDS if the project team decides to change the RANDS questionnaire between fieldlings. If this is the case, an additional 2 rounds of up to 20 interviews will be conducted in the same manner as noted in the above three paragraphs.

We propose to recruit the up-to 80 adult respondents (Age 18 and older) through newspaper advertisements, flyers, special interests groups, word-of-mouth, CCQDER Respondent Database, etc. The newspaper advertisement/flyer used to recruit respondents is shown in Attachment 3. The 5 minute screener used to determine eligibility of individuals responding to the newspaper advertisement/flyer, etc. or the CCQDER Respondent Database is shown in Attachment 4. Note that wording of the template has been approved and is contained within our umbrella package. Only project specific information has been added to the document. It is anticipated that as many as 120 individuals may need to be screened in order to recruit 80 participants. Within these constraints, we hope to recruit participants with some demographic (particularly in terms of gender, education, and race/ethnicity) variety.

Interviews averaging 60 minutes (including the completion of a Respondent Data Collection Sheet) will be conducted by CCQDER staff members with English speaking respondents. All interviews conducted in the Collaborating Center for Questionnaire Design and Evaluation Research will be video recorded to allow researchers to review the behaviors and body language of the respondents. Interviews conducted offsite will only be audio recorded. These recordings will primarily allow researchers to ensure the quality of their interview notes. Individuals who select “yes” for allowing the audio recording on the informed consent form, but “no” for retaining the recording for future research (final text before signatures on informed consent form), will be allowed to participate in the study.

After respondents have been briefed on the purpose of the study and the procedures that CCQDER routinely takes to protect human subjects, respondents will be asked to read and sign an Informed Consent document (Attachment 5). Only project specific information has been added to the document. Respondents will also be asked to fill in their demographic characteristics on the Respondent Data Collection Sheet (Attachment 6). This document is contained in our umbrella package. The burden for completion of this form is captured in the interview.

The interviewer will then ask the respondent to confirm that he/she understands the information in the Informed Consent, and then state that we would like to record the interview. The recorder will be turned on once it is clear that the procedures are understood and agreed upon.

The interviewer will then orient the respondent to the cognitive interview with the following introduction:

[fill staff name] may have told you that we will be working on some questions that will eventually be added to national surveys. Before that happens, we like to test them out on a variety of people. The questions we are testing today are about your health history, behaviors, and opinions. We are interested in your answers, but also in how you go about making them. I may also ask you questions about the questions—whether they make sense, what you think about when you hear certain words, and so on.

I will read each question to you, and I’d like you to answer as best you can. Please try to tell me what you are thinking as you figure out how to answer. Also, please tell me if:

there are words you don’t understand,

the question doesn’t make sense to you,

you could interpret it more than one way,

it seems out of order,

or if the answer you are looking for is not provided.

The more you can tell us, the more useful it will be to us as we try to develop better questions. Okay? Do you have any questions before we start? If yes, answer questions. If not, let’s get started.

The probing in this sub-study will follow the normal semi-structured procedures laid out in the CCQDER generic package. Each interview in this sub-study will be followed by a semi-structured debriefing session with the respondent which will allow the interviewer to discuss any inconsistencies that arose as well as additional questions that may have arisen as part of the scripted portion of the interview.

After the interview, respondents will be given the thank-you letter (document contained in umbrella package) signed by the Charles J. Rothwell, Director of NCHS (Attachment 7), a copy of the informed consent document, and a cash incentive (see below for details of the incentive amounts). After the cognitive interview is over respondents will be asked to read the Special Consent for Expanded Use of Video and Audio Recordings (Attachment 8). There will be no coercion and the respondents will be told that they can call and reverse the decision at any time if they change their minds. If respondents do sign the special consent form they will be given a copy of that as well.

Extreme care will be taken with all recordings and paperwork from the interviews conducted off-site. Recordings and identifying paperwork will be stored in a secured travel case until returned to NCHS, at which point they will be transferred to the usual secured locked storage cabinets.

We plan on doing some experimentation with incentives. We will use this project to begin exploring the impacts of different incentive levels on the recruiting. We plan on running three ads for two weeks each, and tracking metrics including the number of call backs (i.e. the number of potential respondents who call our recruiting phone line), the number of appointments made, the number of actual shows, and the characteristics of respondents recruited. The schedule for this test will be:

• Week 1-2: RANDS Ad for $40

• Week 3-4: RANDS Ad for $50

• Week 5-6: RANDS Ad for $60

Besides the incentive amount (and a reference code that the recruiters will ask the respondents to say during the screening process), all other aspects of the ads will be held constant across the three conditions, including where and when the ads are placed.

We propose giving participants $40 incentives, which is our standard incentive. In total, for this project, the maximum respondent burden will be 90 hours. A burden table for this project is shown below:

Form Name	Number of Participants	Number of Responses/ Participant	Average hours per response	Response Burden (in hours)
Screener	120	1	5/60	10
Questionnaire	80	1	1	80

Attachments (9)

cc:

V. Buie

T. Richardson

DHHS RCO

1 ftp://ftp.cdc.gov/pub/Health_Statistics/NCHS/Survey_Questionnaires/NHIS/2017/english/

2 The CCQDER has already successfully developed and administered these structured probe questions in previous projects. For example, see Chapter 9 in Miller, Willson, Chepp, and Padilla 2014; and Scanlon’s QDET and ESRA presentations and forthcoming chapter (available upon request).

5 | Page