NYTS Cognitive Testing Protocol

Submitted to:

CDC OSH

Submitted by:

ICF

ICF proprietary and confidential. Do not copy, distribute, or disclose.

February 23, 2022

Attachment R1. NYTS Cognitive Testing Protocol

Qualitative Inquiry to Support Tobacco-Related Surveillance and Applied Research

Table of Contents

Introduction 1

I. Segmentation Approach 1

II. Recruitment Plan 3

III. Interview Guides 4

1. Using the Interview Guides 7

IV. Interviewer Training Plan 8

1. Intro to Cognitive Testing / Best Practices 8

2. Interview Approach 9

3. Interview Logistics, Scheduling, and Receipt of Assignments 9

4. Post-Training Activities 9

V. Analysis Plan 10

1. Codebook 10

2. Coding 10

3. Conducting Analysis 11

VI. Data Management Plan 11

Introduction

This effort involves cognitive testing of the National Youth Tobacco Survey (NYTS). In cognitive testing, an interviewer administers survey questions to a respondent while collecting additional verbal information about survey responses with the goal of either evaluating the quality of the responses or determining if the wording is generating the intended information.¹ The results of the cognitive interviews may lead to revisions of survey questions and/or changes in question sequencing and overall questionnaire structure that will enhance response rate and data quality.

The Centers for Disease Control and Prevention (CDC) is partnering with ICF, a survey research organization, to perform cognitive testing of the NYTS. ICF’s cognitive testing plan involves testing the entire proposed NYTS 2023 instrument along with additional questions that may be included on a future NYTS. There are 244 questions to be tested, and interviews will take no longer than two hours to complete testing for this number of questions. The cognitive testing will involve a review of each item and discussion facilitated through cognitive probes or questions related to the respondent perceptions of the survey items. Probes are intended to cover the entire cognitive response process, including encoding, comprehension, retrieval, judgement, and editing.

This document represents the cognitive testing protocol. The protocol consists of the following sections.

1. Segmentation Approach

2. Recruitment Plan

3. Interview Guides

4. Interviewer Training Plan

5. Analysis Plan

6. Data Management Plan

The overall approach to cognitive testing will leverage the overall project Health Equity Framework as part of segmentation, training, and analysis to better understand the significance of NYTS items across varied individual (e.g., behaviors), interpersonal (e.g., health and language literacy, educational attainment, media influences), community (e.g., neighborhood safety, housing and tobacco exposure), and societal (e.g., discrimination) experiences and influences that may affect respondent perception and interpretation.

1. Segmentation Approach

ICF proposes four population segments with associated selection prioritization to ensure representation across the 40 cognitive interviews. The four population segments include: age (youth in grades 6 through 8, youth in grades 9 through 12); race/ethnicity (American Indian or Alaskan Native, Asian, Black or African American, Hispanic, Native Hawaiian or Pacific Islander, or white); low socioeconomic status (household income less than 250% the federal poverty level); and gender identity. This segmentation will provide adequate representation by age to draw meaningful conclusions of youth in middle or high school. Although we intend to recruit a diverse sample, oversampling participants of minority race and ethnicity and participants with low socioeconomic status will provide additional insights into health equity challenges experienced by vulnerable populations.

Two screening surveys will be administered to collect demographic and tobacco product use status data (one for parents, and one for students—following parental consent). These data will be used to identify respondents who qualify for one or more segments of interest. First, we will invite parents to complete the screener survey about their children. Once we achieved a sufficient response from parents, we will first move to balance each segment by age, gender, race/ethnicity, and household income. Following student screening, will diversify each segment using tobacco product use status, environment, sexual orientation, and gender identity. [See the next section on recruitment for more detail on how ICF will achieve diverse segmentation in the sample population.] By monitoring the representation across each segment during recruitment, we will be alerted to fill gaps in underrepresented segments and, as a result, adjust our recruitment efforts. In selecting respondents to invite to participate in the interviews, we will prioritize diversification of selected respondents by tobacco product use status, race/ethnicity, and geography. Exhibit 1 displays the complete systematic selection process for diversifying each segment. Additionally, Exhibit 2 includes a graphic representation of the segmentation flow through the screener surveys. Recruitment is described in more detail in the next section.

Exhibit 1. Segmentation and Selection Prioritization

Exhibit 2. Recruitment Screener Flow

2. Recruitment Plan

ICF will recruit respondents in grades 6 through 12 (age approximately 11 to 18 years) through two stages. Stage 1 ensures national reach, while Stage 2 allows more targeted supplementing of specific groups.

Stage 1 involves recruiting children who are in grades 6 through 12 from the nationwide web panel MFour Surveys on the Go®. ICF has collaborated with MFour Surveys on the Go® for dozens of research studies targeting specific demographic groups. Interview participants will be recruited through adult panelists who have children in their household (which is a characteristic included in the panel profile). These parents/guardians will be prompted to complete a screener survey (see Attachment B. NYTS Cognitive Testing Parent/Guardian Screener), which will allow us to identify those with children aged 11 to 18. Parents with children aged 11 to 18 who are deemed eligible will then be sent further information about the study, as well as a form to obtain consent for their child to participate (see Attachment C. NYTS Cognitive Testing Parent/Guardian Consent). After obtaining parental consent, students will then be sent an invitation (Attachment D. NYTS Cognitive Testing Invitation to Youth Screener) to complete a youth screener survey so that ICF can obtain additional relevant information (see Attachment E. NYTS Cognitive Testing Youth Screener). ICF will draw on this information to achieve the desired segmentation when selecting students to invite to participate. Students who are selected will be sent a consent form to complete prior to participation (see Attachment F. NYTS Cognitive Testing Youth Consent). All students who complete the screener but are not selected will be informed about this (see Attachment G. NYTS Cognitive Testing Notice of Non-Selection). The screener surveys, as well as the consent and assent forms are written for a 6^th grade reading level to ensure ease of access to information for both parents and youth.

To address gaps in age groups or other segments not found using MFour, ICF will initiate Stage 2 of our recruitment strategy. This next step includes working with another recruitment firm, Research America, to address any gaps in recruitment. ICF has worked extensively with Research America on several previous projects to successfully recruit hard-to-reach populations including youth, ethnic minority populations, and people who use drugs. Research America uses a national panel to recruit participants, and conducts extensive outreach with national, regional, and local organizations to supplement their efforts to support specific outreach efforts. ICF will again obtain parental consent, student assent, and conduct a short youth screener with all participants identified through Research America. This two-stage approach helps ensure a cognitive testing sample that fills each of the segments discussed above in Section 1: Segmentation Approach.

3. Interview Guides

In cognitive interviewing, it is essential to provide enough time for respondents to express challenges they have in answering questions, and for the cognitive interviewers to probe. Moving too quickly through a cognitive interview can reduce the quality of cognitive data obtained.² ICF believes that splitting the survey items into three sets will provide the optimal balance of covering a significant portion of the survey, while allowing interviewers to build rapport and achieve a working flow with participants, during an interview that is no longer than two hours.

Each of the three guides contains interviewer introductions, required and optional probes at the section- and item-level as well as the optional generic probes (see Attachment H. NYTS Cognitive Testing Guides). The interview guides also include space for notetaking. Trained ICF interviewers will employ these guides to gather the respondent cognitive feedback.

To accommodate all NYTS 2022 and potential new questions in this cognitive test, we plan to split survey items into three sets, each of which will be tested independently. Each participant will also see items from one of the sets. The item sets include the follow:

• Item Set 1 - Demographics, E-Cigarettes and Heated Tobacco Products, Messaging, Social Media, Advertisements, and Exposure to Secondhand Tobacco Product Emissions

• Item Set 2 - Cigarettes and Other Combustible Tobacco Products, Opinions, and Discrimination

• Item Set 3 - Smokeless Tobacco Products, Healthcare, Opinions, Environment, and Home and School

The following three Exhibits (3-5) document the modules and flow within each item set.

Exhibit 3. Item Set 1 – Demographics, E-Cigarettes and Other Heated Tobacco Products, Messaging, Social Media, Advertisements, and Exposure

Exhibit 4. Item Set 2 – Cigarettes and Other Combustible Tobacco Products, Opinions, Discrimination

Exhibit 5. Item Set 3 – Smokeless Tobacco Products, Healthcare, Opinions, Environment, and Home and School

1. Using the Interview Guides

Respondents will be assigned to one of the three item sets based on their responses to the screener survey to ensure that every item is tested. For example, we will ensure that some e-cigarette users as well as some e-cigarette non-users are assigned to the set of items that includes the e-cigarette items (Item Set 1). This will ensure that both the “current user” and “never user” items are tested in the e-cigarette section. For rarer tobacco products, such as bidis and nicotine pouches, if no users are identified in the screening process, we may show respondents questions that do not apply to them to test the comprehensibility of these items.

Based on their responses to key gate items (e.g., having used e-cigarettes in the past 30 days), respondents may see between 38 to 91 items during the interview. Due to the wide range of possible paths through the questionnaire, we have not included required probes for each individual item – some items that present more difficult cognitive tasks do have required probes, while others will be probed at the section-level. Additionally, interviewers will be instructed to probe whenever a respondent hesitates or seems unsure about an item, using generic probes laid out in the interviewer guides. At the same time, interviewers will ensure that interviews can be completed within the two hours allotted.

Interviews will be conducted virtually through Microsoft Teams video calls, and each interview will be attended by the interviewer and notetaker. One advantage of using MS Teams is that participants do not need to download or purchase anything to access meetings. MS Teams meetings are accessible through standard browsers without the need to download any software. Participants will be provided with instructions for accessing the meeting via their browser. All interviews will also be audio recorded and transcribed through MS Teams. Interviewers will use the MS Teams screen share feature to show participants the relevant survey items. To accommodate the questionnaire’s skip patterns and ensure that every item is tested, respondents will be shown a version of the web questionnaire that has been programmed to present only the items in their assigned item set. After introductions and a simple practice question, the interviewer will guide the respondent to through their item set. The interviewer will instruct the respondent to read and verbally respond to each question. The interviewer will enter the responses and advance through the survey. Interviewers will stop at the end of each section to probe. Interviewers will similarly proceed through each guide, entering participant responses to items, and using specific, generic, or emergent probes for in-the-moment probing on items that appear challenging for the respondent.

4. Interviewer Training Plan

ICF will assemble a team of experienced qualitative researchers from a range of backgrounds and lived experiences. Task Leads and ICF survey methodologists will train these qualitative interviewers to conduct the NYTS cognitive testing. Our training strategy will take advantage of the opportunity to share consistent language, messaging, and qualitative interview approaches. The training on cognitive testing will consist of three primary components: (1) introduction to cognitive testing/best practices, (2) interview approach, and (3) interview logistics, scheduling, and receipt of assignments.

1. Intro to Cognitive Testing / Best Practices

Below is an initial draft of the training topics to be covered for the cognitive testing process. The training will consist of the following elements:

• Cognitive Testing Process in A Nutshell

  • What is Cognitive Testing?

  • Why Do We Do Cognitive Testing?

• The Cognitive Response Process

  • Encoding and Storage

  • Comprehension

  • Retrieval

  • Integration/Estimation/Judgement

  • Mapping

  • Editing

• Common Types of Issues Uncovered by Cognitive Interviewing

• Probing Approaches

  • “Think-Aloud” Approach

  • Verbal Probing Approach

  • Concurrent Probing

  • Retrospective Probing

• Basic Varieties of Verbal Probes

• Tips for Probing

• Example of Neutral Probes

• Common Respondent Behavior

• General Interviewing Best Practices

2. Interview Approach

Trainers will provide a detailed overview of all cognitive testing materials. This will include moderator guides for the three item sets, and the equivalent sets of NYTS questions that respondents will see. Trainers will provide an orientation to the basic set up of the guides, and what information is associated with each of the columns. Trainers will further describe the general flow and mechanics of each interview, including how an interviewer will both use the guide and the respondent-facing question sets. Trainers will then cover each of the three guides in details, including which NYTS questions are in each guide and how to navigate modules to module in each guide. Interviewers will be expected to develop competence across all guides. As noted above, trainers will also leverage the overall project Health Equity Framework to foster interviewing with cultural humility and cultural sensitivity relevant to individuals across the different participant segments.

Notetakers will also attend the training and will be trained on the required format of interview notes and the naming convention for notes and transcripts. Trainers will also instruct interviewers as to what kind of information needs to be coded in notes following interviews to help facilitate future analysis. Following thorough review of the guides and process, interviewers and notetakers will be given the opportunity for some initial practice with the guides. Trainers will observe and provide initial feedback.

3. Interview Logistics, Scheduling, and Receipt of Assignments

Finally, training will cover important logistical considerations for cognitive interviewing, including the overall project schedule, using MS Teams to conduct the interviews, screensharing, recording and transcribing, receiving assignments and scheduling, and how to report both a completed interview and a no-show.

4. Post-Training Activities

Immediately following interviewer training and prior to their first interview, trainers will oversee a dry run with all interviewers and notetakers to ensure they are comfortable with specific study protocols and technology. In addition, after each interviewer has completed their first interview, the trainers will lead a short debrief to talk through any potential issues with the guides for each study, the length/timing of the sessions, using MS Teams, or other challenges that may have arisen. The goal is to resolve challenges before proceeding to subsequent interviews. Interviewer check-ins will continue throughout the course of each task.

5. Analysis Plan

The following represents a preliminary analysis plan. Early in the data collection processes, ICF will submit to OSH expanded plans for data analysis and reporting for cognitive interviews.

1. Codebook

As a first step in qualitative data analysis, the analysis team will use an abductive coding process to develop a codebook to guide thematic analysis. The initial codebook will contain deductive codes, which will be developed based on the interview items and cognitive goals. In addition, the team will conduct an inventory of the data through review of a subset of the transcripts and notes to assess emerging themes and inform the development of inductive codes designed to capture issues or stories that may be unexpected, important to include, and/or unlikely to be captured by the deductive coding scheme. The team will develop inclusion and exclusion criteria for codes, provide explicit guidance for applying the codes, and ensure that codes are both comprehensive and mutually exclusive to avoid misclassification of data. This initial codebook will be provided to OSH for review and feedback.

It is anticipated that the codebook may be updated to include additional codes or revisions as the analysis process proceeds. To ensure coding trustworthiness, analysts will meet to discuss the process, finalize the codebook, and help interpret preliminary results. In addition, analysts will code the same transcript (or sections of multiple transcripts) to compare accuracy and establish intercoder reliability.

2. Coding

All data from the interviews (i.e., transcripts and notes) will be imported into QSR NVivo®, which will be used to organize and manage the data and to conduct analysis in an iterative fashion. As noted, initial codes will be developed based on interview items and cognitive goals. In addition, the software program will facilitate the development of additional codes as the analyses are conducted. The software program will support the development of a coding system and coding process by allowing the text to be marked for subsequent search, retrieval, and classification. The software will facilitate interpretation of the data, identification of themes, tracking of representative quotes and text segments, and reporting of results. Analysts will review the transcripts and notes and select and code sections of text to organize responses according to analytic categories.

To ensure intercoder reliability, all coders will undergo a brief training prior to beginning live coding. Next, coders will code approximately 10% of the transcripts (to be randomly selected). As a team, coders will then review and resolve any differences in coding across these transcripts. Coders will also periodically and proactively share their ongoing coding experience, questions, and potential tweaks. Coders will also regularly share and discuss new codes and/or themes as these emerge. Any new codes and/or themes will be shared with OSH before formally being added to the codebook.

3. Conducting Analysis

Upon completing the coding, analysts will prepare output for analysis by selecting a code and/or codes, retrieving the relevant text, and saving the output in files for analysis. The coding software will allow analysts to organize and pull codes across all interview items, cognitive goals, as well as by respondent types. Analysts will examine all responses and/or categories in the output to identify patterns and themes in the data.

6. Data Management Plan

All documents produced and/or collected before, during, and after each interview (e.g., screener data, audio files, interview notes, transcripts) will be stored in password-protected electronic files accessible only to ICF project team members. Data files will be stored on a secure drive by respondent ID, a five-digit code used to track all data collection. Respondent names, or other identifiers, will never be used. The separate, task specific, secure drive will only be available to a limited number of project staff working on the analysis.

The data storage ‘system’ consists of the ICF corporate laptops of those individuals authorized to participate in the study (estimated to be 6-10 individuals), and a secure shared drive. Each laptop is its own boundary and requires 2-factor authentication to log in. The laptop operating systems are Windows 10. The laptops will run Microsoft Office software which may be used for list data. De-identified response data will be captured to MS Office documents and then uploaded to NVivo for analysis. Microsoft Teams will be used to capture audio recording of interviews. Data sharing between team members within ICF will happen via the secure folder, as will delivery of results to the CDC when done. All list data, response data, and audio recordings will be wiped from the laptops at the conclusion of the project using drive-wipe software with multiple passes.

At the conclusion of the project, analyzed and deidentified data (audio files and transcripts) will be delivered (disassociated from any PII) to CDC.

1 Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage Publications.

2 Willis, G. B. (2005). Cognitive interviewing: A tool for improving questionnaire design. Sage Publications.