Memo Requesting Clearance for CIR Cognitive Testing

U.S. Department of Justice

Office of Justice Programs

Bureau of Justice Statistics

Washington, D.C. 20531

MEMORANDUM

To: Jennifer Park

Office of Statistical and Science Policy

Office of Management and Budget

Through: Melody Braswell

Clearance Officer

Justice Management Division

Jeri M. Mulrow

Acting Director

Bureau of Justice Statistics

From: Jennifer Truman and Lynn Langton

Bureau of Justice Statistics

Date: February 9, 2017

Re: BJS Request for OMB Clearance for Cognitive Testing under the National Crime Victimization Survey (NCVS) Redesign Generic Clearance, OMB Number 1121-0325.

The Bureau of Justice Statistics (BJS) requests clearance for new cognitive interviewing tasks under the OMB generic clearance agreement (OMB Number 1121-0325) for activities related to the National Crime Victimization Survey Redesign Research (NCVS-RR) program. BJS, in consultation with Westat under a cooperative agreement (Award 2013-MU-CX-K054 National Crime Victimization Survey (NCVS) Instrument Redesign and Testing Project), is working to redesign the NCVS survey instrument and mode of administration. The NCVS was last redesigned more than two decades ago, in 1992. Much has changed in the interim, both in the level of public acceptance of surveys and in terms of the nature of crime. The primary purpose of the NCVS Instrument Redesign and Testing Project is to provide scientific and technical support for the redesign and testing of the NCVS roster control card, crime screener (NCVS-1), and crime incident (NCVS-2) instruments in support of BJS’s efforts related to increasing the efficiency, reliability, and utility of the NCVS.

For this clearance, cognitive interviewing will be used to develop and test parts of the revised Crime Incident Report (CIR, based on the current NCVS-2) and proposed new questions about community safety that would be asked of all or a sample of participants regardless of whether they reported a victimization. This clearance includes three rounds of testing with adults who have been victims of specific types of crime, and one round of testing with youth ages 12 to 17, to gather their reactions to police ask-all items, new screening items, and detailed crime incident items.

In-person cognitive testing will be conducted iteratively with up to 120 persons over multiple rounds of testing, including 90 adults and 30 youth ages 12 to 17. These cognitive testing tasks will require up to 180 burden hours, and will help identify issues with question wording or content.

A prior cognitive testing clearance focused on adult testing of the crime screener and police ask-all items. The results from the currently requested interviews, along with the results of the screener testing, will be used to prepare a suite of instruments for web-based administration by interviewers and/or respondents. The web instruments will undergo usability testing and a formal pretest, with clearance requested later under a separate generic agreement. Finally, BJS plans to conduct a large, two-wave field test under a separate clearance request. The field test will involve administering the survey to a representative sample of persons age 12 or older, testing aspects of the design such as mode, response rates, and administration times. OMB approval for usability testing and the pretest will be sought after completion of the cognitive testing in 2017. We anticipate usability testing in late 2017, the pretest in early 2018, and requesting clearance for the field test in mid-2018.

Description and Purpose of Overall Project

This request is for clearance to conduct cognitive interviews to develop and test sections of the redesigned NCVS Crime Incident Report (CIR) in conjunction with a victimization screener. The cognitive interviews will inform the design and the final instruments to be tested in a larger field test. Before describing the proposed cognitive interviews, the goals and research questions for the larger NCVS Instrument Redesign and Testing Project are described.

The NCVS is based on research conducted by the Department of Justice and the Census Bureau in the 1970s (summarized in Lehnen and Skogan, 1981; Skogan, 1990; Skogan and Lehnen, 1985). There was a major redesign implemented in 1992, motivated in part by a National Academy of Sciences (NAS) review (Penick and Owens, 1976). A more recent review by the NAS (Groves and Cork, 2008) provided similar motivation for the current BJS redesign effort.

Since 2008, BJS has initiated a number of research projects to assess and improve upon core NCVS methodology, including redesigning the sample plan, comparing alternative modes of interviewing, testing methods for reducing non-response bias, experimenting with various reference period lengths, testing the effectiveness of new victimization screening questions, and exploring the feasibility of producing sub-national estimates of victimization.

The current effort is part of BJS’s work (with Westat under a cooperative agreement) to come up with a new design for the NCVS. The overarching objective is to provide scientific and technical support for the redesign and testing of the NCVS roster control card, crime screener (NCVS-1), and crime incident report (NCVS-2) in support of BJS’s efforts to increase the efficiency, reliability, and utility of the NCVS. Through this project, BJS aims to evaluate and modernize the organization and content of the NCVS; improve the efficiency of the instruments and the current core-supplement design; and develop a procedure for introducing routine improvements to the survey in order to capture emerging crime types and time-relevant topics.

One of the first steps in the project was a comprehensive assessment of the instruments to determine which survey items are being used and how; which survey items are problematic in their language and placement; and where there are gaps in the content of the instrument. The initial assessment provided a better understanding of the substantive and procedural issues with the instrument and helped to identify areas where the content could be improved to enhance current knowledge of victimization and its correlates.

Additional efforts are on modernizing the crime screener: improving the measurement of highly sensitive crimes like rape and sexual assault and intimate partner violence; increasing the relevance and utility of the crime incident report (CIR); and building in a series of questions on perceptions of police and community safety to be asked of all respondent (‘ask all’/or ‘noncrime’ items). The anticipated changes to the types of crimes measured by the NCVS, and improvement of the measurement process, may require changes to the survey methodology to ensure that the information collected is accurate and reliable. Through this project, we will also examine the feasibility of using self-administered approaches to collect data comparable to those obtained through interviewer administration.

Overall, the NCVS Instrument Redesign and Testing Project will be used to inform improvements in the —

• NCVS screener and crime incident instruments;

• mode of data collection for the NCVS;

• flexibility to measure emerging crime types; and

• ability to capture indictors of safety and security and perceptions of police that go beyond experiences with victimization.

Current Request for Cognitive Testing

In the current request, we are asking for clearance to conduct three additional rounds of cognitive interviewing with adults, and a series of interviews with youth to support this overall redesign effort.

Adults: Cognitive testing with adults will focus on several revised sections of the CIR, on the interview flow from the victimization screener to the CIR, and on community safety items to be asked of both those reporting and those not reporting victimizations. Round 1 will specifically focus on items asking about the location of the crime, the presence of others, the involvement of police, and the use of victim services following the incident (Attachment A1). Round 1 will be interviewer-administered with paper instruments and will include 30 adults, which will allow BJS to test the items with sufficient numbers of victims of different types of crimes, including vehicle theft, theft of other items, attacks, break-ins, and vandalism.

Round 2 will specifically focus on items that assess what happened during the incident, the victim-offender relationship, self-defense items, and the community ask-all items (Attachment A2). Round 2 will be programmed as a self-administered Web-based instrument, and again will include 30 adults who have experienced the different types of crimes mentioned above.

Round 3 will tie all CIR sections developed to date together for a comprehensive test of the self-administered instrument. Cognitive testing will be conducted in person with 30 participants, using primarily concurrent probing techniques to obtain a more nuanced understanding of how respondents are conceptualizing and answering each survey question. The cognitive interviews will address the structure and language of the revised CIR sections, and specifically the following:

• Rounds 1 and 3 (Attachment A1):

  • Reactions to the location items to improve the efficiency of asking where the incident occurred;

  • Reactions to the presence items to ensure BJS is collecting valid and reliable measurement of who was present during the incident and who witnessed the incident;

  • Reactions to the police involvement items to improve the ability of interviewers and respondents to identify reasons why police are/are not contacted; and

  • Reactions to the victim services items to improve the assessment of services received after the incident.

• Rounds 2 and 3 (Attachment A2):

  • Testing the “what happened” series of items in the CIR, which, in conjunction with the revised screening items, should provide a comprehensive picture of the type of crime that occurred;

  • Reactions to the victim-offender relationship and self-defense items to ensure we are accurately and completely capturing these concepts; and

  • Reactions to the community safety items to ensure they are applicable to all types of respondents and are easily understood with low burden.

Youth: Cognitive testing with 30 youth ages 12 to 17 will be conducted with 15 youth ages 12 to 14 and 15 youth ages 15 to 17. We aim to recruit youth such that at least 20 out of the 30 youth have been the victim of a crime in the past 12 months.

• Youth who have not been the victim of a crime will receive the long cue screening items and police ask-all items and their associated cognitive probes. If time permits, they will also be provided with short fictional narratives and asked to answer specific screening items based on their understanding of the narratives (Attachments A3 and A3A).

• Youth who have been the victim of a crime will receive the short cue screening items and one set of the CIR items being tested among adults. If the youth answers no to all of the screening items, indicating they have actually not been the victim of a crime in the past 12 months, they will be provided with short fictional narratives, rather than the CIR items, and will be asked to answer specific screening items based on their understanding of the narratives (Attachments A1 and A2).

Cognitive testing will be conducted with a mix of youth, some of whom live in households in which the adult experienced a crime (and is being recruited for the adult testing), and others whose parents were not necessarily crime victims. To target recruitment activities toward youth victims of crime, we will identify youth victim services organizations and other youth service organizations that may be willing to support recruitment efforts. Parental consent and child assent will be required for all interviews and all recruitment activities will be conducted through the parent or guardian to obtain permission to interview the youth. If parents accompany the youth to the interviews, the youth interviews will be conducted in private without the parent present.

Cognitive Probing

Because the questionnaire design is in a relatively early stage, all of these assessments will be informal and qualitative. Cognitive interviewing will be used to identify potential problems with each version of the questions. Trained cognitive interviewers will administer primarily concurrent probes to gather additional information about comprehension and the response process. (See Attachments A1 through A3A for the instruments and probes). Some probes will seek to determine whether the participant understands specific terms and concepts, such as “near your home” or “notified the police.” Other probes will be used to determine whether the CIR items accurately capture what happened and whether the participants understood the questions easily and as intended. Interviewers will also use unscripted probes if the participant shows signs of difficulty, confusion, or frustration (e.g., “You seem to be having trouble with this question. Can you tell me what the problem might be?”).

Data Collection

The target population for each round of 30 cognitive interviews with adults is persons age 18 or older who have experienced particular types of crimes, including attempted or completed vehicle theft, theft of other items, attacks, break-ins, or vandalism. For the youth testing, the target population is 30 youth ages 12 to 17, with a focus on interviewing 15 younger youth (ages 12 to 14) and 15 older youth (ages 15 to 17). Ideally, we will recruit most of the youth from households that have experienced crime, so that even if the youth has not been the victim of a crime himself/herself, someone in their household will have been a victim. However, we will also work with youth victim services organizations to identify youth who may be recruited for an interview without a parents’ participation in an adult interview. It will be necessary to recruit youth through their parents, directing all recruitment activities towards parents, conducting the screening questionnaire with the parent to obtain permission to interview the youth, and collecting written informed consent from the parent and verbal youth assent prior to launching the interview.

We have selected 6 markets in which to conduct the interviews, three of which will be in person, and three of which will be conducted remotely. Three of the markets (Rockville and Baltimore MD and Cleveland OH) have been selected primarily because Westat can take advantage of local interviewers. All youth interviews will be conducted in person in Rockville, Baltimore, or Cleveland.

We also plan to recruit adult crime victims living in Chicago IL, St. Louis MO, and Oakland CA, metropolitan areas with relatively high rates of crime according to the FBI Uniform Crime Reports. We will conduct those interviews remotely using a web-based meeting tool such as FocusVision, Skype or Goto Meeting. Respondents in the remote locations will be asked to travel to a convenient local focus group facility, and will participate in the interview remotely from that location. Westat maintains a strong network of focus group facilities around the country that use these technologies. This approach allows us to minimize travel costs for Westat researchers while allowing broader geographic coverage, access to more varied types of crimes, and a private setting in which to conduct the interview.

The questions will also use a twelve-month reference period to increase the number of respondents with victimizations. The only other eligibility criterion is that the person speak English. All in-person interviews will take place at the Westat offices, at professional interviewing facilities in Cleveland and Baltimore, or conducted remotely with the respondent sitting at a professional focus group facility and using the facility’s online streaming tool to communicate with the Westat interviewer.

Across the multiple iterations of cognitive testing, in-person interviews will be conducted with up to 120 respondents. One-on-one interviews will last approximately 90 minutes to allow time for the administration of the screener and detailed crime incident items, as well as cognitive probing. Adult participants will receive $60 ($40 per hour) as compensation for their time and to offset the cost of their participation, such as transportation, parking and childcare. Youth will receive $40 (about $25 per hour).

Recruiters will advertise the study to solicit participation, using internal recruiting databases, as well as newspaper ads, fliers, and Craigslist ads. We will also pursue Westat and BJS contacts at victim services organizations to help advertise the study (see Attachment B1 for the letter to be circulated to victim services organizations and Attachment B2 for the recruitment language for those organizations to share with known victims). Recruiters will adapt recruiting strategies as needed to ensure adequate participation. Interested participants who contact study recruiters will be asked a series of screening questions to determine their eligibility. Recruited adults will have been a victim of at least one crime in the past 12 months. (See Attachments C1 and C2 for the recruitment screening questionnaire and the Craigslist ad.) Adults selected to participate will be contacted by the recruiters and scheduled for their interview session. If the adult has a youth ages 12 to 17 in the household, they will also be asked if the youth would be interested in participating in a concurrent interview. Parents will be asked if the youth has been a victim of any crime in the past 12 months. Individuals will be selected to the extent possible to achieve diversity by age, gender, educational attainment, race, and ethnicity across the interviews.

All cognitive interviews will be audio-recorded with the participant’s consent. The audio recordings will only be accessible to project staff directly working on the project and no names or other personally identifying information (other than the participant’s voice itself) will be included interviewer summaries of the audio recordings.

Language

The cognitive interviews will be conducted in English.

Burden Hours for Cognitive Testing

The burden for this task consists of participants being screened for and subsequently participating in in-person or remote cognitive interviews. Each of the first two rounds of adult cognitive testing will cover different CIR sections, and the third round will navigate respondents through the entire CIR, incorporating revisions from prior testing. Similarly, the youth cognitive interviews will focus on testing content that has been revised based on testing with adults, focusing on both screener and CIR items. The burden associated with these activities is presented in the following table.

Table 1. Burden Associated with Planned NCVS-R Cognitive Testing Activities

Data Collection Type	# of Respondents per Round			Total # of Respondents	Average Administration Time (minutes)	Burden (hours)
	Round 1	Round 2	Round 3
In-person cognitive interviewing with adults	30	30	30	90	90	135
In-person cognitive interviewing with youth	30			30	90	45

Justification of Respondent Burden

Every effort has been made to minimize respondent burden and government cost in the conduct of these proposed tests. The size of the sample needed to support analysis of the in-person cognitive interviewing was driven by the goals of the testing. The 90 adult interviews are split across 3 rounds of testing. In each round, we will recruit roughly 5 to 6 victims each of different types of crimes, including vehicle theft, theft of other items, assault, break-ins, and vandalism to ensure that all items in the CIR are tested. The 30 youth interviews will be evenly divided among younger and older youth, and will aim to test the comprehension of items in the long cue and short cue screeners as well as CIR items being tested with adults.

While prior rounds of testing approved by OMB could be completed in one-hour interviews, there is an increased level of content being tested in the current proposed testing. As such, we are anticipating the interviews to take approximately 90 minutes to administer.

Cost to the Federal Government

The total cost of conducting the cognitive interviews, including incentives for in-person interviews, will be approximately $125,000 under the cooperative agreement with Westat (Award 2013-MU-CX-K054) for the National Crime Victimization Survey (NCVS) Instrument Redesign and Testing Project.

Data Analysis

Cognitive interviewers will summarize the findings from each cognitive interview. Interviewers will prepare summary findings on each completed interview based on the completed questionnaire, notes taken during the interview, and associated audio recordings. The summaries will be analyzed using qualitative data analysis software to help identify common themes organized by overall questionnaire issues, individual questionnaire items and sections, and participants’ overall reactions to the questionnaire.

The cognitive interviewing analysis will assess and identify problems, such as comprehension issues, recall problems, difficulties understanding the task, and telescoping. This analysis will be conducted separately for adults and children.

Upon completion of all cognitive testing, a draft cognitive interviewing report will be prepared that will include recommendations for final revisions to the survey for the pretest and field test. These recommendations will provide detailed information on the cognitive testing methodology, basic characteristics of the respondents, and any issues with question comprehension. The report will also document changes made to the initial draft NCVS survey instruments that are being recommended for use in the usability testing, pretest, and field test.

Protection of Human Subjects

There is some risk of emotional distress for the respondents given the sensitive nature of the topic, particularly since the questions are of a personal nature. The planned cognitive testing will be reviewed by Westat’s Institutional Review Board (IRB), which has Federal-wide assurance.

Interviewers will be trained to recognize when respondents are becoming emotionally upset. They will also be trained on how to react when they are upset. Any respondent (adult or youth) who seems to be in distress will be asked if they wish to stop the interview. The respondent will also be offered the opportunity to call a national hotline counselor (e.g., domestic violence or suicide prevention counselor). All respondents will be provided with a list of hotline numbers before leaving the interview session, regardless of whether the interviewer identified any distress.

Informed Consent

Adult participants will review the informed consent document with the aid of the interviewer and will be asked to sign the form before participating. Should an individual refuse informed consent, s/he will be excused from participation and thanked for her/his time. At the time of consent/assent, the interviewer will also ask permission to audio record the interview. (See Attachment D for the informed consent documents).

To interview youth participants we must receive (1) parent/guardian consent and also (2) youth assent. Those youth who are coming to a concurrent interview with a parent will be interviewed in a separate, private room to ensure confidentiality. Written parental/guardian consent will be obtained before the interview begins. Since Westat is covered by BJS Title 42 USC §3735 and §3789g, we are unable to report any instances of abuse and must maintain strict confidentiality of the data. As such, we plan to collect verbal assent from youth rather than written; the interviewer will sign the assent form rather than the youth. We will not maintain the names of the children to ensure we are unable to link the interview data with a child’s identifying information. Should a parent or a child refuse informed consent, the child will be excused from participation and thanked for her/his time. In both the parental consent and youth assent process, the interviewer will ask for permission to audio record the interview.

Use of Information Technology to Reduce Burden

The cognitive testing study will utilize technology to facilitate recruitment and the scheduling process while also reducing participant burden and controlling study costs. Recruitment efforts will use email communications when possible, because participants increasingly prefer to communicate via email so they can respond when it is convenient. Using email for recruitment and scheduling can help to reduce participant burden and save time and money that would otherwise be spent conducting telephone calls, leaving voice messages, and making call-backs.

The study will also take advantage of technology to conduct some of the interviews remotely, allowing access to a broader geographic population without incurring the costs of travel. Respondents in Chicago, St. Louis and Oakland, CA will be asked to go to local focus group facilities and log onto an online streaming tool to interact with a live researcher who will conduct the interview remotely.

Data Confidentiality and Security

BJS’s pledge of confidentiality is based on its governing statutes Title 42 USC §3735 and §3789g, which establish the allowable use of data collected by BJS. Under these sections, data collected by BJS shall be used only for statistical or research purposes and shall be gathered in a manner that precludes their use for law enforcement or any purpose relating to a particular individual other than statistical or research purposes (Section 3735). BJS staff, other federal employees, and Westat staff (the data collection agent) shall not use or reveal any research or statistical information identifiable to any specific private person for any purpose other than the research and statistical purposes for which it was obtained. Pursuant to 42 U.S.C. Sec. 3789g, BJS will not publish any data identifiable specific to a private person (including respondents and decedents). To protect the identity of the respondents, no identifying information will be kept on the final data file.

Participation in this cognitive testing study is voluntary. Personally identifiable information (PII), including names and contact information (phone number and/or email address), will be collected by recruiting facilities for the purpose of scheduling eligible participants for interviews. These data will be securely stored in password protected files to which only project staff will have access, and will be destroyed after the study is finished. Names provided by adult participants on consent and incentive receipt forms will be stored in locked cabinets, separate from data. Participant PII will never be associated with data collected during the interview. PII for individuals not selected for interviews will be destroyed immediately. PII for selected participants will be destroyed per contract requirements.

9