2023 NCVS Redesign Pilot Test Memo

U.S. Department of Justice

Office of Justice Programs

Bureau of Justice Statistics

Washington, D.C. 20531

MEMORANDUM

TO: Robert Sivinski

Office of Statistical and Science Policy

Office of Management and Budget

THROUGH: Alexis R. Piquero, PhD

Director

Bureau of Justice Statistics

Kevin M. Scott, PhD

Principal Deputy Director

Bureau of Justice Statistics

FROM: Heather Brotsos and Jennifer L. Truman PhD

Victimization Statistics Unit

Bureau of Justice Statistics

DATE: April 11, 2023

SUBJECT: BJS Request OMB Generic Clearance for a Pilot Test under the Generic Clearance for Cognitive, Pilot and Field Studies for Bureau of Justice Statistics Data Collection Activities, OMB Number 1121-0339

The Bureau of Justice Statistics (BJS) requests clearance (under OMB Number 1121-0339) for a pilot test of the new NCVS instrument. The objective of this pilot is an operational test to assess the new instrument and protocols and test all systems and operational procedures in the U.S. Census Bureau’s data collection environment.

This memo first provides background on the NCVS Instrument Redesign. Next is a description of the pilot test objectives and proposed testing procedures, followed by a description of language, burden hours, cost, reporting, protection of human subjects, informed consent, and data confidentiality and security.

Background on the NCVS Instrument Redesign

The National Crime Victimization Survey (NCVS) is the nation’s primary source of information on criminal victimization. It provides information on nonfatal personal crimes and household property crimes, both reported and not reported to police. The NCVS instrument was last redesigned more than three decades ago, in 1992. Much has changed in the interim, both in the level of public acceptance of surveys and in the nature of crime. BJS, in consultation with Westat under a cooperative agreement (Award 2019-R2-CX-K001 National Crime Victimization Survey (NCVS) Instrument Redesign and Testing Project), initiated a multiyear effort to improve the efficiency, reliability, and utility of the NCVS. The effort had three main goals: modernize the organization and content of the NCVS instrument, increase the quality of information collected and efficiency of the instrument flow, and improve the measurement and classification of crime. Before phasing the new instrument into production, BJS seeks clearance to conduct a pilot test in coordination with the U.S. Census Bureau.

Prior generic clearances for this project (OMB Numbers 1121-0325, 1121-0339) included cognitive testing, usability testing, and a small-scale pilot test to explore the best approach for the redesigned instrument. This qualitative testing culminated in the design for a full-scale national Field Test in 2019–2020 (OMB Number 1121-0368). That national Field Test involved administering the survey to a representative sample of persons age 12 or older, testing aspects of the design such as mode, victimization-screener approaches, and measured response rates and administration times. The Field Test included three treatment groups: (1) an interviewer-administered version of the current NCVS instrument, (2) an interviewer-administered version of the redesigned NCVS instrument, and (3) a self-administered version of the redesigned NCVS instrument, using a web-based platform to complete the survey. Based on findings from this Field Test, BJS will transition to the redesigned survey instrument.¹ This new interviewer-administered NCVS instrument maintains a two-stage measurement approach in screening and classifying criminal victimization, improves crime screening and flow through the crime incident report, collects expanded information on victimization incidents, and introduces questions on police performance and community safety.

This clearance request is for a pilot test to assess the new instrument in the U.S. Census Bureau production environment prior to implementation. This operational pilot test has two main goals: (1) assess the new survey instrument and protocols in the U.S. Census Bureau data collection environment and (2) test all systems and operational procedures within the U.S. Census Bureau’s data collection environment. This will ensure that the new instrument and all procedures are working before the new instrument is put into production in 2024. It will also inform development of field protocols and interviewer training needs for the split-sample administration in 2024 and full-scale implementation of the new NCVS instrument in 2025. BJS will request full clearances for the split-sample administration and full-scale implementation through separate OMB clearance packages.

Testing Procedures

Research Objectives

BJS, in coordination with the U.S. Census Bureau, will conduct an operational pilot to test the new NCVS instrument in the field prior to implementation. The first objective is to test the new instrument and the protocols in the U.S. Census Bureau data collection environment. The second objective is to ensure that all systems and operational procedures are working within the U.S. Census Bureau’s data collection environment before the new instrument is introduced into production beginning in 2024.

The U.S. Census Bureau will monitor quality indicators and track issues in the computer-assisted personal interviewing instrument and case management systems during pilot data collection. These may include paradata indicators, timing, nonresponse rates, item missingness rates, and other data points that can be tracked during pilot administration. Pilot test data from these systems will be used only to identify and diagnose operational challenges with field protocols and interview procedures. During this period, BJS and the U.S. Census Bureau will assess whether any changes to field procedures or training must be implemented before moving to the split-sample administration.

Sample Design

Approximately 3,500 housing units² will be in sample for the pilot study; 1,575 sample households from the current production sample³ and 1,925 from the reserve sample⁴ that Census selects each year. Once housing units are selected for the pilot study, they will continue to be administered the new instrument at each remaining TIS until they phase out or are dropped from the sample.

The U.S. Census Bureau’s sample design and field staff worked together to identify Primary Sampling Units (PSUs) for the pilot. A total of 150 PSUs out of the 542 PSUs in the current NCVS sample design will be included in the pilot. Key selection criteria included the number of available field staff, expected field staff turnover, hiring forecasts, and current field staff workloads. Additionally, the team looked at internal NCVS response data across three years (2018-2020) to examine response rates and the distribution of reported incidents by PSU. As a result, selected PSUs are not representative of the United States. The 3,500 sample households will be distributed to the PSUs proportionally to the size of the PSUs included in the pilot.

No victimization estimates will be produced from the pilot, as it is not designed to test for differences between the current instrument and the redesigned instrument, nor is it designed to produce regional or national estimates. Any differences between the instruments will be examined using the data collected in 2024.

Data Collection

Each housing unit selected for the pilot will be asked to complete or update a household roster. All persons age 12 or older in the household will be asked to complete a person interview. The Control Card is used to complete a household roster with names and other demographic information of the household members (See Attachment 1: NCVS Redesign Instrument).

The Victimization Screener asks all respondents age 12 or older in the household whether the respondent experienced a personal crime victimization during the prior six months. It collects basic information needed to determine whether the respondent experienced a crime victimization (rape or other sexual assault, robbery, aggravated or simple assault, personal larceny, burglary, motor vehicle theft, other household theft, or vandalism). In each household, one respondent is designated as the head of the household and that head of the household reports about all household property crimes on behalf of the entire household.

When a respondent reports an eligible victimization, the Crime Incident Report (CIR) is administered to collect detailed information about each reported crime incident. It collects information about the offender (e.g. sex, race, Hispanic origin, age, and victim-offender relationship), characteristics of the crime (including time and place of occurrence, use of weapons, nature of injury, and economic consequences), whether the crime was reported to police, reasons the crime was or was not reported, and victim experiences with the criminal justice system.

The new NCVS instrument also includes two new modules that are administered to all respondents: the Police Performance Module measures the types of contact with police and opinions about police performance; and the Community Safety Module measures indicators of public perceptions of community issues related to crime and neighborhood safety. During a given interview month only one of these new modules is administered. The Community Measures module will be administered during the pilot months.

Based on the large-scale national Field Test the average burden of the household roster instrument is estimated to be 10 minutes and the average burden of the person-level instrument, which includes the Police Performance or Community Safety modules, Victimization Screener, CIR, and Interview Info (Debriefing) items, is estimated to be 18 minutes.

Timeline

Milestone	Start Date	End Date
Obtain OMB generic clearance	April 24, 2023	May 15, 2023
Pilot test data collection	July 1, 2023	September 30, 2023
Evaluate pilot test data and complete any recommended changes for full-scale implementation	July 1, 2023	November 30, 2023

Language

All pilot interviews will be conducted in English. The new NCVS instrument will be translated into Spanish in preparation for the full-scale production but will not be ready for use in the pilot. Census interviewers may also conduct the interview in other languages in accordance with the field representative manual for the full-scale production. Participant materials are available in several languages and resources are posted on the BJS participant website at https://bjs.ojp.gov/ncvs-participant

Burden Hours for Testing

Activity	Maximum Number of Respondents	Average Administration Time (hours)	Burden (hours)
NCVS Household Roster	3,500	0.167	585
NCVS Person Interview	4,400	0.367	2,899
TOTAL	7,900		3,484

Cost to the Federal Government

The total cost of conducting the data collection is estimated to be $1,580,000.

Reporting

The U.S. Census Bureau will monitor quality indicators and track issues in the computer-assisted personal interviewing instrument and case management systems during pilot data collection. These may include paradata indicators, timing, nonresponse rates, item missingness rates, and other data points that can be tracked during pilot administration. Upon completion of the pilot test, the U.S. Census Bureau will provide BJS with updates on the findings from this monitoring. This will include any recommended changes to data collection protocol or interview procedures. Pilot test data from these systems will be used to identify and diagnose operational challenges with field protocols and interview procedures. Estimates generated from the pilot will not be released.

Protection of Human Subjects

There is some risk of emotional distress for the respondents given the sensitive nature of the topics, particularly since the questions are of a personal nature; however, appropriate safeguards are in place. All interviewers will receive training in a distress protocol reviewed and approved by the U.S. Census Bureau’s Legal and Policy offices.

Interviewers will be trained to recognize if respondents are becoming emotionally upset (See Attachment 2: Distress Protocol). They will also be trained on ways to respond if the respondent becomes upset to ensure the safety of both the interviewer and respondent. Any respondent who appears to be in distress will be asked if they wish to pause or stop the interview. Interviewers will have a list of resources that they can provide or read to respondents. These resources are also included in the NCVS Fact Sheet (See Attachment 3: NCVS-110, NCVS Fact Sheet).

Informed Consent

The advance letter from the Director of the U.S. Census Bureau (See Attachment 4: NCVS-572(L) Advance Letter) and NCVS Fact Sheet (See Attachment 3: NCVS-110, NCVS Fact Sheet) provide information about the purpose of the NCVS, the voluntary nature of the study, how the respondent was selected, and information about whom to contact with questions about the study. These are the same materials used for the production NCVS.

Use of Information Technology to Reduce Burden

Respondents to the NCVS are individuals living in households. The U.S. Census Bureau collects the data using in-person and telephone interviews, using computer-assisted personal interviewing (CAPI) to reduce cost, improve data quality, and streamline data processing.

Data Confidentiality and Security

The Bureau of Justice Statistics (BJS) is authorized to conduct this data collection under 34 U.S.C. § 10132. BJS will protect and maintain the confidentiality of personally identifiable information (PII) to the fullest extent under federal law. BJS, its employees, and its contractors will only use the information provided for statistical or research purposes pursuant to 34 U.S.C. § 10134, and will not disclose respondent information in identifiable form to anyone outside of the BJS project team. All PII collected under BJS’s authority is protected under the confidentiality provisions of 34 U.S.C. § 10231. Any person who violates these provisions may be punished by a fine up to $10,000, in addition to any other penalties imposed by law. Further, per the Cybersecurity Enhancement Act of 2015 (6 U.S.C. § 151), federal information systems are protected from malicious activities through cybersecurity screening of transmitted data.

All NCVS information about individuals or households is confidential by law under Title 34, U.S.C., § 10231 and 10134 and Title 13, U.S.C., § 9. By law, the U.S. Census Bureau is not permitted to publicly release survey responses in a way that could identify survey respondents or their households. U.S. Census Bureau employees who may see the survey responses are sworn to preserve this confidentiality. Even BJS, as the sponsor of the survey, is not authorized to see or handle the data in its raw form. All unique and identifying information is scrambled or suppressed before it is provided to BJS to analyze. Data are maintained in secure environments and in restricted access locations within the U.S. Census Bureau. All data provided to BJS must meet the confidentiality requirements set forth by the Disclosure Review Board at the U.S. Census Bureau.

In a letter from the Director of the U.S. Census Bureau (See Attachment 4: NCVS-572(L) Advance Letter), sent to all households in the survey, respondents are informed of these laws and assured that it requires the U.S. Census Bureau to keep all information provided by the respondent confidential. The letter also informs respondents that this is a voluntary survey. Furthermore, in addition to the legal authority and voluntary nature of the survey, the letter informs respondents of the public reporting burden for this collection of information, the principal purposes for collecting the information, and the various uses for the data after it is collected which satisfies the requirements of the Privacy Act of 1974.

Contact Information

Questions regarding any aspect of this project can be directed to:

Jennifer L. Truman, PhD

Statistician

Bureau of Justice Statistics

810 Seventh St, NW

Washington, DC 20531

Phone: 202-598-1931

Email: [email protected]

List of Attachments

Attachment 1 – NCVS Redesign Instrument

Attachment 2 – NCVS Distress Protocol

Attachment 3 – NCVS Fact Sheet

Attachment 4 – NCVS Advance Letter

1 For more information, see the NCVS Instrument Redesign page at https: https://bjs.ojp.gov/programs/ncvs/instrument-redesign.

2 Group quarters are out-of-scope to streamline pilot administration.

3 The U.S. Census Bureau interviews NCVS households every six months for a total of seven interviews over a 3-year period. The time-in-sample (TIS) identifies which interview number, out of the seven, the household is receiving. The pilot sample includes households across all TIS groups.

4 The NCVS sample selection process is designed to select twice as much sample as needed to produce the NCVS estimates. Half of the resulting sample is set aside as reserve sample. This reserve sample is not used for interviews unless special situations, such as separate questionnaire experiments, require additional sample.