NSVSP Cognitive Testing Memo

MEMORANDUM

MEMORANDUM TO: Jennifer Park

OMB

THROUGH: Melody Braswell

Justice Management Division

Jeri Mulrow

BJS Acting Director

FROM: Barbara Oudekerk

BJS Statistician

Lynn Langton

BJS Victimization Unit Chief

DATE: August 18, 2017

SUBJECT: Bureau of Justice Statistics (BJS) Request for OMB Clearance for cognitive testing of the questionnaire for National Survey of Victim Service Providers (NSVSP) under the OMB generic clearance agreement (OMB Number 1121-0339).

The Bureau of Justice Statistics (BJS) is requesting clearance for cognitive testing of the National Survey of Victim Service Providers (NSVSP), an instrument designed to gather detailed information about Victim Service Providers (VSPs) and the victims they serve, including the number and socio-demographic characteristics of victims served; the types of crimes victims experienced; details about types of services provided; and information on size of staff and staff salaries, benefits, qualifications, and trainings. This instrumentation effort is part of a program that BJS is developing to capture, on a routine basis, information about how VSPs respond to criminal victimizations. The results from the cognitive interviews will be used to inform the design of full-scale survey implementation with a stratified sample of victim service providers planned for 2018.

The National Survey of Victim Service Providers (NSVSP), funded by the Office for Victims of Crime (OVC) and sponsored by BJS, is a major component of a broader Victim Services Statistical Research Program (VSSRP) by BJS to understand the criminal justice system’s and its complementary agencies’ response to victims of crime. The victim services component of this effort consists of a several phases. We are currently finishing the first phase, the National Census of Victim Service Providers (NCVSP), which will result in a sampling frame for the NSVSP (under OMB #1121-0355). The NCVSP entailed developing the first national roster of victim service providers and collecting basic information on the landscape of the victim service field as a whole. The information collected, including organization type, number of victims served, and staffing sizes, will also allow for stratification of the sample for the subsequent NSVSP. NCVSP data collection ended July 31, 2017, and once the data are cleaned, BJS anticipates a sampling frame of approximately 16,500 VSPs that are actively serving victims of crime or abuse. In the next phase, BJS will use this validated sampling frame to conduct the follow-up NSVSP with a nationally representative sample of VSPs to collect more detailed information about the organizations serving victims, the types of victims receiving services, and the staff providing services.

Other ongoing components of BJS’ effort to understand the response to criminal victimization include the addition of new items to the National Crime Victimization Survey on use and non-use of victim services by victims of crime; a survey of victim services available to persons with and without disabilities within homeless shelters; a survey exploring the number of hospital-based victim service programs conducted in collaboration with the National Center for Health Statistics; and the addition of victim service questions to periodic establishment surveys conducted with subsets of the criminal justice system (e.g., law enforcement, courts).

Under this clearance, we plan to conduct cognitive interviews to identify issues with the wording or content of the NSVSP instrument and to estimate the burden associated with completion of the instrument. The current request for approval, under the BJS Generic Clearance (1121-0339), is for cognitive testing of the NSVSP with 30 service providers. The cognitive testing will require 1 burden hour per VSP for a total of 30 burden hours.

OMB approval for the full-scale fielding of the NSVSP will be sought at a later date.

Purpose of the Research

BJS, and its contractor, RAND, under a cooperative agreement, has planned the NSVSP as a way to collect data from VSPs, including detailed information about the organizational capacity of VSPs, as well as VSP current activities and needs. Under Title 42, United States Code, Section 3732, BJS is authorized to collect and analyze statistical information concerning the operation of the criminal justice system at the federal, state and local levels, and the NSVSP fits within that mission.

The NSVSP is a follow-up to the NCVSP. The NCVSP, which recently wrapped up, was limited in length (20 to 30 minutes) and collected only high-level information about VSPs because it needed to be administered to the full roster of VSPs across the nation. The NSVSP will be administered to a sample of VSPs. The NSVSP is intended to be longer (less than 1 hour), and will provide more in-depth information about the capacity of VSPs to serve victims. Specific goals of the first NSVSP are to gather detailed information about VSPs, the types of services they are able (and not able) to provide, staff working in these organizations, and the victims served by VSPs. Additional goals of the project are to standardize measures of victim services, enabling service providers to benchmark themselves against similar providers serving similar types of victims, and to use data from the NSVSP in connection with data from the National Crime Victimization Survey subnational estimates program and official police statistics to paint a more complete picture of the types of victims who do and do not receive police and victim services, the relationship between reporting to police and the receipt of services, and the magnitude of unmet need for victim services.

In addition to filling important data gaps in knowledge about VSPs, this information will have direct utility for OVC, by identifying gaps in services, documenting the structure and functioning of the service infrastructure, and providing information to assist with more effectively providing assistance to crime victims. This type of information is critical to developing an empirically-based approach to delivering victim services, and is consistent with OVC’s Vision 21 effort to advance the victim services field. Substantively, with no baseline measures about the current services provided or the organizations that provide them, there is no way to measure change in terms of the number and range of victims served by these organizations or the effectiveness of services provided. Victim service organizations lack a systematic way to benchmark their work against that of their peers. The lack of a clear understanding about how current victim service funding is being used limits their ability to work more effectively in providing assistance to crime victims, to seek future funding, or to identify underserved populations.

Because some VSPs included in the study may not have been asked to provide this detailed information before or in the manner requested, it is necessary to cognitively test the survey items, question ordering, and general content of the instrument. It is important to conduct cognitive testing to yield complete information (i.e., to know respondents can answer the questions based on available record); to ensure responses are uniform from one respondent to the next (e.g., how we ask about service categories are perceived similarly across different types of providers); and to increase the likelihood of high response rates by reducing confusion and burden.

The goals of this cognitive interview phase are to:

• Identify any items that cause excessive burden for the respondents;

• Assess comprehension issues associated with any of the questions, including whether respondents interpret questions consistently

• Sharpen the wording of questions so that respondents to the final survey receive the clearest instructions possible concerning how victims and services should be defined;

• Determine VSPs level of comfort and willingness to provide certain types of information (e.g., information on selected staff positions within the organization; gaps in being able to provide certain types of services);

• Roughly estimate the amount of burden needed to complete the instrument.

Design of NSVSP instrument

RAND and BJS began the development of the NSVSP instrument by examining items on existing victim service organization questionnaires, including OVC) and the Office for Violence against Women (OVW) quarterly financial reporting forms. Current reporting by victim services organizations, required by federal funding agencies, served as a starting point for the NSVSP. To the extent possible we modeled questions based on these reporting forms in order to reduce burden on VSPs. However, the NSVSP is designed for a different purpose than the financial reporting forms, and it is also designed to reach a wider range of VSPs.

The project team engaged in extensive discussions with a variety of stakeholders and practitioners in the victim services field to assist with the development of the instrument. Early and continuing discussions were held with OVC and OVW to better understand the needs of the field, the current state of reporting, and where the most important gaps in data could be found. The team also relied on the advice and guidance of experts and practitioners. Two day-long meetings were held with an Expert Panel (EP), comprised of 14 victim services providers and researchers considered experts in their field. During these meetings, the EP provided feedback on developing drafts of the victim service provider items. The EP meetings were instrumental in nominating content and making determinations about the structure of the survey to ensure its utility for policy-makers and the victim services field. Additionally, the project team sought expertise on an as-needed basis from individuals identified as key practitioners representing a range of different types of VSPs.

Some items on the NSVSP instrument are needed to validate information gathered by the NCVSP – for example organizational contact information, organization type, etc., to confirm eligibility for the NSVSP (e.g., a number of providers each year stop serving victims due to lack of funding or other reasons) and to check for changes that might have occurred since the NCVSP (e.g., whether the organization operates on a fiscal or calendar year could have changed). However, the majority of the NSVSP instrument will be new, including more detailed items that were not covered on NCVSP. Cognitive interviews are needed to examine the structure, language, and burden of the draft of the instrument, which will gather detailed information related to these topics. Two versions of the preliminary NSVSP instrument are available in Attachments B (for primary providers) and C (for secondary providers serving victims through dedicated staff or programs). The instruments vary only slightly in language, specifically secondary VSPs are asked to provide information only on the programs or staff that are directly serving victims (e.g., hospitals are not asked to talk about all staff in the hospital, only staff serving victims). The instrument covers the following categories of information:

1. Characteristics of victims served in the past calendar/fiscal year

2. Crime types for which victims sought services in the past calendar/fiscal year

3. Types of services provided to victims in the past calendar/fiscal year

4. Current gaps in services

5. Coordination between local VSPs and referrals to other VSPs

6. Staffing by position type

7. Staff characteristics, activities, training, and benefits

Selecting VSPs for Cognitive Testing

Because many different types of organizations have components that provide victim assistance in a formal capacity, the universe of VSPs is a diverse one. It includes entities such as police departments, YWCA chapters, tribal coalitions, child protective service agencies, family counseling centers, mental health service providers, district attorneys’ offices, and domestic violence shelters. Within the field, some organizations’ principle function is to serve crime victims and these are classified as primary VSPs (e.g, domestic violence shelters). Other organizations provide assistance or services to victims of crime through dedicated programs or staff, but they also serve other functions for both victims and non-victims; these are classified as secondary VSPs (e.g., police departments with a victim services program; YWCAs; hospitals with a sexual assault nurse examiner). The project sampling frame database includes organizations that fall into both of these classifications and a central focus of the cognitive testing is ensuring that respondents across a broad range of types can complete the instrument and provide comparable data.

For the cognitive interviews we will target approximately 15 primary VSPs and 15 secondary VSPs, ensuring that the secondary VSPs understand they are to separate victim-related staffing and services from other functions within in their organizations. To assist in recruiting efforts, we will rely on the aforementioned expert panel to provide recommendations for potential cognitive interview respondents that meet the desired VSP types and characteristics.

Cognitive Interview Procedures

Cognitive interviewing is proposed to begin in September 2017 and close in October 2017. Using the recommendations of the expert panel and other VSP contacts, potential participants will be contacted via phone to explain the purpose of the NSVSP and the cognitive interview testing (attachment A). Willing participants will be informed that they will receive an electronic word document of the instrument to review and that they will subsequently be interviewed about the survey. Participants will not need to complete the survey prior to the interview. Participants will be informed that they will be asked to provide dates that they would be able to participate in an approximately 1 hour interview. Respondents will not be paid for participation.

Once the interview is scheduled, the participant will receive an electronic copy of the survey, which will include a cover letter (attachments B and C). While it is anticipated that the pilot test and full administration will be primarily a web-based survey, for the cognitive testing purposes separate paper and pencil versions of the instrument have been developed for primary and secondary VSP providers.

During the cognitive interview, a research team staff member trained in cognitive interviewing techniques will review the NSVSP with respondents. In general, the cognitive interview questions will test the respondent’s grasp of the meaning of each question, the ability to answer each question, item burden, and appropriateness of data provided in response to questions requiring the gathering of information. Interviews will be conducted over the telephone. Interviews will be recorded with the respondents’ permission to ensure answers are accurately captured. The cognitive interviewer will follow a semi-structured script (see attachments D and E).

Language. The cognitive interviews will be conducted in English.

Burden Hours for Cognitive Testing

We request total 30 burden hours for 30 victim service organizations (1 hour per respondent, including the timing for reviewing instructions and completing the interview).

Analysis Plan

Following the interviews, the research team will listen to the interviews and summarize responses to each survey question and, to the extent possible, create codes based on broad categories of inquiry – e.g., understanding of question, understanding of possible responses, question burden, completion ability, etc. Responses will be categorized by VSP classifications to identify any differences based on organizational structure/type. While it is not possible to conduct any quantitative comparisons with the small number of respondents, the qualitative summaries and codes based on categories of inquiry will allow for identification of any patterns of difficulty across organizations ( including identification of high-burden questions), issues specific to types of organizations (e.g., differences in definitions, terminology, or understanding of questions), and any particularly unique insights into survey issues that discussions with stakeholders thus far have not yet identified. RAND will produce a memo that documents the results of the cognitive testing and summarizes the findings, including question-by-question issues that were identified. Recommended changes to the instrument will also be provided.

Informed Consent, Data Confidentiality and Data Security

The telephone invitations, introduction to the survey, and cognitive interview script read to respondents once on the telephone provide the elements of informed consent. The telephone invitation provides the purpose of the survey, the voluntary nature of the study, how the respondents was selected, and a number to call with questions about the study. The introduction to the survey and the script read to respondents on the telephone will repeat much of this information. The telephone invitation and introduction to the survey will both announce the estimated length of the interview in advance, allowing the participant an opportunity to decline if the burden would be unacceptable.

The data collected for this pretest are protected under the Bureau of Justice Statistics statutory protection. This protects the data from potential subpoena (42 USC 3789g). Access to RAND’s secure computer systems is password protected and data are protected by access privileges, which are assigned by the appropriate system administrator. All systems are backed up on a regular basis and are kept in a secure storage facility. To protect the identity of the respondents, no identifying information will be kept on the final data file. Identifying information includes the name of the sampled organization, address, and telephone number. The survey will not be collecting the name of any of the respondents. The identifying information will be deleted once the analysis file has been created and the link is no longer needed. In addition, the recorded conversations of the interviews will be erased upon completion of the final report. We estimate this to be one month after the interviews have ended. Once the questionnaire is revised and the summary report completed, all copies of the data will be destroyed.

4