NSVSO Cognitive Testing Memo

MEMORANDUM

MEMORANDUM TO: Shelly Wilkie Martinez

Official of Statistical and Science Policy

Office of Management and Budget

THROUGH: Lynn Murray

Clearance Officer

Justice Management Division

William J. Sabol, Ph.D.

Acting Director

Bureau of Justice Statistics

FROM: Lynn Langton

BJS Statistician

DATE: May 16, 2014

SUBJECT: BJS Request for OMB Clearance for cognitive testing of the questionnaire for National Survey of Victim Service Organizations (NSVSO) under the OMB generic clearance agreement (OMB Number 1121-0339).

The Bureau of Justice Statistics (BJS) is requesting clearance for cognitive testing of an instrument to capture data about the organizational attributes, staffing and services provided by victim serving organizations and entities that have victim services programs. This instrumentation effort is part of BJS’s National Survey of Victim Service Organizations (NSVSO), a program that BJS is developing to capture, on a routine basis, information about how victim serving organizations respond to criminal victimizations. The results from the cognitive interviews will be used to inform the design of a field test and full-scale frame validation study planned for summer of 2014 and the winter/spring of 2015, respectively.

The National Survey of Victim Service Organization (NSVSO), jointly funded by the Office for Victims of Crime (OVC) and BJS, is a major component of a broader effort by BJS to understand the criminal justice system’s and its complementary agencies’ response to victims of crime. The NSVSO, an establishment survey, will obtain information about the organization, operations, funding, staffing, and services provided by agencies that serve victims of crime. Other prospective components of this BJS effort to understand the response to criminal victimization include a supplement to the National Crime Victimization Survey (NCVS) on use and non-use of victim services by victims of crime; administrative records-based collections on services provided; periodic establishment surveys of subsets of victim serving organizations on special topics related to victim services; and in conjunction with NCVS subnational estimates and National Incident-Based Reporting Program data, some type of assessment of the services delivered in relation to the incidence of victimization.

These efforts aim to fill important gaps in knowledge about which victims receive services, about the capacity of VSOs to provide services, and about the need to expand or modify how services are delivered. For OVC, this type of information is critical to developing an empirically-based approach to delivering victim services, one that is consistent with OVC’s Vision 21 effort to transform the victim services.

The content of this instrument will be used in conjunction with an effort to validate the frame of VSOs. The frame is designed to capture organizations that actively provide services to crime victims as the primary function of the organization or through dedicated personnel or named programs. BJS has compiled and consolidated rosters of victim service organizations or programs within organizations from federal and state funding entities and victim service associations, as well as through web-scraping efforts to ensure a comprehensive enumeration. While the rosters contain information about the name and location of the organization, they often do not include the name of an executive director or contact person, up-to-date information about whether a listed entity still exists (as an organization or agency that operates a victim services program), or other information about the characteristics of the organization that would be useful for sample stratification. The instrument under consideration for cognitive testing will collect these types of information as well as information about services provided and staffing for all entities in the current list of agencies. The instrument will be administered to all service providers on the sampling frame and will generate basic, but currently unknown information on the landscape of the victim service field as a whole, in addition to the information necessary to stratify the sample for subsequent, more in depth data collection efforts. This approach is similar to that which has been successfully employed for years with the BJS Census of State and Local Law Enforcement Agencies and the Law Enforcement Management and Administrative Statistics collection.  

For this clearance, and prior to the administration of the initial NSVSO frame validation study, we plan to conduct cognitive interviews to identify issues with the wording or content of the instrument and to estimate the burden associated with completion of the instrument. The current request for approval, under the BJS Generic Clearance (1121-0339), is for two rounds of cognitive testing of the NSVSO census instrument with thirteen service providers in the first round and five in the second round. The cognitive testing will require 27 burden hours.

Once the instrument has been finalized through cognitive testing, a field test will be conducted to address issues related to the administration of the full frame validation survey, such as mode, response rates, number of follow-up contacts needed, and successful approaches for gaining respondent cooperation. OMB approval for the field test and subsequent frame validation will be sought at a later point.

Purpose of the Research

BJS, and its contractor, RAND, under a cooperative agreement, has planned the NSVSO as a way to collect data from victim service organizations (VSOs)¹ and begin to understand how they are organized and funded, who they serve and what services they provide. In addition to filling gaps in knowledge about VSOs this information will have direct utility for OVC, by identifying areas in need of service, demonstrating the delivery and effectiveness of the service infrastructure, and providing information to assist with more effectively providing assistance to crime victims. Currently, the victim service field lacks complete and reliable information about the range of services being provided to crime victims and the number of crime victims served, as well as information about the organizations and agencies that comprise the victim service field, including funding sources and the organizational resources required to provide certain services.

Substantively, with no baseline measures about the current services provided or the organizations that provide them, there is no way to measure progress in terms of the number and range of victims served by these organizations or the effectiveness of services provided. Victim service organizations lack a systematic way to benchmark their work against that of their peers. The lack of a clear understanding about how current victim service funding is being used limits the ability of the field to work more effectively in providing assistance to crime victims, to seek future funding, or to identify underserved populations.

OVC and BJS have funded the development of the NSVSO as a method of filling these existing gaps. Additional goals of the project are to standardize measures of victim services, enabling service providers to benchmark themselves against similar providers serving similar types of victims, and to use data from the NSVSO in connection with data from the National Crime Victimization Survey subnational estimates program and official police statistics to paint a more complete picture of the types of victims who do and do not receive police and victim services, the relationship between reporting to police and the receipt of services, and the magnitude of unmet need for victim services. Under Title 42, United States Code, Section 3732, BJS is directed to collect and analyze statistical information concerning the operation of the criminal justice system at the federal, state and local levels, and the NSVSO fits within that mission.

The goals of the initial iteration of the NSVSO are to develop an understanding of the broad range of organizations that provide victim services as their primary function or through specific programs or personnel, including how they are structured, the types of services they offer, and the types of crime victims they serve, and most importantly to enhance the quality of the sampling frame. While there are many directories in place, and many lists of organizations serving specific types of victims, they are not all inclusive and many are not routinely updated. Some of the organizations on frame may no longer be in business, may no longer provide services to victims, or may only assist victims during the regular course of providing other types of services to both victims and non-victims (referred to as incidental VSOs). Therefore, the first full-scale NSVSO effort will be a validation study of the over 21,000 victim service providers in the current database to answer basic questions about the organization and provision of services for frame validation and sample stratification purposes. The current database of providers is being compiled from de-duplicated lists of federal victim service funding recipients, and affiliation lists from national victim service provider membership groups, as well as a web-based state canvassing effort. The database is still being developed so additional information about the frame and the organizations included in the frame will be provided in the request for approval for the pilot testing.

Because the validation study will collect data that VSOs may not have been asked to provide before or provide in the manner requested, it is necessary to cognitively test the survey items, question ordering, and general content of the instrument. Without such testing, the survey could yield a large quantity of incomplete information (because respondents will not be able to furnish it); responses that are not uniform from one respondent to the next (e.g., who is defined as a domestic violence victim); or high rates of nonresponse due to respondent confusion and burden.

The goals of this cognitive interview phase are to:

• Identify any items that cause excessive burden for the respondents;

• Establish the total time needed to complete the frame validation instrument;

• Assess comprehension issues associated with any of the questions, including whether respondents interpret questions consistently

• Sharpen the wording of questions so that respondents to the final survey receive the clearest instructions possible concerning how victims and services should be defined;

• Gain a better understanding of the extent to which organizations that serve both victims and non-victims can separate these for reporting purposes.

Design of NSVSO instrument

RAND and BJS began the development of the NSVSO frame validation instrument by examining items on existing victim service organization questionnaires, including Office for Victims of Crime (OVC) and Office for Violence against Women (OVW) quarterly financial reporting forms. Current reporting by victim services organizations, such as that required by federal funding agencies, served as a jumping off point for the NSVSO. To the extent possible we modeled questions based on these reporting forms in order to reduce burden on VSOs. However, the NSVSO is designed for a different purpose than the financial reporting forms, and it is also designed to reach a wider range of VSOs.

For these reasons, the project team also engaged in extensive discussions with a variety of stakeholders and practitioners in the victim services field to assist with the development of the instrument. Early and continuing discussions were held with OVC and OVW to better understand the needs of the field, the current state of reporting, and where the most important gaps in data could be found. The team also relied on the advice and guidance of experts and practitioners. Two day-long meetings were held with an Expert Panel (EP), comprised of 14 victim services providers and researchers considered experts in their field. During these meetings, the EP provided feedback on developing drafts of the NSVSO questionnaire. The EP meetings were instrumental in nominating content and making determinations about the structure of the survey to ensure its utility for policy-makers and the victim services field. Additionally, the project team sought expertise on an as-needed basis from individuals identified as key practitioners representing a range of different types of VSOs.

The resulting NSVSO frame validation instrument is designed to capture basic, essential information about service providers that will be used to describe and map the field as a whole in the first BJS report on victim service organizations. For the subsequent, more detailed survey efforts that are anticipated down the road, the information will be used to eliminate out-of-scope organizations from the frame. Additionally, data on organization type, type of victims served, and organization size will be used to stratify the sample for later surveys.

The cognitive interviews will address the structure and language of the current draft of the instrument, which will gather information related to these topics:

1. Screening questions

2. Organization/agency type

3. Types of services provided during the past 12 months

4. Number of basic hotline calls in the past 12 months

5. Number of unique victims who received services in the past 12 months

6. Types of crime victims served in the past 12 months

7. Number of paid staff and volunteers by function

8. Funding by source in the past 12 months

9. Use of case management systems

Selecting VSOs for Cognitive Testing

Because many different types of organizations have components that provide victim assistance in a formal capacity, the theoretical universe of federal and state funded VSOs is a diverse one. It includes entities such as police departments, YWCA chapters, tribal coalitions, child protective service agencies, family counseling centers, mental health service providers, district attorneys’ offices, and domestic violence shelters. Within the field, some organizations’ principle function is to serve crime victims and these are classified as primary VSOs (e.g., domestic violence shelters). Other organizations provide assistance or services to victims of crime through dedicated programs or staff, but they primarily serve other functions for both victims and non-victims and are classified as secondary VSOs (e.g., YWCAs, hospitals with a sexual assault nurse examiner). Still others serve victims as part of their regular services but have no designated programs or staff and are classified as incidental VSOs (e.g., a locksmith that changes domestic violence victim locks for free). The current sampling frame database includes organizations that fall into all three classifications and a central focus of the cognitive testing is ensuring that respondents of all three types can complete the instrument and provide comparable data.

Because the instrument was developed in collaboration with a panel of expert practitioners and because a larger pilot study will also be conducted, the number of cognitive interviews will be just large enough to ensure that the interview can be administered to a couple of respondents in each of the three key classification groups. The cognitive interviews will be conducted in two rounds, with 13 VSOs in the first round and 5 in the second round, used to address changes to the instrument after the first round. Since for some questions on the instrument, the instructions and wording will vary based upon the organizational function of the respondent (see attached survey versions), within the first round we will target approximately five primary VSOs, six secondary VSOs, and two incidental VSOs. The focus is largely on the secondary VSOs to better understand whether these organizations are able to separate victim-related staff, funding, and services from other functions within in their organizations. This challenge is less of a concern with the primary VSOs, and the incidental VSOs will be included in the validation study because some are on the frame and may receive federal and state funding, but these providers would likely be screened out of subsequent data collection efforts since it would not make sense to ask them in-depth questions about staffing and services.

Since the current database primarily includes only the organization’s name and contact information, we will rely on the aforementioned expert panel to provide recommendations for potential cognitive interview respondents that meet the desired VSO types and characteristics. We will specifically target several VSO’s based in the Washington DC area to allow for in-person interviews conducted by several research team members who are based in Washington DC.

Cognitive Interview Procedures

Cognitive interviewing is proposed to begin in June 2014 and close in July 2014. The cognitive interviews will be completed in sufficient time to inform the OMB Submission Package for the pilot testing, as well as the main NSVSO data collection. Using the recommendations of the expert panel, potential participants will be contacted via phone to explain the purpose of the NSVSO and the cognitive interview phase of testing. Willing participants will be informed that they will receive an electronic word document of the instrument to complete on their own to the best of their ability and that they will subsequently be interviewed within the following days about their experiences completing the survey. Participants will be informed that they should allot approximately 30 minutes for completing the survey prior to the interview, and will be asked to provide dates that they would be able to participate in a 60 minute interview following their completion of the survey. Respondents will not be paid for participation.

Once the interview is scheduled, the participant will receive an electronic copy of the survey, which will include a cover letter and instructions for completing the survey (attached). While it is anticipated that the pilot test and full administration will be primarily a web-based survey, for the cognitive testing purposes separate paper and pencil version of the instrument have been developed for each of the three VSO classifications (primary, secondary, and incidental provider).

During the cognitive interview, a research team staff member trained in cognitive interviewing techniques will conduct the cognitive interview. Interviews will be conducted primarily over the telephone but several DC-area VSOs will be included in the test to allow for some in person interviewing. Interviews will be recorded with the respondents’ permission to ensure answers are accurately captured. The cognitive interviewer will follow a specific script which will vary slightly depending on the organizational function of the VSO (see attached scripts). In general, the cognitive interview questions will test the respondent’s grasp of the meaning of each question, the ability to answer each question and the amount of burden, and appropriateness of data provided in response to questions requiring the gathering of information. Questions at the end of the survey will be designed to ascertain the costs and burden of the survey.

Language. The cognitive interviews will be conducted in English.

Burden Hours for Cognitive Testing

We request total 27 burden hours for 18 victim service organizations (ninety minutes per respondent).

Analysis Plan

The cognitive interviews will be conducted in two rounds of 13 VSOs in the first round and 5 in the second round. Following the first round of interviews, the research team will listen to the interviews and summarize responses to each survey question and, to the extent possible, create codes based on broad categories of inquiry – e.g., understanding of question, understanding of possible responses, question burden, completion ability, etc. Responses will be categorized by the three VSO classifications to identify any differences based on organizational structure or challenges with the instrument specific to the organizational structure. While it is not possible to conduct any quantitative comparisons with the small number of respondents, the qualitative summaries and codes based on categories of inquiry will allow for identification of any patterns of difficulty across organizations ( including identification of high-burden questions), issues specific to types of organizations (e.g., differences in definitions, terminology, or understanding of questions), and/or any particularly unique insights into survey issues that discussions with stakeholders thus far have not yet identified. The summary of findings will be presented in a joint meeting with BJS and the research team in which the group will determine any changes that should be made the instrument. Following survey instrument edits, we will conduct a second round of interviews with five new VSO respondents, and will follow the same summary and analysis approach described above. Upon completion of both rounds of interviews, RAND will produce a report that documents the results of the cognitive testing and summarizes the findings, including question-by-question issues that were identified. Recommended changes to the instrument will also be provided.

Informed Consent, Data Confidentiality and Data Security

The telephone invitations, the introduction to the survey and the cognitive interview script read to respondents once on the telephone provide the elements of informed consent. The telephone invitation provides the purpose of the survey, the voluntary nature of the study, how the respondents was selected, and a number to call with questions about the study. The introduction to the survey and the script read to respondents on the telephone will repeat much of this information. The telephone invitation and introduction to the survey will both announce the estimated length of the interview in advance, allowing the participant an opportunity to decline if the burden would be unacceptable.

The data collected for this pretest are protected under BJS’s statutory protection. This protects the data from potential subpoena (42 USC 3789g). Access to RAND’s secure computer systems is password protected and data are protected by access privileges, which are assigned by the appropriate system administrator. All systems are backed up on a regular basis and are kept in a secure storage facility. To protect the identity of the respondents, no identifying information will be kept on the final data file. Identifying information includes the name of the sampled organization, address, and telephone number. The survey will not be collecting the name of any of the respondents. The identifying information will be deleted once the analysis file has been created and the link is no longer needed. In addition, the recorded conversations of the interviews will be erased upon completion of the final report. We estimate this to be one month after the pretest has ended. Once the questionnaire is revised and the summary report completed, all copies of the data will be destroyed.

With respect to personnel, all members of the research team are required to sign a pledge of confidentiality. This pledge requires employees to maintain confidentiality of project data and to follow the above procedures when handling confidential information.

1 A “Victim Service Organization” (VSO) is any organization which provides services or assistance to victims of crime

4