UoF Supporting Statement Part B 122316_Final

B. Statistical Methods

1. Universe and Respondent Selection

Respondents to the National Use-of-Force Data Collection include law enforcement agencies that employ sworn officers that meet the definition as set forth by the Law Enforcement Officers Killed and Assaulted (LEOKA) Program. The LEOKA definition and additional criteria are as follows:

All local, county, state, tribal, and federal law enforcement officers (such as municipal officers, county police officers, constables, state police, highway patrol officers, sheriffs, their deputies, federal law enforcement officers, marshals, special agents, etc.) who are sworn by their respective government authorities to uphold the law and to safeguard the rights, lives, and property of American citizens. They must have full arrest powers and be members of a public governmental law enforcement agency, paid from government funds set aside specifically for payment to sworn police law enforcement organized for the purposes of keeping order and for preventing and detecting crimes, and apprehending those responsible.

General Criteria

The data collected by the Federal Bureau of Investigation’s (FBI’s) LEOKA Program pertain to felonious deaths, accidental deaths, and assaults of duly sworn city, university and college, county, state, tribal, and federal law enforcement officers who, at the time of the incident, met the following criteria. These law enforcement officers:

• Wore/carried a badge (ordinarily),

• Carried a firearm (ordinarily),

• Were duly sworn and had full arrest powers,

• Were members of a public governmental law enforcement agencies and were paid from government funds set aside specifically for payment to sworn law enforcement, and

• Were acting in an official capacity, whether on or off duty, at the time of the incident

Exception to the above-listed criteria

Beginning January 1, 2015, the LEOKA Program effected an exception to its collection criteria to include the data of individuals who are killed or assaulted while serving as a law enforcement officer at the request of a law enforcement agency whose officers meet the current collection criteria. (Special circumstances are reviewed by LEOKA staff on a case-by-case basis to determine inclusion.)

Addition to the LEOKA Program’s Data Collection

Effective March 23, 2016, the LEOKA Program expanded its collection criteria to include the data of military and civilian police and law enforcement officers of the Department of Defense (DoD), while performing a law enforcement function or duty, who are not in a combat or deployed (sent outside of the U.S. for a specific military support role mission) status. This includes DoD police and law enforcement officers who perform policing and criminal investigative functions while stationed (not deployed) on overseas bases, just as if they were based in the U.S.

Exclusions from the LEOKA Program’s Data Collection

Examples of job positions not typically included in the LEOKA Program’s statistics (unless they meet the above exception) follow:

• Corrections or correctional officers

• Bailiffs

• Parole or probation officers

• Federal judges

• U.S. and Assistant U.S. Attorneys

• Bureau of Prison officers

The number of local, state, and tribal law enforcement agencies in the Uniform Crime Reporting (UCR) Program’s records that meet that criteria are 18,444. In addition, there are potentially up to 114 federal law enforcement agencies that may also meet the criteria for submission of use-of-force incidents to the National Use-of-Force Data Collection. The first six months of data collection will focus upon a recruited set of law enforcement agencies. Potential participants are those agencies with at least 750 sworn law enforcement officers on their workforce, the four Department of Justice (DOJ) agencies, and additional recruiting from two to five states willing to participate. The goals of the pilot center on data quality and data completeness. More information on the pilot can be found in the response to Supporting Statement Part B, Question 4.

Agency Type	Pilot (minimum numbers)	Full Collection
Municipal	52	11,708
County	30	3,031
Colleges/Universities	0	788
Other Agencies	0	500
Other State Agencies	3	1,075
State Police	6	1,134
Tribal	0	207
Federal	4	114
Total	95	18,558

As the National Use-of-Force Data Collection is intended to collect information on any use of force by law enforcement in the U.S. or on U.S. territory that meets one of the three criteria (death of a person, serious bodily injury of a person, or firearm discharge at or in the direction of a person), sampling methodologies are not used. Instead, the FBI UCR Program relies upon the enumeration of these incidents in total to make statements about the relative frequency and characteristics of the use of force by law enforcement in the U.S. However, the voluntary nature of the UCR Program results in some agencies reporting incomplete information and others not participating in the data collection at all.

Statistical information on law enforcement use of force reported under the definition of justifiable homicide historically has had a low response rate. For the years 2012 to 2014, the total number of justifiable homicides by law enforcement reported by UCR contributors was 1,341, or 447 on average per year. This number represents only about 40 percent of expected reports of justifiable homicide based upon the findings of the Bureau of Justice Statistics (BJS). In its 2015 report¹, the BJS found that approximately 1,200 incidents of justifiable homicide per year are reported through the media and corroborated by law enforcement or the medical examiner. The FBI intends to vigorously address the response rate problem to raise it to a minimum of 80 percent once the data collection is established.

2. Procedures for Collecting Information

Information on law enforcement use of force will be collected initially by law enforcement agencies that employ law enforcement officers that meet the same definition and criteria as the LEOKA Program (see response to Supporting Statement Part B, Question 1). Agencies will record information on the use-of-force incident for their own purposes in case files that may or may not be housed in automated systems. This information will be translated or recoded into standardized answers that correspond to the 41 questions asked in the National Use-of-Force Data Collection.

The process of translating agency information into standardized responses for a UCR data collection more closely aligns with the coding process associated with content analysis, rather than traditional survey design. In the reporting of information on a use of force by law enforcement, the responses will usually be provided by a supervisor of a unit charged with investigating the use of force or one of the staff in such a unit. Rarely would the questionnaire be completed by the individual officer(s) involved in the incident. The FBI will provide both user guides and “just in time” information to guide individuals in the process of responding to questions in a standardized fashion.

Agencies will be encouraged to begin the process of completing the questions regarding a use-of-force incident as soon as possible. All work can be saved within the system and retrieved at a later time for completion. Once an agency has completed the questions related to an incident, a designated individual within each agency will indicate that the information is ready for the next stage in the workflow. At this point, states can directly manage the collection of use-of-force information at the state level, much like the rest of UCR data collections. Alternatively, states can allow for their agencies to report their use-of-force data directly to the FBI. Regardless of whether it would be the state UCR Program or the FBI that receives the data, all incidents will be subject to review for logical inconsistencies by staff of the FBI or state UCR Program. If questions arise regarding the information provided, the original agency will be asked to resolve data quality issues.

In addition to use-of-force incident information, agencies will be able to indicate on a monthly basis that they did not have any use-of-force incidents that are within scope for the data collection. These “zero report” submissions will follow through the same general workflow as the use-of-force incident information.

Some agencies and states have automated systems in place to capture information on law enforcement use of force or have plans in the near future to build those systems. The FBI will be building the capability of “ingesting” a file submission from these systems by a Secure File Transfer Protocol within the first quarter of 2017. At time yet to be scheduled, the FBI will also provide for a means foragencies and states to submit data through Extensible Markup Language web services. The FBI will provide technical specifications to agencies and states wishing to provide data as a bulk file in addition to the instructions.

3. Methods to Maximize Response

Addressing Nonresponse

Analysis of Patterns of Missing Values

To determine if there is a need for a nonresponse bias study, the FBI will be analyzing overall reporting patterns from agencies within the pilot states and agencies for the first six months of data collection and will continue to monitor the response rates once the data collection is considered operational. The analysis will look for patterns of unit missing data (i.e., nonparticipating agencies), as well as item missing data (e.g., not reporting within-scope incidents of firearm discharges) by agency type that fall below a threshold of 80 percent. An additional dimension is that the data collection is also structured in such a way that agencies can leave the majority of data elements as “pending further investigation.” This data value is provided to agencies to accommodate both legal and contractual obligations regarding the release of information. Agencies are frequently bound by either local statute, local policy, or collective bargaining agreements in terms of what information can be released and when it may be released. After the first full year of reporting, the FBI UCR Program will reassess nonresponse patterns and work with the BJS and external experts to provide a methodology for arriving at national estimates.

Technical Response to Address Agency Nonresponse

The recommendation of the Criminal Justice Information Services (CJIS) Advisory Policy Board (APB) to create an FBI-sponsored and FBI-maintained tool is in direct response to issues that have continually provided impediments to the adoption of modifications to the UCR Program. Traditionally, the UCR Program has provided to both agencies and state UCR programs a set of technical specifications for a data submission to any part of the UCR data collections. However, that method is under the presumption that agencies and state UCR programs assume the responsibility to build and maintain a data system for collecting the data. In the case of the use of force data collection, the FBI will sponsor and maintain a data collection tool that will be accessible through the Law Enforcement Enterprise Portal (LEEP). This portal capability will enable agencies to contribute their data directly to the FBI or allow state UCR Program Managers to use the tool to manage the data collection for their states. The tool has been constructed in such a fashion that state UCR Program Managers have enhanced privileges to monitor reporting status and other data quality elements.

The LEEP data collection tool assumes that agencies have consistent connectivity to the Internet and also have maintained an active account on the LEEP. However, it is unlikely that all agencies will have proactively enrolled in the LEEP. The FBI is currently working on a plan for enrolling agencies and state UCR programs into the LEEP. Initially, the FBI will proactively recruit agencies for participation in the pilot study. After the FBI has secured agreement by an agency to participate in the pilot study, the FBI will work with a point of contact within each pilot agency to work on connectivity and LEEP and National Use-of-Force Data Collection Portal access issues. This effort will allow for the FBI to determine the most common problems related to access and develop mitigation against these problems.

Confirming a Report of Zero versus Nonresponse

The final aspect of the needs assessment and technical review addresses the current limitations of the justifiable homicide that prevent the FBI from quantifying the level of reporting by UCR contributing agencies. The current systems do not provide a path for agencies to report that they record no justifiable homicides as part of the UCR data collections. The data collection on use of force will be looking for agencies to positively affirm on a monthly basis that they did not have any use of force the resulted in a fatality, a serious bodily injury to a person, or firearm discharges at or in the direction of a person.

In addition, the BJS will continue to work with the FBI in researching open-source data on law enforcement fatalities to assist in the determination of nonresponse by agencies.

4. Testing of Procedures

Formal Testing

The testing process in the development of the National Use-of-Force Data Collection uses a multi-stage approach. The initial process included a needs assessment and conceptualization of key estimates and data elements. This process involved assessing agency record systems, working through the FBI’s APB process, holding focus groups with key law enforcement components, and developing key data elements through the FBI’s Use of Force Task Force. In addition, the FBI will consult with other agencies, such as the BJS and DOJ, on collections related to force. Given the significant overlap in data elements, the DOJ’s Death in Custody Reporting Act data can provide insight into the quality and completeness of agency reporting. Finally, a description of the formal testing process follows. This testing addresses six questions:

1. What information on use of force do agencies have readily available?

2. Will agencies provide this information?

3. Do agencies correctly interpret the scope of the collection?

4. Do agencies provide the correct information from their record systems?

5. What procedures are optimal for minimizing agency nonresponse and maximizing data quality?

6. Do agencies find the National Use-of-Force Data Collection application intuitive and easy to use?

The proposed testing occurs in an iterative process, with prior testing informing future testing panels in an effort to expedite the full clearance process. The testing plans are a proposed set of generic, small scale tests, building to a full implementation. Permission for the pretesting activities are sought under the existing UCR Generic Clearance, Office of Management and Budget (OMB) Number 1110-0057).

Pre-testing

Pre-testing activities will be conducted prior to the initiation of the pilot study. These activities will provide the preliminary information needed to both construct the sample of targeted agencies for the pilot study and identify early problem areas that can be resolved prior to formal testing. The pre-testing consists of three parts: cognitive testing, a canvass of state UCR Program Managers and CJIS Systems Officers, and testing of questionnaire design and usability.

Cognitive Testing

Purpose of the Research

The purpose of the cognitive testing is to investigate the understanding of the language and wording of the questions in the proposed data collection on law enforcement use of force, as well as their associated instructions by the law enforcement community. The ultimate goal for the development research activities is to ensure that participants have a clear understanding of what information is requested even in complex law enforcement situations. This will aid the UCR Program in its efforts to increase the overall validity and reliability of its data collections. The cognitive testing is a first step to understand the extent to which the law enforcement community has a common understanding of key concepts in the data collection. In addition, there are questions that ask participants to indicate what records are readily available on certain key pieces of information such as time and location.

The cognitive testing instrument was developed with input from the law enforcement community (through the Use of Force Task Force membership), the BJS, and William Bozeman, M.D. Dr. Bozeman is a physician in the Department of Emergency Medicine at Wake Forest University. He has been extensively published in the research area of injury and law enforcement use of force and is a member of the Police Physicians Section of the International Association of Chiefs of Police. We anticipate continuing our collaboration with all three parties for both pre-testing activities and the pilot study. Based upon input from all parties, draft questions were revised to reflect the final version attached to this document.

Methodological Plan

The cognitive testing primarily focused upon the language and construction of the response categories rather than the usability of the Web-form that is under construction or other questions on mode of collection. These usability tests will be conducted as a part of system development prior to the beginning of the pilot study. The purpose of the cognitive test is to identify key concepts that may have the potential for a high amount of variability in their interpretation. These areas will require thorough explanation to promote the reliability of the information measured.

The content and scope of the National Use-of-Force Data Collection was constructed based upon the consensus of representatives from the law enforcement community. Through the CJIS APB and the work of the Use of Force Task Force, the law enforcement community has indicated that this information is valuable for understanding the circumstances surrounding a use of force by law enforcement and that the information exists in local records on these events. However, there is a lack of information to understand the extent to which the law enforcement community applies certain terms on a consistent basis.

The purpose of the questions on the cognitive testing instrument was to identify areas where there might not be a common understanding of the same terminology. In essence, the results of the test provided a general assessment of whether there is an existing normative understanding of some concepts in the National Use-of-Force Data Collection. This questionnaire was not designed to be used to understand how the terminology may be applied on complex law enforcement scenarios. The FBI plans to do further analysis on the application of definitions and guidance during the upcoming pilot study.

The areas that will be addressed in the cognitive testing include the following:

• The assignment at the time of the incident

• The selection of the location and location type (because many location types are not mutually exclusive)

• Further exploration on the request identifying aggression

• The application of the legal definition of serious bodily injury

On each of these particular concepts, the participants will be presented with a series of questions. Some of the questions will involve a simple “yes” or “no” response based on how information is recorded by law enforcement. Other questions will present an array of responses for their ranking or interpretation. For example, on the question of serious bodily injury, a list of potential injuries will be offered to participants. Each participant will indicate the injuries that he or she understands to be “serious” based upon the definition provided.

Participants were recruited by a solicitation to the 280 participants in the FBI National Academy. These 280 potential participants represent the total roster of the current FBI National Academy class. The FBI National Academy is a 10-week training program of leaders and managers of state, local, county, tribal, military, federal, and international law enforcement agencies. The questionnaire was administered during a group assembly in November 2016 with 149 completed surveys returned.

Analysis

Of the 149 participants in the survey, the majority were law enforcement representatives from municipal police agencies (61.4 percent), followed by county sheriffs (12.4 percent) and state police (10.3 percent). The largest share of participants described themselves as “mid-level management” (72.2 percent). The regional representation of participants generally mirrored the regional representation of the UCR Program with most participants indicating that they are from the South and the fewest indicating that they are from the Northeast. In most cases, the results of the testing led to improvements in instructions that will be provided as part of the National Use-of-Force Data Collection. For questions on both injury and resistance/weapons, results indicated the need for additional data values. For more detailed results, a technical report is available upon request.

Canvass of State UCR Programs

Working with the Association of State UCR Programs (ASUCRP) and the BJS, the FBI constructed a survey for state UCR Program Managers. The primary purpose of this survey is to identify potential participants in the pilot study that will commence January 2017. The proposed National Use-of-Force Data Collection will allow for states or other domains (for example, federal or tribal agencies) to determine one of three primary paths to submit and manage their data. The states or domains that have their own data systems to collect and maintain use of force data will be able to submit their data as a bulk submission. In the absence of a state/domain solution, the states or domains will have the ability to use the data collection tool built and maintained by the FBI on the LEEP. The LEEP is a restricted access environment accessible through the Internet that hosts law enforcement services, including the proposed use-of-force data collection tool. Law enforcement agencies within the state will directly access the data collection tool on LEEP to submit their agencies’ information. The state or domain UCR program will indicate whether they will manage the data collection, as is the case with other UCR data collections, or if the state will allow for agencies to submit data as a direct contributor.

Both phases of the pilot study are predicated on selecting a few states for targeted recruitment for each of the phases. The canvass will assist the FBI with identifying state UCR programs that will be using the data collection tool on LEEP fully manage their use of force data collection versus those that have the ability to capture the data in a state system. Additional considerations for the selection of participants in the pilot study include the following information:

• Are there any statutory obligations to collect UCR data?

• Are there any statutory obligations to collect data on law enforcement use of force? If so, what is the scope of that collection?

• Does the state have or anticipate building a system to capture information on law enforcement use of force?

• What are the technical capabilities of existing or proposed state systems to collect information on law enforcement use of force?

• Will the state program retain management authority over the use of force data collection, or will the state allow all law enforcement agencies to provide data directly to the FBI?

• Will the state fully validate data prior to submission to the FBI?

Background Research on Survey Instrument

The survey was constructed with input from various entities both within and outside the FBI which are involved with various aspects of UCR data management—including the ASUCRP and the CJIS APB. Based upon this input, there were six primary areas that were seen to have a potential to influence on participation in the National Use-of-Force Data Collection. These areas were also identified as most likely to affect levels of participation in the National Use-of-Force Data Collection and subsequent data completeness and quality based upon past experience of FBI personnel and early communications with potential data contributors.

The areas covered in the survey include the following:

• How management of the National Use-of-Force Data Collection will be organized (2 questions)

• Technical capabilities with state or domain systems (9 questions)

• State statutes regarding UCR or Use-of-Force data collection (4 questions)

• Data quality, training, and auditing capabilities of state or domain systems (7 questions)

• Publication of use-of-force data by the state or domain (4 questions)

• Use of the LEEP by personnel with the state or domain program (6 questions)

Methodological Plan and Selection of Participants

The survey was distributed to state and domain points-of-contact via email in a fillable portable document format. These individuals are usually described as program managers or CJIS Systems Officers (CSOs). The FBI relies upon these individuals to be the main conduit for the collection of UCR data. This would include all 50 states and territories, as well as tribal and federal domain managers. This was not a sample-based survey and had to 104 potential participants based upon two points-of-contact per state and one per territory. The federal domains were also given the option of completing the survey to include the four DOJ law enforcement entities, the Bureau of Indian Affairs, and other federal law enforcement agencies. The survey had one survey returned from 41 of each of the states and outlying areas.

Analysis Plan

The responses from the survey will be analyzed to identify areas that the FBI needs to address to ensure that the data submissions to the National Use-of-Force Data Collection have a high level of completeness and quality. In addition, the possible state participants in the pilot study will be identified based upon the responses to primarily two variables. The pilot states would need to be both managing their collection of use-of-force data and also using the FBI-constructed collection application on LEEP.

The analysis of the responses involved descriptive statistics. Since the variables measured on the survey are mostly categorical in nature, these statistics involved frequency distributions and cross-tabulations. This survey does not rely upon a statistical sample or seek to test any hypotheses, so there is no need for statistical tests of significance. An internal report will be completed with the results of the analysis and shared with the participants of the survey and others upon request.

Testing of Questionnaire Design/Usability Testing

Background Research

The FBI’s approach to questionnaire design and usability testing is a multistage, iterative approach commonly associated with agile development. Final usability testing will be conducted as a part of Operational Evaluation testing in January and February 2017. In addition to Operational Evaluation testing, FBI will have conducted thirteen separate demonstrations of the application in front of 111 different individuals. These demonstrations included the same users on certain occasions, which allowed the development team the opportunity to receive feedback on adjustments and modifications.

A demonstration was conducted after each development sprint for a total of six. The audience for these demos were the Product Owner and at least one representative each from Contracts, Configuration Management, Testing, Crime Data Modernization, LEEP, and Security each representing particular expertise with web application development. In addition, audience members represented the sworn and civilian law enforcement community and provided feedback on the intuitive nature and ease of use of the application. The smallest number of witnesses for a demo was 11, and the most was 22. Each sprint demonstration consisted of a walk-through of the user stories that had been agreed to for that particular sprint, a demonstration of the applications abilities with particular emphasis on the new capabilities related to the stories, a discussion regarding whether or not the new capabilities fulfilled the agreed-to stories, and an agreement as to what stories were completed and what stories (if any) still needed completed to address the usability of the application.

Of the thirteen total demonstrations, there were two internal demonstrations (one after the third sprint and one after the sixth) conducted for the project stakeholders. These demonstrations were conducted for various unit and section chiefs throughout CJIS as well as the Deputy Assistant Directors of the Operational Programs Branch and the Information Services Branch. The format in both cases was a PowerPoint slideshow defining the system and relaying the approved stories followed by a live demonstration of the application. The rest of the time was spent taking feedback from the attendees for possible incorporation into later iterations of the product. Average attendance was 22.

Recently, the FBI was able to conduct a series of five demonstrations with external stakeholders. A demonstration was conducted for the FBI Inspection Division at FBI Headquarters to get them familiar with the upcoming application. The demonstration was witnessed by an audience of executives to include four FBI Special Agents and subject matter experts at FBI Headquarters. There was an active give-and-take session throughout the demonstration, and several action items were captured to investigate for possible inclusion in future releases of the application. A similar demonstration was later held at FBI Headquarters for representatives from the Drug Enforcement Agency, Bureau of Alcohol, Tobacco, Firearms, and Explosives, and the United States Marshals Service. Finally, three demonstrations were held via Skype with state UCR Program managers and state criminal justice information services managers. In all five demonstrations, attendees had the opportunity to provide feedback and ask questions about the way the application was designed to function.

Methodology

The Operational Evaluation consisted of three testing sessions, each one hour in length. Two sessions were conducted in one of classrooms at the FBI CJIS Division facility on the same day, while a third testing session was conducted approximately one month later. The first session consisted of nine participants who each will be assigned two scenarios to follow. Scenario A consisted of placing a use-of-force incident into the system for review, and Scenario B consisted of creating and submitting a Zero Report. A Zero Report is simply a record submitted by an agency stating that they did not have any use-of-force incidents that month. It is expected to be the most common of all types of submissions.

The second session consisted of fourteen participants who were assigned two scenarios each. Scenarios were a mixture of the earlier Scenarios A and B, as well as new Scenarios C and D. Scenario C consisted of going into the system, locating the incidents created in Scenario A, and approving those incidents for submission. Scenario D was the review of a Zero Report.

Each volunteer was asked to complete System Usability Scale (SUS) questionnaire. Results were evaluated for possible changes in future releases of the product. Based upon the twenty-eight participants providing complete responses in the Operational Evaluation, the average score was 79.464. This score indicates that participants generally found the application to be easy to use. The ratings of four participants resulted in individual scores below 68. Any system scoring above 68 on the SUS is deemed above average in usability. More detailed information on the results is available upon request.

Participant Selection

Volunteers were be pulled from various sources, both internal and external to the CJIS Division. Volunteers were be solicited from FBI Special Agents assigned to the CJIS Division, the FBI Police force assigned to the CJIS Division, various record clerks within the CJIS Division (as record clerks will most assuredly be the individuals inputting incident information, especially in larger law enforcement organizations), and local (city, county, or state) police. Nonfederal volunteers will be limited to nine or fewer, partially due to the difficulty receiving their clearances necessary to enter the facility.

Evaluation/Analysis

The sessions will be evaluated via the SUS² .

The SUS will consist of ten responses:

1. I think that I would like to use this system frequently.

2. I found the system unnecessarily complex.

3. I thought the system was easy to use.

4. I think that I would need the support of a technical person to be able to use this system.

5. I found the various functions in this system were well integrated.

6. I thought there was too much inconsistency in this system.

7. I would imagine that most people would learn to use this system very quickly.

8. I found the system very cumbersome to use.

9. I felt very confident using the system.

10. I needed to learn a lot of things before I could get going with this system.

The SUS will use the following response format:
 

The SUS will be scored as follows:

• For odd items: subtract one from the user response.

• For even-numbered items: subtract the user responses from 5

• This scales all values from 0 to 4 (with four being the most positive response).

• Add up the converted responses for each user and multiply that total by 2.5. This converts the range of possible values from 0 to 100 instead of from 0 to 40.

Pilot

The pilot study will consist of two phases. Each phase will include a set of target agencies and states that will allow for sufficient data to evaluate intercoder reliability in the application of definitions and guidance. While survey design “best practices” can be used to inform the process of eliciting information from individuals providing law enforcement statistics for the UCR Program, the data collection is more similar to an extensive process of content analysis. Information captured within law enforcement records and narratives serve as the basis for the statistical information forwarded to the FBI. The challenge for the FBI is to communicate coding schemes based upon a common set of definitions. Instructions and manuals, as well as training modules and curricula, all serve to help guide individuals at law enforcement agencies to translate their local records into a uniform manner. While basic instructions will be provided during the pilot study, the results of the pilot study will identify concepts with less consensus across locations and types of law enforcement agencies for the development of in-depth instructions, manuals, training modules, and curricula. After each phase of the pilot study, the FBI will file a change request with the OMB to receive clearance to continue with the phased-in approach to this data collection and its movement from a pilot study to full collection. These change requests may be either substantive or nonsubstantive depending upon the nature of the request.

Phase I

The activities of the first phase of the pilot will focus upon a prospective comparison of reported incidents in the National Use-of-Force Data Collection through the data collection tool on LEEP to the original records voluntarily provided by the reporting agency to the FBI. Those agencies that are recruited and agree to participate in the pilot study will understand that local records will be forwarded to the FBI upon submission of statistical information to the National Use-of-Force Data Collection tool. The local case information can be redacted of any personally identifiable information prior to its forwarding to the FBI, and all local records will be destroyed upon completion of the pilot study.

The targeted agencies for participation in the pilot study will include three groups:

• The largest law enforcement agencies with a workforce of 750 or more sworn officers. The group of the largest agencies includes approximately 90 agencies across 32 states. Each agency will be approached through their state UCR Program Manager for their voluntary agreement to provide data for submission in the use of force data collection and participate in the pilot study activities.

• In addition to the subset of largest agencies, the FBI will identify two to five states that will voluntarily participate in the first and second phases of the pilot. These states will be selected based upon the results of the canvass of the states during pretesting. These states will represent those state UCR programs that will be using the data collection tool on LEEP to manage their data collection for their states.

• All four DOJ law enforcement agencies will be asked for voluntary participation.

• Finally, any other agency that voluntarily approaches the FBI to provide their information on incidents that meet the definitional requirements will also be accepted.

The FBI Quality Assurance Review (QAR) staff will review incidents and records provided by the pilot agencies and states. The goal of this review is to ascertain whether the agencies are applying the definitions and using the provided instructions in a uniform manner. In particular, the records comparison will investigate the application of the legal definition of serious bodily injury and the coder’s understanding of how that definition can be operationalized. The records review and comparison will also identify problematic areas where instructions need more detail or more training should be provided to agencies. The data will also be used in the planning of the second phase of the pilot that will involve a site visits to a subset of agencies. Finally, the FBI will work with state UCR Program Managers in the pilot states to identify any potential problems with local and state record-keeping that impedes the ability to provide the use of force information to the FBI.

Phase I of the pilot study will after clearance is received. Assuming clearance is received by the end of January 2017, Phase I would conclude at the end of April 2017.

Phase II

The second phase of the pilot will include the original set of agencies recruited for the first phase. In addition, two more states will be recruited for participation in the second phase of the pilot representing state UCR programs that will provide their use of force data in a bulk data submission. These states will be nominated based upon the information gained from the canvass of state UCR programs during pre-testing. The FBI will also continue to accept agencies and states that provide data voluntarily to the data collection.

The activities of Phase II are primarily centered on an extension of the records review and comparison with targeted, on-site visits with a sample of pilot agencies. Working with the BJS in the development of a statistically defensible sampling strategy, the FBI will use a geographically driven cluster sample of agencies for on-site visits by FBI personnel. The primary goal of the on-site visits is to ascertain the level of underreporting of within-scope incidents, especially those with serious bodily injury or firearm discharges. The on-site visits will also allow for an assessment of local record-keeping capabilities and testing of any adjustments made to the language of instructions and data elements or changes to the data collection.

As with Phase I, the on-site reviews will be conducted by FBI QAR staff who will review all records connected to use-of-force incidents maintained by the sampled agency. The primary purpose of the review is to assess whether any incidents occurred that should have been reported to the use of force data collection, based upon the definition, but were not. The results of Phase II of the pilot study will be an assessment of the amount of underreporting of use of force incidents that meet the definition of the use of force data collection, especially for those instances that do not result in a fatality. The Phase II of the pilot study is anticipated to last three months and will begin at the conclusion of Phase I. The results of Phase II of the pilot study will be provided to the OMB at its conclusion including any substantive or nonsubstantive changes. The full collection of the data would not proceed until clearance from the OMB is received.

Analytical Plan and Publication from Pilot Study

The analysis of information gathered through both phases of the pilot study will begin as soon as possible and will continue in an iterative fashion. The development of the data collection tool is using an agile development process that will easily accommodate changes to the collection tool identified throughout the pre-testing and pilot phases of this plan. A dedicated technical team is under contract to ensure the FBI is able to react quickly to these potential changes.

At the conclusion of Phase II, the FBI will release a report detailing the results of its data collection and analysis. The FBI will provide periodic updates to the OMB throughout the pilot study. The results from the pilot study will be released to the public and will consist of four sections. The first section will provide the results of the on-site assessment regarding underreporting and completeness, as well as an assessment of the reliability of reported data from the Phase I records review. All results in this section will be pooled, and no individual agency will be identified. The second section will provide results of the analysis of nonresponse and missing data. This section will also identify whether a need clearly exists for a nonresponse bias study and a proposed methodology for that study. Again, all results will be pooled, and no individual agency will be identified in the second section. As the pilot study only has two phases, the third section of the report will detail the data collection policies and procedures that will assist with maintaining data quality and completeness as a permanent and final data collection. The third section will also detail any on-going collaboration and partnership between the FBI and the BJS to achieve and maintain a high-level of data quality. Finally, the fourth section will list basic agency-level counts of reported data from all participating agencies as a showcase of item completeness and quality.

Timeline

5. Contacts for Statistical Aspects and Data Collection

The following individuals were consulted on statistical aspects of the design:

Jeri Mulrow, Ph.D.

Acting Director

Bureau of Justice Statistics

(202) 514-9283

William Sabol

Director (past)

Bureau of Justice Statistics

Michael Planty, Ph.D.

Deputy Director

Bureau of Justice Statistics

(202) 514-9746

Shelley Hyland, Ph.D.

Statistician

Bureau of Justice Statistics

(202) 616-1706

The FBI UCR Program does not have immediate plans to use contractors, grantees, or other persons to collect and analyze the information on behalf of the UCR Program.

1 Banks, D, et al. (2015) Arrest-Related Deaths Program Assessment: Technical Report. RTI International, http://www.bjs.gov/index.cfm?ty=pbdetail&iid=5259

2 Information available at https://www.usability.gov/how-to-and-tools/methods/system-usability-scale.html (accessed on December 15, 2016).