LEOKA OMB Generic Clearance Memo - Cognitive Testing

MEMORANDUM

MEMORANDUM TO: Bob Sivinski

Office of the United States Chief Statistician

Office of Information and Regulatory Affairs

Office of Management and Budget (OMB)

THROUGH: Melody Braswell

Clearance Officer

Justice Management Division

Rainer S. Drolshagen

Deputy Assistant Director

Criminal Justice Information Services (CJIS) Division

Amy C. Blasher

Chief of Crime Statistics Management Unit

Federal Bureau of Investigation (FBI)

FROM: Kristi L. Donahue

Management and Program Analyst

FBI

DATE: January 25, 2018

SUBJECT: FBI Request for OMB Clearance for cognitive testing of the new Law Enforcement Officers Killed and Assaulted (LEOKA) database, a web-based instrument to be used for the FBI UCR LEOKA Program’s data collection under the OMB generic clearance agreement (OMB Number 1110-0057).

Background

In March 2010, the OMB approved the collection of 1-701, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Feloniously Killed and Assaulted and 1-701a, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Accidentally Killed which contained additional data elements from the previously approved forms. Although these forms have been disseminated since January 1, 2011, the UCR LEOKA Program has not been able to use all of the data collected from these changes because a database has not been developed to accommodate the expanded Forms 1-701 and 1-701a. The percentage of data collected since 2011 not captured in the UCR LEOKA database includes 68 percent of each felonious incident and 91percent of each accidental death. Since January 1, 2011, the UCR LEOKA Program has received an average of 202 data collection forms per year.

The UCR LEOKA Program seeks to enhance its data collection process with the development and deployment of a LEOKA database. This database will allow users to electronically complete Forms 1-701 and 1-701a (OMB No. 1110-0009). The new UCR LEOKA database is a software application and will be housed on the Law Enforcement Enterprise Portal (LEEP). This database will provide a graphical user interface tool to walk the preparer through a series of questions relevant to the LEOKA incident in which the incident details will automatically be submitted via the database. These incidents will not be used until verified for accuracy by the UCR LEOKA Team.

In 2017, during the first phase of developing the database, the UCR LEOKA Team worked with a contract company to develop a new database providing a solution which allows UCR LEOKA staff to enter the new data elements. This database was implemented in late November 2017.

During the second phase of this initiative, the UCR LEOKA Team will present recommendations to the CJIS Advisory Policy Board (APB) to further expand the 1-701, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Feloniously Killed and Assaulted and 1-701a, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Accidentally Killed. Since approval of these forms in 2010, the UCR LEOKA Team has received numerous requests for additional data elements from various agencies throughout the law enforcement community. Based on these data requests, the UCR LEOKA Team has worked to improve both collection forms through research and analysis to add data elements, reword/rearrange questions, and eliminate unnecessary questions in order to provide the most accurate and useful data to the law enforcement community. In an effort to ensure the data was relevant and accurate, the UCR LEOKA Team created a focus group which included representatives from the FBI, Major City Chiefs Association, International Association of Chiefs of Police, and the National Sheriffs’ Association. The forms were then presented to the focus group and its members were asked to select actual incidents of officers killed and assaulted, within their agency, and complete the forms using those incident reports. This process was done to gain input from the members and to test the usability of the forms. The focus group members provided valuable feedback; their ideas and suggestions were incorporated on the forms.

In addition, the National Use-of-Force Data Collection Task Force was also asked to review the forms to determine the relevance of the data collection. The forms were emailed to each task force member who were given a month to review the documents. The UCR LEOKA Team then spoke to each member via teleconference to discuss any comments, questions, or changes. Following the teleconferences, all of the suggestions were compiled and the appropriate changes were made to the forms.

The UCR LEOKA Team is currently working with the FBI Information Technology staff to complete phase three, development of the new UCR LEOKA database. The two groups plan to finalize the database requirements this month.

The UCR LEOKA Team and the Information Technology Management Section will continue the development and internal testing phases of the new UCR LEOKA database from February through June 2018. During the same timeframe, the new collection recommendations will be introduced during the 2018 Spring CJIS Advisory Policy process. Once approved by the APB, the expanded forms will undergo cognitive testing in July 2018. This process will involve UCR LEOKA personnel conducting cognitive testing and interviews to identify usability issues with the application of the UCR LEOKA database with 20 law enforcement personnel. In addition, the cognitive interviews will provide an opportunity to reveal whether the transition from a paper-based collection instrument to a web-based collection instrument has introduced any issues/concerns with the general understanding of the requested information or confusion with the wording of instructions. The results of the cognitive interviews will allow the UCR LEOKA Team to fine tune the language and question structure.

Finally, the UCR LEOKA Team will document the findings of the cognitive testing/interview process and use this information to finalize the new UCR LEOKA database throughout August and September 2018. In order to meet OMB’s approval timeframe, the final Information Collection Review package for this collection will be submitted in October 2018.

Request

The FBI is requesting clearance for the cognitive testing of changes to an existing instrument used in the UCR LEOKA Program collection in order to capture incident data in regard to the deaths and assaults of law enforcement officers (LEOs). The LEOKA Program is a component of the FBI UCR Program, which collects statistical information on LEOs killed or assaulted in the line of duty on a monthly basis. The results of the cognitive interviews will be used to assess the impact of transitioning from the current paper-based instrument to a web-based instrument.

The UCR LEOKA Program seeks to enhance its data collection process with the development and deployment of the new UCR LEOKA database. The UCR LEOKA database will allow users to electronically complete Forms 1-701, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Feloniously Killed and Assaulted and 1-701a, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Accidentally Killed (OMB No. 1110-0009).

The current request for approval, under the FBI UCR Generic Clearance (1110-0057), is for cognitive testing of the UCR LEOKA Database with 20 law enforcement personnel. The cognitive testing will require 20 burden hours.

For this clearance, the UCR LEOKA Program will conduct cognitive testing and interviews to identify usability issues with the application of the UCR LEOKA Database. In addition, the cognitive interviews will allow for the opportunity to reveal whether the transition from a paper-based instrument to a web-based instrument has introduced any problems with the general understanding of the requested information or confusion with the wording of instructions. The results of the cognitive interviews will allow the UCR LEOKA Program to modify the language and question structure. The information collected will also be used to refine the database if necessary.

Purpose of the Research

The primary goal of the UCR LEOKA Program is to reduce the number of LEO line-of-duty deaths and assaults by providing data, research, and instructional services relative to law enforcement safety. The UCR LEOKA Program provides data regarding officer deaths and assaults via its annual publication and responds to special data requests. The data is used to:

• Identify circumstances and trends in which officers are killed and assaulted in the line of duty;

• Aid LEAs in developing policies and training programs to improve officer safety;

• Include in the curriculum of the UCR LEOKA Officer Safety Awareness Training; and

• Provide information to UCR LEOKA Liaison Specialists for the composition of officer safety articles on a monthly basis.

Currently, when a LEOKA incident occurs, the FBI sends Form 1-701, Analysis of Officers Feloniously Killed and Assaulted, or Form 1-701a, Analysis of Officers Accidentally Killed, to a law enforcement employee (preparer) in the same agency as the victim officer. Much of the form does not apply to the particular incident and can be cumbersome and time consuming for the preparer to complete and return. Since the implementation of Forms 1-701 and 1-701a in 2011, the UCR LEOKA Team has identified many inconsistencies on completed forms which were not identified during the original cognitive testing of the forms. The UCR LEOKA staff believe the inconsistencies exist due to the preparer(s) misperception when completing the form. In an effort to streamline this process for all involved and alleviate these issues, the FBI is seeking to automate this process.

The goals of this cognitive interview and testing phase are to:

• Identify whether the change from a paper-based collection to a web-based collection reduces or introduces any unforeseen issues with comprehension for the respondents;

• Assess the comprehension of terms and definitions included in the collection;

• Improve the response rate; and

• Eliminate reporting errors which are made by the preparer when completing the paper forms.

Design of the LEOKA Database

The new UCR LEOKA database is a software application and will be housed on LEEP. Similar to tax filing preparation software, the database shall provide a graphical user interface tool to walk the preparer through a series of questions relevant to the LEOKA incident for the purpose of reporting deaths and assaults to the LEOKA Program.

Selection of Law Enforcement Participants in Cognitive Interviews

The law enforcement community is typically organized and identified based upon the type of community they serve, and are identified as those which serve incorporated cities, counties, state police, campus police, tribal police, and federal law enforcement. These broad categories can be further divided based upon the size of the populations they serve. In an effort to gain a broader perspective, participants will be selected from a small county police department, a large urban police department, a state police agency, a college/university agency, and a federal agency.

In addition to selecting a set of agencies which represent the variety of viewpoints based upon agency type, participants were also selected based on the role they play within their agency. Since the mode of collection is the primary focus of the testing, participants were selected from both civilian and sworn employee populations, which include record clerks, detectives, patrol officers, and chiefs of police. In most cases, the Forms 1-701 and 1-701a are completed and submitted by personnel in these roles.

Cognitive Interview Procedures

Cognitive interviewing and testing is tentatively scheduled for July 2018. The cognitive interviews will assess two aspects of the revised collection. The first component will focus on whether or not the new web-based collection introduces problems with comprehension or general usability through the transition from one mode of collection to another. The second component will specifically target possible comprehension problems associated with terminology, definitions, and criteria requirements.

The twenty law enforcement personnel will participate in the cognitive interviews once a firm schedule is set. The method involves intensive, one-on-one interviews in which the participant will be asked to "think aloud" as he or she completes a submission based on specific incident details. A number of different techniques may be involved, such as asking respondents to paraphrase questions or asking probing questions to determine how respondents came up with their answers. The objective is to identify problems of ambiguity or misunderstanding, identify potential appearance improvements, flow, and instructions, or highlight other difficulties respondents have answering questions. Participants will be provided three incident examples which include an accidental death, a felonious killing, and an assault with injury which involved a firearm, knife, or other cutting instrument. These incident examples will be used to evaluate the impact of changing from a paper-based instrument to a web-based instrument and identify any other issues as described above. Each interview will take approximately one hour.

A team of two individuals from the UCR LEOKA Team will conduct the cognitive interviews with each participant individually. This will allow for both cognitive interviewers to take detailed notes of responses provided by the participant and record behavioral cues. One cognitive interviewer will be directly viewing how the participant is filling out the data collection on a computer and will be primarily responsible for recording any behavioral cues that would indicate difficulty with the mode of collection. Examples of this behavior may include hesitation, hovering over specific items, or a misunderstanding of terminology. The second cognitive interviewer will be primarily responsible for recording the verbal information provided by the participant as a part of the “think aloud” aspect of the cognitive interview.

Language

The cognitive interviews will be conducted in English.

Burden Hours for Cognitive Testing

The FBI UCR Program is requesting a total of 20 burden hours for 20 law enforcement personnel (sixty minutes per respondent). No incentives will be provided in exchange for participation.

Analysis Plan

During the cognitive interviews, participants will be asked to “think aloud” as they use the UCR LEOKA database to complete incident submissions based on each of the three examples. Due to the limited number of participants, the analysis will be qualitative rather than quantitative. The notes taken by the cognitive interviewers will be analyzed for patterns of problems associated with the comprehension of certain questions and items, as well as any indication of difficulties with the web-based data collection.

Responses will be categorized by whether there are verbal and behavioral indications of comprehension problems and confusion and/or technical problems with the UCR LEOKA database. Particular attention will be paid to differences which emerge from the different types of law enforcement agencies, as well as differences between sworn and civilian law enforcement employees.

The FBI will produce a final report summarizing overall indications of problems with validity and comprehension. In addition, a question-by-question summary will be provided, identifying any difficulties encountered by participants based upon the new web-based mode of collection and recommended changes to questions, instructions, and terminology.

Informed Consent, Data Confidentiality and Data Security

Law enforcement participants will be invited to participate in the cognitive testing and interviews via a telephonic communication. These telephone invitations will provide an introduction to the UCR LEOKA Program’s analysis forms and a description of the cognitive interview process. Further, this communication will provide the purpose of the survey, the voluntary nature of the study (informed consent), how the participants were selected, and a number to call with questions about the study. The telephone invitation and introduction to the survey announced the estimated length of the interview in advance, allowing the participant an opportunity to decline if the burden would be unacceptable.

Once the participants arrive at the location for the cognitive interview, the cognitive interviewers will reiterate the points delivered during the telephone invitation on the voluntary nature of the cognitive interview and the purpose of the research. In addition, the participants will be assured their response will be protected to the extent we are legally allowed and they may stop the interview at any time for any reason. Upon request, this information will be provided to the participants in hard-copy for their signature along with a copy for their own records.

4