Generic Clearance Memo

MEMORANDUM

MEMORANDUM TO: Shelly Wilkie Martinez

Official of Statistical and Science Policy

Office of Management and Budget

THROUGH: Lynn Murray

Clearance Officer

Justice Management Division

John H. Derbas

Chief of Law Enforcement Services Section

Federal Bureau of Investigation

Samuel Berhanu

Chief of Crime Statistics Management Unit

Federal Bureau of Investigation

FROM: Melissa J. Blake

Management and Program Analyst

DATE: May 11, 2015

SUBJECT: FBI Request for OMB Clearance for cognitive testing of the LEOKA Wizard, an automated tool to be used for the Law Enforcement Officers Killed and Assaulted Program Data Collection under the OMB generic clearance agreement (OMB Number 1110-0057).

The Federal Bureau of Investigation (FBI) is requesting clearance for the cognitive testing of changes to an existing instrument used in the Law Enforcement Officers Killed and Assaulted (LEOKA) Program collection in order to capture incident data in regard to the deaths and assaults of law enforcement officers. The LEOKA Program is a component of the Uniform Crime Reporting (UCR) Program, which collects statistical information on the counts and characteristics of crime from law enforcement agencies on an annual basis. The results of the cognitive interviews will be used to assess the impact of transitioning from the current paper-based instrument to a web-based instrument.

The LEOKA Program seeks to enhance its data collection process with the development and deployment of the LEOKA Wizard. The LEOKA Wizard will allow users to electronically complete forms 1-701, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Feloniously Killed and Assaulted and 1-701a, Law Enforcement Officers Killed and Assaulted Program; Analysis of Officers Accidentally Killed (OMB No. 1110-0009). To allow for law enforcement agency technology and policy constraints, paper forms will remain available.

The current request for approval, under the FBI UCR Generic Clearance (1110-0057), is for cognitive testing of the LEOKA Wizard with 20 law enforcement personnel. The cognitive testing will require 20 burden hours.

For this clearance, the LEOKA Program will conduct cognitive testing and interviews to identify usability issues with the application of the LEOKA Wizard. In addition, the cognitive interviews will allow for the opportunity to reveal whether the transition from a paper-based collection instrument to a web-based collection instrument has introduced any problems with the general understanding of the requested information or confusion with the wording of instructions. The results of the cognitive interviews will allow the LEOKA Program to fine tune the language and question structure. The information collected will also be used for possible future refinements depending upon the findings.

Purpose of the Research

The primary goal of the LEOKA Program is to reduce the number of law enforcement officer line-of-duty deaths and assaults by providing data, research, and instructional services relative to law enforcement safety. The LEOKA Program provides data regarding officer deaths and assaults via its annual publication and responds to special data requests. The data is used to:

1. Identify circumstances and trends in which officers are killed and assaulted in the line of duty,

2. Aid law enforcement agencies in developing policies and training programs to improve officer safety,

3. Include in the curriculum of the LEOKA Officer Safety Awareness Training (OSAT), and

4. Provide information to LEOKA Training Instructors for the composition of officer safety articles on a monthly basis.

Currently, when a LEOKA incident occurs, the FBI sends Form 1-701, Analysis of Officers Feloniously Killed and Assaulted, or Form 1-701a, Analysis of Officers Accidentally Killed, to a law enforcement employee (preparer) in the same agency as the victim officer. Much of the form does not apply to the particular incident and can be cumbersome and time consuming for the preparer to complete and return. Since the implementation of Forms 1-701 and 1-701a in 2011, the LEOKA staff has identified many inconsistencies on completed forms that were not identified during the original cognitive testing of the forms. LEOKA staff believes the inconsistencies exist due to the preparer(s) confusion when completing the form. In an effort to streamline this process for all involved and alleviate these issues, the FBI is seeking to automate this process.

The goals of this cognitive interview and testing phase are to:

• Identify whether the change from a paper-based collection to a web-based collection reduces or introduces any unforeseen issues with comprehension for the respondents;

• Assess the comprehension of terms and definitions included in the collection;

• Eliminate reporting errors which are made by the preparer when completing the paper forms.

Design of the LEOKA Wizard

The design of Forms 1-701 and 1-701a was completed in such a way to be automated, submitted and ingested into the New UCR System or another database. However, delays in system development prompted the idea for a web-based instrument which eventually became the LEOKA Wizard concept. An accounting of the history and those involved in the development and design of the LEOKA Wizard is provided below:

1. After implementation of the redesigned LEOKA forms, LEOKA staff began to identify incomplete items, inconsistencies, and a lack of understanding of the skip features built into the forms for efficient completion. These issues were not identified during the cognitive testing of the forms before completion and OMB approval.

2. The LEOKA Wizard concept was proposed by LEOKA staff and presented to the Uniform Crime Reporting Re-development Project (UCRRP) Project Manager. The intention was to replicate the “Turbo Tax concept” to allow for contingency answers and would skip questions and sections which were not required based on previous answers to defining questions or items. It was believed this option would improve accuracy rates and reduce the burden on law enforcement personnel completing the forms and would also reduce the amount of time spent by LEOKA staff processing the death and assault incidents.

3. The project was assigned to an Information Technology Management Section (ITMS) employee. Upon assignment, the employee and the UCRRP Program Manager met with LEOKA staff in regard to details and proposed functionality of the new LEOKA Wizard.

4. LEOKA staff provided a detailed layout to the ITMS employee and he began to build the Wizard. However, a short time later, he was pulled from the project to assist in the development of the New UCR System.

5. After the ITMS employee’s reassignment, a Statement of Work was issued by ITMS and a contracted developer was hired to complete the project.

6. The contracted developer began design of the LEOKA Wizard in September 2014. Current projections show the LEOKA Wizard will be ready for internal usability testing during May 2015, and cognitive testing in June 2015. Internal testing will be conducted by LEOKA staff and will include multiple LEOKA incident examples.

The LEOKA Wizard is a software application that can be downloaded by the preparer to their local workstation. Similar to tax filing preparation software, the wizard shall provide a graphical user interface tool to walk the preparer through a series of questions relevant to the LEOKA incident. The software will then produce/save a file which the preparer will email to the LEOKA Point-of-Contact (POC).

The POC may review the information in their own copy of the wizard, update the information with identifiers and corrections, and then generate the incident details in a format that can be ingested directly into the new UCR System for the purpose of reporting deaths and assaults to the LEOKA Program.

Selection of Law Enforcement Participants in Cognitive Interviews

The law enforcement community is typically organized and identified based upon the type of community they serve, and are identified as those which serve incorporated cities, the unincorporated communities of counties, state police, campus police, and federal law enforcement. These broad categories can be further divided based upon the size of the populations they serve. In an effort to gain a broader perspective, participants were selected from Lewis County Sheriff’s Department (a small, county police department), Pittsburgh Bureau of Police (a large, urban police department), West Virginia State Police, West Virginia University Police Department, and the FBI Police Department.

In addition to selecting a set of agencies that represent the variety of viewpoints based upon agency type, participants were also selected based on the role they play within their agency. Since the mode of collection is the primary focus of the testing, participants were selected from both civilian and sworn employee populations, which include record clerks, detectives, patrol officers, and Chiefs. In most cases, the 1-701 and 1-701a forms are completed and submitted by personnel in these roles.

Cognitive Interview Procedures

Cognitive interviewing and testing is tentatively scheduled for June 2015 based on LEOKA Wizard completion and OMB approval. The cognitive interviews will assess two aspects of the revised collection. The first component will focus on whether or not the new web-based collection introduces problems with comprehension or general usability through the transition from one mode of collection to another. The second component will specifically target possible comprehension problems associated with terminology, definitions and criteria requirements.

The twenty law enforcement personnel will participate in the cognitive interviews once a firm schedule is set. The method involves intensive, one-on-one interviews in which the participant will be asked to "think aloud" as he or she completes a submission based on specific incident details. A number of different techniques may be involved, such as asking respondents to paraphrase questions or asking probing questions to determine how respondents came up with their answers. The objective is to identify problems of ambiguity or misunderstanding, identify potential appearance improvements, flow, and instructions, or highlight other difficulties respondents have answering questions. Participants will be provided three incident examples which include an accidental death, a felonious killing, and an assault with injury which involved a firearm, knife, or other cutting instrument. These incident examples will be used to evaluate the impact of changing from a paper-based instrument to a web-based instrument and identify any other issues as described above. Each interview will take approximately one hour.

A team of two individuals from the LEOKA Program will conduct the cognitive interviews with each participant individually. This will allow for both cognitive interviewers to take detailed notes of responses provided by the participant and record behavioral cues. One cognitive interviewer will be directly viewing how the participant is filling out the data collection on a computer and will primarily be responsible for recording any behavioral cues that would indicate difficulty with the mode of collection. Examples of this behavior may include hesitation, hovering over specific items, or a misunderstanding of terminology. The second cognitive interviewer will be primarily responsible for recording the verbal information provided by the participant as a part of the “think aloud” aspect of the cognitive interview.

Language

The cognitive interviews will be conducted in English.

Burden Hours for Cognitive Testing

We request a total of 20 burden hours for 20 law enforcement personnel (sixty minutes per respondent). No incentives will be provided in exchange for participation.

Analysis Plan

During the cognitive interviews, participants will be asked to “think aloud” as they utilize the LEOKA Wizard to complete incident submissions based on each of the three examples. Due to the limited number of participants, the analysis will be qualitative rather than quantitative. The notes taken by the cognitive interviewers will be analyzed for patterns of problems associated with the comprehension of certain questions and items, as well as any indication of difficulties with the web-based data collection.

Responses will be categorized by whether there were verbal and behavioral indications of comprehension problems and confusion and/or technical problems with the LEOKA Wizard. Particular attention will be paid to differences that emerge from the different types of law enforcement agencies, as well as differences between sworn and civilian law enforcement employees.

The FBI will produce a final report summarizing overall indications of problems with validity and comprehension. In addition, a question-by-question summary will be provided of any difficulties encountered by participants based upon the new web-based mode of collection and recommended changes to questions, instructions, and terminology.

Informed Consent, Data Confidentiality and Data Security

The telephone invitations, introduction to the LEOKA Program analysis forms, and a description of the cognitive interview that was given to the participants previously on the telephone provide the elements of informed consent. The telephone invitation provided the purpose of the survey, the voluntary nature of the study, how the participants were selected, and a number to call with questions about the study. The telephone invitation and introduction to the survey announced the estimated length of the interview in advance, allowing the participant an opportunity to decline if the burden would be unacceptable.

Once the participants arrive at the location for the cognitive interview, the cognitive interviewers will reiterate the points delivered during the telephone invitation on the voluntary nature of the cognitive interview and the purpose of the research. In addition, the participants will be assured that their response will be protected to the extent that we are legally allowed and they may stop the interview at any time for any reason. This information will be provided to the participants in hard-copy for their signature along with a copy for their own records should they wish.

4