SLEPS Cognitive Testing Memo to OMB

MEMORANDUM

MEMORANDUM TO: Jennifer Park

Official of Statistical and Science Policy

Office of Management and Budget

THROUGH: Jeri M. Mulrow

Acting Director

Bureau of Justice Statistics

Gerard Ramker

Deputy Director

Acting Chief, Law Enforcement Statistics Unit

FROM: Elizabeth Davis

Statistician, Law Enforcement Statistics Unit

Bureau of Justice Statistics

DATE: April 10, 2017

SUBJECT: BJS request for OMB Clearance to conduct cognitive interviewing for the Survey of Law Enforcement Personnel in Schools (SLEPS), under the OMB generic clearance agreement (OMB Number 1121-0339).

Introduction

The Bureau of Justice Statistics (BJS) is planning to conduct a new data collection to gain a better understanding of the roles, functions, and supporting infrastructure of police officers in schools through a project titled the Survey of Law Enforcement Personnel in Schools (SLEPS). The SLEPS has two primary goals, the first of which is to identify a national listing of active law enforcement agencies that have law enforcement personnel operating in some capacity in U.S. K—12 schools. The second primary goal of the SLEPS is to generate detailed, accurate, and reliable national statistics describing the scope, size, characteristics, and functions of law enforcement agencies and their personnel that work and interact in a school environment. To accomplish these goals, the SLEPS has a two-phased approach that will survey both law enforcement agencies and officers who work in schools.

This new data collection is a component of a larger school safety agenda. As outlined in the Department of Justice Appropriations Act, 2014 (Public Law 113-76), Congress tasked the National Institute of Justice (NIJ) to undertake the Comprehensive School Safety Initiative (CSSI), a research-focused program to increase the safety of schools nationwide. The CSSI identified a number of school safety-related topics for which more extensive data and research are required; one such topic is the presence of law enforcement in schools. There have been isolated local efforts to empirically examine law enforcement involvement in schools; however, no current comprehensive national-level data exist on the extent of law enforcement involvement in the nation’s schools or on their typical roles and responsibilities. To address this lack of data, NIJ entered into an interagency agreement with BJS, seeking to improve the amount of information pertaining to the roles, responsibilities, and actions of local law enforcement in schools.

Currently, there are limited data available regarding the presence of law enforcement in schools. BJS has collected the number of school resource officers (SROs) or other sworn personnel whose primary duties are related to school safety through the Census of State and Local Law Enforcement Agencies (CSLLEA) and the Law Enforcement Management and Administrative Statistics (LEMAS) Survey. Data are collected about every 4 years for the CSLLEA and about every 3 years for the LEMAS. The question asking about the number of SROs, however, was not included in the most recent completed administration of the LEMAS (2013) but is included in the 2016 LEMAS which is currently in the field. The periodicity of the CSLLEA and the LEMAS along with the narrow scope of the school safety question on these questionnaires limit the utility of these data and provide no insight on the roles of officers in schools or on the infrastructure in place to support these officers.

The National Center for Education Statistics (NCES) conducts the School Survey on Crime and Safety (SSOCS), which surveys principals at a nationally representative sample of over 3,000 public schools. SSOCS asks principals about the presence and activities of school security staff, including sworn law enforcement personnel. The most recent data available cover the 2009-2010 school year and contain only high level information on the school’s security staff as a whole. SSOCS was fielded again in the spring of 2016 with more detailed school security questions, including a distinction between types of security staff and their roles in the school. While there is a level of overlap between the SSOCS and SLEPS data collections, this overlap is not duplicative, but rather complementary. The SSOCS does not provide the information the SLEPS project seeks to gain regarding the number of law enforcement agencies with officers working in schools and the infrastructure within these agencies to support school safety. Additionally, the data collected by the SSOCS on officers working in schools consider only a narrow view of officer roles and responsibilities and do not explore officer characteristics such as training and experience.

To address the need for national-level data on the prevalence and roles of law enforcement in schools, BJS has contracted with RTI International to develop and test the SLEPS. RTI has awarded a subcontract to the Police Executive Research Forum (PERF) to assist with the SLEPS, given PERF’s experience with conducting officer-level surveys and their existing relationships with law enforcement agencies.

Concepts and topics for the SLEPS were developed and prioritized through ongoing discussions between the project team, NIJ, practitioners, researchers, and other key stakeholders. The project team also convened an Expert Working Group (EWG) to solicit input and feedback on topics of interest and utility, following up with a subset of the EWG to review early drafts of SLEPS questionnaires. The EWG was designed to include individuals who would provide the perspective of data providers and consumers of the information that will be gathered by the SLEPS. The group was comprised of officers that work in schools, those who supervise officers who work in schools, a representative from the National Association of School Resource Officers (NASRO), experts from the academic field, and representatives from various federal agencies with a vested interest in school safety topics.

In fall 2016, BJS requested and received OMB approval to conduct cognitive testing under the generic clearance agreement (OMB Number 1121-0339) of (1) a SLEPS law enforcement agency (LEA) survey, which included a school resource officer (SRO) rostering form, and (2) a SLEPS SRO survey. The interviews were designed to (1) thoroughly test the survey instruments to ensure question clarity and reliability and (2) assess the willingness of law enforcement agencies to provide rosters of their officers working in schools. In November and December 2016, the project team completed 18 SRO interviews and received only minor suggestions to improve the clarity of the instrument. The project team completed 20 LEA interviews and the findings led BJS to make significant changes to the LEA survey to make it more streamlined and to improve the flow of and navigation through the instrument. During testing, most LEA participants indicated that they would be willing to complete the roster form attached to the LEA survey, but noted that they may not be able to provide the information at the level of detail requested, or that they may need departmental approval to provide the requested information.

The current request is to conduct cognitive testing of the revised LEA survey under the generic clearance agreement (OMB Number 1121-0339). Project staff from RTI International and PERF will cognitively test the revised LEA survey (Attachment A) to ensure improved question clarity and effectiveness. During this round of testing, the participant will also be asked to complete the SRO roster form (Attachment B), so that project staff can evaluate agencies’ ability to provide a list of their officers working in schools. The project staff will review the results of the cognitive testing and roster requests to determine if a small field test of the LEA and SRO surveys should be employed to test the planned data protocol. If a field test is deemed necessary, BJS will submit a request to conduct a field test of the SLEPS surveys and data collection protocol under the generic clearance agreement, tentatively in September 2017. Following the completion of testing, BJS plans to implement the full-scale law enforcement agency portion of the SLEPS by September 2018 and the full-scale school resource officer portion by January 2019.

Request for Cognitive Interviewing

BJS plans to conduct cognitive testing of the revised LEA survey to ensure that the changes made to the survey following the first round of cognitive testing strengthen the clarity of the content and language of the survey questions and that the new order of questions is logical and appropriate. The LEA survey is designed to capture information on all types of officers employed by LEAs that work in schools, including nonsworn personnel. None of the agencies in the first round of testing reported employing any nonsworn officers working in schools. This round of cognitive testing will ensure inclusion of agencies with nonsworn officers in schools to confirm that the questions are clear and appropriate for these types of agencies as well. The cognitive testing of the agency survey will also include asking participants to complete the officer roster form, so that the project team can assess the ability of agencies to provide this information and learn about potential challenges that agencies may encounter in collecting and submitting this information. Twenty (20) law enforcement agencies will be selected to participate in cognitive interviews for the LEA survey and roster form. The intended respondent for the LEA survey is the agency’s chief or another staff member who is knowledgeable about the infrastructure supporting the officers who work in schools, such as the supervising officer.

The project team will develop a preliminary list of agencies for cognitive testing based on a few select criteria to maximize the potential spread of agency characteristics. Agencies involved in the first round of cognitive testing will excluded. The criteria are as follows: (1) agency size, (2) recency of the agency’s SRO program, (3) agency type, and (4) type(s) of officers employed by the agency that work in schools (sworn and/or nonsworn officers). Criteria 1, 2, and 3 are based upon data from the CSLLEA. Agencies will be randomly selected to represent these three criteria. There is currently no known comprehensive source that identifies agencies with nonsworn personnel working in schools; therefore, criteria 4 will be informed by Internet searches, through which the project team will identify agencies that employ this type of officer. The project team will identify approximately 15 agencies that employ nonsworn officers that work in schools and then assign these agencies to one of the other strata based on one or more of the characteristics of agency size, recency of program, or agency type. Once these agencies have been assigned to one of the other strata, the number of randomly selected agencies for that strata will be reduced based on the number of agencies with nonsworn officers representing that strata. The project team is confident that the resulting list of agencies prioritized for cognitive interviewing will be diverse in characteristics and will therefore provide an informative sampling for testing the survey and identifying potential obstacles to the proposed rostering approach.

The project team will use a ‘brute force’ purposive sampling technique. The sampling will focus on obtaining representation within each strata characteristic, regardless of other agency characteristics. The project team will reach out to 15 unique agencies per strata, with the exception of school district police departments. These departments are a limited population and therefore the project team will reach out to 9 of these agencies. In total, the project team will reach out to 99 agencies. If agencies decline to participate, the project team will select additional agencies depending on the needed characteristic from the randomized pool and targeted phone calls will be made as needed to agencies meeting the needed stratification variables.

Sample Stratification Characteristic	Characteristic Values
Agency size	Less than 100 officers
	100 or more officers
Recency of SRO program	Since 2008
	Before 2008
Agency type	Local police
	Sheriff’s office
	School district PD
Sworn and nonsworn officers	Agency employs nonsworn officers

Upon the selection of agencies for cognitive testing, the project team will conduct the recruitment process as follows:

Stage	Type of Contact	Day
Send email invitation to LEA POC on file (phone call if no email is available)	All selected agencies	1
Send hardcopy USPS letter to LEA POC	Non-responding agencies	5
Send reminder email to LEA POC once per week	Non-responding agencies	11+

The project team has developed two letters to send out to non-responding agencies to enhance recruitment by adding another mode of outreach. One letter is for agencies that had an employee participate in the EWG in April 2015. This letter is framed toward an audience that already has a basic knowledge of the project and leverages previous support and participation in the EWG to support recruitment for testing (Attachment D). The second letter is for agencies that did not have an employee participate in the EWG and therefore likely do not possess any background knowledge on the SLEPS (Attachment E).

Once an agency has agreed to participate, the project team will schedule an interview date and time. One week prior to the scheduled interview, the project team will send the participant a consent form (Attachment F), the LEA survey, and the roster form. Participants will be instructed to complete both the survey and roster form. The initial cognitive interviews found that LEAs would want to see the SRO survey before providing a roster of SROs so that they are aware of what will be asked of their officers. Based upon this finding, participants will also receive the SRO survey with a watermark that states ‘do not complete - for informational purposes only.’ Participants will be instructed to return the completed LEA survey and roster form within 2 business days of their scheduled interview via fax or as a zip file. Interviewers will reference this information while conducting the interview. The project team will follow up via email and phone if the completed forms have not been received prior to the interview and reschedule as needed. At the time of the scheduled interview, the interviewer will call the participant and review the consent form with the participant. Upon receiving the participant’s consent, the interviewer will complete the questions at the bottom of the interviewer’s version of the consent form, acknowledging that the interviewer discussed consent with the participant and the participant provided consent to participate in the interview and to have the interview recorded (Attachment G). Participants will not receive any compensation for the interview.

The LEA survey topics include the practices, policies, and procedures that agencies use to oversee and support officers working in schools; the number of personnel who work in schools; training, common roles and responsibilities of officers in schools as defined by agency policies and agreements with schools; and the types of officer activities monitored by the agency. This survey was developed to collect data leading to the production of detailed information on the characteristics of law enforcement in schools, including a number of key estimates, such as the number of various types of officers working in schools and the number of schools served by law enforcement agencies.

BJS requests approval to conduct cognitive testing under the BJS generic clearance to include more rigorous and thorough testing of the revised LEA questionnaire and roster form. This testing is needed to ensure that the form and content of the instrument are understandable and appropriate across the range of targeted respondents. The project team developed a structured protocol for cognitive testing (Attachment H), providing specific probing questions to help determine that the questions are interpreted as intended and are thereby collecting the targeted data. Participants will be asked to take note of any aspects of the instruments that they found to be unclear, any questions or topics that were omitted, or any answer choices or response categories that were missing or insufficient. The project team will review the feedback from the cognitive interviews and revise the survey instrument as necessary.

The roster form attached to the LEA survey requests the agency point of contact to provide a list of officers working in schools which the project team will ultimately use to draw the sample of officers for the SRO survey. The SLEPS will be BJS’ first attempt at an officer survey and other projects have encountered challenges in their efforts to obtain a list of officers. For these reasons, BJS would like to use the cognitive testing effort to gauge agencies’ ability to provide a list of SROs to help the project team determine the best method to obtain the information needed to draw a sample of officers. To facilitate the collection of this information, included in the attached cognitive interviewing protocol for the LEA survey (Attachment H) are scripted probing questions designed to inform the project team about the officer information that agencies are willing to provide and the mode(s) through which they are willing to provide it.

Cognitive testing will provide an opportunity for BJS to calculate more reliable burden estimates for law enforcement agencies to complete the questionnaire and roster. This information will be taken into consideration as BJS continues the development and design of the SLEPS. BJS will also evaluate the option of conducting a field test following review of the results of the cognitive testing, specifically the willingness and ability of agencies to provide officer rosters. If BJS determines that a field test of survey protocols should follow cognitive testing, the information gathered during the cognitive interviewing and the revised questionnaire will be integrated into a generic clearance request to conduct a small field test, tentatively in September 2017. All information gathered from the SLEPS testing efforts will be integrated into the full information clearance package that is expected to be submitted to OMB by June 2018. BJS plans to fully implement the agency portion of the SLEPS by September 2018 and the officer portion by January 2019. All of the dates noted above are planned and pending final approval.

Burden Hours

The burden hour estimates are provided in the following table:

Agency task	Number of agencies	Estimated burden (in minutes)	Total burden hours
Review cognitive interview request and non-response follow-up requests	99	8	13.2
Schedule cognitive interview	20	10	3.3
Review and complete LEA questionnaire and roster form	20	60	20
Conduct cognitive interview	20	60	20
Total burden			56.5 hours

Institutional Review Board

The project team obtained approval from RTI’s IRB to ensure the cognitive testing protocol is compliant with informed consent and data confidentiality standards.

Contact Information

Questions regarding any aspect of this project can be directed to:

Elizabeth Davis

Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 7th Street, NW

Washington, DC 20531

Office Phone: (202) 305-2667

E-mail: [email protected]

Attachments

Attachment A: SLEPS Law Enforcement Agency (LEA) Survey

Attachment B: SLEPS School Resource Officer (SRO) Rostering Form

Attachment C: SLEPS School Resource Officer (SRO) Survey

Attachment D: SLEPS Letter for EWG Participants

Attachment E: SLEPS Letter for Non-EWG Participants

Attachment F: SLEPS LEA Informed Consent - Participant

Attachment G: SLEPS LEA Informed Consent - Interviewer

Attachment H: SLEPS LEA Survey Cognitive Interviewing Protocol

6