CPSAT_SupportingStatementSectionB 251016

Community Policing Self-Assessment Tool (CP-SAT)

Updated October 25, 2015

Part B. Statistical Methods

Part B. Collections of Information Employing Statistical Methods:

The CP-SAT effort will require the employment of only descriptive statistical methods.

B.1. Respondent Universe and Sampling Methods

For the vast majority of COPS Office hiring program grantees, the CP-SAT is intended to be distributed to all relevant employees and as such, the COPS Office will not be determining sampling methods. There are five groups of individuals who are intended to complete the survey within each participating agency: Officers, Supervisors, Command Staff, Civilian Staff, and Community Partners (specific individuals with whom the agency has a recognized partnership, not the general public).

In very large agencies (e.g., 1,200 or greater sworn staff), there is a point at which gaining more survey responses has less statistical benefit than the burden of the additional officers’ time (i.e., the decrease in standard error resulting from the increased responses is negligible). Thus, the COPS Office provides sampling services for agencies over 1,200 sworn staff in which we identify a sample frame of 600 sworn staff using random sampling. To pull the random sample, the agency provides a roster of all sworn staff to the COPS Office’s CP-SAT provider, ICF International. It is requested that agencies provide a roster with the following pieces of information for each staff member: email address, rank, gender, and shift. Agencies wishing to protect the anonymity of their staff may use a unique identifier instead of email addresses. A set of random numbers are generated in Microsoft Excel and assigned to each staff person. The random numbers are sorted and the sample frame is drawn from the top 600 cases. If demographic information (e.g., rank, gender) is provided, the random sample is checked against the proportion of staff in each level of each demographic variable. In the rare case that a random sample deviates significantly from the demographic proportions in the population, a new random sample is drawn. The goal of providing sampling assistance to some CHP grantees is to minimize the burden on the agency in administering the CP-SAT so they can fulfill their grant requirement. Note that some of these agencies still prefer to distribute the survey to all employees.

B2. Procedures for the Collection of Information

Each CHP grantee is contacted by the COPS Office within 1-4 months of returning their signed award and advised of this grant requirement and their future administration date. Agencies are grouped together and assigned in a specific timeframe (referred to as a “wave”) for administration.

The provider then contacts the grantees who are asked to assign a survey administrator (“key contact”) who will assist in the administration of the CP-SAT to participants. Sample survey invitation and reminders are provided, along with a specific timeline for when and to whom they should be sent. The survey administrator’s responsibilities include sending the initial survey invitation, which includes a hyperlink to the online survey, and at least two reminder emails, as necessary. To further increase response rate, the key contact will be responsible for coordinating a pre-notification letter sent from the agency chief executive prior to the survey invitation email. The provider disseminates instructions on the survey process to the agency prior to their assigned administration date, and also provides other materials to support the administration process (e.g., communication templates to provide more background to staff and convey support from agency leadership). The provider also tracks survey response rates and sends an update to the agency regarding the level of participation and responses needed to complete the assessment.

At the completion of the process and if the agency has an 80% response rate, the provider submits a Results Report to the Chief Executive of the law enforcement agency and the self-assessment point of contact , which summarizes the agency’s aggregate assessment results. Although the report does not interpret the data collected, it allows the agency to assess the extent to which community policing has been implemented across various components of the organization and among units and ranks. The agency will be provided descriptive statistics (e.g., number of responses, mean, and standard deviation) for each item on the CP-SAT to further aid in report interpretation and strategic decision making. There are no individual identifiers in the data, and the agency will not be able to link an individual’s data to the participant. The Chief Executive will determine how the report is distributed in his or her agency and community. Agencies that do not reach an 80% response rate will not receive a report from the provider.

Administration of the CP-SAT is repeated toward the end of a grantee’s award period, at which time they will receive a report that displays their scores for both administrations (i.e., pre/post) to document changes in community policing activities over time.

The CP-SAT is administered in an online format using Verint EFM Community Web-based survey software. Verint’s survey hosting environment has been designed with security as a foremost consideration, with features such as 128 bit SSL encryption and redundant firewalls. Responses to the assessment are anonymous. There are no individual identifiers in the data and there is no way to link an individual’s data to their name or email address. Participants will be invited to participate by sending the survey link via email. If the respondent does not have an email account, participants can be directed to a URL address via alternate means (e.g., via agency memo with URL address, access to a common computer lab with the site loaded onto each computer). The confidentiality statement will also appear at the beginning of the survey. All data exported from the Verint secure Web site will be kept in a secured folder.

Emails to participants will include the following statement prominently displayed notifying them of the confidential nature of their surveys: “Your responses to this survey will be kept confidential. There are no individual identifiers in the data that the law enforcement agency will receive, and the agency will not be able to link an individual’s data to their email address. This is not a test and there are no right or wrong answers. Please answer each question honestly.”

To ensure that the data distributed to participating agencies does not compromise respondent confidentiality, agencies must have at least three participant responses for a given item or set of items in order for the summary data to be displayed on the agency report. If only certain items, subsections, or sections have less than three responses, summary data is provided for all of the report with the exception of those specific sections, subsections, or items with fewer than three responses. If multiple respondent types (e.g., command staff, line officers) each have fewer than three respondents, data from those respondents are not segmented and presented, but rather they are only included with the rest of the respondent types as aggregate findings. These confidentiality protections do not apply to very small agencies with four or fewer sworn staff.

Note that the CP-SAT provider cannot protect respondent confidentiality while delivering agency results in very small agencies under the standard survey administration process. To address this, agencies with sworn force levels of four or fewer will be instructed to complete the survey as a group. They will receive specific instructions in how this should occur, along with a reminder that the confidentiality of individual responses will not be protected under this process. Moreover, an individual review of several data items for these agencies will occur to add additional verification that they followed the instructions to complete the survey as a group. This small-agency modified process will ensure that the agency will receive a report summarizing their results, which would not be possible if the surveys were completed independently.

B3. Methods to Maximize Response Rates and Deal with Issues of Non-Response¹

COPS will not be directly involved in the collection of data. However, we currently provide tips for increasing response rate within an agency. These tips include:

• Chief should stress the importance of the self-assessment process and participation in completing the forms.

• Gain support and “buy in” from each of the groups—officers, supervisors, and command staff—as well as police union leadership or other organized labor bodies. Support from these individuals and groups should be sought at the outset of the project.

• Training/orientation for agency personnel who will be asked to complete the assessment tool form will enhance the overall response rate and reduce the problem of incomplete forms being submitted.

• Respondents should be notified in advance that the assessment tool form will be distributed and that they will be given adequate time to complete the form.

• Respondents should be assured that their responses will be handled in a confidential manner.

• Respondents should be given reminders to complete the assessment tool forms.

• Offer to provide the respondents a summary report of the results of the self-assessment process.

• Convey how the results will be used to make positive changes, as well as to promote the work being done by the agency and its staff.

Additionally, the provider monitors response rates of each agency closely and provides agency-specific updates to the self-assessment point of contact for each agency. For agencies with a low response rate, additional email and phone contact is made by the contractor to help answer questions, identify and rectify challenges encountered, and encourage participation. We will strive to get every agency to reach at least an 80% response rate.

While the current average response rate across all agencies that have participated in the CP-SAT is 77.9%, we plan to take additional steps beyond those described above to encourage an even higher response rate in every single agency, as well as across the group of grantees. To support this goal, we will take the following additional actions:

• Add a pre-notification message to our survey procedures. This will include example email language and required timeline for agencies to notify all staff of the upcoming CP-SAT survey invitation.

• Advertise and encourage an 80% response rate to agency leadership personnel. This will include advertising the 80% desired response rate up front, including benefits of reaching that threshold, with agency executives and key contacts. It will also include response rate updates to each agency during administration on their current response rate and a description of the 80% desired response rate, as well as additional tips for increasing their response rate.

As mentioned above, any agency that fails to meet the 80% response rate will not receive their agency’s CP-SAT Results Report.

B4. Tests of procedures or methods to be undertaken

As stated in earlier supporting statements, the project development and research team conducted six pilot tests of the content of the survey and usability of the format in law enforcement agencies across the country. Based on responses and the analysis of findings, improvements were made to both the content and the online format.

B5. Individuals consulted on statistical aspects of the design and organization/persons collecting and analyzing the data.

Beth Heinen, Ph.D.

Manager

Workforce Research and Performance

ICF International

(314) 918-0373

1 During the most recent OMB approval process (about 1.5 years ago), COPS provider, ICF International, conducted a non-response bias analysis (recognizing limitations within the data) and demonstrated no bias. The OMB response was that it was necessary to conduct a non-response bias study for each agency that did not meet 80% or those agencies could not receive a report. COPS provider, ICF, explained that the project lacked the necessary resources to conduct a non-response bias analysis for each agency, and the COPS Office did not have the resource capacity to provide extra funding for the per-agency non-bias analysis. The solution from OMB was that COPS, and its provider, ICF, would not deliver a report to those agencies that fell below 80%. This is the course of action that was followed and outlined in this section and the preceding section, B.2.

COPS also contacted BJS to devise a plan to address non-response bias and BJS drafted a document that explained the concept of non-response. However BJS agreed with the solution to refrain from delivering a report for agencies with a response rate below 80%.

Given the issues discussed above, COPS introduced many different process improvements to increase response rate (e.g., advertising the 80% goal, revising and clarifying communication about the process, adding a pre-survey notification email from the chief executive prior to the survey launch, etc.) to reduce bias and maximize the number of agencies receiving a report, which are all detailed in this section.