SSLLEASTL Supporting Statement Part B

SSLLEASTL SUPPORTING STATEMENT PART B

1. Universe and Respondent Selection

The Survey of State and Local Law Enforcement Agencies Serving Tribal Lands (SSLLEASTL) intends to collect data from state and local law enforcement agencies that are actively working on tribal lands in states where jurisdiction has been transferred to the states under Public Law 83-280 (PL-280). The collection focuses on the sixteen PL-280 states: Alaska, Arizona, California, Florida, Idaho, Iowa, Minnesota, Montana, Nebraska, Nevada, North Dakota, Oregon, South Dakota, Utah, Washington, and Wisconsin. To be eligible for the SSLLEASTL, state and local law enforcement agencies must operate in the United States, be publicly funded, employ at least one full-time sworn officer with full general arrest powers, and provide services on tribal lands. Eligible agencies include general purpose agencies, such as local police departments, sheriff’s offices, and state police departments. In Alaska, the survey will also collect data from regional native non-profit coordinating agencies responsible for village public safety officers (VPSOs) in their area of the state.

The SSLLEASTL used the universe lists from the 2013 Census of State and Local Law Enforcement Agencies (CSLLEA) to identify the agencies located in counties with tribal lands and counties without tribal lands. Only general purpose law enforcement agencies (state police, county police, city/local police) are included in this effort. The universe is divided into four strata. The first stratum is a census of all sixteen state police agencies. The second stratum is a census of all ten VPSO coordinating agencies in Alaska. The third stratum is a census of all 1,215 general purpose agencies in counties containing tribal lands. The fourth stratum is a sample of the 2,151 general purpose agencies in the remaining counties without tribal lands. The data collector conducted a power analysis to determine sufficient sample size to be able to generate reliable statistics for the 2,151 agencies in stratum 4. Using power level of 0.80 and a confidence level of 95%, the data collector concluded that a total of 400 agencies will need to respond from stratum 4 in order to generate reliable estimates of proportions. Assuming a response rate of 80%, a final stratum sample size of 500 agencies will be selected from the 2,151 general purpose law enforcement agencies in stratum 4. The general purpose law enforcement agencies will be sorted by state, county and proximity to tribal lands, and then every nth office will be drawn until a sample of 500 is reached. This should ensure that every state is represented in the sampled stratum. The total sample size for the SSLLEASTL is 1,741 agencies. Responding agencies that indicate they do not provide services on tribal lands in the screening questions of the SSLLEASTL will be asked to return the survey (online or via prepaid mail) and will be recorded as agencies that do not provide services to tribal lands. Table 1 provides the sampling plan for the SSLLEASTL.

Table 1. Sampling plan for Survey of State and Local Law Enforcement Agencies Serving Tribal Lands
	Description	Total number of law enforcement agencies	Number of agencies needed for sample survey
Stratum 1	Census of all state police agencies	16	n/a
Stratum 2	Census of all Alaska VPSO coordinating non-profits	10	n/a
Stratum 3	Census of all general purpose agencies located in counties that contain tribal lands	1,215	n/a
Stratum 4	Sample of general purpose agencies located in counties that do not contain tribal lands	2,151	500

2. Procedures for Collection Information

Data collection for the SSLLEASTL will involve a series of mailings and non-response follow-up activities, emphasizing questionnaire completion via the electronic fillable PDF version of the survey.

Week One

In the first week of data collection, the 1,741 agencies will be mailed a pre-notification letter on BJS letterhead and signed by the BJS Acting Director (see Attachment 1) announcing the start of the SSLLEASTL data collection and requesting that the agency begin gathering the necessary data to respond. This letter will discuss the dearth of information about services provided by state and local law enforcement agencies on tribal lands, the purpose of the data collection, the type of information to be collected, the benefits to the agency in providing these data and the project partners. A letter of support from the International Association of Chiefs of Police (IACP) (Attachment 2) or the National Sheriff’s Association (NSA) (Attachment 3) will be included with the pre-notification letter. The NSA letter of support will be included in the packets sent to sheriff’s offices, with the IACP letter of support included in the packets for all remaining general purpose law enforcement agencies, including the VPSO nonprofit corporations. The pre-notification letter will include the SSLLEASTL toll-free number and e-mail address so the agencies may contact the data collection agent with any questions.

During the initial phases of data collection, project staff will monitor and respond to e-mails and telephone calls to the project toll-free number and e-mail. When prompting calls begin, the telephone prompters will monitor the project toll-free number. Project staff will also continue to monitor and respond to e-mails. All contacts via e-mail and telephone will be documented and reviewed to assess persistent issues that may occur. Any issues will be addressed as needed.

Staff monitoring the project toll-free number and e-mail will be trained on the purpose and background of the project, eligibility of law enforcement offices, screening questions and the survey questions.

Week Two

Approximately one week after the pre-notification letter mailing, a hardcopy survey packet will be sent to all agencies. This packet will include a cover letter on the data collection agent’s letterhead and signed by the data collection agent’s Project Director (Attachment 4), a hardcopy of the SSLLEASTL questionnaire (see Attachments 5 and 6), and a pre-paid business-reply envelope (BRE). Agencies will have the option of completing and returning the hardcopy survey or the fillable PDF survey. The cover letter will include instructions for accessing and downloading the fillable PDF survey and the agency’s unique PIN. All materials will have telephone and e-mail contact information for the data collection agent.

Week Six

Approximately one month after the survey mailing, the data collection agent will send a reminder postcard (see Attachment 7) to all non-responding agencies. This postcard will continue to encourage non-responding agencies to complete the questionnaire as soon as possible. It will contain the SSLLEASTL toll-free number and e-mail address so that respondents may contact the data collection agent with questions or if they need assistance.

Week Eight

The data collection agent will begin calling non-responding agencies approximately two weeks after the reminder post card mailing (Attachment 9). This will serve to verify receipt of materials, answer questions, determine and attempt to resolve potential problems with timely submission, and prompt for questionnaire completion. Data collection specialists will also offer to collect the survey data by telephone if the information requested is readily available and the agency is not able to complete the questionnaire. Surveys completed over the telephone will be recorded by data collection specialists using the fillable PDF survey.

Week Ten

The data collection agent will mail a replacement hardcopy survey to those agencies that have not returned a completed survey. The packet will also include a cover letter on BJS letterhead (see Attachment 10). This letter will convey the importance of the agency’s participation in the SSLLEASTL, encourage timely submission, stress the importance of the information collection for the law enforcement community providing services on tribal lands, and will include further endorsement from IACP and NSA. It will include instructions for submitting the completed questionnaire via the fillable PDF, as well as mailing, faxing or e-mailing the completed questionnaire back to the data collection agent.

Week Twelve

A reminder letter (Attachment 11) will be sent via fax and email to agencies that have not yet completed the SSLLEASTL survey. The use of a fax/e-mail will provide a different form of contact to gain the agency’s attention, stress the importance of the study, and prompt for return of the questionnaire. The customized e-mail will contain the fillable PDF form as an attachment.

Week Sixteen

NORC will send all remaining non-responding agencies a second replacement hardcopy questionnaire via FedEx, Priority Mail, or other express mail delivery. Based on previous experience collecting data from law enforcement agencies, multiple survey mailings are needed to ensure survey response. This package will contain a personalized letter on BJS letterhead (Attachment 12) that urges the agency to respond, a replacement questionnaire, and a BRE to use when returning the questionnaire via mail. The packaging, mode of delivery and speed by which it is delivered will further stress the importance and urgency for response.

Week Twenty

Near the end of the data collection period and only if needed, a Limited Items Questionnaire will be developed cooperatively with BJS. First, the data collector will complete an item response report. The items with the lowest response rates will be considered for exclusion from the Limited Items Questionnaire. The Limited Items Questionnaire will be a reduced set of questions deemed most critical for analysis. If necessary, it will be sent to those agencies that have not completed the full survey.

Week Twenty-four

During the final weeks of data collection, the data collection agent will mail a postcard or send an e-mail alerting agencies of the scheduled data collection end date. A last chance contact (Attachment 13) has been implemented on previous studies, and serves to motivate non-responding agencies who had failed to complete and return the survey.

Verifying and validating the submitted data

Following OMB’s finalization of the survey instruments, the survey forms will be converted to fillable PDFs. Before the forms are distributed to the sampled agencies, simulated fillable PDFs will be created and extracted into Microsoft Excel to check the accuracy of the extraction. Microsoft Excel will be used to check the data quality. Once quality is assured, the data will be extracted from the PDFs directly to SPSS to avoid introducing any error through a multiple-stage extraction process (e.g., avoid possible errors extracting from PDF – Excel – SPSS). In order to test the accuracy of the data extraction process, data on the simulated fillable PDFs will be compared to the extracted data to confirm that data in the extracted file exactly match the responses recorded in the fillable PDFs.

Once data begin to be received from respondents, the data collector will continue to compare the extracted Excel data to the PDF surveys. Early surveys will all be checked for accuracy and completeness. After the data extraction process is checked, the data will be extracted from the PDFs to SPSS. As data collection progresses, a sample of surveys will be extracted to Excel and tested to ensure that later extracts are as accurate and complete as earlier extracts. If needed, revisions will be made to the fillable PDF documents to ensure that data are correctly exported. Results from these tests will be used to guide data extraction during the data collection stage and quality control procedures. The following is a summary of the data quality assurance steps that will be implemented during the data collection and processing period –

Data review: As surveys are returned, they will be reviewed for completeness. The data collection agent will aim to reconcile missing or erroneous data through a manual edit of each questionnaire. In collaboration with BJS, the data collection agent will develop a list of manual edits that can be completed by referring to other data provided by the respondent on the survey instrument. For example, if a screening question was left blank by the respondent but the follow-up questions were completed, a manual edit could be made to indicate the intended positive response to the screening question. Through this process, the number of cases/questions requiring clarification or retrieval of missing information can be minimized.

Data retrieval: When it is determined that additional data are needed because of errors, inconsistencies, or missing data, the data collection agent will contact (Attachment 14) the data provider for clarification as soon as possible to the date of submission of the questionnaire. Throughout the process, the data collection agent will document the questions needing retrieval (e.g. missing or inconsistent data elements), request clarification on the provided information, obtain values for missing data elements and examine any other issues related to the submission. Data retrieval outreach will occur via telephone and e-mail.

Data entry: Surveys that are completed (i.e. with no errors by the respondent or after retrieval) will be batched and extracted to an Excel file to be checked for accuracy. At the beginning of data collection, the data collection agent will perform 100% confirmation of the extracted data for the first 20 forms received using the fillable PDF format. Once the process is verified, that data will be extracted from the PDFs directly to SPSS. As data collection proceeds, the data collector will randomly select 10% of cases from each batch to compare the responses in the PDF to the extracted data in Excel. The identifier PIN for each record will be checked against a master list of PIN numbers for the batch in order to confirm that all expected surveys have been extracted and stored appropriately. SPSS and Excel files will also be checked to confirm that all variables have been correctly exported.

Following completion of data collection, all SPSS files extracted from the fillable PDF forms will be merged into one SPSS file. The PIN identifiers will be reviewed to ensure that all expected cases are in this merged file. The assignment of response labels, question text, and other metadata will be applied to the SPSS file. The original PDFs, paper surveys, SPSS extracted data and Excel extracted data (if applicable) will be stored in their original forms and delivered to BJS at the end of the project.

Second data review: Once the data have been entered into the database, they will be made available to BJS via the DOJ-approved Box JEFS site. The data collection agent will make data available to BJS on a biweekly basis. To confirm that editing rules are being followed, the data collection agent will review frequencies for the entered data on a biweekly basis. All issues will be investigated and resolved.

Thank you responses: After verification, the respondent will be immediately sent an e-mail to confirm the receipt of the data and to thank them for their participation (Attachment 15). The email serves to thank the respondent, to inform them that their data has been included in the SSLEASLTL, and to provide estimated publication dates of the report. In the event the respondent did not provide a valid e-mail address, the thank-you e-mail will be sent via standard mail.

Adjusting for Non-Response

Although it is anticipated that the steps mentioned above will result in participation from a large number of agencies, there will always be a small percentage that do not complete the survey in a timely manner. For these respondents, offering a Limited Items Questionnaire will reduce unit non-response. While not ideal, reducing the number of items and respondent burden will allow for collection of the most important data items from those respondents who previously could not complete the entire survey because of time and/or reporting constraints and provide data to support a nonresponse bias study. The data collector will complete an item response rate report. Questions that are least likely to be answered by respondents will be considered for elimination from the limited item questionnaire.

After conducting a nonresponse bias analysis, the data collector will employ imputation or weighting adjustments to reduce bias and to provide estimates. While high unit and item response is anticipated, agencies may leave items blank due to lack of access to the information or the fact that the data are not being recorded by the law enforcement agency’s records management system. We anticipate that the inclusion of an “unknown” response option will reduce the item nonresponse; however, “unknown” responses are essentially missing data. Based on the results of the nonresponse bias analysis, the data collection agent plans to use the hot-deck imputation method for missing or unknown responses to ensure complete, quality data files. If the nonresponse bias analysis does not support the hot deck method (i.e., the items missing are not missing at random), the data collection agent’s statisticians will work with BJS statisticians to determine the most appropriate methods of imputing the missing data. Imputed values will be merged into the data file (and flagged) prior to delivery to BJS and subsequent archiving. Traditional non-response weighting adjustments will be used to compensate for unit non-response.

The final response rate will likely be a nonrandom sample of the study population due to differential response rates across subpopulations. Agency characteristics that may affect response include: state vs. local agency, agency size, agency location, agency resources, and other factors. To reduce potential nonresponse bias, NORC proposes to develop an analysis weight for each respondent through a weighting adjustment procedure. The base weight is assumed to be 1 for all sample members where the survey has a 100% response rate. As response rates decline, responding agencies will be weighted to be more than 1 based on adjustment cells and non-responding agencies will be weighted as 0. The adjustment cells will be defined by agency type, population size, and possibly other characteristics that are correlated with nonresponse. Agency type may be defined as state vs. local; size may be defined by the population size served by the law enforcement agency or the number of sworn officers in the agency; and location may be defined as urban vs. rural. The final definitions of these variables and the adjustment cells will be determined collaboratively with BJS based on the results of a nonresponse bias analysis. To avoid introducing unnecessary weight variation, we will require that each adjustment cell contain at least 20 cases. The final weights may be interpreted as the number of eligible agencies that each responding agency represents in the target population. Therefore, the total weight represents the estimated size of the target population, both overall and by the adjustment cells.

3. Methods to Maximize Response Rates

As a new data collection effort, historical response rates for state and local law enforcement agencies serving tribal lands are not available. For the 2004 and 2008 CSLLEA, BJS obtained response rates of over 99%. The response rate for the 2014 CSLLEA was 80%. BJS expects a response rate of at least 80% for the SSLLEASTL.

As described in the data collection section, the data collection agent will implement a series of data collection strategies to encourage timely and complete participation. Law enforcement agencies will have several modes in which to respond to the SSLLEASTL, including mail, fax, e-mail, or by telephone. These strategies also incorporate a variety of contacts to encourage response, such as mailing, fax, e-mail and telephone.

A letter of support from the International Association of Chiefs of Police (IACP) and the National Sheriffs’ Association (NSA) will accompany the initial survey mailing and both organizations will communicate with their membership about the survey, stressing the importance of the data collection and need for the information.

The data collection agent has opted to utilize a ‘fillable’ or interactive PDF document for electronic survey completion in lieu of providing respondents a web survey. The primary reason for pursuing this option is to reduce cost of survey administration and burden on the respondents. During the pilot survey, respondents expressed the desire to complete the survey electronically, but not necessarily while connected to the Internet or online. Recent administration of the 2013 CSLLEA found that providing a web-only option increased the amount of time between receipt of the survey and submission of the survey, reducing the initial response rates when compared to the 2008 CSLLEA. Later, the CSLLEA included a web option and a hardcopy option, which increased the response rate. Based on the sample size for this survey, we expect that the web survey would be minimally used by respondents. Additionally, providing the fillable PDF form allows respondents the flexibility of sharing a partially-completed form with others in their office via email or hard copy, as well as returning the survey electronically or by mail as preferred by the agency.

Agencies will be able to contact the data collection agent via e-mail or a toll-free telephone number. This will allow respondents to contact the data collection agent via their preferred methods at no additional cost to them. Further, respondents will be able to email the data collection agent at any time and retrieve the responses when it is most convenient to them.

4. Testing of procedures

Pilot test methodology

Working with the IACP and the NSA, the data collection agent recruited 9 state and local law enforcement agencies that provide services on tribal lands. These agencies were in Alaska, Arizona, California (2 agencies), Idaho, Montana (did not complete debriefing), Oklahoma (non-responder), Oregon, and Washington. Eight of the nine agencies completed the survey and seven completed the phone debriefing after the pilot survey. Each pilot agency was contacted and the SSLLEASTL project was described by telephone and/or e-mail before obtaining agreement that the agency would participate in the SSLLEASTL pilot. The data collection agent then sent the pilot SSLLEASTL survey to each law enforcement agency with a cover letter on BJS letterhead (Attachment 16) and a pre-addressed FedEx return label/envelope. Pilot respondents were asked to complete the survey within 4 weeks of the mailing. The pilot survey also contained a short series of questions to elicit feedback on the survey.

The data collection agent made reminder telephone calls and sent reminder e-mails to non-responding agencies to encourage return of the survey and answer questions. As completed surveys were returned to the data collection agent, the data provider was contacted and a debriefing telephone call scheduled. During the one hour debriefing call, survey questions and response categories were reviewed for clarity and completeness (Attachment 17). Pilot respondents were also asked about the availability of the requested data and burden. The pilot responses culminated in a pilot survey report (Attachment 18).

Use of pilot feedback

Based on feedback from the pilot, questions were modified to include an “unknown” option, allowing agencies to provide estimates of counts and to select calendar year or fiscal year for providing caseload data. Agencies will be sent a hardcopy survey as one of the initial mailings and the cover letter accompanying the initial and replacement survey mailings will request that the survey be completed and returned within four weeks. Pilot agencies indicated that this would be helpful in assigning priority to the survey. A fillable PDF form will be created to facilitate forwarding of the partially completed survey to agency staff. This will also facilitate saving of the file by an agency for future reference, reduce the need for the agency to print the survey form and will facilitate the submission of the completed form by e-mail.

5. Contacts for Statistical Aspects and Data Collection

The Bureau of Justice Statistics, U. S. Department is the funding agency for the SSLLEASTL. NORC is the data collection agent.

1. BJS contacts:

Suzanne M. Strong

U.S. Department of Justice

Bureau of Justice Statistics

810 Seventh Street, N.W.

Washington, D.C. 20531

[email protected]

Howard Snyder

U.S. Department of Justice

Bureau of Justice Statistics

810 Seventh Street, N.W.

Washington, D.C. 20531

[email protected]

2. NORC contact:

David Herda

Senior Survey Director II

NORC at the University of Chicago

55 East Monroe Street, 20^th floor

Chicago, IL 60603

Attachments:

Attachment 01- SSLLEASTL -16- Law Enforcement Pre-notification Letter

Attachment 02- SSLLEASTL -16- Law Enforcement IACP Letter of Support

Attachment 03- SSLLEASTL -16- Law Enforcement NSA Letter of Support

Attachment 04- SSLLEASTL -16- Law Enforcement Initial Questionnaire Cover Letter

Attachment 05- SSLLEASTL -16- Law Enforcement Questionnaire for Alaska

Attachment 06- SSLLEASTL -16- Law Enforcement Questionnaire Remaining PL-280 States

Attachment 07- SSLLEASTL -16- Law Enforcement Reminder Postcard Text

Attachment 08- SSLLEASTL -16- Law Enforcement Telephone Contacting Script

Attachment 09- SSLLEASTL -16- Law Enforcement Telephone Voicemail Script

Attachment 10- SSLLEASTL -16- Law Enforcement Replacement Questionnaire Cover Letter

Attachment 11- SSLLEASTL -16- Law Enforcement Reminder Letter

Attachment 12- SSLLEASTL -16- Law Enforcement Second Replacement Questionnaire Cover

Letter

Attachment 13- SSLLEASTL -16- Law Enforcement Last Chance Postcard/E-mail

Attachment 14- SSLLEASTL -16- Law Enforcement Telephone Retrieval Script

Attachment 15- SSLLEASTL -16- Law Enforcement Thank You E-mail

Attachment 16- SSLLEASTL -16- Law Enforcement Pilot Cover Letter

Attachment 17- SSLLEASTL -16- Law Enforcement Pilot Debriefing Script

Attachment 18- SSLLEASTL -16- Law Enforcement Pilot Survey Report

9