CSLLEASTL SUPPORTING STATEMENT PART B
Universe and Respondent Selection
The Census of State and Local Law Enforcement Agencies Serving Tribal Lands (CSLLEASTL) intends to collect data from state and local law enforcement agency that are actively working on tribal lands. To be eligible for the CSLLEASTL, state and local law enforcement agencies must operate in the United States, be publicly funded, employ at least one full-time sworn officer with full general arrest powers, and provide services on tribal lands. Eligible agencies may include several types of law enforcement agencies, such as local police departments, sheriff’s offices, state police departments, constables and special jurisdiction agencies (e.g. airport or campus police).
All state and local law enforcement agencies indicating in response to the 2014 Census of State and Local Law Enforcement Agencies (CSLLEA) that their agency provided law enforcement services on tribal lands in 2014 will be included on the CSLLEASTL universe list. Added to this list of tribal-serving agencies will be all of the state and local law enforcement agencies identified by the respondents to the 2014 National Survey of Tribal Court Systems (NSTCS) as providing services on tribal lands. It is anticipated that some unknown percentage of tribal-serving agencies will not be identified by either the CSLLEA or the NSTCS. To capture agencies not identified by either CSLLEA or NSTCS, the data collector has mapped all tribal lands in the U.S. and identified the counties contiguous to those lands. Any law enforcement agency functioning in those contiguous counties (including state law enforcement) that did not respond to the CSLLEA or that was not identified in NSTCS will be included as valid members of the CSLLEASTL universe. That is, any agency located in counties contiguous to tribal lands that did not respond to the CSLLEA or that were not identified in the NSTCS will follow the same schedule of data collection outlined below. Agencies that indicate they do not provide services on tribal lands in the screening questions of the CSLLEASTL will be asked to return the survey (online or via prepaid mail) and will be deemed out of scope.
Given the current response rate to the CSLLEA, the data collector used mapping technologies to identify all counties contiguous to federally registered tribal lands. The data collector then identified all law enforcement agencies located in those counties and estimates a total response universe of 1,657 law enforcement agencies that could provide services to tribal lands. Of those agencies, 1,105 have not responded to the CSLLEA, meaning that we do not know whether these 1,105 agencies provide services to tribal lands.
Based on responses to the 2014 CSLLEA, the data collector estimates that approximately 740 agencies will be in the CSLLEASTL universe. Agency contact information from the 2014 CSLLEA will be utilized for the CSLLEASTL. The 2014 CSLLEA contact information includes agency name, contact person, mailing address, telephone number, fax number, if available, and e-mail address, if available.
Procedures for Collection Information
Data collection for the CSLLEASTL will follow two tracks: one for the known universe members and one for the unknown universe members (i.e., law enforcement agencies located geographically contiguous to tribal lands, but did not respond to the CSLLEA or were not identified as providing services to tribal lands in the NSTCS). Eventually, the two tracks will align and similar procedures will be implemented for both tracks. The data collection will involve a series of mailings and non-response follow-up activities, emphasizing questionnaire completion via the electronic fillable PDF version of the survey.
Week One – Unknown Universe
In the first week of data collection, the 1,105 agencies that have not responded to the CSLLEA but are contiguous to tribal lands will be mailed a postcard (Attachment 1) with the first 3 screening questions from the CSLLEASTL questionnaire. The postcard will provide a brief explanation of the CSLLEASTL and allows respondents to mail, fax or scan and email the postcard back to the data collector. The postcard also includes the e-mail address and toll-free phone number of the data collector for the agency to contact with any questions. Agencies that indicate they do not provide services to tribal lands will be removed from the universe. Agencies that indicate they do provide services will be immediately moved into the known universe track and mailed the pre-notification letter describing the study with the correct supporting letter (see week one – known universe).
Week One – Known Universe
In the first week of data collection, all agencies will be mailed a pre-notification letter on BJS letterhead and signed by the BJS Acting Director (see Attachment 2) announcing the start of the CSLLEASTL data collection and requesting that the agency begin gathering the necessary data to respond. This letter will discuss the dearth of information about services provided by state and local law enforcement agencies on tribal lands, the purpose of the data collection, the type of information to be collected, the benefits to the agency in providing these data and the project partners. A letter of support from the International Association of Chiefs of Police (IACP) (Attachment 3) or the National Sheriff’s Association (NSA) (Attachment 4) will be included with the pre-notification letter. The NSA letter of support will be included in the packets sent to sheriff’s offices, with the IACP letter of support included in the packets for all other law enforcement agencies. The pre-notification letter will include the CSLLEASTL toll-free number and e-mail address so the agencies may contact the data collection agent with any questions.
During the initial phases of data collection, project staff will monitor and respond to e-mails and telephone calls to the project toll-free number and e-mail. When prompting calls begin, the telephone prompters will monitor the project toll-free number. Project staff will continue to monitor and respond to e-mails. All contacts via e-mail and telephone will be documented and reviewed to assess persistent issues that may occur. Any issues will be addressed as needed.
Staff monitoring the project toll-free number and e-mail will be trained on the purpose and background of the project, eligibility of law enforcement offices, screening questions and the survey questions.
Week Two – Known Universe
Approximately one week after the pre-notification letter mailing, a hardcopy survey packet will be sent to all agencies. This packet will include a cover letter on the data collection agent’s letterhead and signed by the data collection agent’s Project Director (Attachment 5), a hardcopy of the CSLLEASTL questionnaire (see Attachment 6), and a pre-paid business-reply envelope (BRE). Agencies will have the option of completing and returning the hardcopy survey or the fillable PDF survey. The cover letter will include instructions for accessing and downloading the fillable PDF survey and the agency’s unique PIN. All materials will have telephone and e-mail contact information for the data collection agent.
Week Three – Unknown Universe
The data collector will begin telephone outreach (see Attachment 7) to all non-responding unknown universe members. Telephone outreach will include the screener questions. Agencies that indicate they do not provide services to tribal lands will be removed from the universe. Agencies that indicate they do provide services will be immediately moved into the known universe track and mailed the pre-notification letter describing the study with the correct supporting letter (see week one – known universe). Approximately one week later, the respondent will receive the complete pack (see week two – known universe).
Week Six – Unknown Universe
If no response has been received from the unknown agencies, the data collector will mail all remaining non-responding unknown universe agencies the hardcopy survey packet. This packet will include a cover letter on the data collection agent’s letterhead and signed by the data collection agent’s Project Director (Attachment 5), a hardcopy of the CSLLEASTL questionnaire (see Attachment 6), and a pre-paid business-reply envelope (BRE). Agencies will have the option of completing and returning the hardcopy survey or the fillable PDF survey. The cover letter will include instructions for accessing and downloading the fillable PDF survey and the agency’s unique PIN. All materials will have telephone and e-mail contact information for the data collection agent.
Week Six – Known Universe
Approximately one month after the survey mailing, the data collection agent will send a reminder postcard (see Attachment 8) to all non-responding agencies. This postcard will continue to encourage non-responding agencies to complete the questionnaire as soon as possible. It will contain the CSLLEASTL toll-free number and e-mail address so that respondents may contact the data collection agent with questions or if they need assistance.
Week Eight – Known and Unknown Universe
The data collection agent will begin telephoning non-responding agencies approximately two weeks after the reminder post card mailing (known universe) or the hardcopy survey packet (unknown universe) (Attachments 9 and 10). This will serve to verify receipt of materials, answer questions, determine and attempt to resolve potential problems with timely submission, and prompt for questionnaire completion. Data collection specialists will also offer to complete the survey by telephone if the information requested is readily available and the agency is not able to complete the questionnaire. Surveys completed over the telephone will be recorded by data collection specialists using the fillable PDF survey.
Week Ten
The data collection agent will mail a replacement hardcopy survey to those agencies that have not returned a completed survey. The packet will also include a cover letter on BJS letterhead (see Attachment 11). This letter will convey the importance of the agency’s participation in the CSLLEASTL, encourage timely submission, stress the importance of the information collection for the law enforcement community providing services on tribal lands, and will include further endorsement from IACP and NSA. It will include instructions for submitting the completed questionnaire via the fillable PDF, as well as mailing, faxing or e-mailing the completed questionnaire back to the data collection agent.
Week Twelve
A reminder letter (Attachment 12) will be sent via fax and email to agencies that have not yet completed the CSLLEASTL survey. The mass fax/e-mail will provide a different form of contact to gain the agency’s attention, stress the importance of the study, and prompt for return of the questionnaire. The customized e-mail will contain the fillable PDF form as an attachment.
Week Sixteen
NORC will send all remaining non-responding agencies a second replacement hardcopy questionnaire via FedEx, Priority Mail, or other express mail delivery. Based on previous experience collecting data from law enforcement agencies, multiple survey mailings are needed to ensure survey response. This package will contain a personalized letter on BJS letterhead (Attachment 13) that urges the agency to respond, a replacement questionnaire, and a BRE to use when returning the questionnaire via mail. The packaging, mode of delivery and speed by which it is delivered will further stress the importance and urgency for response.
Week Twenty
Near the end of the data collection period and only if needed, a Limited Items Questionnaire will be developed cooperatively with BJS. First, the data collector will complete an item response report. The items with the lowest response rates will be considered for exclusion from the Limited Items Questionnaire. The Limited Items Questionnaire will be a reduced set of questions deemed most critical for analysis. If necessary, it will be sent to these agencies that have not completed the full survey.
Week Twenty-four
During the final weeks of data collection, the data collection agent will mail a postcard or send an e-mail alerting agencies of the scheduled data collection end date. A last chance contact (Attachment 14) has been implemented on previous studies, and serves to motivate non-responding agencies who had failed to complete and return the survey.
Verifying and validating the submitted data
Following the finalization of the survey, the survey forms will be converted to fillable PDFs. The data collector created a testing plan with different scenarios in which to enter and extract responses from the fillable pdf. Responses will be entered into the fillable PDF and compared to the extracted data to confirm that data in the extracted file exactly matches the responses recorded in the fillable PDF. Comparisons will be made between the extracted data and the PDF surveys from these scenarios to check for accuracy of the extracted responses, completeness, and ensure that each question is being appropriately exported and associated with the appropriate variable for a particular question. Prior to contacting the respondent, the data collection agent will attempt to address data inconsistencies via BJS-approved editing specifications. Staff will also ensure that responses fall within the proper coding schemes specified by BJS. If needed, revisions will be made to the fillable PDF documents to ensure that data is correctly exported. Results from these tests will be used to guide data extraction during the data collection stage and quality control procedures. The following is a summary of the data quality assurance steps that will be implemented during the data collection and processing period –
Data review: As surveys are returned, they will be reviewed for completeness following the same protocols that are used for paper hardcopy surveys and retrieval will be conducted as needed. The data collection agent will aim to reconcile missing or erroneous data through a manual edit of each questionnaire. In collaboration with BJS, the data collection agent will develop a list of manual edits that can be completed by referring to other data provided by the respondent on the survey instrument. For example, if a screening question was left blank by the respondent but the follow-up questions were completed, a manual edit could be made to indicate the intended positive response to the screening question. Through this process, the number of cases/questions requiring clarification or retrieval of missing information can be minimized.
Data retrieval: When it is determined that additional data are needed because of errors, inconsistencies, or missing data, the data collection agent will contact (Attachment 15) the data provider for clarification as soon as possible to the date of submission of the questionnaire. Throughout the process, the data collection agent will document the questions needing retrieval (e.g. missing or inconsistent data elements), request clarification on the provided information, obtain values for missing data elements and examine any other issues related to the submission. Data retrieval outreach will occur via telephone and e-mail.
Data entry: Surveys that are completed (i.e. with no errors by the respondent or after retrieval) will be batched and extracted to an Excel file. At the beginning of data collection, the data collection agent will perform 100% confirmation of the extracted data for the first 20 forms received using the fillable PDF format. As data collection proceeds, the data collector will randomly select 10% of cases from each batch to compare the responses in the pdf to the extracted data. The identifier PIN for each record will be checked against a master list of PIN numbers for the batch in order to confirm that all expected surveys have been extracted and stored appropriately. Files will also be checked to confirm that all variables have been exported and that Excel files will be saved using the .xlsx file format that will ensure that no column or row limits are met.
Following completion of data collection, all Excel files extracted from the fillable PDF forms will be merged into one Excel file. The PIN identifiers will be reviewed to ensure that all expected cases are in this merged file. This Excel file with the PDF responses will be merged with the Excel file with the hardcopy survey data. The combined file will be converted to SPSS. The assignment of response labels, question text, and other metadata will be applied to the SPSS file.
Second data review: Once the data have been entered into the database, they will be made available to BJS via an SFTP site. The data collection agent will make data available to BJS on a biweekly basis. To confirm that editing rules are being followed, the data collection agent will review frequencies for the entered data on a biweekly basis. All issues will be investigated and resolved.
Thank you: After verification, the respondent will be sent an e-mail to confirm the receipt of the data and to thank them for their participation (Attachment 16). The email serves to thank the respondent, to inform them that their data has been included in the CSLLEASTL, and to provide estimated publication dates of the report. The e-mail will be sent via standard mail in the event the respondent does not provide a valid email address.
Adjusting for Non-Response
Although it is anticipated that the steps mentioned above will result in participation from a large number of agencies, there will always be a small percentage that do not complete the survey in a timely manner. For these respondents, offering a Critical Item Questionnaire will reduce unit non-response. While not ideal, reducing the number of items and respondent burden will allow for collection of the most important data items from those respondents who previously could not complete the entire survey because of time and/or reporting constraints.
If necessary, the data collection agent will employ imputation or weighting adjustments to reduce non-response bias in the estimates. While high item response is anticipated, agencies may leave items blank due to lack of access to the information or the fact that the data are not being recorded by the law enforcement agency’s records management system. The data collection agent plans to primarily use the hot-deck method for imputation to ensure complete, quality data files. The data collection agent will work collaboratively with BJS on an imputation plan for the CSLLEASTL that is deemed both cost-efficient and suitable for the project and implement the plan. Imputed values will be merged into the data file (and identified) prior to delivery to BJS and subsequent archiving. Traditional non-response weighting adjustments will be used to compensate for unit non-response.
CSLLEASTL is intended to be a census. However, the final respondent sample is likely to be a nonrandom sample of the study population due to differential response rate across subpopulations. Agency characteristics that may affect response rate include: state vs. local agency, agency size, agency location, agency resources, among other factors. To reduce potential nonresponse bias, NORC proposes to develop an analysis weight for each respondent through a two-step weighting adjustment procedure. The sample base weight is assumed to be 1 for all sample members. The first step adjusts the weight for unknown eligibility to compensate for cases whose eligibility status is unknown at the end of data collection. Through this adjustment, the weight assigned to eligibility unknown cases is transferred to eligibility known cases within each adjustment cell.
The second step adjusts the weight further for nonresponse to compensate for known eligible cases that fail to respond to the census. Through this adjustment, the weight carried by nonrespondents is transferred to final respondents within each adjustment cell. For both adjustments, the adjustment cells will be defined by location in a PL-280 or non-PL-280 state, agency type, population size, and possibly other characteristics that are correlated with unknown eligibility and nonresponse. Agency type may be defined as state vs. local; size may be defined by the population size of the county in which the agency is located; and location may be defined as urban vs. rural. The final definitions of these variables and the adjustment cells will be determined collaboratively with BJS based on the results of a nonresponse bias analysis. To avoid introducing unnecessary weight variation, we will require that each adjustment cell contain at least 20 cases. The final weight may be interpreted as the number of eligible agencies that each responding agency represents in the target population. Therefore, the total weight represents the estimated size of the target population, both overall and by the adjustment cells.
Methods to Maximize Response Rates
The standard BJS requirement for administrative surveys is 90%. As a new data collection effort, historical response rates for state and local law enforcement agencies serving tribal lands are not available. For the 2004 and 2008 CSLLEA, BJS obtained response rates of over 99%. The response rate for the 2014 CSLLEA was 80%.
As described in the data collection section, the data collection agent will implement a series of data collection strategies to encourage timely and complete participation. Law enforcement agencies will have several modes in which to respond to the CSLLEASTL, including mail, fax, e-mail, or by telephone. These strategies also incorporate a variety of contacts to encourage response, such as mailing, fax, e-mail and telephone.
A letter of support from the International Association of Chiefs of Police (IACP) and the National Sheriffs’ Association (NSA) will accompany the initial survey mailing and both organizations will communicate with their membership about the survey, stressing the importance of the data collection and need for the information.
The data collection agent has opted to utilize a ‘fillable’ or interactive PDF document for electronic survey completion in lieu of providing respondents a web survey. The primary reason for pursuing this option is to reduce cost of survey administration and burden on the respondents. During the pilot survey, respondents expressed the desire to complete the survey electronically, but not necessarily while connected to the Internet or online. Recent administration of the CSLLEA found that providing a web option reduced the rate and speed of response as compared to prior rounds of data collection for the same survey. Based on sample size estimates for this survey, we expect that the web survey would be minimally used by respondents. Additionally, providing the fillable PDF form allows respondents the flexibility of sharing a partially-completed form with others in their office via email or hard copy, as well as returning the survey electronically or by mail as preferred by the agency.
Agencies will be able to contact the data collection agent via e-mail or a toll-free telephone number. This will allow respondents to contact the data collection agent via their preferred methods at no additional cost to them. Further, respondents will be able to email the data collection agent at any time and retrieve the responses when it is most convenient to them.
Testing of procedures
Pilot test methodology
Working with the IACP and the NSA, the data collection agent recruited 9 state and local law enforcement agencies that provide services on tribal lands. These agencies were in Alaska, Arizona, California (2 agencies), Idaho, Montana (did not complete debriefing), Oklahoma (non-responder), Oregon, and Washington. Eight of the nine agencies completed the survey and seven completed the phone debriefing after the pilot survey. Each pilot agency was contacted and the CSLLEASTL project was described by telephone and/or e-mail before obtaining agreement that the agency would participate in the CSLLEASTL pilot. The data collection agent then sent the pilot CSLLEASTL survey to each law enforcement agency with a cover letter on BJS letterhead (Attachment 17) and a pre-addressed FedEx return label/envelope. Pilot respondents were asked to complete the survey within 4 weeks of the mailing. The pilot survey also contained a short series of questions to elicit feedback on the survey.
The data collection agent made reminder telephone calls and sent reminder e-mails to non-responding agencies to encourage return of the survey and answer questions. As completed surveys were returned to the data collection agent, the data provider was contacted and a debriefing telephone call scheduled. During the one hour debriefing call, survey questions and response categories were reviewed for clarity and completeness (Attachment 18). Pilot respondents were also asked about the availability of the requested data and burden. The pilot responses culminated in a pilot survey report (Attachment 19).
Use of pilot feedback
Based on feedback from the pilot, questions were modified to include an “unknown” option, allows agencies to provide estimates of counts and allows agencies to select calendar year or fiscal year for providing caseload data. Agencies will be sent a hardcopy survey as one of the initial mailings and the cover letter accompanying the initial and replacement survey mailings will request that the survey be completed and returned within four weeks. Pilot agencies indicated that this would be helpful in assigning priority to the survey. A fillable PDF form will be created to facilitate forwarding of the partially completed survey to agency staff. This will also facilitate saving of the file by an agency for future reference, reduce the need for the agency to print the survey form and will facilitate the submission of the completed form by e-mail.
Contacts for Statistical Aspects and Data Collection
The Bureau of Justice Statistics, U. S. Department is the funding agency for the CSLLEASTL. NORC is the data collection agent.
BJS contacts:
Suzanne M. Strong
U.S. Department of Justice
Bureau of Justice Statistics
810 Seventh Street, N.W.
Washington, D.C. 20531
Howard Snyder
U.S. Department of Justice
Bureau of Justice Statistics
810 Seventh Street, N.W.
Washington, D.C. 20531
NORC contact:
David Herda
Senior Survey Director II
NORC at the University of Chicago
55 East Monroe Street, 20th floor
Chicago, IL 60603
Attachments:
Attachment 01- CSLLEASTL -15- Law Enforcement Unknown Universe Screener Postcard
Attachment 02- CSLLEASTL -15- Law Enforcement Pre-notification Letter
Attachment 03- CSLLEASTL -15- Law Enforcement IACP Letter of Support
Attachment 04- CSLLEASTL -15- Law Enforcement NSA Letter of Support
Attachment 05- CSLLEASTL -15- Law Enforcement Initial Questionnaire Cover Letter
Attachment 06- CSLLEASTL -15- Law Enforcement Questionnaire
Attachment 07- CSLLEASTL -15- Law Enforcement Unknown Universe Telephone Contacting Script
Attachment 08- CSLLEASTL -15- Law Enforcement Reminder Postcard Text
Attachment 09- CSLLEASTL -15- Law Enforcement Telephone Contacting Script
Attachment 10- CSLLEASTL -15- Law Enforcement Telephone Voicemail Script
Attachment 11- CSLLEASTL -15- Law Enforcement Replacement Questionnaire Cover Letter
Attachment 12- CSLLEASTL -15- Law Enforcement Reminder Letter
Attachment 13- CSLLEASTL -15- Law Enforcement Second Replacement Questionnaire Cover
Letter
Attachment 14- CSLLEASTL -15- Law Enforcement Last Chance Postcard/E-mail
Attachment 15- CSLLEASTL -15- Law Enforcement Telephone Retrieval Script
Attachment 16- CSLLEASTL -15- Law Enforcement Thank You Letter
Attachment 17- CSLLEASTL -15- Law Enforcement Pilot Cover Letter
Attachment 18- CSLLEASTL -15- Law Enforcement Pilot Debriefing Script
Attachment 19- CSLLEASTL -15- Law Enforcement Pilot Survey Report
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-23 |