CTLEA OMB Part B

Part B. Collection of Information Employing Statistical Methods

1. Respondent Universe and Selection

Defining the tribal LE agency universe

BJS and its data collection agent conducted an extensive process to develop the universe list for the 2019 Census of Tribal Law Enforcement Agencies (CTLEA). Since the CTLEA is the first census of tribal LE agencies operating in the U.S., the first task was to develop agency eligibility criteria. BJS and its data collection agents (NORC at the University of Chicago and the International Association of Chiefs of Police [IACP]) collaborated with federal, state, and tribal LE agencies in the lower 48 states and Alaska to develop these criteria. The CTLEA will collect data from tribally-operated LE agencies that 1) provide direct law enforcement services on tribal lands, 2) are funded by a tribe, 3) are responsible for maintaining public order and enforcing the law, 4) are responsible for the detection and investigation of crimes and apprehension of criminals and 5) have general arrest powers or the authority to issue citations.

Authority for tribal LE agencies is granted by federally recognized tribes through tribal constitutions, statutes or codes or by 638 compacts through the Bureau of Indian Affairs (BIA).¹ These tribal LE agencies include any and all tribal police departments and those agencies with special authority and/or functions operated by or on the behalf of tribal governments in Indian country, including conservation and wildlife agencies and tribal college or university police operating in Indian country. A screener question in the CTLEA questionnaire will confirm eligibility, asking specifically if the agency employed at least one sworn law enforcement officer with general arrest powers and/or the ability to issue citations. While eligibility was confirmed during the creation of the CTLEA universe list, this question confirms that agencies are still eligible at the time of data collection.

No comprehensive list of tribal LE agencies existed prior to the development of the 2019 CTLEA universe development. To ensure that LE agencies for each federally recognized tribe (whether qualifying for the CTLEA or not) were identified, the development was initiated using the 2016 Federal Register to first obtain a list of the 567 federally recognized American Indian tribes.² BIA posts a list of all federally recognized tribes in the Federal Register on an annual basis. Each tribe and native village listed on the Federal Register was assigned a CTLEA universe identification number. This number allowed for consistent tracking and reference as research and outreach were conducted to determine whether each of the 567 federally recognized tribes is eligible for the CTLEA or not.

Using the 2015 Tribal Leadership Directory, downloaded from the BIA website, leadership contact and reservation information was obtained for each federally recognized tribe. The BIA Tribal Leadership Directory does not have information about the presence of LE agencies working on tribal lands, so other sources were assessed and reconciled to determine the tribal LE agency contact information, agency type, and other pertinent data to determine initial eligibility. The following data sets were analyzed, cross-referenced, and compiled to produce a comprehensive list of contact information for tribal and BIA LE agencies operating in the U.S. as of calendar year 2016:

• Census of State and Local Law Enforcement Agencies, (CSLLEA) – Since 1992, BJS has conducted the Census of State and Local Law Enforcement Agencies (CSLLEA) about every 4 years to provide a complete enumeration of state and local agencies in the U.S. CSLLEA data files from 2004 and 2008 were reviewed to identify agencies that identified themselves as tribal police departments. For agencies that had both a primary listing and a criminal investigations unit, it was determined that only the main agency listing would be included, as it would encompass the investigations unit. The CSLLEA is being conducted in 2019 and the CTLEA universe will be updated with any new contact information or agencies found during the CSLLEA data collection. The CSLLEA does not include federal agencies (e.g., the Bureau of Indian Affairs police agencies).

• BIA Office of Justice Services’ Law Enforcement Directory – Through its work with the Indian Country Law Enforcement Section, IACP obtained the 2015 directory of tribal LE agencies from the BIA Office of Justice Services. This directory included BIA direct service agencies, tribal LE agencies, and tribal conservation agencies with a law enforcement function.

• FBI Tribal Law Enforcement and Justice Agencies Lists – The Federal Bureau of Investigation (FBI) provided BJS with three lists, one of tribal LE agencies, one of tribal justice agencies, and one of agencies identified as tribal with an assigned Originating Agency Identification (ORI) number. The tribal justice agencies list included records for tribal courts, housing authorities, prosecutors, and other tribal departments, many of which were deemed ineligible during the vetting process because they lacked a law enforcement function. The FBI Tribal Law Enforcement and ORI lists included tribal LE agencies, BIA agencies, and tribal conservation agencies.

• IACP Indian Country Section Roster and membership database – IACP staff gathered lists from several IACP data sources. This included the current IACP Indian Country Law Enforcement Section (ICLES) roster. Records for nontribal agencies were excluded from the universe file. Queries of IACP’s customer management system database were run on the terms ‘tribal,’ ‘Nation,’ ‘Indian,’ and ‘Pueblo’ to find all IACP member and non-member tribal LE agency contacts that had interacted with IACP.

• National Survey of Tribal Court Systems, 2014 (NSTCS) – The National Survey of Tribal Court Systems (NSTCS) data collection provided tribe name, tribal agency name, city name of the tribal court, state of the tribal court and zip code of the tribal court. These data were used to verify and validate tribal LE agency information on the CTLEA list with the LE agency information reported by tribal courts on the 2014 NSTCS universe list.

The source files listed above were used to create an initial comprehensive listing of tribal LE agencies with contact information. When there were discrepancies between the sources for an agency name (e.g., Nez Perce Police Department versus Nez Perce Tribal Police Department), contacts, or addresses (i.e., mailing versus physical address), follow up was made by checking online sources or, in the event that the information could not be verified through other sources, contacting the tribe to ensure the most current and correct information was included.

After compiling the draft universe list from the sources described above, there were general categories of agencies that needed follow up to determine eligibility. These included the following:

• No Known Law Enforcement Agency – Based on the universe sources listed above 87 tribes could not be confirmed to have a LE agency. Each of these tribes was called and/or web research conducted to determine if the tribe had a qualifying LE agency or if it was covered by state, local, or other tribal public safety services.

• Conservation and wildlife agencies – 33 tribal wildlife and conservation agencies were found through the various universe sources. Eighteen of the identified conservation and wildlife agencies were derived from the CSLLEA data file and were deemed eligible because CSLLEA eligible agencies must have at least one full time sworn officer with general arrest powers. The remaining 15 agencies that were derived from other source files were contacted by telephone to determine if the agency was eligible. Overall, 31 of the 33 were found to meet the criteria for inclusion.

• Judiciary – There were 80 tribal court agencies identified on the FBI Justice Agencies list, including courts, prosecutor’s offices, detention facilities, and related agencies. None were determined to have a qualifying law enforcement role as officers do not make arrests or issue citations.

• Housing – There were 27 tribal housing agencies, identified primarily from the FBI Justice Agencies list. Each agency was called to determine if the agency had a qualifying law enforcement function. None were determined to meet the criteria for the survey.

• Tribal Colleges and Universities – A list of the 32 fully-accredited tribal colleges and universities was downloaded from the U.S. Department of Education’s White House Initiative on American Indian and Alaska Native Education website. Phone and website research was done to determine which of these colleges and universities had internal LE agencies. Of these, six tribal college police agencies were determined to meet the criteria for inclusion.

• Alaska – Alaska has one reservation and over 200 Alaska Native Villages (ANVs). Law enforcement services are provided to the ANVs by the State of Alaska through Village Public Safety Officers (VPSOs), a component of the Alaska State Troopers. Two tribal LE agencies serve reservation or Indian trust land in Alaska and both are included on the CTLEA universe list, in addition to the Alaska State Troopers. Through contact with the Alaska State Troopers, it was determined that a central office would be able to complete the TLE survey on behalf of all VPSOs and was considered eligible to participate.

• BIA - The Bureau of Indian Affairs is responsible for the management and oversight of federally recognized tribes that do not have their own tribal LE agency, in addition to providing guidance and resources to other tribal police organizations. BIA LE agencies are typically responsible for one or more tribes. BIA LE agencies can respond to calls for service on tribal lands and currently have 27 active BIA offices.

As a result of the development process outlined above, 308 total LE agencies were identified as meeting the CTLEA eligibility criteria. Data on the 27 agencies operated by the BIA will be collected directly from the Department of Interior, Office of Justice Services. Data from VPSOs will be collected from the central point of contact that manages the VPSO program, which provides services to the vast majority of ANVs in the state of Alaska. The remaining 280 tribal-government-operated agencies provide services to a subset of the 573 federally recognized tribes.

To maintain the accuracy of the universe list, IACP regularly checks eligibility within its member directory for new and updated tribal agency contacts and monitors online alerts so that updates to chief and contact name are included in the universe file. Since 2016, four additional agencies were identified through these efforts. BJS identified two tribes that might have tribal LE agencies in late 2018. These agencies were found as part of the universe development for the CSLLEA. One was confirmed as not having a tribal police agency and the second agency was added to the universe list. On January 29, 2018, the Thomasina E. Jordan Indian Tribes of Virginia Federal Recognition Act of 2017 recognized the Chickahominy, Eastern Chickahominy, Upper Mattaponi, Rappahannock, Monacan and Nansemond tribes as sovereign nations, taking the total number of federally recognized tribes to 573. It was confirmed that these tribes did not have qualifying LE agencies. Additional edits to agency contact information will be incorporated into the universe list as new information is made publicly available by the tribes via website updates, by BIA, or through other public sources.

All eligible tribal police, conservation and wildlife offices, and tribal university or college police agencies will be surveyed. BIA agencies will receive the survey through a central point of contact at the Department of the Interior, described further below.

A breakdown of the 2019 CTLEA universe list can be found in Table 1.

Table 1. 2019 CTLEA universe of tribal LE agencies

Type of agency	Count	Percent of universe
CTLEA Universe	308	100%
Total tribal operated agencies	280	91%
Tribal law enforcement agencies	229	74%
Conservation/Wildlife enforcement agencies	45	15%
Tribal university/college police	6	1.9%
Total Federal and State operated agencies/programs	28	9%
Bureau of Indian Affairs police agencies	27	8.7%
Alaska State Troopers	1	0.3%

2. Survey Development and Procedures for Collection of Information

Survey development

Given the unique nature of this data collection effort and that much of the information requested in the survey is not available from any existing source, it was necessary to develop new survey questions to collect data on topics most critical to tribes, tribal justice experts, tribal LE agencies and federal agencies.

The initial questionnaire development involved a literature review, a review of legislation, and interviews with subject matter experts. The literature review identified existing research on tribal law enforcement, as well as the gaps in knowledge that could be addressed through the CTLEA.

The CTLEA survey went through an extensive development process as described in Part A. Once the CTLEA questions were identified and developed, nine eligible agencies were selected for a survey development pilot test. The goal of the survey development pilot was to obtain feedback from tribal LE agencies on the survey content and burden. Agencies were selected from a list provided by members of IACP’s ICLES. Selection criteria was based on geographic location and the size of the population over which the agency had jurisdiction. Eight of the nine agencies selected for the initial survey development pilot test completed the survey and five agencies participated in a debriefing interview.

The table below summarizes the pilot test agency locations and public law 280 status (Table 2).

Table 2. 2019 CTLEA survey development pilot test sites 2017

Tribe/Agency	Agency Type	City	PL 280 Status
Fort McDowell Police Department	Tribal	Fort McDowell, AZ	Optional, concurrent Federal juris.
Sycuan Tribal Police Department	Tribal	El Cajon, CA	Mandatory
Southern Ute Tribal Police Department	Tribal	Ignacio, CO	Non-PL 280
Leech Lake Police Department	Tribal	Cass Lake, MN	Mandatory
Duckwater Tribal Police	Tribal	Duckwater, NV	Optional, concurrent Federal juris.
Lummi Nation Police Department	Tribal	Bellingham, WA	Optional, concurrent Federal juris.
St. Croix Tribal Police Department	Tribal	Webster, WI	Mandatory
Northern Cheyenne Agency	BIA	Lame Deer, MT	Optional, concurrent Federal juris.
Mescalero Agency	BIA	Mescalero, NM	Non-Pl 280

Survey development pilot test procedures

The survey development test of the pilot survey began on December 6, 2016, and closed on January 27, 2017. At the time of the initial survey development pilot test, both tribal and BIA LE agencies were to be included in the CTLEA survey data collection activity. Therefore, BIA agencies were included in the survey development pilot test and received a survey specific to BIA agencies. The survey development test served to pilot the survey questions and response choices. It also assessed survey burden. Selected agencies were contacted by both mail and email at the start of the data collection period. Included in the correspondence was a letter from BJS (Attachment 20), a project summary page (Attachment 21), and a copy of the draft questionnaire appropriate for the agency type (Attachments 22 and 23). For those agencies that had not returned a completed survey within two weeks of the initial mailing, a follow up email was sent to each agency with an attached PDF letter from BJS (Attachment 24). A replacement survey was mailed as needed with a cover letter signed by BJS (Attachment 25).

Based on the results of the survey development pilot test and to reduce the respondent burden hours and ensure high unit and item response rates with tribal LE agencies, significant revisions were made to the tribal LE agency survey (CTLEA) including the removal of the most burdensome questions, specifically those which asked agencies for specific types of arrest statistics and counts of incidents investigated by offense category. Also after the pilot test, the scope of the survey data collection was modified. As BIA LE agencies are operated by the U.S. Department of Interior, a federal agency, they will be excluded from the CTLEA’s direct data collection. Instead of collection data directly from each BIA LE agency, BJS will work directly with the BIA to collect the required information.

Cognitive pilot test procedures

With BIA agencies being collected separately, there is now one CTLEA survey form that will be sent to eligible tribal LE agencies (Attachment 15). A separate survey, specific to BIA, will be circulated by BIA. As extensive edits were made to the tribal LE agency survey after the survey development pilot test, it was determined that a cognitive pilot test would be conducted to re-assess the survey burden and question content.

CTLEA cognitive testing. BJS conducted a pilot test using the revised CTLEA survey to perform both cognitive testing and assess the respondent burden. The pilot testing was conducted under the BJS “Generic Clearance for Cognitive, Pilot and Field Studies for Bureau of Justice Statistics Data Collection Activities” (OMB Control Number 1121-0339), with the request approved by OMB in May 2018 (ICR Reference Number 201512-1121-004). Twenty tribal LE agencies were selected from both Public Law 280 jurisdictions and non-PL 280 areas, based on size of resident population, geographic location, and PL-280 status. Two additional sites were identified as alternates. A break out of the agency selection criteria for the cognitive testing is shown in Table 3.

Table 3. Cognitive testing agency selection criteria

Tribe/Agency	Geographic Region	Size	PL 280 Status	City
Coquille Tribal Police Department	West	Small	PL-280	North Bend, OR
Cahto Tribal Police Department	West	Small	PL-280	Laytonville, CA
Prairie Island Tribal Police Department	North	Small	PL-280	Welch, MN
Lac Du Flambeau Police Department	North	Medium	PL-280	Lac Du Flambeau, WI
Hoopa Valley Tribal Police Department	West	Medium	PL-280	Hoopa, CA
Fond Du Lac Police Department	North	Medium	PL-280	Cloquet, MN
Lac Courte Oreilles Police Department	North	Medium	PL-280	Hayward, WI
Oneida Police Department	North	Large	PL-280	Oneida, WI
Chickaloon Tribal Justice Law Enforcement Department	North	Large	PL-280	Chickaloon, AK
Penobscot Nation Police Department	East	Small	Non-PL-280	Indian Island, ME
Chitimacha Tribal Police Department	South	Small	Non-PL-280	Charenton, LA
Laguna Tribal Police Department	West	Small	Non-PL-280	Laguna, NM
Zuni Tribal Police Department	West	Medium	Non-PL-280	Zuni, NM
Chickasaw Nation Lighthorse Police	South	Large	Non-PL-280	Ada, OK
Choctaw Tribal Police Department	South	Large	Non-PL-280	Durant, OK
Navajo Nation Police Department	West	Mega	Optional PL-280	Window Rock, AZ
Muscogee Creek Lighthorse Police Department	West	Mega	Non-PL-280	Okmulgee, OK
White Mountain Apache Police Department	West	Medium	Optional PL-280	Whiteriver, AZ

The collection period for the cognitive testing was 12 weeks. A total of 18 of the 20 agencies completed and returned surveys within the testing period, which lasted from May 15, 2018 to August 15, 2018. Of the 18 sites that returned completed surveys, 16 also completed a cognitive interview.

The CTLEA cognitive test was divided into three phases: outreach and data collection; nonresponse follow-up; and cognitive interviews and feedback.

Phase 1: Selection, outreach and data collection for cognitive testing of the CTLEA.

• Step 1. Send participation request letter and email. The CTLEA cognitive test began with a mailing and e-mail. A letter (Attachment 14) was mailed to the twenty sites selected by the IACP ICLES and BJS requesting the agency’s participation in the cognitive test, describing the requested information, and underscoring the importance of participation and benefits of the project to all tribes. The packet also included the CTLEA questionnaire (Attachment 15) and a pre-paid business-reply envelope for respondents to send the completed hard copy questionnaire to the data collection agent. The letter listed the telephone and email contact information for the BJS Project Officer.

• Step 2. Nonresponse follow-up. One week after the initial mailing, NORC and IACP attempted to make contact by phone to confirm receipt of the questionnaire (Attachment 16). Agencies were provided an opportunity to ask questions about the test survey, the project, and about specific questions on the questionnaire. Once receipt of the survey was confirmed with an agency, they were not contacted again by phone until either they returned the completed survey so that a debriefing could be scheduled or the deadline for returning the survey had elapsed and a prompting call was needed. During week 4 of the cognitive testing, dependent on contact information on file for the agency, a reminder email or second hard copy of the survey was sent to all non-responding agencies (Attachment 17). If an agency contact had an email address, an email reminder was also sent (Attachment 18). The email contact provided a different form of contact to gain the respondent’s attention, stress the importance of the study, and prompt for the return of the questionnaire.

In general, agencies typically required a combination of at least three follow up contacts by phone or email to complete and return the survey, but some agencies were contacted many more times to first return the survey and then schedule a cognitive interview. Contacts included phone calls and emails based on the method that the agency was most responsive to. Some agencies were contacted only by staff at NORC while others that had a prior working relationship with IACP were first contacted by IACP staff.

• Step 3. Cognitive interviews and feedback. As surveys were completed and returned, research staff attempted to immediately contact the pilot agencies to schedule a debriefing call. The calls were anticipated to last between 10 to 20 minutes, with some completed on the spot due to the relatively short time period required. The intent of the call was to obtain general feedback on agency contacting protocols and to review specific questions and the provided responses. Toward the end of the data collection period, a list of the cognitive interview questions were emailed to three agencies that had not yet completed the cognitive interview. One of the 17 agencies returned their responses by email, with the rest responding by telephone.

Response burden reported by agencies was relatively low compared to previous iterations of the survey, with agencies reporting that the time to complete the survey for all but one was between 15 minutes to about an hour. On average, disregarding the high estimate of four hours provided by one site, the average time to complete the survey was 30 minutes. Across agencies, the vast majority reported no difficulties in providing numeric information regarding staffing counts for sworn officers, calls for service, or adult and juvenile arrests. Agencies reported that the planning and thought put into the survey was evident, with questions being clear, response categories exhaustive, and appropriate terminology was used to capture the concepts being measured. Item non-response was low across all questions. One agency, which had a survey completed by a staff member that was not the chief of police, initially had many missing responses. Upon prompting, the agency was able to provide responses to most questions.

Based on feedback from the cognitive test, minor revisions were made to the CTLEA survey, including clarification of one term, modifying the order of two questions in the survey, and making contact information more prominent throughout the survey if assistance was needed to complete the survey.

Procedures for full 2019 CTLEA data collection

The full data collection effort will include all 308 eligible agencies identified in the tribal LE agency universe list. The collection period for the CTLEA is 16 weeks, anticipated to begin in September 2019, pending OMB approval.

Based on the experience with the cognitive testing, full data collection will be divided into four phases: outreach and data collection; nonresponse follow-up; post data collection verification, validation and editing; and weighting for unit non-response and item imputation phases.

Phase 1: Outreach and data collection for the Tribal Law Enforcement survey

• Step 1. Data collection begins. During week 1 of data collection, NORC will mail a questionnaire packet to all 280 tribal-government-operated agencies via first class USPS mail. (The remaining 28 eligible agencies—27 BIA LE agencies and the Alaska State Troopers—will be surveyed through a separate protocol detailed below.) The questionnaire packet will contain the CTLEA agency invitation letter from the BJS director addressed to the tribal LE agency (Attachment 2); the CTLEA questionnaire (Attachments 1); a letter of support from IACP and their Indian Country Law Enforcement Section (Attachment 3); and a pre-paid business-reply envelope for respondents to send the completed hard copy questionnaire to the data collection agent.

• Step 2. Confirm receipt instruments. Two weeks after the initial survey mailing, NORC will conduct one round of telephone outreach to all agencies that have not responded to the survey, confirming receipt of the materials, responding to questions, and encouraging timely submission. A copy of the confirmation script is included as Attachment 4.

Phase 2. Nonresponse follow-up strategy for the Tribal Law Enforcement survey

• Step 3. Reminder postcard prompt. One month after the start of data collection, NORC will mail a reminder postcard (Attachment 5) to non-responding agencies. The agency-specific postcard will be mailed approximately 2 weeks after the telephone calls. This postcard will encourage non-responders to complete and return the survey.

• Step 4. Telephone prompt. Two weeks after the reminder postcard is mailed, NORC and IACP will conduct one round of telephone outreach to all non-responding agencies. This round of telephone calls will serve to determine progress on the survey, obtain an anticipated timeline for submission and encourage response. Telephone interviewers will also be offered to those who want to complete the survey over the telephone. A copy of the prompting script is included as Attachment 6.

• Step 5. Email and/or fax prompt. During week 8 of the data collection a fax and/or email reminder (Attachment 7) will be sent to all non-responding agencies. If an agency contact has an email address, an email reminder will be sent. If a fax number only is available, a fax will be sent to the agency. If both a fax and e-mail are available, both will be sent. The fax/email will provide a different form of contact to gain the respondent’s attention, stress the importance of the study, and prompt for the return of the questionnaire.

• Step 6. Second hardcopy mailing. One month prior to the end of data collection, all remaining non-responding agencies will be mailed a second hardcopy questionnaire via USPS first class mail. The package will contain a personalized letter from the BJS director that urges the agency to respond (Attachment 8), a copy of the questionnaire based on the agency’s sample type, and an envelope in which to return the completed survey.

• Step 7. Nonresponse follow-up. Some partner organizations that participated in the development of the CTLEA surveys will be asked to assist in the nonresponse follow-up. These partnering organizations have regular contact with or work within Indian country. One week following the second hardcopy mailing, IACP Indian Country Section members will conduct outreach to non-responding agencies (Attachment 18).

• Step 8. Last chance post card. Two weeks prior to the end of data collection, a Last Chance postcard notice (Attachment 9) will be mailed via USPS first class mail alerting non-responding agencies of the scheduled data collection end date. An email announcement will also be sent to those agencies with an email address on file.

Phase 1 and 2 for BIA data collection and nonresponse follow up.

Unlike the data collection and follow up effort described for tribal LE agencies, the project will rely on central offices within the BIA and Alaska State Troopers for distribution, follow up, and, where appropriate, completion of surveys. At the start of the data collection period, the following actions will be taken by the project:

• Alaska State Troopers: A copy of the CTLEA survey will be forwarded to the central office for the Alaska State Troopers. One survey will be completed on behalf of all VPSOs, with aggregate data used to populate the survey. The Alaska State Troopers have confirmed their ability to provide aggregate data and complete the CTLEA survey as written.

• Bureau of Indian Affairs offices: Copies of the BIA survey (Attachment 10) will be provided to BIA’s Office of Justice Services. In turn, the Office of Justice Services will forward the survey to individual BIA LE agencies for completion. BIA surveys will be returned to NORC, the BJS contractor, upon completion. Similar steps as outlined above will be followed to ensure receipt of the surveys, though all direct follow up activities will be conducted by the Office of Justice Services.

Phase 3. Post data collection verification, validation and editing

• Step 1. Data editing and retrieval. NORC staff will edit completed surveys to assess whether missing information can be imputed based on survey responses, if values are within range and if inconsistencies can be resolved. When it is determined that additional data are needed because of errors, inconsistencies, or missing data that cannot be corrected using the editing specifications, the data collection agent will contact (Attachment 11) the data provider for clarification as close as possible to the date of submission of the questionnaire.

• Step 2. Data entry. As editing and data retrieval is completed, survey data will be keyed into a database by NORC.

Phase 4: Weighting for unit non-response and item imputation

Weighting for Unit Non-Response

Although it is anticipated that the steps mentioned above will result in participation from a large number of agencies, there will likely be a percentage that do not complete the survey in a timely manner. While the CTLEA is intended to be a census, the final list of responding agencies is likely to be a nonrandom sample of the study population due to differential response rate across subpopulations. NORC will develop an analysis weight for each respondent through a two-step weighting adjustment procedure. The sample base weight is 1 for all sample members because it is a census. The first step is an eligibility step. If any respondents are found to be ineligible, the base weight will be set to zero at this step. It is not expected that many agencies will be found to be ineligible as all agencies were screened during the universe development stage.

The second step is a nonresponse weight adjustment step. Through this adjustment, the weight carried by non-respondents is transferred to respondents within each adjustment cell so each responding agency will represent a portion of the non-responding agencies and the sum of the weights will be the total number of eligible agencies (280). To determine which variables should be used to create the adjustment cells, a non-response bias analysis to compare response rates among different subgroups will be carried out. Variables analyzed will include the source of inclusion (i.e. the IACP, CSLLEA, FBI, BIA, and Tribal Leader lists), Law Enforcement Agency Type, PL-280 Status, Native American Land type, agency location (state or region), American Community Survey (ACS) Reservation Population Estimate (as a proxy for agency population size), and other ACS variables such as the percentage of the population that is American Indian. Response rates by land size (in square mile) categories will also be explored. Variables where subgroups have the largest differential response rates will be used to define the adjustment cells for the nonresponse weight adjustment. The details of these analyses will be included in a non-response bias analysis report. To avoid introducing unnecessary weight variation, each adjustment cell must contain at least 20 cases.

Imputation for Item Non-Response

While high item response is anticipated, agencies may leave items blank due to lack of access to the information or the fact that the data are not being recorded by the agency’s records management system. NORC proposes to use hot-deck method for imputation to ensure a complete data file. Hot deck imputation is a cost-efficient imputation method that protects relationships between variables that are observed in the non-missing data. The proposed method and program has been used for many other NORC studies, including the Survey of Doctorate Recipients and the National Immunization Survey. Any imputed values will be merged into the data file (and flagged) prior to delivery to BJS and subsequent archiving. This single-imputation method does result in an under-representation in variance. If there is a high non-response rate for certain items (20 percent or more), hot-deck will not work as well due to the necessary re-use of donors. In this case, the more costly multiple imputation could be used for some or all of the variables.

Multiple imputation involves model building for multiple related variables in which all are imputed together or sequentially. NORC has multiply-imputed data for many projects, including the National Immunization Survey and the Truth Initiative project. Multiple imputation can better reduce non-response bias in items with high missing data rates (greater than 20%) than hot deck imputation. For each variable to be imputed with our hot-deck method, the file will be sorted by variables that have correlations with the variable to be imputed. These “sort” variables will be chosen based on models in which the variable to be imputed is the dependent variable and the independent variables (sort variable candidates) will be other questionnaire items or variables known for all 280 eligible agencies considered for weighting (see above). These sort variable candidates can be quantitative or qualitative, but quantitative sort variables chosen will be converted to categorical variables during imputation so that later variables in the sort order can still have an impact on the sorting of the file. Once the file is sorted, our hot-deck imputation then uses the nearest neighbor as the donor for the missing value. Each variable to be imputed will be sorted according to its own set of sort variables.

3. Methods to Maximize Response Rates

The CTLEA aims to obtain a response rate of 90% or more. Since the CTLEA is a new tribal specific data collection effort, historical response rates are not available. However, the BJS 2008 Census of State and Local Law Enforcement Agencies (CSLLEA) data collection included 178 tribal LE agencies and obtained a response rate around 95%.

BJS and NORC have and will continue to undertake various steps to help facilitate high response rates for both versions of the CTLEA survey. In addition to the various survey methodology strategies discussed in the data collection section above, best practices learned for collecting data from tribal justice agencies will be incorporated. For example, BJS has taken a proactive approach to seek information from and inform tribal nations about the CTLEA, as well as coordinate with BIA and tribal organizations and tribal LE experts from around the country to design the CTLEA survey instruments. The development of the CTLEA surveys with the input and support of tribal LE practitioners working in Indian country was essential to the development of the culturally-centered survey and also served to inform tribal nations about the project early on. In September 2016, an introduction letter (Attachment 19) and one-page flyer summarizing project activities (Attachment 21) were sent to tribal leaders of all federally recognized tribes informing them of the CTLEA project, the types of data to be collected, the reasons for the collection, and requesting their support.

Indian country law enforcement expert panel. As described in Part A, to ensure that the CTLEA survey captures key measures on the administration and operation of tribal LE agencies with clarity and an informed perspective from tribal LE professionals, BJS hosted a two-day expert panel meeting in Phoenix, Arizona in October 2016 comprised of representatives from various LE agencies that work directly in Indian country. Twenty-four representatives from tribal police departments, BIA agencies, Alaska State Troopers and Village Public Safety Officer Coordinators, DOJ’s Office of Tribal Justice, the FBI’s Indian Crimes Unit, Community Oriented Policing Services (COPS) and the U.S. Attorneys served as panelists. Panel members helped to improve the initial draft survey questions and provided content and survey item suggestions to ensure a high response rate. The panel members conducted a detailed review of the survey content and questions section by section. In addition, the panelists also discussed the utility of the information gathered for tribal LE agencies and whether the types of data requested in the CTLEA would be readily available to the respondents. The CTLEA tribal LE expert panel participants included the following tribes or organizations:

Indian Country justice agencies marketing strategy. At the start of data collection, a CTLEA project summary will be sent by NORC to various Native American media, professional associations and organizations asking that they distribute materials about the project. A NORC press release will describe the utility and benefits of CTLEA results in understanding the challenges of crime and justice in Indian country and it will also indicate the types of information that will be collected, including budgets, staffing and work activities.

Letter of support. A letter of support (Attachment 3) from IACP/ICLES will accompany the initial survey mailing. Subsequent updates about the value of the survey will continue to be communicated to IACP’s membership through its ICLES, which has many tribal law enforcement chiefs as members. IACP will also issue project updates via their Twitter account plus other social media outlets.

Response modes. The data collection plan includes a multi-mode collection strategy so that surveys may be returned to the data collection agent via mail, fax or email. A web option is not offered for this data collection, as the small sample size and inconsistent internet availability and access across Indian country does not warrant the costs.

Additionally, NORC will monitor a project specific email address and toll-free number, both of which allows respondents to contact NORC with questions on the survey or issues they encounter.

Contacts for Statistical Aspects and Data Collection

The Bureau of Justice Statistics within the U. S. Department of Justice is the sponsoring agency for the CTLEA. NORC is the data collection agent. NORC’s efforts are supported through a subcontract to IACP, specifically IACP’s Indian Country Law Enforcement Section.

• BJS contact –

Steven W. Perry, Statistician

810 Seventh Street NW

Washington, DC 20531

(202)-307-0777

[email protected]

• NORC contact –

Pam Loose, Senior Research Director

NORC at the University of Chicago

55 E. Monroe

Chicago, IL 60603

(312)-759-4000

[email protected]

Attachments

Final Data Collection Phase

1. CTLEA Data Collection TLE Survey

2. CTLEA Data Collection Invite Letter

3. IACP/ICLES Letter of Support

4. CTLEA Data Collection Confirmation Script

5. CTLEA Data Collection Reminder Postcard

6. CTLEA Data Collection Prompting Script

7. CTLEA Data Collection Email/Fax

8. CTLEA Data Collection Follow Up Letter

9. CTLEA Data Collection Last Chance Postcard

10. CTLEA Data Collection BIA Survey

11. CTLEA Telephone Retrieval Script

12. Tribal Leader Announcement Letter

13. CTLEA Data Collection Project Summary BJS

Cognitive Interview Phase

14. CTLEA Cognitive Test Invite Letter

15. CTLEA Cognitive Test Survey

16. CTLEA Cognitive Test Prompting Script

17. CTLEA Cognitive Test Follow Up Letter

18. CTLEA Cognitive Test Survey Email Template

Initial Survey Development Phase

19. CTLEA Tribal Leader Announcement Letter

20. CTLEA Pilot Invite Letter

21. BJS CTLEA Summary December 2015

22. Pilot Survey CTLEA

23. Pilot Survey BIA

24. CTLEA Pilot Survey Email Template

25. CTLEA Pilot Follow Up Letter

1 Title 25 U.S. Code § 2802 - Indian law enforcement responsibilities

2 “Indian Entities Recognized and Eligible To Receive Services From the United States Bureau of Indian Affairs.” Federal Register 82: 10 (January 17, 2017) p. 4915: https://www.gpo.gov/fdsys/pkg/FR-2017-01-17/pdf/2017-00912.pdf