NISMART-4 Supporting Statement B – Statistical Methodology

PART B

1. Respondent Universe and Sampling Methods

The Fourth National Incidence Studies of Missing, Abducted, Runaway and Thrownaway Children (NISMART-4) will include three pilot studies to test new approaches to collecting information on stereotypical kidnappings (i.e., stranger abductions), family abductions and other types of missing children reported to law enforcement and a full survey administration designed to provide national estimates of stereotypical kidnappings.

1. Law Enforcement Survey – Stereotypical Kidnappings (LES-SK) Pilot

The design for the pilot test of the LES-SK focuses on testing the new methodology of the self-administered online questionnaire to collect details about stereotypical kidnapping cases. The project staff will identify 20 law enforcement investigators to complete the survey based on the National Center for Missing and Exploited Children (NCMEC) providing 20 stereotypical kidnapping cases from their database. The pilot test will not include the methodology for administering the mail screener to identify jurisdictions that have stereotypical kidnapping cases. The reasons for this are two-fold. First, the mail screener is a methodology that has been demonstrated effective in the past. This method yielded high response rates of 91 percent for NISMART-2 1999 and 86 percent for NISMART-3 2011. The design principles of the methodology proposed continue to conform to best practices within the survey industry for mail surveys.¹ There is no reason to believe the mail survey, with its repeated follow-up components by mail and telephone, will not be successful today.

The second consideration for not testing the mail screener methodology is cost and level of effort. An estimated 20 stereotypical kidnapping cases are needed to conduct a solid pilot test of the feasibility of collecting detailed case information from an online survey of child abductions. Stereotypical kidnappings are rare events. NISMART-3 identified just 142 cases that seemed to meet the definition from the 4,644 screeners mailed to law enforcement agencies (LEAs) in 433 counties.² Replicating the sampling methodology of the LES-SK to find 20 stereotypical kidnapping cases for a pilot test would require administering the mail screener with multiple follow-up contacts via mail and telephone to a minimum of 670 law enforcement agencies and would take at least 4 months to implement. Consequently, the project team has determined that such an effort would not contribute any new information to the planned methodology for the mail screener. As such, focusing the pilot test on testing the new methodology of the online survey appears to be the best use of available resources. Results from the LES-SK pilot study will inform the final methodology for the national implementation of the LES-SK, discussed below.

2. National Law Enforcement Survey – Stereotypical Kidnappings (LES-SK) Full Survey

The sampling approach for the NISMART-4 national LES-SK will use a similar design to that employed in NISMART-3 and NISMART-2.

Considerations for the Sample Design. The 2011 NISMART-3 sampled 433 counties from a sample frame of 3,143 counties after clustering small adjacent counties. The sample selected these first stage units using a stratified probability-proportional-to-size design based on size of the population ages 0 to17. The largest counties were selected with certainty and smaller counties had a lower chance of being selected. Within the sampled first stage counties, the 4,644 LEAs identified in the sampled counties were sent a screener. The 2011 LES-SK estimate of the national number of stereotypical kidnapping victims was 105, with a 95 percent confidence interval of the estimate ranging from 40 to 165.³ While the estimate and confidence interval clearly showed the number of victims of stereotypical kidnapping was very small, the breadth of the confidence interval was larger than desired.

Upon the conclusion of NISMART-3, the project staff investigated options to improve the precision of the estimates for NISMART-4 in the LES Redesign Planning Papers.⁴ With any sample survey, the natural approach to improving precision of the estimates is to increase the sample size. This is not a viable option for the LES-SK because stereotypical kidnapping cases are rare and the sample already contained almost half the estimated total that would be obtained if all LEAs in the nation were canvassed.⁵ The vast majority of the qualifying cases occurred within the certainty counties. The statisticians concluded that increasing the number of counties in the first stage sample above 433 would bring in relatively few additional cases because the larger counties are already included with certainty and the additional counties would all be smaller with lower probabilities of yielding relevant cases. Of the qualifying cases the NISMART-3 LES-SK identified in the sample of 4,644 LEAs, only 40 cases occurred in the designated study year; so increasing the sample of counties would be relatively expensive while producing a very low yield of additional cases to increase precision.

The NISMART-3 team also explored an alternative approach to improve the precision of survey estimates, as reported in the planning papers (Appendix 7). The team considered using auxiliary data from other sources in the estimation stage. The general idea is that if an estimate from the survey is highly correlated with data that are known from an alternative data source then the survey weights can be adjusted by taking this relationship into account, and this can result in more precise estimates. The team explored the potential for improving the precision of LES-SK estimates through the use of auxiliary data. Several data sources were considered, with the most promising being the National Incident-Based Reporting System (NIBRS) and the Uniform Crime Reporting (UCR) Program. However, the results from the analysis showed that adjusting survey weights by using the auxiliary data, in its current form,⁶ was ineffective at improving the precision of LES estimates. If NIBRS or UCR were to change the structure and coverage of their programs, further investigation of their utility in improving LES-SK precision could be worthwhile in the future.

In some sense, the best that can be done with a sample survey on kidnapping is to determine if the number of stereotypical kidnappings remains low. The current design and estimation strategy is useful for collecting data on the outcomes of interest. Large increases in the number of stereotypical kidnapping, such as an increase from 100 to 200, would be detected by the sample design used for NISMART-3 and proposed for the NISMART-4 LES-SK national study.

The NISMART-4 LES-SK Sample. The NISMART-4 LES-SK agencies were sampled according to a stratified cluster design, where the clusters are PSUs (Primary Sampling Units) consisting of a single county or a group of small counties. The sampling frame was created from a national list of all counties in the United States and stratified by Census region (Northeast, Midwest, South, West) and metro status (Core Based Statistical Areas, or CBSA vs. non-CBSA). Counties on the list with an age 0-17 population of less than 2,500 were grouped within the same stratum and state to form PSUs until the minimum size criteria of 2,500 was met.⁷ This resulted in a sampling frame of 2,499 PSUs from the national list of 3,142 counties. The sample size of 400 PSUs was allocated proportionally to the eight strata based on age 0-17 population from the 2016 Census Population Estimates. Prior to sampling, 73 PSUs on the frame were identified to sample with certainty because their measure of size exceeded 184,105, the overall sampling interval ( ). These certainty PSUs were placed into their own stratum. PSUs in the remaining strata were then selected with probability proportional to the number of children aged 0 to 17 years old, using systematic sampling. Of the 400 PSUs sampled, 22 consisted of more than one county, for a total of 427 counties. Table B-1 summarizes the PSU sampling for the NISMART-4 LES-SK.

Table B-1. Stratum Definitions and Allocation for Sample of 400 PSUs

Stratum	CBSA Status	Census Region	PSUs in Frame	Counties in Frame	2016 Population Age 0-17	PSUs Sampled	Counties Sampled
Certainty	CBSA		73	73	27,371,284	73	73
Noncertainty	CBSA	Northeast	160	162	7,129,013	50	50
		Midwest	477	526	10,248,766	72	73
		South	769	840	17,830,642	126	136
		West	210	224	7,052,611	50	54
	NonCBSA	Northeast	35	40	264,473	2	2
		Midwest	296	517	1,367,016	10	18
		South	386	557	1,923,298	14	16
		West	93	203	455,182	3	5
Total			2,499	3,142	73,642,285	400	427

All law enforcement agencies located in the sampled counties were identified using the 2016 Law Enforcement Agency Roster (LEAR) downloaded from the National Archive of Criminal Justice Data (NACJD).⁸ This source was developed by the Bureau of Justices Statistics (BJS) and provides a census of 15,810 active, general purpose law enforcement agencies, including 12,695 local and county police departments, 2,066 sheriffs’ offices and 49 primary state police departments. Excluded from LEAR are special purpose agencies including constables, marshal offices that could not be confirmed to have a primary law enforcement jurisdiction, tribal agencies, criminal investigation agencies, special enforcement (e.g., alcohol, gaming, narcotics, racing) and agencies providing law enforcement services for transportation systems/facilities, natural resources/parks, public buildings/facilities (e.g., primary schools, colleges/universities, state buildings, and public housing). After excluding sheriff’s offices that LEAR stated do not engage in any law enforcement activities (e.g., those that solely operate jails), a total of 4,707 agencies from the LEAR were included in the sample. An additional 20 statewide criminal investigative agencies that were not already part of the LEAR based on the membership directory of the Association of State Criminal Investigative Agencies (ASCIA). All 4,727 of these agencies will receive the Phase 1 mail screener survey. Agencies will be determined to be in scope if they have jurisdiction to investigate child abductions, a criteria ascertained in question 1 of the Phase 1 survey as well as in a final abbreviated questionnaire (Appendix 9, Attachments D and I), which is sent as the final contact to all outstanding, non-respondent agencies in the Phase 1 survey.

The project staff anticipates that the number of stereotypical child kidnapping cases will likely fall within the ranges reported in the 2011 NISMART-3 LES-SK (105, with a 95 percent confidence interval of the estimate ranging from 40 to 165) and 1997 NISMART-2 LES-SK (115, with a 95 percent confidence interval of the estimate ranging from 60 to 170).⁹ Most agencies surveyed will not identify any stereotypical kidnapping cases, either because they do not have jurisdiction to conduct criminal investigations of cases in which a child is reported as missing or abducted or because they did not have any such cases in the reference year (question 2 of the Phase 1 survey).

c. Law Enforcement Survey – Family Abductions (LES-FA) Pilot

The LES-FA pilot will be focused on testing the recruitment letters, data collection instruments, and agencies’ search strategies with a limited number of purposively selected agencies. For Component 1 of the LES-FA pilot, five LEAs will be selected of both medium and large sizes based on consultation with the project’s expert panel, prioritizing agencies that have a history of cooperation on missing children issues. The LEAs will be provided with definitional elements and search strategy suggestions, and asked to monitor the problems and questions they encounter as they try to conduct a search with these criteria. For example, LEAs will be asked to search their case management system for items related to “abduction,” “family offense,” and “custody,” and apply the criteria of involvement of family perpetrators and the removal or detention of a child without permission. Upon completion of the searches, project staff will review with officials their selections and questions, and use the analysis and feedback to design subsequent recruitment materials and protocol.

In Component 2, a formal recruitment letter and protocol will be tested with 25 other agencies of different sizes. This sample will be chosen at random from a list of LEAs stratified by the number of sworn officers. The pilot will include typical procedures for following up with emailed and written reminders and, subsequently, phone calls. The goal will be to confirm that LEAs agree to participate based on the recruitment materials and followup procedures and will further test the ability of the agencies to search for and identify relevant cases in their management systems.

In Component 3, 30 FA cases will be selected from the participating agencies to test the self-administered online questionnaire. These cases may be purposively selected to check the questionnaire’s efficacy with a variety of FA incident characteristics. Results from this pilot will inform the national implementation of the LES-FA at a later date.

4. Law Enforcement Survey – Missing Children (LES-MC) Pilot

The pilot LES-MC survey will, like the LES-FA pilot, test the procedures for helping LEAs identify missing children cases in their system/database, and search out the relevant information. Here again, in Component 1, five LEAs will be sampled, different from the 5 selected for the LES-FA pilot, based on recommendations from the project’s expert panel as having cooperated in missing children training and activities. The LEAs will be provided with definitional elements and search strategy suggestions and asked to monitor the problems and questions they encounter as they try to conduct a search with these criteria. Based on guidance provided in conversations with police management information system experts, the searches will focus on record codes referring to “missing person” and dispatch note searches for the term “missing.” Upon completion of the searches, project staff will review with officials their selections and questions and use the analysis and feedback to design a recruitment protocol.

In Component 2, 25 additional agencies of different sizes will be selected. This sample also will be chosen at random from a list of LEAs stratified by the number of sworn officers. The pilot will include typical procedures for following up with reminders and, subsequently, phone calls. The goal will be to confirm that LEAs agree to participate based on the recruitment material and that they are able to search for and identify relevant cases in their management systems.

In Component 3, 30 MC cases will be selected from those the participating agencies identify to test the self-administered online questionnaire. These cases may be purposively selected to check the questionnaire’s efficacy with a variety of missing child case types. Results and feedback from the three components of the LES-MC will inform the final methodology of the national LES-MC.

2. Procedures for the Collection of Information

1. LES-SK Pilot

The self-administered web survey to collect data on specific SK episodes will be the central focus of our pilot test plans. For the pilot study, NCMEC will be asked to identify 20 recent stranger abduction episodes in their databases that would be out of scope for the national LES-SK, because of their dates of occurrence. Working with NCMEC records, the key investigating officer or current staff member most knowledgeable about each case will be contacted by mail and asked to complete (within 2 weeks if possible) the online Phase 2 questionnaire about the details of the case(s).¹⁰ The invitation letter will be sent with a copy of the statistical bulletin on the LES-SK findings from NISMART-3 and a sheet of Frequently Asked Questions (FAQs). (See Appendix 8, Attachment A for the invitation letter and Appendix 8, Attachment B for the FAQs).

Nonrespondent protocol. Based on previous NISMART response rates, a high rate of cooperation for the pilot is expected with minimal effort expended for nonrespondent follow up. In previous NISMARTs, almost all of the investigating officers were willing to discuss their child abduction cases. The response rate for the Phase 2 survey was 91 percent for NISMART-3 and 99 percent for NISMART-2.¹¹ As necessary, project staff will use email and telephone reminders if no response is received within 2 weeks. After three reminders, the investigator will be contacted by telephone and asked for the reasons s/he has been unable to complete the survey online and will be offered the option of doing the survey as a telephone interview. Text for the reminder is found in Appendix 8, Attachment C.

Administration of the pilot survey. The web survey will be programmed using the Westat Survey Framework, which Westat developed using the best tools from .Net, JavaScript, JQuery, and Bootstrap. Westat has successfully used this framework on hundreds of web data collections. The Westat Survey Framework uses MS SQLServer 2016 as its search engine. This web-based, user-friendly survey framework meets Section 508D requirements and exceeds required accessibility requirements. The framework uses current, standard, and secure tools that operate effectively on all of the popular web browsers.

To ensure survey respondent anonymity, no personally identifiable information (PII) will be stored in the web survey application’s database. Westat will generate a random identifier (data type GUID) for each potential survey respondent. Timings automatically captured in the survey will be used to determine the burden of administering the survey for the full survey administration of the LES-SK.

A paper version of the LES-SK survey is found in Appendix 8, Attachment D. Investigators will be asked to complete the online survey about the selected case. Along with completing the survey, participating investigators will be asked to note any problems they encounter with the survey, such as holes in content (e.g., missing response categories, omitted questions critical to the case), language issues, and the appearance and functionality of the web survey itself. Comment fields in the programmed pilot survey will allow investigators to record easily any problems as they complete the survey (see Appendix 8, Attachment E for an illustration of the text regarding the recording of comments).

Follow-up interviews. After survey completion, project staff will review the collected data and comments and follow up by telephone to discuss any problems encountered or suggestions for improving the online questionnaire. Project staff will simultaneously conduct data retrieval for inconsistencies and item nonresponse. The script for the telephone follow-up interview is found in Appendix 8, Attachment F. These interviews are estimated to take about 20 minutes.

Report. Following completion of the pilot test, a report on the pilot test implementation and results will be prepared and delivered to NIJ. The report will incorporate insights and comments gleaned from participating investigators with regard to content, language, and functionality of the web survey, as well as a complete item response analysis of problematic or unnecessary questions.

Revisions for the national LES-SK. Based on these findings, revisions to the LES-SK and data collection protocols will be proposed, and the revised draft plan for the national administration of the LES-SK will be delivered. Following review, comment, and approval by NIJ in consultation with OJJDP, the final plans for national implementation of the LES-SK survey will be delivered. An amendment will be submitted to OMB for any substantive changes required for the LES-SK based on pilot findings.

2. National LES-SK

Publicizing the Survey. In the months leading up to the launch of the national survey, project staff will work to publicize the survey to law enforcement agencies to generate interest and expectations for the survey’s completion. This will take the form of announcements on relevant listservs, hosting informational webinars with NCMEC, and distributing fliers at conferences or symposiums such as the National Amber Alert Symposium.

Project staff will also seek to publicize the survey by enlisting the support of state investigative partners (SIPs) or state MCC managers. A few weeks prior to the launch of the survey, SIPs and/or state MCC managers will send letters of support (Appendix 9, Attachment A) to investigators in their state, endorsing the survey and encouraging participation.

Data Collection. The name and contact information for the chief of police or sheriff for each of the sampled LEAs will be obtained from the 2018 National Directory of Law Enforcement Administrators. In Phase 1 (detailed below), the chief of police or sheriff of each agency will receive the mailing with the mail screener to determine whether they had any stereotypical kidnappings in their jurisdiction during the timeframe of the survey. In most cases, the chief of police or sheriff will assign another officer to complete this task. If agencies have investigated any candidate stereotypical kidnappings cases within the 1-year timeframe of the survey, the screener will ask them to provide case numbers along with names and contact information for the primary investigating officer for each case. In the second phase, those agencies that reported they had any potentially relevant cases will be recontacted and the investigating officers will be asked to provide detailed information about each case through the web survey.

In addition to these surveys, project staff will search relevant databases to identify possibly qualifying LES-SK cases in the sampled counties not already identified in the Phase 1 survey. Two FBI databases include its Crimes against Children Unit (CACU) and its Violent Criminal Apprehension Program (VICAP) data. Similarly, NCMEC and state MCCs will be asked to provide information on any nonfamily abductions during the LES-SK timeframe that they may have in their databases. Project staff will also conduct systematic Internet searches of newspaper databases (e.g., ProQuest, Nexis Uni) for additional kidnapping cases in the sampled jurisdictions. Any additional cases identified through these methods in sampled jurisdictions will be included in the Phase 2 survey.

Phase 1. The Phase 1 survey packet will include a letter of support from the NIJ social scientist (Appendix 9, Attachment B), an invitation letter from the principal investigators describing the study and the voluntary nature of participation (Appendix 9, Attachment C), the mail survey screener (Appendix 9, Attachment D), frequently asked questions for the Phase 1 mail screener with a glossary of study terms (Appendix 9, Attachment E), and a business-reply envelope. The FAQs provide answers to common concerns or questions about the study and the survey, with contact information of individuals (names, email addresses and toll-free numbers) should respondents need additional information about the study. Several follow-up mailings are planned in order to obtain a high response rate (Table B-2). The protocols are based on both past successful experience with the NISMART-2 and -3 LES-SK and on current best practices for mail survey administration cited by the well-known survey methodologist Don Dillman.¹²

Table B-2. Schedule of Mailings

Activity	Date
1st mailing of survey	Week 1
Thank you note/reminder postcard	Week 3
Reminder letter with 2nd copy of survey	Week 6
Reminder letter with 3rd copy of survey	Week 8
Shortened four-question survey	Week 16

About 2 weeks after the initial mailing, a postcard will be sent to all agencies thanking those that have returned the survey and asking nonrespondents to complete it (Appendix 9, Attachment F). A second reminder (Appendix 9, Attachment G) with a copy of the survey and business reply envelope will be sent to nonrespondents 3 weeks after the first reminder. A third reminder (Appendix 9, Attachment H) will be sent to nonrespondents about 2 weeks later. This mailing will contain a copy of the survey (stamped “Third Request”), another copy of the NIJ social scientist letter of support and a Fed-Ex return envelope to convey the urgency and importance of the study. A shortened four-question survey (Appendix 9, Attachment I) with key questions will be sent to the remaining nonrespondents in a fifth mailing. The purpose of this short survey is to determine if the nonresponding agency had any jurisdiction to investigate relevant cases so those without jurisdiction can be classified as out-of-scope rather than as eligible nonrespondents during the weighting. Nonrespondents will be contacted after the fifth mailing by telephone and asked to complete the screener by phone. Telephone follow up was highly successful in increasing the response rate on NISMART-3 from 60 percent to 86 percent.

Phase 2. In the second phase, the key investigating officer for each stereotypical kidnapping case identified in Phase 1 will be contacted and asked to complete the LES-SK self-administered web survey (Appendix 9, Attachment J) providing details about the case. The web survey covers several topics broken down as follows:

• Preliminary Questions: This section includes questions on the stereotypical kidnapping criteria, case status, agency role in the investigation, involvement of other agencies, and criteria for a case screening out as ineligible if they fail to meet the criteria for a stereotypical kidnapping or if the investigation was closed or suspended for all of the study timeframe.

• Child Characteristics: This section asks about the child victim in the incident, demographics, perpetrator information, family living arrangement, and whether the child victim has a physical, mental disabilities, or medical conditions or other problems such substance use. It also includes items on the child victim’s involvement in the criminal justice system and whether the child was recovered, killed, or still missing.

• Perpetrator Characteristics: This section asks about perpetrator demographic characteristics, life circumstances at the time of the crime, prior offenses, and status in the criminal justice system. Other items include perpetrator social interactions, intelligence, history of substance abuse and information on any prior arrest.

• Crime Characteristics: This section includes questions about the discovery and report of the crime; recovery of the victim, the site where the victim was last seen, initial contact between the victim and perpetrator(s), the movement, detainment, or concealment of the victim, maltreatment or injuries to the victim; criminal motive and plan, and characteristics of key sites in the abduction.

• Investigation Information: This section includes the investigative activities and tasks undertaken by other law enforcement agencies in cases where multiple law enforcement agencies are involved. Items in this section also cover questions about what evidence or leads were key to recovering the victim or identifying the perpetrator, and whether technology such as cell phones or the internet played a role in the investigation.

Invitation letters will be sent via the U.S. Postal Service to each designated respondent. The letter will provide a brief description about the study, explaining that their agency is participating in the survey, and that the recipient is asked to complete their agency’s participation by providing details about the case(s) they investigated (Appendix 9, Attachment K). The letter will note the voluntary nature of participation but will also stress the importance of the project. The letter will contain instructions for logging into the survey website and completing the self-administered web questionnaire. The instructions to the LES-SK self-administered questionnaire will direct respondents to refer to agency records of the incident when they fill out the survey. In addition to the letter, the mailing will include FAQs (Appendix 9, Attachment L) specific to the Phase 2 web survey data collection.

Based on preliminary testing of the LES-SK using records on old missing child cases provided by NCMEC, the Phase 2 survey is expected to take about 40 minutes, including time to read the survey invitation materials and FAQs. More accurate estimates of the respondent’s time burden will come from the pilot test.

Email reminders will be used to ask respondents to complete the web survey. The first reminder (Appendix 9, Attachment M) and third reminder (Appendix 9, Attachment N) emails will be sent by the project staff, while the second reminder email (Appendix 9, Attachment O), will come from NIJ. After three reminders, the nonresponding investigators will be contacted by telephone and asked if they would like to complete the survey online or by telephone. Phone interviewers will be trained to administer the web survey as a telephone interview. The training will involve a review of question-by-question specifications, practice interviews, and instruction on completing call records, scheduling appointments for telephone interviews, responding to questions from participants, and entering information into the web survey. The interviewers will read the questions from the web survey and enter responses directly into the web application. Telephone follow up will continue until the survey is completed in either mode or the 4-month Phase 2 field period ends. If response rates are lower than anticipated, SIPs may be asked to assist in obtaining a response.

Kidnapping cases may also be identified through media reports within the sampled counties, as described above. If they do not yield an online or telephone survey with an investigator, they will be coded in the web survey if there is sufficient published detail provided about them to answer the key survey questions that are critical for deciding whether the case meets the NISMART definition and does not duplicate another case in the jurisdiction.

Data Retrieval for Inconsistencies and Item Nonresponse. As agencies submit data, the project staff will review submitted surveys for completeness. The project staff will perform quality checks to identify questions that are critical but were left unanswered, inconsistencies in reported data across items, and other instances where follow up is needed. The team will contact respondents when necessary to resolve questions, acquire missing data and understand ambiguous descriptions. To minimize the need for such follow ups, the project staff will identify key questions that are most critical to the study and build soft edits into the programming of the web survey to encourage respondents to answer the questions with a valid response. In addition, a quality control program will be developed to run checks and produce a summary report on every submitted survey. The checks will flag inconsistencies or missing data. Data retrieval staff will use the summary report listing the inconsistencies to facilitate follow-up with individual investigators to retrieve missing data on key elements. Where there are missing data or inconsistencies that need clarification, the team will call the investigating officer to collect the missing information over the telephone. If a case does not present any issues needing follow up, a thank you letter (Appendix 9, Attachment P) will be sent to both the chief of police or sheriff and the investigating officer who completed the web survey.

Data Coding. After data collection, project staff will clean and code the data. Initial attention will be paid to identify and exclude any duplicate case handled by more than one agency. In the pre-evaluative coding phase, all completed Phase 2 surveys will be reviewed to verify that (1) cases were actively investigated during the study timeframe and (2) each case had at least one victim/perpetrator pair in which the perpetrator was a stranger or slight acquaintance. In NISMART-3, 31 cases were disqualified as not actively investigated during the study timeframe. These were largely cases from prior years involving children who had vanished or unsolved homicides, which police considered open although no investigative steps were taken during the study timeframe. An additional 3 cases were disqualified in which all victim/perpetrator pairs were family members or close acquaintances.

The primary function of evaluative coding is to identify whether a perpetrator’s actions in relation to a specific victim qualified as a stereotypical kidnapping based on the definitions established for the NISMART-3 LES-SK and used in NISMART-4. Consistency in NISMART definitions over time is important to be able to track trends and changes to stranger abduction episodes.

The evaluative coding will use procedures established for the NISMART-3 LES-SK. The procedures will entail reviewing all documentation associated with each case (i.e., the Phase 1 and 2 surveys) and applying NISMART criteria to determine whether perpetrator’s actions toward a specific victim qualified under the NISMART definitions as (1) a nonfamily abduction and (2) a stereotypical kidnapping. It is essential that both these qualifications be met before the victim/perpetrator pair can be considered “countable” for the NISMART-4 LES-SK and included in the cases used for the study estimates.

Qualifying as a Nonfamily Abduction. In order to qualify as a stereotypical kidnapping, the perpetrator’s actions have to first qualify as a nonfamily abduction. There are two principal types of nonfamily abductions—Nonfamily Abduction Type 1 (NFA1) and Nonfamily Abduction Type 2 (NFA2). For the first type (NFA1), the child has to be taken by use of force or threat or detained by the use of force or threat for a substantial period and in a place of isolation by a nonfamily member without either lawful authority or parental permission. For the second type (NFA2), force or threat is not required. Rather, the child (14 or younger or mentally incompetent and 17 or younger) has to be lured, taken, or detained by a nonfamily member in an isolated place for a substantial period of time, without either the lawful authority or permission of the parent/guardian and the perpetrator has to have (a) concealed the child’s whereabouts; (b) required ransom, goods, or services; or (c) expressed an intention to keep the child permanently.

Qualifying as a Stereotypical Kidnapping. In order to also qualify as a stereotypical kidnapping, a nonfamily abduction has to fulfill additional requirements: (a) the perpetrator must have been a stranger, a recent or only slight acquaintance, or still of unknown identity but deemed likely to have been a stranger or slight acquaintance¹³; and (b) at least one of the following markers of severity must apply:

• The child was detained overnight or longer

• The child was killed

• The child was transported at least 50 miles

• The child was held for ransom

• The perpetrator intended to keep the child permanently

The coding guidelines do allow the coders to make reasonable inferences when case information is limited. For example, if the child was sexually assaulted, coders can infer that the perpetrator had to detain the child with force or threat for that to occur. In addition to deciding whether the nonfamily abduction criteria and stereotypical kidnapping requirements were met, the evaluative coders will document other case characteristics as part of the evaluative coding task. Evaluative coding sheets will capture the evaluators’ conclusions as to whether: (a) the victim/perpetrator pair met the NFA1 criteria, (b) the victim/perpetrator pair met the NFA2 criteria, (c) whether the events involving the victim/perpetrator fulfilled any of the stereotypical kidnapping markers, (d) specific aspects of the abduction (such as the perpetrator’s intent; whether the incident involved a transformation or escalation of events; perpetrator, victim, and parent characteristics like drug use and previous criminal justice involvement); and (e) other key information, such as whether the victim was still missing at the time of the follow-up interview, and whether the perpetrator was a serial killer.

Coders will review all cases and assign codes independently. Coders will then randomly select 10 cases to review and check for disagreements to ensure high inter-coder reliability. Disagreements will be discussed and resolved between the two coders. If decisions are made that influence coding in other cases, each coder will then independently recode the other cases to ensure consistency.

Countability. Based on the evaluative coding decisions recorded on the coding sheets for victim/perpetrator pairs, countability classifications will be assigned separately for each case and each victim and perpetrator involved in the case. These classifications begin with the victim/perpetrator pair, the focus of all evaluative coding decisions. The first decision is whether the victim/perpetrator pair meet the NISMART stereotypical kidnapping criteria. If so, then the pair will be considered to be countable in the stereotypical kidnapping incidence estimates. Following from this, any victim in a countable victim/perpetrator pair will be classified as a countable victim and any perpetrator in a countable pair will be classified as a countable perpetrator. Finally, cases with any countable victim-perpetrator pair will be classified as countable cases.

Weighting and Variance Estimation. A weight will be created for each agency and case to allow the responding agencies and case interviews to represent all law enforcement agencies in the United States. A set of 80 jackknife replicate weights will also be created for each agency and case for estimating variances.

Agency Weights. The agency weight reflects the PSU probability of selection and adjusts for nonresponse at the agency level. Since there is no sampling of agencies within the PSU, the agency base weight = PSU weight. The agency base weight will be probably be adjusted for agency level nonresponse by Census region and agency size, because experience from NISMART-3 showed that response rates were lower for smaller agencies and those of unknown size, and for those in the south. The approach used in NISMART-3 was to create four size classes defined by the quartiles of the distribution of number of officers, plus an additional category for agencies of unknown size. It is likely that NISMART-4 will have a similar classification of agency size for nonresponse adjustment. The nonresponse adjustment factor will be calculated separately for the region x size cells. The final agency weight can be written as:

Final agency weight = PSU weight x agency nonresponse adjustment factor

=

Where MOS_i is the age 0-17 pop for the i-th PSU, N_h is the number of PSUs in the stratum on the frame, and n_h is the number of PSUs sampled in stratum h. The numerator of the nonresponse adjustment factor is summed over the eligible agencies within the nonresponse adjustment cell, and the denominator is summed over the eligible responding agencies in the cell. The nonresponse adjustment factor distributes the agency base weights of the eligible agencies that refused or did not respond to the eligible agencies within the same region/size class who did respond. The final agency weight is zero for nonresponding agencies, and is equal to the agency base weight for ineligible agencies.

Case Weights. The case base weight is equal to the final agency weight from which the case originated, since there is no sampling of cases within agencies.

The case base weights will be adjusted for case interview nonresponse, probably by region and agency size class, as done for NISMART3. The final case weight can be written as:

Final case weight = final agency weight x case nonresponse adjustment factor

= final agency weight

where the case interview nonresponse adjustment factor is calculated within region/size class. The numerator is summed over cases that were eligible (in-scope) for NISMART-4; the denominator is summed over the eligible cases that had a completed interview.

Replicate Weights. To account for the stratification, clustering and unequal weighting in the LES sample design, special procedures are required to produce correct standard errors for the survey estimates. Replication techniques compute standard errors by measuring the variability among “replicates” of the full sample.¹⁴ The replicate samples are subsets of the full sample created to mirror the design of the full sample. As in NISMART-2 and NISMART-3, the jackknife replication method will be used to create a set of replicate weights for this purpose. The paired stratified jackknife method (JK2) will be used to create a set of 80 replicate weights for each agency and for each case to estimate agency level and case level standard errors.

Variance Estimation. The formula for calculating standard errors using the jackknife replicate weights is:

; k = 1, 2, ….80, where is the estimate of based on the k-th replicate and is the estimate of based on the full-sample.

Nonresponse Bias Assessment. If the response rate either at the agency level or the case level is lower than 80 percent, then a nonresponse bias analysis will be carried out. At a minimum, this will include a comparison of the characteristics of early and late respondents, and a comparison of respondents and nonrespondents on characteristics known for both groups. Comparing early and late responders can be useful, as some nonresponse models assume that those units that require more effort to respond (more callbacks, incentives, refusal conversion) are similar to the units that do not respond. A classification tree algorithm, such as CHAID, will be used in both analyses, to identify characteristics related to the propensity to respond.

Analysis and Reporting. Following data cleaning and weighting, three analysis files will be developed—victim level, episode level, and perpetrator-level files to conduct data analysis according to the plans developed by the project staff and approved by NIJ. In consultation with NIJ, the project team will generate estimates for children with countable episodes within the LES-SK survey. The project team will produce tables with estimates of findings on the incidence of stereotypical kidnappings of children known to law enforcement showing the number of victims, the characteristics of the kidnapping episodes, the characteristics of the victims, and the characteristics of the perpetrators. Estimates of key items will be compared with findings from the previous cycles of NISMART-2 and NISMART-3. Following the same approach as in the previous waves, trend analyses, comparing the NISMART-4 estimates of episode children and of missing children with earlier NISMART-3 and NISMART-2 estimates will be conducted. The statistical tables will include comparisons of trends over time and changes in the composition of cases between this survey and prior NISMARTs by examining characteristics of child victims of stereotypical kidnapping, characteristics of episodes, characteristics of perpetrators and relationship to the victim, and characteristics of the kidnapping events as reported to law enforcement.

3. LES-FA Pilot

The LES-FA pilot will have three components. Component 1 will test the procedures for identifying relevant cases from LEA records in a small number of LEAs. Component 2 will test the efficacy of the recruitment materials to recruit LEAs to participate in the study and will further test their ability to identify relevant cases from LEA records. Component 3 will test the new online instrument for gathering detailed case-level information from investigators on specific FA cases.

Component 1. Five LEAs of both medium and large sizes will be selected based on consultation with the project’s expert panel, prioritizing agencies that have a history of cooperation in missing children issues. LEAs will be provided with definitional elements and search strategy suggestions, and asked to monitor the problems and questions they encounter as they try to conduct a search with these criteria. To inform this process project staff have already conducted discussions with experts familiar with LEA data management systems, including their computerized Records Management System (RMS) and their dispatch monitoring software. These experts include executives who design and help install software systems for LEAs, as well as LEA officials who work with these systems. They have advised that police will be able to use codes and fields embedded in their systems as well as free text searching to identify possible cases.

The pilot study will ask investigators to search their case management system for codes related to “abduction,” and “family offense,” as well as free text searches for the word “custody.” The investigators will be asked to apply to the episodes revealed by these searches several other critical criteria: (1) involvement of a juvenile age 0-17, (2) involvement of a family perpetrator, and (3) the removal or detention of a child without permission. A copy of the invitation letter with search instructions, the instrument for recording cases and FAQs and are found in Appendix 10, Attachments A, B and C. MCCs that serve the jurisdiction in the sample will be contacted and asked to perform a database search for all FA cases in their system reported by the agency within the timeframe of the study. A similar process will occur with the NCMEC. The results of these searches will be used to evaluate whether all FA cases known to the MCCs and the NCMEC were found in the law enforcement searches conducted for the study.

Upon completion of their searches, project staff will review with officials their selections, problems they encountered, and any additional cases identified by the MCCs or the NCMEC that did not come up in their searches (Appendix 10, Attachment D). Cases identified by the MCCs or NCMEC but missing from the law enforcement searches will be discussed with the investigators to determine if they should have been included and if changes need to be made in the search procedures. As the NCMEC and MCCs get their cases from the LEAs, this step will help the project understand if the study’s instructions and search protocols missed relevant cases and what adjustments need to be made. The analysis and feedback from these discussions will be used to revise the recruitment letter and search instructions.

Component 2. The invitation letter and protocol will be pilot tested with 25 new LEAs of different sizes (Appendix 11, Attachments A, B, and C). This sample will be chosen at random from a list of LEAs organized by the number of sworn officers. The pilot will include typical procedures for following up with reminders and, subsequently, phone calls (Appendix 11, Attachments D-G). The goal will be to confirm that LEAs agree to participate based on the recruitment material, and to monitor and verify that the agencies can use the searching instructions to compile a complete listing of the eligible FA cases from their records. The MCCs and NCMEC will again be asked to do a database search to identify all FA cases that were reported to them in the study timeframe to compare with the cases identified in the law enforcement searches.

Component 3. In the last component of the LES-FA pilot, the self-administered questionnaire will be tested to gather information about the FA episodes. In the completion phase of NISMART-3, the project team drafted an abbreviated LES-FA questionnaire that can be programmed for online self-administration (Appendix 12, Attachment A). This incorporates data items to capture all the definitional elements used to identify family abductions in prior NISMART cycles. The past definition has been “the taking or keeping of a child by a family member in violation of a custody order, a decree, or other legitimate custodial rights, where the taking or keeping involved some element of concealment, flight, or intent to deprive a lawful custodian indefinitely of custodial privileges.” The questionnaire operationalizes these elements.

Project staff will select 30 FA cases from all those identified by the 30 LEAs who participated in Components 1 and 2. Ten investigators will be asked to fill out the questionnaire in a telephone interview; this will allow project staff to interact immediately with respondents about any difficulty they encounter in answering a question (Appendix 12, Attachments B and C).

The balance of 20 investigators will receive an invitation letter by email (Appendix 12, Attachment D) or mail (Appendix 12, Attachment E), along with Frequently Asked Questions (Appendix 12, Attachment F). The investigators will be directed to the online version of the questionnaire and asked to fill it out on their own about the selected case and make note of any confusion or problem they encounter in the comment boxes programmed into the instrument.

As necessary, project staff will use email and telephone reminders if no response is received. After three reminders coming from the project staff (Appendix 12, Attachment G and H) and the NIJ social scientist (Appendix 12, Attachment I) the investigator will be contacted by telephone and asked for the reasons s/he has been unable to complete the survey online and will be offered the option of doing the survey as a telephone interview.

After the online survey is complete, respondents will be contacted for a debriefing interview about their experience (Appendix 12, Attachment J). The debriefing interviews are estimated to take about 20 minutes. The online questionnaire will be revised based on the feedback from the online survey and telephone interviews in preparation for the national study.

Reporting. Upon completion of the pilot test, project staff will prepare a detailed evaluation report, incorporating insights and comments from participating agencies on the workability of the search procedures and from participating investigators regarding the content, language, and functionality of the web-based instrument. Based on analyses of the strengths and weaknesses of the design revealed in the pilot, revisions will be proposed to the instruments and data collection protocols in a draft plan for the full national implementation of the LES-FA.

d. LES-MC Pilot

Assuming that the data collection approach is effective and successful for the LES-FA pilot, the data collection component of the LES-MC pilot will be virtually identical to the LES-FA pilot. It will have the same three components for (1) testing the procedures for identifying relevant cases from LEA records, (2) testing the efficacy of the invitation letter and followup protocols to recruit LEAs to participate in the study and conduct the searches, comparing the law enforcement search results to the those from the state MCCs and NCMEC, and (3) testing the new online instrument developed for gathering detailed cases information from investigators on specific MC cases.

The only differences will be the following:

1. In the recruitment and search protocol of Phase 1, LEAs will be asked to search their data systems for episodes occurring in a much more abbreviated window of time – a month period occurring at least 6 months prior to the search date (not a full year of data as with the LES-FA).

2. The search criteria will be different. Based on feedback from police management information system experts, LEAs will be asked to search for codes associated with “missing person” and free text for “missing child.” Only episodes involving a child ages 0-17 would be enumerated.

The data collection protocols are found in Appendix 13 for the LES-MC Component 1, Appendix 14 for the LES-MC Component 2 and in Appendix 15 for the LES-MC Component 3.

In Component 3 of the LES-MC the online questionnaire will be tested to gather information about the MC episodes. In the completion phase of NISMART-3, the project team drafted an abbreviated MC questionnaire to be programmed for online self-administration (see Appendix 15, Attachment A). This incorporates data items to capture all the definitional elements used to identify missing children. This definition will differ from past definitions because systematic information from the point of view of caretakers will not be collected.

The questionnaire operationalizes these elements:

• A missing child would be defined as “any child for whom police help was sought (and some record created) to either locate or recover the child.”

• A recovered child would be defined as the record indicating that the child has returned to the household.

Reporting. Upon completion of the pilot test, project staff will prepare a detailed evaluation report, incorporating insights and comments from participating agencies on the workability of the search procedures and from participating investigators regarding the content, language, and functionality of the web-based instrument. Based on analyses of the strengths and weaknesses of the design revealed in the pilot, revisions to the instruments and data collection protocols will be proposed in a draft plan for the national implementation of the LES-MC.

3. Method to Maximize Response rates and Deal with Nonresponse

Several methods to maximize the response rate on the LES-SK will be implemented. Most of these methods have been employed in the previous LES-SK studies and have resulted in high response rates for the mail screener under NISMART-2 (91 percent) and NISMART-3 (86 percent). These methods include designing a screening questionnaire that is easy to understand and complete (the current screener is largely unchanged from the one used in NISMART-3), providing a toll-free number to answer questions, using a series of reminder mailings with additional copies of the questionnaire, and finally, conducting telephone follow-up calls to nonrespondents to either collect data on the abbreviated 4-questionnaire or ask them to return the abbreviated survey.

If these methods are unsuccessful in getting some investigators to complete Phase 2 of the LES-SK either online or through a telephone interview, project staff will look for media reports to provide answers to the questions in the survey to maximize response. For NISMART-3, media reports were sufficiently detailed to allow project staff to code an additional 20 cases, resulting in an increase in the response rate to the Phase 2 survey from 79 percent (based solely on telephone interviews) to 91 percent (telephone interviews plus detailed media reports).

For the NISMART-4 LES-SK, four enhancements will be introduced to increase response rates. The first is publicizing the survey to generate interest and expectation for the survey’s completion among law enforcement agencies. In the months leading up to the launch of the full LES-SK, the project team will make announcements on relevant listservs, host informational webinars with NCMEC and distribute fliers at relevant conferences or symposiums such as the National Amber Alert Symposium.

The second enhancement is using advance letters from SIPs and/or state MCC mangers to sampled LEAs to encourage participation in the screening survey in Phase 1. The third enhancement is the major change in the design from a telephone interview to a self-administered web survey to collect detailed data on child abductions in Phase 2. Finding a time when investigators (who spend much of their days out of the office) could participate in a telephone interview was a difficult and time-consuming task under previous NISMARTs. Given the option to fill out a questionnaire at a time convenient to them, some respondents might participate who would otherwise not. A telephone option will be offered to those who do not wish to participate via web, so these dual options should help maintain a high response rate for the LES-SK.

The final enhancement is a much shorter and more streamlined Phase 2 questionnaire, compared to the lengthy telephone protocol administered under NISMART-3. The shorter time required for the web administration versus the prior telephone instrument should help maximize response rates.

For the pilot studies of the LEA-FA and LEA-MC, the goal is not maximizing response rate, so samples will be intentionally skewed toward those LEAs that are likely to be cooperative and available. However, one of the goals of the pilot is to assess elements of the design that may be burdensome or confusing and thus an obstacle to participation in the ultimate national study. Prompts and follow-up questions in the interviews (see Appendix 12, Attachment J and Appendix 15, Attachment J) will help to identify some of these obstacles.

4. Tests of Procedures to be Undertaken

As described in this submission, NISMART-4 will include a pilot test of the new methodology of the online survey for providing details about stereotypical child kidnapping cases for the LES-SK. This submission also describes pilot tests of the new methodology of having law enforcement search for and provide case details in an online survey for the LES-FA and LES-MC.

5. Consultation Information

The NIJ contact is:

Benjamin Adams

Social Science Analyst

NIJ

810 Seventh St., NW

Washington, DC 20531

[email protected]

(202) 616-3687

The Principal Investigators are:

Andrea Sedlak

Vice President

Westat

1600 Research Blvd.

Rockville, MD 20850

(301) 251-4211

David Finkelhor

Crimes against Children Research Center
Department of Sociology, University of New Hampshire
Durham, NH 03824
(603) 862-2761

1 Dillman, D. A., Smyth, J. D., and Christian, L. M. (2014). Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method. (4^th ed.). Hoboken, NJ: Wiley.

2 Wolak, J., Finkelhor, D., and Sedlak, A. (2016). Child Victims of Stereotypical Kidnappings Known to Law Enforcement in 2011. Juvenile Justice Bulletin – NCJ 249249. OJJDP: Washington, DC. A total of 204 seemingly eligible cases were identified for telephone followup for NISMART-3. Of these, 142 were identified from the screener and another 62 cases came from searches of other databases and emails to agencies about cases from previous years.

3 Ibid.

4 Sedlak, A., Finkelhor, D., Brick, M. and Wolak, J. (2016). Law Enforcement Survey (LES) Redesign: Planning Papers and Draft Instruments. Rockville Institute: Rockville, MD.

5 The confidence intervals for the estimates do not contain a finite population adjustment for the variance estimate to account for the high sampling rates.

6 The 2011 NIBRS and UCR data were used in this exploration.

7 The one exception is Nantucket County, MA (FIPS=25019). Since it is the only non-CBSA county in MA, it was combined with Dukes County, MA (FIPS=25007) to form a single PSU. These two counties were also combined in NISMART3, but Dukes County is now classified as CBSA. Although this means two counties from different strata were combined, this was preferable to combining counties from different states.

8 https://www.icpsr.umich.edu/icpsrweb/NACJD/studies/36697

9 Sedlak, A., Schultz, D., Sayer, J., Hammer, H., and Finkelhor, D. (2004). Second National Incidence Studies of Missing, Abducted, Runaway, and Thrownaway Children (NISMART-2): Stereotypical Kidnappings: National Estimates and Case Profiles Research Report. Philadelphia: Temple.

10 If schedule or workload do not allow the officer to complete the survey in 2 weeks, we will work with them on a suitable timeframe if the officer is willing to participate.

11 Lounsbury, K., Wolak, J., Broene, P. (2016). NISMART-3: Law Enforcement Study (LES-3) Technical Report.  Office of Juvenile Justice and Delinquency Prevention (OJJDP), U.S. Department of Justice, Washington, DC; Sedlak, A., Schultz, D., Sayer, J. (2004).  NISMART 2: Law Enforcement Study Technical Report.  Office of Juvenile Justice and Delinquency Prevention (OJJDP), Office of Justice Programs (OJP), U.S. Department of Justice, Washington, DC.

12 Dillman, D. A., Smyth, J. D., and Christian, L. M. (2014). Internet, Phone, Mail and Mixed-Mode Surveys: The Tailored Design Method. (4^th ed.). Hoboken, NJ: Wiley.

13 A slight acquaintance or person with limited previous contact is a nonfamily perpetrator who was a recent acquaintance whom the child or family have known for less than 6 months, or someone the family or child have known for longer than 6 months but have seen infrequently (e.g., less than once a month).

14 Krewski, D., and Roa, J.N.K. (1981). Inference from stratified samples: Properties of the linearization, jackknife and balanced repeated replication methods. The Annuals of Statistics, 9, i1010-1019.

29