2014 CPFFCL Support Statement_Part B_updated 1-23-15

B. Statistical Methods

1. Universe and Respondent Selection

The Census of Publicly Funded Forensic Crime Laboratories (CPFFCL) collects administrative data from all agencies in the United States that are solely funded by government (or whose parent organization is a government agency) and that employ at least one full-time scientist who possesses a minimum of a bachelor’s degree in the natural science disciplines of chemistry, physics or biology; analyzes physical evidence in criminal matters; and provides reports and testimony to courts of law on such evidence. The CPFFCL-14 will be expanded from previous iterations to include all publicly funded crime labs that analyze digital and multimedia evidence in support of criminal investigations or prosecutions, regardless of whether they employed a natural scientist. The nation’s digital evidence labs are primarily operated by federal government agencies, including the Federal Bureau of Investigation (FBI), Drug Enforcement Administration, Department of Homeland Security, and Department of Defense. The FBI operates 16 Regional Computer Forensic Laboratories that facilitate joint efforts between federal, state, and local law enforcement agencies in the collection and analysis of digital evidence.

The traditional CPFFCL definition of a crime lab limited the information collected about digital evidence because some crime labs only handle this type of evidence and employ forensic experts with training in computer science or information technology as opposed to natural sciences. About 1 in 5 labs that met the eligibility criteria for the CPFFCL-09 reported analyzing digital or multimedia evidence in 2009. These labs were eligible because they were part of a multi-disciplinary lab system that also included natural sciences. Additional details regarding this change to the data collection are available in part A of the supporting statement.

Each individual public lab, including those in multi-lab systems, will be asked to respond to the CPFFCL-14.  Entities outside of the scope of the CPFFCL include privately operated labs and police identification units. The latter may perform forensic services including crime scene investigations and latent print examinations but do so outside of the crime laboratory setting.

In 2015, the Urban Institute will send the CPFFCL-14 questionnaire on behalf of BJS to the universe of approximately 410 publicly funded forensic crime labs identified from the CPFFCL-09 as well as an estimated 100 public labs that solely analyze digital evidence and that were added to the scope of the data collection. State, local, and federal crime labs will provide data on their personnel, budgets, workloads, and backlogs in requests for forensic services. They will also provide information on quality assurance measures such as lab accreditations, proficiency testing programs, examiner certifications, and resources devoted to research.

No sampling will be involved with this project. BJS and Urban have the organizational capacity to successfully complete a data collection from the entire universe of respondents within a responsible timeframe. The census population frame and a complete listing of the contact information from the CPFFCL-09 will serve as a starting point for creating a contact list of eligible labs for the CPFFCL-14. A total of 411 labs met the eligibility criteria for the CPFFCL-09. In addition to the 2009 list, a variety of methods were used to identify additional federal, state, and local labs that are within the scope of the CPFFCL-14 as well as labs that consolidated with other labs or no longer exist.

To develop the complete and updated roster, Urban has conducted a systematic review of the federal, state, and local law enforcement agencies in all 50 states to identify their current organization structure of forensic science services, including the processing of digital evidence. Urban has also reviewed the latest membership lists from professional forensic science organizations [e.g., the American Society of Crime Laboratory Directors (ASCLD), American Academy of Forensic Science (AAFS), and Scientific Working Group on Digital Evidence] and conducted an extensive internet search to identify eligible labs. As a starting point for producing a complete enumeration of all federal, state, and local digital evidence labs, the White House Subcommittee on Forensic Science provided BJS with a list of federal forensic science service providers that includes about 50 crime labs whose sole function is to analyze digital evidence. The contact information (including names, mailing addresses, telephone numbers, and email addresses) for the director of each eligible lab will be finalized just prior to fielding the CPFFCL-14 to minimize the effects of turnover and respondent burden.

2. Procedures for Collecting Information

The CPFFCL-14 data collection will begin with an introductory letter from the BJS director that is mailed to the directors of each publicly funded crime lab in the CPFFCL-14 (Attachment J). These communications will explain the purpose and importance of the data collection, the necessary data confidentiality statements, and the URL link to the previous CPFFCL reports. Labs will be informed that they will be receiving instructions on how to complete the questionnaire in approximately two weeks. They also will be informed that ASCLD has expressed strong support for this project. The letter will also include instructions for labs to access the project website (https://cfcl.urban.org ) and verify their eligibility for the CPFFCL-14 and the contact information of the designated respondent who should receive the questionnaire. Each respondent will be provided a username and password. Labs will also have the option to verify their contact information by calling the Urban Institute.

Following of verification of the contact information, a personalized letter will be mailed to each lab from BJS’s director that includes instructions for accessing the web-based questionnaire (Attachment K). The contact information for the Urban Institute’s help desk, including a toll-free number, will also be provided in the letter, along with the deadline date for submitting responses, which will initially be set for 60 days. Labs that did not verify their eligibility for the CPFFCL-14 and contact information during the initial verification stage will be able to do so at this point. To help encourage participation through web-based system, Urban will not initially enclose a paper copy of the questionnaire, although one will be sent to non-respondents four weeks into the data collection (Attachment D). A postage-paid envelope to the Urban Institute’s address will be included with the paper copy of the questionnaire for those that wish to mail in the completed form.

During the first month of the data collection, Urban will closely monitor activity on the web-based system, as well as the responses to each data item. After the first month, Urban will begin contacting each of the non-responding crime labs by email to verify receipt of survey materials and to confirm each lab’s designated point-of-contact. This contact will also provide an opportunity for Urban to answer any questions that an individual might have about the CPFFCL. Where necessary, new survey packets will be sent to labs that did not receive the original one or that changed directors. Another 30-day deadline will be set for this initial non-respondent group.

Following the initial deadlines, a multi-stage follow-up approach will be initiated. In the first stage, an automated email reminder will be sent to non-respondents on the day after the deadline. Next, Urban will call each non-respondent reminding them of the importance of the CPFFCL and that assistance in completing the questionnaire is available. If a non-respondent does not submit a completed questionnaire within the next two weeks, a telephone call will follow from Susan Narveson, a consultant on the CPFFCL-14 project who is a former lab director and ASCLD president. This call will once again emphasize the importance of the CPFFCL and note the endorsements from nationally recognized professional organizations. During this period, email reminders will also be sent to non-respondents and those that have started but not completed the questionnaire.

Verifying and validating the submitted data

Each response received from the crime labs to the questionnaire will be reviewed and edited by the Urban Institute. They will contact the respondents, as needed, to clarify responses or obtain missing information. Prior to contacting the respondents, Urban will examine the data for inconsistencies and ensure that the responses comply with BJS’s coding specifications. The following is a summary of the quality assurances that will be implemented during the data collection period:

Data editing: Throughout the data collection, Urban will review each submission from the crime labs for completeness and accuracy and reconcile missing or erroneous data through a series of manual and automated validation checks. All paper-based submissions will be manually reviewed and entered into the web-based data collection system where the data will receive additional automated checks. For example, if a screening question was left blank, but the follow-up questions were completed, a manual edit could be made to indicate the intended positive response to the screening question. Automated skip patterns built into the web-based data collection system will ensure that the questions presented to each respondent are applicable to the lab’s operations and significantly reduce the editing needed on the data. In addition, each lab’s 2014 budget, personnel, and workload counts will be compared to those reported in the previous CPFFCL data collections and any significant increases or decreases will be verify with the respondent. The total number of requests received and backlogged in 2014 reported in items D1 and D2 will also be compared for consistency to the combined total of requests reported separately for each individual forensic service in items D3 through D17.

In the web-based data collection system, respondents will be able to login and make revisions as many times as they wish during the data collection period up to the point when they decide that the questionnaire is complete. To submit, the respondent will navigate to a complete and submit page. When they access this page, a system-initiated validation procedure will be automatically run and incomplete items or those that otherwise do not pass a validation check will be identified. These will be listed for respondents to reconsider and correct. However, designation of the questionnaire as complete and submitting it will not require changing items that failed the validation check. Rather, respondents will be advised that Urban will contact them to follow up on items that had been flagged.

Data retrieval: As responses are submitted into the web-based system, the information will be made available to BJS to monitor the data collection. In cases where data are missing, incomplete or unclear, Urban will attempt to contact the respondent to clarify any problematic responses. Urban will document these issues and the methods used to resolve them in an automated tracking system.

Data entry: Responses submitted via the web-based system will be automatically captured, and paper responses will be manually entered into the web system. When respondents are contacted regarding missing or incomplete responses, they will have the option to either enter the data through web system or provide the information by phone or hard copy to Urban who will then entered the data into the system. To confirm that editing rules are being followed, Urban will review frequencies for the entered data after the first 10 percent of cases are received and processed. Any issues will be investigated and resolved. Urban will review frequencies of the data items and perform logic checks throughout the collection period to evaluate the quality and completeness of the responses.

Adjusting for non-response

There are two major types of non-response within the CPFFCL data collection: “unit,” when a crime lab answers no questions, and “item,” when a crime lab only answers some questions. While every effort will be made to obtain 100 percent unit and item response rates on the CPFFCL-14, some data will inevitably be missing at the end of the collection period. During the most recent CPFFCL in 2009, BJS obtained a 97 percent unit response rate. One way to address missing data will be to provide respondents who could not complete the entire questionnaire because of time or reporting constraints with the option of completing a shortened version of the questionnaire that only requests key information, such as staffing, budgets, workloads, and quality assurances. In the CPFFCL-09, a shorter form with the basic census items was sent to 12 non-responding labs in a final effort to improve response rates. This process helped to ensure the statistical information on the most critical data elements was nationally representative.

Several methods will be used to account for item non-response and reduce bias in the national estimates on staffing, budget, and workload totals. Depending on the amount and nature of missing data, Urban will work collaboratively with BJS on an imputation plan that is deemed both cost-efficient and suitable for the project. Once all missing values have been imputed, the database can then be analyzed using standard techniques for complete data. The imputation procedures will be fully documented within the database and codebook. Flag variables will be included to indicate where imputations have occurred, with the values indicating the nature of the imputation.

Non-response weighting adjustments will also be used to compensate for unit non-response. The first step under the weighting approach involves computing the sampling weight or base weight, defined as the inverse of the selection probability. Since we are conducting a census of crime labs, the base weight will be one for all agencies. The second step adjusts the base weight for non-response through a weighting class or response propensity-based adjustment to compensate for the non-respondents. Crime labs will be grouped together (based on characteristics such as budget, staff size, and workload) to form weighting classes and adjust the weights of respondents so that they represent the non-respondents within the same weighting class. The final weighting procedures will be determined empirically based on a non-response bias analysis to identify variables that influence non-response. BJS plans to produce separate weighting procedures for the agencies that meet the traditional project definition of a crime lab and those that solely analyze digital evidence and were added to the data collection in 2014.

3. Methods to Maximize Response

BJS obtained high response rates on the CPFFCL in 2005 (90 percent) and 2009 (97 percent). Given that the scope of the CPFFCL will expand (for the first time) to include all publicly funded digital evidence labs, the CPFFCL-14 will include a response rate on the approximately 410 agencies that meet the traditional project definition of a crime lab and a separate response rate on the estimated 100 public labs that solely analyze digital evidence. The response rate for each group is expected to be 90 percent or more.

The design of the CPFFCL questionnaire is consistent with the leading research on methods to maximize response, as outlined in Dillman, Smyth, & Christian (2010). Related questions are grouped together in topical sections. In addition, questions, instructions, and definitions are presented in a consistent manner throughout the data collection instrument to make each data item easier to comprehend. Proper alignment and vertical spacing is also used to increase the ease of reading the response options for each question.

The content of the CPFFCL questionnaire was designed with the input from forensic experts and practitioners from all parts of the country and all levels of government. BJS and Urban have sought the counsel of the forensic community to ensure that the questionnaire reflects the current issues of interest to the relevant community, avoids overly burdensome or irrelevant questions, and maintains the core structure of past instruments. Maintaining a core group of questions already familiar to the crime labs on staffing, budgets, workload and quality assurances is vital to maximizing response.

To encourage labs to participate in the CPFFCL-14, ASCLD will post announcements in its newsletter prior to and during the data collection. These announcements will alert respondents to expect the questionnaire, as well as provide examples of some of the more burdensome items to help respondents prepare the necessary information to efficiently respond. This publication is widely received by crime lab directors nationwide.

The data collection launch coincides with the largest national forensic science conference in the United States hosted by AAFS. Members of the project team will attend the conference, announce the data collection launch, and make themselves available on-site for one-on-one assistance with crime laboratory staff.

Urban will utilize a variety of techniques to obtain high response rates, including the option for crime labs to response through a web-based questionnaire or paper-copy via mail or fax. Follow-up operations will include email, mail, and telephone prompts. Mailing attempts will include letters of support from ASCLD and NIJ to convey the importance of the data collection. In addition, a subject matter expert (who is a former lab director and ASCLD president) will reach out to non-respondents to prompt their completion of the questionnaire.

Phone outreach designed to encourage response will be guided by data captured in Urban’s online performance monitoring and response tracking system. This system will log all contacts, in all modes (e.g., email or telephone), and be updated in near real-time. Urban will use these data to devise weekly call strategies designed to target non-respondents, respondents close to completion, and those with specific issues to address. Whenever possible the call team will reduce burden by aiding sites with calculations or their understanding of data items. When terms in the questionnaire differ from individual laboratory terms, call-team members will help respondents correctly interpret between the two terminologies to report accurate data.

Urban will also employ several data collection strategies that were effective at maximizing response rates during the CPFFCL-09. Both the web-based and paper versions of the questionnaire will contain instructions and definitions for the data items. In addition, extensive support and assistance will be available to respondents during the data collection. A help desk at the Urban Institute will be available through a toll-free number during normal business hours. Each team member from Urban will be assigned to separate groups of crime labs to establish a good rapport with the respondents and their assistants.

4. Testing of Procedures

Pilot test methodology

A pretest version of the CPFFCL-14 questionnaire was administered to nine crime labs in order to assess new or revised data items and revised response options, as well as to calculate an estimate of respondent time burden. These agencies included the Utah Bureau of Forensic Services, Allegheny County (Maryland) Medical Examiner’s Office, Indiana State Police Crime Laboratory, Virginia Department of Forensics, Baltimore County (Maryland) Police Crime Laboratory, Kansas City (Missouri) Police Crime Laboratory, Montgomery County (Maryland) Crime Laboratory, Orange County (California) Crime Laboratory, and the Department of Veterans Affairs Office of the Inspector General.

An initial list of more than a dozen labs was generated and ranked based on recommendations from the project consultant and her knowledge of labs and their directors. County, municipal, state, and federal labs were included in this list to ensure all groups would have the opportunity to participate and test items that were applicable to them. Contact logs and response times from the CPFFCL-09 were also examined to determine potential labs for the pilot study. In addition, data collected from the CPFFCL-09 were used to determine (1) whether labs had a laboratory information management system (LIMS), which can ease the reporting burden, to include respondents in the pilot test with and without LIMS and (2) the types of forensic requests that labs completed in 2009 to obtain a variation in workload. In addition to these considerations, several labs that processed digital and multimedia evidence were included in the pilot test to provide an additional layer of input on the new questions and definitions related to this topic.

Description of process

Urban began recruiting labs for the pilot test by phone during the first week of April 2014. Once the lab director had confirmed that he or she was willing to participate, a packet of materials was sent to the director by email and mail. The packet contained an introductory letter from BJS, an instruction sheet for returning the questionnaire, a pretest version of the questionnaire, and a survey asking about completion times and resources expended to complete the questionnaire. Respondents tested a 43-item instrument that was inclusive of the CPFFCL-09 items and new topical items on emerging issues in forensic science. Labs were given one month to complete the questionnaire. Urban sent reminder emails and made telephone calls to labs that had not responded by the deadline. Due to unforeseen circumstances that eventually impacted their ability to participate in the pretest, Allegheny County did not complete the questionnaire prior to the final deadline.

Following receipt of each lab’s responses, Urban conducted a debriefing interview with each respondent to discuss the experience of completing the questionnaire, obtain opinions on the data items being considered for revision, and clarify any unclear responses. The response times ranged from 15 minutes to almost six hours (over the course of several days), and the average time to complete the survey per agency was 2.9 hours. While the majority of respondents indicated that the information requested was readily available, it was at times necessary to contact other individuals in order to complete certain items. Conversations with pretest sites did not elicit strong complaints about the questionnaire or overall burden in completing it. Each site provided helpful feedback on certain items and sections, which has been incorporated into the questionnaire. Further details on the feedback provided by pretest sites and the changes made to the instrument are available in Attachment I.

5. Consultation Information

The BJS Law Enforcement Statistics Unit takes responsibility for the overall design and management of the activities described in this submission, including data collection procedures, development of the questionnaire, and analysis of the data. BJS contacts include the following:

Matthew R. Durose

Statistician

Bureau of Justice Statistics

810 Seventh St., NW

Washington, DC 20531

202-307-6119

Andrea M. Burch

Statistician

Bureau of Justice Statistics

810 Seventh St., NW

Washington, DC 20531

(202) 307-1138

The project director at the Urban Institute is:

Kelly A. Walsh, Ph.D

Research Associate

Urban Institute

2100 M St., NW

Washington, DC 20037

202-261-5434

6