CPFFCL_SupportingStatement-PartB_20250128

Part B. Collection of Information Employing Statistical Methods

1. Universe and Respondent Selection

The 2024 Census of Publicly Funded Forensic Crime Laboratories (CPFFCL) will use procedures successfully employed for the 2020 CPFFCL to identify the universe of eligible labs. The 2024 CPFFCL will be conducted among all federal, state, and local crime labs that meet the following eligibility criteria— 

 

1. Employs one or more full-time analysts whose principal function is the examination of physical or digital evidence in criminal and investigative matters;  

2. Provides reports and testimony to courts of law with respect to such evidence; 

3. Does not engage exclusively in evidence collection and documentation (such as finger print recovery and development, crime scene response, and photography).  

 

The CPFFCL excludes all private labs and publicly funded agencies that engage exclusively in:  

1. Evidence collection and documentation, such as latent print recovery and development, crime scene response, and photography; or 

2. Analysis of digital evidence and perform no other forensic functions. 

BJS and RTI used the American Society of Crime Laboratory Directors (ASCLD) membership list and the 2020 CPFFCL frame as a starting place for updating the 2024 CPFFCL frame. In addition, the current BJS Census of State and Local Law Enforcement Agencies (CSLLEA) (OMB #1121-0346, expires on 8/31/2025), the current roster of federal agencies in the BJS Census of Federal Law Enforcement Officers (CFLEO) frame, and the DEA National Forensic Laboratory Information Management System (NFLIS) (OMB #1117-0034, expires on 5/31/2022) were used to verify and update the 2024 CPFFCL frame. About 30 new individual labs were added to the roster of 423 individual labs in the 2020 CPFFCL, increasing the expected total for the 2024 CPFFCL to 453. Among the individual labs that are part of a multi-laboratory system, BJS expects that in most situations the main laboratory will be able to respond on behalf of the entire multi-lab system, reducing the actual number of respondents from 453 individual labs to 350 standalone labs and multi-laboratory systems. Any duplicates among the approximately 453 labs in the 2024 CPFFCL frame will be removed based on identical or nearly identical addresses, phone numbers, and/or names of the lab directors. Frame verification (as approved under the OMB generic clearance approval OMB #1121-0339) with email and telephone outreach will be conducted to confirm or collect contact information for the CPFFCL frame.

As in previous administrations, the 2024 CPFFCL will be a census rather than a sample survey. The reasons for this decision include the following:

• Moving to a sample survey with a universe of this size will not result in significant cost savings given the stratification dimensions needed to capture critical aspects of the universe, such as size and government oversight. The differences in individual labs and multi-lab systems also complicates efforts to create representative sampling strata.

• A census provides BJS with an opportunity to show how crime labs vary. Being able to compare crime labs is particularly important considering the variability that exists among these agencies in terms of administration, caseload, policies, procedures, resources, staffing, and infrastructure.

• Other federal, state, and local agencies are interested in CPFFCL data because the data can be used to describe the programs and infrastructure needs. With a census design, these data can be used to support expansion and enhancement of crime labs through funding from the Department of Justice and other sources.

• A census provides an opportunity to build a foundation for conducting future surveys of crime labs by other federal agencies. Completing the CPFFCL, for example, will provide the information necessary to produce samples based on a more comprehensive and fuller understanding of how each lab operates given the variability that exists within and across states. Attempting to generate samples of labs without this crucial information would be more time intensive and costly.

• While not essential for national estimates, the small increase in effort to conduct a census over a sample would allow BJS to report on labs in all 50 states and Washington DC.

2. Procedures for Collecting Information

Data Collection Procedures

The 2024 CPFFCL will be a multi-mode data collection approach using web as the primary mode, with hard copy data collection as an alternative, and email, mail and telephone follow-up as needed. The data collection period will last approximately eight months and will involve initial invitations, several reminders, and an end-of-study letter. In addition, there will be data quality and non-response follow-up (see Table 1 for schedule).

Table 1. Outreach schedule for CPFFCL

Stage	Timing	Type of contact	Attachment number(s)
Survey invitation letter (with URL and login instructions) and endorsement letter posted to web and linked	Week 1	All	F,G
Email invitation (with URL and login instructions) and CPFFCL flyer	Week 2	All	H,I
ASCLD survey launch announcement	Week 3	All	J
First reminder email and postcard	Week 4	Non-respondents	K
Data Quality follow-up phone/email	Week 6	Respondents with errors	Q
Second reminder email/letter mailed with questionnaire and business return envelope	Week 6	Non-respondents	L
Third reminder email and mail	Week 9	Non-respondents	M
Fourth reminder letter	Week 12	Non-respondents	N
Fifth reminder email and postcard	Week 14	Non-respondents	O
Telephone prompting	Week 17	Non-respondents	R
Sixth reminder email from ASCLD	Week 20	Non-respondents	P
Critical items letter and survey sent with business return envelope; login credentials provided	Week 24	Non-respondents	T
Critical items email	Week 25	Non-respondents	U
Critical items prompting calls	Week 27	Non-respondents	R
End-of-study letter	Week 29	Non-respondents	V
End-of-study email	Week 30	Non-respondents	W
Close data collection	Week 32
Completion thank you email	Rolling	Respondents	X

The data collection will begin with an invitation letter mailed via the United States Postal Service (USPS) to the head of each lab to inform them about the survey (Attachment F). The survey invitation letter will be signed by the BJS Acting Director and will stress the purpose and importance of CPFFCL and the need for participant’s participation. Further, it will notify the recipient of the survey due date, provide instructions for submitting the survey online, and will provide the project Help Desk email address and telephone number should there be any questions. Included with the invitation will be a letter of support from ASCLD (Attachment G). About 5 business days after the mailed invitation letter is sent, an email invitation and CPFFCL flyer will be sent to those directors/designees for whom an email address is available (Attachments H-I). This invitation will closely align to the mailed invitation letter and will contain a hyperlink to the web survey and endorsement letter. Two weeks after the invitation is sent, ASCLD will reach out to their membership by email encouraging participation in the study for those invited (Attachment J).

Approximately three weeks after the invitation is sent, we will email the first reminder letter and postcard to nonrespondents (Attachment K). Beyond the initial reminder letter, over the course of approximately eight months, a total of six reminder mailings and emails will be sent to those who have not yet responded and will alternate between mediums to keep the survey reminders fresh to the respondents (Attachments L-P). Both the mail and email reminders will contain information for completing the web survey.

Approximately five weeks after data collection begins, RTI will begin reviewing the data received. As data discrepancies or missing data values are discovered, RTI staff will conduct data quality follow up with respondents via telephone or email to clarify responses or obtain missing information (Attachment Q).

Approximately four months before the survey due date, telephone follow-up with nonrespondents will begin as described earlier (see Attachment R). Respondents will be reminded of the purpose and importance of the survey and informed of the goal of receiving a completed survey from each office. They will be asked to submit the survey online but will be sent another hard copy version of the survey if requested. Up to 8 calls will be made by RTI until surveys are received (or an office refuses to participate) and will reference the most recent communication (e.g., reminder letters, reminder emails, etc.).

By month 5, BJS will determine if a shorter critical item survey is needed to bolster response rates. During the 2020 CPFFCL, certain questions that were deemed to be critical for BJS to obtain, including staffing, budget, and workload data; during the previous data collection, RTI staff followed up with labs on a case-by-case basis to ensure these critical items were answered for each respondent, even if other non-critical items were left unanswered. BJS has already identified the critical items that would be needed for the 2024 CPFFCL (Attachment S). Should this effort be necessary, we will begin critical item capture following the sixth reminder email. The abbreviated critical items survey would be addressed to the office head with an invitation to fill out the critical items survey on the web or via paper response via the USPS (Attachment T). This mailing would include a business return envelope to facilitate paper response. An email would be sent to all agency heads about 5 business days after the critical items survey packet was sent (Attachment U). Within each letter and email, the login credentials and the Help Desk email and toll-free telephone number would be provided. If deemed necessary, nonresponse follow-up that would incorporate critical item data capture would take place 3 weeks after that mailing. Using this approach, when the 2020 CPFFCL ended data collection, there was an overall response rate of 90% and 95% responded via the web.

RTI will send the end-of-study notification both via mail and email to notify nonrespondents that the study is coming to an end and that their response is needed within two weeks (Attachments V-W). Data collection will continue for approximately three more weeks to allow for receipt of the remaining questionnaires. Immediately after surveys are submitted respondents receive a thank you email (Attachment X). These emails will thank them for the time and effort necessary to complete the survey. The text will formally acknowledge receipt of the survey and state that the agency may be contacted for clarification once their survey responses are processed.

Data Processing

Upon receipt of a survey (web or hardcopy), data will be reviewed and edited, and if needed, the respondent will be contacted to clarify answers or provide missing information. Respondents who submit via web will be prompted with real-time validation checks when submitting missing or inconsistent data. Any unresolved items that remain after the respondent submits will result in recontact by RTI staff to the respondent to attempt to resolve these issues.

The hardcopy survey will be developed and keyed using a fillable PDF. This will ensure that the same post-collection data quality review procedures, which mirror and expand upon the web validation checks, are applied to all survey data, regardless of response mode. The following is a summary of the data quality assurance steps that RTI will take during the data collection and processing period:

Data Editing. RTI will reconcile missing or erroneous data through automated and manual edits of each questionnaire. In collaboration with BJS, RTI will develop a list of edits that can be completed by referring to other data provided by the respondent on the survey instrument. For example, if a screening question was left blank, but the follow-up questions were completed, a manual edit could be made to indicate the intended positive response to the screening question. Through this process, RTI can quickly identify which hardcopy cases require follow-up and indicate the items that need clarification or retrieval from the respondent.

Data Retrieval. When the project team identifies a potential data issue, such as missing or inconsistent answers, an RTI professional staff member will contact the respondent for clarification. Throughout the data retrieval process, RTI will document the critical questions needing retrieval (e.g., missing or inconsistent data elements), request clarification on the provided information, obtain values for missing data elements, and examine any other issues related to the respondent’s submission.

Data Entry. Respondents completing the survey via the web instrument will enter their responses directly into the online instrument. For those respondents returning the survey via hardcopy (mail or fax), data will be keyed by hand once received and determined complete. To confirm that editing rules are being followed, RTI will review frequencies for the entered data after the first 10% of cases are received. Any issues will be investigated and resolved. Throughout the remainder of the data collection period, RTI staff will conduct regular data frequency reviews to evaluate the quality and completeness of data captured in both the web and hardcopy modes.

3. Methods to Maximize Response Rates

The 2020 CPFFCL achieved an 90% response rate for critical items. BJS and RTI will undertake various activities to ensure that high response rates are again achieved for the 2024 CPFFCL. CPFFCL will use a web-based instrument supported by various online help functions to maximize response rates. A toll-free number will also be provided for both substantive and technical assistance. RTI staff will respond to these requests for assistance.

The previous CPFFCL administrations received widespread support by ASCLD, which was enlisted to help with the development of the questionnaire and to encourage individual labs to respond to the survey. This continues to be the case for the 2024 CPFFCL.

The survey instrument was reviewed to ensure that it captured the most relevant information, removing any unnecessary questions to reduce burden. An item-level assessment of the 2020 CPFFCL was conducted to look for patterns of non-response. Items with lower response rates were reviewed to determine whether to be removed from the survey or revise. In addition, the questionnaire was reviewed for ease of use, flow, and additional survey methodology best practices to ensure ease in administration by expert panel reviewers and by BJS and RTI.

To promote 100% item completion by respondents, RTI will monitor item response rates as surveys are submitted. RTI will have a survey management system linked to the web-based application that will flag missing items and invalid responses. RTI will also flag missing items on hard copy submissions on a flow basis. The data collection manager will oversee phone and email outreach to respondents to clarify missing or invalid responses and to take corrective action. Changes to survey responses obtained through this follow-up effort will be tracked and entered in the data collection database.

Nonresponse Adjustments

As the 2024 CPFFCL is planned to be a census of publicly funded crime labs, sampling weights are not necessary. However, to ensure that nonresponding labs are not fundamentally different than those that participate, a nonresponse bias analysis will be conducted on the agency-level response rates to develop adjustment weights to account for unit nonresponse in the statistical estimates, using the same weighting methods from the 2020 CPFFCL. Administrative data on the government-level of the lab (i.e., federal, state, county, or municipal) and lab size according to the number of full-time personnel will be used in the nonresponse bias analysis as weighting classes. RTI will compare the distribution of respondents to nonrespondents to determine if the potential for bias varies by agency type and size. Weights will be calculated such that the weights sum to the frame population totals within each weighting class. Because this is a census, design weights are effectively “1” and every lab in the frame is eligible to participate in the census. Thus, the nonresponse adjustment weight for the 2024 CPFFCL will simplify to the inverse proportion of nonrespondents to respondents for every responding lab in the weighting class. Among the labs that responded to the 2020 CPFFCL, item nonresponse was low with response rates of 90% or higher on critical items. Therefore, additional imputations were not used to adjust for item nonresponse in the 2020 CPFFCL. BJS does not plan to impute for item nonresponse in 2024 CPFFCL either as item response rates are expected to be high.

Final Testing of Procedures

The proposed new questions in the 2024 CPFFCL and the revisions made to those retained from the 2020 CPFFCL were reviewed by BJS and RTI staff, suggested and discussed by the expert panelists, and cognitively tested with nine laboratories. The cognitive testing provided insight into whether respondents fully understood questions, provided expected answers, informed our phrasing and response options, and provided an estimate for burden. The instrument was modified to increase comprehension as a result of these interviews. In addition, RTI will thoroughly test the web-based survey administration system through systematic user testing, including testing skip patterns, attempting to “break” the instrument, and back-end data checks on entered responses.

The 2024 CPFFCL will maintain similar respondent recruitment and support procedures as the 2020 CPFFCL, which was field tested and successfully employed. RTI has used web-based survey instruments for other BJS surveys including the Census of Medical Examiner and Coroner Offices (CMEC), Law Enforcement Management and Administrative Statistics (LEMAS) and Census of State and Local Law Enforcement Agencies (CSLLEA). The web-based survey procedures successfully employed for these data collectionswill be substantially retained but modified as necessary to accommodate the 2024 CPFFCL instrument and respondents.

4. Contacts for Statistical Aspects and Data Collection

1. BJS contacts include:

Matt Durose

CPFFCL Program Manager

202-598-0295

[email protected]

Charlotte Lopez-Jauffret

Statistician

202-812-6754

[email protected]

Alexia Cooper

Law Enforcement Statistics Unit Chief

202-307-0582

[email protected]

2. Persons consulted on statistical methodology:

Heather Meier

RTI International

3. Persons consulted on data collection and analysis:

BeLinda Weimer

RTI International

Katherine Bollinger

RTI International

Carlye Arcas

RTI International

Ashley Murray

RTI International

Ryan Weber

RTI International

Kathryn Greenwell

RTI International

8