CPDO Cog Test Memo to OMB

I would but then I wI’m sorr

MEMORANDUM

TO: Robert Sivinski

Office of Statistical and Science Policy

Office of Management and Budget

THROUGH: Kevin M. Scott

Acting Director

Bureau of Justice Statistics

Shelley Hyland

Senior Statistical Advisor

Bureau of Justice Statistics

Heather Brotsos

Deputy, Statistical Operations

Bureau of Justice Statistics

Suzanne M. Strong

Chief, Judicial Statistics Unit

Bureau of Justice Statistics

FROM: Erica Grasmick

Statistician, Judicial Statistics Unit

Bureau of Justice Statistics

DATE: January 10, 2024

SUBJECT: BJS requests to conduct cognitive testing for the Census of Public Defender Offices (CPDO), under the OMB generic clearance agreement (OMB Number 1121-0339).

Summary of Current Request

BJS requests clearance under its generic clearance agreement (OMB Control Number 1121-0339) to: 1) contact up to 20 leaders of statewide public defense organizations or leaders of county (or other jurisdiction) public defense organizations to complete and return the survey; 2) contact up to 20 replacement public defense leaders to replace refusals; 3) conduct debriefing interviews with up to 20 public defense leaders from that sample, and 4) conduct usability testing of the programmed instrument with up to five public defense leaders.

Introduction

The Bureau of Justice Statistics (BJS) is planning its second iteration collecting data on public defense providers through the Census of Public Defender Offices (CPDO). The CPDO is the latest in a series of BJS-sponsored research efforts dedicated to the study of public defense systems and providers. The CPDO will enhance a growing BJS Public Defense program by generating a complete census of all public defense offices which will serve as the foundation for administering future surveys capturing key aspects of office operations nationwide.

A previous generic clearance request covered the frame building for the CPDO. Under this generic clearance request, BJS and its data collection agents, NORC at the University of Chicago (NORC), the Urban Institute, and the National Association for Public Defense (NAPD), propose to cognitively test the 2023 CPDO survey instrument. The cognitive test will involve up to 20 public defender office leaders selected based on a series of elements representative of public defense administration across the U.S. The office leader or designee will be asked to complete the survey and participate in a one-hour debriefing interview to review the overall burden of the survey and the challenges with individual questions.

After compiling feedback from the cognitive test, the survey will be programmed in Voxco. Up to five public defense leaders will complete a usability test of the instrument online. The five participants will be informed that the purpose of the test is to assess the online data collection mode and not the survey itself. After completing the survey and submitting it online, the project team will schedule a call with the participants.

History of the Collection

Within BJS’s Public Defense portfolio are the 2007 Census of Public Defender Offices (CPDO, OMB Control Number 1121-0095, expired 7/31/2010), the National Survey of Indigent Defense Systems (NSIDS, OMB Control Number 1121-0095, expired 5/31/2017) as a survey in 1999 and attempted census in 2013, and the Survey of Public Defenders (SPD) Pilot test, conducted under OMB generic clearance 1121-0339 in 2021. The 2007 CPDO was a census of all public defender offices in centralized and decentralized states. The CPDO provided administrative data on the staffing, budgets, training, and caseloads of public defender systems. The NSIDS collected similar information as the CPDO, except that it included all forms of indigent defense – public defender, contract counsel, and appointed attorneys. The NSIDS was conducted as a survey of all indigent defense providers in 1999, and as an attempted census in 2013. The statewide NSIDS was successful, with 100% response rate. The county-based NSIDS had a 68% response rate. Data were not published from the county-based portion of NSIDS. The SPD pilot tested the ability of state and county public defender offices to generate rosters of their practicing public defenders, and of the public defenders to answer a survey about their work as a public defender.

Survey Instrument Revisions

The 2023 CPDO will add to our body of administrative surveys by updating the 2007 CPDO information. The updated instrument is a balance of established questions addressing continuing public defense issues as well as new questions capturing changes in the field over the past 16 years. For instance, the instrument continues to collect detailed expenditure information, staffing distributions, caseload metrics, eligibility standards, public defender attrition rates, and salary ranges. Additionally, it addresses the increased burden on investigators and attorneys associated with enhancements in technology-based evidence. At the recommendation of the expert panel, questions pertaining specifically to death penalty case expenditures have been removed due to the burdensome nature of those questions and concerns that the responses may lack reliability. Questions addressing how an office handles conflict cases have also been removed in an effort to limit burden on office chiefs while preserving the critical information of interest to stakeholders.

To prepare for the cognitive test, the project team reviewed the 2007 survey instrument. The project team considered the expected burden for each question and whether the items are critical for a census of public defender offices in 2023 (the reference year). The project team flagged problematic questions for review and convened an expert panel to assess the revisions, offer further revisions or suggest new areas of inquiry. The resulting instrument has 5 sections: general office information, staffing, caseload, eligibility for services, and office resources (Attachment A).

Frame and Sample Selection

The 2023 CPDO frame was updated under a previous generic clearance request. In determining who to select for cognitive interviewing, BJS and the project team considered the major structural and contextual differences in public defense offices across the country and developed a list of characteristics to include when selecting cognitive participants. The different office profiles may influence the availability and accessibility of financial and institutional data. By including at least one office from each of these categories, the team seeks to identify whether there are barriers in the survey instrument for offices that fit into each subset. Selected offices may fulfill more than one criterion (e.g., an urban office with an elected public defender). Although the “office that participated in the 2007 CPDO or 2013 NSIDS” category does not reflect a specific office characteristic, including such an office could provide insight about this survey instrument compared to previous surveys. Additionally, we will select geographically diverse offices across the country.

BJS intends to identify 20 offices based on the following criteria:

• Single county office in a county-based public defense state

• Multi-county office in a county-based public defense state

• Single county office in a state with public defense districts/circuits

• Multi-county office in a state with public defense districts/circuits

• Statewide office

• Office with a mix of state/county funding

• Contracted non-profit

• Office in an urban jurisdiction

• Office in a rural jurisdiction

• Office with an elected public defender

• Office that participated in 2007 CPDO or 2013 NSIDS

• Regional office

• Specialty offices, for example:

  • Capital

  • Appellate

  • Juvenile

  • Misdemeanor

  • Mental Health

Testing Procedures

Under a previous generic clearance request, the project team finalized the universe of public defender offices in the U.S. BJS plans to identify the statewide and county offices for the cognitive test upon receipt of OMB approval.

BJS seeks OMB approval to survey up to 20 leaders in statewide systems and public defense leaders in counties, county clusters, or other combined jurisdictions, such as regional offices or judicial districts. Additionally, BJS seeks OMB approval to test the usability of the programmed instrument with up to 5 public defense offices leaders. Participation in the cognitive and usability tests is voluntary.

The cognitive test will be conducted using a paper version of the instrument to center respondent feedback around the survey content rather than usability and functionality. Once the cognitive test results have been incorporated into the instrument, the project team will program the instrument and conduct a small usability test with up to five chief public defenders. Respondents identified for the usability test will differ from those who completed the cognitive test.

Informed Consent and Data Collection

Prior to participation, the identified public defense office leader will be informed via email (or by mail if BJS does not have a valid email) of the purpose of the survey and cognitive test (Attachment B). This email will provide the elements of informed consent. The initial invitation provides the purpose of the CPDO, the voluntary nature of the study, how the respondent was selected, and information about whom to contact with questions about the study. The emails will be addressed to the chief public defender in the office. The participant is likely to be the chief public defender or designated to a public defender knowledgeable about the entire office. The respondent will be asked to respond to the email to provide consent for participation.

If a potential participant is unresponsive to the initial email, the project team will follow-up with a phone call to request participation. In the event that the potential participant refuses or we are unable to secure their participation, a replacement respondent matching the initial respondent’s selection criteria will be contacted immediately. The project team will follow a similar protocol if a respondent’s consent is initially secured and then withdrawn. BJS will continue this process until 20 participants consent and are interviewed or a maximum of 40 participants are contacted.

After obtaining consent, respondents will receive the PDF survey (Attachment A) and instructions (i.e., how to complete the survey, the voluntary nature of the survey, the importance of noting challenges experienced during the survey, and the expected length of time to complete the survey; see Attachment C) via email and mail. Respondents will be instructed to:

1. complete the survey by highlighting their answers in Adobe, if they are able to, and return via email; or

2. print out the survey, complete and scan the survey to return via email; or

3. complete the hard copy and return via self-addressed stamped envelope.

Respondents will be asked to track the time they start and finish the survey as well as the number of individuals consulted to complete the instrument. The survey also includes an optional notes section at the end to record item numbers of challenging or unclear questions. If a question is too difficult or would require too much time to answer, the respondent will be asked to estimate the time it would take to answer the question and proceed to the next question. The participant will be asked to complete the instrument in this manner prior to the cognitive interview. Respondents will be asked to return the survey prior to the cognitive interview to provide areas of focus for the interview. Once the project team has received the response, a team member will reach out to the respondent via email to schedule the cognitive interview.

The cognitive interviews will include questions about how the respondent interpreted the questions, how much time was necessary to answer challenging questions, and whether the questions required the respondent to research the response or involve other staff members (Attachment D). The interviewer will ask the respondent if any questions were unclear, any response options were missing, any questions were missing, or any questions should be revised or reconsidered because of respondent burden. The project team will make every effort to incorporate the data provided by the cognitive test respondents into the full dataset and not contact those offices again. The interviews will be recorded (with interviewee consent) to allow the interviewer to focus on conducting an informative interview rather than capturing the details of the responses.

Participants will not receive any compensation for their time, but a thank you email will be sent to each respondent within 48 hours of the interview (Attachment E). The project team will review the feedback and determine any revisions to the survey instrument. Table 1 provides the timeline for the informed consent and data collection activities.

The usability test will follow a similar protocol of outreach to the cognitive test. That is, BJS will secure the consent and participation of up to 5 public defender leaders by reaching out to a maximum of 10 potential respondents. Participants will receive an email requesting their participation in the usability test and, if unresponsive, receive a subsequent phone call (Attachment F). Once participants have consented to participation, they will receive usability test instructions with the survey link via email (Attachment G). Upon survey completion, usability tests will be scheduled via email and conducted via video conference to uncover instrument compatibility issues, programming bugs and ease of use (Attachment H). Upon completion of the usability test interview, participants will receive a thank you email (Attachment I).

Table 1. Informed Consent and Data Collection Timeline

Task #	Task Description	Number of participants contacted	Method of contact	Timing of contact
1	Contact potential cognitive test participants via email	20	Email, mail	Day 1
2	Phone call to request participation from nonrespondents	20	Phone	Day 5
3	Contacts to replace participants who decline	Up to 20	Email, mail	Up to day 14
4	Respondents complete and return survey	Up to 20	Email, mail	Up to day 28
5	Email to schedule cognitive interview	Up to 20	Email	Up to day 28
6	Cognitive interviews	20 total completed	Video Conference	Day 5-day 40
7	Contact potential usability test participants via email	Up to 10	Email, phone	Day 40-day 50
8	Conduct usability test	5 total completed	Email, phone	Day 50-day 60

Reporting

Upon completion of the cognitive interviewing, NORC and Urban Institute will provide BJS with a report describing the findings including final recommendations. It will also include any suggested changes to the instrument based on the cognitive interviewing. The report will provide detailed information on the testing methodology, respondent characteristics, and findings related to the new or revised questions tested.

Burden Hours

The burden hour estimates for the participants are provided in Table 2. The project team expects the initial contact for recruitment and scheduling the cognitive interview to take ten minutes per participant. Up to 40 public defenders may be contacted to secure 20 cognitive interviews. The estimated burden to complete the 2023 CPDO instrument prior to the interview is 60 minutes per respondent. The cognitive interview will require up to a 60-minute video conference interview with each participant.

An additional maximum of 10 public defense leaders may be contacted to secure 5 instrument usability interviews. Recruiting for the usability tests is estimated to take ten minutes per participant. The burden for respondents to complete the CPDO instrument online is 60 minutes per respondent. BJS estimates a 60-minute video conference interview with each usability participant. The total burden time for all contacts under this request is 58.4 hours.

Table 2. Burden Hour Estimates for Respondents

Task #	Task Description	Number of respondents	Estimated burden (in minutes)	Total burden (in hours)
1	Public defender outreach (email or mail) – cog test	Up to 40	10	6.7
2	Public defender survey completion	20	60	20
3	Public defender cognitive interview	20	60	20
4	Public defender outreach – usability test	Up to 10	10	1.7
4	Public defender usability test	5	120	10
	Total burden			58.4

Cost to the Federal Government

The estimated annual cost to the Federal government is $19,999.88 for NORC at the University of Chicago, the National Association for Public Defense, the Urban Institute, and the Deason Criminal Justice Reform Center portion of the work and an estimated $1,610.88 for 32 hours of the GS-13 project manager’s work for a total of $21,610.76.

Data Confidentiality and Security

BJS is authorized to conduct this data collection under 34 U.S.C. § 10132. The testing will collect identifying information including the name of the testing organization, address, email, and telephone number. All information related to the cognitive interviews and usability tests, including the recordings of interviews, will be stored on a secure drive at NORC and Urban Institute with restricted access to those staff members who are directly involved in cognitive or usability testing. Draft reports will be shared, via email, with all project team members. To protect the identities of the respondents, including the cognitive interviewees, no identifying information will be kept in the final data file. In addition, the recorded conversations of the interviews will be erased one month after completion of the cognitive testing report. Once the instrument is revised and the summary report completed, all copies of the cognitive interview data will be destroyed. The responses to the survey will be entered electronically for incorporation into the full collection. Paper copies of the survey responses will be retained until the end of the project and destroyed in accordance with the project’s data destruction guidelines.

All project staff are required to sign a pledge of confidentiality and privacy certificate which confirms the maintaining of data and following the procedures outlined above. All cognitive interviews will be conducted by project staff at Urban Institute.

Protection of Human Subjects

The protocol described in this OMB Generic Clearance was submitted for review by the NORC institutional review board and it was determined that the activities as described by the protocol do not meet the definition of human subjects’ research (Attachment J). The research was certified as not meeting the definition of human subjects’ research as described under Department of Justice regulations at 28 CFR 46.

Project Timeline

Milestone	Start Date	End Date
Obtain OMB generic clearance	01/11/24	-
Recruitment and testing period	Upon OMB approval	30 days after OMB approval
Analyze data and develop instrument recommendations	2/1/24	2/29/24
Draft cognitive report	2/1/24	3/15/24

Contact Information

Questions regarding any aspect of this project can be directed to:

Erica Grasmick

Statistician

Bureau of Justice Statistics

U.S. Department of Justice

810 7^th Street, NW

Washington, DC 20531

Office Phone: (202) 598-7355

E-mail: Erica.Grasmick@usdoj.gov

Attachments

Attachment A: Census of Public Defender Offices survey

Attachment B: Contact scripts to participate in cognitive interviews

Attachment C: Instructions to complete survey

Attachment D: Cognitive interview script

Attachment E: Cognitive test thank you script

Attachment F: Contact scripts to participate in usability testing

Attachment G: Usability test instructions

Attachment H: Usability test

Attachment I: Usability test thank you script

Attachment J: IRB approval

10