Cognitive Interviewing for 2022 Census of Law Enforcement Training Academies

2022 CLETA: Cognitive Interview Protocol

DATE: ___ ___ / ___ ___ / 2 0 22

M M D D

START TIME: ___ ___ : ___ ___ AM / PM

Section 1: Introduction

Hello, I’m [NAME] with RTI, speaking with you on behalf of the Bureau of Justice Statistics.

Thanks for agreeing to help us develop the questionnaire for the 2022 Census of Law Enforcement Training Academies, or CLETA. The call will take about an hour. I’d like to start with a short summary of the goals for today’s call and explain a bit about how I’ll conduct the interview.

• IF NO LONGER A GOOD TIME, OFFER TO RESCHEDULE

• IF STILL GOOD TIME, CONTINUE

As you may know, the Bureau of Justice Statistics and RTI are preparing to conduct the CLETA in early 2023. As we get ready for the study, we are asking academy staff like you to review the draft questionnaire. During this call, I’ll ask for your reactions and feedback to these draft questions– including things like how the questions are worded, your understanding of concepts, ways to clarify instructions, and the information your academy tracks that is related to the survey questions.

Your participation in this interview is completely voluntary. You may stop at any time or skip questions that you do not want to answer. The data we collect will be de-identified, meaning that your personal information, such as your name, will never be stored together with your responses. With your permission, I would like to record this call for the sole purpose of capturing comments that I may miss when taking notes. The recording would only be accessible to me and would be deleted no later than 2 weeks from this interview when the responses from this call are finalized. Do I have your permission to record this call?

Please keep in mind that there are no right or wrong answers to my questions. One of our main goals is to draft questions that make sense, so if anything about the questions is confusing or unclear, you can help by pointing this out to me. Also, if you’re not sure how you would respond to any of the questions, please tell me that, too. Finally, please let me know if you believe any questions or topics were omitted, or any answer choices or response categories that were missing or insufficient.

I am interested in hearing all of your feedback on the survey, but because there are a lot of topics to discuss and we only have an hour, sometimes I might ask that we move on to the next question before you’ve had a chance to share everything on your mind. At the end of the interview, you can share any important feedback that you didn’t have a chance to share earlier.

Do you have any questions before we begin?

Before we start discussing the questions, did you review all questions and answer them or think about how you would go about answering them?

• IF YES: Do you happen to recall approximately how much time you spent reviewing and completing the selected portions of the questionnaire?

__________ HOURS __________ MINUTES  Go to Section 2.

• IF NO: [INTERVIEWER: RECORD NOTES ABOUT REASONS OF NOT COMPLETING THE INSTRUMENT. RECORD ANYTHING THE R MENTIONED EARLIER OR AFTER SAYING THEY DID NOT COMPELTE IT]

Section 2: Item reviews

I’m planning to discuss only some of the questions on the questionnaire, but if you have comments or concerns about any of the questions I skip, please feel free to share them with me at any time.

P1. (BOTH) The first question I’d like to discuss is Question 2. It asks about BASIC training. Please read the question to yourself.

PROBE: Do you have any difficulty understanding the definition of BASIC training?

PROBE: Is there anything you would add/remove to the definition of BASIC training?

P2. (GROUP 1) Now let’s review Question 3.

PROBE: Is it clear that this question asks about accreditation from a body that is not affiliated with a state POST requirement?

P3. (BOTH) Next I’d like to review Question 4 which asks about degrees and college credits.

PROBE: Is it clear what “partner institution” means here?

PROBE: Do you interpret this question to mean degrees and/or college credits are awarded based on completing BASIC training?

P4. (BOTH) Now we will move on to Question 8. It asks about the length of your academy’s BASIC training program.

PROBE: What do you think of this question?

PROBE: Can you tell me how you went about answering this question?

PROBE: What does, “core BASIC training program” mean to you?

PROBE: How do you interpret, “TOTAL number of hours required to complete basic training”?

How about, “Number of hours mandated by state POST…”?

PROBE: Did you read the instruction under the question? Did your response include any of these?

PROBE: Does your academy measure the length of BASIC training in hours?

[IF NOT] What process would you use to convert your answer if you needed to report the length of your program in hours?

P5. (BOTH) [ONLY ASK IF Q10 = YES] Please now read aloud Question 11. It asks about the length of field training for academies that oversee these programs.

PROBE: In your own words, what does, “…core field training program” mean to you?

PROBE: Does your academy measure length of its field training program in hours?

[IF NOT] What process would you use to convert your answer if you needed to report the length of your program in hours?

P6. (GROUP 2) Now let’s review Question 14. This question asks about how community members are involved with different aspects of your academy. Please read it to yourself.

PROBE: Do you feel this question is clear or unclear? Why?

PROBE: What does “directly involved community members” means to you as used in this question?

PROBE: In your own words, what do “formal” and “informal” methods mean to you?

[IF NO EXAMPLES ARE PROVIDED BY R IN THEIR DEFINITION: Could you give us an example of a [formal] and/or [informal] method?]

PROBE: FOR ANY YES RESPONSE: Can you tell me more about why you answered “Yes” to X?

P7. (BOTH) Now we are going to review questions related to academy personnel. We’ll start with Question 15. Please read the question and each category to yourself. Let me know if there are any descriptions that are unclear or difficult to understand.

PROBE: Does your academy record this information separately for full-time and part-time instructors?

PROBE: How difficult is it to obtain this information for each of the categories listed in the table?

PROBE: Do any of your trainers or instructors fit within more than one category?

[IF YES: PROBE TO FIND OUT IF THEY REPORTED THEM IN MULTIPLE ROWS. IF ONLY REPORTED IN ONE ROW, HOW DID THEY DECIDE WHICH ROW?]

PROBE: When determining who to count as full-time and who to count as part-time, did you think about their status as a trainer or instructor, or did you think about their status with respect to all job responsibilities, not just training?

P8. (BOTH) Now let’s go to Question 16. This question asks about education requirements for full and part time instructors.

PROBE: What challenges, if any, did you have in answering this question?

PROBE: Was it clear that you could only provide one response for each column?

P9 (BOTH). Question 17 asks about requirements for instructors as sworn officers.

PROBE: Did you have any challenges answering the field for, “Sworn officer status…”?

PROBE: [IF SWORN OFFICER OPTION IS SELECTED, DETERMINE HOW NUMBER OF YEARS OF EXPERIENCE WAS CALCULATED]

P10. (BOTH) Please now turn your attention to Question 18. It asks about certifications required for your academy’s instructors.

PROBE: Can you explain to me with your own words what this question is asking about?

PROBE: Is the format of the question clear? [PROBE TO SEE IF ITS CLEAR OR UNCLEAR THAT AN ANSWER IS REQUIRED FOR EACH ROW AND THE DISTINCTION OF FULL-TIME/PART-TIME]

PROBE: I am going to show you the same question, but with a different format. What do you think about this alternative format? Do you have a preference? [INTERVIEWER SHARES SCREEN AND DISPLAY ALTERNATIVE FORMAT]

P11. (GROUP 1) Now let’s review Question 20, which ask about refresher training for academy instructors.

PROBE: What does “general instruction” mean to you?

PROBE: Did you have any challenges understanding or responding to this question? How so?

P12. (GROUP 1) [ASK ONLY IF Q19 = 1 OR 3] Question 21 is a follow up of the previous question. Please review this question.

PROBE: What does “provided internally” mean as used in the context of this question?

PROBE: What does “internally-provided training sessions” mean to you?

P13. (GROUP 2) Next, I would like to discuss items related to academy resources. Please go to Question 22. This table contains five general categories of resources. Let’s look at section A. Please let me know if you find anything unclear or confusing [ASK FOR EACH SECTION AFTERWARDS. ASK PROBES BELOW AFTER THE CORRESPONDING SECTION IS REVIEWED BY THE RESPONDENT]

PROBE: [INDICATE OVERALL SECTION COMMENTS HERE]

PROBE: [AFTER SECTION B] What does “Use of force training simulators” this mean to you?

• Do you think this is similar or not to “firearms training simulators”?

PROBE: [AFTER SECTION C]: What does “Mental Health Services” mean to you?

PROBE: [AFTER R FINISHES READING ALL SECTIONS] Are there any resources that are missing from this list?

PROBE: How did you know what to select for each resource? For example, were you thinking about resources within your academy or also considering external resources?

P14. (GROUP 1) The CLETA questionnaire asks two questions related to your academy’s budget. Please have a look at Question 23. Please read through the question to yourself and let me know if anything is unclear or confusing.

PROBE: Can you tell me, in your own words, what you consider as total operating budget?

PROBE: [IF R COMPLETED THE SURVEY]: How did you define the total amount? [IF ESTIMATE – HOW DID THEY COME UP WITH THE ESTIMATE?]

PROBE: How difficult or easy do you think it would be to get an accurate value for this question? Would that value be an estimate?

P15. (GROUP 1) Now let’s look at Question 24. Again, please read through the question to yourself and let me know if anything is unclear or confusing.

PROBE: Can you tell me, in your own words, what you consider as budget for equipment?

PROBE: [IF R COMPLETED THE SURVEY]: How did you define the total amount? [IF ESTIMATE – HOW DID THEY COME UP WITH THE ESTIMATE?]

PROBE: How difficult or easy do you think it would be to get an accurate value for this question? Would that value be an estimate?

P16. (BOTH) The next section of the questionnaire contains questions about basic academy trainees. Let’s start by discussing Question 25.

PROBE: Did you have any challenges completing this question? What challenges did you have?

PROBE: [IF NOT COVERED] How difficult was it to obtain the values for this question?

PROBE: The question asks to check a box if your values are estimates and not exact. How likely would you be to provide an estimate as opposed to providing exact values?

PROBE: [IF ESTIMATE BOX WAS SELECTED] Why did you provide an estimate? / Do you have access to the exact values? / How difficult would it be to obtain exact values?

PROBE: Does your academy collect information on recruits who do not align with the categories “Male” and “Female” (e.g. non-binary, unknown)? How would you report such recruits? What response options would fit how you track recruits’ gender identity?

P17. (BOTH) Now I’d like to hear your feedback about Question 26. It asks about the number of recruits that started and completed basic training by race and ethnicity.

PROBE: Did you have any challenges completing this question? What challenges did you have?

PROBE: [IF ESTIMATE NOT CHECKED] Did you see the instruction under the question to check the box if your response was an estimate? Were any of your responses’ estimates?

PROBE: [IF THERE IS A VALUE IN “NOT KNOWN” ROW] Could you tell me more about those recruits included in the “not know” category? [PROBE TO UNDERSTAND IF THEY DON’T COLLECT DATA ON RACE/ETHNICITY]

P18. (BOTH) Imagine that the survey asked you the following question [INTERVIEWER SHARES SCREEN WITH QUESTION TEMPLATE]: “How many recruits, upon entering BASIC training were a) military veterans, b) Possessed an associate degree, c) Possessed a bachelor’s degree?”

PROBE: Would you be able to provide the total number of recruits that were military veterans?

PROBE: How about those have earned an associate’s degree?

PROBE: How about those have earned a bachelor’s degree?

PROBE: How burdensome would this question be to answer?

P19. (BOTH) Let’s look at Question 27. Please read each reason listed for why recruits did not complete BASIC training and let me know if anything is unclear or hard to understand.

PROBE: How difficult was it for you to report these data for each of the categories, by sex classification?

PROBE: Are these accurate values or are they estimates?

PROBE: Are there any other frequent reasons not listed in the table?

P20. (BOTH) The final portion of the questionnaire contains questions related to your academies core basic training curriculum. Let’s first review Question 28.

PROBE: What does “…training philosophy of academies…” mean to you as used in this question?

P21A. (BOTH) Question 29 asks you about the topics covered in BASIC training. If the topic is covered, then you are asked to provide the number of hours of instructions required. It also requests you to select if the number of hours is an estimate or if it is an exact number. Please read all topics under “A. Operations” and “B. Weapons/Defensive Tactics” to yourself.

PROBE: What do you think about these subject areas? Are any subject areas unclear?

PROBE: How easy or difficult is it for your academy to report the number of hours on each topic covered in BASIC training?

PROBE: For those topics covered, can you provide an exact number of hours or an estimate?

P21B. (BOTH) Please keep reading the remaining of the topics to yourself. Please let me know if you think there is any topic that is unclear to you, or you find any difficulty on responding.

PROBE: Do you think there is overlap between these subjects? IF YES: Which ones?

How would you report the hours of those overlapping subjects? [PROBE TO SEE IF R WOULD INCLUDE HOURS ON ALL SUBJECTS, ADD THEM TO ONE AND EXCLUDE THEM FROM THE REST, OR SOME OTHER WAY]

P22. (GROUP 2) Next, let’s go to Question 30.

PROBE: What does FORMAL ASSESSMENTS mean to you as used in this question?

PROBE: Do these names of formal assessment match with the way they are called in your academy? IF NO: Which one?

How is it called in your academy?

PROBE: [IF YES TO ANY] For those required assessments, how did you define the number of assessments required?

PROBE: Is the “Number of assessments required” a value that you can access at your academy? IF NOT READILY AVAILABLE: How difficult would it be to report on these?

P23. (GROUP 2) Question 31 asks about the use of role play scenarios for different activities during BASIC training.

PROBE: What does “role play scenarios and activities” mean to you as used in this question?

PROBE: Can you tell me what types of training do you think this term include? For example, do you think it includes less formal classroom scenario exercises?

PROBE: In the last CLETA, the term “reality-based (mock) scenarios” was used instead. Do you think there is a difference between the terms “role play scenarios and activities” and “reality-based (mock) scenarios?

PROBE: Are the categories provided in the list clear or unclear to you? Did you have difficulty understanding any of them?

P24. (GROUP 2) Now let’s review Question 32, which asks about various types of weapons used in training.

PROBE: Do you have any confusion on the use of the term “Revolver” in 31a? Do you have a better suggestion for a term to use?

P25. (GROUP 2) Moving to Question 33. This question asks about firearms or related training.

PROBE: In the last CLETA, 33a was referred to as: Firearms Training Simulator. Do you feel “Use of force training simulators” is better to include here? Why?

DEBRIEF QUESTIONS (BOTH)

P26. Now I’d like you to think about the entire survey. Did you have difficulty with any aspects of the survey that we have not already discussed?

P27. How do you feel about the length of the survey and the time needed to answer the questions?

P28. Before we conclude, do you have any other feedback or suggestions to improve the questionnaire?

Thank you very much for taking the time to provide your thoughts and feedback, it has been very helpful. We will combine your comments with feedback from other participants into an overall report. That report will help BJS evaluate the survey and determine whether to make changes to the questionnaire.

Thanks again!

END TIME: ___ ___ : ___ ___ AM / PM

3