Cognitive interviewing for the Law Enforcement Management and Administrative Statistics (LEMAS) supplement survey

Attachment C: LEMAS Supplement Survey Cognitive Testing Protocol

DATE: ___ ___ / ___ ___ / 2 0 22

M M D D

START TIME: ___ ___ : ___ ___ AM / PM

Hello, I’m [NAME] with RTI, calling on behalf of the Bureau of Justice Statistics.

Thanks for agreeing to help us develop the questionnaires for the 2023 Law Enforcement Management and Administrative Statistics Supplemental Surveys. This call will take about an hour. If this time still works for you, I’d like to start with a short summary of the goals for today’s call and explain a bit about how I’ll conduct the interview.

IF NO LONGER A GOOD TIME, OFFER TO RESCHEDULE

IF STILL GOOD TIME, CONTINUE

The purpose of this conversation is to get your feedback on draft questions for the Law Enforcement Management and Administrative Statistics, or LEMAS, survey program. The LEMAS program incorporates use of a core survey and periodic supplemental surveys to address special topics. The core and supplemental surveys are conducted every 4 years – with the next supplements taking place in 2023 and the next core taking place in 2025.

BJS is interested in exploring two topics through upcoming LEMAS supplemental surveys: a.) post-academy training and b.) officer mental health, well-being, and resources to support officer health. As we get ready for the supplemental surveys, we’re asking staff from law enforcement agencies to review the draft questionnaires. During this call, I’ll ask for your reactions to the draft questions – including things like question wording, clarity of instructions, and information your agency tracks related to the survey questions.

Please keep in mind that there are no right or wrong answers to my questions. One of our main goals is to draft questions that make sense, so if anything about the questions is confusing or unclear, you can help by pointing this out. Also, if you’re not sure how you would respond to any of the questions, please tell me that, too. Finally, please let me know if you believe any questions or topics were omitted, or any answer choices or response categories that were missing or insufficient.

I am interested in hearing all your feedback on the survey, but because there is a lot to discuss and we only have an hour, sometimes I might ask that we move on to the next question before you’ve had a chance to share everything on your mind. At the end of the interview, you can share any important feedback that you didn’t have a chance to share earlier. Your participation is completely voluntary and you do not have to answer any question you do not wish to, but your responses will help improve the LEMAS supplemental surveys.

Do you have any questions before we begin?

First, do you happen to recall approximately how much time you spent completing the questionnaires? And please include the time you and any others at your agency spent gathering information needed to answer the questions.

Post-Academy Training Response to Police Suicide Total

__________ HOURS __________ HOURS __________ HOURS

__________ MINUTES __________ MINUTES __________ MINUTES

I’m planning to discuss only some of the questions on the questionnaires, but if you have comments or concerns about any of the questions I skip, please feel free to share them with me at any time.

1. The first thing I’d like to discuss is the paragraph at the beginning of the Post-Academy Characteristics section. Please read the first paragraph, starting at “For purposes of this survey…”

   1. In your own words, what does “post-academy training” mean?



2. Looking at that paragraph and the bullet points that follow, is there anything confusing or unclear about this section?

IF YES: What do you find unclear?

2. Question 3 asks about the post-academy training budget for the fiscal year.

   1. How easy or difficult is it for you to report this number? What makes it easy/difficult?

2. What training costs are you including when reporting this number?

   1. How easy or difficult is it to exclude academy training costs from this number? What makes it easy/difficult?

3. Did you include any funding received for training through grants in this number? If NO: Why not?

3. Now let’s look at Question 4. It asks about the various ways you can provide full-time sworn personnel with training.

   1. The previous question referenced “the fiscal year that included December 31, 2022”, and this question references “the calendar year 2022”. Did you notice that? Are those time frames different or are they the same? IF DIFFERENT: How are they different?

2. How easy or difficult was it to answer this question with “Yes” or “No”? What makes it easy/difficult?

3. IF YES TO 4B OR 4C: How many officers did you send to receive outside training? IF NECESSARY: If your agency only sent 1 officer to outside training, would you have reported “Yes” or “No”?

4. For your agency, what does it mean to “provide any post-academy training”?

5. Would it be easier for you to answer this question with the number of full-time sworn officers who participated, or do you prefer the question the way it is?

4. I’d now like to discuss Question 5. It asks about the minimum number of hours of training required.

   1. How easy or difficult was it for you to answer this question? What made it easy/difficult?

2. What time period were you thinking about when you answered this question?

3. What other situations where a training might be mandated are not listed here?

5. The next question I’d like to discuss is Question 5a. It asks about how you developed trainings not mandated by law.

   1. In your own words, what is this question asking?

2. How did you decide on your answer for “State legislative requirements” and “Local legislative requirements”? How did you decide on your answer for “Consent decree”?

3. What time period were you thinking about when answering this question?

4. Was there anything about this question that was confusing or unclear?

6. Ok, let’s look at Question 6. It asks about specific trainings provided following a promotion or assignment.

   1. How did you interpret “N/A” in the context of this question?

2. IF YES OR NO TO CHIEF EXECUTIVE TRAINING: What does “training on the Chief Executive position” typically entail?

7. Turning to Question 7. It asks whether your agency authorized or provided various benefits to full-time sworn personnel.

   1. How easy or difficult was it to answer this question with “Yes” or “No”? What makes it easy/difficult?

2. How did you interpret “authorize” as it is used in this question? What does that mean?

8. I’d now like to look at Question 8. It asks about individuals serving as post-academy trainers or instructors.

   1. This question asks specifically about “trainers or instructors.” Did that lead you to exclude any other individuals who are paid to facilitate trainings?

2. What time period were you thinking of when you answered this question?

9. Now let’s look at Question 10.

   1. In your own words, what does the term ”certifications” refer to in this question?

2. Are any of the certifications listed unclear or confusing?

3. What additional certifications should be included in this question?

10. Question 12 asks about agencies/organizations your agency may have used for post-academy training.

    1. How easy or difficult was it to answer this question with “Yes” or “No”? What makes it easy/difficult?

2. What additional groups, agencies, or organizations should be included?

3. In your own words, what is a “community group”? IF YES TO NON-PROFITS: If “community groups” was not included under “Non-profit organizations,” would your response change to this question?

11. Let’s turn to the table in Question 14. This table asks about various kinds of content that could be covered by post-academy trainings.

    1. Were the instructions helpful or not helpful? How easy or difficult was it for you to fill out this table? What made it easy/difficult?

2. How easy or difficult was it for you to get the data needed to answer these questions? What made it easy/difficult?

3. How did you calculate the number of hours for the most recent training?

4. IF MARKED MORE THAN ONCE PER YEAR:

   1. How did you determine that a training is offered more than once per year?

5. Are there any column headings that are unclear or confusing? Are there any training subject areas that are unclear or confusing?

6. Are there any training subject areas that are not listed that you think should be added to this table?

7. When filling out this table, what time period were you thinking of?

8. How easy or difficult was it to answer in terms of programs offered since 2018? What made it easy/difficult?

9. How consistently have these programs been offered (or not offered) since 2018? How did that affect your ability to fill out this table?

12. I’d now like to look at the table included as part of Question 14a. It mentions possible specialized content areas that could be covered by post-academy trainings.

    1. Thinking about the section in the bottom left, labeled “Agency Technical” --

       1. Do you think any of these subjects overlap with subjects already listed in the previous table in Question 14?

2. Would any of these trainings be provided as part of the trainings listed for Question 14?

13. Now let’s go to the table for Question 15. This table lists additional training subject areas that we might ask more information about.

    1. Thinking about the training subjects listed in the table —

       1. Do you think any of these subjects overlap with subjects already listed in the previous tables?

          1. Were there any stand-alone trainings specifically on these subjects that you considered when responding to a previous question?

2. How common is it for stand-alone trainings to be provided on these specific subjects? (IF COMMON, PLEASE ELABORATE.)

3. Are any of the subject areas confusing or unclear? Are any of the column headings confusing or unclear?

14. Now, let’s look at Question 16. It asks about possible agencies or organizations you may have collaborated with to provide training. For [ITEMS ANSWERED AS YES], could you tell me a bit more about that?

15. Overall, when answering these questions about post-academy training, were you thinking about in-person training, web-based distance learning, or both?

16. Were there any questions or items you would recommend cutting from this survey? [FOR ANY RESPONSES] Could you tell me why?

17. What other issues did you see with the survey questions that we did not discuss?

END TIME: ___ ___ : ___ ___ AM / PM

[TRANSITION TO THE ORGANIZATIONAL PROGRAMS, POLICIES, AND TRAINING TO ADDRESS POLICE SUICIDE MODULE]

I would next like to ask about the questionnaire that focuses on resources responding to police suicide. Would you be able to answer questions related to this section, or would I need to speak with someone else?

R WILL ANSWER – CONTINUE WITH QUESTION 17

SOMEONE ELSE WILL ANSWER – COLLECT CONTACT INFORMATION, THANK RESPONDENT AND END CALL.

DATE: ___ ___ / ___ ___ / 2 0 22 [IF SECOND RESPONDENT]

M M D D

START TIME: ___ ___ : ___ ___ AM / PM

IF SECOND RESPONDENT: RE-READ INTRODUCTION AT BEGINNING OF PROTOCOL. First, do you happen to recall approximately how much time you spent completing this questionnaire? And please include the time you and any others at your agency spent gathering information needed to answer the questions.

__________ HOURS

__________ MINUTES

18. Question 1 in this survey asks about formal wellness programs that your agency might offer to full-time sworn personnel.

1. Is “wellness program” the best term to use?  Does your agency use a different term?

2. In your own words, what does the word “offer” mean to you as it’s used in this question?

3. If your agency had offered a single training that covered two topics in this table, how would you respond to this question?

4. IF MARKED “YES,” MANAGED BY GOVERNMENT: You marked “Yes, managed by government.” Does the government manage these programs directly, or through a contract?

5. If your agency offered a temporary wellness program, such as peer support for 3 months after an incident, would you include that here? Would you include it under one of the programs already listed or should it be listed on its own?

6. Are any of the column headings confusing or unclear?

19. Looking at Question 1A, it asks you to estimate the budget dedicated to formal wellness programs in calendar year 2022.

1. IF ANSWERED: How easy or difficult was it to answer this question? What makes it easy/difficult?

2. IF NOT ANSWERED: This question was not answered. Can you tell me more about why?

20. Looking at Question 1B, it asks about full-time agency employees who work on these wellness programs.

1. In your own words, what does the phrase “full-time agency employees work in wellness programs as their primary responsibility” mean?

1. PROBE: Would you include contractors under this definition?

2. When answering this question, did you include staff who were partially assigned to work on these wellness programs? Did you include contractors?

21. Let’s look at Question 2. It asks about the data collected and used to assess formal wellness programs offered.

1. How easy or difficult was it for you to answer this question? What made it easy/difficult?

2. What time frame were you thinking about when answering this question?

22. Question 3 asks about the confidentiality of your formal wellness programs. What does “confidentially” mean to you as it’s used in this question?

23. Question 4 asks about policies your organization might mandate related to officer mental health. Was there anything about this question that was confusing or unclear?

24. IF YES TO ANY ITEM IN QUESTION 6. Let’s look at Question 6. You answered “Yes” to [ITEM]. Can you tell me more about that?

25. Is there anything else about these questions that you would like to bring up?

Thank you very much for taking the time to provide feedback. We will combine your comments with feedback from other participants into an overall report. That report will help BJS evaluate the questionnaire and determine whether to make any changes.

Is there anything else you would like to talk about today?

Thanks again!

END TIME: ___ ___ : ___ ___ AM / PM

6