Instrument - CISA Exercise Interview Guide

OMB CONTROL NUMBER: 1670-0027

OMB EXPIRATION DATE: 05/31/2024

CISA Exercises In-depth Interview Guide

Hello. Thank you so much for taking the time today to talk with us. As we mentioned when we emailed or spoke with you previously, the Cybersecurity and Infrastructure Security Agency (CISA) partnered with Team Grant Thornton to conduct an evaluation to learn more about what happens after a customer participates in exercises.

We would like to get your perspective on how things are going since you participated in your latest CISA Exercises session with CISA and received your After-Action Report (AAR). We are also interested in learning what recommendations your organization and its partners have been able to implement and/or what barriers or challenges you’ve encountered with implementing the recommendations in the AAR.

We want to assure you that what you say is confidential and that identified responses will not be released. Also, please know that no specific names, of either individuals or organizations, will be identified in our reports, and if we include quotes, they will not be attached to a person’s name or organization. Additionally, this is not an assessment of you or your agency; it is intended to help inform CISA’s products and services.

Today’s call will last up to an hour. We may follow up with you or others afterwards if we have remaining questions after reviewing our notes.

Do you have any questions before we get started?

Paperwork Reduction Act

The public reporting burden to complete this information collection is estimated at 10 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and the completing and reviewing the collected information.  The collection of information is voluntary. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB control number and expiration date.  Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to DHS’s Office of the Chief Procurement Officer, Office of Acquisition Policy and Legislation, 7^th and D Street, Washington, D.C.,  ATTN: PRA [OMB Control No. 1601-0027].

• Introductions/ (Who is on the call and) what is/are your title(s) and primary responsibilities at (ORGANIZATION NAME)?

• Please tell me about your knowledge and experience with security threats at (ORGANIZATION NAME)?

• How long have you been at (ORGANIZATION NAME)? How long have you been in this position?

• Before we talk about the recommendations in the AAR, I’m interested in learning more about your participation in specific parts of the exercise process .

[Note: some of these questions are asked directly but if we have this information from other sources (i.e. intake forms) we will confirm this information during the interview]

• Were you the person from your organization that made the initial request?

  • If yes, what prompted you to put in a request?

• My understanding is that you participated in the pre-planning meetings.

  • Did you find those helpful? Why/Why not?

  • Now that you have completed the exercise, is there anything that could have been done differently during those planning meetings? Or, is there something that could have made the pre-planning meeting better for the overall process?

  • Did you feel that the right participants from your organization and partner organizations were in the pre-planning meetings? If not: Who did not attend that should have?

• Did you attend the [EXERCISE NAME] on [DATE]?

  • If yes: What were your overall impressions?

  • What was your role in the exercise? Probe if needed: Were you an observer, player, planner, controller, facilitator, evaluator, simulator, or something else?

  • Do you think the right people from your organization or partner organizations attended? If not: Why did they not attend?

  • Is there anything that you think was missing from that exercise that you think would’ve been helpful to your organization?

  • Were the scenarios presented in the exercise relevant to your organization?

Did you read the After-Action Report?

• How soon after the exercise did you receive the report?

• What were your overall impressions of the report?

• Were the identified gaps and recommendations clear?

• Did you agree with the gaps and recommendations? If not, what did you disagree with and why?

• Were there any corrective actions based on the exercise that were taken by your organization even before you received the AAR? If yes, what actions did you take and why?

• Have you taken, or plan to take, additional corrective actions beyond those identified in the report?

• Ok, let’s talk about (some of) the recommendations your organization received in the After-Action Report. What would you say was the most important recommendation in the AAR?

• Who is most impacted by this recommendation? Probe if needed: your organization, your partners, customers, staff, third party vendors?

• What is the status of this recommendation as of today? Did your organization implement that change; are you in the process of making that change, or has your organization not made that change?

  • If implemented: Walk me through the process of how the change was made. Were there any challenges with implementing this change? How quickly was that change made?

  • If in progress: Have you encountered any challenges or barriers with implementing this change? How did you overcome these challenges, or are you still facing them? When do you expect to be able to complete the process?

  • If not implemented: Why has your organization not implemented this recommendation? What barriers/challenges did you run into? Do you expect that the recommendation will eventually be implemented?

    • If yes: If you had to guess, when do you think it would be implemented?

    • If no: Why not?

• Do/Did you consider addressing this issue as a priority for your organization? Why or why not?

• What was your level of commitment to implementing this recommendation when you received the AAR? (very high, high, moderate, very low, low)

• What is your level of commitment to implementing this recommendation now?

• How would you rate the level of commitment of others within your organization? How about partner agencies?

• Ok, moving on to the second most important recommendation.

• Who is most impacted by this recommendation? Probe if needed: your organization, your partners, customers, staff, third party vendors?

• What is the status of this recommendation as of today? Did your organization implement that change; are you in the process of making that change, or has your organization not made that change?

  • If implemented: Walk me through the process of how the change was made. Were there any challenges with implementing this change? How quickly was that change made?

  • If in progress: Have you encountered any challenges or barriers with implementing this change? How did you overcome these challenges, or are you still facing them? When do you expect to be able to complete the process?

  • If not implemented: Why has your organization not implemented this recommendation? What barriers/challenges did you run into? Do you expect that the recommendation will eventually be implemented?

    • If yes: If you had to guess, when do you think it would be implemented?

    • If no: Why not?

• Do/Did you consider addressing this issue as a priority for your organization? Why or why not?

• What was your level of commitment to implementing this recommendation when you received the AAR? (very high, high, moderate, very low, low)

• What is your level of commitment to implementing this recommendation now?

• How would you rate the level of commitment of others within your organization? How about partner agencies?

IF THERE IS TIME, REPEAT QUESTION SERIES ABOUT ADDITIONAL RECOMMENDATION OR IF THERE IS ANOTHER ONE IN PARTICULAR THEY WANT TO DISCUSS

• If some changes were implemented but not others: How did your organization decide what recommendations to implement first, and which to delay or not implement? How were priorities set?

• Now that X amount of months have passed since you received the AAR, thinking back through the whole process and what your organization has learned, what has been the most valuable takeaway?

  • How has your organization benefited as a result of this exercise?

• If you could go back and change anything in the exercises process, is there anything you would do differently? Why?

• Did you make any other changes to your organization as a result of the exercise?

If yes, please specify

• Have you recommended CISA’s exercise services to others? If yes, who?

• What advice would you give another organization who was considering requesting an exercise?

Closing remarks: Thank participants for their input and provide contact information for future questions or concerns.