Form 0923-0047 User Testing Script and Questionnaire

Attachment B.

User Testing – Script and Questionnaire

Form Approved

OMB No. 0923-0047

Exp. Date 1/31/2022

CDC estimates the average public reporting burden for this collection of information as 90 minutes per response, including the time for reviewing instructions, searching existing data/information sources, gathering and maintaining the data/information needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to a collection of information unless it displays a currently valid OMB Control Number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden to CDC/ATSDR Information Collection Review Office, 1600 Clifton Road NE, MS D-74, Atlanta, Georgia 30333; ATTN: PRA (0923.0047).

Date: __________

Interview number:

Good afternoon. (Introductions if not done already) Thank you for joining us and helping us in getting feedback on our new interface for our Community Reception Center Planning Tool. This tool is meant to help planners evaluate their current resource capacity and potential resource needs for population monitoring during a radiation emergency.

Background Questions

Before we get started, we have a few background questions to ask:

Have you participated in emergency response planning?
Y N	Any planning
Y N	Radiation specific
Y N	Community reception center/population monitoring

Have you participated in emergency response exercises?

4. Y N Any exercises

5. Y N Radiation specific

6. Y N Community Reception Center/population monitoring

7. Are you familiar with the term community reception center?

   • Yes

   • No

     1. A Community Reception Center is the location where potentially contaminated individuals would be directed during a radiation emergency. These individuals would be screened for contamination, decontaminated if needed, and be entered in a registry for follow up after the event.

8. Are you familiar with CDC’s population monitoring guide?

• Yes

• No

• I don’t know

9. Have you used a throughput-planning tool for any exercise or event? Examples of throughput tools include CRC-STEP or RealOpt for Points of Dispensing.

• Yes

• No

• I don’t know

Narrative: We have recently developed a new tool that emergency planners can use to understand their current capacity, potential bottlenecks, and additional resource needs when responding to a radiation emergency, specifically for population monitoring. This tool focuses on typical or anticipated activities that are needed to conduct population monitoring, which includes but is not limited to providing services such as: basic first aid, contamination screening, decontamination, registration and mental health counseling. Our goal is that this tool will help planners assess their current population monitoring capacity and help them plan for potential needs in a way that is simple to understand, quick to interpret, and can be taken or presented to decision makers if/when they need to ask for additional resources. The tool was developed using modelling software and incorporates real timing data collected from CRC exercises across the country.

Narrative: To help us assess how intuitive our tool is I’m going to ask that you go through the process of entering your existing population monitoring resources to estimate throughput capacity using our tool. If you are unsure or do not have population monitoring plans, we can provide a typical list of resources for a basic community reception center. As you slowly complete the query and observe the results feel free to “think out loud” and express what you are observing about the tool. We may also interject from time to time with specific questions or additional query parameters for you to use. There is no right or wrong answer to our tasks. We are simply judging how well our user interface helps guide you. Do you have any questions at this time?

Narrative: We will start off with a very simple scenario and we’ll observe you as you enter the information into the tool. Before you execute it, by hitting the “estimate” button, I’ll ask you to pause so I can ask a few questions.

Simple Input

Narrative: We are going to simulate a radiation dispersal device emergency response. Using the resource list given to you and the scenario provided, please fill in the tool inputs to estimate your throughput. The arriving population is 1000, and the predicted percent of population assumed contaminated is 25%. Please wait to click the “estimate” button.

1. Where the instructions presented on the opening screen of the tool useful? Y : N

2. What additional instructions would be helpful to input your resources?

   1. Feedback:

3. Are the labels on the inputs clear? Y : N

   1. If there were labels that were unclear, please list them and explain what you think they mean?

      1. Feedback:

4. Did you notice the tool tips and if so, were they helpful? Y : N

   1. If No, Feedback:

5. Can you click on the showers drop-down menu? Does the zero option make sense to you?

   1. Yes : No

6. What do you think it means?

   1. Expected : Other

   2. Feedback:

7. User was able to fill in inputs: Easy : Difficult : Needed help

   1. Feedback :

8. What information do you expect the tool will give you when you ask it to “estimate”?

   1. Feedback:

9. Did you understand the sequential process of making a query? Y : N

Simple Results/Outputs

Narrative: Press the “Estimate” button and let’s observe what results you get.

10. Is this what you expected to see? Y : N

    1. Feedback

Data Extrapolation Here we will be looking to see if the user is able to find specific outputs on the interface and provide us with the “right” answer. For certain headers such as in questions 15 and 16 we are looking to see if they understand the intended meaning and provide the definition of that header (expected) or a different definition (other) which would mean that clarification needs to be made for those sections.

11. What is your average hourly throughput for this CRC? Easy : Difficult : Needed help

12. What station has the longest average wait time? Easy : Difficult : Needed help

13. What is the total Throughput at the end of the shift? Easy : Difficult : Needed help

14. What is the extended time past shift? Easy : Difficult : Needed help

15. What do you think “extended time past shift” means? Expected: Other

Feedback:

16. What do you think “average time spent at CRC” means? Expected: Other

Feedback:

Graphs

Narrative: Take a moment to look at the graphs.

17. Do the three charts presented make sense with regards to what information they are showing and how it’s organized in the charts?

    1. Yes: No

    2. If No, please explain what information presented is unclear and why.

18. Is the labeling on the axis clear?

    1. Yes: No

19. Looking at the utilization graph, what station has the highest utilization, and what does that represent to you?

1. Expected : Other

20. Please identify the station with the longest line and the station name.

    1. Expected : Other

21. Knowing this is the longest line, how would you reduce the line length from what you see here?

    1. Expected : Other

22. Which graph provides you with the strongest justification to add a resource and why?

    1. Wait : Utilization : Length : Unsure

Feedback:

23. Would you like it if the program made an educated suggestion about exactly what resource to add based upon the modeling it is using as a resource?

    1. Yes: No : Don’t care

24. If you do, would you like a description of how it concluded what resource to add?

    1. Yes : No

25. Can you describe what you would need in the description if one was available?

    1. Feedback:

26. Do you understand the modeling used to create this tool?

    1. Yes : No

    2. Feedback:

27) Would it help you if we described the modeling assumptions?

Yes: No

Feedback

27. Would you and your management trust the results from this tool?

    1. Yes : No

    2. Feedback:

Tables

Narrative: Click on the table tab and observe what is presented to you.

28. Does the table presented make sense with regards to what information is showing and how it’s organized in the table?

    1. Yes: No

29. Is the column labeling clear?

    1. Yes: No

Saving/Exporting Data

Narrative: Please try to export the data.

30. Go thru the steps of saving, is this intuitive?

    1. Yes: No

    2. If no, Feedback:

31. Open the text document produced by saving the estimate. Take a moment to look over its entirety. Is there anything missing or not presented clearly? If so, please describe.

    1. Yes: No

    2. Feedback:

32. Open the excel document produced by saving the estimate. Take a moment to look over its entirety. Is there anything missing or not presented clearly? If so, please describe.

    1. Yes: No

    2. Feedback:

Summary Questions

33. Based off the results, what information is most useful when providing justification to decision makers so they can provide additional support to you?

1. Feedback:

34. Based on the scenario and resources you input, can you achieve your current anticipated throughput goals? This would mean that all persons were serviced within the actual shift time and there was no need for extended shift time.

    1. Yes: No: Unsure

    2. If unsure – When looking at the summary data, what parameter would suggest that your goals were achieved?

       1. Expected: Other

35. What is the best way to improve throughput?

    1. Expected: Other

36. After using this tool and viewing its summary information, do you feel empowered to plan to conduct population monitoring in your locality?

    1. Yes: No: Unsure

37. Did this tool provide any information/results that you weren’t expecting?

    1. Yes: No

    2. If yes, Feedback:

38. What additional information could the tool provide that might help you in your planning to conduct population monitoring?

    1. Feedback:

39. There are pros and cons when developing web-based or PC-based tools. For example, web-based tools can be updated without the user’s need to continually download new versions or install on their computer. PC-tools can be used with or without internet connectivity. For you and your agency, is it easier to use web-based tools vs tools installed on a PC?

1. Web Tool: PC Tool

Narrative:

This is the last question we had for our testing. Is there anything more you would like to add, or do you have any questions for us?

Thank you very much for helping us assess the user interface of the CRC SimPLER prototype.

9