Protocol

PHMSA 5800.1 Testing Protocol

General Research Questions:

• How would filers describe their overall experience with completing the form?

• Do filers understand what information they are being asked to provide?

• What type of responses do they provide?

• (HAZMATICS-online form) How are filers interacting with the online navigation/functionality of the form?

  • Does the flow of the instrument screens support filers’ decision-making needs with regard to reporting options and instrument functionality? If not, what other information do they need, and what are the optimal locations and formats for presenting such information?

  • Do button labels and descriptions make sense to filers and inform them of the correct functions? If not, what terms do filers suggest?

  • Are filers able to discern the functions of individual screens? If not, what are the sources of confusion, and how do they suggest making them clearer?

  • What features did filers use when completing the form, and did these features meet their needs?

  • What additional features do filers want in the form?

  • What type of errors did filers encounter?

    • Are filers able to successfully resolve errors associated with the form?

• Are the instructions helpful to filers

  • Do filers understand the terminology used?

• Do filers utilize help materials (e.g., definitions, manual)?

  • If so, were the materials helpful in completing the form?

  • Do filers understand the terminology used?

  • Did the definitions/help content prompt filers to change responses/make corrections?

• How are the requested data maintained in records?

• How accessible is the information requested?

• How relevant is the requested information for each mode of transport?

• Do available response categories make sense to the filer?

Expected Length of Debriefing Interview: 1 hour

Required Materials

• 5800.1 Incident reporting form (paper/online)

Introduction

Thank you for your time today. My name is XX and I work here with the XXXX. I will be working with you today. We evaluate how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it.

Before we start, there is a form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to record the session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. [Usability] In addition, today we will be utilizing eyetracking technology to help us understand how you are interacting with the form online.

[Hand consent form; give time for participant to read and sign; sign own name and date, start recording.]

Thank you.

[Usability] Ok great. I now have a demographic questionnaire and a questionnaire about computer and internet experience that I’d like you to fill out.

[Have the participant complete the Demographic and the Computer Use and Internet experience Questionnaire]

Thank you.

• I am going to give you a little background about what you will be working on today. Today you will be helping us to evaluate the Hazardous Materials Incident Report form. We are interested in learning about your experience using the form and reactions to some redesign suggestions for the form. To do this, we will have you complete the form thinking about a recent incident report you have submitted. I may also ask you questions about your experience as we move through each section of the form. There are no right or wrong answers as we go through the form, we are mainly interested in your open impressions.

• [Usability]: Before we get started, I need to calibrate your eyes for our eyetracking technology. This will help us to understand how you are interacting with the form visually. Follow the red dot on the screen with your eyes without moving your head. Great, thank you.

[Administer general questions to learn more about the filer]

General Questions

ABOUT THE FILER

First, I’d like to learn a bit about you, and your role here at the company.

• What is your job title?

• How much experience do you have completing the Hazardous Materials Incident Reporting form?

• What has your experience completing the Hazardous Materials Incident Reporting form been like?

• Do you work with anyone else in your company to get the data requested in the form?

  • How are those other people involved?

  • How many people are involved?

• Do you have experience completing the paper version, online version, or both?

  • If both, which do you prefer?

Now I’d like to learn a little about your company.

• Can you give us a brief description of what your company does?

  • What kind of products do you transport?

[Administer interview probes to address research questions]

Interview Questions BY Section

As you go through the form, think of a recent incident you had to report using the Hazardous Materials Incident report form. As you go through each question, think aloud and verbally tell me how you came about your answer. I may ask additional questions as we go along to get your thoughts on particular sections of the form.

[Allow respondent to complete the section prior to administering probes]

Introduction

• Overall impressions of the instruction content?

• Is it clear whom should report the incident? When? Why?

• How could one obtain additional guidance completing the form?

Part I: Carrier reporter information

• For which types of incidents have you reported in the past?

• Overall impressions of this section of the form?

• Whose contact information would you provide?

• Are you familiar with your companies 9 digit Hazmat registration number?

  • How easy or difficult is it to access this information

Part II: Incident information

• What are your overall impressions of this section of the form?

• When providing information about the location- what do you typically provide (i.e., do you include descriptions, landmarks etc.)

  • How easy/difficult is this to report?

• [Existing form] Describe the transportation phases outlined in the existing form as they relate to your mode of transport. Are the options clear? Describe.

Part III: Shipment information

• How easy or difficult would it be to provide the requested details about the hazardous material involved in the incident? Describe

  • Hazard zone

  • EPA manifest number

• Is it clear what you should report for the Carrier information (personal, company)?

  • Is there an alternative way that we may be able to obtain this information?

• How easy or difficult is it to report the descriptions for the hazardous materials? Describe

  • What sources would you reference to provide this information?

• Are there any additional details that you think would be useful to provide regarding the shipment?

Part III: Packaging information (existing form)

• What are your overall impressions of this section of the form? Describe.

• How easy or difficult was it to identify how the packaging failed? Describe.

  • Did you refer the instructions in the guide?

    • If yes, tell me more.

• How easy or difficult would it be to identify the packaging type? Describe.

• How are materials like those reported in your recent incident report typically packaged?

• Where can the packaging information requested be found (Items 26-29)?

  • How easy or difficult would it be to obtain this information?

  • Which is the most challenging to obtain/report?

• [Radioactive transports only]

  • What are your overall impressions of this section of the form? Describe.

  • Is it clear what each packaging category represents? Any missing?

  • How easy or difficult would it be to obtain the packaging certification information requested? Describe.

  • Would any of the requested information in this section be difficult/impossible to report?

Part III: Packaging information (continued, revised form)

• To help determine additional changes for this section of the form tell us your thoughts on the following?

  • Reporting of how a package failed vs why it failed and what failed?

    • In your opinion how does these differ?

  • What type of information would you provide in the open field for additional details about the incident?

  • Thoughts on reporting by transport mode?

    • Based on the information outlined, in the existing form, which fields are most applicable for your mode of transport?

Part IV: Consequences

• What are your overall impressions of this section of the form? Describe.

• How easy or difficult is it to report the result of the incident?

  • Are there any categories that are confusing? Missing?

• Are you typically able to report the EMS/police report #?

• In your own words, what does in-house clean up mean?

• How easy or difficult is it to assess damages resulting from an incident? Fatalities? Injuries- both hospitalized and non-hospitalized?

Part V: Air Incident Information (existing form) [Air transports only]

• What are your overall impressions of this section of the form? Describe.

• How easy or difficult is it typically to report the occurrence of the incident? Phase of the shipment? Describe

• Is there any information not requested, that you think would be useful in describing the incident?

Part VI: Description of events and Package failure and Part VII: Recommendations

• What are your overall impressions of this section of the form? Describe.

• Describe what you would typically report in the open field for additional details.

Usability only debriefing questions [Administer at the conclusion of the form]

• [Review responses from the satisfaction questionnaire- probe about any items rated <3]

• Does the flow of the instrument screens aid in reporting to the incident? How so?

• How easy or difficult was it to navigate through the form

• How clear are the button labels and descriptions?

• Are the various sections of the online form clear? Describe each?

• Which features of the online form did you find to be the most helpful/useful? Which can be improved? Added?

• Discuss any areas of the form where difficulty navigating was observed during testing

• Overall, what was your impression of the form?

• Is there anything you liked / disliked about the form?

• In general, would you say that it was easy or difficult to complete the form using the online application? Why?

• If you could change anything about the online form what would that be?

Wrap-up

We are almost done, and I just have a few wrap-up questions.

• What are your overall impressions about the revised form?

  • Layout?

  • Organization of questions?

  • Additional details missing from the revised version of the form that may be useful (with the exception of the packaging/air incident information requested)

• Is there anything else we haven’t touched on today, that you think is relevant?

• Do you have any questions for us?

Thank you very much for your time today!

General probes that may be used throughout the interview:

• Reflect back on answers: “you said…”

• In your own words, what is this question asking?

• How did you arrive at this number/answer this question?

• What records (if any) did you look at? What line or lines were of interest?

• Specifically, what did you include in reporting this information? What did you exclude?

• Did you consult other people to obtain this answer?