Example of Instrument_Usability_Cognitive Interview Protocol_2022

Usability/Cognitive Protocol for Form EIA-XXX

My name is __________________ and I work for the Energy Information Administration. Thank you for agreeing to participate in our study.

What: In order to help us improve our surveys, we turn to our respondents, like you, to find out if our questions make sense and are easy to understand and answer. We have found that the best way to do that is to actually conduct the survey with respondents and see how it works for them. We then evaluate how easy or difficult EIA products are to use. What works well, we keep. When users, such as yourself, have difficulty with something, we have an opportunity to fix it. You will be helping us test the redesigned XXX Form EIA-XXX.

Confidentiality: Our session today is completely confidential. Any names you provide will never be used in our reports. Your participation in this study is voluntary and you can decline to answer any particular question. In addition to me, there will be a person from the program office. His/Her name is _________ (Name of PO). ___________ (Name of PO) can answer any technical questions you may have.

Think aloud method: Today you will be completing an EIA survey form. I want you to answer the questions exactly the way you would if you were completing it on your computer in your office, but with one major difference. I would like you to think aloud as you answer the questions. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice: Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[IF NEEDED:]Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

Great that’s what I want you to do throughout our session.

I will remind you to think aloud if you get quiet. From time to time, I’ll ask you some questions about your answers, or about the questions themselves. Remember, there are no right or wrong answers, because only you know what you are thinking.

If you run into any difficulties as you work on the survey, please let us know as this will help us improve the survey.

Ok, let’s begin:

You would have received a welcome letter, this form and instructions at your office. Please answer the questions as you would if you were in your office answering the form.

Do you currently respond to Form EIA-XXX, XXX?

Give the respondent some time to read the instructions. If needed remind the respondent to think aloud (whenever possible).

Instructions Section: If respondent hesitates or expresses confusion: “Tell me about your hesitation here.”

1. After R finishes reading the instructions: What are your thoughts about the instructions?

   1. If necessary: What if anything did you find confusing?

2. After reading the instructions on how to report… How easy or difficult is it to file the report?

   1. What, if any, are potential issues you foresee in filing the report?

Ok let’s move on to the first section, please complete this section.

Instructions Section: If respondent hesitates or expresses confusion: “Tell me about your hesitation here.”

1. After R finishes reading the instructions: What are your thoughts about the instructions?

   1. If necessary: What if anything did you find confusing?

2. After reading the instructions on how to file… How easy or difficult is it to file the report?

   1. What, if any, are potential issues you foresee in filing the report?

Ok let’s move on to the first section, please complete this section.

Part 1 Identification Information Section: If respondent hesitates or expresses confusion during the task: “Tell me about your hesitation here.”

(Wait until participant completes this section before asking questions)

3. Who would typically complete and file this form?

4. Are there any other relevant information that you think we need to ask here?

Ok let’s move on to the second section, please complete this section.

Part 2 Submission/Resubmission: If respondent hesitates or expresses confusion: “Tell me about your hesitation here.”

(Wait until participant completes this section before asking questions)

5. What, if any, is unclear in this section?

   1. If yes, what makes this unclear?

Ok let’s move on to the third section, please complete this section.

Part 3 xxx: If respondent hesitates or expresses confusion: “Tell me about your hesitation here.”

(Wait until participant completes this section before asking questions)

1. What, if any, is unclear in this section?

2. What does the term “XXX” mean to you?

3. Are there any items that are missing from the item description?

4. What does “XXX” mean to you?

Conclusion: General Questions

5. What are your thoughts about the color and design of the survey form?

1. If you would have to change something in the form, what would it be?

6. Approximately how long do you think it will take you to complete this form?

2. How long would it take you to gather the information you need?

3. How long would it take you to fill in the information and submit to EIA?

4. Does your outlet submit price or volume data to any other organizations? If yes:

   1. How frequent do you submit data to these organizations? If needed… Is it on a frequent basis or one time/ad hoc?

   2. About how long does it take you (or the preparer) to gather and submit the information?

   3. How is the information transmitted? If needed… Is it manual process or automatic/passive process?

7. Do you have any other feedback regarding your experience completing this form?

Thank you for your participation in this study!