EIA-112 Usability Protocol and Interview Guide_9.4.2025

O MB Control Number: 1905-0186

Expiration Date: 7/31/2028

Usability Protocol and Interview Guide – EIA-112

Respondent ID:

Interviewer:

Observer:

Date:

Time:

(This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview. Interviewers may also ask emergent probes as necessary.)

Introduction:

Thank you for taking the time to talk with us today!

My name is _______ and I work for EIA. With me on this call today is ____. The purpose of this study is to assess the functionality of the Residential Utility Disconnections Survey, EIA-112. The way we will do this is by you sharing your screen with us as you complete the webform EIA-112. When you are done completing the questionnaire we will have a few questions for you regarding your experience completing this survey.

(If provided consent to recording) Thank you for allowing us to record this interview. As part of our procedures, I have to read the following: This MS Teams meeting is being recorded and will be used internally to supplement our notes. If you do not wish to have your voice recorded, please do not speak during the call or disconnect now. If you do not wish to have your image recorded, please turn off your camera or participate only by phone. If you speak during the call or use a video connection, you are presumed to consent to recording and to the use of your voice or image.

We would like you to go through the webform and complete it as you normally would. This is not a test of your ability, so just proceed in whatever way makes the most sense to you. You do not need to enter real data; you may enter mock data. The EIA-112 data you will be submitting during the usability interview will not be used in any way as the purpose of this study is to understand the functionality of the web instrument. If you need to get information from records or others within the company in order to answer a question, just let us know what you would need to do. Through usability testing we will be able to understand your experience in completing this webform and be able to improve the webform experience in future data collections. Thank you for allowing us to observe you while you complete the EIA-112.

[Only after reading this section, share the login information with respondent through MS Teams chat] In the Teams chat I will be sending you the login information for you to complete the EIA-112. Please share your screen prior to logging into the instrument so we are able to capture the process of you logging into the webform.

When you are ready you can begin. I’m just going to observe and take notes of any issues I see with the webform. As you are completing the EIA-112 webform, I may interrupt you to ask you some questions regarding your experience in certain sections of the webform. If you have any comments on the functionality of the webform or if there are survey questions that are confusing, feel free to let me know. I will have you stop at the end of the survey to ask some additional questions. Please do not submit until I tell you to.

Observation Session

As the respondent moves through the survey, make notes of any usability issues and ask concurrent probes.

Note any content issues that arise.

Login Screen

Note any areas of confusion. Use emergent probes when necessary.

Observation/Concurrent probes

1. Do you think that the information given on the Login Screen was sufficient?

2. What are your overall reactions to the process of logging into the system?

3. Is there any other information you’d like to see on any of the screens you saw while logging in?

4. Was it clear how to proceed to the next step?

Schedule 1: Part 1

Concurrent Probes (use the following probes if necessary):

1. Were there any issues in selecting a type of respondent in the dropdown menu?

   1. Any source of confusion (Probe if necessary)

2. Does the functionality (e.g., process of entering your information) of this page match your expectations? If not, please elaborate.

If there are any soft or hard edits use the following:

3. Is it clear to you based on the edit messages you are receiving on the screen that there is an issue? In your own words, can you explain to me what the problem(s) are?

4. Is it clear to you what action you need to take? What would you need to do if you wanted to ignore the message you were receiving and move on to the next question/screen?

5. Is there a better way that we could notify you about the problem(s) on this screen?

6. What other information should we provide you with in the error messages and how to correct them?

Schedule 1: Part 2

Observation/Concurrent Probes

1. Did the respondent read the definitions? If not, what did they do? (Note if later the respondent navigates back to the definitions from the next screen. If so, why?)

Example: I noticed you scrolled past these instructions, is there a reason you did so? Did you read these instructions in full, why not? What did you focus on?

Schedule 1: Part 3:

Observation/Concurrent Probes

1. Did the respondent encounter any issues on this screen? Please describe.

2. Did the respondent hover over “states”?

   1. If the respondent does not hover explain that they can hover over the “States”

      1. Ask, what do you think about this functionality.

If there are any soft or hard edits use the following:

2. Is it clear to you based on the edit messages you are receiving on the screen that there is an issue? In your own words, can you explain to me what the problem(s) are?

3. Is it clear to you what action you need to take? What would you need to do if you wanted to ignore the message you were receiving and move on to the next question/screen?

4. Is there a better way that we could notify you about the problem(s) on this screen?

5. What other information should we provide you with in the error messages and how to correct them?

Schedule 2: Part 4

Observation/Concurrent Probes:

1. Did the respondent read through the descriptions? Please explain.

2. If not, what did they do?

Example: I noticed you scrolled past these instructions, is there a reason you did so? Did you read these instructions in full, why not? What did you focus on?

Observation/Concurrent Probes:

1. Did R hover over the additional instructions?

If the respondent does not hover explain that they can hover over the “additional instructions”

2. Ask, what do you think about this functionality.

3. Is there any information here that seems unnecessary or distracting?

The way respondent answers Part 3 questions 1 and 2 will determine how many tables the respondent will see. For example, if the respondent entered 2 on question 1 (number of states) and entered both electric and natural gas in question 2, the respondent will have to complete a total of four tables. State 1 Electricity, State 1 Gas; State 2 Electricity, State 2 Gas.

Observations/Concurrent Probes

1. As you're navigating, please try to tell me what you're thinking, what you're trying to do, and why.

Observe how the respondent enters the data. How are they navigating through the tables? Note any issues.

2. How easy or difficult was it to enter that information?

3. Is the way you are entering the data here consistent with how you would enter the data on your company’s records? (does the current functionality match their expectations)

4. Was it clear how to proceed to the next step?

5. If you needed to go back, would you know how?

6. Are there any issues when removing or editing data?

7. Are they clicking on the next box?

8. Are they using the tab key to go to the next box?

9. Is this table behaving the way you expected it to?

10. What is your understanding of what this table is asking for?

11. Is the purpose of this page clear to you?

12. What are your thoughts on filling out this particular section?

    1. Do you feel like you're spending too much time on any particular part?

If there are any soft or hard edits use the following:

1. Is it clear to you based on the edit messages you are receiving on the screen that there is an issue? In your own words, can you explain to me what the problem(s) are?

2. Is it clear to you what action you need to take? What would you need to do if you wanted to ignore the message you were receiving and move on to the next question/screen?

3. Is there a better way that we could notify you about the problem(s) on this screen?

4. What other information should we provide you with in the error messages and how to correct them?

Observation/Concurrent Probes

1. Did respondent enter any comments here?

2. Were any issues encountered here?

3. Do respondents skip this box?

4. Did they focus on the box?

5. Is the size of the text box the appropriate size for what it is asking you to enter?

Review Page

Observation/Concurrent Probes

1. What are your first thoughts when seeing this Tab?

2. Did the respondent review their work?

3. Can you explain to me what is expected of you?

   1. What is this screen telling you?

4. If edits were necessary, did they make the changes in this page?

   1. Did they navigate to different schedules to make the edits?

5. Does this feel like an efficient way to complete this review?

6. Does this feel intuitive to you?

7. If you made a mistake, would you know how to correct it?

8. Was it easy to find what you were looking for on this page?

9. Is the purpose of this section clear to you?

10. What do you think of the title of this screen?

11. What other types of review would you be interested in performing prior to submission?

12. What do you think of the print option?

STOP the respondents from submitting. We want to ask additional questions before submitting.

Soft and Hard Edits: If they have not triggered any edits, have respondent complete an action that would trigger an edit. Ask them the following:

1. Is it clear to you based on the edit messages you are receiving on the screen that there is an issue? In your own words, can you explain to me what the problem(s) are?

2. Is it clear to you what action you need to take? What would you need to do if you wanted to ignore the message you were receiving and move on to the next question/screen?

3. Is there a better way that we could notify you about the problem(s) on this screen?

4. What other information should we provide you with in the error messages and how to correct them?

Submitting the Form

Have the respondent submit the survey form.

1. Did they get any edits?

2. Did they get an error message?

3. What are you seeing on the screen right now?

4. If need to edit: What are you expecting to happen after you fix this?

5. Is it clear how to proceed to the next step?

6. What kind of feedback are you getting from the system?

7. Did the error message (if applicable) help you understand the problem and how to fix it?

8. Is this behaving the way you expected it to?

Debriefing Question

Before asking these questions, ask questions on the issues the respondents encountered while completing the webform.

1. What position do you hold in your organization?

2. What is your experience in completing surveys for EIA?

3. Have you competed EIA-112 before?

   1. (If yes) Did you encounter any issues when completing this survey?

      1. (if necessary) Please elaborate.

4. When you completed this survey in the past, did you complete it on your own or with the help of others? If needed, can you tell me more about that? For which questions did you need the help from others?

5. Did you consult records before or while completing this survey? If needed, can you tell me more about that? For which questions did you need to consult your records?

6. What browser are you using? Microsoft Edge, Chrome, Safari, Internet Explorer, Firefox, other____ (note: we recommend respondents use Chrome)

7. Are you using a desktop, laptop, tablet or smartphone?

8. Do you have any other suggestions for how to make this online form easier for you to use?

Thank you for your time!