Protocol

2022 Annual Business Survey

Web Usability Testing Protocol (04/2023)

(This protocol is a guide – the questions herein will not necessarily be asked exactly as worded or necessarily in this order. Please note that not all questions will be asked in every interview.)

Part A – Introduction

Introduce observers and their background

Thank you for your time today. My name is XX and I work with the United States Census Bureau. My research team evaluates how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it.

Purpose of this meeting:

Let me start by telling you a little about what we will be doing today. Today you will be helping us to evaluate the design of the online 2022 Annual Business Survey instrument. The survey is in the early stages of development and we are interested in obtaining feedback to ensure that it has all of the information and features needed.

To do this, we will have you go through the web survey instrument and complete it just as you would if it were the actual survey. Please keep in mind that there are no right or wrong answers, we are mainly interested in your impressions both good and bad about your experience. I did not create the instrument so please feel free to share both positive and negative reactions.

While you are completing the survey I would like for you to think aloud. Essentially what this means is to verbally express what you are doing and experiencing as you use the site.

I may ask you additional questions about some of the screens you see today and of your overall impressions.

Do you have any questions before we begin? Ok let’s get started.

Permission to audiorecord discussion? Have R sign consent form electronically.

Before we start, there is a consent form I would like you to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to record the session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session.

Part B – Background Information

Respondent Background

• What is your role in this company/location?

• Can you tell us about your experience completing the ABS?

Now I would like to show you an electronic draft of the ABS form. Here is the website address for accessing the survey (show the respondent the URL, survey key (if applicable), test username and test password).

I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records or others within the facility in order to answer a question, please just describe what you would need to do.

Do not hesitate to tell me anything you find surprising on the site, or if you have questions about the material on the site, just ask. We will try our best to answer your questions.

Part C – Testing

(Note: The primary interview technique is retrospective; R is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when R indicates some confusion or other problem, e.g., by frowning, re-reading a question, looking away in thought, saying “Hmm”, etc.)

Survey Item Screens – Sections E, F, G, H, and I

General Questions

• Do you have any recommendations for improving this screen?

Instructions – Note whether R appears to read instructions

• Do you recall whether you read any instructions on this screen?

• If you had questions about definitions or the survey items, where would you go?

Edit failure messages – If the R has not yet triggered an edit, have them do so

on purpose

• Do the messages on the screen clearly indicate to you that there is an issue, or are they not clear? Can you explain to me what the problem(s) are?

• Is there a better way that we could notify you about the problem(s) on this screen?

• What would you need to do if you wanted to ignore the problem message you were receiving and move on to the next questions? Is it clear to you what action you need to take?

Navigation

• If you needed to return to a prior section, how would you do this?

• Did you have any problems navigating back to a prior screen/section? Do you have any recommendations for improving the navigation?

• A future version of this website will have a drop-down box with a list of the screens to let you navigate directly to any screen; do you have any feelings about this kind of function?

• What did you think about the layout of the items on this screen? Was it a comfortable amount of information? Is there anything we could do to improve this screen?

Save

• If you wanted or needed to save your work and return to it later, how would you go about doing so?

• Does the save feature function as you would it expect it to?

• Is the save feature something that you would use?

“Select all” instructions

• Note whether R attends to the “select all that apply” instructions

• Would it ever be possible for more than one answer to be correct in this question? What would you do if that were the case?

Capitalized text

• Did you notice that some words were printed in capital letters in the question on this page? What does that mean to you?

“Select only one” instructions

• Note whether R attends to the “select only one” instructions

• Do you recall seeing the instruction to select only one answer?

• (If R attempted to select more than one answer) Can you tell me about how you decided to select more than one answer in this question?

Part D – Wrap Up

• Would you complete this on paper before completing online?

• Do you think that you would complete the online form in one sitting?

• If you were completing this form electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?

• Now that you have gotten a good feel for the online form, what is your overall impression of it? Was it easy, difficult, or somewhere in between?

• Did you have any issues with the navigation on the site?

• What are some of the things that you liked the most about the online form?

• What are some of the things that you liked the least about the online form?

• Are there any other options that we should provide in the online form?

• Do you have any suggestions for how to make this online form easier for you to use?

Thanks for your time and input!

4

ABS Usability Testing Protocol

April 2023