Protocol

2022 NAPCS Usability Evaluation Testing Protocol

1. Participant Background

• How long have you been with the company/in your current position?

• What is your title/role?

• What are your major responsibilities?

• Are you typically responsible for completing surveys like the Economic Census?

2. General Probes (Use these whenever necessary)

• In your opinion, are the information and instructions on this screen helpful or not helpful?

• Can you tell me what you are thinking about?

• Can you tell me more about that?

• I want to make sure I understand; can you explain that again?

• Reflect back on R’s answer (“I want to make sure I have it right. I think you said, “…?”)

3. NAPCS Background

4. Can you tell me about your process for filling out Economic Census?

   1. Additional probes (if necessary):

      1. Was that your first time filling out this survey?

      2. What was your role completing the Economic Census? Are others involved in helping gather the data? If so, Did you complete Item 22 which asks about the Details of Sales, Shipments, Receipts or Revenue in the 2017 Economic Census survey? –may reference content that was emailed if needed

      3. How do you typically gather the data for this survey item?

      4. Did someone else have to review the data for this survey item before you submitted it?

Now I would like to ask you specific questions about this survey item. Please open the attachment/ click on the link that I sent you via e-mail (if haven’t already done so).

5. Item 22 (Part A)

• In your own words, what is the purpose of this survey item?

• Describe to me how you reported to this item?

  • Via the spreadsheet or form (screen by screen) view?

    • Tell me about your past experience reporting to this survey item? Any recommendations for improvements?

    • How easy or difficult was it to find the services/products that you needed to report?

      • Please tell me about how you went about identifying your products/services for the items that were listed.

      • Were certain items harder to answer than others? If so, why?

• Were the products/services listed relevant/applicable to your company? Did they accurately reflect your products/services?

• To what extent did the descriptions used for your pre listed products/services match the way your business describes them in your records?

• Were you able to report all of your products/services from the listing? Were there any missing?

  • If so, how did you account for the products and services that were not pre listed?

• Is there a criteria you used for deciding which products/services to identify?

Machine learning

• Let’s say that a product/service for your establishment is not on the listing, how would you proceed? Present option A and B to respondent (random)

  • Option A: What do you think is the purpose of line item #X?

  • Option B: What do you think is the purpose of the Add additional products/services button at the bottom of the screen? What do you expect to happen upon clicking on this button?

• Preference: Which option would you prefer to use to write in additional products/services that you did not find on the listing. Why?

• Any thoughts on having the ability to type in products/services not listed on this screen?

• Would you be able to select a check box for a pre listed description and enter another product/service in the field that isn’t listed?

  • What makes you think that is/is not an option?

• What could you type/enter in the field?

• Is it clear whether there is a criteria for the types of products/services you should enter?

• (If applicable) Are there any words you may be inclined to use from your past experience with this survey?

• Alternative A (no interactive machine learning functionality): Let’s say someone entered (X- varied write in response by industry) into the field because they did not see it on the pre listed products/services for their establishment. After typing in those keywords and clicking on the Save and Continue button to proceed, what would you expect for them to see next?

  • The survey presents the respondent with the following screen. Is it clear what happened here? Describe.

• Alternative B: Describe to me what you may enter based on your establishment.

• Let’s enter those words from this screen. After typing in those keywords and clicking on the Save and Continue button to proceed, the survey presents you with the following screen. Is it clear what happened here? Describe.

• What are your thoughts on the results displayed including the following:

  • Relevancy of responses?

  • Description?

  • Order?

• Is it clear that there is a criteria for the types of products/services you should have entered?

• Based on the results presented, how would you report?

• Are the instructions clear? What should you do on this screen?

  • Any thoughts on the display including the number of results displayed, order, details provided, or instructions?

• How would you go about making a selection from the results displayed on the screen?

  • Are there any additional details that may be helpful in making a selection?

• How many do you think that you can select from the list displayed?

• If the results returned no descriptions that matched well to the product/service entered, how would you proceed?

  • Note if the respondent indicates that they would try to accessing additional results or use the Not applicable option.

• If you entered the wrong words resulting in results that were not applicable, how would you proceed?

  • If needed, how would you generate a new search?

• Do you think there may be limits on the type of searches that can be generated? Tell me more.

  • Abbreviations?

  • Product codes?

  • Multiple products?

6. Item 22 (Part B)

• Now I would like for you to navigate to Item 22 Details of Sales, Shipments, Receipts or Revenue

• In your own words, what is the purpose of this screen?

  • Do you think anything is missing from this screen? (e.g., expectations)

• Any thoughts on this screen? What is it showing you/asking you to do here?

• Are the products/services you added available?

  • Is this how you would have expected them to appear?

• Are you able to make changes to the product/service you added?

• How would you change a product that was added?

  • Note if the respondent chooses to go back and make a change on the search screen or if they would like the ability to retype their entry here.

Wrap-up

• Do you have any final recommendations or comments on any of the screens we went over today?

This concludes our meeting. Thank you for your time and valuable feedback.