North American Products Classification System (NAPCS)

NAPCS Search Usability Testing

1. Research Questions:

   1. How usable are different designs for presenting pre-listed NAPCS products?

   2. How do the enhancements to a search function for product look-up work for respondents?

2. Before beginning

• Introduction of Census employees

• Introduction of company employee(s)

• Audio taping

• Permission to audio-tape discussion? Have R sign consent form.

• Before we get started: I'd like to audio tape this interview, so I don't have to rely on my memory later. This session is confidential. Only people connected with this project will have direct access to your recording. If that's all right with you, please sign this consent form. It also tells you about the confidentiality of this session.

3. Background

• How long have you been with the company?

• What is your title/role?

• What are your major responsibilities?

• What are your government reporting responsibilities?

• What other government surveys or filings, if any, do you also handle?

4. Introduction

Today, you are going to be helping us gather feedback on different ways to present some of the questions that you might answer in the 2017 Economic Census. One of the questions asks you about the goods and services you provide and any revenue they might generate. (Show respondent 2012 form for 22 and make sure the respondent remembers and is familiar with the process). Today, we are going to look at one of the ways that we can ask you about this information.

We are going to be looking at a very rough draft of what this question might look like on screen. It won’t function like it would if it were actually on the Web. As you are taking a look at this question, I would like you to tell me what you are thinking about it. We are interested in whether it is clear what you need to do to answer it, and what information you might provide.

5. Instrument walk-through

First, you are going to see an idea for how to present a list of goods and services that might be tailored to your business activities. Because this is a rough draft, I ask that you not focus so much on how it looks on the screen, but more on the flow of the question and how the information is presented.

I would like you to go through this question and answer it as if you were actually filling it out It is not important that you provide exact numbers or figures, we just want you to get a feel what you might select.

As you are going through the screen(s), I will ask you about how you would complete some of the items. I also will be asking questions about your reactions to what you are thinking.

(Have R look at pre-printed question version)

(Have R go through the item. Make careful note of what goods or services they would select. Probe any difficulties the respondent seems to be having)

Overall impressions of the design:

• Overall, how did you find the question?

• What did you think of how the information was laid out?

• Was it clear or not clear what you would need to do to make a selection?

• Does this list cover all of the goods and services you provide?

  • (If not) What would you do? (Determine if respondent understands the write-in or search options)

  • (If R indicates he or she would look something up) what would you look up/write in?

  • Do you have a preference for searching or writing something in? If yes, which one?

• If you could, what changes, if any would you make to this question?

Probes to ask for designs that have multiple screens:

• What did you think about how you moved through this item?

• Is this what you expected to happen? (If not – what would you expect to see instead?)

• Does it make sense to see this next screen? If not, how would you change the way this item flowed?

(If R indicates that he or she would search for a product, continue on to look-up)

6. Product Look-up

You mentioned that this list does not cover all of the goods and services that your business provides. Now, I’d like you to use an online tool to look up some of those goods and services. This tool may be a part of the questions

As you are looking things up, I’d like you to walk me through what you are thinking. Keep in mind that like the question, this is also a rough draft of what the tool might look like. It won’t be perfect. What we would like to know is what you are thinking about and expecting to see as you are looking up additional goods and services.

First, start by telling me what goods and services that your business provides that were not listed already.

(Have R jot down those products.)

(Have R start the look-up process.)

(Use the Error Sheet to record what the respondent is entering in and how they react to the search).

(Refrain from helping the respondent unless they have been searching a long time and task needs to be terminated, or the respondent requests to stop out of frustration. – Make a note if this happens).

(If the respondent selects/adds a product, ask about their confidence that it is correct).

1. ASQ

(Administer the ASQ after respondent completes looking up products in the task)

2. Follow-up questions

1. Follow-ups on performance measures

   1. Find out more detail on the usability issues behind errors and/or task success or failure.

“You indicated that you thought you were (un)successful at this task. Can you tell me more about that?”

2. Find out how to fix search to prevent or solve those problems.

“What changes would you make to fix that?”

2. Follow-ups on satisfactions measures

   1. Find out more detail on ratings and answers to the satisfaction questions.

E.g., “You said you felt ______. Can you tell me more about that?”

2. Find out how to increase user-satisfaction for the search.

E.g., “What would make this search easier/more satisfying/etc.?”

E.g., “Is there another way you would like to provide this information?”

3. General Questions

General Search function questions:

• In general, how do you feel about having a search function to look up goods and services?

• Other than what we discussed are there features for that function that you would like to see?

7. Overall Questions

Next I have just a few general questions about answering this question.

Reporting in dollars or percent:

• Here we have shown you the option to report in either dollars or percents. How do you feel about that?

• Is one easier than the other, or is it just as easy to report in dollars or percents? (If one is easier, probe to find out which one and why)

Web Response:

• (For paper responders) For 2017, we will have all-Web collections. How do you feel about that?

Role of Questionnaire:

• What role, if any, does the paper form play in your process?

• If we did not provide paper copies of the form, what would you do?

• (If R mentions printing from online) If you were unable to print out a copy of the form from online, what would you do?

• If you could not print a form online, what information would you like to see available to take its place? Can you describe it to me? What would be in it? Where would you expect to find it?

• Do you want to keep copies of what you filled out and submitted for your records?

• In what format would you want these copies to be?

Thanks the respondent for their time.

4