Various Economic Area Pretesting Activities

2012 Commodity Flow Survey

Usability Testing Protocol

(This protocol is a guide – the questions presented here won’t necessarily be asked exactly as worded in the protocol or in this order. It also is important to note that not all questions will be asked in every interview.)

Background Information

Respondent Background

• What is your role here at the company?

• How long have you worked in your current position?

• Have you completed other government surveys?

  • If so, which ones?

  • Have you ever filled any of these forms out online?

  • (If yes) How did you find the process? Easy? Difficult?

• What benefits would there be to reporting electronically to a survey. Any drawbacks?

Filling out the Form

Now I’d like to show you an electronic draft of the CFS form. I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records or others within the company in order to answer a question, just let me know what you would need to do. If at any point something isn’t clear or you have questions about something please ask and we’ll try our best to answer them.

I’ll periodically stop you and ask you questions about some of the pages and get your feedback on them.

(Note: The primary interview technique is retrospective; R is asked about each question and/or group of questions after s/he has reported. Concurrent “think-aloud” probes are used when R indicates some confusion or other problem, e.g. by frowning, re-reading a question, pausing for lengthy periods, looking away in thought, saying “Hmmm”, etc.)

Logging In

First, I’d like you to walk me through how you would log into the survey (hand R the paper with the URL, User ID and Passcode information)

• How easy or difficult was it to login?

• Do you think that the information given on the login screen was sufficient?

• What do you think of the overall layout of this page?

• Is there any other information you’d like to see on this screen? Anything that doesn’t belong?

Dashboard/Main Menu

• Can you describe to me what you see on this page?

• Where do you need to go to start completing your form?

  • Is it clear what you need to do to start? If not what could be done to make the start of the form clearer?

• Is there anything else you would like to see on this screen? Is there any other information that you need before starting to complete the survey? Anything that doesn’t belong?

• Where would you go if you wanted to find out more information about this survey?

  • How easy or difficult was it finding that information?

  • Would you look at this information before completing the form or when you needed it?

Items A and B – Verification of Physical Location and Mailing Address

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• (If respondent doesn’t change address information) Suppose you wanted to update the physical address information. How would you do that?

• If you had a question about how to answer Item B and needed to contact Census, where would you find information about contacting us?

Item C – Operating Status

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Item D – Total Number of Outbound Shipments

• What do you think of the overall layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• The first bullet instructions reference Item A. Suppose you had to go back to Item A to double check your answer. How would you do that from this page?

• What do you think would happen if you didn’t respond to the Total Number of outbound shipments and attempted to proceed? (Have respondent leave it blank and hit “Next”)

  • What did you expect to see?

  • (If an error message appears) Is this

  • message helpful?

  • Does it help you to figure out what you need to do to fix the problem?

  • What do you think would happen if you don’t fix the error? (Have R try to move on to the next page – leave error)

  • Is that what you expected to happen?

Item E – Sampling Instructions

(see if R realizes that the system has just done the sampling for them)

• What is your initial reaction to this page?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Item F – Shipment Characteristics

• What do you think of the layout of this page?

• Is it clear what information you are supposed to provide?

• What do you think about the amount of information on this screen?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

• Do the answer boxes provide enough space to enter your response?

• (If R doesn’t use commas for Shipment value question) I notice you didn’t use commas. Do you think you could enter commas in the answer blank?

• Where would you go to find the commodity code?

• What information would you provide for the Foreign Destination Export mode?

• How would you respond to selecting your mode of transportation?

  • How would you communicate the order used?

• What do you think would happen if you didn’t enter any Foreign Destination information? (Have respondent mark “Yes” for Export and hit “Next”.)

  • What did you expect to see?

  • (If an error message appears) Is this error message helpful?

  • Does it help you figure out what you need to do to fix the problem?

  • What do you think will happen if you do not fix the error? (Have R try to move on to the next page – leave error)

  • Is that what you expected to see?

• What do you think will happen once you hit “Add/Save” Shipment? (Have R proceed)

  • Is this what you expected to see?

  • What do you think of the layout of this page?

  • What would you do if you have another shipment to add?

  • What would you do if you are finished adding shipments?

Item G – Rush Deliveries

• What do you think of the layout of this page?

• Is it clear what information you are supposed to provide?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Item H – Monthly Value of Outbound Shipments

• What do you think of the layout of this page?

• Is it clear what information you are supposed to provide?

• What do you think about the amount of information on this screen?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Contact/Remarks

• What do you think of the layout of this page?

• Is it clear what information you are supposed to provide?

• Do you think you would use the remarks section? (If yes) What information do you think you would put there? What do you think about the size of the remarks box?

Review Page (no screen shots available yet)

• What is your initial reaction to this page?

• Is it clear what this screen is telling you? (Note: There will be an indication that there is a problem with one of the sections)

  • Do you find this information helpful?

  • Does it help you figure out what you need to do?

• How would you go about correcting the errors (Have R try and correct the problems)

  • (When R has fixed the problems) What do you need to do next? How would you submit the form?

• If you weren’t ready to submit your data at this point, what would you do?

• What other types of review, if any, would you be interested in performing prior to submission?

• Is there anything else you would like to see on this screen? Anything that doesn’t belong?

Completion Page (no screen shot available yet)

• What is your initial reaction to this page?

• Would you save a copy of this information for your records?

• If you wanted to print a copy of your answers to this form for your own records, what could you do?

  • Do you find this feature helpful?

• Is there any other information that you would need to have for your records?

Wrap Up

• Now that you’ve gone through this form electronically, would you prefer to do this survey on paper or electronically?

• If you were completing this electronically, would you still want/need the paper form? If so, what purpose would the paper form serve?

• Do you think you would complete the online form in one sitting?

• Did you have any issues with the navigation on the site?

• What are some of the things you liked most about the online form?

• What are some of the things you liked least about the online form?

• Do you have any suggestions for how make this online form easier for you to use?

Thank you so much for assisting us today.