2017 Commodity Flow Survey
Usability Testing Protocol
(This protocol is a guide – the questions will not necessarily be asked exactly as worded in the protocol or in this order. It also is important to note that not all questions will be asked in every interview.)
Introduction
Introduce observers and their background
Purpose of visit
Let me start by telling you a little about what we will be doing today. The Commodity Flow Survey collects data on the movement of goods within the United States. It will be mailed out quarterly in 2017. We are in the process of updating the questions on this survey and evaluating the Web questionnaire. Our goal is to find out how well the Web questionnaire works, and we would like your feedback on it.
Permission to audio tape discussion? Have R sign consent form
Before we get started I’d like to audio tape this interview so I don’t have to rely on my memory later. This session is confidential. Only people connected with this project will have access to your tape. If that is all right with you, please sign this consent form. It also tells you about the confidentiality of this session.
Background Information
Respondent Background
What is your role here at the company?
How long have you worked in your current position?
Have you completed other government surveys?
If so, which ones?
Have you ever filled out any of these government surveys online?
(If yes) How did you find the process? Easy? Difficult?
General Usability Probes
Is that what you expected to happen?
Can you show me what you would do/do next?
General Question Probes
In your own words, what was that question asking?
What are you thinking about?
Can you tell me more about that?
You answered, “…” because…?
I want to make sure I understand, can you explain that again?
Can you tell me how you arrived at that answer? What did you include? What did you exclude? What records did you use?
Reflect back on R’s answer and ask R to correct (“I want to make sure I have it right. I think you said, “…?”)
Shipping records and other information
Before we start the questionnaire, I would like to find out a little bit about your company keeps records of your shipments.
What do they look like?
What information is contained within them?
How your shipping records are kept? (Are they paper-based, spreadsheet-based, etc? If paper based, do they get lost or misplaced and how are they stored? What system do you use for creating spreadsheets? Are records in more than one format?)
How are they organized? By carrier? By mode of transportation? By customer, etc.?
Are they easily accessible or is there a process you need to go through in order to access them? Who has access to these records?
How long do you keep the records on hand?
Filling out the Questionnaire
Now I’d like to show you a draft of the 2017 CFS Web questionnaire. I would like you to go through the electronic version of the survey and complete it as you normally would. This is not a test of your ability or Internet savvy, so just proceed in whatever way makes the most sense to you. If you need to get information from records or others within the company in order to answer a question, just let me know what you would need to do. If at any point something isn’t clear or you have questions about something please ask and we’ll try our best to answer them.
I’ll periodically stop you and ask you questions about some of the pages to get your feedback on them.
(Note: We will employ both concurrent and retrospective probing techniques during the interviews. We will ask respondents about each question and/or group of questions after s/he has answered. We will also follow up on any difficulties that respondents had with the overall process of navigating the online instrument once they have completed it)
Logging In
First, I’d like you to walk me through how you would log into the survey (Observe how the respondents enters the password and ID information. Note any problems.)
What did you think of logging in?
How easy or difficult was it to login?
Do you think that the information given on the login screen was sufficient?
Is there any other information you would have liked to have seen on the log-in screen? Anything that did not belong?
Dashboard/Main Menu
What are your initial reactions to this screen?
What do you think this screen is telling you? (Make sure the respondent understands what the different features are. Also observe how the respondent reacts to see all four quarters. Note any reactions to the greyed-out quarters. )
Is it clear or not clear what you need to do to on this screen? If not what could be done to make the start of the form clearer?
Is there anything else you would like to see on this screen? Is there any other information that you need before starting to complete the survey? Anything that doesn’t belong?
(Point out the “view/print” PDF”) Did you notice this?
(If yes) What do you think you might find there?
Items A and B – Verification of Shipping and Mailing Address
What do you think of the overall layout of this page?
Is it clear or not clear what you need to do?
Is there anything else you would like to see on this screen? Anything that doesn’t belong?
(If respondent doesn’t change address information) Suppose you wanted to update the shipping address information. How would you do that?
Item C – Operating Status
What do you think of the overall layout of this page?
Is it clear or not clear what you need to do?
Is there anything else you would like to see on this screen? Anything that doesn’t belong?
Item D – Total Number of Outbound Shipments
What do you think of the overall layout of this page?
Is it clear or not clear what you need to do?
Is there anything else you would like to see on this screen? Anything that doesn’t belong?
The first bullet instructions reference Item A. Suppose you had to go back to Item A to double check your answer. How would you do that from this page?
What do you think would happen if you didn’t respond to the Total Number of outbound shipments and attempted to proceed? (Have respondent leave it blank and hit “Next”)
Is this what you expected to see, or did you expect to see something else?
(If an error message appears) Does it help you or not help you to figure out what you need to do to fix the problem? Can you tell me more about that?
What do you think would happen if you don’t fix the error? (Have R try to move on to the next page – leave error.
Is that what you expected to happen?
Item E – Sampling Instructions
(observe whether R realizes that the system has just done the sampling for them)
What is your initial reaction to this page?
Is there anything else you would like to see on this screen? Anything that doesn’t belong?
How would you go about selecting your shipments? (Probe as necessary. Pay particular attention to how R’s with electronic records will sample their shipments and report).
Item F – Shipment Characteristics
Shipment Level Option
What is your initial reaction to this page?
What is it telling you to do?
Which option would you most likely choose to enter your data?
Option One – Download/Upload
Say you wanted to choose option 1. How would you go about doing that?
What would you do first? Second/Etc.? (Have R show you the steps of downloading, completing one row and uploading the data. Probe when necessary.)
How did you find that process? Easy? Difficult?
Now say that you have finished entering your data.
What are your thoughts of that process?
Option 2 – Question by Question
Now say you want to choose Option 2. How would you proceed? (A statement should appear at this time instructing R that if they change Options then whatever data they currently have entered will be cleared).
In your own words what is this statement telling you?
Is there any way we can improve this?
What do you think of the layout of this page?
Is it clear what information you are supposed to provide?
What do you think about the amount of information on this screen?
Is there anything else you would like to see on this screen? Anything that doesn’t belong?
Do the answer boxes provide enough space to enter your response?
How would you respond to selecting your mode of transportation?
How would you communicate the order used?
Shipment Characteristics - Main Menu
You are done entering this shipment. What would you do next? (see if R can figure out the add/Save feature)
Is this what you expected to see?
What do you think it is telling you?
What do you think of the layout of this page?
What would you do if you have another shipment to add?
Now say that you need to make a change to one of the shipments you entered? How would you go about doing that?
What would you do if you are finished adding all of your shipments?
Shipment options
Now that you have seen both options for entering your shipments, which one do you think you would be more likely to choose? Can you tell me more about that/
(If R chooses Spreadsheet) Would you use the spreadsheet you downloaded here, paste your shipping information into this spreadsheet, or something else?
Item G – Verification of Primary Industry
What do you think of the layout of this page?
Is there anything else you would like to see on this screen? Anything that doesn’t belong?
In your own words, what do you think this item is asking?
What kind of description would you expect to see here?
I’d like to show you an example of the description. (Show R NAICS description) Is this the description you would expect to see, or would you expect to see something else?
If respondent indicates it is not the right description: you indicated that this was not the industry in which you do business. Can you tell me about more about that?
(If not obvious) How accurate or inaccurate was the description? (If inaccurate, find out what made it inaccurate)
Would you change anything about this description? If so, what would you change?
Item H – Time to Complete Survey
What do you think of the layout of this page?
How would you come up with your answer to this question?
What kinds of activities would you be including in your estimate? (Find out if R is including time it takes to gather the data.)
Contact/Remarks
What do you think of the layout of this page?
What do you think it is telling you?
Is it clear or unclear what information you are supposed to provide?
Do you think you would use the remarks section? (If yes) What information do you think you would put there? What do you think about the size of the remarks box?
Review Page
What is your initial reaction to this page?
Is it clear or not clear what this screen is telling you? (Note: There will be an indication that there is a problem with one of the sections)?
How would you go about correcting the errors (Have R try and correct the problems)
(When R has fixed the problems) What do you need to do next? How would you submit the form?
If you weren’t ready to submit your data at this point, what would you do?
What other types of review, if any, would you be interested in performing prior to submission?
Is there anything else you would like to see on this screen? Anything that doesn’t belong?
Completion/Confirmation Page
What is your initial reaction to this page?
Would you save a copy of this information for your records?
If you wanted to print a copy of your answers to this form for your own records, what could you do?
Do you find this feature helpful or not helpful?
Is there any other information that you would need to have for your records?
Where would you go next? (See if R selects “Next” to go to the next section or just logs out)
Wrap Up
If R responded on paper: It looks like you responded on paper for at least some of the quarters in 2012. Can you tell me about your decision to respond on paper? How can we encourage companies like yours to respond on the Web?
What are the benefits, if any, for completing the questionnaire online?
What is the downside, if any, to completing the questionnaire online?
If we did not send you a paper questionnaire in the mail, how would that affect your response process?
What role, if any, does the paper questionnaire play in your process for filling this questionnaire out?
Do you need to keep a copy of your answers? If so, which format do you prefer: hard copy, pdf, both, or some other format?
What are some of the things you liked most about the Web questionnaire?
What are some of the things you liked least about the Web questionnaire?
Other than what we have already discussed, do you have any suggestions for how make this easier for you to use?
Thank the respondent(s) for his or her time.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jennifer L Beck |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |