Protocol

AIES Usability Testing Protocol

Paths:

• Manufacturing Status

• MU SU Status

• Add a location [MU only]

• Error Check

• Online or Upload

Introduction
Thank you for your time today. My name is XX and I work for the United States Census Bureau on a research team that evaluates how easy or difficult Census surveys are to use. We conduct these interviews to get a sense of what works well, and what areas need improvement. We recommend changes based on your feedback.

[Confirm they signed the Consent Form sent prior to interview.]

Thank you for signing the consent form, I just want to reiterate that we would like to record the session to get an accurate record of your feedback, but neither your name or your company name will be mentioned in our final report. Only those of us connected with the project will review the recording and it will be used solely for research purposes. We plan to use your feedback to improve the design of this survey instrument and make sure it makes sense to respondents like you. I'm going to start the recording now, if that's okay with you. 

[start Snagit/Camtasia screen recording, if yes.] 

Thank you.

Background 
I am going to give you a little background about what we will be working on today. Today you will be helping us to evaluate the design of the online Annual Integrated Economic Survey, or AIES, instrument.

The AIES is the Census Bureau’s attempt to consolidate seven of our annual economic surveys which span each major sector- for example retail, wholesale, services etc. This is a new initiative so, the website was only recently developed, and this is an opportunity to make sure it works as smoothly as possible.

To do this, we will have you complete various tasks using the site. These will be consistent with tasks you would normally complete if you were requested to complete the actual AIES in the future. There are no right or wrong answers, we are mainly interested in your impressions both good and bad about your experience. I did not create the instrument so please feel free to share both positive and negative reactions.

The data on the survey website should be your own company data, but any changes we make here will remain in the testing environment and won’t reflect on your real data. The goal is just to give you a sense of what you will actually see when you receive the real survey. 

We will provide you any data you need to fill in on the Answer Key. [Refer respondents to the printed document].

To wrap up at the end, I’ll ask you additional questions about some of the screens you see today and your overall impressions.

Do you have any questions before we begin? Ok let’s get started. First, I would like to get some information to give me some context about your business. 

Can you tell me about the business, like what types of goods or services it provides? And how it is organized?

What is your role within the company?

Are you typically the person responsible for government surveys?

• Do you typically have access to all of the data needed?

• If not, what areas or positions do you usually reach out to?

Great thank you.

Now I’m going to read each task question out loud. Then you will use the website/[spreadsheet] to complete the task. Again, I will prompt you to use the answer key for any areas of the survey you need to fill out data for.

While you are completing the tasks, I would like for you to think aloud. It will be helpful for us to hear your thoughts as you move through the survey. So, try and explain what’s on your mind as you move through the survey website. Tell us about expectations you have as you explore and whether the site is or isn’t meeting those expectations. Once you have completed each task just let me know by saying finished or done- then we can move on to the next task.

To get started, I’ll navigate you to the opening screen of the survey. Let’s begin with having you read the first task.
 

Usability Testing Tasks

STEP ONE ITEMS

Path: Online Spreadsheet

[PROBE] Before you make a selection, can you tell me which you would normally select if you were actually filling out this survey?

For this task, we’ll have you select Online Spreadsheet here to begin.

Path: Download Spreadsheet

[PROBE] Before you make a selection, can you tell me which you would normally select if you were actually filling out this survey?

For this task, we’ll have you select Upload a Spreadsheet here to begin.

Special Task: Download Spreadsheet (Download only)

You want to access the survey from this page, how would you go about doing that here? -Use the site to do so.

Core Task: Verify Location(s)

Begin the survey by verifying the location data. 

For this task, let’s assume all these locations are [this location is] operational.
Feel free to take a moment and explore, as you normally would.  Remember to think out loud.

Let me know when you feel you’ve accomplished this task.

• Requirement for success: Update Operational Status

Special Task: Add a location (Download or Online)

For Non-SU’s

Your company opened a new location. Is there a way to add a location to the listing?

Tell me when you feel you've completed this task.

• Note to researcher: Download path; requires R to navigate back to the online spreadsheet to successfully complete this task.

Special Task: Remove Added location (Online only)

For Non-SU’s

You realize the location you just added is not in scope for this survey. How would you remove the location you just added?

Step One Probes:

• Run prechecks What would you do next? (what are their assumptions)

• Any comments about the Overview? 

STEP ONE B.

Special Task: Step One B

IF Added Location not removed; Half of Online participants; Half of Download participants

Probe: Can you describe the purpose of this screen? Is there any action you need to take?

Update the operational status for the location you just added.

STEP TWO ITEMS

Core Task: Provide Company Level Data

[Use answer key]

OK here we’re on Step 2, company-level Questions, where you’ll fill out high level company data.

For the first question, ‘Company Mailing Address’ fill out the information as you normally would.

[Researcher Note] Tell R to feel free to enter fake data for any cells they don’t know off the top of their head, (such as zip code).

We will have you answer NO for all of the Yes/No Questions, until you reach the reporting period question. Then, use the answer key guide to fill in the remaining questions.

Step Two Probes:

• Is the reporting of data in thousands clear?

  • What if your data is less than 1,000?? What would you do?

    • Is your data ever less than 1,000?

  • If your answer to this question is none/nothing how would you enter that?

• Had you noticed the auto summing on this question? Any thoughts on that?

• If you wanted to go back to the beginning of this section, how would you do that most efficiently?

  • Would a menu be beneficial here, or no? Would you use it if it were available?

• There’s no option to download these questions into a spreadsheet to answer them, is that something you would do if the option were available?

STEP THREE ITEMS

Path: Online Spreadsheet

[PROBE] Before you make a selection, can you tell me which you would normally select if you were actually filling out this survey?

For this task, we’ll have you select Online Spreadsheet here to begin.

Path: Download Spreadsheet

[PROBE] Before you make a selection, can you tell me which you would normally select if you were actually filling out this survey?

For this task, we’ll have you select Upload a Spreadsheet here to begin.

Special Task: Download Spreadsheet (Download only)

You want to access the survey from this page, go ahead and move through the steps to do that.

Core Task: Navigate Detailed Data Tab

Feel free to take a moment and explore, as you normally would.  Remember to think out loud.

• Where would you go to begin filling in your data?

• Pick any location. You want to indicate that there are 500 employees at this location.

• You want to fill out data about revenue, can you locate the section for questions about revenue?

Special Task: Manufacturing Tab, NAPCS

• Is there any other data you would need to fill out before moving forward?

• The survey asks that you report the value for each product you produce at your company. Can you do that from this page?

• Select a product of your choosing and give it a value of $100,000.

Core Task: KAU Row(s)

• [Researcher, point out the KAU row, beginning with 30]

• Can you describe what the row(s) here represent?

• [If unsure], Is there a way to find out?

• [Researcher, explain if necessary] Do you have any thoughts about this?

Special Task: KAU split [PR, Wholesale, Tax]

Only if PR; multiple NAICS in Wholesale (42); Tax Op Status

• Can you describe what the row(s) here represent?

• [Researcher, explain if necessary] Do you have any thoughts about this?

Core Task: Fix an Error

• Part A: Before you submit your data, you’d like to check your data for errors. Is there a way you can do that from this screen?

• Part B: If there is an error, how would you fix it?

• Probe: 

  • Do you have any thoughts about the current functionality?

  • How would you expect to be notified of errors?

Special Task: Save and log out (Online Only)

• You have to leave your workstation and will be gone for a while, but are in the middle of reporting, what would you do?

• Do you think your work on the spreadsheet will be saved?

STEP THREE PROBES:

Run prechecks

• What would you do next? (what are their assumptions)

• Is this running a check just for this spreadsheet? Or all of the data you’ve entered?

Manipulating Spreadsheet: Freezing / Filtering / Copy Paste

• Would you be able to manipulate the spreadsheet to better suit your needs? Are there certain functions you would try to use?

Entering in Thousands (auto-rounding)

• Do you have any thoughts about the auto-rounding function? E.g. In ‘First Quarter Payroll’ typing 1 auto changes to 1,000

STEP FOUR TASKS

Core Task: Submit Data

You would like to submit your data to the Census Bureau.  Let me know when you feel you're done.

For the submit screen, fill out your own information as you normally would. Feel free to enter fake data for any cells you don’t know off the top of your head such as zip code.

Probe: Did this process meet your expectations? Or would you expect something different?
Would you have any expectation of reviewing your data, or no?

Downloading Data:

• At what point in the survey would it be most useful for you to view/download a record of your responses?

• Do you have any thoughts about the current placement of this option, which is available after submission?

Wrap Up 

Thank you for taking the time to explore the site. I’d like to ask some wrap up questions:

• Overall, what was your impression of the site?

• Is there anything you liked / disliked about the site?

• In general, would you say that it was easy or difficult to complete the tasks using the site? Which tasks, if any, were difficult to complete?

-----

• If you could change anything about the site, what would that be?

• Were there any features that were not intuitive?  

Delegation

• If you wanted to share or delegate certain sections or questions of the survey, how would you do that?

• If a feature were available allowing you to send certain sections of the survey to another department, is that something you would utilize, or no?

• What would you expect a delegation option to look like?

  • Where in the survey flow would you expect to have that option?

-----

• Are there any other features or information that would support your response, that you think are currently missing from the website?    

• Is there anything else you would like to mention that we haven’t talked about?

This concludes our session. Thank you for your time and valuable feedback!

Page 10 of 10