Protocol

2023 Annual Integrated Economic Survey (AIES) Usability Evaluation

Introduction

Thank you for your time today. My name is XX and I work with the United States Census Bureau on a research team that evaluates how easy or difficult Census surveys are to use. We conduct these interviews to get a sense of what works well, and what areas need improvement. We recommend changes based on your feedback.

[Confirm they signed the Consent Form; should be sent prior to interview.]

Thank you for signing the consent form, I just want to reiterate that we would like to record the session to get an accurate record of your feedback, but neither your name or your company name will be mentioned in our final report. Only those of us connected with the project will review the recording and it will be used solely for research purposes. We plan to use your feedback to improve the design of this survey instrument and make sure it makes sense to respondents like you. Do you agree to participate? Thank you.

[start CAMTASIA/Snagit screen recording, if yes.]

Thank you.

Background

I am going to give you a little background about what we will be working on today. Today you will be helping us to evaluate the design of the online Annual Integrated Economic Survey, or AIES, instrument. The website is in the early stages of development, so this is an opportunity to make sure it works as smoothly as possible.

To do this, we will have you complete various tasks using the site. These will be consistent with tasks you would normally complete if you were requested to complete the actual AIES in the future. There are no right or wrong answers, we are mainly interested in your impressions both good and bad about your experience. I did not create the instrument so please feel free to share both positive and negative reactions.

I may ask you additional questions about some of the screens you see today and your overall impressions.

Do you have any questions before we begin? Ok let’s get started.

First, I would like to get some information to give me some context.

• Can you tell me about the business, like what types of goods or services it provides? And how it is organized?

• What is your role within the company?

• Are you typically the person responsible for government surveys?

  • Do you typically have access to all of the data needed?

  • If not, what areas or positions do you usually reach out to?

Next, I’m going to read each task question out loud, then you will use the website to complete the task. While you are completing the tasks, I would like for you to think aloud. Once you have completed each task just let me know by saying finished or done- then we can move on to the next task.

Tasks

[During the interview, respondents will use Census Laptops which will be screen/audio recording. Have participant go through each screen, remind them to think out loud, and note participant’s questions or signs of difficulty. Note any content issues if they arise.]

Task One: Login

Welcome Screen

Read over this screen as you normally would. Do you have any questions or comments about this page? Is anything confusing?

Task Two: Provide Company Level Data

You first want to fill out high level data about your company. Provide the answers to ‘Employment & Payroll’, ‘Revenue’, and ‘Capital Expenditures’ using the data provided.

Fill out the information for:

• Company Name (Row 6)

Task Three: Choice Answer by Establishment or Industry

Provide the answers to question, “What was the total number of employees for the pay period including March 12, 2022?” using the following information:

You have 4,000 employees in your industry; Broken out you have 4 establishments with 1,000 employees each, you may choose how to fill out this data.

Task Four: Establishment Reporting

Part One: Navigate to the section Capital Expenses within the main Survey Tab. Provide the answers to the question “What were the total capital expenditures for new machinery and equipment in 2022?” Please provide the data provided at the establishment level.

Fill out the information for:

• Location 1 (Row 6)

• Location 2 (Row 7)

• Location 3 (Row 8)

Part Two: Auto Sum: Compare the total figure of the establishments against what’s listed for the industry section, do the figures match?

Probes: Are you aware of any instructions regarding the formatting of your answers? If you wanted to indicate numerical value of 125,000, how would you type that in?

Task Five: Industry & Company Total

Part One: Navigate to the section Revenue within the main Survey Tab. Provide the answers to the question “What were the total sales, shipments, receipts, or revenue in 2022?” Please provide the data provided at the Industry level.

For each industry, fill in the number ‘100’.

Part Two: Company Total: Does the total from the industry section match the company total?

Other Tasks: Functionality Unknown

NAPCS Data if applicable Navigate to the Products section. Your company recently started manufacturing ‘chew toys for dogs’, how would you add a new product to the listing?

Add a Location & PBA: Your company opened a new location. First add a new location, then select a Primary Business Activity that best matches your company’s overall activities.

Errors: Is there anything that needs to be corrected? How would you fix an error in the data? Is this how you would expect to be notified of an error, or would you expect a different method?

Delegation: You would like to send the capital expenditures section to another department. Is there a way within this website to send/delegate a section?

Forward Navigation: You have completed the company data section of the survey, continue to the location data section.

Backward Navigation: Once the detailed spreadsheet is filled out- you would like to go back and check the additional information pdf to ensure you didn’t forget anything.

Location Help/Instructions: You would like clarification on [undetermined content], is there a way for you to find more information about this question? (Click into instructions)

Submit Data: You would like to submit your data to the Census Bureau

Wrap up:

• Overall, what was your impression of the site? Is there anything you liked / disliked about the site?

• In general, would you say that it was easy or difficult to complete the tasks using the site? Which tasks, if any, were difficult to complete?

• If you could change anything about the site, what would that be?

  • Were there any features that were not intuitive?

• Do you have any thoughts about delegation? If a feature were available allowing you to send certain sections of the survey to another department, is that something you would utilize, or no?

  • What would you expect a delegation option to look like?

• Are there any other features or information that would support your response, that you think are currently missing from the website?

• If you were having trouble with using this site, would you be interested in watching video tutorials?

  • Which tasks do you think need a video tutorial?

  • Where would you expect to find the videos?

  • What is the longest video tutorial you would watch?

• Is there anything else you would like to mention that we haven’t talked about?

This concludes our session. Thank you for your time and valuable feedback.

4