Protocol

Protocol: 2022 Economic Census Usability Evaluation

Introduction

Thank you for your time today. My name is XX and I work with the United States Census Bureau. My research team evaluates how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it.

[Email/ Send link to the PIA/ consent form; give time for participant to read and sign]

I just sent you an email. Can you please click on the link to the Qualtrics survey (or open the pdf) and read over the first page? This study is being conducted under the authority of Title 13 USC. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to record the session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. We plan to use your feedback to improve the design of an economic survey instrument and make sure it makes sense to respondents like you. Do you agree to participate? Thank you. Please click on the checkbox at the bottom of the screen.

[start CAMTASIA screen recording.]

Thank you.

Next, I would like for you to complete a Computer and Internet experience questionnaire. This provides us with additional information about the participants taking part in the evaluation.

[provide link to the Computer/Internet Experience survey; give time for participant to read and complete both screeners]

Thank you.

I am going to give you a little background about what we will be working on today. Today you will be helping us to evaluate the design of the online 2022 Economic Census instrument. The survey is in the early stages of development and we are interested in obtaining feedback to ensure that it has all of the information and features needed.

To do this, we will have you complete various tasks using the site. These will be consistent with tasks you would normally complete if you were requested to complete the actual 2022 Economic Census survey in the future. There are no right or wrong answers, we are mainly interested in your impressions both good and bad about your experience. I did not create the instrument so please feel free to share both positive and negative reactions.

While you are completing the tasks I would like for you to think aloud. Essentially what this means is to verbally express what you are doing and experiencing as you use the site.

I may ask you additional questions about some of the screens you see today and your overall impressions.

Do you have any questions before we begin? Ok let’s get started. I want you to read each task question and use the website to complete the task. Once you have completed the task just let me know by saying finished or done- then we can move on to the next task.

[Provide participant tasks- have them complete tasks one at a time]

During the interview, note participant’s questions or signs of difficulty.

Once all tasks are complete:

Next I would like for you to rate your overall experience using the site to complete those tasks.

[Hand the participant the Satisfaction questionnaire]

Administer the following probes by task/screen at the end of testing.

Feature	User task/screen	Probes
Welcome screen content and functionality	To get started, log into the survey using the information provided.
Adding/Deleting locations (MU)	(MU): Your company has gone through some changes and you need to make some updates to the location listing. Part A: Your company has added 2 new locations. Please update your listing to reflect these additional locations. Part B: The locations you added in Part A no longer exist. Remove the locations from the listing and Return to the Dashboard	MU: How easy or difficult was it to add locations? Why? Is it clear for which locations you are able to delete from the listing? Describe which can be removed/deleted.
Data review and Edits/Warnings	You would like to view what you have already reported prior to sending. Part A: Please review your data on the site Part B: You receive an error on your listing, please resolve the error displayed	Any comments on this screen? (use of color, layout, etc.) Any thoughts on available tools/features such as the Totals by EIN? What do you think is the purpose of this feature? Does the label make sense/do you have suggestions for changing the label of this feature? How easy or difficult was it to identify and resolve issues with your data? Why? Part B: Describe your process for resolving errors associated with multiple survey screens? Part B: Did you notice the ability to navigate using the drop down option? Part B: Is it clear how the drop down works once inside the form view? Describe.
Submission process	You are now done with the survey and ready to send your data. Part A: Please send your data for the Economic Census. Part B (MU): You’ve returned to the Report screen- identify which surveys have already been submitted.	What keywords were you looking for when trying to determine how to submit? How easy or difficult was it to submit your data? Why? (MU) Does the survey allow you to submit partially (i.e., for only a few locations at a time) How confident are you that your data was submitted successfully? Why? Any thoughts on the placement of the submission button? Which would be your preference? Why? How easy or difficult was it to identify surveys that were submitted. Describe.
PBA functionality [SU cases only]	We are investigating a new strategy to allow respondents to access the appropriate survey/form for their establishment. The next set of slides will walk you through several screen prototypes that we plan to test.	Prescreening After reading this description, what do you expect to see on the next screen? The next few screens will ask a series of questions about your establishment. After answering the questions, you will be presented with the following screen Any thoughts? Is it clear what is happening? Describe. Once your survey has been created you will receive the following message. Your survey will then appear on the Report screen What are your overall impressions of this functionality? PBA Search: Another option for allowing respondents to identify their Principal Business Activity is through a series of screens within the survey. The first screen allows the respondent to select the PBA associated with their predetermined industry. If none of the options on the listing are applicable, how should one report? Clicking on the Search allows respondents to view a complete listing of sectors/industries. After making a selection, the respondent clicks the Save and Continue Button. What do you expect to be displayed on the next screen? On the next screen, the respondent is presented with the ability to search for a PBA associated with the industry they selected from the prior screen. Is it clear how one should generate a search from here? If none of the PBAs are applicable, how should one proceed? Let’s take a look at another option for generating a search. Any thoughts on this? What could one enter in the write in box to generate a search? If none of the PBAs generated from the search are applicable, how should one proceed? If search results do not generate an applicable PBA (and respondents select the Other Principal Business or Activity option), respondents have the ability to write in a response. Any thoughts on this functionality?

[Debriefing]

General:

[Review Satisfaction responses- probe about any items rated <3)]

• Overall, what was your impression of the site? Is there anything you liked / disliked about the site?

• In general, would you say that it was easy or difficult to complete the tasks using the site? Which tasks, if any, were difficult to complete?

• If you could change anything about the site, what would that be?

• If you were having trouble with using this site, would you be interested in using a chat support feature, or would you prefer another way to get help? If yes to chat support, describe how you envision this working? Any potential challenges you see with something like this?

• (MU/SU) Based on the available reporting options, how has your establishment reported in the past? Why? How was your experience reporting this way?

• [MU]If reported via spreadsheet in the past (MU) How would you describe the process of downloading and uploading a spreadsheet?

  • How may we improve your experience with the spreadsheet functionality?

  • (MU/SU) If reported via form view in the past: What are your thoughts on using the form to report to the survey?

    • Review the three options for going between screens in the form. Is it clear for which screens you are able to visit? Describe which screens you are able to revisit.

    • Which of the displays would you prefer? Why?

• Is there anything else you would like to mention that we haven’t talked about?

This concludes our session. Thank you for your time and valuable feedback.

3