User Research and Testing for Data Transparency Office Products

Example User Testing

Moderated User Testing Guide

Date:

Name:

Persona:

Demographics: (for ensuring an unbiased, diverse set of participants; not published)

Intro

Thanks again for your time today to give us your feedback. My name is _______, and I’m helping with the design of a new Treasury Department website.

We know your time is valuable and we appreciate your time and attention. Your feedback today will help how we [information for how their feedback will be used].

For Fiscal Service to speak with the public, we are required to have approval from the Office of Management and Budget.  You are not required to respond unless a currently valid OMB approval number is provided. The approval number for this project is 1530-0023.  This study is estimated to average 60 minutes. If you have any comments regarding this study, please write to: Bureau of the Fiscal Service, Forms Management Officer, Parkersburg, WV 26106-1328.

RECORDING PERMISSION

Before we get started, we'd like to ask for your permission to record the audio and screen capture of our session. Recordings are helpful for our future reference and when compiling the findings of our research.

Do we have your permission to record our session today?

[ ] Yes

[ ] No

[If yes:]

I’m going to start the recording process. It should start recording within a few minutes.

[Turn on recording function within conference app]

I wanted to run through a few logistical things before we get started.

• [Observers] We do have [my colleague _____] acting as observers or notetakers on the call. This allows me to stay focused on what you’re saying. Is that ok with you?

• [Stopping] Next, I wanted to let you know that we can stop at any time if you would like and you are welcome to decline or skip any question, just let me know.

• [Aggregation] Last, we will be taking your answers and aggregating them with others’ feedback, so no information will be directly tied to you. Is that alright?

To give a quick overview of what we’ll run through, I’ll quickly outline our agenda:

• This session is scheduled for [time period]. If you do have more time, we’d love to get all the feedback and opinions you have!

• I’ll send over a link [in Slack/via email/in Zoom] shortly for the activity and then ask you to screenshare.

• For this activity…

  • I’ll start by asking some information about your demographics, technology preferences, and level of interest in the subject matter

  • Next, I’ll ask that you think back to a time where you are looking for data to complete a task.

  • We’ll be using static designs of what our website could be, so it’s not a fully developed website and it’s still a work in progress.

  • There are no right or wrong answers, so if anything is confusing, please say so! Your feedback will help us improve that section before we start coding things up.

  • We did not design this ourselves, so you won’t hurt our feelings if you are honest about what you think!

  • I’ll have a mix of questions and pose some hypothetical tasks for you to complete.

  • For each task, I ask that you think out loud and basically narrate what’s going on in your head – again, because these are static designs, you can narrate what you’d do on the page or what you’d click.

After the activity, I’ll ask some follow up questions.

Do you have any questions at this point?

Ok, great. Let’s get started.

User Background

Before we get started, I have a few questions for you so I can get a better understanding of your background, technology preferences, and level of interest.

General

1. Do you mind if I ask your age range? (I will give you 5 options or you can decline to answer).

   1. 22-30

   2. 31-40

   3. 41-55

   4. 56-65

   5. 66+

   6. Decline to answer

2. Where do you live? [Zipcode]

3. Which of the following categories best describes the industry you primarily work in?

• Agriculture, Forestry, Fishing and Hunting,

• Utilities

• Computer and Electronics Manufacturing

• Wholesale

• Transportation and Warehousing

• Software

• Journalism or Broadcasting

• Other Information Industry

• Real Estate, Rental, and Leasing

• Primary/Secondary (K-12) Education

• Health Care and Social Assistance

• Hotel and Food Services

• Legal Services

• Homemaker

• Religious

• Mining

• Construction

• Other Manufacturing

• Retail

• Publishing

• Telecommunications

• Information Services and Data Processing

• Finance and Insurance

• College, University, and Adult Education

• Other Education Industry

• Arts, Entertainment, and Recreation

• Government and Public Administration

• Scientific or Technical Services

• Military

• Other Industry

4. What device do you see yourself using most often to browse the internet?

   1. Cell phone

   2. Tablet

   3. Laptop

   4. Desktop computer

   5. Other

5. Which browser do you use most often when surfing the internet?

6. What is your primary reason for wanting to gather more information about federal financial data?

Brand, Mission, Marketing or Other Non-Visual Questions

Before we get started with the design discussion, I’d like to get your thoughts on [our site’s mission.]

I’ll read you a statement and then ask you a few questions about it:

We want to inspire trust in the government by providing high quality Treasury data in one easy to use platform. Treasury data for the people.

1. Do you have an idea of what the site is going to offer based on my previous statement?

2. Based on that statement, do you think we will be different than any of the other Treasury data websites you’ve used in the past?

3. And lastly based on the above statement, would you use this site?

I’m sending a link via [Slack/Email/Zoom chat] and once you have that up and running, can you share your screen?

Activity URL:
Webex:

Search & Filter User Testing Questions

Section 1

1. Ok let’s get started! Imagine you are looking for data to complete a task, answer a question, write an article, create a data visualization, etc. Can you think of a time where you were looking for a dataset?

2. Ok, let’s say you want to find some datasets for your task. Where would you go to get started with this? [Ok, go ahead and click there.]

3. I’ll give you a second to scan the page. Can you think out loud - what are your initial impressions?

4. From looking at this page, what data would you say is available on this site?

5. Let’s say you wanted to find all datasets related to some specific concept or category for your task. How would you do that?

6. I see you chose [options A]. Is there anything confusing or unclear about those other two options, [Option B or C]? Or now that we’ve mentioned all 3, why would you say you chose [the option they chose]?

7. Ok, and where would you go to search for specific terms?

   1. What information do you expect that to be searching?

Section 2

1. There are some--what I’ll refer to--as cards or tiles on the page, such as that tile for “XYZ.” What do these tiles represent to you?

2. Can you point out everything you would expect to be clickable within one of those tiles?

3. When you click on the tile, where would you expect it to take you?

4. What information in the search results are you looking at to decide whether to either use or view more about that dataset?

5. Is there information you wish you could see in the search results in order to decide what to view or explore?

Section 3

1. Ok, let’s say now, you only want to only explore datasets if they are related to ABC. How would you do that?

2. Let’s say you’re only interested in data that is current and up-to-date. How would you find ABC data, but only if it’s been updated within the last day?

   1. How would you clear just the 24-hour filter? Ok, go ahead and click there.

3. How would you clear all filters? Ok, go ahead and click there.

   1. Is this page where you expected to return to after clearing filters?

4. How would you clear this keyword search? Ok, go ahead and click there.

5. Let’s say you wanted to view all datasets the site has available in CSV or XML format. How would you do that?

Secondary Questions if Time Allows

Misc

1. [Mobile] Would you expect to use this on a mobile device?

2. [Filters] Under filters on the left, there is “XYZ” filter. What would you expect to see under the XYZ dropdown menu?

3. [Topics] Looking at the Rates of Exchange tile, what do you expect would happen when you click on “Accounting and Financial Reporting”?

   1. Do you use tags like these? What do they help you with? Do they waste space?

Notetaker questions?

Future Outreach?

Great, thanks. Here’s my last question for you:

We will be talking with people throughout the life cycle of this project. If you’d be interested in giving us your opinion in the future, we can take your name down, and it’d be great to get your feedback in the next month or two.

If not, no worries - we really appreciate your time today!

Yes / No

Well, again, on behalf of our team, I’d like to thank you for your time today.

Your feedback and insights will play an important role in helping us shape our

Thank you. Enjoy the rest of your day. Bye.