Remote Website and Mobile Application Usability Focus Groups

Objective: Gather feedback from NEW customers to identify areas for improvement on catalog.usmint.gov.

Developed FoR:

U.S. Mint Web Team

U.S. mint Catalog Website

NEW Customers

UX Test Plan

UX TEST PLAN

WHAT IS THE PURPOSE OF THE U.S. MINT UX TEST?

The purpose of the usability test is to gather data on how new users interact with the United States Mint catalog website. The usability test will evaluate the effectiveness of website changes that have occurred over the last year, with a specific focus on the integrated navigation and improved content pages.

This data gathered during the usability test will be used to better understand users’ perceptions of the website, to evaluate users’ abilities to find information using website, and to inform future improvements to the website.

The UX study will:

Gather users’ impressions of website

Measure users’ abilities to find information using the website

Identify difficulties in using the website

Suggest research-based recommendations for improvement

When & where will the testing be conducted?

24 UX sessions will be conducted remotely on both desktop and mobile devices using screensharing software:

Members from the U.S. Mint marketing, web, and content teams are encouraged to observe the sessions remotely, as all of the sessions will be streamed live and recorded. Audio/video recordings will be uploaded at the end of each week.

To the extent possible, users will have a mix of demographic characteristics, including age, gender, education, ethnicity, internet experience, geographic location, and purchasing history with the U.S. Mint.

What will happen during the UX test?

Each session will last approximately 60 minutes.

During the session, participants will:

• Provide initial impressions of the homepage and email newsletter.

• Perform a series of tasks using the catalog website.

• Provide overall reactions to using the catalog website.

The facilitator will observe and objectively probe on issues. The facilitator will not coach or guide the participants.

Following is a high-level breakdown of the 60-minute sessions (these time limits are simply an approximation of the test session):

Duration	Activity	Administered By…
0:00 – 0:03	Introduction / Greeting	Facilitator
0:03 – 0:05	Think Aloud Instructions	Facilitator
0:05 – 0:10	Initial Impressions	Facilitator
0:10 – 0:50	Usability Tasks	Facilitator
0:50 – 0:55	Overall Impressions	Facilitator
0:55 – 1:00	Closing – Thank You	Facilitator

Test Protocol

The following protocol will be used during in-person moderated usability tests:

1. Prior to the test, recruiting firm gathers demographic information from participant.

2. Prior to the test, participants will participate in a 15 min set-up session to ensure they are able to connect remotely via their desktop/mobile device.

3. Facilitator greets participant and introduces self.

4. Facilitator very briefly explains hour, sets expectations, and asks if participant has questions.

5. Facilitator asks participant to share their screen.

6. Facilitator shows participant webpage and gathers initial impressions.

7. Facilitator explains thinking aloud and begins with scenarios.

8. Participant performs scenarios one at a time.

9. Facilitator monitors participant’s actions, comments, frustrations, etc.

10. Facilitator monitors time so that there is time at the end for final questions.

11. Facilitator elicits overall reactions and answers to specific end-of-session questions.

12. Facilitator thanks the participant.

Please note: All participants will receive the same instructions before the usability test, based on the sample script (next page). Great care will be taken to ensure participants all receive similar instructions before beginning the usability test.

Facilitator Introduction

The following introduction is a sample and will be used as a guide, not read to participants verbatim.

Hello, my name is Cari Wolfson/Kristina Sommer and I am a usability researcher who is working with the United States Mint in an effort to improve their website. Today we are talking with users like you who may use the website to learn more about where improvements could be made to enhance the overall user experience. Thank you for agreeing to participate in our usability study.

Today, I am going to show you a website and ask for your honest feedback. So, please let us know what you like about the site as well as what you don’t like. Your opinions are very important to us and will help us improve the site so that it meets the needs of people like you.

During our session, I am going to ask you to perform a few activities, or tasks, using the site. I am planning to give you each task one at a time and ask you to read the task out loud. As you go through each of the tasks, please try to “think out loud” and tell us what you are doing and what you expect to find on the site. I’ll demonstrate what we mean by “thinking aloud” in a minute.

 

Please feel free to ask me a question about the tasks or the site. However, I may not be able to answer your question, as I was not involved in the development of the website. If that happens, please continue to work as you would if you were at home or at work.

Your comments are very important, so I’ll be taking notes as well as recording our voices and screen interactions during the session.

This information will be kept strictly confidential and used only for analysis.

The session should take about an hour. At the end of the session, we’ll then have a chance for you to tell us about your experience and give us any feedback you may have.

 

Do you have any questions before we begin?

What will be tested?

The following scenarios have been developed to test various aspects of the United States Mint catalog website. While the participants complete the scenarios, qualitative and quantitative data will be captured by the facilitator and note takers.

Please note:

Prior to the usability scenarios, the facilitator will explain the purpose of the usability test and will conduct pre-test activities.

Please refer to:

• Test Protocol

• Facilitator Introduction

• Facilitator Outline

• Consent forms

The scenarios on the following pages will be provided to the users verbatim, so that participants hear the exact same questions.

The probing questions made by the facilitator will vary from participant to participant and will be based on the facilitator’s judgment of what to test and the time allotted in the session. The probing questions that follow are simply a guide to assist discussion and will be used by the facilitator to learn more about specific concepts.

Lastly, the estimated times below are simply an estimate of the time it will take for each scenario. Some users may complete tasks faster and others may struggle and not be able to complete all of the following scenarios.

How will the tasks be measured?

In order to measure users’ performance on the website, several quantitative metrics and qualitative data will be collected, including:

Successful Task Completion

Each scenario will ask participants to complete a task and find specific information using the website. The scenario is considered a successful completion when the participant says he has achieved the goal AND has done so successfully. To ensure that tasks are judged correctly, successful task completion criteria have been defined in this plan so that all team members can review and contribute to the definitions of success. This will help to ensure that observers all score each task in the same way.

Critical Errors

Critical errors are large deviations from the intended path for the scenario. Participants may or may not be aware that the task goal is incorrect or incomplete. As an example, a participant providing the wrong answer to the scenario question, while thinking he was correct, is a critical error. Other examples include intervention from a facilitator because either the participant had gone so off-course or had made mistakes that prevented him from completing the scenario correctly.

Non-critical Errors

Non-critical errors are mistakes that the participant is able to recover from. These may be clicking the wrong link, choosing the wrong feature, trying to select something that is disabled, or another minor mistake.

Wrong Path

Wrong path errors include those for which the participant had a “false start” and went down the wrong navigation path or workflow path. These are also reflected by critical and non-critical errors but are captured at a higher level. Wrong path errors reflect major issues with navigation or workflow design. To help measure wrong path, users’ first clicks are recorded and identify where users initially went to complete a task.

Subjective Measures

The participants’ satisfaction, ease of use, ease of finding information, and other relevant subjective measures will be captured via a post-test questionnaire. Participant confidence and judgments on ease of use will be collected after each usability test.

Likes, Dislikes, and Recommendations

As the participant completes each scenario, he may comment on features and content he likes, dislikes, or would change. We will capture these and look across participants for trends.

What equipment will be used?

Recording & Observation

All of the sessions will be recorded and streamed live using screensharing software. We invite members of the U.S. Mint marketing, web, and content teams to login and watch the sessions live as they are happening. If observers miss a session all will be recorded an uploaded so that observers may watch at a later time if needed.

UX TAsks

#	UX GOALS	SCENARIOS / QUESTIONS
0.	Warm-up	Today, I’d like to ask for your feedback on the United States Mint catalog website. First, can you tell me about your interest in coins and your experience collecting coins? Have you ever visited the United States Mint website? If so, do you remember what you were looking for? Do you remember your reaction to the website? How do you learn about coins and purchasing coins? What types of coins are you most interested in?
1.	Gold Coins Gold Coins	Part 1: Let’s pretend you saw this ad while scrolling your Facebook feed. What is your reaction to this ad? If you saw this while on Facebook what would you do? Would an ad like this motivate you to click on “Shop Now”? Why or why not? Part 2: Go ahead and click on shop now. From Gold Coins page: What is your reaction to this page? When looking at coins on this page, what order do you think they are shown in? Which of these coins would be most interesting to you and why? Part 3: Let’s pretend you have a budget and would like to see the coins in order of least expensive, how would you do this? What do you think it means when it says “Limited”? Take a look at item that says “Learn More” or “Shop Now”, how do those differ from the other products on the page? To Note: Do users notice the shelf toppers?
2.	Silver Coins Silver Coins	Part 1: Now that we’ve looked at gold coins, without using the search, how would you find all the silver coins offered by the U.S. Mint? What is your reaction to this page? What would you do next? Would you ever want to see all silver coins on one page, or do you prefer to have them categorized like this? Why? Part 2: Let’s take a look at the menu again, please click on “Shop All Precious Metals Coins”. What do you think of the organization of this page? Would you ever want to see all the precious metal coins on one page, or do you prefer to have them categorized like this? Why?
3.	Proof Sets Proof Sets	Part 1: Let’s pretend that you’d like to purchase a set of coins from 2020 that includes the $1 coin, Kennedy half dollar, dime, nickel, and penny. How would you do this? (if users try to search, allow it, but then ask them to find this using the menu) What do you think the difference is between these sets? Which set would you choose and why? Let’s take a look at the item you chose. What do you think of this product page? What would you do first on this page? What information on this page is most helpful to you? How would you look at images of the coin? Which image is most interesting to you and why? What do you think about the information presented on this page? Is there anything unclear or confusing to you? Did you notice the mintage limit? What do you think that is? What about Product and Household Limits, what do you think those are? Without clicking anything, can you tell me how you would purchase this item? Part 2: Would you be interested in purchasing an annual set every year, why or why not? If you wanted to purchase this set every year, how would you do that? (Do users notice the enrollment section and know how to use it?) To Note: Where do users go to find this information? How do they use the menu? Do users know where to look? Do they know the difference between proof sets, uncirculated sets, and circulating sets? Do they know what is meant by a ‘bundled set’? If they go to the “uncirculated set” or “circulating set” do they see the “also purchased with section”? Do they notice the “proof set” available from this section?
4.	Product Schedule Product Schedule	Part 1: How would you see a list of all the coins that the U.S. Mint will be releasing this year? What do you think of the organization of this page? How would you like to see this page organized? What would be most helpful to you? Part 2: How would you find out when the 2020 Proof set was released? Part 3: How would you see the coins that are coming out next month? What do you think of the ‘remind me’ feature? What do you think this will do? Would you ever sign up to get a reminder? Why or why not? If so, would you prefer an email or a text alert? Why? To Note: Do users expect to land on current month or on January?
5.	American Eagle Coins American Eagle Coins	Part 1: Without searching, how would you find a list of the American Eagle Coins on this site? What is your reaction to this page? What do you think of the organization of this page? Part 2: How would you see all of the uncirculated coins on this page? To Note: Do users know where to look? Are they able to effectively use the menu structure? Do users know how to use the filters?
6.	Gift Ideas Gift Ideas (Graduation)	Your niece recently graduated from school and you would like to get her something that she will remember forever. How would you select something to give to her? What is your reaction to the page? What do you think of the organization of this page? How would you like to see this page organized? What would be most helpful to you? What do you think “Graduation Gifts from the U.S. Mint” are? How do they differ from what is on this page? Click on “Shop Now” What is your reaction to this page? What would you do next? To Note: How do users approach this scenario? Do users go to “gift ideas”? If so, what do they think of the items shown to them? From the “Hats off to 2020 Grads” do users realize “Shop Our Catalog” takes them back to the Graduation Gift page?
7.	American Eagle Search & Personalization Scenario American Eagle	Part 1: Let’s pretend that you decided to purchase an American Eagle One Ounce Silver Proof Coin for your niece to celebrate her graduation. How would you search for this coin? What do you think of the search results? (Allow users to search) If users search: What is your reaction to the results? What do you think the difference is between “Products” and “Articles”? What is the difference between the “American Eagle 2020 One Ounce Silver Proof Coin” and the “American Eagle 2020 One Ounce Silver Proof Coin Enrollment”? I’m going to send you a link to view the “American Eagle 2020 One Ounce Silver Proof Coin” What is your reaction to the page? What information on this page is most important to you? Why? What do you think of the quality of the image shown? What is your reaction to the description? Is there anything about the description you don’t understand? Part 2: How would you have this item personalized for your niece? (If time permits) Please select the color you would like and the message. What is your reaction to the messages? Are there other messages you think should be included? If so, what? Would you like to be able to create your own message? What do you expect will be personalized? To Note: Do users click on pre-populated results or view the search page? Do users understand the difference between “Products” and “Articles”? Do users understand the difference between the coin and those marked “enrollments”?
8.	Homepage: Initial Impressions Homepage	Let’s take a look at the catalog homepage (http://www.catalog.usmint.gov/). What is your general reaction to this page? Before you click on anything on this page, I’d like to ask you a few questions about this homepage: What is your initial impression of this page? What draws your attention first on this page? What would you click on first? Why? How would you describe the information on this page? Is there too much information, not enough, or the right amount? Why? What is your reaction to the video? [will need to have video available during testing] Would you watch it? Why or why not? Which areas of this website would you most likely visit? If you could make one improvement to this page what would it be?
9.	Menu	Let’s take a look at the menu at the top of this page. You told me earlier that you are most interested in XYZ coin. How would you find that coin using this site? What do you think of the categories under the “coins” menu? Are there any categories that are unclear or confusing? What could we do to make this menu easier to use for someone who is new to coin collecting?
10.	Overall Reactions	Now that you have had a chance to use the website: What is your overall reaction to the website? Do you think this website works for someone who is new to coin collecting? Why or why not? What did you like best about this website? What did you like least about this website? If you could suggest one improvement, what would it be?

POST-TEST SATISFACTION QUESTIONAIRES

How would you rate your satisfaction with the following?

Very Dissatisfied Very Satisfied

Catalog homepage	1 2 3 4 5 6 7
Organization of content on the website	1 2 3 4 5 6 7
Overall look and feel (images, colors, etc.)	1 2 3 4 5 6 7
Product pages	1 2 3 4 5 6 7
Product descriptions	1 2 3 4 5 6 7
Product images	1 2 3 4 5 6 7
Gift purchase experience	1 2 3 4 5 6 7
Personalization experience	1 2 3 4 5 6 7
Search experience	1 2 3 4 5 6 7
Overall site navigation	1 2 3 4 5 6 7
OVERALL SATISFACTION WITH THE WEBSITE	1 2 3 4 5 6 7

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	UX Test Plan
Author	Jackie
File Modified	0000-00-00
File Created	2021-10-15