Remote Website and Mobile Application Usability Focus Groups

Developed FoR:

U.S. Mint Web Team

U.S. mint Catalog Website

Existing Customers

UX Test Plan

Objective: Gather feedback from customers to identify areas for improvement on catalog.usmint.gov.

UX TEST PLAN

WHAT IS THE PURPOSE OF THE U.S. MINT UX TEST?

The purpose of the usability test is to gather data on how users interact with the United States Mint catalog website. The usability test will evaluate the effectiveness of website changes that have occurred over the last year, with a specific focus on the integrated navigation and improved content pages.

This data gathered during the usability test will be used to better understand users’ perceptions of the website, to evaluate users’ abilities to find information using website, and to inform future improvements to the website.

The UX study will:

Gather users’ impressions of website

Measure users’ abilities to find information using the website

Identify difficulties in using the website

Suggest research-based recommendations for improvement

When & where will the testing be conducted?

16 UX sessions will be conducted remotely on both desktop and mobile devices using screensharing software:

Members from the U.S. Mint marketing, web, and content teams are encouraged to observe the sessions remotely, as all of the sessions will be streamed live and recorded. Audio/video recordings will be uploaded at the end of each week.

To the extent possible, users will have a mix of demographic characteristics, including age, gender, education, ethnicity, internet experience, geographic location, and purchasing history with the U.S. Mint.

What will happen during the UX test?

Each session will last approximately 60 minutes.

During the session, participants will:

• Provide initial impressions of the homepage and email newsletter.

• Perform a series of tasks using the catalog website.

• Provide overall reactions to using the catalog website.

The facilitator will observe and objectively probe on issues. The facilitator will not coach or guide the participants.

Following is a high-level breakdown of the 60-minute sessions (these time limits are simply an approximation of the test session):

Duration	Activity	Administered By…
0:00 – 0:03	Introduction / Greeting	Facilitator
0:03 – 0:05	Think Aloud Instructions	Facilitator
0:05 – 0:10	Initial Impressions	Facilitator
0:10 – 0:50	Usability Tasks	Facilitator
0:50 – 0:55	Overall Impressions	Facilitator
0:55 – 1:00	Closing – Thank You	Facilitator

Test Protocol

The following protocol will be used during in-person moderated usability tests:

1. Prior to the test, recruiting firm gathers demographic information from participant.

2. Prior to the test, participants will participate in a 15 min set-up session to ensure they are able to connect remotely via their desktop/mobile device.

3. Facilitator greets participant and introduces self.

4. Facilitator very briefly explains hour, sets expectations, and asks if participant has questions.

5. Facilitator asks participant to share their screen.

6. Facilitator shows participant webpage and gathers initial impressions.

7. Facilitator explains thinking aloud and begins with scenarios.

8. Participant performs scenarios one at a time.

9. Facilitator monitors participant’s actions, comments, frustrations, etc.

10. Facilitator monitors time so that there is time at the end for final questions.

11. Facilitator elicits overall reactions and answers to specific end-of-session questions.

12. Facilitator thanks the participant.

Please note: All participants will receive the same instructions before the usability test, based on the sample script (next page). Great care will be taken to ensure participants all receive similar instructions before beginning the usability test.

Facilitator Introduction

The following introduction is a sample and will be used as a guide, not read to participants verbatim.

Hello, my name is Cari Wolfson/Kristina Sommer and I am a usability researcher who is working with the United States Mint in an effort to improve their website. Today we are talking with users like you who may use the website to learn more about where improvements could be made to enhance the overall user experience. Thank you for agreeing to participate in our usability study.

Today, I am going to show you a website and ask for your honest feedback. So, please let us know what you like about the site as well as what you don’t like. Your opinions are very important to us and will help us improve the site so that it meets the needs of people like you.

During our session, I am going to ask you to perform a few activities, or tasks, using the site. I am planning to give you each task one at a time and ask you to read the task out loud. As you go through each of the tasks, please try to “think out loud” and tell us what you are doing and what you expect to find on the site. I’ll demonstrate what we mean by “thinking aloud” in a minute.

 

Please feel free to ask me a question about the tasks or the site. However, I may not be able to answer your question, as I was not involved in the development of the website. If that happens, please continue to work as you would if you were at home or at work.

Your comments are very important, so I’ll be taking notes as well as recording our voices and screen interactions during the session.

This information will be kept strictly confidential and used only for analysis.

The session should take about an hour. At the end of the session, we’ll then have a chance for you to tell us about your experience and give us any feedback you may have.

 

Do you have any questions before we begin?

What will be tested?

The following scenarios have been developed to test various aspects of the United States Mint catalog website. While the participants complete the scenarios, qualitative and quantitative data will be captured by the facilitator and note takers.

Please note:

Prior to the usability scenarios, the facilitator will explain the purpose of the usability test and will conduct pre-test activities.

Please refer to:

• Test Protocol

• Facilitator Introduction

• Facilitator Outline

• Consent forms

The scenarios on the following pages will be provided to the users verbatim, so that participants hear the exact same questions.

The probing questions made by the facilitator will vary from participant to participant and will be based on the facilitator’s judgment of what to test and the time allotted in the session. The probing questions that follow are simply a guide to assist discussion and will be used by the facilitator to learn more about specific concepts.

Lastly, the estimated times below are simply an estimate of the time it will take for each scenario. Some users may complete tasks faster and others may struggle and not be able to complete all of the following scenarios.

How will the tasks be measured?

In order to measure users’ performance on the website, several quantitative metrics and qualitative data will be collected, including:

Successful Task Completion

Each scenario will ask participants to complete a task and find specific information using the website. The scenario is considered a successful completion when the participant says he has achieved the goal AND has done so successfully. To ensure that tasks are judged correctly, successful task completion criteria have been defined in this plan so that all team members can review and contribute to the definitions of success. This will help to ensure that observers all score each task in the same way.

Critical Errors

Critical errors are large deviations from the intended path for the scenario. Participants may or may not be aware that the task goal is incorrect or incomplete. As an example, a participant providing the wrong answer to the scenario question, while thinking he was correct, is a critical error. Other examples include intervention from a facilitator because either the participant had gone so off-course or had made mistakes that prevented him from completing the scenario correctly.

Non-critical Errors

Non-critical errors are mistakes that the participant is able to recover from. These may be clicking the wrong link, choosing the wrong feature, trying to select something that is disabled, or another minor mistake.

Wrong Path

Wrong path errors include those for which the participant had a “false start” and went down the wrong navigation path or workflow path. These are also reflected by critical and non-critical errors but are captured at a higher level. Wrong path errors reflect major issues with navigation or workflow design. To help measure wrong path, users’ first clicks are recorded and identify where users initially went to complete a task.

Subjective Measures

The participants’ satisfaction, ease of use, ease of finding information, and other relevant subjective measures will be captured via a post-test questionnaire. Participant confidence and judgments on ease of use will be collected after each usability test.

Likes, Dislikes, and Recommendations

As the participant completes each scenario, he may comment on features and content he likes, dislikes, or would change. We will capture these and look across participants for trends.

What equipment will be used?

Recording & Observation

All of the sessions will be recorded and streamed live using screensharing software. We invite members of the U.S. Mint marketing, web, and content teams to login and watch the sessions live as they are happening. If observers miss a session all will be recorded an uploaded so that observers may watch at a later time if needed.

UX TAsks

#	UX GOALS	SCENARIOS / QUESTIONS
0.	Warm-up	Today, I’d like to ask for your feedback on the United States Mint catalog website. Before we get started, I’d like to briefly ask about your purchasing history. In just a few sentences, can you tell me which of the following ways you typically purchase products: do you have a certain type of coin(s) or coin set that you purchase every year; or do you purchase new coins after receiving mailings or emails; or do you browse items on the website to find new releases? Desktop Users: Do you ever view the products from your phone?
1.	Email Newsletter Desktop Mobile	Do you currently receive email updates/newsletters from the U.S. Mint? If yes: How often would you say you receive an email from the U.S. Mint? Do you typically read the emails when they arrive? Why or why not? What type of content would you like to receive in an email newsletter? What other types of promotional emails would you be interested in receiving? How often would you like to receive email updates from the U.S. Mint? If no: Would you consider signing up to receive email updates/newsletter from the U.S. Mint? Why or why not? What type of content would you like to receive in an email newsletter? How often would you like to receive email updates from the U.S. Mint? Before we look at the website, I’d like to share with you a recent newsletter that was sent out by the U.S. Mint. [Facilitator will share link via Zoom or TeamViewer Chat]. What is your reaction to the newsletter? What information is most interesting to you? What would you click on and why? What do you expect you will see? Possible probes: What do you think the “Sign Up for Reminder” feature and “Add to My Calendar” will do? Would you use either of these features? Which ones and why? Click on “Sign Up for Reminder”. Is this what you expected? Why or why not?
2.	Silver Coins Silver Coins	You are interested in purchasing a silver coin. How could you view all the silver coins that are available? What is your reaction to this page? What would you do next? Would you ever want to see all silver coins on one page, or do you prefer to have them categorized like this? Why?
3. 3.	Military Medals / Campaign Military Appreciation Military Medals / Campaign Medals	Part 1: You heard that the U.S. Mint now sells military-inspired products saluting the history of our troops. Let’s pretend you googled military coins and ended up on this page. What would you look at first and why? Campaign Page: Let’s take a look at 2-3 of those boxes. What is your reaction to the content displayed? Is there anything else you would want to know about a collection like this? If so, what? Let’s take a look at the bottom of the page. What is your reaction to the three coins and print collection? What do you think those are? What do you like best about this page? What one improvement would you make to this page? Part 2: How would you view ALL the military medals? To Note: Do users see the “Explore the Collection” button? Military Medals Page: Take a minute to scroll the page to look at the full collection. What is your reaction to the additional coins that appear the further you scroll the page? How would you like this page to be organized? By branch of service? By war or engagement? Part 3: You have a budget of $25, how would you view coins in your price range? If users use “Sort By” – is there another way to do this? If users use “price $1-$25” – is there another way to do this? How could this page be improved to make it easier for you to find coins? To Note: Do users use the “sort by” feature or select the “price” feature?
4.	Add Product + Enrollment Silver Proof Coin	Part 1: You are interested in the 2020 silver proof set. How would you find this collection? What do you think of this page? Which information is most important to you on this page? Why? Is there anything additional you would like to see on this page? If so, what? Did you notice the “Also purchased with section”? What is your reaction to those? If you could make one improvement to this page what would it be? Part 2: How would you purchase this set? At end of task moderator will ask user to empty their bag before performing the next task. To Note: Do users see the “add product + enrollment” button? Do they understand the difference between “add to bag” and “add product + enrollment”? If users did not comment on enrollment, ask in “bag summary” if users would like an option to “enroll” available to them.
5.	Gift Certificate Purchase Experience Gift Certificate	Your son is interested in expanding his collection, but you aren’t sure which coin he would like to purchase next. How would you purchase a $100 gift certificate for him? From Gift Certificate page: What do you think of the gift certificate value amounts? Would you want to be able to select your own value amount? From Checkout page: Whose information would you enter in the “shipping and delivery” section? How will the gift certificate be delivered? What email address would you enter? Yours or your son’s? Would you like to be able to add a personal note to accompany the gift certificate? To Note: Do users understand that gift certificates are sent via email? Is “shipping address” confusing to users?
6.	Personalization American Eagle Personalization (CON’T) Desktop Mobile	Part 1: Your niece’s birthday is coming up and you would like to get her something that she will remember forever. How would you purchase an American Eagle One Ounce Silver Proof Coin? When user gets to American Eagle 2020 One Ounce Silver Proof Coin page – moderator will give them a URL to prototype I’d like for you to view the American Eagle 2019 One Ounce Silver Proof Coin page, can you click on the link I put in the chat? What is your reaction to the page? What information on this page is most important to you? Why? What do you think of the quality of the image shown? What is your reaction to the description? Is there anything about the description you don’t understand? To Note: Do users see the personalization feature? Part 2: How would you have this item personalized for your niece? Please select the color you would like and the message. What is your reaction to the messages? Are there other messages you think should be included? If so, what? Would you like to be able to create your own message? What do you expect will be personalized? Can you view an example? Did you notice the personalization in your cart? How would you add gift wrapping? To Note: Do users understand the packaging will be engraved or do they think the coin will be engraved? What are users’ reactions after they select a color and message and see the sample vs. the actual text they entered?
7.	Manage Account My Account Manage Account (CON’T)	Moderator will ask user to click on U.S. Mint logo to be taken to home screen [½ users will be shown confirm payment details email and ½ will be shown verify payment details email]. Task 1: Let’s pretend you got this email about your upcoming order. After looking at this email, what would you do? To Note: Do users know how to update their credit card? Do they know which card is going to be used? Do users know how to update their address based on this email? Task 2: You want to update your payment method with a new credit card. How would you do that? Which payment method do you prefer? Why? What do you think will happen if you click on “Connect PayPal Account”? What other payment methods would you like to use? Do you currently have a saved payment option? Why or why not? To Note: Do users click on “Add Credit Card”, “Delete Card”, or “Connect PayPal Account” Task 3: Let’s pretend that you moved, how would you change your shipping address so that your products go to the new house? [Need to be able to give users fake account info to log in if they don’t want to use their own]. To Note: Do users click on “Create New Address” or the “Edit” button below the current address? Do users want to edit their current address? Do they want to delete an old address? Do users want to be able to make one address the “default” address?
8.	Homepage: Initial Impressions Homepage	Let’s take a look at the catalog homepage (http://www.catalog.usmint.gov/). What is your general reaction to this page? Before you click on anything on this page, I’d like to ask you a few questions about this homepage: What is your initial impression of this page? What draws your attention first on this page? What would you click on first? Why? How would you describe the information on this page? Is there too much information, not enough, or the right amount? Why? What is your reaction to the video? [will need to have video available during testing] Would you watch it? Why or why not? Which areas of this website would you most likely visit? If you could make one improvement to this page what would it be?
9.	Overall Reactions	Now that you have had a chance to use the website: What is your overall reaction to the website? What did you like best about this website? What did you like least about this website? Did you encounter any challenges or have any difficulties using the site today? What recommendations for improvement do you have?

POST-TEST SATISFACTION QUESTIONAIRES

How would you rate your satisfaction with the following?

Very Dissatisfied Very Satisfied

Catalog homepage	1 2 3 4 5 6 7
Organization of content on the website	1 2 3 4 5 6 7
Overall look and feel (images, colors, etc.)	1 2 3 4 5 6 7
Product pages	1 2 3 4 5 6 7
Product descriptions	1 2 3 4 5 6 7
Product images	1 2 3 4 5 6 7
Gift card experience	1 2 3 4 5 6 7
Product enrollments experience	1 2 3 4 5 6 7
Personalization experience	1 2 3 4 5 6 7
Managing account	1 2 3 4 5 6 7
Subscribing/email newsletter	1 2 3 4 5 6 7
OVERALL SATISFACTION WITH THE WEBSITE	1 2 3 4 5 6 7