CEE Usability Testing (ASK and PFC)

Moderator’s Guide for Ask CFPB Usability Test

Introduction

Thank you for participating in this study today. My name is _____, and I work with the User
Experience Team at Fors Marsh Group. Today we will be evaluating a website that has
financial information. I will give you a series of tasks that I would like you to complete using
the website. Your comments and feedback will help the developers improve the site. I did
not create the website, so please do not feel like you have to hold back on your thoughts to
be polite to me. We’re interested in both your positive and negative reactions to the website.
Difficulties you may run into reflect the design of the website, not your skills or abilities.
Please keep in mind that I am not testing you or your knowledge. Rather you are helping us
to see how we can improve this site.
Some elements of the website are not fully functioning, but I would like you to use the site
as if it were fully functioning. So I would like you to scroll and click as you normally would.
The entire session should last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• We are making a video recording of this session. Only those of us associated with this
project will see the recordings, and we will not share your name or personal
information. The recordings are used as a memory aid for me so I can go back later
to recall what happened during each session.
• I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help. So while you are working, I would like you to think aloud. I would like
you to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
o Tell me if you are looking for something and what it is and whether you can
find it or not.
o Tell me why you click on a link and what you expect to happen.
o Basically, just tell me everything you are thinking as you work.
• There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
• Any questions so far?
We are going to track where you are looking as you work.
• This is the eye tracker [point to tracker].
• Let’s get you situated in your chair in front of the screen in a good position.
Ask CFPB Moderators Guide  1

•

•

•

Please try to refrain from making any large head movements during the session. You
do not need to remain perfectly still like a statue, but if you move your head too
much, the eye tracker may lose your eyes.
Let’s do a quick task so the computer can find your eyes. Please follow the red circle
as it moves on the screen [calibration].

•

Ok, we are ready to begin. I will work from the other room, and we will communicate
through microphones and speakers. I will be able to see you with this camera [point
to camera].
Do you have any questions?

•

Can you hear me ok? Ok, we are ready to begin.

Notes:
• All tasks (except pre-tasks) begin on the home page. Participants will be redirected to
the home page at the conclusion of each task.
• After the participant reads the task, the moderator will start the eye tracker, and the
home page will load. The eye tracker will close the site at the end of each task.
• Participants will have the tasks on paper on the desk in front of them. Each task will
be on its own piece of paper, in large font.
• All participants will begin with the pre-tasks. After the pre-tasks, tasks 1 through 7
will be randomly assigned to prevent order effects. The moderator will place tasks in
the correct order before the session begins.
• Accuracy = percent of steps successfully completed for each task.
• Path = clicks made
• Task Time = page load to end of task (when they verbalize that they found the
answer)
• Moderator: Note any inconsistencies between users’ terminology and the site’s.

Ask CFPB Moderators Guide  2

Pre-Tasks (will always come first)
Pre1. You recently were interested in [fill in what they recently did or will recently do]. Tell
me a little bit about that process. What information did you need? What did you do to
start the process? [Moderator note resources e.g., websites, family/friends, etc.].
[Start on www.slate.com]
Now I would like you to use the Internet to find information. Just do what you normally would
do, and remember to think aloud. [Moderator: See if they go to CFPB on their own. If
not, bring them back to the search results. Ask them what they think about the CFPB
links, and then ask them to select a CFPB link. If there are none on the search
results, ask them the next question.]
[If they do not get to CFPB links in their search results on their own, lead them there.]
Pre2. [once within Ask CFPB] What would you do next? [Moderator: Probe further. If they do
not stay on the site, why? Did they get the answer they were looking for? What do
they think about the quality of the answer? What is missing?]
Pre3. Would you come back to this site for more information in the future? Tell me about
that – what would you use this site for?
Ok, now we are going to use this website for a few tasks. I would like you to start with the
first task in front of you and begin by reading the task aloud. Then I will pull up the
website, and I would like you to use it to find the information. When you feel you have
found it, I would like you to tell me “This is my final answer.” Ready?

Task 1: Student Loans (Difficulty: Easy)
1a.

Let’s say you are interested in taking out a loan for college courses. You have been
getting a lot of information in the mail with offers for private student loans. You want
to know the similarities and differences between private and federal student loans.
Please find information that compares these two types of loans.
Accuracy: ________
Correct Paths:
1. Prototypes: 1 (Home page)  2 (Search results page)  4 (Answer page 1
with left nav, office name, number of visits, and rating)
2. Search “private and federal student loans”  Search  Select second option
3. Student loans (Other categories)  Select second option
http://www.consumerfinance.gov/askcfpb/545/what-are-main-differences-between-federalstudent-loans-and-private-student-loans.html

Ask CFPB Moderators Guide  3

1b.

How trustworthy is the information on this page?

Not
Trustworthy
At All
1
1c.

Slightly
Trustworthy
2

Moderately
Trustworthy
3

Very
Trustworthy
4

Extremely
Trustworthy
5

(If 1a is successful) If you wanted to share this information with someone who is
helping you with this decision, how would you do that? [Do participants bookmark,
email, print, etc.? Do they use the left nav to go back?]
Accuracy: ________
Path: __________

1d. How confident are that the website answered your question? [Do participants mention
the office that posted content, number of visits and rating?]
Not Confident
At All
1

Slightly
Confident
2

Moderately
Confident
3

Very
Confident
4

Extremely
Confident
5

Task 2: Payday Loans (Difficulty: Intermediate)
2a.

You have a bill due and you are short on cash. A friend has told you that you can take
out a cash advance on your paycheck, and you’re interested in finding out more
about this. When would you have to repay this cash advance?
Correct Answer: Next paycheck
Accuracy: ________
Task Time:___________
Correct Paths:
1. Prototypes: 1 (Homepage)  live search results page  6 (Answer page 3
with text box that allows users to post info, shows few posts, office name,
number of visits, rating)
2. Search “cash advance”  Search  Select last option
3. Search “payday loan”  Search  Select first option
4. Payday loans (Other categories)  Select first option
http://www.consumerfinance.gov/askcfpb/1567/what-payday-loan.html
2c. How confident are that the website answered your question? [Do participants mention
the office that posted content, number of visits and rating?]
Not Confident
At All
1

Slightly
Confident
2

Moderately
Confident
3

Very
Confident
4

Extremely
Confident
5

Ask CFPB Moderators Guide  4

2d. How trustworthy is the information on this page?
Not
Trustworthy
At All
1

Slightly
Trustworthy
2

Moderately
Trustworthy
3

Very
Trustworthy
4

Extremely
Trustworthy
5

2e. Let’s say that you have been through this process and found the experience very
different from the way the website explains it. How would you report this difference to the
CFPB team? [Do participants mention the text box at the bottom of the page or office that
posted content?]
2f.

What do you expect to happen next?

2g.

How confident are you that it would happen this way?

Not Confident
At All
1

Slightly
Confident
2

Moderately
Confident
3

Very
Confident
4

Extremely
Confident
5

Task 3: Credit Report and Scores (Difficulty: Intermediate)
3a.

Let’s say you’re interested in buying a new car, and you have heard that only credit
scores above 700 qualify for the 0% interest loan. Before you go to the dealership,
you want to review your credit report and score. How would you obtain your credit
report?
Correct Answer: You can request your free report through the website
AnnualCreditReport.com, by calling 1-877-322-8228, or by filling out the Annual
Credit Report Request form (available at AnnualCreditReport.com) and mailing it to
Annual Credit Report Request Service, P.O. Box 105281, Atlanta, GA 30348-5281.
Accuracy: ________
Task Time:___________
Correct Paths:
1. Search “credit report”  Search  Select fifth option
2. Credit reports and scores (Other categories)  Select third option
http://www.consumerfinance.gov/askcfpb/5/can-i-review-my-credit-report.html
3. Search “credit score”  Search  Select fourth option
http://www.consumerfinance.gov/askcfpb/6/i-got-my-free-credit-reports-but-they-do-notinclude-my-credit-scores-can-i-get-my-credit-score-for-free-too.html
3b. How confident are that the website answered your question? [This question will be on
the live website. Compare their reasons to the prototypes that have the office that posted
content, number of visits and rating?]
Not Confident
At All
1

Slightly
Confident
2

Moderately
Confident
3

Very
Confident
4

Extremely
Confident
5
Ask CFPB Moderators Guide  5

3c. How trustworthy is the information on this page?
Not
Trustworthy
At All
1

Slightly
Trustworthy
2

Moderately
Trustworthy

Very
Trustworthy

3

4

Extremely
Trustworthy
5

Task 4: Mortgages (Difficulty: Hard)
4a.

Let’s say that you paid off your mortgage a few months ago, but you just checked
your credit report and the mortgage is still listed as a debt. What can you do to
correct your credit report?
Correct Answer: You should file a dispute with the credit bureaus about the error.
You can also file a complaint with the CFPB online or call 1-855-411-CFPB (2372) to
file a complaint and be connected to a U.S. Department of Housing and Urban
Development (HUD)-approved housing counselor.
Accuracy: ________
Task Time:___________
Correct Paths:
1. Search “credit report error”  Search  Select fifth option
2. Mortgages  housing counseling filter (left)  Select third option
http://www.consumerfinance.gov/askcfpb/222/i-paid-off-my-mortgage-but-it-is-still-listedon-my-credit-report-as-outstanding-what-can-i-do.html
4b. How confident are that the website answered your question? [This question will be on
the live website. Compare their reasons to the prototypes that have the office that posted
content, number of visits and rating?]
Not Confident
At All
1

Slightly
Confident
2

Moderately
Confident
3

Very
Confident
4

Extremely
Confident
5

4c. How trustworthy is the information on this page?
Not
Trustworthy
At All
1

Slightly
Trustworthy
2

Moderately
Trustworthy
3

Very
Trustworthy
4

Extremely
Trustworthy
5

Task 5: Print multiple questions (Difficulty: Hard)
5a.

Your friend would like to wire money, and you are going to give him information from
this site. Find information about wiring money and print the top 3 most visited items
about wiring money.
Accuracy: ________
Task Time:___________
Correct Path:
Ask CFPB Moderators Guide  6

1. Prototypes: 1 (Home page)  3 (“Bundles” Results page)
2. Scroll to bottom  wire money  most viewed  click on one  print icon 
right click  Print  Save  back  repeat 3 times (6 clicks each)
5b. What would you expect to happen next? [Probe about how the printing would work.]

Task 6: Quality of Answer (Difficulty: Medium)
6a.

You have heard about negative amortization as a way to pay off loans, and you are
not sure what this means. Find out what this means using this site.
Accuracy: ________
Task Time:___________
Correct Paths:
1. Prototypes: 1 (Home page)  live search results  5 (Answer page 2, with
images, actionable items, office name, number of visits, and rating)
2. Search “negative amortization”  Search  Select second option
http://www.consumerfinance.gov/askcfpb/103/what-is-negative-amortization.html
6b.

How would you explain ‘negative amortization’ to a friend?

6c. How trustworthy is the information on this page?
Not
Trustworthy
At All
1

Slightly
Trustworthy
2

Moderately
Trustworthy
3

Very
Trustworthy
4

Extremely
Trustworthy
5

6d.
How clear is the information on this page? [Do participants mention the images,
actionable items, office that posted content, number of visits and rating?]
Not Clear
At All
1

Slightly
Clear
2

Moderately
Clear
3

Very
Clear
4

Extremely
Clear
5

Task 7: Filters (Difficulty: Medium)
7a.

You want to help your grandmother with her finances, and you are looking for tips.
Find information on this site for the elderly.
Correct Paths:
1. Older Americans (lower right)
http://www.consumerfinance.gov/askcfpb/search?q=&selected_facets=audience_exact%3
AOlder%20Americans

Ask CFPB Moderators Guide  7

7b.

Now you are interested in learning about reverse mortgages for your grandmother.
Correct Paths:
1. Reverse mortgage filter (left)
http://www.consumerfinance.gov/askcfpb/search?selected_facets=audience_exact%3AOld
er+Americans&selected_facets=tag_exact%3Areverse+mortgage
7c. Do you need to purchase an annuity to get a reverse mortgage?
Correct Answer: No
Accuracy: ________
Task Time:___________
Correct Paths:
1. Annuity filter (left)  select first option
http://www.consumerfinance.gov/askcfpb/239/i-was-told-i-had-to-buy-an-annuity-or-aninvestment-product-to-get-a-hecm-reverse-mortgage-is-this-true.html
7d. How confident are that the website answered your question?
Not Confident
At All
1

Slightly
Confident
2

Moderately
Confident
3

Very
Confident
4

Extremely
Confident
5

7e. How trustworthy is the information on this page?
Not
Trustworthy
At All
1

Slightly
Trustworthy
2

Moderately
Trustworthy
3

Very
Trustworthy
4

Extremely
Trustworthy
5

7f. In your own words, what did you learn from this task? [Moderator probe about the key
takeways.]

Ask CFPB Moderators Guide  8

Satisfaction Questionnaire (SATQ)

Please circle the numbers that most appropriately reflect your impressions about using this device.

1. Please rate your overall experience using the website:
Did not like it
Did not like it
Neutral
Liked it
at all
1
2
3
4

Liked it a lot
5

2. How helpful were the responses to the questions on the website?
Not Helpful
Slightly
Moderately
Very
Extremely
At All
Helpful
Helpful
Helpful
Helpful
1
2
3
4
5
3. How trustworthy is the information on the site?
Not
Slightly
Moderately
Very
Trustworthy
Trustworthy
Trustworthy Trustworthy
At All
1
2
3
4
4. How likely would you be to use this site in the future?
Not Likely
Slightly
Moderately
Very
At All
Likely
Likely
Likely
1
2
3
4
5.

How likely would you be to recommend this site to a friend?
Not Likely
Slightly
Moderately
Very
At All
Likely
Likely
Likely
1
2
3
4

6.

Please rate the organization of the website in general:
Not Clear
Slightly
Moderately
Very
At All
Clear
Clear
Clear
1

7.

3

Was the search function of the website helpful?
Not Helpful
Slightly
Moderately
At All
Helpful
Helpful
1

8.

2

2

3

Was the site easy or difficult to navigate?
Somewhat
Neither Easy
Very Easy
Easy
nor Difficult
1
2
3

Extremely
Trustworthy
5
Extremely
Likely
5
Extremely
Likely
5
Extremely
Clear

4

5

Very
Helpful

Extremely
Helpful

4

5

Somewhat
Difficult
4

Extremely
Difficult
5

Ask CFPB Moderators Guide  9

Debriefing Questionnaire
(NOTE: Some are used as follow-ups to actions during session and to SATQ
responses; participants do not receive all debriefing questions.)
1. How would you describe this site to a friend?
2. Tell me about your overall experience using this website. What worked well; what did
not work well? [Probe further as necessary.]
3. How would you describe this website in your own words?
4. How did you feel about the navigation? Were you able to find the information you
were looking for?
5. How about the filters on the left – tell me about your experience using those.
6. What do you think about the “most viewed,” “most helpful,” and “recently updated”
links? How about the pages that had boxes on the bottom – what did you think about
those?
7. Tell me about your experience using Search. Did you get what you were looking for?
8. What do you think about the process of asking a question? [Probe further as
necessary. Did they like it? What will happen next? How will they get the answers to
their posed questions?]
a. Does this seem like something you might use on your own?
9. What was your level of trust like for the different pages you viewed? [Probe about why
they do or do not trust content.]
10. What did you think about the process of printing?
11. If you were using this on your own, would you want to print? [Probe: What would you
want to print? How would you expect it to work and look? Would you want more than
one question on a single page?]
12. What do you think about the overall look and feel of the website?
13. What else were you expecting to do on this website that you couldn’t do? What is
missing?
14. What would you say is the best thing about this website?
15. What would you say is the worst thing about this website?
16. Do you think you would use this site in the future when you have a financial
question? Why or why not?
17. [Follow up regarding likihood to recommend (SATQ4)] Why are you likely? What about
the site makes you likely to recommend. [If not likely] What is the one thing that you
would recommend on this site?
18. [Follow up regarding some of the SATQ items.]
19. [Follow up re Pre-questionnaire 6e.]
20. On the live website, people tend to use a search engine to get to the page on this
website, and they leave quickly. Now that you’ve interacted with the website, why do you
think this may be happening? Is there a way that we can keep people on the website?
21a. [For round 1 sessions: Half of the participants get the “open-sort task”] Now we are
going to do one final task. I would like you to look at the text on these three pieces of
paper, and I would like you to describe to me, in your own words, what these
functionalities are. Imagine you see these on the website you just worked with – what do
Ask CFPB Moderators Guide  10

you think would happen if you clicked on these?
21b. [Half of the participants get the “closed-sort task”] Now we are going to do one final
task. I would like you to look at the text on this piece of paper, and I would like you to
label the functionality. Imagine that the website you just worked with has this
functionality. What would you call it?

Ask CFPB Moderators Guide  11