CEE Usability Testing (ASK and PFC)

Moderator’s Guide for Paying for College Usability Test

*Note: Tasks are written for high school students who wish to explore college options. Tasks will be tweaked
for parents and couselors/teachers who help students with college decisions.

Introduction

Thank you for participating in this study today. My name is _____, and I work with the User
Experience Team at Fors Marsh Group. Today we will be evaluating a website that has
financial information. I will give you a series of tasks that I would like you to complete using
the website. Your comments and feedback will help the developers improve the site. I did
not create the website, so please do not feel like you have to hold back on your thoughts to
be polite to me. We’re interested in both your positive and negative reactions to the website.
Difficulties you may run into reflect the design of the website, not your skills or abilities.
Please keep in mind that I am not testing you or your knowledge. Rather you are helping us
to see how we can improve this site.
The entire session should last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• We are making a video recording of this session. Only those of us associated with this
project will see the recordings, and we will not share your name or personal
information. The recordings are used as a memory aid for me so I can go back later
to recall what happened during each session.
• I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help.So while you are working, I would like you to think aloud. I would like
you to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
o Tell me if you are looking for something and what it is and whether you can
find it or not.
o Tell me why you click on a link and what you expect to happen.
o Basically, just tell me everything you are thinking as you work.
• There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
• Any questions so far?
We are going to track where you are looking as you work.
• This is the eye tracker [point to tracker].
• Let’s get you situated in your chair in front of the screen in a good position.
PFC Moderators Guide  1

•

•

•

•
•
•
•

Please try to refrain from making any large head movements during the session. You
do not need to remain perfectly still like a statue, but if you move your head too
much, the eye tracker may lose your eyes.
Let’s do a quick task so the computer can find your eyes. Please follow the red circle
as it moves on the screen [calibration].
Ok, we are ready to begin. I will work from the other room, and we will communicate
through microphones and speakers. I will be able to see you with this camera [point
to camera].
Please complete this questionnaire as I go to the other room now and do a sound
check.
Do you have any questions?
Can you hear me ok?
Ok, we are ready to begin. I would like you to start with the first task in front of you
and begin by reading the task aloud. Then I will pull up the website, and I would like
you to use it to find the information. When you feel you have found it, I would like you
to tell me “This is my final answer.” Ready?

Notes:
• All tasks (except pre-task) begin on the home page. Participants will be redirected to
the home page at the conclusion of each task.
• After the participant reads the task, the moderator will start the eye tracker, and the
home page will load. The eye tracker will close the site at the end of each task.
• Participants will have the tasks on paper on the desk in front of them. Each task will
be on its own piece of paper, in large font.
• All participants will begin with the pre-task. After the pre-task, tasks will be randomly
assigned to prevent order effects. The moderator will place tasks in the correct order
before the session begins.
• Accuracy = percent of steps successfully completed for each task.
• Path = clicks made
• Task Time = page load to end of task (when they verbalize that they found the
answer)
• The first time participants use search, the moderator will probe about terms they
would enter and their expectations (before clicking search).
• After 3 tasks, if participants ONLY use search, prompt them with “without using
search.”
• Moderator: Note any inconsistencies between users’ terminology and the site’s.

PFC Moderators Guide  2

Pre-task interview: Process in Comparing Schools

Let’s say you interested in finding out more about the different financial decisions that you
need to make about education after graduating high school. What would you do? [If
necessary] Who would you talk to? How would you research this? [Moderator: What
resources does participant mention? What process? Websites? Search terms they use?]

Task 1: Navigation to pay for college/first impressions (Difficulty: Easy)
Start on main CFPB home screen.

1a. Let’s say you interested in finding out more about the different financial decisions that
you need to make about education after high school. What should happen before you
compare schools? (e.g., Apply, Find Financial Aid, FAFSA – on timeline) [Moderator: Does
participant understand that the timeline reflects a linear process? Or something else?]
Correct path: home page  Pay for college (or Get started)
1b. What are your first impressions of this website?
1c. Who is this site for?
1d. What would you do next? [Moderator: Probe further – Tell me more about that. What
would get you to explore this site?]

Task 2: Compare colleges (Difficulty: Medium)
Start on Paying for College.
2a. Let’s say that you are interested in going to college, but you are having a hard time
deciding between a public and private university. Approximately how much more is a 4-year
private university likely to cost?
Correct Answer: ~80K
Correct Paths:
1. Get started (under Compare financial aid and college cost)  Add a school  Set
values to average private 4-year university
2. Choose a school (on timeline)  Add a School  Set values to average private 4year university
2b. Now let’s say that you have your two schools narrowed down, and you’re deciding
between the University of Virginia - Main Campus and Georgetown, and you are interested in
graduation rates. Which university graduates a higher percentage of students who enroll?
[Moderator note: $$ does not auto-populate. What do they say/do?]
Correct Answer: University of Virginia – Main Campus
Correct path:
PFC Moderators Guide  3

1. Remove this school  Yes (Are you sure you want to remove)  Add a school 
Search University of Virginia – Main Campus  Click George Mason University 
Continue (Bachelors)  Continue (4 years)  Continue (It looks like this school
adopted the Financial Aid Shopping Sheet.)  Add a school  Search Georgetown
University  Click Georgetown University Continue (Bachelors)  Continue (4
years)  Continue (You’ll need your financial aid offer info…)  Scroll to information
on page
2c. Let’s say you want to keep a copy of this information so you can show it to a family
member. Get a copy of this information to use later.
Correct path:
1. Save & Share  enter email address  Send
2. Save & Share  copy URL (will save #s entered)  email to self
3. Ctrl-P
2d. What do you expect to happen next, when you return? [Do they know that info is/isn’t
saved?]

Task 3: Compare loans (Difficulty: Medium)
Start on paying for college
3a. Let’s say you have decided on the university you want to attend and you know you’re
going to need to take out a loan to pay for tuition. What are the different types of loans that
are available to you? [Moderator: Which section of the page does the participant use to find
this information?]
Correct Answer: Federal or Private
Correct path:
1. Choose a loan  Scroll to compare federal and private
2. Get started (under Choose a Loan)  Scroll
3b. Which is better for you? [Do they understand? Does the content help them?]
3c. Ok, you now know that you need to take out a federal loan. What are the approximate
interest rates for federal loans?
Correct Answer: Interest rates range from 3.86% to 6.41%, depending on the loan program.
Correct path
1. Scroll down  Detailed comparison of federal and private loans  Look at “Interest
Rates” section
3d. You think your friends would be interested in this information too. How would you share
this with them? [What do they do? Share on social media? Bookmark? Print? If print, what
would that look like? How do they want it to look? What would they expect?]

PFC Moderators Guide  4

Task 4: Financial Aid Disbursement (Difficulty: Easy)
Start on paying for college

4a. You got a flier in the mail from a university you are interested in, and the flier advertises
a bank on campus. Using this site, find out if you have to use that bank or if you are free to
open an account at a bank off campus.
Correct Answer: People can open an account off campus
Correct path:
1. Manage Your Money  Do I have to get an account with the bank at my school?
2. Manage Your Money

PFC Moderators Guide  5

Satisfaction Questionnaire (SATQ)

Please circle the numbers that most appropriately reflect your impressions about using this device.

1. Please rate your overall experience with the Paying for College website:
Not Difficult
At All
1

Slightly
Difficult
2

Moderately
Difficult
3

Very
Difficult
4

Extremely
Difficult
5

2. Please rate the clarity of the information presented to you on the Paying for College
website:
Not Clear
At All
1

Slightly
Clear
2

Moderately
Clear
3

Very
Clear
4

Extremely
Clear
5

3. Please rate your difficulty with navigating on the Paying for College website:
Not Difficult
At All
1

Slightly
Difficult
2

Moderately
Difficult
3

Very
Difficult
4

Extremely
Difficult
5

4. How likely would you be to use this site in the future?
Not Likely
At All
1

Slightly
Likely
2

Moderately
Likely
3

Very
Likely
4

Extremely
Likely
5

5. How likely would you be to recommend this site to a friend?
Not Likely
At All
1

Slightly
Likely
2

Moderately
Likely
3

Very
Likely
4

Extremely
Likely
5

6. Please rate the organization of the website in general:
Not Clear
At All

Slightly
Clear

Moderately
Clear

Very
Clear

Extremely
Clear

1

2

3

4

5

7. Please rate whether the Paying for College website met your expectations:
Did not meet
expectations
1

Met
expectations
2

Exceeded
expectations
3
PFC Moderators Guide  6

Debriefing Questionnaire
(NOTE: Some are used as follow-ups to actions during session and to SATQ
responses; participants do not receive all debriefing questions.)
1. How would you describe this site to a friend?
2. What can this organization hel-p you with? What can’t they help you with? [Do users
understand the difference between this site and Dept of Ed?]
3. Tell me about your overall experience using this website. What worked well; what did
not work well? [Probe further as necessary.]
4. Let’s imagine you were to use this site on your own. What information would you look
for?
5. Would you share any of the information on social media? What would you share?
6. [If participant uses search] Tell me about your experience using Search. Did you get
what you were looking for?
7. [Follow up to SATQ4] What could be changed to increase the likelihood that you
would use this website?
8. [Follow up to SATQ4] As the website currently is, what is decreasing the likelihood
that you would use the Paying for College website?
9. What do you think the “Download action guide” is? [Probe further as necessary.]
10. What do you think about the process of comparing colleges? [Probe further as
necessary. Did they like it? What will happen next?]
a. Does this seem like something you might use on your own?
11. What else were you expecting to do on this website that you couldn’t do? What is
missing?
12. [For counselors] What is the one thing you would share with a student who was not
concerned with debt associated with college?
a. What about with a student who is very concerned with debt?
13. What did you think about this process timeline? [If necessary: Is in in the right order?
Anything missing?]
14. What did you think about the icons [on the timeline]?
15. What did you think about the video? [If participant did not watch video, ask him/her
to and then ask what they think.]
16. What did you think about the questions on the right [show the items in green boxes]?
17. [Follow up regarding some of the SATQ items.]
18. What would you say is the best thing about this website?
19. What would you say is the worst thing about this website?
20. How do you think this experience should be different on mobile? If you accessed this
site from your phone or tablet, what should be different?
21. What information/links should be the same/different?
22. Do you think you would use the comparison tool on mobile?

PFC Moderators Guide  7