CFPB Paying for College usability study (Round 2)

Moderator’s Guide for Paying for College Usability Test
Group B: Consumers with School-Related Debt

Introduction

Thank you for participating in this study today. My name is _____, and I work with the User
Experience Team at Fors Marsh Group. Today we will be evaluating a website that has
financial information. I will give you a series of tasks that I would like you to complete using
the website. Your comments and feedback will help the developers improve the site. I did
not create the website, so please do not feel like you have to hold back on your thoughts to
be polite to me. We’re interested in both your positive and negative reactions to the website.
Difficulties you may run into reflect the design of the website, not your skills or abilities.
Please keep in mind that I am not testing you or your knowledge. Rather you are helping us
to see how we can improve this site.
The entire session should last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• We are making a video recording of this session. Only those of us associated with this
project will see the recordings, and we will not share your name or personal
information. The recordings are used as a memory aid for me so I can go back later
to recall what happened during each session.
• I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help.So while you are working, I would like you to think aloud. I would like
you to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
o Tell me if you are looking for something and what it is and whether you can
find it or not.
o Tell me why you click on a link and what you expect to happen.
o Basically, just tell me everything you are thinking as you work.
• There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
• Any questions so far?
We are going to track where you are looking as you work.
• This is the eye tracker [point to tracker].
• Let’s get you situated in your chair in front of the screen in a good position.

PFC Moderators Guide  1

•

•

•

•
•
•
•

Please try to refrain from making any large head movements during the session. You
do not need to remain perfectly still like a statue, but if you move your head too
much, the eye tracker may lose your eyes.
Let’s do a quick task so the computer can find your eyes. Please follow the red circle
as it moves on the screen [calibration].
Ok, we are ready to begin. I will work from the other room, and we will communicate
through microphones and speakers. I will be able to see you with this camera [point
to camera].
Please complete this questionnaire as I go to the other room now and do a sound
check.
Do you have any questions?
Can you hear me ok?
Ok, we are ready to begin. I would like you to start with the first task in front of you
and begin by reading the task aloud. Then I will pull up the website, and I would like
you to use it to find the information. When you feel you have found it, I would like you
to tell me “This is my final answer.” Ready?

Notes:
• All tasks (except pre-task) begin on the home page. Participants will be redirected to
the home page at the conclusion of each task.
• After the participant reads the task, the moderator will start the eye tracker, and the
home page will load. The eye tracker will close the site at the end of each task.
• Participants will have the tasks on paper on the desk in front of them. Each task will
be on its own piece of paper, in large font.
• All participants will begin with the pre-task. After the pre-task, tasks will be randomly
assigned to prevent order effects. The moderator will place tasks in the correct order
before the session begins.
• Accuracy = percent of steps successfully completed for each task.
• Path = clicks made
• Task Time = page load to end of task (when they verbalize that they found the
answer)
• The first time participants use search, the moderator will probe about terms they
would enter and their expectations (before clicking search).
• After 3 tasks, if participants ONLY use search, prompt them with “without using
search.”
• Moderator: Note any inconsistencies between users’ terminology and the site’s.

PFC Moderators Guide  2

Pre-task interview: Process in Comparing Schools

Let’s say you interested in finding out more about repayment of your student loans that
you’re currently making payments on. What would you do? [If necessary] Who would you talk
to? How would you research this? [Moderator: What resources does participant mention?
What process? Websites? Search terms they use?]

Task 1: Navigation to Paying for College/first impressions
Start on main CFPB home screen.

1a. Let’s say you interested in finding out more about the different types of repayment
options you have on your current student loans. Where would you go?
Correct path: home page  Pay for college (or Get started)
1b. What are your first impressions of this website?
1c. Who is this site for?
1d. Is there any content on this site that you feel might be relevant to you?
Correct answer: Repay Student Debt
1e. What would you do next? [Moderator: Probe further – Tell me more about that. What
would get you to explore this site?]

Task 2: Debt Repayment
Start on Paying for College.

2a. Let’s say that you’re interested in finding out about repayment options. Go ahead and
click on “Repay Student Debt” and explore this tool using information about you and
your student loans. Please think aloud as you work and tell me when you’re done.
2b. Did you find anything useful to your situation when using this tool? [If yes], what
information did you find useful? [If no], what information would have been useful to you
in this scenario?
Start on Repay Student Debt
2c. Let’s say that you’re not confident that you can keep making your monthly payments on
time. You haven’t missed a payment yet, but you were wondering what options you
have to consolidate your federal student loans to lower your monthly payments. Where
would you go to find this information?
Correct path: Repay Student Debt  Federal  No (haven’t missed a payment)  No
(not confident in full payment)  Yes OR No (servicemember)
Correct answer: Income-based repayment (IBR) option.
2d. Let’s say you have a private student loan and you’re interested in repayment options.
Find out what a graduated repayment plan is. Once again, you haven’t missed a
PFC Moderators Guide  3

payment and you’re not confident that you can keep making your monthly payments on
time.
Correct path: Repay Student Debt  Non-Federal  No (haven’t missed a payment) 
No (not confident in full payment)  No (servicemember)
Correct answer: Graduated repayment has low payments in the early years that
increase over time.
2e. If you’ve missed a payment and are now in default and are unable to make payments on
a defaulted loan, what is the fastest way to get out of default?
Correct path: Repay Student Debt  Federal  Yes (have missed a payment)  Yes
(currently in default)  No (unable to make payments)
Correct answer: Direct consolidation loan

PFC Moderators Guide  4

Satisfaction Questionnaire (SATQ)

Please circle the numbers that most appropriately reflect your impressions about using this device.

1. Please rate your overall experience with the Paying for College website:
Not Difficult
At All
1

Slightly
Difficult
2

Moderately
Difficult
3

Very
Difficult
4

Extremely
Difficult
5

2. Please rate the clarity of the information presented to you on the Paying for College
website:
Not Clear
At All
1

Slightly
Clear
2

Moderately
Clear
3

Very
Clear
4

Extremely
Clear
5

3. Please rate your difficulty with navigating on the Paying for College website:
Not Difficult
At All
1

Slightly
Difficult
2

Moderately
Difficult
3

Very
Difficult
4

Extremely
Difficult
5

4. How likely would you be to use this site in the future?
Not Likely
At All
1

Slightly
Likely
2

Moderately
Likely
3

Very
Likely
4

Extremely
Likely
5

5. How likely would you be to recommend this site to a friend?
Not Likely
At All
1

Slightly
Likely
2

Moderately
Likely
3

Very
Likely
4

Extremely
Likely
5

6. Please rate the organization of the website in general:
Not Clear
At All

Slightly
Clear

Moderately
Clear

Very
Clear

Extremely
Clear

1

2

3

4

5

7. Please rate whether the Paying for College website met your expectations:
Did not meet
expectations
1

Met
expectations
2

Exceeded
expectations
3
PFC Moderators Guide  5

Debriefing Questionnaire
(NOTE: Some are used as follow-ups to actions during session and to SATQ
responses; participants do not receive all debriefing questions.)
1. How would you describe this site to a friend?

2. What can this organization help you with? What can’t they help you with? [Do users
understand the difference between this site and Dept of Ed?]
3. What information is most/least important to you (ask participant to rank the top 3
most/least)
4. Tell me a little bit about the way information is organized on the site. What do you
like? What don’t you like? (If necessary, probe about if information should be split by
repayment program, personal circumstance, or both?)
5. Tell me about your overall experience using this website. What worked well; what did
not work well? [Probe further as necessary.]
6. Let’s imagine you were to use this site on your own. What information would you look
for?
7. Would you share any of the information on social media? What would you share?

8. [If participant uses search] Tell me about your experience using Search. Did you get
what you were looking for?
9. [Follow up to SATQ4] What could be changed to increase the likelihood that you
would use this website?
10. [Follow up to SATQ4] As the website currently is, what is decreasing the likelihood
that you would use the Paying for College website?
11. What else were you expecting to do on this website that you couldn’t do? What is
missing?

PFC Moderators Guide  6

12. [Follow up regarding some of the SATQ items.]
13. What would you say is the best thing about this website?
14. What would you say is the worst thing about this website?
15. How do you think this experience should be different on mobile? If you accessed this
site from your phone or tablet, what should be different?
16. What information/links should be the same/different?

PFC Moderators Guide  7