CFPB Paying for College usability study (Round 2)

Moderator’s Guide for Paying for College Usability Test
Group A: Students With Offers in Hand

Introduction

Thank you for participating in this study today. My name is _____, and I work with the User
Experience Team at Fors Marsh Group. Today we will be evaluating a website that has
financial information. I will give you a series of tasks that I would like you to complete using
the website. Your comments and feedback will help the developers improve the site. I did
not create the website, so please do not feel like you have to hold back on your thoughts to
be polite to me. We’re interested in both your positive and negative reactions to the website.
Difficulties you may run into reflect the design of the website, not your skills or abilities.
Please keep in mind that I am not testing you or your knowledge. Rather you are helping us
to see how we can improve this site.
The entire session should last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• We are making a video recording of this session. Only those of us associated with this
project will see the recordings, and we will not share your name or personal
information. The recordings are used as a memory aid for me so I can go back later
to recall what happened during each session.
• I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help.So while you are working, I would like you to think aloud. I would like
you to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
o Tell me if you are looking for something and what it is and whether you can
find it or not.
o Tell me why you click on a link and what you expect to happen.
o Basically, just tell me everything you are thinking as you work.
• There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
• Any questions so far?
We are going to track where you are looking as you work.
• This is the eye tracker [point to tracker].
• Let’s get you situated in your chair in front of the screen in a good position.

PFC Moderators Guide  1

•

•

•

•
•
•
•

Please try to refrain from making any large head movements during the session. You
do not need to remain perfectly still like a statue, but if you move your head too
much, the eye tracker may lose your eyes.
Let’s do a quick task so the computer can find your eyes. Please follow the red circle
as it moves on the screen [calibration].
Ok, we are ready to begin. I will work from the other room, and we will communicate
through microphones and speakers. I will be able to see you with this camera [point
to camera].
Please complete this questionnaire as I go to the other room now and do a sound
check.
Do you have any questions?
Can you hear me ok?
Ok, we are ready to begin. I would like you to start with the first task in front of you
and begin by reading the task aloud. Then I will pull up the website, and I would like
you to use it to find the information. When you feel you have found it, I would like you
to tell me “This is my final answer.” Ready?

Notes:
• All tasks (except pre-task) begin on the home page. Participants will be redirected to
the home page at the conclusion of each task.
• After the participant reads the task, the moderator will start the eye tracker, and the
home page will load. The eye tracker will close the site at the end of each task.
• Participants will have the tasks on paper on the desk in front of them. Each task will
be on its own piece of paper, in large font.
• All participants will begin with the pre-task. After the pre-task, tasks will be randomly
assigned to prevent order effects. The moderator will place tasks in the correct order
before the session begins.
• Accuracy = percent of steps successfully completed for each task.
• Path = clicks made
• Task Time = page load to end of task (when they verbalize that they found the
answer)
• The first time participants use search, the moderator will probe about terms they
would enter and their expectations (before clicking search).
• After 3 tasks, if participants ONLY use search, prompt them with “without using
search.”
• Moderator: Note any inconsistencies between users’ terminology and the site’s.

PFC Moderators Guide  2

Pre-task interview: Process in Comparing Schools

Let’s say you interested in finding out more about the different financial decisions that you
need to make about education after graduating high school. What would you do? [If
necessary] Who would you talk to? How would you research this? [Moderator: What
resources does participant mention? What process? Websites? Search terms they use?]

Task 1: Navigation to pay for college/first impressions (Difficulty: Easy)
Start on main CFPB home screen.

1a. Let’s say you interested in finding out more about the different financial decisions that
you need to make about education after high school. What should happen before you
compare schools? (e.g., Apply, Find Financial Aid, FAFSA)
Correct path: home page  Pay for college (or Get started)

1b. What are your first impressions of this website?

1c. Who is this site for?

1d. What would you do next? [Moderator: Probe further – Tell me more about that. What
would get you to explore this site?]

PFC Moderators Guide  3

Task 2: Compare colleges (Difficulty: Medium)
Start on Paying for College.
2a. Let’s say that you have your list narrowed down to the two schools that you have
financial aid offers from. You are interested in graduation rates of those two schools.
Which university graduates a higher percentage of students who enroll? [Moderator
note: $$ does not auto-populate. What do they say/do?]
Correct path:
1. Compare Financial Aid Offers  Get Started  Search School X  Click School X 
Check ‘I have a financial aid offer from this school”  Continue (This school pledged
to use the Financial Aid Shopping Sheet.)  Continue  Add another school 
Search School Y  Click School Y  Check ‘I have a financial aid offer from this
school”  Continue (This school pledged to use the Financial Aid Shopping Sheet.)
Scroll to information on page
2b. Let’s say you want to keep a copy of this information so you can show it to a family
member. Get a copy of this information to use later.
Correct path:
1. Save & Share  enter email address  Send
2. Save & Share  copy URL (will save #s entered)  email to self
3. Ctrl-P
2c. Did you expect to be able to save information on this site?
What did you expect that to be like?

Task 3: Compare loans (Difficulty: Medium)
Start on Paying for College.
3a. Let’s say you have decided on the university you want to attend and you know you’re
going to need to take out a loan to pay for tuition. What are the different types of loans
that are available to you? [Moderator: Which section of the page does the participant
use to find this information?]
Correct Answer: Federal or Private
Correct path:
1. Student Financial Guides  Scroll to compare federal and private
2. More about student loans (under Student Financial Guides)  Scroll

PFC Moderators Guide  4

3b. Which is better for you? [Do they understand? Does the content help them?]

3c. Ok, you now know that you need to take out a federal loan. What are the approximate
interest rates for federal loans?
Correct Answer: Interest rates range from 3.86% to 6.41%, depending on the loan
program.
Correct path
1. Scroll down  Detailed comparison of federal and private loans  Look at “Interest
Rates” section

3d. You think your friends would be interested in this information too. How would you share
this with them? [What do they do? Share on social media? Bookmark? Print? If print, what
would that look like? How do they want it to look? What would they expect?]
3e. After you graduate, what would be the debt burden for [THE SECOND SCHOOL THEY
ENTER]?
3f. After you graduate, what is the monthly payment for the loans you would take out while
attending [THE FIRST SCHOOL THEY ENTER]?
3g. How much debt burden would you have at [SECOND SCHOOL] vs. [FIRST SCHOOL]?
3h. Now enter some basic information about a school that you don’t have an offer for.
Is this helpful?

PFC Moderators Guide  5

Task 4: Financial Aid Disbursement (Difficulty: Easy)
Start on Paying for College.

4a. Let’s say you have found out that you have money left over from your loan after tuition
and fees were paid. You have heard that you have a few options for how you can receive
this money. What are your options?
Correct Answer: Direct deposit to a bank account, to a card that might also double as
your student ID, check, or cash.
Correct path:
1. Student financial guides  Student banking  Scroll to ”Financial Aid
Disbursement”  View Aid Disbursement Options

PFC Moderators Guide  6

Satisfaction Questionnaire (SATQ)

Please circle the numbers that most appropriately reflect your impressions about using this device.

1. Please rate your overall experience with the Paying for College website:
Not Difficult
At All
1

Slightly
Difficult
2

Moderately
Difficult
3

Very
Difficult
4

Extremely
Difficult
5

2. Please rate the clarity of the information presented to you on the Paying for College
website:
Not Clear
At All
1

Slightly
Clear
2

Moderately
Clear
3

Very
Clear
4

Extremely
Clear
5

3. Please rate your difficulty with navigating on the Paying for College website:
Not Difficult
At All
1

Slightly
Difficult
2

Moderately
Difficult
3

Very
Difficult
4

Extremely
Difficult
5

4. How likely would you be to use this site in the future?
Not Likely
At All
1

Slightly
Likely
2

Moderately
Likely
3

Very
Likely
4

Extremely
Likely
5

5. How likely would you be to recommend this site to a friend?
Not Likely
At All
1

Slightly
Likely
2

Moderately
Likely
3

Very
Likely
4

Extremely
Likely
5

6. Please rate the organization of the website in general:
Not Clear
At All

Slightly
Clear

Moderately
Clear

Very
Clear

Extremely
Clear

1

2

3

4

5

7. Please rate whether the Paying for College website met your expectations:
Did not meet
expectations
1

Met
expectations
2

Exceeded
expectations
3
PFC Moderators Guide  7

8. How hard was it to compare colleges using the comparison tool?
Not Difficult
Slightly
Moderately
Very
Extremely
At All
Difficult
Difficult
Difficult
Difficult
1
2
3
4
5

PFC Moderators Guide  8

Debriefing Questionnaire
(NOTE: Some are used as follow-ups to actions during session and to SATQ
responses; participants do not receive all debriefing questions.)
1. How would you describe this site to a friend?

2. What can this organization help you with? What can’t they help you with? [Do users
understand the difference between this site and Dept of Ed?]

3. Tell me about your overall experience using this website. What worked well; what did
not work well? [Probe further as necessary.]

4. Let’s imagine you were to use this site on your own. What information would you look
for?

5. Would you share any of the information on social media? What would you share?

6. [If participant uses search] Tell me about your experience using Search. Did you get
what you were looking for?

7. [Follow up to SATQ4] What could be changed to increase the likelihood that you
would use this website?

8. [Follow up to SATQ4] As the website currently is, what is decreasing the likelihood
that you would use the Paying for College website?

PFC Moderators Guide  9

9. What do you think the “Download action guide” is? [Probe further as necessary.]

10. What do you think about the process of comparing colleges? [Probe further as
necessary. Did they like it? What will happen next?]
a. Does this seem like something you might use on your own?
b. What did you think when you saw the zeros appear?
c. What else did you expect that you couldn’t do?
d. What was the best part?
e. What was the worst part?
f. Is this something you would use in the future?
g. What did you think abou the different sections like contributions, federal
loans, private loans, etc?
11. What categories in the compare tool were hard to understand?

12. What do you think about the school indicators at the bottom?
a. Do you like them?
b. Why did/didn’t you pay attention to them?

13. What do you think about this interest rate toggle?

14. Did the compare tool progresses in a logical order?

15. What categories were the hardest to work through?

16. Where would you go on this site to find out more information about these various
loan options?

17. What else were you expecting to do on this website that you couldn’t do? What is
missing?

PFC Moderators Guide  10

18. [For counselors] What is the one thing you would share with a student who was not
concerned with debt associated with college?
a. What about with a student who is very concerned with debt?

19. What did you think about this process timeline? [If necessary: Is in in the right order?
Anything missing?

20. What did you think about the questions on the right [show the items in green boxes]?

21. [Follow up regarding some of the SATQ items.]

22. What would you say is the best thing about this website?

23. What would you say is the worst thing about this website?

24. How do you think this experience should be different on mobile? If you accessed this
site from your phone or tablet, what should be different?

25. What information/links should be the same/different?

26. Do you think you would use the comparison tool on mobile?

PFC Moderators Guide  11