Paying for College - Dynamic Disclosures Usability Test (Round 1)

MODERATOR’S GUIDE
Dynamic Disclosures
Usability Test (Round 1)
Data Collection: [insert dates]

Last revision: 12/07/2015

Section 1: Introduction
Thank you for participating in this study today. My name is _____, and I am a researcher with
the Fors Marsh Group User Experience Team. Today we will be evaluating a website that has
information about receiving and understanding college financial aid offers. Some of the
pages you will be interacting with are still in development, so not all the links will be
clickable and others may have missing elements. If you encounter something isn’t working,
please let me know so I can get us back on track.
I will ask you questions and give you some tasks to complete using the website. Your
comments and feedback will help the design team improve the site. I am a researcher and
did not create the site, so please do not feel like you have to hold back on your thoughts to
be polite to me. I’m interested in both your positive and negative reactions. Difficulties you
may run into reflect issues with the design of the sites, not your skills or abilities. Please
keep in mind that I am not testing you or your knowledge. Rather, you are helping us to see
how we can improve this site.
The entire session will last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• We are making a video recording of the computer screen during this session, but your
name and personal information will not be associated with the recordings.
• Please take a moment to review the Privacy Act Statement (Hand Privacy Act
Statement to participant.)
Privacy Act Statement
5 U.S.C. 552a(e)(3)
The information you provide through your responses to Fors Marsh will assist the study
sponsor, the Consumer Financial Protection Bureau (“CFPB”), in evaluating a prototype
website that has financial information.
The CFPB will not obtain or access any directly identifying information from Fors Marsh
about study participants. The agency will only obtain and access de-identified results and
aggregated analyses of those results.
Information collected on behalf of the Bureau by Fors Marsh will be treated in
accordance with the System of Records Notice (“SORN”), CFPB.021 – CFPB Consumer
Education and Engagement Records, 79 F.R. 78839. Direct identifying information will
only be used by Fors Marsh to facilitate the study and will be kept private except as
required by law.
This collection of information is authorized by Pub. L. No. 111-203, Title X, Sections
1013 and 1022, codified at 12 U.S.C. §§ 5493 and 5512.
Dynamic Disclosures Usability Test (Round 1), January 2016  1

Last revision: 12/07/2015
Participation in this study is voluntary, you are not required to participate or share any
identifying information with Fors Marsh, including name and email address, and you may
withdraw participation at any time. However, if you do not include the requested
information, you may not participate in the study.
•

•
•
•

I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help. While you are working, I would like you to think aloud. I would like you
to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
o Tell me why you clicked on a link and what you expect to happen next.
o Basically, just tell me everything you are thinking as you work.
There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
Any questions so far?
Ok, we are ready to begin. I would like you to start by asking you some background
questions.

Dynamic Disclosures Usability Test (Round 1), January 2016  2

Last revision: 12/07/2015

Section 2: Questions and Tasks
Item 1: Pre-task interview

Goal: To get to know participants and explore their relevant past behavior.
1a. What types of electronic devices do you use to visit websites?
1b. What devices do you usually use to track or manage your financial accounts online?
1c. What is your main online resource when you are looking for financial information?
1d. What other resources might you use to find similar information?
1e. Have you heard of a website called consumerfinance.gov before?
o [IF YES]
o How or when did you hear about it?
o Have you used consumerfinance.gov before?
 [IF YES] How often on average do you use it?
 [IF YES] For what purposes do you use it?
o [IF NO]
o What do you think the website consumerfinance.gov would be useful for?
o What kind of information do you think you could find on
consumerfinance.gov?

Item 2: General thoughts around financial aid and student aid offer
acceptance

Goal: To explore participants’ beliefs and impressions around financial aid and receiving
student aid offers from schools.
2a. We'd like to talk to you specifically today about financial aid and student loans when it
comes to paying for college. What comes to mind when you think about financial aid in
regards to going to college?
2b. When you think about your financial future, what does going to college mean to you?
How does going to college impact your future financial success?
2c. When you think about your financial future, what does accepting financial aid or taking
out student loans mean to you? How do financial aid and student loans impact your future
financial success?
2d. Please tell us about the process of applying and accepting your financial aid package.
What was presented to you as part of the process, and what was explained to you?
2e. How would you describe the communication from your school about financial aid and
student loans? How understandable was it to you?
2f. If you had questions about your financial aid package, who could you talk to and where
did you look for answers?
2g. Did you visit any websites to help you understand or make decisions about financial aid?
If so, which websites?
[If a “for-profit student”, proceed to Item 3. Else, skip to Item 4.]

Dynamic Disclosures Usability Test (Round 1), January 2016  3

Last revision: 12/07/2015

Item 3: For-profit students and the “Understanding financial aid offers”
page

Audience: Students who fall into the “For-profit student” recruitment category
Starting Page: Paying for College “Understanding financial aid offers” web page
Goal: To evaluate for-profit students’ first impressions.
3a. What is the purpose of this web page?
3b. What is the first thing that caught your eye?
3c. What would be the first thing you would interact with or select on this site? Why?

Starting Page: Paying for College “Understanding financial aid offers” web page
Goal: To evaluate the overall usability, learnability, and usefulness of the Dynamic
Disclosures tool for for-profit students.
3d. Please use and interact with this web page until you consider your experience on the
page complete.
[Correct Path (with some ability to reverse direction and modify choices): Verify general
information as correct
 Review and modify financial aid offer as needed
 Review the evaluation information (modifying expense data as needed)
 Give opinion of the given offer for the student’s financial future
 Review options
 Read next steps]
3e. Does this page match your expectations for a website about receiving and
understanding college financial aid offers? Why or why not?
Goal: To evaluate the usability, learnability, and usefulness of the financial aid offer review
section for for-profit students.
[Scroll to “Step 1. Review your first year offer”]
3f. Would you walk me through your thought process when you were reviewing your financial
aid offer on this page?
3g. What information was presented to you as part of the financial aid offer?
3h. Did it match your expectations for what would be included in a financial aid offer? Why?
3i. Was the information easy to understand? How could the writers improve this section of
the site?
3j. Was it easy to enter the data? How could the designers improve this section of the site?
3k. Were there any concerns about your privacy or the security of the data entered?
3l. Was there anything missing that you expected to find?
Goal: To evaluate the usability, learnability, and usefulness of the offer evaluation section
for for-profit students.
[Scroll to “Step 2. Evaluating your offer”]
3m. Would you walk me through your thought process when you were reviewing the offer
evaluation on this page?
3n. What information was presented to you as part of the financial aid offer evaluation?
3o. Did it match your expectations for what would be included in an evaluation of a financial
aid offer? Why?
Dynamic Disclosures Usability Test (Round 1), January 2016  4

Last revision: 12/07/2015
3p. Hpw easy were the text and explanations to understand? How could the writers improve
this section of the site?
3q. Was it easy to understand the graphs or data provided? How could the designers
improve this section of the site?
3r. Were there any concerns about your privacy or the security of the data presented?
3s. Was there anything missing that you expected to find?
Goal: To evaluate the usability, learnability, and usefulness of the options and next steps
section for for-profit students.
[Scroll to “Step 3. You have options”]
3t. Would you walk me through your thought process when you were reading and reviewing
the last sections on this page?
3u. What information was presented as your options for your financial aid offer?
3v. Did the information given match your expectations for what the next steps would be?
Why?
3w. How easy were the text and explanations to understand? How could the writers improve
this section of the site?
3x. Was it easy to understand the visuals provided? How could the designers improve this
section of the site?
3y. Was there anything missing that you expected to find?

Item 4: General students and the “Understanding financial aid offers”
page
Audience: Students who fall into the “Not a for-profit student” recruitment category
Starting Page: Paying for College “Understanding financial aid offers” web page
Goal: To evaluate students’ first impressions.
4a. What is the purpose of this web page?
4b. What is the first thing that caught your eye?
4c. What would be the first thing you would interact with or select on this site? Why?

Starting Page: Paying for College “Understanding financial aid offers” web page
Goal: To evaluate the overall usability, learnability, and usefulness of the Dynamic
Disclosures tool for students.
4d. Please use and interact with this web page until you consider your experience on the
page complete.
[Correct Path (with some ability to reverse direction and modify choices): Select and
enter information about college
 Enter information for financial aid offer
 Review the evaluation information (modifying expense data as needed)
 Give opinion of the given offer for the student’s financial future
 Review options
 Read next steps]
4e. Does this page match your expectations for a website about receiving and
understanding college financial aid offers? Why or why not?
Dynamic Disclosures Usability Test (Round 1), January 2016  5

Last revision: 12/07/2015
Goal: To evaluate the usability, learnability, and usefulness of the college selection and
financial aid offer review section for students.
[Scroll to “Step 1. Review your first year offer”]
4f. Would you walk me through your thought process when you were selecting the
information about your college and entering your financial aid offer on this page?
4g. What information was presented to you as part of the financial aid offer?
4h. Did it match your expectations for what would be included in a financial aid offer? Why?
4i. Was the information easy to understand? How could the writers improve this section of
the site?
4j. Was it easy to enter the data? How could the designers improve this section of the site?
4k. Were there any concerns about your privacy or the security of the data entered?
4l. Was there anything missing that you expected to find?
Goal: To evaluate the usability, learnability, and usefulness of the offer evaluation section
for students.
[Scroll to “Step 2. Evaluating your offer”]
4m. Would you walk me through your thought process when you were reviewing the offer
evaluation on this page?
4n. What information was presented to you as part of the financial aid offer evaluation?
4o. Did it match your expectations for what would be included in an evaluation of a financial
aid offer? Why?
4p. Were the text and explanations easy to understand? How could the writers improve this
section of the site?
4q. Was it easy to understand the graphs or data provided? How could the designers
improve this section of the site?
4r. Were there any concerns about your privacy or the security of the data presented?
4s. Was there anything missing that you expected to find?
Goal: To evaluate the usability, learnability, and usefulness of the options and next steps
section for students.
[Scroll to “Step 3. You have options”]
4t. Would you walk me through your thought process when you were reading and reviewing
the last sections on this page?
4u. What information was presented as your options for your financial aid offer?
4v. Did the information given match your expectations for what the next steps would be?
Why?
4w. Were the text and explanations easy to understand? How could the writers improve this
section of the site?
4x. Was it easy to understand the visuals provided? How could the designers improve this
section of the site?
4y. Was there anything missing that you expected to find?

Item 5: Collection of usability and marketing metrics

Goal: To numerically evaluate the usability, learnability, and of the site

System Usability Scale (SUS)
Dynamic Disclosures Usability Test (Round 1), January 2016  6

Last revision: 12/07/2015
Please circle the numbers that most appropriately reflect your impressions about using this website.

1. I think that I would like to use the “Understanding financial aid offers” web page frequently.

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

2. I found the “Understanding financial aid offers” web page unnecessarily complex:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

3. I thought the “Understanding financial aid offers” web page was easy to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

4. I think that I would need the support of a technical person to be able to use the “Understanding
financial aid offers” web page:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

5. I found the various functions in the “Understanding financial aid offers” web page were well
integrated with each other:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

6. I thought there was too much inconsistency in the “Understanding financial aid offers” web page:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

7. I would imagine that most people would learn to use the “Understanding financial aid offers” web
page very quickly:
Dynamic Disclosures Usability Test (Round 1), January 2016  7

Last revision: 12/07/2015
Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

8. I found the “Understanding financial aid offers” web page very cumbersome to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

9. I felt very confident using the “Understanding financial aid offers” web page:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

10. I needed to learn a lot of things before I could get going with the “Understanding financial aid
offers” web page:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

Net Promoter Score (NPS)

Please circle the numbers that most appropriately reflect your impressions about using this web
page.
On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this website to
a friend, colleague, or family member as a place to consult about understanding their financial aid offers?
Why?

1

2

3

4

5

6

7

8

9

10

Debriefing Questions

Dynamic Disclosures Usability Test (Round 1), January 2016  8

Last revision: 12/07/2015
(NOTE: Some are used as follow-ups to actions during session and to SUS or NPS responses;
participants do not receive all debriefing questions.)
1. What stands out to you about your experience using the site? Tell me about your overall
experience using this website. What worked well; what did not work well? [Probe further
as necessary]
2. How would you describe this site to a friend?
3. Did you learn any new information on the website that you were unaware of before, or
that was a surprise to you?
4. Earlier, we asked you how financial aid and student loans impact your future financial
success. After going through this website, is your answer still the same? Why or why not?
5. Was there anything missing from the website shown to you that you expected to see, or
expected to have access to, that you haven’t called out already?
Extra questions
6. Where would you go or what would you do after using this site?
7. What can this organization help you with? What can’t they help you with?
8. [Follow up to NPS] What could be changed to increase the likelihood that you would
recommend this site to a friend?
9. [Follow up to SUS1] What could be improved on the site that would increase the
likelihood that you would use it in the future?
10. What did you like most about the site?
11. What did you dislike the most about the site?
12. If you could change one thing on the site, what would it be?
13. Do you have any other feedback/input that we did not cover?

Dynamic Disclosures Usability Test (Round 1), January 2016  9