Mortgage e-Closing Usability Study

Moderator’s Guide for Mortgage Closing Usability Test

Introduction

Thank you for participating in this study today. My name is _____, and I work with the User
Experience Team at Fors Marsh Group. Fors Marsh is a research company that conducts
itnerviews and focus groups on behalf of different organizations across the country. For this
specific study, we are wolking on behalf of the Consumer Financial Protection Bureau. Today
we will be evaluating a prototype that has financial information. The tool is still in
development, but we are testing some of the early functionalities so there may be some
missing elements or broken links. If you encounter any missing elements, please ignore
them and try to complete your task to the best of your ability.
I will give you a series of tasks that I would like you to complete using the tool. Your
comments and feedback will help the developers improve the tool. I did not create the it, so
please do not feel like you have to hold back on your thoughts to be polite to me. We’re
interested in both your positive and negative reactions to the tool. Difficulties you may run
into reflect its design, not your skills or abilities. Please keep in mind that I am not testing
you or your knowledge. Rather you are helping us to see how we can make improvements.
The entire session should last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• Your participation in this study is totally voluntary. You can choose not to answer any
questions you do not want to answer.
• The study has been approved by the US Government Office of Management and
Budget and has been assigned OMB Control Number 3170-0024) and information
will be handled in accordance with the Privacy Act Statement provided.
• We are making a video recording of this session, but your name and personal
information will not be associated with the recordings.
• Only the researchers working on this project will have access to the information you
provide in the study.
o CFPB may view the sessions and interview notes. CFPB receives final results
that are stripped of direct identifiers like name and email.
• I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help.So while you are working, I would like you to think aloud. I would like
you to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
Mortgage Closing Iterative Testing Mar 2014 Moderators Guide  1

•
•
•

o Tell me if you are looking for something and what it is and whether you can
find it or not.
o Tell me why you click on a link and what you expect to happen.
o Basically, just tell me everything you are thinking as you work.
There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
Any questions so far?
Ok, we are ready to begin. I would like you to start with the first task in front of you
and begin by reading the task aloud. Then I will pull up the prototype, and I would like
you to use it to find the information. When you feel you have found it, I would like you
to tell me “This is my final answer.” Ready?

Notes:
• Participants will have the tasks on paper on the desk in front of them. Each task will
be on its own piece of paper, in large font.
• Accuracy = percent of steps successfully completed for each task and/or first click
accuracy
• Path = clicks made
• Moderator: Note any inconsistencies between users’ terminology and the site’s.

Mortgage Closing Iterative Testing Mar 2014 Moderators Guide  2

Task 1: First Impressions
1a.
1b.

Where would you click first? Why?
What are your first impressions of this tool?

Task 2: Closing Documents
2a.

Imagine that you are going to close on a mortgage in two days and are trying to get
prepared. Where can you find a list of the documents that you are going to be signing
at the closing?
Accuracy: ________
Click Path:

2b.

From your review of these documents, which appear to be the most important to
focus on? [ANSWER: PROMISSARY NOTE, DEED, DISCLOSURE]
Accuracy:___________
Click Path:
Explanation noted:

2c.

[Once participant finds the list] What is the purpose of [INSERT DOCUMENT – ask
about all 3 documents]?
Accuracy: ________
Click Path:

Task 3: Comparing Information
3a.

You are closing on your mortgage tomorrow and want to review the mortgage amount
on your Good Faith Estimate (GFE) and compare it to the HUD-1. How would you do
this?
Accuracy: ________
Click Path:

3b.

Are there any differences between the two documents?
Accuracy: ________
Discrepancies noted:

3c. [If yes] Are the changes you’ve identified legal?
[If no, show the discrepancy and ask the question.]
Accuracy: ________

Task 4: Understanding Terminology
4.

You are currently closing on a mortgage, but you are confused about some of the
terminology on [INSERT FORM - PROMISSARY NOTE, DEED, DISCLOSURE]. Find out
what [INSERT SPECIFIC TERM] means?
Accuracy: ________
Click Path:
Explanation noted:
Mortgage Closing Iterative Testing Mar 2014 Moderators Guide  3

Task 5: Accessing Live Chat
5.

You have looked around this tool, but you still have more questions. You want
someone to answer your questions, how would you do that?
Accuracy:___________
Click Path:

Task 6: The Title Process
6.

Find more information about the title process. For example, when do you get your
name on the title of your new home?
Accuracy: ________
Click Path:

Task 7: Taxes: Escrow
7a.

When and how are taxes on your loan paid?
Accuracy: ________
Click Path:

Task 8: Understanding Escrow
8a.

What is escrow and how does it pertain to your loan?
Accuracy: ________
Click Path

Task 9: Assessing Changes in a Loan
9a.

You have a friend who recently closed on a home and the amounts at closing were
different than what she expected. You want to make sure that doesn’t happen to you.
Find out what can change from your Good Faith Estimate (GFE), or loan offer, and
what cannot.
Accuracy: ________
Click Path:

9b.

[If something can change] Explain why it is allowed to change.
Accuracy: ________
Click Path:

9c.

What should you do at closing if you find a discrepancy? In your own words, how
would you explain this to someone else?
Accuracy: ________
Click Path:

Mortgage Closing Iterative Testing Mar 2014 Moderators Guide  4

Satisfaction Questionnaire (SATQ)

Please circle the numbers that most appropriately reflect your impressions about using this tool.

1. Please rate your overall experience using the tool:
Did not like it
Did not like it
Neutral
Liked it
at all
1
2
3
4
2. How helpful was this tool?
Not Helpful
Slightly
At All
Helpful
1
2

Moderately
Helpful
3

3. How trustworthy was this closing tool?
Not
Slightly
Moderately
Trustworthy
Trustworthy
Trustworthy
At All
1
2
3

Liked it a lot
5

Very
Helpful
4

Extremely
Helpful
5

Very
Trustworthy

Extremely
Trustworthy

4

5

4. How likely would you be to use this tool if you close on a house again/in the
future?
Not Likely
Slightly
Moderately
Very
Extremely
At All
Likely
Likely
Likely
Likely
1
2
3
4
5
5.

How likely would you be to recommend this tool to a friend?
Not Likely
Slightly
Moderately
Very
At All
Likely
Likely
Likely
1
2
3
4

6.

Please rate the organization of the tool in general:
Not Clear
Slightly
Moderately
Very
At All
Clear
Clear
Clear
1

7.

3

4

Was the ability to compare the GFE to HUD-1 helpful?
Not Helpful
Slightly
Moderately
Very
At All
Helpful
Helpful
Helpful
1

8.

2

2

3

Was the tool easy or difficult to navigate?
Very
Somewhat
Neither Easy
Easy
Easy
nor Difficult
1
2
3

Extremely
Likely
5
Extremely
Clear
5
Extremely
Helpful

4

5

Somewhat
Difficult
4

Extremely
Difficult
5

Mortgage Closing Iterative Testing Mar 2014 Moderators Guide  5

Debriefing Questionnaire
(NOTE: Some are used as follow-ups to actions during session and to SATQ
responses; participants do not receive all debriefing questions.)
1. “You mentioned that you would [insert what they said in task 5]. Are there any other
ways you would seek assistance?
2. What do you think about live chat features? What has been your previous experience
with live chats/phone calls with a person? Would you want to see that kind of
function? Has this kind of function been helpful in the past?
3. How far in advance before you closing would you like to get this information?
4. What would you expect to see in this [INSERT SECTION] area of the tool?
5. How could the layout of the tool be improved?
6. How could the tool’s navigation be improved?
7. Followup to SAT Q2: What parts were helpful on the tool?
8. Followup to SAT Q7: Why was/wasn’t the ability to compare information helpful?
9. How would you describe this tool in your own words?
10. What kind of information about escrows would you expect to find on this tool?
11. What was your level of trust for the different pages you viewed? Why?
12. Do you think you would use this tool in the future when you have a question about
closing a home loan? Why or why not?
13. If you had a choice between electronic closing and traditional closing, which would
you chose?
14. What else were you expecting to do on this tool that you could not do?
15. What would you say is the best thing about this tool?
16. What would you say is the worst thing about this tool?

Mortgage Closing Iterative Testing Mar 2014 Moderators Guide  6