Owning a Home Project User Experience Testing – Round 1

MODERATOR’S GUIDE

Owning a Home
Monthly Payment Worksheet
UX Test (OaH3) - DRAFT
Data Collection: [date], 2015

DRAFT

Introduction (5 minutes)

Thank you for participating in this study today. My name is _____, and I am a researcher with
the Fors Marsh Group User Experience Team. Today we will be evaluating a worksheet.
I will ask you questions and give you some tasks to complete while using this worksheet.
Your comments and feedback will help the team improve the worksheet. I did not create this
worksheet, so please do not feel like you have to hold back on your thoughts to be polite to
me. I’m interested in both your positive and negative reactions. Difficulties you may run into
reflect issues with the design of the worksheet, not your skills or abilities. Please keep in
mind that I am not testing you or your knowledge. Rather, you are helping us to see how we
can improve this tool.
The entire session will last about 60 minutes. In accordance with the Papework Reduction
Act of 1995, this study as been approved by the Federal government’s Office of
Management and Budget (OMB) under OMB control number 3170-0024. Without this
approval we could not conduct this study. Do you have any questions so far?
Let’s cover a few things before we get started.
You might have noticed the cameras. I am recording the session. The recording is used as a
memory aid for me so I can recall what happened during the session. Please speak openly
about your opinions and experiences. We want to learn from you, so it is important that you
share your honest opinions.
I am interested in your thoughts and reactions as we proceed. This is important because I
can see what you are doing, but I won’t know why you are doing it. So I need your help. While
you are working, I would like you to think aloud. I would like you to:
•
•
•
•
•

Tell me what you are thinking
Describe the steps that you are taking.
Tell me why you are doing what you are doing, what you are going to do, and why.
Tell me why you clicked on a link and what you expect to happen next.
Basically, just tell me everything you are thinking as you work.

There are no right or wrong answers, and your comments and opinions will only be used in
combination with the feedback that we get from other people.
What we talk about here is private. That means your name will not be associated with
anything you say in our reports and your responses will not be linked to your identity in any
way.
Likewise, we want to respect everyone’s privacy in this room and not share any of our
discussion from today with others who are not actively working on this project.
Please also make sure to review the Privacy Act Statement provided to you in hard copy.
Owning a Home Monthly Payment Worksheet – Moderator Guide  1

DRAFT

[Provided in hard copy.]
5 U.S.C. 552(a)(e)(3)
The information you provide through your responses to Fors Marsh will assist the study
sponsor, the Consumer Financial Protection Bureau (“CFPB”), in participating in usability
testing that evaluates consumers’ experiences with a financial website.
The CFPB will obtain and access identifying information from Fors Marsh about study
participants by observing recordings made during this usability testing.
Information collected on behalf of the Bureau by Fors Marsh will be treated in accordance
with the System of Records Notice (“SORN”), CFPB.022,
https://www.federalregister.gov/articles/2012/11/14/2012-27582/privacy-act-of-1974as-amended. This information will not be disclosed. The Routine Uses for the SORN
specifies how we may use this information. Direct identifying information will only be used by
Fors Marsh to facilitate the usability testing and will be kept private except as required by
law.
This collection of information is authorized by Pub. L. No. 111-203, Title X, Sections 1013
and 1022, codified at 12 U.S.C. §§ 5493 and 5512.
Participation in this study is voluntary, you are not required to participate or share any
personally identifying information with Fors Marsh during the usability testing, including
name, age, email address, mobile number, alternate number, race, ethnic origin, gender,
city and state of residence, education level, household income, current employment status,
and occupation. You may withdraw from participation at any time.
•
Any questions so far?
•
Ok, we are ready to begin. I would like you to start by asking you some background
questions.

Owning a Home Monthly Payment Worksheet – Moderator Guide  2

DRAFT

Section 1: Interview about steps taken in the homebuying process (3
minutes)

Section 1 Goal: Understand the stage each participant in the homebuying process.
You had indicated that you are [in the process of buying a home/preparing to buy a home].
I’m going to ask you a few general questions about your experience with this process.
1.1. Would you tell me about your experience so far?
1.2. What are the major challenges you have been facing as you’ve been shopping? What
have you been struggling with?
1.3. What resources have you consulted (friends or tools/resources)? What tools or
resources do you wish you had access to but didn’t?

Section 2: First Impressions (5 minutes)

Goal: Assess if users are able to find the link to the tool from the landing page
2.1. First, let’s say you’re not sure how much you can afford to pay for your new home.
Please find a tool to help determine the home you are able to afford. Please think aloud as
you do so.
2.2. What was your experience like finding the tool?
Goal: Assess if users are able to understand that it targets first-time homebuyers
2.3. What do you think is the purpose of this worksheet? (What do you think this worksheet
is for?)
Goal: Assess first impressions of the tool
2.4. What are your first impressions of this worksheet?
Goal: Assess if users are comfortable with the structure/layout of the worksheet
2.5. Is there anything confusing or unclear on first glance?

Section 3: Using the worksheet (10 minutes)

Goal: Assess whether users understand how to use the tool
3.1. Please take a few minutes to explore the web tool and customize it to your own
situation. Put in whatever information is appropriate for you. Please talk out loud as you’re
going through the tool, and walk through eash step you are taking.
Goal: Determine what users would do next after using the monthly payment web tool
3.2. What would you do next?

Owning a Home Monthly Payment Worksheet – Moderator Guide  3

DRAFT

Section 4: Questionnaires (3 minutes)
System Usability Scale

Please circle the numbers that most appropriately reflect your impressions about using this website.
1. I think that I would like to use the Monthly Payment Worksheet frequently:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

2. I found the Monthly Payment Worksheet unnecessarily complex:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

3. I thought the Monthly Payment Workhseet was easy to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

4. I think that I would need the support of a technical person to be able to use the Monthly Payment
Worksheet:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

5. I found the various functions in the Monthly Paymnet Worksheet were well integrated with each
other:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

6. I thought there was too much inconsistency in the Monthly Payment Worksheet:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

Owning a Home Monthly Payment Worksheet – Moderator Guide  4

DRAFT
7. I would imagine that most people would learn to use the Monthly Payment Worksheet very
quickly:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

8. I found the Monthly Payment Worksheet very cumbersome to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

9. I felt very confident using the Monthly Payment Worksheet website:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

10. I needed to learn a lot of things before I could get going with the Monthly Payment Worksheet:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

Net Promoter Score (NPS)

Please circle the numbers that most appropriately reflect your impressions about using this website.
On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this worksheet
to a friend, colleague, or family member?

1

2

3

4

5

6

7

8

9

10

Owning a Home Monthly Payment Worksheet – Moderator Guide  5

DRAFT

Section 5: Cognitive Walkthrough (15 minutes)
Goal: Assess if users are able to understand the structure/flow of the tool
5.1. Could you please walk me through your thought process as you were entering your
information in this worksheet?
Goal: Assess if users are able to understand how the tool functions
5.2. Could you explain at a high-level how the tool operates?
5.3. Was it clear why you needed to enter the information in each of the fields?
5.4. Were there any fields where you were uncertain why you needed to provide that
information?
Goal: Assess whether users understand how to complete the Monthly Income section
5.5. Were you able to understand how to fill out the “Monthly Income” section?
[If necessary] How difficult was it to figure out your pre-tax and post-tax income? Is there
anything specific that makes this section hard for you to fill out?
Goal: Assess whether users feel that there are enough budget categories or would want
more
5.6. Did you feel like there were any fields in the “Monthly Spending” section that you would
add? Are there any fields that you would change?
Goal: Assess whether any other fields/instructions were confusing
5.7. Looking at this middle section of the worksheet [steps 3-6], were there any fields or
instructions that were confusing? Did you feel confident filling out this part?
Goal: Assess whether it is clear how the inputs affect the ouputs
5.8. How do the numbers you inputted affect the results on the worksheet? [If necessary]
How did the tool arrive at the final results on the page?
5.9. How difficult did you find it to understand the final results?
5.10. What do these numbers mean to you?
5.11. Are there any results that are confusing?
Goal: Assess users’ understanding of the tool’s purpose to guide them in figuring out what
they should spend
5.11. What do you think the purpose of this final page is? What would it specifically help you
to figure out? [if needed] Do you think it’s related to how much you would qualify for? Or just
a guide for your spending?
5.12. Can you tell me what these numbers are based on, in your own words?
Owning a Home Monthly Payment Worksheet – Moderator Guide  6

DRAFT
5.13. How would you use the information presented in this section?

Section 6: General Debrief (5 minutes)

Goal: Determine the most memorable aspect of the tool.
6.1. What stands out to you about your experience using this tool? What worked well? What
didn’t work well?
Goal: Determine whether there were any elements/aspects that were frustrating
6.2. Did you find anything in this tool frustrating?
Goal: Assess if there are any unclear terms
6.3. Were there any terms used that you had questions about or that were hard to
understand at first?
Goal: Assess whether users understand the helper prompts and are able to use them
6.4. Did the text under each field [moderator points to an example of the text on the page]
help you to better understand what you needed to enter in each field?
Goal: Assess if the categories are comprehensive, or if there are any more that would be
helpful
6.5. What do you think about the completeness of the information?

Section 7: Design and Imagery Debrief (3 minutes)
Goal: Assess users’ reaction to the design of the form
Now I’ll ask you to think again about the tool overall.
7.1. What were your thoughts about the design of the page?

7.2. Was it easy or difficult to understand the layout of each of the sections?
7.3. Do you have any other thoughts about the design of the tool?

Section 8: Behavior Change (10 minutes)

Goal: Assess whether users would use this tool repeatedly; assess when they use it and
under what circumstances
8.1. Do you think you would use this tool during your mortgage process? Why or why not?
When do you think you would do so?
8.2. [if more than once] How many times do you think you would refer to it: repeatedly, or
once in a while?
Goal: Assess whether users’ approach to buying a home and finding a mortgage reportedly
changes after using this tool
8.3. Do you feel like your approach to buying a home and finding a mortgage would change
after visiting this site and using these tools? How so?
Goal: Assess what next steps users would take
Owning a Home Monthly Payment Worksheet – Moderator Guide  7

DRAFT
8.4. After using the tool, what would you do next? Are there any specific next steps you
would take?
8.6. Would you want to keep the information you were provided? [If necessary] Would you
print it out?
Goal: Assess whether users would recommend tool to friend or family member.
8.7. You had said that you would be [insert Net Promoter Score rating] to recommend this
tool to a friend or family member who was in the process of buying a home. Could you tell
me a little about your rating?
Goal:Assess whether users would find this tool helpful in their mortgage shopping process.
8.8. Would this be helpful in your mortgage shopping process?
8.9. What information on this sheet would you find to be the most helpful to you in your
honebuying process?
Goal: Assess whether there is additional content that users desire.
8.10. Was there anything else were you expecting to do with this tool that you couldn’t do?
(If yes) What were you expecting?

Owning a Home Monthly Payment Worksheet – Moderator Guide  8