Owning a Home Project User Experience Testing – Round 1

MODERATOR’S GUIDE

Owning a Home
Know the Process - Interaction
UX Test (OaH3) - DRAFT
Data Collection: [Insert Dates]

DRAFT

Introduction (5 minutes)

Thank you for participating in this study today. My name is _____, and I am a researcher with
the Fors Marsh Group User Experience Team. Today we will be evaluating a prototype of a
website.
I will ask you questions and give you some tasks to complete using the website. Your
comments and feedback will help the team improve the site. I did not create the sites, so
please do not feel like you have to hold back on your thoughts to be polite to me. I’m
interested in both your positive and negative reactions. Difficulties you may run into reflect
issues with the design of the site, not your skills or abilities. Please keep in mind that I am
not testing you or your knowledge. Rather, you are helping us to see how we can improve
this site.
The entire session will last about 60 minutes. In accordance with the Papework Reduction
Act of 1995, this study as been approved by the Federal government’s Office of
Management and Budget (OMB) under OMB control number 3170-0024. Without this
approval we could not conduct this study. Do you have any questions so far?
Let’s cover a few things before we get started.
You might have noticed the cameras. I am recording the session. The recording is used as a
memory aid for me so I can recall what happened during the session. Please speak openly
about your opinions and experiences. We want to learn from you, so it is important that you
share your honest opinions.
I am interested in your thoughts and reactions as we proceed. This is important because I
can see what you are doing, but I won’t know why you are doing it. So I need your help. While
you are working, I would like you to think aloud. I would like you to:
•
•
•
•
•

Tell me what you are thinking
Describe the steps that you are taking.
Tell me why you are doing what you are doing, what you are going to do, and why.
Tell me why you clicked on a link and what you expect to happen next.
Basically, just tell me everything you are thinking as you work.

There are no right or wrong answers, and your comments and opinions will only be used in
combination with the feedback that we get from other people.
What we talk about here is private. That means your name will not be associated with
anything you say in our reports and your responses will not be linked to your identity in any
way.
Likewise, we want to respect everyone’s privacy in this room and not share any of our
discussion from today with others who are not actively working on this project.
Owning a Home Know the Process - Interaction – Moderator Guide  1

DRAFT

Please also make sure to review the Privacy Act Statement provided to you in hard copy.
[Provided in hard copy.]
5 U.S.C. 552(a)(e)(3)
The information you provide through your responses to Fors Marsh will assist the study
sponsor, the Consumer Financial Protection Bureau (“CFPB”), in participating in usability
testing that evaluates consumers’ experiences with a financial website.
The CFPB will obtain and access identifying information from Fors Marsh about study
participants by observing recordings made during this usability testing.
Information collected on behalf of the Bureau by Fors Marsh will be treated in accordance
with the System of Records Notice (“SORN”), CFPB.022,
https://www.federalregister.gov/articles/2012/11/14/2012-27582/privacy-act-of-1974as-amended. This information will not be disclosed. The Routine Uses for the SORN
specifies how we may use this information. Direct identifying information will only be used by
Fors Marsh to facilitate the usability testing and will be kept private except as required by
law.
This collection of information is authorized by Pub. L. No. 111-203, Title X, Sections 1013
and 1022, codified at 12 U.S.C. §§ 5493 and 5512.
Participation in this study is voluntary, you are not required to participate or share any
personally identifying information with Fors Marsh during the usability testing, including
name, age, email address, mobile number, alternate number, race, ethnic origin, gender,
city and state of residence, education level, household income, current employment status,
and occupation. You may withdraw from participation at any time.
•
•

Any questions so far?
Ok, we are ready to begin. I would like you to start by asking you some background
questions.

Owning a Home Know the Process - Interaction – Moderator Guide  2

DRAFT

Section 1: Interview about steps taken in the homebuying/mortgage
process (3 minutes)

Section 1 Goal: Understand the stage each participant in the homebuying process.
You had indicated that you are [in the process of buying a home/preparing to buy a home].
I’m going to ask you a few general questions about your experience with this process.
1.1. Would you tell me about your experience so far? What has been your experience with
looking for a mortgage?
1.2. What are the major challenges you have been facing as you’ve been shopping? What
have you been struggling with?
1.3. What resources have you consulted (friends or tools/resources)? What tools or
resources do you wish you had access to but didn’t?

Section 2: First Impressions (5 minutes)
[Rotate order of devices]

Goal: Assess users’ first impressions with the main landing page
Now I’m going to show you the site we’ll be reviewing today.
2.1. What are your first impressions of it?
Goal: Assess what users think this site is supposed to help them do.
2.2. What do you think is the purpose of this site?

Section 3: Free Exploration – Landing Page (5 minutes)

Goal: Assess what section users’ feel would help them most in reaching their goal
3.1. We had discussed earlier how you are [Insert homebuying stage]. Now, I will ask you to
go through the site and use it for your own homebuying process. Without making a click,
please show me what you would do first.
Goal: Assess users’ expectations when looking for information related to their homebuying
goals
3.2. What are you expecting to find after clicking on that link?
3.3. Could you talk about why you would click there? What are you hoping to find?
3.4. [After click] Does this match what you were expecting to find?
3.5. What are your first impressions?
Ask questions 3.6 – 3.10 if participant does not click on Know the Process page links.
3.6. [If participant did not click on a Know the Processlink, redirect back to the Owning a
Home landing page] Let’s go back to the first page we viewed. Similar to the stage you are
at in the home buying process, let’s say you’re looking for more information about what to
Owning a Home Know the Process - Interaction – Moderator Guide  3

DRAFT
do next as you are [insert participant’s stage in the home buying process]. Without making a
click, please show me what you would do next.
3.7. What are you expecting to find after clicking on that link?
3.8. Could you talk about why you would click there? What are you hoping to find?
3.9. [After click] Does this match what you were expecting to find?
3.10. What are your first impressions?

Section 4: Know the Process Tool Tasks (15 minutes)

Overarching goals of Section 4: Assess whether the navigation is effective in helping users
have clear sense of place and of the process. Assess whether the amount of entry points on
the individual and landing pages are easy to digest and skim.
Goal: Assess users’ initial interaction with the Know the Process pages
4.1. Considering where you are at in the home buying process, please take a few minutes to
explore this page. Please think aloud as you are doing so.
Goal: Assess whether users can find their place in the in the homebuying process among
the presented action steps
4.2. [If they didn’t pick an action step] Let’s say you wanted more in-depth information about
the step you are at in the home buying process. Please think aloud as you’re trying to find
more in-depth information.
Goal: Assess how users navigate to the next action step in the Know the Process tool
4.3. Let’s say you want to go to the next step in the process; please show me what you
would do.
Goal: Assess how users navigate to the next section in the Know the Process tool
4.4. Let’s say you’ve progressed through all of the action steps on this page, what would you
do next if you wanted to learn more about the next stage in the homebuying process?
4.5. [If participant is at the end of the home buying process] Let’s imagine that you are
actually in the beginning of the home buying process rather than towards the end, please
show me what you would do to find information that would relate to you in this scenario.
Goal: Assess users’ expectations about what they want to learn from the site
4.6. Now that you have gone through the [insert section] section of the site, what other kinds
of information would you specifically look for on this site? What would you want to learn
more about?
Goal: Assess where users look for financial information, and how it has influenced their
homebuying process
4.7. What next steps would you take after reviewing the information on this site, in regards
to your mortgage shopping process? What other kinds of information would you look for?
Owning a Home Know the Process - Interaction – Moderator Guide  4

DRAFT

Section 5: Questionnaires (3 minutes)
System Usability Scale

Please circle the numbers that most appropriately reflect your impressions about using this website.
1. I think that I would like to use the Owning a Home site frequently:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

2. I found the Owning a Home site unnecessarily complex:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

3. I thought the Owning a Home site was easy to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

4. I think that I would need the support of a technical person to be able to use the Owning a
Home site:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

5. I found the various functions in the Owning a Home site were well integrated with each other:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

6. I thought there was too much inconsistency in the Owning a Home site:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

Owning a Home Know the Process - Interaction – Moderator Guide  5

DRAFT
7. I would imagine that most people would learn to use the Owning a Home site very quickly:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

8. I found the Owning a Home site very cumbersome to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

9. I felt very confident using the Owning a Home site:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

10. I needed to learn a lot of things before I could get going with the Owning a Home site:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

Net Promoter Score (NPS)

Please circle the numbers that most appropriately reflect your impressions about using this website.
On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this site
to a friend, colleague, or family member with similar goals in mind?

1

2

3

4

5

6

7

8

9

10

Owning a Home Know the Process - Interaction – Moderator Guide  6

DRAFT

Section 6: Design and imagery (5 minutes)

Goal: Assess users’ response to the illustrations and other graphics, including the amount
of them
6.1. Now I’d like to ask you to look at the graphics on the home page as well as the Know
the Process page we reviewed. Please feel free to click around as we are discussing. Did you
notice them before?
6.2. What are your general impressions of them?
6.3. Are there any that confuse you?
6.4. Are there any that you really like or dislike?
6.5. Is there anything you would change about them?
6.6. What do you think about the amount of graphics and illustrations? Are there too many
or not enough?
Goal: Assess how the graphics help users to find the information they need on the site
6.7. Do the graphics help you to understand the purpose of the page? [On homepage] Did
the image on the page help you to find what you were looking for in that section?

Section 7: Information Architecture and Navigation (10 minutes)

Goal: Determine the most memorable aspect of the tool.
7.1. What stands out to you about your experience using this site? What worked well? What
didn’t work well?
Goal: Assess ease of navigation within the Know the Process pages
7.2. What was it like trying to locate your point in the process?
7.3. What was it like navigating through each of the action steps? Did everything work as
you expected? Was anything confusing about navigating through these different steps?
Goal: Assess how the page organization affects users’ ability to navigate the page, for the
Know the Process page they review, and the site overall
7.4. What do you think about how the information on the site is organized? What do you like
or dislike about how the information is organized?
7.5. What do you think about the way in which the information was broken up into different
steps on each page?
7.6. [If necessary] How did the way in which this information was organized into steps affect
how you navigated the page? Was it helpful to have the information and tools broken up into
different steps as you tried to locate information?
Owning a Home Know the Process - Interaction – Moderator Guide  7

DRAFT

7.7. What do you think about the way in which the information was broken up into different
sections on different pages?
7.8. [If necessary] How did the way in which this information was organized into sections
affect how you navigated the site? Was it helpful to have the information and tools broken
up into different sections as you tried to locate information?
Goal: Assess how easily users are able to navigate the number of entry points to content on
the landing page
7.9. Was the information you were looking for about [insert stage in the homebuying
process] in the place that you expected it to be?

Section 8: Trust (5 minutes)

Goal: Assess whether users trust the information they receive through this tool
8.1. On a scale from 1-5, with 1 being not trustworthy at all and 5 being extremely
trustworthy, how much do you trust the information on this site? Why or why not?
Goal: Assess whether the process outlined on the site matches the users’ mental models
8.2. Does the process on this page match your own understanding of the process? What is
the same? What is different? How does this affect how you trust the information?
Goal: Assess what factors affect that trust (or lack thereof): other sources? Info from people
they know? Etc.
8.3. Is there any information on this site that you have also received from other sources? Is
there any information that contradicts information you’ve previously received? How does
that difference/similarity affect your level of trust?
8.4a. [if they trust the site] Can you explain what makes you trust the information on this
site?
8.4b. [if they don’t trust the site] Is there anything that could be added to the site that would
make you more likely to trust this site?

Section 9: Behavior Change (3 minutes)

Goal: Assess whether users would use this tool repeatedly; assess when they use it and
under what circumstances
9.1. Do you think you would access the information on this site during your mortgage
process? When do you think you would do so?
9.2. [if more than once] How many times do you think you would refer to it: repeatedly, or
once in a while?
Goal: Assess whether users’ approach to buying a home and getting a mortgage reportedly
changes after using this tool
Owning a Home Know the Process - Interaction – Moderator Guide  8

DRAFT
9.3. Do you feel like your approach to buying a home and getting a mortgage would change
after visiting this site and using these tools? How so?

Owning a Home Know the Process - Interaction – Moderator Guide  9