Consumer Response Intake Form Improvement Study

Moderator’s Guide for the Consumer Response (CR1) Usability Test
Group A: People who had issues with financial products and services

Introduction

Thank you for participating in this study today. My name is _____, and I work with the User
Experience Team at Fors Marsh Group. Today we will be evaluating a website that has
financial information. I will give you a series of tasks that I would like you to complete using
the website. Your comments and feedback will help the developers improve the site. I did
not create the website, so please do not feel like you have to hold back on your thoughts to
be polite to me. We’re interested in both your positive and negative reactions to the website.
Difficulties you may run into reflect the design of the website, not your skills or abilities.
Please keep in mind that I am not testing you or your knowledge. Rather you are helping us
to see how we can improve this site.
The entire session should last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• We are making a video recording of this session. Only those of us associated with this
project will see the recordings, and we will not share your name or personal
information. The recordings are used as a memory aid for me so I can go back later
to recall what happened during each session.
• I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help. So while you are working, I would like you to think aloud. I would like
you to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
o Tell me if you are looking for something and what it is and whether you can
find it or not.
o Tell me why you click on a link and what you expect to happen.
o Basically, just tell me everything you are thinking as you work.
• There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
PFC Moderators Guide  1

•

Any questions so far?

We are going to track where you are looking as you work.
• This is our eye tracker [point to tracker].
• Let’s get you situated in your chair in front of the screen in a good position.
• Please try to refrain from making any large head movements during the session. You
do not need to remain perfectly still like a statue, but if you move your head too
much, the eye tracker may lose your eyes.
• Let’s do a quick task so the computer can find your eyes. Please follow the red circle
as it moves on the screen [calibration].
•

•
•
•
•

Ok, we are ready to begin. I will work from the other room, and we will communicate
through microphones and speakers. I will be able to see you with this camera [point
to camera].
Please complete this questionnaire as I go to the other room now and do a sound
check.
Do you have any questions?
Can you hear me ok?
Ok, we are ready to begin. I would like you to start with the first task in front of you
and begin by reading the task aloud. Then I will pull up the website, and I would like
you to use it to find the information. When you feel you have found it, I would like you
to tell me “This is my final answer.” Ready?

Notes:
• All tasks (except pre-task) begin on the home page. Participants will be redirected to
the home page at the conclusion of each task.
• After the participant reads the task, the moderator will start the eye tracker, and the
home page will load. The eye tracker will close the site at the end of each task.
• Participants will have the tasks on paper on the desk in front of them. Each task will
be on its own piece of paper, in large font.
• All participants will begin with the pre-task. After the pre-task, tasks will be randomly
assigned to prevent order effects. The moderator will place tasks in the correct order
before the session begins.
• Accuracy = percent of steps successfully completed for each task.
• Path = clicks made
• Task Time = page load to end of task (when they verbalize that they found the
answer)
• The first time participants use search, the moderator will probe about terms they
would enter and their expectations (before clicking search).
• Moderator: Note any inconsistencies between users’ terminology and the site’s.

PFC Moderators Guide  2

Pre-task interview: Steps taken with a complaint

You had indicated that you’ve had issues with a financial product or service in the past year.
Would you tell me about that issue? What steps did you take once you identified that there
was an issue? Who did you talk to? How did you research the problem? What resources did
you consult to try to resolve the issue? Did you visit any websites? Were there any resources
that you would have liked to have in the process but didn’t?

Task 1: Navigation to Submit a Complaint/First impressions (Difficulty:
Low) (Alternate between mobile and desktop)
Start on search engine of choice

Correct path: Search (Help with a financial complaint)  consumerfinance.gov/complaint
1a. Let’s say you have an issue with a financial product or service and you’ve heard that
there is a site that can help you get a response from the company. Please show me
where you would go and what you would do to get help from a 3rd party resource?
1b. What are your first impressions of this website?
1c. Who is this site for?
1d. What would you do next? [Moderator: Probe further – Tell me more about that. What
would get you to explore this site?]

Task 2: Submit his/her own complaint (Difficulty: Moderate) (Alternate
between mobile and desktop).
Start on consumerfinance.gov/complaint

2a. Let’s use your own complaint for the 1st scenario. Please submit a complaint about the
issue you had with the financial product or service.
2b. What would you do next?
2c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

PFC Moderators Guide  3

Task 3: Debt collection complaints - Lenders, Bank Accounts, & CRA
(Difficulty: Moderate) (Alternate between mobile and desktop).
Start on consumerfinance.gov/complaint

Correct path: Debt collection  Get started  What type of debt is this? (Other)  Which of
these best describes your issue (Continued attempts to collect debt I do not owe)  Debt
was paid  Describe issue  Provide desired resolution  Continue  Debt collection
company name  Do you know where this debt came from? (same company)  Continue
 Enter personal information  Enter contact information  Enter military affiliation 
Continue  Check box  Submit
3a. A debt collection company has been frequently calling you about the balance for your
Comcast service which you have canceled. You have tried telling the caller that the service
was fully paid for, but they insist that the debt exists and that your credit report will be
affected if you don’t pay the balance of $138.45. You have started to simply ignore the calls,
but you just noticed that the debt has shown up on your credit report and caused a
significant drop in your credit score.

3b. What would you do next?
3c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

Task 4: Credit reporting complaints - Lenders, Debt Collection, & Bank
Account (Difficulty: Moderate) (Alternate between mobile and desktop).
Start on consumerfinance.gov/complaint

Correct path: Credit reporting  Get started  Which of these best describes your issue?
(Incorrect information on my credit report)  Account status  Have you disputed…(Yes) 
Describe what happened  Continue  Enter company name  Continue  Enter desired
resolution  Enter consumer information  Continue  Check boxes  Submit
4a. You recently applied for a Home Depot credit card to buy a new washer and dryer and
were declined because your credit history said that you are delinquent on your auto loan
with Bankwell loans. You call your auto loan company and they tell you that you are current
on your payments. Very confused, you call the credit reporting agency that provided your
credit history. They insist that the delinquency information is accurate because it was
supplied by your auto loan company. You don’t know why this is happening and know that
you can’t buy the kitchen appliance until this is fixed. You are frustrated that each company
blames the other and isn’t doing anything to help you. You decide to submit a complaint to
the CFPB.
PFC Moderators Guide  4

4b. What would you do next?
4c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

Task 5: Bank account complaints – CRA, Lenders, & Debt Collection
(Difficulty: Moderate) (Alternate between mobile and desktop).
Start on consumerfinance.gov/complaint

Correct path: Bank account or service  Get started  Describe what happened  This is
about a (checking account).  Problems caused by my funds being low  Continue 
Desired resolution  Enter my contact information  Continue  Enter company name 
Continue  Check box  Submit
5a. Four weeks ago, you purchased a short-term payday loan from Kwik Cash to pay for car
repairs. You had planned to pay off the loan in two weeks, but the money wasn’t available so
you asked to extend the loan for another two weeks. You had planned to extend the loan
again this week when you realized that Kwik Cash had already deducted the amount from
your account, which resulted in an overdraft fee of $35.
5b. What would you do next?
5c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

PFC Moderators Guide  5

Task 6: Mortgage (Difficulty: High) (Alternate between mobile and
desktop).
Start on consumerfinance.gov/complaint

Correct path: Mortgage  Get started  Describe what happened  Problems when you
are unable to pay  Are you concerned about losing your home to foreclosure? (yes)  Is
there a date scheduled…(Don’t know)  Did you pay a company to help you avoid
foreclosure? (No).  My loan is a…conventional adjustable mortgage.  Continue  Enter
desired resolution  Enter my contact information  Continue  Enter company name 
Continue  Check box  Submit
6a. Eight years ago, you purchased an adjustable rate mortgage for your home from Athens
Bank and in the past year you haven’t been able to make all of your payments. Two months
ago, you received a letter from a law firm called Castle & Mallone stating that West Bank
was now the servicer of your loan and also that your house was now in foreclosure. You
immediately called Athens Bank to figure out if this was accurate and a Vice President from
Athens Bank sent an email on your behalf to a representative at West Bank. The
representative at West Bank said that they do not own your loan.
You found this very confusing and you don’t understand how West Bank has the right to
foreclose on your home which you purchased with Athens Bank, so you filed a complaint
with the CFPB asking to find and speak to the person who actually owns your loan.
6b. What would you do next?
6c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

Task 7: Vehicle lending (Difficulty: Moderate) (Alternate between mobile
and desktop).
Start on consumerfinance.gov/complaint

Correct path: Vehicle lending  Get started  Describe what happened  My loan is a
…vehicle loan  Taking out the loan or lease  Enter desired resolution  Continue 
Enter my information  Continue  Enter company name  Continue  Check box 
Submit
7a. You recently bought a used car at Al’s Best Cars, a local dealership. Though you had
secured a pre-approved loan with your local bank beforehand, the financing Al offered you
had a lower APR and you ultimately decided that his loan was better.
When you get your first monthly bill in the mail, you are very confused that it is from
Northern Bank and not Al or Ford. You also look at the account terms and interest rate and it
PFC Moderators Guide  6

seems to be considerably more than you remember. You don’t know who the new company
is and believe that you are being billed for the wrong loan.
7b. What would you do next?
7c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

Task 8: Money transfer (Difficulty: High) (Alternate between mobile and
desktop).
Start on consumerfinance.gov/complaint

Correct path: Money transfer or virtual currency  Money transfer  Get started 
Describe what happened This was about an…international money transfer  Describe
issue (wrong amount charged or received)  Enter desired resolution  Continue  Enter
my information  Continue  Enter company name  Enter sender information  Enter
amount of error.  Enter person receiving funds  Continue  Check box  Submit
8a. A close relative is abroad on a month-long vacation when you get an urgent message
that they need cash to pay for an earlier return flight. You remember StoreMart advertising
instant money transfers. While shopping at StoreMart that afternoon, you pay $75 to send
an instant transfer of $600 to your relative. Your receipt indicates that the money will be
delivered by InstaMoney and available for pick up in an hour at a branch of International
Bank. You called your relative late that night and it seems that the money is still not
available. You call StoreMart and InstaMoney but can’t get through to anyone. The next
morning you finally get through to a InstaMoney agent but they are unable to provide you an
explanation. Finally, two days later the money becomes available for your relative but you
notice that amount delivered is about $50 less than you had anticipated. You believe that
International Bank may have charged you more. You are upset about the entire experience.
8b. What would you do next?
8c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

PFC Moderators Guide  7

Task 9: Installment loan (Difficulty: Medium) (Alternate between mobile
and desktop).
Start on consumerfinance.gov/complaint

Correct path: Other consumer loan  Get started  Installment loan  Describe what
happened Managing the loan or lease  Enter desired resolution  Continue  I am the
borrower  Enter your information  Continue  Enter company name  Continue 
Submit
9a. You saw a great sale in the weekend flier for a refrigerator at Home Depot. You go to
Home Depot and take out a no-interest installment loan on the refrigerator which you’ll
make regular fixed payments on. When you get your first bill in the mail though, you notice
that the loan is owned by a bank, not Home Depot. You’re very confused because you
purchased this refrigerator and loan from Home Depot. You’ve tried contacting Home Depot
for further information but all of the customer service representatives you’ve talked to are
not sure why a bank would contact you about your loan. You want to make sure that this
bank that has contacted you actually represents Home Depot for this loan. You decide to
contact the CFPB to find out if the bank actually owns your Home Depot loan.
(If correct) – How clear was it that Citi Bank manages your loan with Home Depot?
Not clear at all
Slightly clear
Moderately clear
Very clear
Extremely clear
1
2
3
4
5

PFC Moderators Guide  8

Satisfaction Questionnaire (SATQ)

Please circle the numbers that most appropriately reflect your impressions about using this device.

1. Please rate your overall experience with the Submit a Complaint website:
Not Difficult
At All
1

Slightly
Difficult
2

Moderately
Difficult
3

Very
Difficult
4

Extremely
Difficult
5

2. Please rate the clarity of the information presented to you on the Submit a Complaint
website:
Not Clear
At All
1

Slightly
Clear
2

Moderately
Clear
3

Very
Clear
4

Extremely
Clear
5

3. Please rate your difficulty with navigating on the Submit a Complaint website:
Not Difficult
At All
1

Slightly
Difficult
2

Moderately
Difficult
3

Very
Difficult
4

Extremely
Difficult
5

4. How likely would you be to use this site in the future?
Not Likely
At All
1

Slightly
Likely
2

Moderately
Likely
3

Very
Likely
4

Extremely
Likely
5

5. How likely would you be to recommend this site to a friend who has a similar issue?
Not Likely
At All
1

Slightly
Likely
2

Moderately
Likely
3

Very
Likely
4

Extremely
Likely
5

6. Please rate the organization of the website in general:
Not Clear
At All

Slightly
Clear

Moderately
Clear

Very
Clear

Extremely
Clear

1

2

3

4

5

7. Please rate whether the Submit a Complaint website met your expectations:
Did not meet
expectations
1

Met
expectations
2

Exceeded
expectations
3
PFC Moderators Guide  9

8. How hard was it to submit a complaint using the website?
Not Difficult
Slightly
Moderately
Very
At All
Difficult
Difficult
Difficult
1
2
3
4

Extremely
Difficult
5

PFC Moderators Guide  10

Debriefing Questionnaire
(NOTE: Some are used as follow-ups to actions during session and to SATQ
responses; participants do not receive all debriefing questions.)
1. What stands out to you about your experience using the site?

2. How would you describe this site to a friend?

3. What can this organization help you with? What can’t they help you with?]

4. Tell me about your overall experience using this website. What worked well; what did
not work well? [Probe further as necessary.]

5. Let’s imagine you were to use this site on your own. What information would you use?

6. Would you share any of the information on social media? What would you share?

7. [If participant uses search] Tell me about your experience using Search. Did you get
what you were looking for?

8. [Follow up to SATQ5] What could be changed to increase the likelihood that you
would recommend this site to a friend?

9. [Follow up to SATQ4] What could be improved on the site that would increase the
likelihood that you would use it in the future?

PFC Moderators Guide  11

10. What did you think about the steps involved with filing a complaint? Do you think the
steps were appropriate? Anything difficult or confusing?

11. What else were you expecting to do on this website that you couldn’t do? What is
missing?

12. Where would you go or what would you do after using this site?

13. [Follow up regarding some of the SATQ items.]

14. What would you say is the best thing about this website?

15. What would you say is the worst thing about this website?

16. What did you think about the mobile experience? What do you think should be
different on the mobile site?

PFC Moderators Guide  12