CFPB's Consumer Response Intake Form Improvement Study, Second Iteration (Intake Form Improvement Study II)

Moderator’s Guide for the Consumer Response (CR3) Usability Test
Group A: People who had issues with financial products and services

Introduction

Thank you for participating in this study today. My name is _____, and I work with the User
Experience Team at Fors Marsh Group. Today we will be evaluating a website that has
information about financial products and services. I will give you a series of tasks that I
would like you to complete using the website. Your comments and feedback will help the
developers improve the site. I did not create the website, so please do not feel like you have
to hold back on your thoughts to be polite to me. We’re interested in both your positive and
negative reactions to the website. Difficulties you may run into reflect the design of the
website, not your skills or abilities. Please keep in mind that I am not testing you or your
knowledge. Rather you are helping us to see how we can improve this site.
The entire session should last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• We are making a video recording of this session. Only those of us associated with this
project will see the recordings, and we will not share your name or personal
information. The recordings are used as a memory aid for me so I can go back later
to recall what happened during each session.
• I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help. So while you are working, I would like you to think aloud. I would like
you to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
o Tell me if you are looking for something and what it is and whether you can
find it or not.
o Tell me why you click on a link and what you expect to happen.
o Basically, just tell me everything you are thinking as you work.
• The site is still in development so not everything will work perfectly. I’d like you to use
the site though as if it were.
PFC Moderators Guide  1

•
•
•

We do not want you to use your personal information when completing the form.
Instead, we’d like you to use this information here.
There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
Any questions so far?

We are going to track where you are looking as you work.
• This is our eye tracker [point to tracker].
• Let’s get you situated in your chair in front of the screen in a good position.
• Please try to refrain from making any large head movements during the session. You
do not need to remain perfectly still like a statue, but if you move your head too
much, the eye tracker may lose your eyes.
• Let’s do a quick task so the computer can find your eyes. Please follow the red circle
as it moves on the screen [calibration].
•

•
•
•
•

Ok, we are ready to begin. I will work from the other room, and we will communicate
through microphones and speakers. I will be able to see you with this camera [point
to camera].
Please complete this questionnaire as I go to the other room now and do a sound
check.
Do you have any questions?
Can you hear me ok?
Ok, we are ready to begin. I would like you to start with the first task in front of you
and begin by reading the task aloud. Then I will pull up the website, and I would like
you to use it to find the information. When you feel you have found it, I would like you
to tell me “This is my final answer.” Ready?

Notes:
• All tasks (except pre-task) begin on the home page. Participants will be redirected to
the home page at the conclusion of each task.
• After the participant reads the task, the moderator will start the eye tracker, and the
home page will load. The eye tracker will close the site at the end of each task.
• Participants will have the tasks on paper on the desk in front of them. Each task will
be on its own piece of paper, in large font.
• All participants will begin with the pre-task. After the pre-task, tasks will be randomly
assigned to prevent order effects. The moderator will place tasks in the correct order
before the session begins.
• Accuracy = percent of steps successfully completed for each task.
• Path = clicks made
• Task Time = page load to end of task (when they verbalize that they found the
answer)
PFC Moderators Guide  2

•
•

The first time participants use search, the moderator will probe about terms they
would enter and their expectations (before clicking search).
Moderator: Note any inconsistencies between users’ terminology and the site’s.

Pre-task interview: Steps taken with a complaint

You had indicated that you’ve had issues with a financial product or service in the past year.
Would you tell me about that issue? What steps did you take once you identified that there
was an issue? Who did you talk to? How did you research the problem? What resources did
you consult to try to resolve the issue? Did you visit any websites? Were there any resources
that you would have liked to have in the process but didn’t?

PFC Moderators Guide  3

Task 1: Submit his/her own complaint (Difficulty: Moderate) (Alternate
between mobile and desktop).
Start on consumerfinance.gov/complaint

1a. Let’s use your own complaint for the 1st scenario. Please submit a complaint about the
issue you had with the financial product or service.

1b. What would you do next?

1c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

1d. (Take participant back to first page) Would you walk me through your thought process
when you were selecting the product? Would you walk me through your thought process
when you were selecting the type?

1e. Would you walk me through your thought process when you were selecting what best
describes your issue?

1f. Would you tell me a bit about which documents you would attach in this scenario?
PFC Moderators Guide  4

Task 2: Submit his/her own complaint (Difficulty: Moderate) (Alternate
between mobile and desktop).
Start on consumerfinance.gov/complaint

2a. Let’s use your own complaint for the 1st scenario. Please submit a complaint about the
issue you had with the financial product or service.

2b. What would you do next?

2c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

2d. (Take participant back to first page) Would you walk me through your thought process
when you were selecting the product? Would you walk me through your thought process
when you were selecting the type?

2e. Would you walk me through your thought process when you were selecting what best
describes your issue?

2f. Would you tell me a bit about which documents you would attach in this scenario?

PFC Moderators Guide  5

Task 7: Vehicle lending (Difficulty: Moderate) (Alternate between mobile
and desktop).
Start on consumerfinance.gov/complaint

Correct path: Submit a complaint  Vehicle loan or lease  Loan or lease? (Loan)  Best
describes issue (Problems when making payments)  Select one (Issues related to the
transfer of your loan to another company  Company  Enter what happened  Enter fair
resolution  Enter primary consumer information (Aunt)  Enter additional point of contact
(self)  Continue  Vehicle loan company information (Northern Bank)  Additional
company (Golden Valley Used Cars)  Continue  Submit
7a. Your aunt, Martha Stevens, recently bought a used car at Golden Valley Used Cars, a
local dealership. Though she had secured a pre-approved loan with her local bank at 6.5%,
the dealership offered her closer to 4% with no down payment so she decided that the
dealership’s loan was better.
When she gets her first monthly bill in the mail, she is very confused that it is from Northern
Bank and not Golden Valley Used Cars. The monthly payment and interest rate are also
considerably higher than she remembers. She doesn’t know who Northern Bank is and
believes that she is being billed for the wrong loan. Your aunt does not have strong
computer skills so you decide to submit a complaint on her behalf.
7b. What would you do next?

7c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

7d. (Take participant back to first page) Would you walk me through your thought process
when you were selecting the product? Would you walk me through your thought process
when you were selecting the type?

7e. Would you walk me through your thought process when you were selecting what best
describes your issue?

7f. Would you tell me a bit about which documents you would attach in this scenario?

PFC Moderators Guide  6

Task 8: Money transfer (Difficulty: High) (Alternate between mobile and
desktop).
Start on consumerfinance.gov/complaint

Correct path: Submit a complaint  Money transfer or money service  International
money transfer  Continue  Best describes issue (Money was not available when
promised)  Continue  Describe what happened  Enter fair resolution  Continue 
Enter primary consumer information  Enter additional consumer (Recepient)  Continue
 Enter company name (Instamoney)  Enter additional company (Banco Popular) 
Continue  Submit
8a. Your cousin, Mark Smith, is in Argentina on a month-long vacation when you get an
urgent message that they need cash to pay for an earlier return flight. You remember
StoreMart advertising instant money transfers. While shopping at StoreMart that afternoon,
you pay $75 to send an instant transfer of $600 to your relative. Your receipt indicates that
the money will be delivered by InstaMoney and available for pick up in an hour at a branch
of the bank Banco Popular. You called your relative late that night and it seems that the
money is still not available. You call StoreMart and InstaMoney but can’t get through to
anyone. The next morning you finally get through to a InstaMoney agent but they are unable
to provide you an explanation. Finally, two days later the money becomes available for your
relative but you notice that amount delivered is about $50 less than what you had
anticipated. You believe that Banco Popular may have charged you more. You are upset
about the entire experience.

8b. What would you do next?
8c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?

8d. (Take participant back to first page) Would you walk me through your thought process
when you were selecting the product? Would you walk me through your thought process
when you were selecting the type?
8e. Would you walk me through your thought process when you were selecting what best
describes your issue?

8f. Would you tell me a bit about which documents you would attach in this scenario?
PFC Moderators Guide  7

Task 9: Installment loan (Difficulty: Medium) (Alternate between mobile
and desktop).
Start on consumerfinance.gov/complaint

Correct path: Submit a complaint  Credit reporting  Credit reporting or credit score 
Continue  Incorrect information on my credit report  Incorrect status of my account 
Continue  Enter what happened  Enter fair resolution  Enter primary consumer 
Continue  Credit reporting company (Transunion)  Additional company (Home Depot) 
Continue  Submit
9a. You recently received an email alert that your credit score had gone down 30 points and
pulled a credit report from Transunion to investigate. You noticed that the drop occurred
because your Home Depot credit card was marked 90 days overdue. You remember paying
your bill on time every month, so you logged in to your credit card account and it also said
that you are current on your account. You called Home Depot and their customer service
representatives said that they don’t know why the error exists and that you should contact
Transunion. You called Transunion and they told you that they received that information
from your credit card company, but they’re willing to start an investigation, but it may take
up to 90 days. You’re frustrated because you always paid your bills on time and don’t know
why the information can’t be removed from your credit report.
9b. What would you do next?
9c. What would you expect CFPB to do next? Would you expect to hear back from CFPB?
When would you expect to hear back? What would you expect from them when you do hear
back?
9d. (Take participant back to first page) Would you walk me through your thought process
when you were selecting the product? Would you walk me through your thought process
when you were selecting the type?
9e. Would you walk me through your thought process when you were selecting what best
describes your issue?

9f. Would you tell me a bit about which documents you would attach in this scenario?

PFC Moderators Guide  8

Satisfaction Questionnaire (SATQ)

Please circle the numbers that most appropriately reflect your impressions about using this device.

1. Please rate your overall experience with the Submit a Complaint website:
Not Difficult
Slightly
Moderately
Very
Extremely
At All
Difficult
Difficult
Difficult
Difficult
1
2
3
4
5
2. Please rate the clarity of the information presented to you on the Submit a Complaint
website:
Not Clear
At All
1

Slightly
Clear
2

Moderately
Clear
3

Very
Clear
4

Extremely
Clear
5

3. Please rate your difficulty with navigating on the Submit a Complaint website:
Not Difficult
Slightly
Moderately
Very
Extremely
At All
Difficult
Difficult
Difficult
Difficult
1
2
3
4
5
4. How likely would you be to use this site in the future?
Not Likely
Slightly
Moderately
Very
At All
Likely
Likely
Likely
1
2
3
4

Extremely
Likely
5

5. How likely would you be to recommend this site to a friend who has a similar issue?
Not Likely
Slightly
Moderately
Very
Extremely
At All
Likely
Likely
Likely
Likely
1
2
3
4
5
6. Please rate the organization of the website in general:
Not Clear
At All

Slightly
Clear

Moderately
Clear

Very
Clear

Extremely
Clear

1

2

3

4

5

7. Please rate whether the Submit a Complaint website met your expectations:
Did not meet
Met
Exceeded
expectations
expectations
expectations
1
2
3
8. How hard was it to submit a complaint using the website?
Not Difficult
Slightly
Moderately
Very
At All
Difficult
Difficult
Difficult
1
2
3
4

Extremely
Difficult
5
PFC Moderators Guide  9

Debriefing Questionnaire
(NOTE: Some are used as follow-ups to actions during session and to SATQ
responses; participants do not receive all debriefing questions.)
Opening questions
1. What stands out to you about your experience using the site? Tell me about your
overall experience using this website. What worked well; what did not work well?
[Probe further as necessary]

2. How would you describe this site to a friend?

3. What did you think about the order in which you entered information?

4. How would you check the status of your complaint?

5. How would you submit your next complaint?

6. What do you consider a fair resolution?

7. After you submit the complaint to CFPB, what happens next? How do you think the
CFPB will handle your complaint?

PFC Moderators Guide  10

Home page
8. What did you think about the home page?

9. If you already submitted a complaint and you hadn’t heard a response, how would
you submit a new complaint about the same issue?

10. What did you think about the login link?

Extra questions
11. What did you think about the steps involved with filing a complaint? What did you
think about the order of the step? Do you think the steps were appropriate? Anything
difficult or confusing?
12. Where would you go or what would you do after using this site?

13. What can this organization help you with? What can’t they help you with?

14. [Follow up to SATQ5] What could be changed to increase the likelihood that you
would recommend this site to a friend?

15. [Follow up to SATQ4] What could be improved on the site that would increase the
likelihood that you would use it in the future?

PFC Moderators Guide  11

16. What else were you expecting to do on this website that you couldn’t do? What is
missing?

17. [Follow up regarding some of the SATQ items.]

18. What would you say is the best thing about this website?

19. What would you say is the worst thing about this website?

What product or service is this complaint about?
20. Walk me through your experience using this page.

21. What did you think about the text below the bolded text?

Which of these best describes your issue?
22. Walk me through your experience using this page.

23. What did you think about the text below the bolded text?

Which of these best describes your issue?
24. Walk me through your experience using this page.
PFC Moderators Guide  12

25. What did you think about the text below the bolded text?

Who is submitting this complaint?
26. Walk me through your experience using this page.

27. What did you think about selecting the option for the primary consumer?

28. Would you tell me about how you would use the email box? Which email would you
use? How frequent do you check that email?

29. What would you expect to happen if you clicked “I don’t have an email address.”

30. Please tell me about your experience with adding an additional consumer. What
would you expect to happen if you checked the “Allow this person to access this
complaint and receive status updates.”

31. Please tell me about your experience with adding an additional point of contact.
What would you expect to happen if you entered your email?

PFC Moderators Guide  13

What company should receive this complaint?
32. Please tell me about your experience entering in the company information. How
about providing an account number?

33. Please tell me about your experience with entering in an additional company. How
about selecting the product on this page?

Review and attach documents
34. What are your thoughts about the attach documents page? How would you use this
page?

PFC Moderators Guide  14