Round 2 of User testing to solicit feedback on the Bureau's Planning for retirement- Before you claim English website

MODERATOR’S GUIDE

Claiming Social Security
User Experience Test (Round 2)
Data Collection: November 5-6, 2015

Section 1: Introduction
Thank you for participating in this study today. My name is _____, and I am a researcher with
the Fors Marsh Group User Experience Team. Today we will evaluate a website that has
information about claiming Social Security retirement benefits and retirement planning.
Some of the pages you will interact with are still in development, so not all the links will be
clickable and others may have missing elements. If you encounter something that isn’t
working, please let me know so I can get us back on track.
I will ask you questions and give you some tasks to complete using the website. Your
comments and feedback will help the design team improve the site. I am a researcher and
did not create the site, so please do not feel like you have to hold back on your thoughts to
be polite to me. I’m interested in both your positive and negative reactions to them.
Difficulties you may run into reflect issues with the design of the sites, not your skills or
abilities. Please keep in mind that I am not testing you or your knowledge. Rather, you are
helping us to see how we can improve this site.
The entire session will last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• We are making a video recording of the computer screen during this session, but your
name and personal information will not be associated with the recordings.
• I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help. While you are working, I would like you to think aloud. I would like you
to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
o Tell me why you clicked on a link and what you expect to happen next.
o Basically, just tell me everything you are thinking as you work.
• There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
• Any questions so far?
•
•

Before we begin, we’d like to provide you with a copy of our privacy act
statement. Feel free to read it now, if you’d like. Do you have any questions about it?
Ok, we are ready to begin. I would like you to start by asking you some background
questions.
Claiming Social Security Usability Test (Round 2), Dev. September 2015  1

Claiming Social Security Usability Test (Round 2), Dev. September 2015  2

Section 2: Questions and Tasks
Item 1: Pre-task interview

Goal: To get to know participants and explore their relevant past behavior.
1a. What types of electronic devices do you use to visit websites?
1b. What devices do you usually use to track or manage your financial accounts online?
1c. What is your main online resource when you are looking for financial information?
1d. What other resources might you use to find similar information?

Item 2: Planning around retirement and Social Security benefit claiming

Goal: To explore participants’ behavior around retirement planning and Social Security
retirement benefits.
2a. We'd like to talk to you specifically today about the idea of retirement. Do you think
about it?
o [IF YES] Can you share some of your recent thinking?
o [IF YES] What does retirement mean to you?
o [IF NO] What does retirement mean to you?
2b. Have you created a retirement plan?
o [IF YES] Please tell us in general, non-specific terms about what you've included
in it.
o [IF YES] What do you expect will happen if you stick to your plan?
o [IF NO] What are your feelings about retirement planning?
2c. Have you thought about when to claim Social Security retirement benefits?
o [IF YES] How did you come to that decision?
o [IF YES] Did you visit any websites to help you make that decision?
o [IF NO] What are your feelings about the idea of claiming Social Security
retirement benefits?
2d. When did you last check with the Social Security Administration about the status of your
future retirement benefits?

Item 3: Interaction with the Claiming Social Secuirty tool

Starting Page: Claiming Social Security main web page
Goal: To evaluate the overall usability, learnability, and usefulness of the Claiming Social
Security tool on desktop.
3a. Please use and interact with this web page until you consider your experience on the
page complete.
(Moderator: Ask participant to think aloud when you notice them reading the content around
the graph and reading the answers to the multiple-choice questions.)
[Correct Path (with some ability to reverse direction and modify choices): Enter date of
birth  Enter highes annual salary  Get estimates  Interact with graph (several
clicks/taps possible)  Answer “Are you married?”  Answer “Do you plan to continue
working in your 60s?” Answer “Will your expenses decrease after you retire?” 
Answer “Will you have sources of income in addition to Social Security benefits?” 
Claiming Social Security Usability Test (Round 2), Dev. September 2015  3

Answer “Do you expect to live a long time?”  Enter age planned to claim Social Security
benefits  Answer if page was helpful in thinking about claiming age decision]

Item 4: Collection of usability and marketing metrics

Goal: To numerically evaluate the usability, learnability, and of the site

System Usability Scale (SUS)

Please circle the numbers that most appropriately reflect your impressions about using this website.
1. I think that I would like to use the Claiming Social Security website frequently.

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

2. I found the Claiming Social Security website unnecessarily complex:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

3. I thought the Claiming Social Security website was easy to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

4. I think that I would need the support of a technical person to be able to use the Claiming Social
Security website:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

5. I found the various functions in the Claiming Social Security website were well integrated with
each other:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

6. I thought there was too much inconsistency in the Claiming Social Security website:

Claiming Social Security Usability Test (Round 2), Dev. September 2015  4

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

7. I would imagine that most people would learn to use the Claiming Social Security website very
quickly:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

8. I found the Claiming Social Security website very cumbersome to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

9. I felt very confident using the Claiming Social Security website:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

10. I needed to learn a lot of things before I could get going with the Claiming Social Security website:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

Net Promoter Score (NPS)

Please circle the numbers that most appropriately reflect your impressions about using this website.
On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this website to
a friend, colleague, or family member as a place to consult about retirement? Why?

1

2

3

4

5

6

7

8

9

10

Claiming Social Security Usability Test (Round 2), Dev. September 2015  5

Item 5: First impressions of Claiming Social Security web page

Starting Page: Claiming Social Security main web page
Goal: To evaluate consumers’ first impressions.
5a. What comes to mind when you are looking at this page?
5b. What is the first thing that caught your eye?
5c. What can you do here on this page?
5d. What is the purpose of this site?
5e. Explain for me what YOUR full retirement age is, the number and the definition.
5ei. Where on the page does it tell you this information?
(Note which of the many references he/she credits for this information.)
5f. How much confidence do you have in the benefit estimate given on this website?

Item 6: Comprehension

Starting page: None. (Please turn off the screen that the participant is viewing, or retrieve
the device from them.)
Goal: To evaluate understanding of pivotal content intended to inform consumers of lessknown details about Social Security claiming
Please answer the following questions to the best of your ability. Please also tell us where
you recall learning the information or answer, e.g, a friend, this website, a commercial.
6a. Fill in the blank. Your Social Security Retirement benefits are based on your highest ___
years of earnings.
6b. If a friend who is married asked you for your advice about when he/she should claim
Social Security Retirement benefits, what would you tell him/her?
6c. At what age is your maximum benefit claiming age?
6d. At what age is your earliest benefit claiming age?
6e. If you live to age 85, which of the following will bring you a higher dollar benefit over your
lifetime?
A. Claiming at 62 so you maximize the number of checks you receive.
B. Claiming at full benefit claiming age because you’ll receive your full benefit for
each check.
C. The age you claim matters less by the time you get to age 85.
D. I don’t know.
6f. True or False. When you claim Social Security Retirement benefits, you will need to stop
working.
6g. Which of the following is true:
A. It’s possible to claim social security retirement benefits only at 62, 67, or 70.
B. It’s possible to claim social security retirement benefits at any point between 62
and 70.
C. It’s possible to claim social security retirement benefits only at my full retirement
benefit age.
D. I’m not sure.

Claiming Social Security Usability Test (Round 2), Dev. September 2015  6

Item 7: Follow-up questions to interaction with the tool on desktop

Goal: To evaluate the overall usability, learnability, and usefulness of the Claiming Social
Security tool on desktop.
7a. Does this site match your expectations for a website about Social Security and
retirement? Why or why not?
Goal: To evaluate the usability, learnability, and usefulness of the data entry required.
7b. Would you walk me through your thought process when you entered your data on this
page?
7c. Did it match your expectations for what would be needed to calculate Social Security
retirement benefits? Why?
7d. Was it easy to enter the data? How could the designers improve this section of the site?
7e. Were there any concerns about your privacy or the security of the data entered?
7f. Was there anything missing that you would expect to find?
Goal: To evaluate the usability, learnability, and usefulness of the graphs used to visualize
Social Security retirement benefits.
7g. Would you walk me through your thought process when you explored the graphs on this
website?
7h. What information is the graph portraying?
7i. Was it easy to use the graph? How could the designers improve this section of the site?
7j. Was there anything missing that you would expect to find?
Goal: To evaluate the usability, learnability, and usefulness of the visual graphics used and
textual content used.
7k. Would you tell me a bit about what the visuals on this website mean to you?
7l. Would you walk me through your thought process about the textual information and tips
found on this page?
7m. Was it easy to understand the text given?
7n. How could the designers improve the textual information on the site?
7o. Was there anything missing in the text that you would expect to find?

Item 8: Interaction with mobile device (smartphone or tablet)

Goal: To evaluate the overall usability and usefulness of the Claiming Social Security tool on
mobile.
8a. Please use and interact with this web page until you consider your experience on the
page complete.
[Correct Path (with some ability to reverse direction and modify choices): Enter date of
birth  Enter highest annual salary  Get estimates  Interact with graph (several
clicks/taps possible)  Answer “Are you married?”  Answer “Do you plan to continue
working in your 60s?” Answer “Will your expenses decrease after you retire?” 
Claiming Social Security Usability Test (Round 2), Dev. September 2015  7

Answer “Will you have sources of income in addition to Social Security benefits?” 
Answer “Do you expect to live a long time?”  Enter age planned to claim Social Security
benefits  Answer if page was helpful in thinking about claiming age decision]
8b. Would you walk me through your thought process when you were entering your data on
this page?
8c. Was it easy to enter the data?
8d. Was it easy to use the graph?
8e. Was it easy to understand the text given?

Section 3: Debriefing Questions
(NOTE: Some are used as follow-ups to actions during session and to SUS or NPS responses;
participants do not receive all debriefing questions.)
1. What stands out to you about your experience using the site? Tell me about your overall
experience using this website. What worked well; what did not work well? [Probe further
as necessary]
2. How would you describe this site to a friend?
3. Did you learn any new information on the website that you were unaware of before, or
that was a surprise to you?
4. Earlier, we asked you if you had thought about when to claim Social Security. After going
through this website, is that answer still the same? Why or why not?
5. Was there anything missing from the website shown to you that you expected to see, or
expected to have access to, that you haven’t called out already?
Extra questions
6. Where would you go or what would you do after using this site?
7. What can this organization help you with? What can’t they help you with?

Claiming Social Security Usability Test (Round 2), Dev. September 2015  8

8. [Follow up to NPS] What could be changed to increase the likelihood that you would
recommend this site to a friend?
9. [Follow up to SUS1] What could be improved on the site that would increase the
likelihood that you would use it in the future?
10. What did you like most about the site?
11. What did you dislike the most about the site?
12. If you could change one thing on the site, what would it be?
13. Do you have any other feedback/input that we did not cover?

Claiming Social Security Usability Test (Round 2), Dev. September 2015  9