User testing for soliciting feedback on the Bureau's Planning for retirement- Before you claim website (both in English and Spanish)

MODERATOR’S GUIDE

Claiming Social Security
Usability Test (Round 1)
--Spanish Edition-Data Collection: [insert dates]

Section 1: Introduction
Thank you for participating in this study today. My name is _____, and I am a researcher with
the Fors Marsh Group User Experience Team. Today we will evaluate a website that has
information about claiming Social Security retirement benefits and retirement planning.
Some of the pages you will interact with are still in development, so not all the links will be
clickable and others may have missing elements. If you encounter something isn’t working,
please let me know so I can get us back on track.
I will ask you questions and give you some tasks to complete using the website. Your
comments and feedback will help the design team improve the site. I am a researcher and
did not create the site, so please do not feel like you have to hold back on your thoughts to
be polite to me. I’m interested in both your positive and negative reactions to them.
Difficulties you may run into reflect issues with the design of the sites, not your skills or
abilities. Please keep in mind that I am not testing you or your knowledge. Rather, you are
helping us to see how we can improve this site.
The entire session will last about 60 minutes. Do you have any questions so far?
Let’s cover a couple things before we get started.
• We are making a video recording of the computer screen during this session, but your
name and personal information will not be associated with the recordings.
• I am interested in your thoughts and reactions as we proceed. This is important
because I can see what you are doing, but I don’t know why you are doing it. So I
need your help. While you are working, I would like you to think aloud. I would like you
to:
o Tell me what you are thinking
o Describe the steps that you are taking.
o Tell me why you are doing what you are doing, what you are going to do, and
why.
o Tell me why you clicked on a link and what you expect to happen next.
o Basically, just tell me everything you are thinking as you work.
• There are no right or wrong answers, and your comments and opinions will only be
used in combination with the feedback that we get from other people.
• Please make sure that you have reviewed the Privacy Act Statement has been
provided in hard copy, if you haven’t already.
• Any questions so far?
• Ok, we are ready to begin. I would like you to start by asking you some background
questions.

Claiming Social Security Usability Test (Spanish Round 1), August 2015  1

Section 2: Questions and Tasks
Item 1: Pre-task interview

Goal: To get to know participants and explore their relevant past behavior.
1a. What types of electronic devices do you use to visit websites?
1b. What devices do you usually use to track or manage your financial accounts online?
1c. What is your main online resource when you are looking for financial information?
1d. What other resources might you use to find similar information?
1e. Have you heard of a website called consumerfinance.gov before?
o [IF YES] How or when did you hear about it?
o [IF YES] Have you used consumerfinance.gov before?
o [IF YES] How often on average do you use it?
o [IF YES] For what purposes do you use it?
o [IF NO] What do you think it’s for?
o [IF NO] What kind of information do you think you can find on
consumerfinance.gov?

Item 2: Planning around retirement and Social Security benefit claiming

Goal: To explore participants’ behavior around retirement planning and Social Security
retirement benefits.
2a. We'd like to talk to you specifically today about the idea of retirement. Do you think
about it?
o [IF YES] Please share some of the thoughts you’ve had recently?
o [IF YES] What does retirement mean to you?
o [IF NO] What does retirement mean to you?
2b. Have you created a retirement plan?
o [IF YES] Please tell us in general, non-specific terms about what you've included
in it.
o [IF YES] What are your expectations around using your plan in the future?
o [IF NO] What are your feelings about retirement planning?
2c. Have you thought about when to claim Social Security retirement benefits?
o [IF YES] How did you come to that decision?
o [IF YES] Did you visit any websites to help you make that decision?
o [IF NO] What are your feelings about the idea of claiming Social Security
retirement benefits?

Item 3: First impressions of Claiming Social Security web page
Starting Page: Claiming Social Security main web page
Goal: To evaluate consumers’ first impressions.
3a. What comes to mind when you are looking at this page?
3b. What is the first thing that caught your eye?
3c. What can you do here on this page?
3d. What is the purpose of this site?

Claiming Social Security Usability Test (Spanish Round 1), August 2015  2

3e. What would be the first thing you would interact/select on this site? Why?

Item 4: Learn about claiming Social Security retirement benefits

Starting Page: Claiming Social Security main web page
Goal: To evaluate the overall usability, learnability, and usefulness of the Claiming Social
Security tool.
4a. Please use and interact with this web page until you consider your experience on the
page complete.
[Correct Path (with some ability to reverse direction and modify choices): Enter date of
birth  Enter current annual salary  Get estimates  Interact with graph (several
clicks/taps possible)  Answer “Are you married?”  Answer “Do you plan to continue
working in your 60s?” Answer “Will your expenses decrease after you retire?” 
Answer “Will you have sources of income in addition to Social Security benefits?” 
Answer “Do you expect to live a long time?”  Enter age planned to claim Social Security
benefits  Answer if page was helpful in thinking about claiming age decision]
4b. Does this site match your expectations for a website about Social Security and
retirement? Why or why not?
Goal: To evaluate the usability, learnability, and usefulness of the data entry required.
4c. Would you walk me through your thought process when you were entering your data on
this page?
4d. Did it match your expectations for what would be needed to calculate Social Security
retirement benefits? Why?
4e. Was it easy to enter the data? How could the designers improve this section of the site?
4f. Were there any concerns about your privacy or the security of the data entered?
4g. Was there anything missing that you would expect to find?
Goal: To evaluate the usability, learnability, and usefulness of the graphs used to visualize
Social Security retirement benefits.
4h. Would you walk me through your thought process when you were exploring the graphs
on this website?
4i. What information is the graph portraying?
4j. Was it easy to use the graph? How could the designers improve this section of the site?
4k. Was there anything missing that you would expect to find?
Goal: To evaluate the usability, learnability, and usefulness of the visual graphics used and
textual content used.
4l. Would you tell me a bit about what the visuals on this website mean to you?
4m. Would you walk me through your thought process about the textual information and tips
found on this page?
4n. Was the text easy to understand?
4o. How could the designers improve the textual information on the site?
4p. Was there anything missing in the text that you would expect to find?
Claiming Social Security Usability Test (Spanish Round 1), August 2015  3

Item 5: Collection of usability and marketing metrics

Goal: To numerically evaluate the usability, learnability, and of the site

System Usability Scale (SUS)

Please circle the numbers that most appropriately reflect your impressions about using this website.

1. I think that I would like to use the Claiming Social Security website frequently.

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

2. I found the Claiming Social Security website unnecessarily complex:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

3. I thought the Claiming Social Security website was easy to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

4. I think that I would need the support of a technical person to be able to use the Claiming Social
Security website:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

5. I found the various functions in the Claiming Social Security website were well integrated with
each other:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

6. I thought there was too much inconsistency in the Claiming Social Security website:

Claiming Social Security Usability Test (Spanish Round 1), August 2015  4

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

7. I would imagine that most people would learn to use the Claiming Social Security website very
quickly:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

8. I found the Claiming Social Security website very cumbersome to use:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

9. I felt very confident using the Claiming Social Security website:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

10. I needed to learn a lot of things before I could get going with the Claiming Social Security website:

Strongly
Disagree
1

Disagree

Neutral

Agree

2

3

4

Strongly
Agree
5

Net Promoter Score (NPS)

Please circle the numbers that most appropriately reflect your impressions about using this website.
On a scale of 1 to 10, with 10 being more likely, how likely is it that you would recommend this website to
a friend, colleague, or family member as a place to consult about retirement? Why?

1

2

3

4

5

6

7

8

9

10

Claiming Social Security Usability Test (Spanish Round 1), August 2015  5

Debriefing Questions
(NOTE: Some are used as follow-ups to actions during session and to SUS or NPS responses;
participants do not receive all debriefing questions.)
1. What stands out to you about your experience using the site? Tell me about your overall
experience using this website. What worked well; what did not work well? [Probe further
as necessary]
2. How would you describe this site to a friend?
3. Did you learn any new information on the website that you were unaware of before, or
that was a surprise to you?
4. Earlier, we asked you if you had thought about when to claim Social Security. After going
through this website, is that answer still the same? Why or why not?
5. Was there anything missing from the website shown to you that you expected to see, or
expected to have access to, that you haven’t called out already?
6. If you could name this site, what would you call it?
Extra questions
7. Where would you go or what would you do after using this site?
8. What can this organization help you with? What can’t they help you with?
9. [Follow up to NPS] What could be changed to increase the likelihood that you would
recommend this site to a friend?
10. [Follow up to SUS1] What could be improved on the site that would increase the
likelihood that you would use it in the future?
11. What did you like most about the site?
12. What did you dislike the most about the site?
13. If you could change one thing on the site, what would it be?
14. Do you have any other feedback/input that we did not cover?
Claiming Social Security Usability Test (Spanish Round 1), August 2015  6