IRS2Go Validation Usability Study Plan
Page 1 of 22
Table of Contents
Executive Summary....................................................................................................................................... 2
RESEARCH PLAN ............................................................................................................................................ 2
Background and Study Objectives/Goals ............................................................................................... 2
Investigators or Sponsors ....................................................................................................................... 2
Study Participants................................................................................................................................... 2
Study Methodology ................................................................................................................................ 3
Study Schedule ....................................................................................................................................... 5
RESEARCH REPORT........................................................................................................................................ 6
Execution ................................................................................................................................................ 6
Plan Deviations ....................................................................................................................................... 6
Results .................................................................................................................................................... 6
Discussion ............................................................................................................................................... 7
Appendix A – Consent Form ......................................................................................................................... 8
Appendix B – Introduction Script .................................................................................................................. 9
Appendix C – Data Collection Sheets .......................................................................................................... 11
Appendix D – Study Materials..................................................................................................................... 17
Appendix E – Recruitment Email................................................................................................................. 18
Appendix F – Reminder Email (In Person Study) ........................................................................................ 19
Appendix G – Confirmation Email (In Person Study) .................................................................................. 20
Appendix H – Usability/User Research Brief............................................................................................... 21
IRS2Go Validation Usability Study Plan
Page 2 of 22
Executive Summary
This section will be completed after the study is conducted.
RESEARCH PLAN
This section of the document, along with the appendices, contains the study plan content.
Background and Study Objectives/Goals
This is a validation usability study to determine whether improvements and added features to the
IRS2Go mobile application served the participant’s needs and present any usability issues. The
specific objectives for this study are as follows:
1. Measure new features including pay bank account, pay by credit card and find help nearby.
2. Assess usability of the app menu.
3. Test the app on various operating systems to assure that the interface does not conflict with
native controls and functionality.
4. Observe and record participant behaviors while completing tasks
5. Obtain comments and feedback about the product, which can be fed into the design of future
versions of the product.
Investigators or Sponsors
IRS UX Team Lead: Andrew Hughey ([email protected])
IRS UX Team Member: William Woodard ([email protected])
IRS UX Team Support: Michael Weiss ([email protected])
Study Recruitment, Facilitation, and Data Analysis: Daniel Vickers ([email protected])
Study Participants
General description - Individuals who are taxpayers will be recruited for participation. The
recruited participants will represent a mix of demographics including age, gender, amount of
internet usage, and mobile device platforms and operating system experience.
Geographical location – Testing will take place in Miami, Florida.
Sample size - A total of 16 participants will be recruited with a goal of segmenting the sampling
into 8 iOS users and 8 Android users. Due to limitations presented with recruiting this
segmenting may not be exact. The minimum sample size is 10.
Inclusion Criteria - Participants must be U.S. taxpayers. Participants can be government
employees, but must participate in the study on their own time to in order to receive the
incentive. Federal government participants cannot be paid incentives.
Compensation - Participants will be compensated with a $40 Visa gift card for their
participation.
IRS2Go Validation Usability Study Plan
Page 3 of 22
Study Methodology
Design
Method - Testing will performed using an in-person moderated study method and will consist
participants using the prototype and live version of IRS.gov to complete tasks. Participants will
also be asked questions to learn more about their thoughts on the products.
Measures - Quantitative data will be gathered pertaining to time on task, number of clicks to
complete each task, and task success rate. Qualitative data will be collected through contextual
inquiry throughout the test, as well as a 7 point ease of use scale scored by the participant after
each task.
Equipment – Testing will take place on either an Apple iPhone or an Android smartphone based
upon the participant’s own experience. A laptop will be set up with Morae Recorder software
running. The computer’s onboard camera will be used to record the participant’s reactions, while
a lamp cam will be placed above the device to record activity on the device.
Experimenter - The facilitator of the testing will be Daniel Vickers of Flux Forward.
Tasks
Participants will use the IRS2Go (v5.2) mobile application to complete four tasks:
1.
2.
3.
4.
Find and use the “Where’s My Refund” feature
Make a payment using a bank account
Make a payment using a credit card
Find a local IRS office to contact for help
During each task it should be noted how participants interact with the software (i.e., what menus
are used, what options/information on the screen is noticed, etc.).
Participants will complete each task using test data, so that none of their personal information
will be used during the study session. The test data will be added in a future version of this study
plan.
Task Order - Tasks will be completed in procedural order.
Task Success Criteria and Usability Metrics Task Success/Failure
Each task will require, or request, that the participant obtains or inputs specific data that would
be used in course of a typical task. The scenario is completed when the participant indicates the
scenario's goal has been obtained (whether successfully or unsuccessfully) or the participant
requests and receives sufficient guidance as to warrant scoring the scenario as a critical error.
IRS2Go Validation Usability Study Plan
Page 4 of 22
Critical Errors
Critical errors are deviations at completion from the targets of the task. Obtaining or otherwise
reporting of the wrong data value due to participant workflow is a critical error. Participants may
or may not be aware that the task goal is incorrect or incomplete.
Independent completion of the scenario is a universal goal; help obtained from the other usability
test roles is cause to score the scenario a critical error. Critical errors can also be assigned when
the participant initiates an action that will result in the goal state becoming unobtainable. In
general, critical errors are unresolved errors during the process of completing the task or errors
that produce an incorrect outcome.
Non-critical Errors
Non-critical errors are errors that are recovered from by the participant or, if not detected, do not
result in processing problems or unexpected results. Although non-critical errors can be
undetected by the participant, when they are detected they are generally frustrating to the
participant.
These errors may be procedural, in which the participant does not complete a scenario in the
most optimal means (e.g., excessive steps and keystrokes). These errors may also be errors of
confusion (ex., initially selecting the wrong function, using a user-interface control incorrectly
such as attempting to edit an un-editable field).
Noncritical errors can always be recovered from during the process of completing the scenario.
Exploratory behavior, such as opening the wrong menu while searching for a function, selecting
help text links, selecting the “Contact Us” links, and footer links will be coded as a non-critical
error.
Subjective Evaluations
Subjective evaluations regarding ease of use and satisfaction will be collected via questionnaires,
and during debriefing at the conclusion of the session. The questionnaires will utilize free-form
responses and rating scales. Combined with the interview/debriefing session, these data are used
to assess attitudes of the participants.
Task Completion Time (time on task)
The time taken to complete the task (including each page the participant uses during a task), not
including subjective evaluation durations, will be recorded. This data will be captured using time
stamps from recorded session videos.
Task Error Severity - To prioritize recommendations, a method of problem severity classification
will be used in the analysis of the data collected during evaluation activities. The approach treats
problem severity as a combination of two factors - the impact of the problem and the frequency
of participants experiencing the problem during the evaluation.
Impact
IRS2Go Validation Usability Study Plan
Page 5 of 22
Impact is the evaluation of the severity of an error and how much it affects the participant’s
ability to complete a task. There are three levels of impact:
High - prevents the participant from completing the task (critical error)
Moderate - causes participant difficulty but the task can be completed (non-critical error)
Low - minor problems that do not significantly affect the task completion (non-critical error)
Frequency
Frequency is the evaluation of how often an error occurs. It is measured by the percentage of
participants who experience the error.
High: 30% or more of the participants experience the problem
Moderate: 11% - 29% of participants experience the problem
Low: 10% or fewer of the participants experience the problem
[For studies with less than ten participants in a group, the percentages may need to be adjusted.
For example, for a study with 8 participants the low frequency should be 12.5% (1/8 = .1250]
Procedure
The study will begin with the participant executing permission to record the session. The
facilitator will begin the recording and then ask the introductory and demographic questions
included in the test plan. The participant will then be asked to perform a series of tasks. After
each task they will be asked to score the ease of use on a scale of 1-7 with 1 being most difficult
and 7 being easiest.
Study Schedule
The objectives addressed by this study are needed to validate the development of additional
features within IRS2Go v5.2 and develop recommendations for future releases. Completion of
data collection, analysis, and reporting is planned for September 4th, 2015.
IRS2Go Validation Usability Study Plan
Page 6 of 22
RESEARCH REPORT
For Plan release: Remove all information below and provide the following statement: “This section will
be completed after the study is conducted. This section will provide the evaluation of the usability metrics
collected and provide formal recommendations.” For Report release: Use the statement below.
This section of the document contains the study report content. Because it was created after the
study was completed, it is written in the past tense. Transcribed raw data can be found in
, and all audio/video files can be in found .
Execution
For Plan release: Remove section. For Report release: Provide a paragraph explaining when the study
occurred, who conducted the session, where the sessions were conducted, and who performed the
analysis.
The study was conducted between and by
in .
Plan Deviations
For Plan release: Remove section. For Report release: Provide a paragraph that indicates whether the
study was conducted as planned or whether deviations in objectives, participants, methods, deliverables,
or data collection tools occurred. If deviations were made, list all deviations and provide rationale for
why the changes occurred.
Results
For Plan release: remove section. For Report release: Start with a presentation of demographic data for
the participant sample. Then describe the outcomes for each task. When describing the outcome for each
task state the exact number of participants that performed each reported action. Subsections, headings,
may be added, removed or retitled as needed.
Participant Background Information
IRS2Go Validation Usability Study Plan
Page 7 of 22
Task 1
TASK DESCRIPTION
SUCCESSFUL
WITH NO
ERRORS
SUCCESSFUL
WITH NONCRITICAL
ERRORS
FAILURE/CRITI
CAL ERROR
AVERAGE
TIME ON
TASK
(SECONDS)
xxx% (xxx)
xxx% (xxx)
xxx% (xxx)
xx
List the task details/description
List task events –
Post-Session Questions
Discussion
For Plan release: remove section. For Report release: Provide a discussion of the results with expert
interpretation of why particular outcomes occurred (e.g., design issues, study method, limitations of
prototype, product issues). This is where you qualify, justify, and/or interpret outcomes. Include
information about how reported deviations affected the study results.
RECOMMENDATIONS
For Plan release: remove section. For Report release, complete the table below. Recommendations
should be solution free statements where possible so the implementation team has room to work out a
feasible approach. List the priority of each action. Consider the severity of the error(s) that led to the
recommendation when determining a recommended action’s priority.
NO.
ACTION
FROM
PRIORIT
Y
1
Design product to user’s need to accurately understand how to make a
tax payment
Task 1
High
2
Add instructional information to help users find the appropriate form.
Task 1, 2, and 3
High
CONCLUSION
For Plan release: remove section. For Report release: This section should provide a clear statement of
whether or not your objectives were met. If objectives were not met, you should discuss whether
redesign and/or retesting is needed. Provide any recommended actions or next steps for the product
development team.
IRS2Go Validation Usability Study Plan
Page 8 of 22
Appendix A – Consent Form
I agree to participate in the study conducted and recorded by the Internal Revenue Service (IRS).
I understand that the study will take approximately .
I understand and consent to the use and release of the audio, video, and/or digital recording by
the IRS. I understand that the information and recording is for research purposes only and that
my name and image will not be used for any other purpose. I relinquish any rights to the
recording and understand the recording may be copied and used by IRS without further
permission.
I understand that participation in this usability study is voluntary and I agree to immediately raise
any concerns or areas of discomfort during the session with the study administrator.
The Paperwork Reduction Act requires that the IRS display an OMB control number on all
public information requests. The OMB Control Number for this study is 1545-2256. Also, if you
have any comments regarding the time estimates associated with this study or suggestions on
making this process simpler, please write to:
Internal Revenue Service
Tax Products Coordinating Committee
SE:W:CAR:MP:T:T:SP
1111 Constitution Ave. NW
Washington, DC 20224
Please sign below to indicate that you have read and you understand the information on this form
and that any questions you might have about the session have been answered.
Date
Please print your name
Please sign your name
Thank you! We appreciate your participation.
IRS2Go Validation Usability Study Plan
Page 9 of 22
Appendix B – Introduction Script
Welcome and Purpose
Thank you so much for coming in today. I want to give you a little information about what you will be
looking at and give you time to ask any questions you might have before we get started.
Today we will be asking you to complete a set of tasks. These tasks were derived from actual business
scenarios. Our goal is to use the results of this study to provide us with insights that will help us improve
the design of our online services. The overall goal of our testing is to provide better services to the
American public.
Test Facilitator’s Role
Our role is to record your reactions and comments about the tasks we will present. My colleague(s) will
be helping me to take notes and observe your interactions and responses.
We will not be able to offer any suggestions or hints, but from time to time, we may ask you to clarify
what you have said or ask you for information on what you were looking for or what you expect to have
happened.
Please remember, as you perform the tasks, there are no right or wrong responses. You are helping us
test our programs and designs. If you have a problem it is our problem and we appreciate you letting us
know.
Test Participant’s Role
• Your role today is to perform a series of tasks that we give you.
• There is no right or wrong answer. We are not testing you, we are testing our services.
• If you have any questions, comments or areas of confusion while performing a task, please let us
know. Any problem you have points to a problem we have with our services.
• As you perform a task, please do so as you would at home or your office. I ask that you work through
the tasks based on what you see on screen. If you reach a point where you are not sure where to go or
how to proceed, please feel free to ask one of us.
• We will be recording this session. We will record your expressions, your voice and what you see on the
screen. Your identity will not be associated or reported with any data or findings from this evaluation.
• I may ask questions as you work and then we will finish up with questions at the end.
• Remember: Your decision to participate is completely voluntary. You are free to stop participating at
any time.
Do you have any questions before we begin?
Okay, one last thing before we continue. I need to have you to read the following:
Paperwork Reduction Act Statement Notice
The Paperwork Reduction Act requires that the IRS display an OMB control number on all public
information requests. The OMB Control Number for this study is 1545-2256. Also, if you have
any comments regarding the time estimates associated with this study or suggestions on
making this process simpler, please write to:
IRS2Go Validation Usability Study Plan
Internal Revenue Service
Tax Products Coordinating Committee
SE:W:CAR:MP:T:T:SP
1111 Constitution Ave. NW
Washington, DC 20224
Page 10 of 22
IRS2Go Validation Usability Study Plan
Appendix C – Data Collection Sheets
1. Which option best describes your race?
Asian
Black or African American
Hispanic or Latino
Native American or American Indian
Native Hawaiian or Other Pacific Islander
White
2. What is your gender?
Female
Male
3. What category describes your current age?
18 to 24 years
25 to 34 years
35 to 44 years
45 to 54 years
55 to 64 years
65 to 74 years
75 to 84 years
85 years and over
4. What is the highest level of education you have completed?
Less than 9th grade
9th grade to 12th grade, no diploma
High school graduate (or GED)
Some technical or vocational school
Technical or vocational school graduate
Some college, no degree
Associate Degree
Bachelor’s Degree
Master’s Degree
Post-Master’s Degree
5. What category best describes your annual household income?
Less than $15,000
$15,000 but less than $25,000
$25,000 but less than $35,000
$35,000 but less than $50,000
$50,000 but less than $75,000
$75,000 but less than $100,000
$100,000 or more
6. What is the primary language spoken at home? (Please select one)
English
Spanish
Chinese
Vietnamese
Korean
Russian
Other (please specify):__________
7. Have you ever made a payment online to the IRS?
Page 11 of 22
IRS2Go Validation Usability Study Plan
Yes
No
8. If yes, what methods have you use?
Pay by card
Electronic Funds Withdrawal
Same Day Wire
Check or Money Order
Other
9. Have you ever used the IRS2Go mobile app?
Yes
No
Page 12 of 22
IRS2Go Validation Usability Study Plan
Page 13 of 22
Task 1 – Where’s My Refund
You have completed and filed your taxes and are due a refund. You would like to find the status
of your refund. Using the app, find the status of your refund. Please use the tax information to
review your refund information.
SUCCESS
(CIRCLE ONE)
PATHWAY(S)
0
Failure/Critical
Error
TBD
1
Completed with
non-critical error
ERROR INFORMATION
Possible critical error:
TBD
Possible non-critical error:
TBD
2
Completed with
no errors
NOTES/OBSERVATIONS
(Note: why was the participant successful or not successful, e.g., wrong pathways, confusing page
layout, navigation issues, terminology, record number of times an incorrect/unexpected button or link is
pressed)
1. Overall, how difficult or easy was it to complete that process?
Very
Difficult
Very
Easy
1
2
3
4
5
6
7
IRS2Go Validation Usability Study Plan
Page 14 of 22
Task 2 – Make a payment from bank account
You have an outstanding balance to the IRS and have made a payment arrangement. You need to
schedule your first payment to be withdrawn from your bank account. Using the app show me how you
would go about doing this. Please use the provided bank account information when setting up your bank
payments.
SUCCESS
(CIRCLE ONE)
PATHWAY(S)
TBD
0
Failure/Critical
Error
ERROR INFORMATION
Possible non-critical error:
TBD
1
Completed with
non-critical error
2
Completed with
no errors
NOTES/OBSERVATIONS
(Note: why was the participant successful or not successful, e.g., wrong pathways, confusing page
layout, navigation issues, terminology, record number of times an incorrect/unexpected button or link is
pressed)
1. Overall, how difficult or easy was it to complete that process?
Very
Easy
Very Difficult
1
2
3
4
5
6
7
IRS2Go Validation Usability Study Plan
Page 15 of 22
Task 3 – Make a payment by credit card
You’ve decided that you would rather make your payment by credit card. Using the provided credit card
payment information, how would you go about doing this?
SUCCESS
(CIRCLE ONE)
PATHWAY(S)
TBD
0
Failure/Critical
Error
ERROR INFORMATION
Possible non-critical error:
TBD
1
Completed with
non-critical error
2
Completed with
no errors
NOTES/OBSERVATIONS
(Note: why was the participant successful or not successful, e.g., wrong pathways, confusing page
layout, navigation issues, terminology, record number of times an incorrect/unexpected button or link is
pressed)
1. Overall, how difficult or easy was it to complete that process?
Very
Easy
Very Difficult
1
2
3
4
5
6
7
IRS2Go Validation Usability Study Plan
Page 16 of 22
Task 4 – Find local help
You have questions about your taxes and would like to talk to someone in person. Using the app find the
closest place to you to accomplish this. Please use the provided personal information when determining
the closest IRS office location.
SUCCESS
(CIRCLE ONE)
PATHWAY(S)
TBD
0
Failure/Critical
Error
ERROR INFORMATION
Possible non-critical error:
TBD
1
Completed with
non-critical error
2
Completed with
no errors
NOTES/OBSERVATIONS
(Note: why was the participant successful or not successful, e.g., wrong pathways, confusing page
layout, navigation issues, terminology, record number of times an incorrect/unexpected button or link is
pressed)
1. Overall, how difficult or easy was it to complete that process?
Very
Easy
Very Difficult
1
2
3
4
5
6
7
IRS2Go Validation Usability Study Plan
Appendix D – Study Materials
TBD
Page 17 of 22
IRS2Go Validation Usability Study Plan
Page 18 of 22
Appendix E – Recruitment Email
SUBJECT LINE: Internet Study: Let the IRS pay to serve you better!
Hello,
My name is Daniel Vickers. I do research for the IRS and we plan to conduct a usability study. The goal
of this study is to improve our services by asking people to try out our mobile app and give us feedback.
If you qualify, you will receive $40 to particípate.
What will you do in the usability study?
You will be asked to do several short tasks to test a mobile software application. We may also ask you
questions about your thoughts and perceptions of the website.
How long is a session?
The testing will take about 30 minutes.
When and where?
The study will be held August 16th and 17th 2015. You will be asked to participate at 350 Lincoln Rd.,
Miami Beach.
Interested in participating?
Please reply to this email with your contact information. When we talk I will ask you a short list of
questions to help decide if you qualify for the study.
If you have questions, please contact me at [email protected].
Thank you for interest,
Daniel Vickers
Flux Forward
IRS2Go Validation Usability Study Plan
Page 19 of 22
Appendix F – Reminder Email (In Person Study)
SUBJECT LINE: Reminder: Website study tomorrow
Dear :
Thanks again for agreeing to help test the IRS2Go. We are looking forward to talking with you.
You are scheduled to participate as follows:
DATE:
TIME: