OMB Package
A Customer Satisfaction Study for Taxpayers and Tax Practitioners using
IRS.gov Frequently Asked Questions (FAQs) – Usability Study
Introduction
Customer On-Line Decision Support (COLDS) Release 1 will enhance the Wage and Investment’s (W&I) Frequently Asked Questions (FAQs) content and functionalities currently available on IRS.gov web site. COLDS will modify the existing Content Management Application (CMA), Search Engine, and WebTrends components to improve FAQs. The solution will enable taxpayers to conduct key word and natural language queries on the Frequently Asked Questions (FAQs). Taxpayers will have the opportunity to provide feedback on the usefulness of the content, thereby enabling FAQ content authors to improve content effectiveness. COLDS will also provide FAQ content authors the capability to easily update the FAQ content to the web site.
Release 1 will be implemented by the 2009 Filing Season.
Background
In response to a Congressional directive, the IRS conducted a comprehensive review of its current portfolio of taxpayer services to create a Taxpayer Assistance Blueprint (TAB). The TAB team identified the need to improve website self-service applications. The W&I Strategy and Program Plan addressed the need to provide solutions to ensure that Taxpayers, Practitioners and other IRS partners have access to accurate and timely responses to tax law issues and inquiries at all hours during the day. This included the need to increase availability of self-assist applications through multiple channels, including the Internet.
COLDS will provide an efficient electronic alternative on IRS.gov for taxpayers seeking assistance with their federal tax questions who would otherwise require assistance from a Call Site or a Tax Assistance Center (TAC). COLDS will also support the TAB initiatives as well as the W&I Strategy and Program plan.
An improved on-line FAQ solution for routine questions will mitigate the burden on telephone assistance centers during peak traffic times, freeing the assistors to handle more complex questions. This project will move the IRS customers away from the more expensive channels to the less expensive internet customer self-service channel
Objectives of Research
The purpose of this test is to determine if the W&I’s Frequently Asked Questions portion of the web site is logical and easy to understand.
The Usability testing is intended to gather user data on the overall content, navigation, and user interface design of the site.
We will determine the user’s ability to navigate to the FAQs on the web site.
We will determine the user’s ability to use the search features to search for the appropriate FAQs.
We will determine whether the listing options are easily understood by users.
We will determine the user’s ability to understand the categorization of the topics on the site.
Methodology
The test is designed to take approximately 60 minutes for each participant to complete. Test participants will consist of individuals and tax practitioners that prepare or have knowledge about individual tax filing. Testing will occur in Ogden, Utah at the IRS Usability Lab. Testing will occur in one lab. Consumer Opinion Services will recruit the test participants using a series of screening questions approved by the COLDS R1 team.
Sample Design
The goal for each user group (individuals and tax practitioners) is between 6 and 8 participants for a total of 12 – 16 participants. The test will have a user complete a variety of tasks using the IRS.gov web site. The participant will be asked to talk-aloud as they complete the tasks.
Testing will be based on scenarios created by Accenture and the IRS. Scenarios are short statements that include a task to be completed by using the FAQs portion of IRS.gov. The scenarios will be unique for each user group (i.e. individuals and tax practitioners).
The moderator will complete an introduction, and provide an overview of the test process and ask the participant to sign a consent form. The form will explain what the participant is going to be doing and how the IRS plans to use the information gathered from testing.
No personally identifiable information will be shared outside of the test environment. The IRS team will destroy all background information upon completion of the test.
After the introduction each participant will work each scenario individually on a computer. Participants will be required to complete each task without assistance from the moderator. They will be asked to “think aloud”, so that the usability team can capture as much data as possible.
When the participant has completed the scenarios there will be a comprehension question about the FAQ page in general. After that there will be a debriefing session that will consist of open discussion and an exit questionnaire. We are rewarding $75 to each participant.
Participants will use a PC running Windows XP with a 17-inch monitor. The screen resolution of the monitor will be set to 800 x 600 with Windows display properties set to “Small Fonts”. Participants will access the product by using Microsoft Internet Explorer 6.0.
Data
Collection Date
Research will be conducted the week of August 18 – 22, 2008. Testing will occur in Ogden, Utah at the IRS Usability Lab.
Data to be collected
This test will gather usability information regarding the FAQs portion of the IRS.gov web site. The testing team will gather information metrics for the following
Time spent to complete a task
Navigation paths used
Search vs. navigation
User satisfaction ratings (Post Test Questionnaire)
Number of times user required help
Task success rate
Error recovery
The participant will be monitored by having a test facilitator get them started on the test. In addition, they will be monitored by COLDS R1 usability team members through a two way mirror and video. Testing sessions are expected to last approximately 60 minutes per participant.
How data will be used
Data will be used to evaluate taxpayers’ satisfaction and identify areas for improvement for the FAQs redesign on the IRS.gov web site. All information collected is strictly for research, and will not be used to personally identify participants nor will it be shared for commercial purposes.
How data will be analyzed
Analysis of the usability testing data will be conducted by Wage & Investment (W&I), Customer Account Services (CAS), and the IRS.gov contractor, Accenture. Observations are going to be used to asses the performance (design) of the FAQs pages.
Who is conducting research?
The COLDS R1 test team includes one IRS.gov usability specialist, a management analyst assigned to the Public Portal Branch of ETARC, two internal usability specialists from the IRS – Ogden Usability Lab, Cindy Williams (COLDS R1 project manager), and a contractor from Accenture.
Observer-team member |
Cindy Williams |
|
|
|
|
Role |
Name(s) |
Business Operating Division |
Organization |
Job Title / Role |
Contact Information |
Planner and coordinator |
Shawn Tafoya |
MITS |
AD |
Usability Administrator |
(801)620-4060 |
Participant recruiter |
Consumer Opinion Services |
|
|
|
|
Test facilitator |
Jana Roberts |
MITS |
AD |
Usability Administrator |
801-620-4283 |
Test data logger |
Jana Roberts |
MITS |
AD |
Usability Administrator |
801-620-4283 |
Test team lead |
Accenture
Beth Krappweis |
W&I |
ETARC: PPB |
IT Specialist |
202-283-4844 |
Observer-stakeholder |
Cindy Williams |
W&I |
CAS |
Project Manager |
678-530-5238 |
Observer-team member |
Corliss Brooks |
W&I |
ETARC:PPB |
PPB COLDS R1 Lead |
202-283-0833 |
Observer-team member |
Accenture |
|
|
|
|
The following roles are required before and during the period over which usability testing takes place:
Planner and coordinator – Usability Administrator ensures that everything is in place to allow the usability test sessions to take place as planned and without problems.
Participant recruiter – recruits participants and schedules them for available time slots; provides test facilitator with honorarium checks to give to each participant at the end of their test session.
The following roles are required in order to conduct a usability test session:
Test facilitator –Usability Consultant oversees the test itself, introduces and explains the test session to the participant, observes participant’s actions and obtains participant feedback at completion of test session, conducts issues assessment session. The facilitator is also available to answer questions and ensures that the observer stakeholders stay on task and have a productive and successful experience.
Test data logger – creates a written record of the participant’s actions and comments as they use the site using data logging software, a word processor, or a spreadsheet.
Test Team Lead – responsible for coordination meetings and set up for testing.
Observer-team member – observes test to identify functionality and performance, compares notes with other team members, and actively participates in Issues Assessment at end of each test session.
Cost of Study
The one-time cost for this usability test is $2,000.00.
Location
Testing will occur in Ogden, Utah, at the IRS’ Ogden Usability Lab.
Stipend
A $75.00 per participant stipend will be paid to each participant. This is necessary in order to recruit a sufficient number of study participants.
Recruitment Efforts
Consumer Opinion Services, an external market research firm, will recruit 12 participants from two user groups: Individuals and Tax Professionals. Participants will be recruited the week prior to usability testing. With the usability study being completed by August 22, 2008.
Participant Criteria
External Customers |
Target # |
Individuals – Individual tax payers (6 participants). Students (newer / first time tax payers), homemakers, retired citizens, average taxpayers (employed individuals). |
6 |
Tax Practitioners – (6 participants). Tax practitioners that prepare taxes for individuals, use the individual’s portion of the website on a regular basis. |
6 |
General knowledge of the internet with frequent use of the internet
General knowledge of tax filing and processing
Knowledge of / responsible for individual tax filing
Burden Hours
Individual Participants
Total Screened: 20 @ 10 minute screening interview = 3.2 hours
Total Recruited: 6 people (30% of screened individuals qualify)
Travel Time estimated 20 minutes per participant to travel to testing facility and return = 2 hours
Total testing time averaged at 1 hour/participant = 6 hours
Total
estimated burden is 11.2 hours
Tax Professional Participants
Total Screened: 20 @ 10 minute screening interview = 3.2 hours
Total Recruited: 6 people (30% of screened individuals qualify)
Travel time estimated 20 minutes per participant to travel to testing facility and return = 2 hours
Total testing time averaged at 1 hour/participant = 6 hours
Total estimated burden is 11.2 hours
Efforts to not duplicate research
Usability testing has been conducted on the FAQs portion of the IRS.gov web site previously. Some tasks may be repeated to gather metrics on improvement.
Privacy, Security, Disclosures, Confidentiality
The IRS will ensure compliance with the Taxpayers Bill of Rights II. All participants will be treated fairly and appropriately.
Confidentially will be safeguarded. During the testing sessions, participants will be identified to IRS personal by their first names only. In addition, no participant names will be mentioned in the usability study final report. Participants will be advised that observers from the IRS may be observing the session and via video monitoring equipment and that their comments may be recorded to ensure accurate data capture.
Attachments
Telephone/ Participant Screeners Questionnaire
Recruiting Service Cover Letter
Moderator Checklist
Consent Form
Scenarios
Exit Questionnaires
Confirmation Phone Call Script
Usability Test Plan
| File Type | application/msword |
| File Title | OMB Package |
| Author | 401fb |
| Last Modified By | qhrfb |
| File Modified | 2008-07-11 |
| File Created | 2008-07-11 |