SOI-306 - Focus Group Testing of IRS Forms 941X, 6251EX and 8910; SOI-307 - Taxpayers and Tax Practitioners using IRS.gov Frequently Asked Questions (FAQs) – Usability Study

IRS.gov Redesign Usability Study

Individuals Screener

Job #

Hello, I’m ______________from Consumer Opinion Services calling on behalf of the Internal Revenue Service. The IRS wants to hear from you. Have you heard about the website IRS DOT GOV? What we would like to do is ask you a few questions to see if you would be able to participate in a 60 minute evaluation of an IRS web site to be scheduled during the week of Aug 18th. The IRS is interested in finding out your opinions on the way they have designed their site. These evaluations are for research purposes only – we just want your opinions. We also want you to know the IRS will NOT in anyway use your personal information for anything other than the strict usage of it for this particular study. We would pay you to come to the IRS testing facility and provide your feedback and input on how well it works. Would you be willing to answer a few questions to see if you qualify?

IF NO, schedule time to call back. If Yes, continue.

[Reference the OMB Control Number and PRA Statement as noted below before continuing with the interview]

The Paperwork Reduction Act requires that the IRS display an OMB control number on all public information requests. The OMB Control Number for this study is 1545-1349. Also, if you have any comments regarding the time estimates associated with this study or suggestions on making this process simpler, please write to:

Internal Revenue Service

Tax Products Coordinating Committee

SE:W:CAR:MP:T:T:SP

1111 Constitution Ave. NW

Washington, DC 20224

Because we are seeking a wide variety of people, let me start by asking you some questions about yourself…

1. Have you ever taken part in a market research interview or usability study, either one on one or in a focus group type setting? IF NO SKIP TO Q4

2. When was the last time you participated in a market research interview or usability study? TERMINATE IF LESS THAN 6 MONTHS

3. Do you or any member of your immediate family currently work for or did they ever work for any of the following?

__	Marketing, Market Research Firm
__	Advertising
__	Web Site Design Company

IF YES TO ANY TERMINATE

__	For a city, state or federal government agency? If so, which one?

If IRS, terminate

4. Do you have access to the internet where you can use it for your own personal use?
   YES
   NO TERMINATE

5. On a scale of 1 to 5 where 1 means ‘not at all comfortable’ and 5 means ‘very comfortable’, how comfortable are you with… TERMINATE IF 1 or 2

using a mouse	1	2	3	4	5
using a keyboard	1	2	3	4	5
using the Internet	1	2	3	4	5
using a scroll bar	1	2	3	4	5

6. Have you currently or in the past personally prepared an income tax return for yourself or your family, or have you always used an outside agency?

Personally prepare the taxes………………..Continue

Uses an outside agency……………………..Terminate

7. Do you use the Internet to do any of the following?

e-commerce (web shopping)	YES	NO
web site surfing	YES	NO
pay bills online	YES	NO
work related reasons	YES	NO
research	YES	NO
online banking	YES	NO
pay taxes online/software	YES	NO

8. Are you employed? IF NOT EMPLOYED SKIP TO 8A

__

Full time or Part Time outside the home

Need 3

If employed:

What is your occupation? _____________________________________________

What industry are you in? __________________________________________

GET A GOOD MIX OF DIFFERENT OCCUPATIONS AND INDUSTRIES

8a. If Not Employed:

__	Retired
_	Student
__	Homemaker
__	In between jobs

GET AT LEAST 1 NOT EMPLOYED.

9. What is the highest level of education you have completed?

DO NOT READ ANSWER CHOICES - GET A MIX

__	Some high school or less
__	High school graduate
__	Some college, vocational or trade school
__	College graduate
__	Graduate school

10. Can you please tell me which of the following categories your age falls into?

GET A MIX - NO MORE TWO ONE PER AGE GROUP

Under 18	TERMINATE
18-29
30-39
40-49
50-59
60-65
66+

11. On a scale of 1 to 5 where 1 is ‘not at all knowledgeable’ and 5 is ‘extremely knowledgeable’, how much do you know and understand about federal income taxes and procedures, i.e. filling out tax forms, etc.

1

2

3

4

5

Terminate if 1 – Get a Mix

12. Have you ever visited the website IRS DOT GOV?

YES

NO SKIP TO Q17

13. What was the nature of your visit to the website?

__	to find forms
__	to review tax regulations
__	to find phone numbers
__	file downloads
__	online tax tools
__	other___________________

14. Approximately how long was the duration of time for your visit to the website?

__	less than 15 minutes
__	15 minutes to 30 minutes
__	30 minutes to 1 hour
__	longer than 1 hour

15. How many times have you visited the website? ___________

16. Was the information you were seeking on the website different depending on the season? (such as tax filing season or calculating withholding changes)

YES NO

17. What ways have you ever contacted the IRS?

__	visited a local branch
__	phone call
__	spoken with a specific representative

Thank you, the following questions are for classification purposes only.

18. What is your current marital status? RECRUIT A MIX

__	Single
__	Married/Living with partner
__	Widowed
__	Divorced/Separated

Mix of men and women

Thank you for answering my questions. As you can imagine, this survey was part of a larger research project. Your responses to the questions I have asked have prompted me to inquire if you would be interested in further helping by participating in a paid research project. You will receive $75, simply for sharing more of your opinions with us in a 60 minute evaluation. This is strictly a research project, and we do not, and are not selling anything, ever.

This is how our process works: As part of our research, we will be conducting evaluations sessions with a number of people in your area similar to yourself. You would be attempting tasks on a web site and discussing your experiences w/ that web site. Let me stress again that these evaluations are for research purposes only – we just want your opinions. We also want you to know the IRS will NOT in anyway use your personal information for anything other than the strict usage of it for this particular study. The IRS is just interested in finding out your opinions on the way they have designed their website. Most importantly we want to ensure that the system ends up helping taxpayers.

These evaluation sessions will be held on (DATE, TIME) at 1001 South 1200 West, Ogden UT. They will last approximately 60 minutes. Would you be available for that particular date and time?

No  POLITELY DISCONTINUE

Yes  CONTINUE

Do you have any questions up to this point or special needs for coming to our facility?

(Answer inquiries appropriately and thoroughly)

Again, let me thank you for your participation. This research is important to all of us, and your help and participation is highly appreciated. Here are the rest of the steps in the process:

After I have gathered your information, we will mail you a confirmation letter and map to the testing facility. You will also receive a follow-up / reminder call prior to the evaluation session. In a moment, I will need your name and address, as well as a phone number where you can be reached during the day and in the evening.

We will give you a reminder call prior to your scheduled appointment on (date) (time) to re-confirm that you will be attending and that you have no further questions. If we are unable to reach you and we leave a message, please call us back. Because we only schedule a limited number of sessions, and invite a limited number of research participants, it is our policy to continue calling until we do confirm your attendance.

Should something change in your participation availability, please phone our call center at 1-888-xxx-xxxx and reference “XXXXX”.

IRS.gov Redesign Usability Study

Tax Professionals Screener

Job #

Hello, I’m ______________from Consumer Opinion Services, Inc. On behalf of the Internal Revenue Service we are currently seeking out decision makers and persons who have involvement with the tax preparations for their place of business. The IRS wants to hear from you. Have you heard about the website IRS DOT GOV? What we would like to do is ask you a few questions to see if you would be able to participate in a 60 minute evaluation of an IRS web site to be scheduled during the week of Aug. 18th. The IRS is interested in finding out your opinions on the way they have designed their site. These evaluations are for research purposes only – we just want your opinions. We also want you to know the IRS will NOT in anyway use your personal information for anything other than the strict usage of it for this particular study. We would pay you to come to the IRS testing facility and provide your feedback and input on how well it works. Would you be willing to answer a few questions to see if you qualify?

IF NO, schedule time to call back. If Yes, continue.

[Reference the OMB Control Number and PRA Statement as noted below before continuing with the interview]

The Paperwork Reduction Act requires that the IRS display an OMB control number on all public information requests. The OMB Control Number for this study is 1545-1349. Also, if you have any comments regarding the time estimates associated with this study or suggestions on making this process simpler, please write to:

Internal Revenue Service

Tax Products Coordinating Committee

SE:W:CAR:MP:T:T:SP

1111 Constitution Ave. NW

Washington, DC 20224

Because we are seeking a wide variety of people, let me start by asking you some questions about yourself…

1. Do you currently conduct professional tax preparations? Yes Continue No Terminate

2. Do you prepare both individual and business returns? Yes CONTINUE No TERMINATE

3. What size of tax Preparation Company do you work for? Recruit a mix if possible

__	1 – 10 Employees
__	10 – 50 Employees
__	50 – 1000 Employees
__	1000 + Employees

3. Have you ever taken part in a market research interview or usability study, either one on one or in a focus group type setting? IF NO SKIP TO Q3

3. When was the last time you participated in a market research interview or usability study? TERMINATE IF LESS THAN 6 MONTHS

3. Do you or any member of your immediate family currently work for or did they ever work for any of the following?

__	Marketing, Market Research Firm
__	Advertising
__	Web Site Design Company

IF YES TO ANY TERMINATE

__	For a city, state or federal government agency? If so, which one?

If IRS, terminate

3. Do you have access to the internet at your company? IF NO TERMINATE

3. On a scale of 1 to 5 where 1 means ‘not at all comfortable’ and 5 means ‘very comfortable’, how comfortable are you with… TERMINATE IF 1 or 2

using a mouse	1	2	3	4	5
using a keyboard	1	2	3	4	5
using email	1	2	3	4	5
using the Internet	1	2	3	4	5

3. Do you use the Internet to do any of the following?

e-commerce (web shopping)	YES	NO
web site surfing	YES	NO
pay bills online	YES	NO
work related reasons	YES	NO
research	YES	NO
online banking	YES	NO
pay taxes online/software	YES	NO

3. Have you ever visited the website IRS DOT GOV?

YES

NO SKIP TO Q14

3. What was the nature of your visit to the website?

__	to find forms
__	to review tax regulations
__	to find phone numbers
__	file downloads
__	online tax tools
__	other___________________

3. Approximately how long was the duration of time for your visit to the website?

__	less than 15 minutes
__	15 minutes to 30 minutes
__	30 minutes to 1 hour
__	longer than 1 hour

3. How many times have you visited the website? ___________

3. Was the information you were seeking on the website different depending on the season? (such as quarterly filing or tax law changes)

YES NO

3. What ways have you ever contacted the IRS?

__	visited a local branch
__	phone call
__	spoken with a specific representative

Recruit to obtain 6 Tax Professionals / Tax Practitioners

Recruit A MIX OF GENDERS

Thank you for answering my questions. As you can imagine, this survey was part of a larger research project. Your responses to the questions I have asked have prompted me to inquire if you would be interested in further helping by participating in a paid research project. You will receive $75, simply for sharing more of your opinions with us in a 60 minute evaluation. This is strictly a research project, and we do not, and are not selling anything, ever.

This is how our process works: As part of our research, we will be conducting evaluation sessions with a number of people in your area similar to yourself. You would be attempting tasks on a web site and discussing your experiences with that web site. Let me stress again that these evaluations are for research purposes only – we just want your opinions.

We also want you to know that the IRS will NOT in any way use your personal information for anything other than the strict usage of it for this particular study. The IRS is just interested in finding out your opinions on the way they have designed their web site. Most importantly we want to ensure that the system ends up helping taxpayers.

These evaluation sessions will be held on (DATE, TIME) at 1001 South 1200 West, Ogden UT. They will last approximately 60 minutes. Would you be available for that particular date and time?

No  POLITELY DISCONTINUE

Yes  CONTINUE

Do you have any questions up to this point?

(Answer inquiries appropriately and thoroughly).

Again, let me thank you for your participation. This research is important to all of us, and your help and participation is highly appreciated. Here are the rest of the steps in the process:

After I have gathered your information, we will mail you a confirmation letter and map to our facility. You will also receive a follow-up / reminder call prior to the interviews. In a moment, I will need your name and address, as well as a phone number where you can be reached during the day and in the evening.

We will give you a reminder call prior to your scheduled appointment on (date) (time) to re-confirm that you will be attending and that you have no further questions. If we are unable to reach you and we leave a message, please call us back. Because we only schedule a limited number of sessions, and invite a limited number of research participants, it is our policy to continue calling until we do confirm your attendance.

Should something change in your participation availability, please phone our call center at 1-888-xxx-xxxx and reference “XXXXX”.

2. Recruiting Cover Letter

August, 2008

Dear ____________,

Thank you for agreeing to participate in our upcoming research usability study for the Internal Revenue Service about feedback of the IRS.gov Web site. We want to assure you that this is only a research study and your identity remains anonymous. During the study we will ask you to navigate a website to ensure it is helpful to taxpayers. Your appointment is as follows:

DATE:	(insert date)
LOCATION:	South 1200 West, Ogden UT
TIME:	(insert time)
LENGTH:	1.5 hours (including travel time and 60 minute interview)
HONORARIUM:	$75.00
Study Title:	COLDS R1 Usability Study
DIRECTIONS:	Contact Jana Roberts if you get lost. See enclosure
Please check in 10 minutes before the session start time to ensure that we proceed on schedule.

We have invited only a limited number of people, about 6, and we are counting on your timely attendance and participation for completion of our study. There is no back up person to participate if you do not show up to the research study, so it is very important that you keep your commitment. You will be alone in a conference room with a moderator and the IRS staff will be observing you while you are reviewing the website. Your opinions are very important to the IRS, as they will act on what you have to say. If for any reason you cannot make this appointment, call us upon receipt of this letter so that we may find a replacement for you before the day of the study.

Please bring your photo ID upon check-in. If you use reading glasses, please bring these with you. Because of limited space, please do not bring children or others along with you, as we will not be able to accommodate them. If you need to bring an assistant to help you please contact us to make arrangements. Only the people we have contacted are accepted in to the research study.

All of our interviews are conducted strictly for research. At no time will someone attempt to sell you any product or service, nor will your name be supplied to any manufacturer for sales purposes.

Your honorarium will be paid at the end of your interview. Again, thank you for your time, and remember, your opinion counts!

Sincerely,

Beth Fitzpatrick, Call Center Manager

3. Moderator Checklist

1. Look at and remember person’s name prior to going to meet him/her.

2. Introduce yourself and other team members who might interact with participant
   (i.e., logger, observation facilitator).

3. Ask if person would like something to drink and/or to use the restroom.

4. Ask person to read and sign the consent form.

5. Reference the OMB Control Number and PRA Statement:

The Paperwork Reduction Act requires that the IRS display an OMB control number on all public information requests. The OMB Control Number for this study is 1545-1349. Also, if you have any comments regarding the time estimates associated with this study or suggestions on making this process simpler, please write to:

Internal Revenue Service

Tax Products Coordinating Committee

SE:W:CAR:MP:T:T:SP

1111 Constitution Ave. NW

Washington, DC 20224

6. Talking points to cover in verbal session overview:

• Discuss the purpose of usability testing / this usability study.

• Inform that the session will be videotaped and audio taped (as mentioned in consent form).

• Ask person to wear microphone (if applicable).

• Explain that people will be observing (and remotely, if applicable) and that they will see what is happening on the computer screen and the video from the video camera.

• Remind participant to “think-aloud” – ask the participant to provide a running commentary about what he/she is doing at all times in the site, and that it may seem awkward but to try to talk as much as possible.

• Remind that (s)he is helping us evaluate the site—we are not evaluating him/her

7. Instruct the participant to read the first page of the participant package.

8. Ask the participant if (s)he has any questions.

9. Ask the participant to turn the page and complete the preliminary question about information (s)he typically would want to find on IRS.gov.

10. Ask the participant to turn the page and read the first task aloud.

11. After each task, ensure that participant completes the post-task questions and discuss his/her responses with him/her.

12. After each task, check whether more than 15 minutes of session time remains. If so, let participant proceed to next task. If not, ask him/her to respond to design and do an “exploration” task where the participant can look through other sections – observe navigation technique.

13. Ensure that participant completes post-study questionnaire and discuss his/her responses with him/her. Ask if (s)he has any other comments about the site.

14. Have participant sign receipt form and record honorarium check number on signed form.

4. Participant Consent Form

The purpose of this study is to help us evaluate the new design for the IRS.gov Web site. We would like you to help us identify ways we can change the design to make finding information easier. By participating in this study, you will help us shape the future of the IRS.gov Web site.

This session will take approximately 1 hour. You will be given a brief questionnaire to find out more about your previous web experiences. While you evaluate the Web site, you will be asked to perform specific tasks and answer some questions after each task is performed. The session will be recorded using microphones, a video camera, and a device to capture the computer screen.

You may experience frustration and confusion during the test. The Search function will not be working, but all content will be in the site. You may ask questions at any time during the study, but please note that due to the type of information we are trying to collect, we may not be able to fully answer your question until after the study.

If you do not want to be in this study, you do not have to participate. You do not have to answer any questions you do not want to, and you may stop at any time without any consequences. All information that you provide in this research study will be kept confidential and any report of this research will not identify you personally in any way.

If you agree to help us, please sign below.

( Signature) (Date)

( Print Name)

Paperwork Reduction Act (PRA) Statement:

The Paperwork Reduction Act requires that the IRS display an OMB control number on all public information requests. The OMB Control Number for this study is OMB 1545-1349. Also, if you have any comments regarding the time estimates associated with this study or suggestions on making this process simpler, please write to the:

Internal Revenue Service

   Tax Products Coordinating Committee

   SE:W:CAR:MP:T:T:SP

   1111 Constitution Ave. NW

   Washington, DC  20224

5. Scenarios

INDIVIDUALS

1. You have money in a Roth IRA and are wondering if you are unsure if you have to report your nondeductbible contributions. Using the FAQs find out if you have to report your nondeductible contributions.

Target = “More online tools” < “Frequently asked questions” < “Keyword” < “R” < “Roth IRA” OR “Category” < “Individual retirement arrangements” < “Roth IRA”

Answers:

a. Yes, you have to report

b. No, the financial institution reports for you

2. You are a Canadian citizen both living and working in the United States. You are unsure if you need to file a US tax return or a Canadian tax return or both. Using the FAQs, what are you required to do?

Target = “More online tools” < “Frequently asked questions” < “Keyword” < “Aliens and US Citizens living abroad”

Answers:

a. You only file a US tax return

b. You only file a Canadian tax return

c. You file both a Canadian and a US tax return

d. You don’t need to file with either country

TAX PROFESSIONALS

1. Your client has money in a Roth IRA and you are unsure if you have to report their nondeductible contributions. Using the FAQs find out if you have to report their nondeductible contributions.

Target = “Frequently asked questions” < “Keyword” < “R” < “Roth IRA” OR “Category” < “Individual retirement arrangements” < “Roth IRA”

Answers:

a. Yes, you have to report

b. No, the financial institution reports for you

2. Your client is a Canadian citizen both living and working in the United States. You are unsure if you need to file a US tax return or a Canadian tax return or both. Using the FAQs, what are you required to do?

Target = “Frequently asked questions” < “Keyword” < “Aliens and US Citizens living abroad”

Answers:

a. You only file a US tax return

b. You only file a Canadian tax return

c. You file both a Canadian and a US tax return

d. You don’t need to file with either country

3. A client you did taxes for last year calls you and states that he/she feels that someone may have filed a return using their social security number because the IRS is calling and wants to know why 2 returns were filed with the same SSN. You want to halp your client, but aren’t 100% sure what to do. Using the FAQs, find who you should call to report this identity theft problem.

Target = “Frequently asked questions” < FAQ by subcategory” < “Reporting Fraud”

Answers:

1. The IRS

2. The FBI

3. The Federal Trade Commission

4. Your local police station

6. Exit Questionnaire

Instructions:
Please circle the single response that best matches your answer where there is a multiple choice question and print your response in the space provided for open ended questions. Please feel free to include your feedback in the comments areas provided.

Please answer the following questions and statements based on all the tasks you performed today.

Navigating the Demo

1. How easy was it to find your way through the Frequently Asked Questions (FAQs) using the navigation within the application?

Very Hard

Hard

Slightly Hard

Slightly Easy

Easy

Very Easy

Comments:

2. Did the the navigation make sense to you?

Yes No

Comments:

3. Did you have difficulty completing tasks?

Yes No

Comments:

4. If you lost your way, how easy was it to recover and continue searching for the answer to your task?

Very Hard

Hard

Slightly Hard

Slightly Easy

Easy

Very Easy

Comments:

Search Feature

5. How would you rate the difficulty of initially finding the search box on the FAQs landing pagepage?

Very Hard

Hard

Slightly Hard

Slightly Easy

Easy

Very Easy

Comments:

6. Did you understand the purpose of the search term field?

Yes No

Comments:

7. Did the FAQs search function return useful results?

Yes No

Comments:

Aesthetics

8. To what degree did you find the FAQs landing page visually attractive?

Not Attractive At All

Not Attractive

Slightly Unattractive

Slightly Attractive

Attractive

Very Attractive

Comments:

9. How would you rate the amount of information on each page?

Far Too Much Information—Overload

Too Much Information

About the Correct Amount per Page

Too Little Information

Far Too Little Information—Unsatisfied

Comments:

Other Features

10. Please list any features you expected on the FAQs pages but did not find.

11. What other improvements would make it easier to find and read content?

12. Do you have overall suggestions for improvement of the FAQs?

Paperwork Reduction Act (PRA) Statement:

The Paperwork Reduction Act requires that the IRS display an OMB control number on all public information requests. The OMB Control Number for this study is 1545-1349. Also, if you have any comments regarding the time estimates associated with this study or suggestions on making this process simpler, please write to the:

Internal Revenue Service

   Tax Products Coordinating Committee

   SE:W:CAR:MP:T:T:SP

   1111 Constitution Ave. NW

   Washington, DC  20224

7. Confirmation Phone Call Script

Good morning/afternoon, I am calling to confirm your participation in our upcoming research usability study for the Internal Revenue Service about the IRS.gov Web site.

Reference the OMB Control Number and PRA Statement as noted below before continuing with the interview:

The Paperwork Reduction Act requires that the IRS display an OMB control number on all public information requests. The OMB Control Number for this study is 1545-1349. Also, if you have any comments regarding the time estimates associated with this study or suggestions on making this process simpler, please write to:

Internal Revenue Service

Tax Products Coordinating Committee

SE:W:CAR:MP:T:T:SP

1111 Constitution Ave. NW

Washington, DC 20224

Did you receive a package from us with the date and time of your interview and directions to the site?

Yes  Confirm Date and Time are OK and that Directions are clear (see below). Verify that they have transportation to get to the facility.

No  Detail the appointment [See below] and ask if they have FAX # or email address that you can send the information to.

DATE:	(insert date)
TIME:	(insert time)
LOCATION:	South 1200 West, Ogden UT
DIRECTIONS:	See enclosure
LENGTH:	1.5 hours (including travel time and 60 minute interview)

Please check in 10 minutes before the session start time to ensure that we proceed on schedule. Please bring your photo ID upon check-in. If you use reading glasses, please bring these with you.

We have invited only a limited number of people, and we are counting on your timely attendance and participation for completion of our study. There is no back up person to participate if you do not show up to the research study, so it is very important that you keep your commitment. Should something change in your participation availability, please phone our call center at 1-888-xxx-xxxx and reference “IRS.gov FAQs Usability Study”.

All of our interviews are conducted strictly for research. At no time will someone attempt to sell you any product or service, nor will your name be supplied to any manufacturer for sales purposes. Your honorarium of $75 will be paid at the end of your interview.

Do you have any questions?

[Answer inquiries appropriately and thoroughly]

Again, thank you for your time and we look forward to seeing you on [day of week] at [time].

Internal Revenue Service

C ustomer Online Decision Support (COLDS)R1

Usability Test Plan

August 18-22, 2008

Version 2

Prepared by IRS’ Public User Portal, Electronic Tax Administration, Wage and Investment Business Operating Division, Accenture and IRS-Ogden Usability Staff, Ogden, Utah.

Product Overview

Description

Customer On-Line Decision Support (COLDS) Release 1 will enhance the W&I’s Frequently Asked Questions (FAQs) content and functionalities currently available on IRS.gov web site. COLDS will modify the existing Content Management Application (CMA), Search Engine, and WebTrends components to improve FAQs. The solution will enable taxpayers to conduct key word and natural language queries on the Frequently Asked Questions (FAQs).  Taxpayers will have the opportunity to provide feedback on the usefulness of the content, thereby enabling FAQ content authors to improve content effectiveness. COLDS will also provide FAQ content authors the capability to easily update the FAQ content to the web site.

Release 1 will be implemented by the 2009 Filing Season.

 

Background

In response to a Congressional directive, the IRS conducted a comprehensive review of its current portfolio of taxpayer services to create a Taxpayer Assistance Blueprint (TAB). The TAB team identified the need to improve website self-service applications. The W&I Strategy and Program Plan addressed the need to provide solutions to ensure that Taxpayers, Practitioners and other IRS partners have access to accurate and timely responses to tax law issues and inquiries at all hours during the day. This included the need to increase availability of self-assist applications through multiple channels, including the Internet.

COLDS will provide an efficient electronic alternative on IRS.gov for taxpayers seeking assistance with their federal tax questions who would otherwise require assistance from a Call Site or a Tax Assistance Center (TAC). COLDS will also support the TAB initiatives as well as the W&I Strategy and Program plan.

An improved on-line FAQ solution for routine questions will mitigate the burden on telephone assistance centers during peak traffic times, freeing the assistors to handle more complex questions. This project will move the IRS customers away from the more expensive channels to the less expensive internet customer self-service channel.

Purpose of Usability Test

The purpose of this test is to determine if the information in the Frequently Asked Questions portion of the web site is logical. In addition, we are testing to confirm that the layout of the information is logical and easy to understand. Testing will also be conducted to determine if the navigation of the FAQs is logical and intuitive.

.

Test Objectives

During this usability test the team would like to verify that the navigation, search and listing options are easily understood by users. In addition, the team would like to verify that users understand how to find the information they are looking for quickly and without any question. We would also like to test navigation to the FAQs from the IRS.gov home page.

The most important aspects of the FAQs are the search, navigation and understanding of categorization of the topics. Categorization of the topics is not part of the COLDS R1 scope, however, we would like to test it and share our findings with the SME’s. Users need to be able to navigate through the FAQs in a short period of time. The categories and the content need to make sense to the users. This should be a very simple application that users can navigate or search without problems.

Usability testing is intended to gather user data on the overall content, navigation, and user interface design of the site.

The following are examples of objectives that address these areas.

Test Objective: Content

Findings in the section will not be addressed by COLDS R1. Information from testing of content will be shared with SME’s for future enhancements.

Frequently Asked Questions is usually one of the last resorts for users to find information. In addition, FAQs are supposed to supplement content that is currently on the site. During testing we want to ensure that the answer in the FAQ is answering the questions being asked.

• Content needs to be clear and concise

• Writing for the web practices need to be used

• Bullets, instead of long paragraphs

• Examples if necessary

Test Objective: Navigation

Navigation of the FAQs portion of the web site needs to be clearly understood by users. Users will have the option to either search for the content they need, navigate to through the different categories, or use the “Top FAQs” listing. Navigation of the FAQs has been created so that there is not a long listing of information to read through to get to the answer. Each category has a set of subcategories that users should select to see the different questions asked. The navigation should be easy for users to understand. Findings related to content will be shared with the business, but is not part of the scope of COLDS R1. In addition, findings related to functionality will be addressed by the COLDS R1 team.

Test Objective: User Interface (UI)

The overall objective for this usability test is to verify that the navigation paths offered are useful and make sense to the user. In addition, we want to verify that users understand where they are while navigating through the FAQs. The team would like to make sure that the FAQs search is useful and in an appropriate location.

• Are the FAQ screens consistent between all navigation paths? (compliant with style guide)

• Is the information written using writing for the web principles?

• Are the navigation paths easily understood?

• Do users understand where they are within the navigation path?

• Are colors pleasing? Is color used in a manner that is inconsistent with section 508? (compliant with style guide)

• Does the application functionality address user’s concerns?

Usability Measures

• Time spent to complete a task

• Navigation paths used

• Search vs. navigation (behavior)

• User satisfaction ratings (Post Test Questionnaire)

• Number of times user required help

• Task success rate

• Error recovery

• 1^st clicks (using UTE tool)

• Search Terms if search is used)

User Profile

Public – External Customers

External Customers	Target #
Individuals – Individual tax payers (6 participants). Students (newer / first time tax payers), homemakers, retired citizens, average taxpayers (employed individuals).	6
Tax Practitioners – (6 participants). Tax practitioners that prepare taxes for individuals, use the individuals portion of the website on a regular basis.	6

• General knowledge of the internet with frequent use of the internet

• General knowledge of tax filing and processing

• Knowledge of / responsible for individual tax filing

Usability Test Procedures

The test is designed to take approximately 60 minutes for each participant to complete. Test participants will consist of individuals and tax practitioners that prepare or have knowledge about individual tax filing. Testing will occur in Ogden, Utah at the IRS Usability Lab. Testing will occur in one lab. Consumer Opinion Services will recruit the test participants using a series of screening questions approved by the COLDS R1 team. The goal for each user group is between 6 and 8 participants for a total of 12 – 16 participants. The test will have a user complete a variety of tasks using the IRS.gov web site. The participant will be asked to talk-aloud as they complete the tasks.

Testing will be based on scenarios created by Accenture and the IRS. Scenarios are short statements that include a task to be completed by using the FAQs portion of IRS.gov. The scenarios will be unique for each user group (i.e. individuals and tax practitioners).

The moderator will complete an introduction, and provide an overview of the test process and ask the participant to sign a consent form. The form will explain what the participant is going to be doing and how the IRS plans to use the information gathered from testing.

No personally identifiable information will be shared outside of the test environment. The IRS team will destroy all background information upon completion of the test.

After the introduction each participant will work each scenario individually on a computer. Participants will be required to complete each task without assistance from the moderator. They will be asked to “think aloud”, so that the usability team can capture as much data as possible.

When the participant has completed the scenarios there will be a comprehension question about the FAQ page in general. After that there will be a debriefing session that will consist of open discussion and an exit questionnaire. We are rewarding $75 to each participant. .

Participants will use a PC running Windows XP with a 17-inch monitor. The screen resolution of the monitor will be set to 800 x 600 with Windows display properties set to “Small Fonts”. Participants will access the product by using Microsoft Internet Explorer 6.0.

Tasks/ Scenarios

Information will be added once scenarios are completed…

Test Environment and Equipment

This usability test will be conducted in Ogden’s Usability Lab, located in Ogden, Utah. The lab consists of an evaluation/test room and an observation viewing room.

The evaluation room includes the following equipment:

• Three ceiling mounted video cameras

• One-way mirror

• Workstation and chair

• Telephone, which on occasion is used as a line to the “help desk,” (actually a team member in the observation room).

• Lapel microphone to be worn by the participant to ensure all comments and feedback can be heard by the test team in the observation room.

• Computer with Windows XP operating system, a 17-inch monitor and Microsoft Internet Explorer 6.0.

• Printer

The control/observation room, which is adjacent to the evaluation room, includes the following equipment:

• Audio mixer and amplifier

• DVD Video recorder

• Video switcher

• Scan Converter

• Data logging workstation

• 6 monitors to view the application interface

.

Usability Test Roles

This section describes the roles of people involved directly or indirectly in the usability test sessions and describes the procedure that will be followed for each test session and the task scenarios that participants will attempt during the test sessions

The following roles are required before and during the period over which usability testing takes place:

• Planner and coordinator – Usability Administrator ensures that everything is in place to allow the usability test sessions to take place as planned and without problems.

• Participant recruiter – recruits participants and schedules them for available time slots; provides test facilitator with honorarium checks to give to each participant at the end of their test session.

The following roles are required in order to conduct a usability test session:

• Test facilitator –Usability Consultant oversees the test itself, introduces and explains the test session to the participant, observes participant’s actions and obtains participant feedback at completion of test session, conducts issues assessment session. The facilitator is also available to answer questions and ensures that the observer stakeholders stay on task and have a productive and successful experience.

• Test data logger – creates a written record of the participant’s actions and comments as they use the site using data logging software, a word processor, or a spreadsheet.

• Observer-team member – observes test to identify functionality and performance, compares notes with other team members, and actively participates in Issues Assessment at end of each test session.

Project Team

The following table shows which people will be serving in which roles:

Observer-team member	Cindy Williams
Role	Name(s)	Business Operating Division	Organization	Job Title	Contact Information
Planner and coordinator	Shawn Tafoya	MITS	AD	Usability Administrator	[email protected] (801)620-4060
Participant recruiter	Consumer Opinion Services
Test facilitator	Jana Roberts	MITS	AD	Usability Administrator	[email protected] 801-620-4283
Test data logger	Jana Roberts	MITS	AD	Usability Administrator	[email protected] 801-620-4283
Test team lead	Accenture Beth Krappweis	W&I	ETARC: PPB	IT Specialist	[email protected] 202-283-4844
Observer-stakeholder	Cindy Williams	W&I	CAS	Project Manager	[email protected] 678-530-5238
Observer-team member	Corliss Brooks	W&I	ETARC:PPB	PPB COLDS R1 Lead	[email protected] 202-283-0833
Observer-team member	Accenture

Deliverables to Client

Final Usability Report

• Executive Summary

• List of Usability Issues

• Task success rates

• Questionnaire results

• PowerPoint presentation of findings / report

System Requirements

Application URL	TBD
Server Requirements	TBD
Hardware needed	Screen Resolution of_____800 X 600_______
Software needed	Internet explorer
Project Technical Contact Point
Usability Technical Contact Point	Howard Schuyler 801-620-4056 [email protected]
Remote Viewing	No
	Yes	Number Viewing	Sites Viewing

Test Agenda

Date	Time	Activity	Participant
Tues, Aug. 19	8:00 a.m.– 10:00 a.m.	Test	#1
Tues, Aug. 19	10:00 a.m.– 12:00 p.m.	Test	#2
	12:00 p.m. – 1:00 p.m.	Lunch
Tues, Aug. 19	1:00 p.m.– 3:00 p.m.	Test	#3
Tues, Aug. 19	3:00 p.m.– 5:00 p.m.	Test	#4
Wed, Aug. 20	8:00 a.m.– 10:00 a.m.	Test	#5
Wed, Aug. 20	10:00 a.m.– 12:00 p.m.	Test	#6
	12:00 p.m. – 1:00 p.m.	Lunch
Wed, Aug. 20	1:00 p.m.– 3:00 p.m.	Test	#7
Wed, Aug. 20	3:00 p.m.– 5:00 p.m.	Test	#8
Thurs, Aug. 21	8:00 a.m.– 10:00 a.m.	Test	#9
Thurs, Aug. 21	10:00 a.m.– 12:00 p.m.	Test	#10
	12:00 p.m. – 1:00 p.m.	Lunch
Thurs, Aug. 21	1:00 p.m.– 3:00 p.m.	Test	#11
Thurs, Aug. 21	3:00 p.m.– 5:00 p.m.	Test	#12

Assumptions

• All usability tests will be physically conducted in Ogden, Utah.

• Development project along with Consumer Opinion Services will be responsible for participant recruitment and compensation if necessary with guidance from members of the IRS Usability Team.

• This study is not evaluating the products’ conformance with Section 508 regulations.

20