SOI-274 - Supporting Statement

Submission for
OMB Clearance

U sability Testing of the 1040 and 4933 Call Scripts for the Internal Revenue Service (IRS)- Round 2

Prepared for the
Office of Management and Budget

August 1, 2007

Table of Contents

I. Introduction 1

Background 1

Objectives of Data Collection 1

II. Methodology 2

Who Is Conducting Research 2

Sample Design 2

Data to Be Collected 2

How Data Will Be Used 2

How Data Will Be Analyzed 2

Data Collection Date 3

Location 3

Cost of study…………………………………………………………………………………………. 3

Stipend 3

Recruitment Efforts 3

Test Structure/Design 4

III. Participant Criteria 5

IV. Privacy, Security, Disclosure, Confidentiality 6

V. Burden Hours 7

VI. Attachments 8

I. Introduction

Background

The Primary Abandons Taskforce Report drafted by Internal Revenue Service (IRS) outlined the need to address primary abandons to the call center environment (CCE) phone system (IRS’ 1-800 lines). IRS defines primary abandons as an abandon (caller hang-up) that occurs anytime prior to a caller being placed in a queue (on hold) or routed directly to a live agent. According to this report, in Filing Year (FY) 2006 the enterprise primary abandon rate for the Customer Account Services (CAS) Toll Free Product Lines was 16.36 percent. In order to reduce this primary abandon rate, IRS contracted with Kleimann Communication Group, Inc. (KCG) (under subcontract to Lockheed Martin/MSD, Inc.) to conduct focus groups and one-on-one usability tests on IRS call scripts to test the usability of the phone system and identify elements that can be improved.

In these focus groups and usability testing, Lockheed Martin, MSD/KCG will test call scripts heard by taxpayers calling IRS toll free numbers, specifically the 1040 (individual taxpayer) and 4933 (business and specialty tax) product lines. We will evaluate how effectively these call scripts perform across a range of taxpayer demographics, identify possible areas for improvement, and identify causes for call abandons. With this information, IRS will be able to improve the usability of the call scripts and the customer experience.

Objectives of Data Collection

The specific objectives of the focus groups and usability testing of the call scripts are to determine the following:

• Why do taxpayers abandon calls?

• What perceptions do taxpayers have about calling the IRS toll free numbers?

• How can IRS improve call scripts so that taxpayers can effectively get the information they seek?

• How do taxpayers feel about entering personal information over the IRS phone system?

II. Methodology

Who Is Conducting Research

IRS has contracted with the Lockheed Martin, MSD/KCG to conduct focus groups and usability testing of call scripts for the IRS toll free numbers (1040 and 4933).

Sample Design

Lockheed Martin, MSD/KCG will conduct four focus groups with eight participants each (for a total of 32 focus group participants) and one-on-one 90-minute usability tests with six participants at each site across four sites (for a total of 24 usability participants).

Lockheed Martin, MSD/KCG will confirmatory test the effectiveness of changes made to the 1040 and 4933 call scripts. Participants will be recruited to mirror typical U.S. taxpayers in race/ethnicity, age, marital status, gender, income, and filing method (electronic vs. paper). Our sample will include at least three BMF (business filing) and three IMF (individual filing) taxpayers at each location.

Data to Be Collected

Since the focus group and usability testing analysis is strictly qualitative in nature, we will report feedback and impressions noted during the sessions. Our data will result from critical questions posed in the Moderator’s Guide (Attachment A-Parts 1 and 2) that are answered by the participants during the focus group and usability testing sessions.

How Data Will Be Used

We will use the data to evaluate potential new changes to the phone script and system to assess whether taxpayers have problems, why they are having problems, and their attitudinal opinions, particularly issues concerning navigation, decision-making, and comprehension.

How Data Will Be Analyzed

Researchers will identify critical incidents from each of the testing sessions that report

• Participant perceptions about calling the IRS toll free numbers;

• Whether participants can find information they need by navigating the call scripts;

• Whether participants can understand the information in the call scripts; and

• Reasons for primary abandons.

Data Collection Date

Focus group and usability testing is scheduled to be conducted from October 1 - October 18, 2007.

Location

Testing will be conducted in the following geographic locations that represent diverse settings and populations:

• Focus group testing

— San Francisco, CA

— Denver, CO

— Tulsa, OK

— Boston, MA

• Usability testing

— San Francisco, CA

— Denver, CO

— Tulsa, OK

— Boston, MA

In each location we will use professional testing and recruiting facilities that include a testing room equipped with a one-way mirror, observation room, and video- and audio-taping equipment.

Cost of Study

The approximate cost of this study is $197,901. This number includes site costs, travel costs, labor costs, and other direct costs.

Stipend

Each individual participant (IMF) will receive $50 for participation in the study; each small business participant (BMF) will receive $100. The current industry stipend for participation in a testing session or focus group is $100-$200. Remuneration at the $50 rate for individuals and $100 rate for business owners is the minimum amount recommended for successfully recruiting participants for a 90-minute interview.

Recruitment Efforts

Attachment B includes the participant recruiting package. We will recruit eight participants at each site for usability testing and eight participants at each site for focus group testing. All participants will meet the eligibility requirements listed in Attachment B, Part 1. Recruiting will begin as soon as possible after receiving OMB approval and recruiting will be based on a participant screener that will be used by all four locations (Attachment B, Part 2). The recruiting facilities will keep track of recruitment burden hours. Participants will receive a confirmation letter in advance of the testing date (Attachment B, Part 3) and will be reminded, via telephone, the day before the testing appointment in order to ensure participation. Attachment B, Part 4 contains a telephone script for this reminder telephone call.

Test Structure/Design

Each focus group and usability testing session will take no more than 90 minutes. Prior to the testing session, participants will be asked to read and sign a confidentiality statement (Attachment C, Part 1). This confidentiality statement will inform participants that their participation is voluntary and will explain how the confidentiality will be maintained. We will also ask participants to complete a video release form (Attachment C, Part 2) to allow us to record their image for use in reporting data. Participants will be able to elect whether or not they will allow us to use their image and may participate in the interview regardless of which option they select.

Below, we outline the details of the 90-minute sessions, including participant tasks and time allotted for each task:

• In the introduction, we will introduce ourselves, briefly describe the purpose of the testing in general terms, and have the participants complete a demographic questionnaire (Attachment D) to capture the demographic characteristics of the testing sample. Estimated time: approximately 15 minutes.

• In the focus group testing session, we will have the participants (1) voice specific opinions about calling the IRS toll free number and toll free numbers in general, (2) voice specific likes and dislikes about call systems, and (3) voice opinions on the security of entering personal information into phone systems. Participants will review and discuss these topics for approximately 25 minutes each.

• In the one-on-one usability testing session, we will have the participants (1) complete a series of tasks using the IRS 1040 and 4933 product lines, (2) explain their thought processes and decisions in navigating the phone system, and (3) respond to specific questions related to the call scripts and tasks completed. Participants will spend approximately 30 minutes on each of these sections and will take a 10-minute break about halfway through the testing session.

If participant(s) take less than 90 minutes to complete the tasks, we will end the focus group or usability testing when all tasks have been completed. If participant(s) do not complete all tasks within the time allotted for the testing sessions, we will terminate the sessions after 90 minutes.

III. Participant Criteria

Gender	Mix
Race/Ethnicity	Mix
Marital Status	Mix
Age	Mix 10%-20% over 65 10%-20% under 25
Education	Mix 20% high school or less No more than 20% advanced or professional degrees
IMF/BMF	At least three IMF at each site At least three BMF at each site
ESL	10%-20%
File own tax return	100%
Total	56 participants (32 focus group and 24 usability testing)

IV. Privacy, Security, Disclosure, Confidentiality

We will protect the privacy of the participants in the testing sessions by not using names in our reports. We will also control official access to the information and will not allow public access to the information. The questionnaires and data collected during the interviews will be destroyed when we have completed the project and there is no further need for the information.

V. Burden Hours

The estimated time to complete the participant screening is 10 minutes and the estimated time for each participant session is 90 minutes. We anticipate that approximately 50% of taxpayers contacted for this study will qualify. With this percentage, we will need to screen 128 individuals to recruit our 64 participants (56 actual test participants and 8 floater participants).

Total number of potential participants screened: 128 individuals

Estimated time to complete screening: 10 minutes

Estimated participant screening burden: 21 hours (128 x 10/60)

Estimated number of participants: 56 individuals

Time to conduct study: 90 minutes

Estimated travel time to and from site: 30 minutes

Estimated participation burden: 112 hours (56 x 2)

Total Burden (screening and study participation) = 133 hours (21+ 112)

Lockheed Martin, MSD/Kleimann Communication Group, Inc. 0