IRS W&I Independent Review of IRS Free File Program

Free File Program Usability Testing Protocol

SCHEDULE

1. Welcome participant – 5min

2. Introduction and Consent – 5min

3. Free Search Engine Testing – 15min

4. Search Engine Debriefing – 5 min

5. Free File Site Testing – 45 min

6. Debriefing and Gift Card – 3mins

7. Wrap-Up – 2mins

PRE-INTERVIEW

Prepare materials:

• Protocols

• Schedule that includes participant names and participant ID

• 2 signed consent forms

• Scenario used for session

• Questionnaire

• VISA GC

• GC Receipt

• Computer for session

• Hotspot for internet access

INTRODUCTION

Thank you for helping us out today. Our names are [NAMES] and we work as researchers with the MITRE Corporation. MITRE is a not-for-profit company that runs federally funded research and development centers, including for the IRS. I’d like to start out by telling you a little bit about what we’re doing today. As you may know, IRS has a Free File program that allows taxpayers free access to electronic filing of their tax returns if they meet the eligibility requirements set annually by the IRS. The IRS has asked MITRE to conduct interviews with taxpayers, such as yourself, to better understand your experience using the Free File Program. Today, you’ll be helping us out by using fictitious tax information complete a few activities using the online Free File program systems. We will observe you while you complete these activities, and after you are done, I’ll ask a few questions.

I want to stress that my purpose here today is not to collect your personal information, but rather to understand if there are any problems or difficulties you encounter while using the Free File system, so we can make recommendations to the IRS on how to improve the system or better support taxpayers as they fill out the new form. Our team did not have a role in designing the online software, so please don’t feel like you need to hold back or worry that we will take your criticisms personally. Also, MITRE has no commercial interest in the Free File program.

Before we get started, I’d like you to read and sign this consent form. This consent form explains your rights as a participant in this research study. It explains that your information will be kept confidential, and that your name will not be associated with anything you tell me when I prepare my report for the IRS. It also explains that you have the right to end the interview at any time.

[GIVE CONSENT FORM AND HAVE PARTICIPANT SIGN]

Thank you. Again, I want to remind you that your participation in this interview is completely voluntary. We will be here for about 90 minutes. To focus on this interview, I ask that you place your phone on silent and not use it during the study. At the completion of the interview, I will give you a prepaid VISA card for $100.

Do you have any questions before we start?

USABILITY SECTION

Usability Activity 1. Free File Search Path Testing

For your first activity, imagine that you have learned that the IRS provides a service that allows you to file your tax return for free. In front of you is a computer we have provided that has access to the internet. Please imagine that you are at home, or in a private space that is comfortable you, and use this computer to locate the free online tax service provide.

Our goal is to understand how you would go about locating the Free File program. If you have questions for me, we can discuss those when the interview is over. That said, it will help me to understand your process if you think aloud about what’s going on. What I’m most interested in is your thoughts and feelings as you complete the activities during this session. If you fall quiet, I may encourage you to share your thoughts with me.

Please go ahead locate the Free File online program. We will observe you and take some notes, and then I will have some questions about your experience when you finish.

[ALLOW PARTICIPANT TO USE COMPUTER TO LOCATE FREE FILE]

[ADMINISTER SINGLE QUESTION SEQ]

Debriefing Questions

1. Can you please walk us through the process you used to locate the free file program?

2. [if applicable] What were your impressions of the search engine results that you encountered?

3. [if applicable] What are your impressions of the website that you encountered when you searched for the Free File program?

4. [if needed] I noticed you expressed [hesitation/confusion/frustration] at [note point of difficulty]. Can you tell me more about that?

Free File Lookup Tool/Provider Selection

Our next activity is for you to navigate this page, which helps taxpayers select a Free File software provider. Please take a moment to look at this site.

[navigate to www.freefilealliance.org]

Questions

1. What are your initial impressions of this site?

2. If you were looking to use Free File and encountered this site, what would you do? Can you tell me more about that?

[navigate to https://www.irs.gov/filing/free-file-do-your-federal-taxes-for-free /]

Our next activity is for you to select an online Free File offer. Again, we are not looking for information about you personally, but rather, your experience using the Free File Program.

To protect your confidentiality, I am going to give you a fictitious test scenario. The test scenario might not reflect your real-life situation, but I’d like to use it when you are selecting a Free File offer.

[GIVE SCENARIO DOC]

Here you will find a write-up about the scenario. Please use the information provided in the scenario to select a Free File software program to use to complete your tax return.

Please go ahead and find and select an online Free File offer with the test scenario in mind.

[INTERVIEWER: ALLOW P TO NAVIGATE SITE; ALLOW TO USE SOFTWARE TOOL IF DESIRED; STOP P AT POINT WHERE THEY WOULD SELECT A VENDOR]

[IF P dOES NOT SELECT LOOK UP TOOL, STOP THEM BEFORE THEY SELECT A VENDOR]:

[if needed] Thank you for picking a vendor. Now I’d like to back up a minute and look at site here [redirect to https://apps.irs.gov/app/freeFile/]. Did you notice this link to the Lookup Tool?

Now I’d like you to try to use the Lookup Tool to select a vendor, using the test scenario information I already gave you.

[ADMINISTER SINGLE QUESTION SEQ FOR LOOKUP TOOL ONLY, NOT PATH BEFOREHAND]

Debriefing Questions (based on participants selection)

1. Can you walk me through how you went through this process of looking for a vendor?

2. [if needed] I noticed that you expressed [confusion/hesitation/frustration] at [note point of difficulty]. Can you tell me more about that?

3. If used: What are your impressions about the “Lookup Tool”?

1. If needed, prompt: How did you decide whether or not to use the Lookup Tool?

2. [if taxpayer used lookup tool] How easy or difficult was it to use the Lookup Tool?

4. What are your impressions of the results returned from the lookup tool?

Free File Site Testing
Our next activity is for you to complete an online tax return using one of the software program offerings provided through the IRS Free File Program. I’d like you to pretend that you are completing your return at home as the person in the scenario. Please take the next minute to read through it.

[LOAD https://freefilealliance.org/ ON COMPUTER]

Using this website, please complete an online Federal tax return. You do not need to complete a state tax return. Please use the website to fill out the tax return as though we were not here. I understand that sometimes you might want to ask me a question, but again, my goal is to understand your experience completing the online tax return. If you have questions for me, we can discuss those when the interview is over. That said, if you wish to talk aloud about what you’re thinking as you go through the form, please feel free to do so. In fact, we encourage it.

Please go ahead and complete the online federal tax return. We will observe you and take some notes, and then I will have some questions about your experience when you finish.

Today I’d like you to test [SOFTWARE PROVIDER], so please select [SOFTWARE PROVIDER LINK].

[AFTER TASK, ADMINISTER SINGLE QUESTION SQE]

Debriefing

Now I will ask you a few questions to get a better sense of your experience with the Free File Program. There are no right or wrong answers to these questions.

Questions

1. Can you walk through the process that you followed to fill out your return?

2. Were there any aspects of the process that you found particularly easy or difficult?

3. [at points of confusion] I noticed that you expressed confusion/hesitation/frustration at [note point]. Can you tell me more about that?

4. Were there any instructions that were unclear? If so, what was unclear and why?

5. When completing the online return, were there any question or prompts you found to be confusing? If so, please tell me which ones and why they were confusing?

That concludes our testing activities. Now I have some general questions about your experiences with the Free File program.

[ADMINISTER UMUX QUESTIONS ABOUT ENTIRE PROCESS]

Overall Reactions

1. What, if anything, did you know about the Free File Program prior to today’s session?

2. Have you used the Free File Program previously? [if yes] When you used the Free File program previously, what were your overall impressions? Would you say that your experience was easy or difficult?

   1. [if needed] How did your previous experience compare to what you did today? Other than using a test scenario.

3. If you were completing the online tax return by yourself and couldn’t figure something out, what would you normally do?

4. Is there anything that could be done to make the process of filling out the form [and anything else from picking a vendor onwards] easier?

6. Is there anything that could be done to make using the IRS Free File program easier?

   1. [Deepen: Is there anything that could be done to make the tasks or websites you encountered in our session today easier?]

   2. [Deepen: Are there any changes you recommend to the IRS Free File page or the site you used to complete the online return?]

CLOSE OUT

Ok, that wraps up the study. The information you provided is extremely helpful and will allow us to make recommendations to the IRS. In a minute I will provide you a $100 gift card. First, do you have any questions about the study?

[ANSWER QUESTIONS]

[HAND PARTICIPANT GIFT CARD AND RECEIPT]

Here is your gift card. Please sign the receipt acknowledging receiving it.

That’s all we have. You’re free to go. Thank you very much for your help!

Page | 6