Attachment 1
Testing Scenarios
(questions to be used for pilot test if participants do not have their own questions, and to be used for the in-depth test event)
Attachment 1
Spanish and English Application “Guide to Federal Income Taxes” Testing
Test Scenarios
(Moderators will select from these scenarios for both the pilot and in-depth test events)
Task 1/Scenario #1: (for educational value)
You are an English-speaking taxpayer. You want to find out how to file a tax return. You go through the topics in the application to find one which might provide the information you need. You perform the exercise and take the quiz at the end of the topic. How many did you get right?
Task 1/Scenario #2: (educational value)
You are a taxpayer who only understands Spanish (and/or English). You don’t know it, but you have to leave in 30 minutes. How much of the application were you able to go through and retain a reasonable amount of information? (i.e., go through the application carefully enough to have tried some of its features and complete a topic)
Task 1/Scenario #3: (educational value)
You are a taxpayer who only understands Spanish (and/or English). You don’t want to go through the whole application, but only to get information about what to do if you receive an IRS notice or letter. Did you find out what to do from the application? How prepared do you feel to respond to the notice or letter after going through the application?
Task 2/Scenario #1: (presentation)
You are a taxpayer who only understands Spanish (and/or English). You find something in the application that you don’t understand. How easy is it to clarify your understanding?
Task 2/Scenario #2 (presentation)
You are a taxpayer who only understands Spanish (and/or English). You are reviewing a topic. Did you have to stop and go back at any point? Why?
Task 2/Scenario #3 (presentation)
You are a taxpayer who only understands Spanish (and/or English). You are reviewing a topic. You want to find out more information about something than the application gives. Does the application help you know how to get more information?
Task 2/Scenario #4 (presentation)
You are a taxpayer who only understands Spanish (and/or English). You need to go through the entire application. Are you able to in one sitting, or do you take breaks?
Task 3/Scenario #1: (user interface)
You are a taxpayer who only understands Spanish. You try to download a document (or form or publication) covered in the application. How successful are you?
Task 3/Scenario #2: (user interface)
You are a taxpayer who only understands Spanish (and/or English). You have very little understanding about how to operate a computer or computer-based application. Were you able to get the application to work without assistance? What did you have to do to figure it out?
Task 3/Scenario #3: (user interface)
You are a taxpayer who only understands Spanish (and/or English). You complete part of the application, then go to lunch. How easy is it to find your place again in the application?
Task 4/Scenario #4 (user interface)
You are a taxpayer who only understands Spanish (and/or English). You try 3 links in each topic of the application / try to download one document in each topic / invoke 3 pop-ups in each topic. Was the correct information arrived at? Do the links operate as you would expect them to? Did the link and the retrieved information correspond?
Attachment 2
Interview Questions
Attachment 2
Spanish and English Application “Guide to Federal Income Taxes” Testing
Interview Questions
Pilot Test Event
Did the application cover any questions about returns that you might have had before you tested it, or do you still have outstanding questions? (if so, what?)
Do you think you will now need less time to speak to the TAC assister?
Is the information explained such that it is easy to understand (by topic)?
How easy is the navigation? (easy? intuitive? clear?)
Interview Questions
In-Depth Test Event
Do you feel that you have a better understanding of tax returns, including what information you need, how to file, and benefits/consequences of compliance or non-filing?
Did the application cover any questions about returns that you might have had before you tested it, or do you still have outstanding questions? (if so, what?)
Is the amount of information overwhelming, thereby reducing knowledge retention? Are there appropriate points where the application(s) could be broken down into separate applications or topics?
Can the application be used one section at a time, or must you go through the whole thing to obtain value? (i.e., it is easy to come back to without losing context?)
Is the presentation compelling (interesting)?
Is the information explained such that it is easy to understand (by topic)?
Does it take an appropriate amount of time to go through each topic? (too long? too short?)
Is the presentation too busy or confusing in any way? Are all the features in the presentation distracting such that it detracts from learning?
Do all the features work properly as you would expect them to (e.g., do the hyperlinks go to where they say they’re going, do the definitions in the glossary correspond to what is being shown on the video screen?)
Are the instructions adequate?
How easy is the navigation? (easy? intuitive? clear?)
For bilinguals, do the English and Spanish versions correspond?
Attachment 3
Screener Guide (Pilot and In-Depth Tests)
Confirmation Letter (In-Depth Test only)
Reminder Phone Script (In-Depth Test only)
Attachment 3
Screener Guide
Pilot Test Event
INTRODUCTION
“Hello, my name is ___________. I am work for IRS. We are looking for people to test a new product we developed. It is an interactive computer application to teach people about tax returns. Before we release this application to the public, we would like to test it for real taxpayers’ reactions.
We’re talking to you now because we would like to know if you might be interested helping us test. Your input could help thousands of people get a better understanding of their tax responsibilities and know what to do.
Please understand this is a “test” of the application, not of you or your understanding of taxes. There is no “grading” of your answers.Participation is completely optional. If you do participate in the test, your responses are completely private, and will in no way affect your tax or any other standing with the government. You identity will not be obtained, nor given to IRS. IRS is interested in your reaction to the application, not your tax situation.
Testing will be done in [point to test location]. The test will take about 20 minutes, you will return to your same place in line (or put to the front of the line if your name was called), and you will be paid $15 for your time.
We are looking for people with certain backgrounds to test the application. We need people who speak _______________(only Spanish, only English, both English and Spanish). If you think you might be interested, I would like to ask you some questions to determine whether or not you are the kind of tester we are looking for.”
IF ASKED: (EACH INTERVIEWER WILL HAVE THIS INFORMATION)
If respondents ask how they were selected, tell them you are recruiting up to 10 people in the waiting area for that day. If respondents are concerned about participating, tell them their names or any other personal information will not be given them to the IRS or anyone else. Participation will not affect their taxes or filing status.
If respondents ask what the test will be like, tell them IRS is testing a computer application similar to a computer game. It is designed to teach Spanish- or English-speaking people about tax returns. They will be asked to try some of the application and later asked for their opinions.
This project has been approved by the U.S. Office of Management and Budget (OMB). The OMB Clearance Number is xxxx-xxxx. If they wish, you can give them a name and address where you can send comments and questions regarding these time estimates or suggestions for making this process simpler.
Internal Revenue Service
Tax Products Coordinating Committee
1111 Constitution Ave, NW IR-6526
SE:W:CAR:MP:T:T:SP
Washington, DC 20024
TEST DESCRIPTION
IRS developed an interactive computer application to teach people about Federal income tax returns. It is in both English and Spanish. It does not create a tax return. It teaches people about tax returns, like who needs to file one or how.
They will be asked to try the application to see if it helps them answer either a question they have or one IRS can give them. After they use it for a while, IRS will ask questions about how they liked it.
There are a few questions we would like to ask you to determine if you are the type of tester we are looking for. Your answers to these questions will be completely anonymous and we will not provide your name or any identifying information to the IRS.
Are you interested in continuing? (if yes, ask the following questions; if no, thank them for their time and wish them a good day)
SCREENING QUESTIONS
Are you fluent in Spanish?
( ) Yes Continue
until five people Spanish-only and three bilingual people per day are selected
( ) No Continue
Until two English-only people are selected
2. Do you [also] understand in English?
( ) Yes Continue
(in conjunction with question # 1, continue until three bilinguals and two English-only speakers are selected)
( ) No Continue
(in conjunction with question # 1, continue until five Spanish-only speakers are selected)
3. Are you (or were you) a tax professional? (Individual who prepares 25 or Federal tax returns each year or works in a tax-related occupation)
( ) Yes ……………………………………………………………………................. Terminate
( ) No Continue
4. Might you, or have you ever filed a Federal income tax return?
( ) Yes ……… ……………………. Continue
( ) No Terminate
5. Are you familiar with using computers and software?
( ) Yes ………………............................................... Continue
( ) No Terminate
6. Are you over 18 years of age?
( ) Yes ………………............................................... Continue
( ) No Terminate
For questions 7-14, recruit a mix of individuals. Answers will not affect their eligibility to participate.
7. Into which of the following categories does your age fall?
( ) 18 to 33
( ) 34-44
( ) 45 to 64
( ) 65 and older
Please identify your country of origin: ____________________________________________
Please identify the primary language spoken in your household (if single, primary language used) : _________________________________________________________________
Please identify your highest completed level of education. [may opt-out if they wish to]
( ) Less than 9th grade
( ) 9th grade to 12th grade, no diploma
( ) High school graduate (or GED)
( ) Some technical or vocational school
( )Technical or vocational school graduate
( )Some college, no degree
( )Associate Degree
( )Bachelor’s Degree
( ) Master’s Degree
( )Post-Master’s Degree
So that we can be sure that all backgrounds are represented in our project, could you please tell me the race and ethnic origin you identify with [ may opt-out if they wish to].
11. Are you of Hispanic or Latino origin?
( ) Yes
( ) No
12. What is your race? Please select one or more. [1 or more responses OK; may opt-out if they wish to].
( ) American Indian or Alaska Native
( ) Asian
( ) Black or African American
( ) Native Hawaiian or Other Pacific Islander
( ) White
Which of these categories does your total annual household income fall into? [may opt-out if they wish to].
13
( ) Less than $10,000
( ) $10,000 but less than $15,000
( ) $15,000 but less than $25,000
( ) $25,000 but less than $35,000
( ) $35,000 but less than $50,000
( ) $50,000 but less than $75,000
( ) $75,000 but less than $100,000
( ) $100,000 or more
Record Gender— DO NOT ASK IF UNABLE TO DETERMINE; may opt-out if they wish to].
14
( ) Female
( ) Male
Have you ever gone to an IRS Taxpayer Assistance Center before to resolve a tax issue or have your return prepared?
( ) Yes
( ) No
INVITATION
“Thank you for answering our questions. Based on your responses, we would like to invite you to participate in the test which will be held [indicate location]. The total time should be about twenty minutes. After the test you will return to your same place in line or, if it has passed, moved to the front of the line. You do not need to bring any information about yourself. Your participation is completely voluntary.
Are you interested in participating?
( ) Yes Continue
( ) No Terminate
I’m glad that you will be able to join us! We will provide you with a [$15] stipend completing the test.
Attachment 3
Screener Guide
In-Depth Test Event
If suitable candidates are identified but not needed because of sufficient numbers, ask them if they would like to be an alternate. If so, collect contact information.
INTRODUCTION
“Hello, my name is ___________. I am calling on behalf of ICF Macro, a research and consulting firm in the Washington DC area. We are looking for people to test a new product developed by the IRS. It is an interactive computer application to teach people about tax returns. We are calling to see if you would be interested in testing this application. Your input could help thousands of people get a better understanding of their tax responsibilities and know what to do.
Please understand this is a “test” of the application, not of you or your understanding of taxes. There is no “grading” of your answers. Participation in this test is completely optional. If you do participate in the test, you will receive a stipend. Your responses will be kept completely private, and will in no way affect your tax or any other standing with the government. You identity will not be obtained, nor given to IRS. IRS is interested in your reaction to the application, not your tax situation.
Testing will be held in ____________(where) on _______________ (when). The test will last about 3 hours, and you will be paid for your time.
We are looking for people with certain backgrounds to test the application. We need people who speak _______________(only Spanish, only English, both English and Spanish). If you think you might be interested, I would like to ask you some questions to determine whether or not you are the kind of tester we are looking for.”
IF ASKED: (EACH INTERVIEWER WILL HAVE THIS INFORMATION)
If respondents ask how their name was obtained, tell them their phone number was randomly selected from a list of individuals in the local calling area. If respondents are concerned about participating, tell them their names or any other personal information will not be given them to the IRS or anyone else. Participation will not affect their taxes or filing status.
If respondents ask what the test will be like, tell them IRS is testing a computer application similar to a computer game. It is designed to teach Spanish- or English-speaking people about tax returns. They will be asked to try some of the application and later asked for their opinions.
This project has been approved by the U.S. Office of Management and Budget (OMB). The OMB Clearance Number is xxxx-xxxx. If they wish, you can give them a name and address where you can send comments and questions regarding these time estimates or suggestions for making this process simpler.
Internal Revenue Service
Tax Products Coordinating Committee
1111 Constitution Ave, NW IR-6526
SE:W:CAR:MP:T:T:SP
Washington, DC 20024
TEST DESCRIPTION
“IRS developed an interactive computer application to teach people about Federal income tax returns. It is in both English and Spanish. It does not create a tax return. It teaches people about tax returns, like who needs to file one or how to file one.
We will be asking you to see if it helps you answer certain questions. After you use it for a while, we will be asking you questions about how you liked it.
There are a few questions we would like to ask you to determine if you are the type of tester we are looking for. Your answers to these questions will be completely anonymous and we will not provide your name or any identifying information to the IRS.
Are you interested in continuing?” (if yes, ask the following questions; if no, thank them for their time and wish them a good day)
SCREENING QUESTIONS
Are you fluent in Spanish?
( ) Yes Continue
( ) No Continue
2. Do you [also] understand in English?
( ) Yes Continue
( ) No Continue
3. Are you (or were you) a tax professional? (Individual who prepares 25 or Federal tax returns each year or works in a tax-related occupation)
( ) Yes ……………………………………………………………………................. Terminate
( ) No Continue
4. Might you, or have you ever filed a Federal income tax return?
( ) Yes ……… ……………………. Continue
( ) No Terminate
5. Are you familiar with using computers and software?
( ) Yes ………………............................................... Continue
( ) No Terminate
For questions 6-14, recruit a mix of individuals
6. Into which of the following categories does your age fall? [may opt-out if they wish to].
( ) less than 18 Terminate
( ) 18 to 33
( ) 34-44
( ) 45 to 64
( ) 65 and older
Please identify your country of origin: ____________________________________________
Please identify the primary language spoken in your household (if single, primary language used) : _________________________________________________________________
Please identify your highest completed level of education [may opt-out if they wish to].
( ) Less than 9th grade
( ) 9th grade to 12th grade, no diploma
( ) High school graduate (or GED)
( ) Some technical or vocational school
( )Technical or vocational school graduate
( )Some college, no degree
( )Associate Degree
( )Bachelor’s Degree
( ) Master’s Degree
( )Post-Master’s Degree
So that we can be sure that all backgrounds are represented in our project, could you please tell me the race and ethnic origin you identify with [1 or more responses OK; may opt-out if they wish to].
10. Are you of Hispanic or Latino origin? [may opt-out if they wish to].
( ) Yes
( ) No
11. What is your race? Please select one or more [1 or more responses OK; may opt-out if they wish to].
( ) American Indian or Alaska Native
( ) Asian
( ) Black or African American
( ) Native Hawaiian or Other Pacific Islander
( ) White
Which of these categories does your total annual household income fall into? [may opt-out if they wish to].
( ) Less than $10,000
( ) $10,000 but less than $15,000
( ) $15,000 but less than $25,000
( ) $25,000 but less than $35,000
( ) $35,000 but less than $50,000
( ) $50,000 but less than $75,000
( ) $75,000 but less than $100,000
( ) $100,000 or more
Have you ever gone to an IRS Taxpayer Assistance Center to resolve a tax issue or have your return prepared?
( ) Yes
( ) No
Record Gender—DO NOT ASK IF UNABLE TO DETERMINE
( ) Female
( ) Male
INVITATION
“Thank you for answering our questions. Based on your responses, we would like to invite you to participate in the test which will be held in [insert city] [insert date] at [insert time]. The total time will be about three hours plus the time it takes to arrive and return from the testing location. We will provide the materials you need to participate in the testing. You do not need to bring any information about yourself. Your participation is completely voluntary.
Are you interested in participating?”
( ) Yes Continue
( ) No Terminate
“I’m glad that you will be able to join us! We will provide you with a [$100-$125] stipend upon completing the test.
At this point I need some contact information from you. We will send you a confirmation letter and directions to the facility via mail, or e-mail if you have it.
We are only inviting a few people, so it is very important that you notify us as soon as possible if for some reason you are unable to participate. Please call [insert recruiter contact and phone] if this should happen. We look forward to having you participate on [insert day] at [insert time].
Do you have any questions?
Great! Thank you for your time and we will be in touch again. Please remember to bring reading glasses to the test if you use them.”
TERMINATE TEXT
“Thank you very much for your time, and thank you for answering our questions. Unfortunately, based on your answers, we cannot extend you an invitation. Perhaps in the future we can include you in a test of this type. Have a good [day/evening].”
Attachment 3
Confirmation Letter
In-Depth Test Event Only
<contact information>
< date>
Dear <name>:
Thank you for agreeing to test a tax-return-related computer application on <insert date> at <insert time> at <insert facility name>. This letter is a reminder and gives you the information you need to participate.
Please arrive no later than [time]. During the15 minutes before the start of the test, you will be asked to complete some paperwork and receive an explanation of what you will do. Snacks and refreshments will be provided.
The directions to the facility are attached. We are accessible by public transportation [insert bus number/subway station as appropriate]. If you would like to drive, parking is available [insert driving directions, and information about parking and any costs].
As a thank you for participating, you will be given [$amount] at the end of the test. If you have decided not to participate in this test, please let us know as soon as possible (contact information above) so that we can find a replacement.
Please understand this is a “test” of the application, not of you or your understanding of taxes. There is no “grading” of your answers. If you have any questions, please call <insert facility manager’s name> at <facility phone number>.
Thank you for your time and participation.
[Insert facility manager’s name]
[Insert name of facility] on behalf of ICF Macro.
Reminder Phone Script
In-Depth Test Event Only
To be used no more than 24 hours prior to the scheduled test. If participant is not available, leave voice mail message and try back later.
Hello, may I speak to <insert name>? Hi, my name is <insert recruiter’s name> and I work for IFC Macro. I’m calling to remind you about the testing you agreed to participate in tomorrow at <insert time> at <insert location>. Are you still able to participate? Great! [If no, thank them for their time and notify manager to try to find an alternate tester] I <mailed/e-mailed> you directions to our facility. Did you receive the directions? [If no, read directions and follow with an e-mail.] [If yes] Do you have any questions about how to get to the facility?
Please remember to bring reading glasses if you use them. If you need to cancel for any reason, I would appreciate if you would call me at [insert phone number].
We look forward to seeing you tomorrow.
Attachment 3
Moderator’s Guide
Attachment 3
Moderator’s Guide
Pilot Test Event
[Note: Information in regular type is the moderator’s “script.” Information in italics is notes to the moderator].
Spanish and English Application “Guide to Federal Income Taxes” Testing
[Moderators will work in a one-on-one fashion unless several participants are available at the same time]
INTRODUCTION
Introduction of the moderator(s) [name, affiliation, what moderator will do, including that one is fluent in Spanish].
Explanation of the test — to get taxpayer feedback on a Spanish-English interactive computer application, “Guide to Federal Income Taxes.”
Explanation of the purpose of the test — to explore, in a semi-structured manner, what people think about the application by:
Using the application, to try to answer some questions and run through some scenarios
Discussing the tester’s experiences, impressions of, and suggestions for the application
“Playing with” the application to see the effectiveness of its interactive features
Asking questions or making comments freely.
Please understand this is a “test” of the application, not of you or your understanding of taxes. There is no “grading” of your test.
We will ask you to work on the application and then ask you questions about your experience. I will explain more shortly.
Are there any questions at this point?
BASIC GROUND RULES:
Everything you say here will be kept strictly private. You will remain anonymous to the IRS. Participation does not affect your tax situation or status with IRS or any other government agency.
We are required to have approval from the Office of Management and Budget for IRS to speak with the public (you). Their approval number for this project is xxxx-xxxx. Also, if you have any comments regarding the time estimates associated with this study or suggestions on making this process simpler, please write to the: IRS Tax Products Coordinating Committee, 1111 Constitution Avenue, NW, IR-6526, Washington, DC 20224. [have the OMB clearance number and contact address on a board, or on the handouts]
We may make audio tapes of your one-on-one interviews. If you are concerned about your voice being recognized, you may request that the session not be taped. We will, however, need a moderator to take notes in order to capture your comments.
You may be in a room with other participants who are doing something similar to, but not the same as, you. We ask you to refrain from side conversations when you are testing. (the need for this comment depends on how the test site is set up).
Moderators will be around to answer questions. They speak both Spanish and English.
None of us are in tax law experts so we cannot answer tax questions.
There are no right and wrong answers. We are testing the application, not you.
If you have a cell phone, please turn it off, or set it to vibrate.
The test will last 15-20 minutes. This includes talking to you after you look at the application.
Are there any questions at this point?
SCENARIOS AND DISCUSSION
The application teaches you about Federal income tax returns. It does not produce a tax, it teaches the user about the tax return process.
There is one complete version in Spanish and one in English. Each version is divided into six topics plus instructions. We will be testing both versions.
You will sit at a computer and the application will be running. There will be a form for you to fill out as you are going through your test.
What will your test be? We would like you to see if the application answers the questions or addresses the issues you came to this TAC for. If your own question isn’t related to tax returns, we have a list of questions you can choose from.
Please write the question and your language on the form. Fill out one form per question if you have time for more than one.
The first thing we would like you do is to spend some time becoming familiar with the format and features of the application. We will give you a few minutes to “play with” the application. We will not explain the features or operation of the application because one of the things we are testing for is how easy it is to learn. If we find that someone is truly stuck, one of the moderators will come over to help.
While you are completing your question, please consider the look, layout, and language of the application. We will discuss these items after you complete your work.
Are there any questions at this point?
[Lead participants to the testing area and get them set up to test. Tell them when to begin the test, if necessary.]
INTERVIEWS
[After the participants worked with the application for about 10-15 minutes, ask them to wrap up and guide them to a place where you can ask them the following interview questions:]
Is the information explained such that it is easy to understand (by topic)?
How easy is the navigation? (easy? intuitive? clear?)
FALSE CLOSE
Explain that participants can discuss the application among themselves while you check in with the observers to see if they have any addition questions they would like you to ask.
CLOSE
Thinking about what you saw, is there anything else you would like to share regarding the application we looked at today?
THANK YOU/WRAP UP
Thank you very much for coming this evening, and for sharing your ideas with us—we really appreciate your time
[Give incentive, and then direct participants back to the waiting room. Check to see that they haven’t missed their appointment. If they have, work with the TAC receptionist to move them up in line.]
WORKING WITH THE TAC ASSISTER
[After the participant completes his/her meeting with the TAC assister, ask the assister whether the participant seemed well informed about the issue, and whether the participant mentioned that the application helped them with the reason they came in.]
In-Depth Test Event
[Note: Information in regular type is the moderator’s “script.” Information in italics is notes to the moderator].
Spanish and English Application “Guide to Federal Income Taxes” Testing
INTRODUCTION
Introduction of the moderator(s) [name, affiliation, what moderator will do, including that one is fluent in Spanish].
Explanation of the test — to get taxpayer feedback on a Spanish-English interactive computer application, “Guide to Federal Income Taxes.”
Explanation of the purpose of the test — to explore, in a semi-structured manner, what people think about the application by:
Using the application, to try to answer some questions and run through some scenarios
Discussing the tester’s experiences, impressions of, and suggestions for the application
“Playing with” the application to see the effectiveness of its interactive features
Asking questions or making comments freely.
Please understand this is a “test” of the application, not of you or your understanding of taxes. There is no “grading” of your test.
We will ask you to work on the application and then ask you questions about your experience. I will explain more shortly.
Are there any questions at this point?
BASIC GROUND RULES:
Everything you say here will be kept strictly private. You are free to tell me what you think. We use first names only, and names will not be used in the findings report for IRS. You will remain anonymous to the IRS. Participation does not affect your tax situation or status with IRS or any other government agency.
We are required to have approval from the Office of Management and Budget for IRS to speak with the public (you). Their approval number for this project is xxxx-xxxx. Also, if you have any comments regarding the time estimates associated with this study or suggestions on making this process simpler, please write to the: IRS Tax Products Coordinating Committee, 1111 Constitution Avenue, NW, IR-6526, Washington, DC 20224. [have the OMB clearance number and contact address on a board, or on the handouts]
We may make audio tapes of your one-on-one interviews. If you are concerned about your voice being recognized, you may request that the session not be taped. We will, however, need a moderator to take notes in order to capture your comments.
If we tape your discussion, I ask that you speak loudly and clearly. If I think you are speaking too softly to be heard on the tape, I will ask you to speak up.
We ask you to refrain from side conversations when you are testing. You will be in a room with other participants who are doing something similar to, but not the same as, you. (the need for this comment depends on how the test site is set up).
Moderators will walk around during the testing to answer questions or address comments. They will speak both Spanish and English.
Participation—we need to get a full range of opinions, and we need everyone’s participation. You are here to talk. I am here to listen. Be aware that none of us are in tax law experts so we cannot answer questions that involve tax law.
There are no right and wrong answers. You are free to express your opinions.
If you have a cell phone, please turn it off, or set it to vibrate. We don’t want the group to be disturbed and we also need your full attention.
The test will last about 2 ¼ hours and you may take breaks if you wish. The interview will take about ¾ hour and you may request a break if necessary.
Are there any questions at this point?
INTRODUCTION OF PARTICIPANTS
Go around the room and ask each participant why they were interested in the testing. [This section is optional, at the moderators’ discretion if “ice-breaking” is needed]
SCENARIOS AND DISCUSSION
The application is a tutorial about Federal income tax returns. It does not produce a tax return at the end. Instead, it teaches the user about the tax return process. There is one complete version in Spanish and one in English. Each version is divided into six topics plus instructions.
We will be testing both versions. Some of you only speak Spanish, some only speak English, and some speak both languages. Part of our test is to see if there are differences in your opinions of it based on what language you speak.
Each of you will sit at your own computer and the application will be running. There will be a form you to fill out as you are going through the scenario or scenarios we assign to you. Please write the scenario and your language on the form. Fill out one form per scenario.
The first thing we would like you do is to spend some time becoming familiar with the format and features of the application. We will give you a few minutes to look around it. We will not explain the features or operation of the application because one of the things we are testing for is how easy it is to learn and use. If we find that someone is truly stuck using it, one of the moderators will come over to help.
Now let’s take a look at the scenarios. (Distribute copies of the scenario). Each one of you will be given one or more of these to explore. We would like you to address the issues of the scenario(s) using the application.
(Read scenarios to participants. Please point out that they (the participants) represent taxpayers and should complete tests the same they would if they were the taxpayer.)
Each scenario presents a situation or a problem to be solved. Be sure to use the instructions provided to complete the exercise. You will have about 20-25 minutes to complete the scenario.
The first thing we would like you do is to spend some time becoming familiar with the format and features of the application. We will give you a few minutes to “play with” the application. We will not explain the features or operation of the application because one of the things we are testing for is how easy it is to learn. If we find that someone is truly stuck, one of the moderators will come over to help.
While you are completing your scenario, please consider the look, layout, and language of the application. We will discuss these items after you complete your scenario.
Are there any questions at this point?
[lead participants to the testing area and get them set up to test. Tell them when to begin the test, if necessary.]
INTERVIEWS
[After the participants worked with the application for about 2.5 hours, or if they finished some work and look like they are no longer interested in working on the application, ask them to wrap up and guide them to a place where you can ask them the following interview questions:]
Do you feel that you have a better understanding of tax returns, including what information you need, how to file, and benefits/consequences of compliance or non-filing?
Did the application cover any questions about returns that you might have had before you tested it, or do you still have outstanding questions? (if so, what?)
Is the amount of information overwhelming, thereby reducing knowledge retention? Are there appropriate points where the application(s) could be broken down into separate applications or topics?
Can the application be used one section at a time, or must you go through the whole thing to obtain value? (i.e., it is easy to come back to without losing context?)
Is the presentation compelling (interesting)?
Is the information explained such that it is easy to understand (by topic)?
Does it take an appropriate amount of time to go through each topic? (too long? too short?)
Is the presentation too busy or confusing in any way? Are all the features in the presentation distracting such that it detracts from learning?
Do all the features work properly as you would expect them to (e.g., do the hyperlinks go to where they say they’re going, do the definitions in the glossary correspond to what is being shown on the video screen?)
Are the instructions adequate?
How easy is the navigation? (easy? intuitive? clear?)
For bilinguals, do the English and Spanish versions correspond?
Did the application cover any questions about returns that you might have had before you tested it, or do you still have outstanding questions? (if so, what?)
Do you think you will now need less time to speak to the TAC assister?
FALSE CLOSE
Explain that participants can discuss the application among themselves while you check in with the observers to see if they have any addition questions they would like you to ask.
CLOSE
Thinking about what you saw, is there anything else you would like to share regarding the application we looked at today?
THANK YOU/WRAP UP
Thank you very much for coming this evening, and for sharing your ideas with us—we really appreciate your time
[Direct participants to area outside of the room to pick up their incentive.]
| File Type | application/msword |
| File Title | OMB Package |
| Author | mawueb43 |
| Last Modified By | qhrfb |
| File Modified | 2010-07-20 |
| File Created | 2010-07-19 |