IRS W&I Adjustments Customer Satisfaction Survey Usability Testing OMB #: 1545-1349
Screener Guide
Ten 60 minute usability sessions total. Recruit 13 for 10 to show for total number of sessions.
Use survey participants that have indicated that they are interesting in participating in additional IRS-related research by providing their phone numbers at the end of the Adjustments survey. We would like a mix of:
Census geographic locations
Both Satisfied and Dissatisfied participants (this information will be pulled from survey responses to the overall satisfaction item Q1k).
Hello, my name is __________________and I work for Fors Marsh Group. We are a market research firm that is working with the IRS to schedule people to participate in one-on-one online interview sessions. We received your contact information from a survey you completed in [INSERT SURVEY DATE] about a correspondence that you had with the IRS regarding your tax return or amended return. We are contacting you today because we are looking for participants in an upcoming study. The study will be conducted remotely via phone and Internet.
Your identity will not be shared with the IRS or other agencies. We would like to ask you a couple of questions to see if you qualify for this study. Your participation is voluntary, but your help on this project would be very much appreciated.
Note to recruiter: If respondent asks were we got their contact info from tell them: “At the end of the Adjustments survey, you were given the option to provide your contact information if you were interested in participating in future research.”
First, we would like to know if you are interested in participating in this study.
1 Yes
2 No Thank and Terminate
Thank you, we would like to now ask you a few questions for qualification purposes.
Do you recall having contact with the IRS regarding your tax return or an amended return?
Yes
No Thank and Terminate
Do you feel that you can recall this experience well enough to answer some detailed questions on the process?
1 Yes
2 No Thank and Terminate
Do you have a computer at home?
1 Yes
2 No Thank and Terminate
Do you have Internet access at home?
1 Yes
2 No Thank and Terminate
How often do you use the Internet other than using e-mail?
1 Very often, daily
2 Often, several times a week
3 Not very often, once a week
4 Hardly ever, less than once a week Thank and Terminate
Have you ever used an online video chat program such as Skype or Google Video Chat?
1 Yes
2 No
Do you feel confident that you could download and use a video chat program if provided with detailed instructions?
1 Yes
2 No Thank and Terminate
If respondent qualifies for the study:
This study will take about 60 minutes to complete and we will give you $50.00 for your time. We would like to go ahead and schedule an interview time for you. [Provide Options].
Would you please provide us with your email address so that we can send you a reminder and instruction?
[EMAIL]
What online video chat program do you prefer?
[TEXT BOX]
The Paperwork Reduction Act required that IRS provide an OMB Control Number on all approved public information requests. That number is OMB 1545-1349. Also, if you like, I can give you a name and address where you can send comments and questions regarding this process or suggestions for making it simpler.
[Provide the following information only if respondent asks for address:]
Internal Revenue Service
Tax Products Coordinating Committee, Room #IR-6406
1111 Constitution Avenue, NW
Washington, DC 20224
Thank you and have a nice day. We look forward to speaking with you on (DATE) at (TIME)
IRS W&I Adjustments Customer Satisfaction Survey Usability Testing
Moderator’s Guide
Protocol – Usability Testing for Accounts Management/Adjustments Survey
Introduction
“Hello. Thank you for joining us today. My name is [Insert name of moderator] and I will be working with you today. We’ve invited you here so we can discuss your experience with your tax return or filing an amended return. What we are doing today is part of a larger effort being conducted by the Internal Revenue Service. I do not work for the Internal Revenue Service. I am an independent researcher who has been asked to help them learn more about taxpayers’ experiences with a specific branch of the IRS. Please be assured that nobody from the Internal Revenue Service will attempt to contact you after your participation today to discuss this topic and I will not be providing any of your personal information to them.
Today, we’ll ask you to answer questions that will be used in a national survey. The questions are about your correspondence and interactions with the IRS regarding your amended return, and the procedures involved in filing your return.
Since we are interested in your thoughts and reactions, I would like you to think aloud as you are figuring out the responses to the questions. So I would like you to tell me what you are thinking while you are working on your answers. Tell me how you are coming up with your answer and if words or terms in the question are confusing or unclear. This is important because we want to ensure that the questions are clear to people who are responding to them. Do you have any questions?”
“Let’s begin with a practice task. I am going to ask you a question, and I would like you to think aloud while you come up with your answer. Tell me what you are thinking as you determine your final answer. Ready?”
“How many doors are in your home?” Ensure that respondents are discussing how they are determining their answer. For example, they may say that there is a front and back door, and there is a door at the bathroom and at the bedroom…any others? No, that is all…So there are 4 doors total. Then ask if there are any more, such as on a closet or laundry room. Then ask if they would include cabinet doors and garage doors. The point is to get them to think through this question and consider all possible options.
When participants indicate something is unclear, probe about the perceived meaning of the item. Some potential probes are:
What does this mean to you?
Is this word/phrase/question clear?
How would you say this in your own words?
Why do you think that?
“Please take a moment to recall your most recent experience with filing an amended return.
What are a couple things that you can remember going right with the process?
What are a couple things that you can remember going wrong with the process?”
“We will now start working our way through a new version of the survey that you completed a few months back. Please keep in mind that we would like to hear about your experience with filing your amended return throughout the interview. For the items on the survey we would like you to let us know if they were applicable to your experience or if you feel that they don’t relate to the process.”
Do you recall having written contact with the IRS regarding your tax return within the last year?
Was it difficult for you to remember how long ago your most recent contact with the IRS was?
What does “written contact” mean to you?
Was the first contact made by you or by the IRS?
Was it difficult for you to remember who made the first contact?
Did you file an amended return to the IRS within the last year?
Yes, the notice prompted me to file the amended return
No, I filed the amended return, but not because I received a notice/letter
Are you familiar with the term amended return?
Was it difficult for you to answer this question?
Did you file the amended return because a notice or letter from the IRS prompted you to do so?
Yes, the notice prompted me to file the amended return
No, I filed the amended return, but not because I received a notice/letter
What does “prompted” mean?
Was it difficult for you to answer this question?
Regardless of whether you agree or disagree with the final outcome, how would you rate your overall satisfaction with the way your issue was handled?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
In your own words, what do you think this question is asking?
What experiences or criteria did you use to develop your answer?
What does overall satisfaction mean?
For the following questions, please focus on the outcome of the issue you recently resolved with the IRS.
Was it difficult for you to think of the issue that you recently resolved with the IRS?
6a) How satisfied were you with the final outcome of your recent issue with the IRS?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A= Don’t know/Not applicable
What does “final outcome” mean to you?
Why do you agree/disagree with the final outcome?
6b) How satisfied were you that your outcome was appropriate based on information you provided the IRS?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
In your own words, what do you think this question is asking?
Was it difficult for you to select a response option?
At the end of this section, ask:
Why do you agree/disagree with the outcome of your case?
Do you understand why this outcome was reached?
For the next set of questions, regardless of your satisfaction with the outcome of your issue, please focus on the process and procedures that the IRS used to address and resolve your issue.
In your own words, please explain what “regardless of your satisfaction with the outcome of your issue” means.
What do “process” and “procedures” mean?
7a) How satisfied were you with the ease of getting more information about your issue?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
How did you get more information about your issue?
What additional information about your issue did you get?
7b) How satisfied were you with the ease of providing information requested by the IRS?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
What information did the IRS request?
How did you provide this information?
What did you find easy or difficult about providing information requested by the IRS?
7c) How satisfied were you with the length of time it took to resolve the issue?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
What does “resolve” mean?
How long did it take to resolve your issue?
Did any correspondence with the IRS indicate that your issue would be resolved in a shorter timeframe?
7d) How satisfied were you with the extent to which the IRS used accurate information about you to process your issue?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
In your own words, what is this question asking?
Can you provide an example of how the IRS might use inaccurate information about you to process your issue?
At the end of this section ask:
How do you see the process working? What steps were involved for you?
For the next set of questions, regardless of your satisfaction with the outcome of your issue, please focus on the clarity and timeliness of the information regarding your issue that was provided by the IRS.
What does “clarity” mean?
What does “timeliness of the information regarding your issue” mean?
8a) How satisfied were you with the ease of understanding the initial notice and what was requested of you?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
Was it difficult for you to remember the initial notice you received?
What did the IRS request of you?
Please describe your experience with finding out about your amended return.
8b) How satisfied were you with the completeness of instructions you received for resolving your issue?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
What type of instructions did you receive?
[If dissatisfied] why were you dissatisfied with the instructions? What additional information would you have wanted?
8c) How satisfied were you with the ease of understanding responses from the IRS?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
What types of responses did you receive from the IRS?
8d) How satisfied were you with the IRS keeping you informed about the status of your case?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
How did the IRS keep you informed about the status of your case, or where did they fall short?
8e) How satisfied were you with the explanation regarding the resolution of your issue?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
How did you find about the resolution of your issue?
Why were you satisfied or dissatisfied with the explanation regarding the resolution of your issue?
[Repeat if participant needs prompting] “Again, please keep in mind that we would like to hear about your experience with the process throughout the interview. As with the previous items, we would like to you to let us know if they were applicable to your experience or if you feel that they don’t relate to the process.”
For the next set of questions, regardless of your satisfaction with the outcome of your issue, please focus on the personal treatment you received throughout the process of resolving your issue.
9a) How satisfied were you with the appropriateness of the tone of the IRS correspondence concerning your issue?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
How do you interpret the phrase “appropriateness of tone”?
Why was the tone of the correspondence appropriate or inappropriate?
9b) How satisfied were you with the politeness of any individuals you spoke with at the IRS concerning your issue?
1 = Very Dissatisfied
2 = Dissatisfied
3 = Neither Satisfied nor Dissatisfied
4 = Satisfied
5 = Very Satisfied
N/A = Don’t know/Not applicable
What does “politeness” mean to you?
How many individuals at the IRS did you speak with?
Were you treated differently by different individuals, or did all the individuals you interacted with treat you equally?
When you answered, were you thinking of all of the representatives you interacted with?
At the end of this section ask:
Can you please describe some of the correspondence and interactions that you had with the IRS during this process?
What was positive about them? What was negative?
If you were ‘very dissatisfied’ or ‘dissatisfied’ with any of the aspects in question 6-9 above, please provide a brief explanation of why you gave this rating in the box provided.
If you were dissatisfied, how did you decide what explanation to give?
Please mark the topic that best describes your main issue.
Status of refund
Status of payment
Penalty/interest charges
Earned income credit
Exemptions/Dependents
Name/Address change
Credits (child care, education, etc.)
Other changes to original return
Other
Are there response options missing from this list (besides “Other”) that you or someone you know might choose?
Are there any items on this list that you find confusing?
For this recent interaction, how many days elapsed between the time you submitted your correspondence or amended return and the time you received a reply?
Less than 15 days
15 – 29 days
30-44 days
45-60 days
Over 60 days
Did not receive a reply
Did you first think about your response in number of days, weeks, or months?
Was it difficult for you to remember the number of days you waited for a reply?
What do you think is a reasonable time frame to wait for the IRS to respond to your issue?
Less than 15 days
15 – 29 days
30-44 days
45-60 days
Over 60 days
How did you come up with this answer?
Why do you think this is a reasonable time frame?
Was your issue with the IRS completely resolved?
Yes
No
Not sure
What does “completely resolved” mean?
How many days did it take to resolve your issue from the time you were contacted, or were contacted, by the IRS about this issue?
Less than 15 days
15 – 29 days
30-44 days
45-60 days
Over 60 days
Did you first think about your response in number of days, weeks, or months?
How did you develop your answer to this question?
Was it difficult for you to answer this question?
Did you…?
Use a tax professional to assist you with resolving your issue
Represent yourself
Other
Are there response options missing from this list (besides “Other”) that you or someone you know might choose?
Did you use any of the following methods to contact the IRS about this issue? (Respondent can select multiple responses)
Toll-free line
In person
Other (please specify)
Are there response options missing from this list (besides “Other”) that you or someone you know might choose?
Have you contacted the IRS about the same issue for any prior year’s tax return?
Yes
No (skip to question 20)
How would you rate the level of service received from this contact versus previous contacts?
Better
Worse
The same
Was it difficult for you to remember the level of service received from previous contacts?
Please describe your experience with previous contacts. What made them better, worse, or the same as the last contact?
Regardless of the outcome of your case, how much do you agree with the following statements?
20a) I received an adequate description of the Adjustments process
1 = Strongly Disagree
2 = Disagree
3 = Neutral
4 = Agree
5 = Strongly Agree
N/A = Don’t know/Not applicable
How do you interpret the word “adequate”?
Who or what provided you with a description of the Adjustments process?
20b) My experience reflected the described Adjustments process
1 = Strongly Disagree
2 = Disagree
3 = Neutral
4 = Agree
5 = Strongly Agree
N/A = Don’t know/Not applicable
Who or what are you referring to when it asks for the “described Adjustments” process?
Why did you agree or disagree?
20c) I had the opportunity to provide information important to my case
1 = Strongly Disagree
2 = Disagree
3 = Neutral
4 = Agree
5 = Strongly Agree
N/A = Don’t know/Not applicable
How could you provide information about your case?
20d) I was treated with respect during the Adjustments process
1 = Strongly Disagree
2 = Disagree
3 = Neutral
4 = Agree
5 = Strongly Agree
N/A = Don’t know/Not applicable
What experiences or interactions did you take into account while answering this question?
What do you consider being “treated with respect”?
Overall, how well did the IRS meet your expectations while handling your Adjustments case?
1 = Much Worse Than Expected
2 = Worse Than Expected
3 = As Expected
4 = Better Than Expected
5 = Much Better Than Expected
Can you describe where your “expectations” were drawn from?
Can you explain what factors you considered while answering this question?
If you answered “Worse than expected” or “Much worse than expected” to the above question, can you describe what caused you to feel that way?
Use this space for comments or suggestions for improvements
At the end of the survey ask the participant items to better understand the experience that they had with filing their tax returns:
Does this survey seem to capture your experience with the Adjustments process?
Did this order of the survey items make sense to you?
Please describe some of the difficulties that you experienced.
Are there any aspects of the Adjustments process that you feel are missing from this survey that you would suggest including?
Was there anything that you feel the IRS did well on that was not included in this survey?
Were there any deficiencies that that you experienced throughout the process? What were they? Do you feel that they were captured on the survey?
Do you have any suggestions on how the IRS could better your experience with the Adjustments process?
| File Type | application/msword |
| File Title | ATTACHMENT A: USABILITY TESTING RECRUITER’S GUIDE |
| Author | mdsloa00 |
| Last Modified By | mdsloa00 |
| File Modified | 2012-06-07 |
| File Created | 2012-06-06 |