OMB 1545-1432 Customer Satisfaction
Taxpayer Advocate Service (TAS) 2010 Evaluative Focus Groups
The Taxpayer Advocate Service (TAS) is looking to identify ways to improve business results within the IRS by working with taxpayers identified as underserved by TAS. Communications and Liaison (C&L) is working with an outside vendor, Porter Novelli Public Services (PNPS), to better understand how to communicate with these audiences about receiving TAS assistance. In fall 2009, TAS conducted formative discussion groups with these underserved audiences to obtain their thoughts and opinions, particularly to gauge relevance and comprehension of our current mission statement and other messages, and how different segments of TAS target audiences interpret this information. Specifically, TAS shared various versions of the mission statement and other draft messages with the participants to determine the extent to which respondents correctly interpreted current language and what elements of alternative language fostered increased resonance among TAS target audiences.
These groups successfully met the aforementioned goals and gleaned useful data to inform future messaging development. In conjunction with PNPS, TAS is developing revised messaging and promotional materials to be used among key target audiences. To ensure that TAS allocates its budget efficiently, it wants to garner 1) whether these promotional materials communicate as intended, 2) whether images and other visual aids improve overall comprehension, and 3) whether the materials increase the likelihood that TAS target audiences will solicit TAS assistance when they have an appropriate tax issue.
To meet this need, the Taxpayer Advocate Service would like to conduct twelve focus groups in February 2010. The focus groups will be held in Richmond, VA and Chicago, IL to facilitate geographic and socio-economic diversity over the course of the project.1 Six groups each will be held in these locations. Although the specific dates for the sessions have not been determined, it is TAS’s intention to hold these groups in February to allow findings to inform TAS strategy associated with the 2010 tax season and to ensure that sessions do not complicate data collection associated with the 2010 U.S. Census.
The formative phase of this research program employed triads for qualitative data collection. Those sessions necessitated an intense review of multiple statements enabling the moderator to probe for specific thoughts and feelings that underlie preferences for each statement. Since this phase of research requires a review of executional materials, focus groups will be utilized. Focus groups enable the research program to balance the need to present materials to multiple respondents with the opportunity to garner justification for preferences and to determine strengths and weaknesses of the materials. The results of the focus groups will be presented to the appropriate areas of TAS.
The C&L division within TAS has been working with PNPS to identify the four priority underserved audiences of TAS. PNPS reviewed two previous TAS studies conducted by Russell Research as well as other datasets (e.g., Simmons Two-Year Combined Study) to identify a set of TAS target audiences defined by socio-demographic segmentations, with the primary determinants being income and marital status. The TAS Underserved segments are as follows:
Middle-Income Families
Affluent Families
Low-Income Families
Financially Distressed
As was the case with the formative round of research, TAS would like to conduct focus groups within these underserved audiences:
Four with mixed Middle/Affluent-Income Families
Four with Low-Income Families
Four with Financially Distressed Individuals
Effectiveness of Taxpayer Advocate Service (TAS) Messages and Its Presentation in Audience-Facing Materials
TAS is interested in improving the effectiveness of its consumer-facing communications materials. This will involve testing revised messages for 1) comprehension, 2) effectiveness in conveying how TAS can help and 3) establishing trust. It will also involve the assessment of a consumer-facing brochure which will be evaluated for its clarity, attractiveness, and its ability to 1) communicate when target audience members should involve TAS, and 2) increase the likelihood that the target audience members will involve TAS if they face a tax issue in the future.
The overall objective of this project is to provide qualitative data to TAS on materials designed to communicate TAS’ mission and capacity of TAS to assist taxpayers with their tax problems. The specific business questions to be explored by this project include:
What are the opinions of the underserved audiences regarding the effectiveness of revised TAS messages?
How effectively to TAS materials communicate these messages?
Do fears/concerns associated with the upcoming tax season impact the ways in which the underserved audiences react to these materials?
Do these materials garner attention from TAS audiences?
In what ways can these materials be improved to maximize their effectiveness?
The project will encompass: recruiting participants, moderating the focus groups, and preparing a report for TAS summarizing the qualitative feedback from underserved participants. Comments will be used to refine overall messaging and improve/finalize outreach materials.
To accomplish the objectives of this project we will conduct 12 focus groups in conjunction with our vendor, PNPS. These discussion sessions will be held in Richmond, VA and Chicago, IL.
Focus groups are directed discussion on specific messages and language used by TAS with a small group of no more than eight people. Focus groups employ group interaction to stimulate relatively spontaneous responses to the supplied topics and creative materials. The goal is to explore the feelings, opinions, and beliefs people hold, and to learn how these feelings shape their responses to both TAS materials and overt behavior. This methodology provides insight and direction rather than quantitatively precise or absolute measures. They are useful for gathering information and generating hypotheses based on participants’ opinions. The main advantage focus groups offer is the opportunity to observe considerable interaction on a topic in a limited period of time, and among a larger number of individuals than triads, dyads, or in-person interviews allow. In fact, the number of individuals per session affords considerable reaction and subsequent probing on tested materials and increases the opportunity for multiple opinions and reactions to be gleaned. An important aspect of these groups is that they give rise synergistically to insights and solutions that may not come about without them, and they serve as a useful red-flag check to unintended interpretations of materials.
Focus groups are also a useful methodology in that they create a comfortable environment in which respondents are not daunted by many individuals in the same room. As the guide indicates (see Appendix D), a number of responses will be gleaned via worksheet to reduce the possibility of “group think” and it creates a dynamic that enables the moderator to better control respondents that seek to dominate the discussion.
The market segments for this project consist of the identified underserved TAS audiences: Middle-Income/Affluent Families,2 Low-Income Families, and the Financially Distressed. TAS executives and managers will use the data gathered during the focus group interviews to better finalize marketing materials designed to reach these important audiences.
Our step by step approach follows:
Step 1 – We met with PNPS to discuss the findings from the formative research phase and to determine the objectives for this follow-on project. After clarifying objectives for the qualitative sessions PNPS developed screening questionnaires (see Appendices A, B, and C) to recruit participants and a moderator’s guide3 (see Appendix D) to be used during the focus groups.
Step 2 – So that we obtain a diverse group of participants PNPS prepared a demographic sheet (see Appendix E) to be used by professional recruitment facilities in each location to track recruitment. The facilities will use the demographic sheet and the screener’s guide to solicit and select focus group participants.
Step 3 – Professional moderators contracted by PNPS will conduct the discussion sessions. All focus groups will be audio-taped and video-taped (participants informed in advance). Their identities will be kept private.
Step 4 – Notes taken while watching all focus group sessions and on-site recordings will serve as the data collection method and the foundation of the analysis. Porter Novelli Public Services will review each set of data for recurring themes identified during the review. More specifically, study analysts will review notes and responses to handout exercises to identify patterns in participants’ answers to the questions in each segment of the discussion guide. (See Appendix F for handouts used during the discussions.) Recordings will be reviewed to explore additional themes and to confirm results. Differences across gender and target audience will be noted in the report when appropriate; however, given the small number of participants in this research, analyses focus on commonalities across groups, as opposed to within specific demographic characteristics. No respondents’ names will be used in reporting documents. Upon conclusion of the research, PNPS will submit a summary report of findings.
It should be noted that focus groups are a qualitative research technique. Qualitative research provides valuable insight into a particular group’s thoughts, feelings, and perspectives and is especially useful in identifying “red flags” or unintended interpretations of creative material. Recruiting techniques and small sample sizes, however, mean that results are not statistically representative of a larger population. Consequently, findings will be considered descriptive and directional but not definitive.
We want to include a diverse mix of our population based on the following aspects:
Gender
Self-identified as “bad money managers” (assessed via responses to screening items)
Self-identified as having a “life-changing event” in past 12 months (assessed via responses to screening items)
Household income
Marital status
The discussions will gather qualitative data only. The data will not be, nor will it be presented to be, representative of the population. The study will obtain geographic diversity via the different cities used for the focus groups.
TAS proposes to use the qualitative sessions to identify and capture a range of ideas regarding revised TAS messaging surrounding TAS’ purpose and when individuals should seek TAS services, with the goal to increase the likelihood that these audience view TAS as a viable option to help solve their tax-related problems. This messaging will be placed in a draft brochure for participants to review and evaluate. Resulting data will be used to provide final revisions to these materials and to correct any unintended interpretations of the messaging and supporting content in the marketing materials.
The participants’ comments will be used to provide insight into TAS underserved audiences’ understanding of our purpose and how our relationship to the IRS strengthens our ability to solve tax problems. Specifically, they will gauge the effectiveness of draft marketing materials and will serve as a communications/red-flag check of the information provided in them. The results are not designed to, nor is it expected that they will, be projected to the population for formulation of conclusions about the general population. It is anticipated that the study objectives can be met in a qualitative, not quantitative measure.
The focus groups are planned for mid to late February 2010. The groups should be completed over the course of one week. Three groups would be conducted per day, with one day allowing for travel between focus group locations.
Porter Novelli Public Services is responsible for recruiting and screening participants, preparing screener and moderator guides, moderating the groups, and compiling and summarizing the results. They have research analysts trained in all aspects of qualitative research, including moderating the focus groups, analyzing the qualitative data, and preparing report findings. Focus groups will be conducted in Richmond, VA and Chicago, IL.
The contract with the vendor is for $108,000. There is also an estimated cost of $1,600 for travel of IRS personnel for a total estimated cost of $109,600. This includes all vendor costs and IRS costs for travel, printing, postage, and overtime payments (but not regular salary payments).
Participants will receive a cash stipend for their participation ($75-$100), intended to reimburse for expenses such as transportation, child care services, etc. The incentive amount will be standard for the locations and a recruitment of this nature.
The vendor will be responsible for recruiting the focus group participants. The vendor has agreed to use specially designed screeners (see Appendices A, B, & C) to ensure the participants meet the requirements of our study.
The Taxpayer Advocate Service would like to conduct 12 focus groups in February 2010. The focus groups will be held in Richmond, VA and Chicago, IL, with six sessions held in each location. The groups will be held in facilities selected by the vendor. All facilities will be suitable for conducting focus groups.
The expected response rate is 10%. This is based upon the vendor’s prior experience in conducting similar research. With regard to the low response rate, the IRS will assume that all data collected from this survey is qualitative in nature, and that no critical decisions will be made by TAS solely from the analysis of data from this survey. The results from this survey are simply one piece of a larger set of information needed to assess underserved taxpayer needs related to services provided by TAS.
A stipend will be provided to all participants who show up and participate in the focus groups. This incentive has proven itself as a valuable tool for increasing participation rates.
Focus groups are a directed discussion on specific messages and draft marketing materials used by TAS with a small group of no more than eight people per session. As a type of small group interview, these sessions use group interaction to stimulate relatively spontaneous responses to the supplied topics. The goal is to explore the feelings, opinions, and beliefs people hold, and to learn how these feelings shape reactions to tested materials and overt behavior. Focus groups provide insight and direction rather than quantitatively precise or absolute measures. They are useful for gathering information and generating hypotheses based on participants’ opinions. And, they are very useful as “red-flag” checks in that they can efficiently explore whether any proposed messaging or materials garner unintended interpretations. The main advantage focus groups offer is the opportunity to observe considerable interaction on a topic in a limited period of time, and among a larger number of individuals than triads, dyads, or in-person interviews allow. In fact, the number of individuals per session affords considerable reaction and subsequent probing on tested materials and increases the opportunity for multiple opinions and reactions to be gleaned. An important aspect of these groups is that they give rise synergistically to insights and solutions that may not come about without them.
In addition, these groups are designed to include exercises that are completed via handout. This technique allows each respondent to answer questions independently, before questions are opened for group discussion. It also reduces the possibility of “group think” and it creates a dynamic that enables the moderator to better control respondents that seek to dominate the discussion.
The formative phase of this research project provided invaluable insight into techniques, words, and phrase TAS should consider when communicating its mission and criteria for participation. This initial phase of research, though, did not assess whether revised messaging, communicated via marketing materials, would communicate as intended and effectively garner interest in TAS services among our underserved audiences. There has been no official research conducted to date to gauge reactions to TAS materials designed for consumers. Also, to date, messaging testing has been conducted outside of tax season. Since these groups are slated for February, TAS will be able to ascertain reaction to TAS and TAS materials while members of our target audiences are facing (or have recently faced) tax filing for 2009. This fact, coupled with the need to conduct a review and red-flag check of consumer-facing materials represent key elements of the importance of this research.
We will include a diverse mix of our population based on the following aspects:
Gender
Self-identified as “bad money managers” (assessed via responses to screening items)
Self-identified as having a “life-changing event” in past 12 months (assessed via responses to screening items)
Household income
Marital status
TAS will ensure that the utmost scrutiny is given to privacy, security, and disclosure when reporting the comments from the focus groups. Only first names will be used in both recruiting and groups (first initial of last name will be used if there are duplicate names). No individual taxpayer data will be used for this project. Only qualitative data derived from discussions will be used in the report. We will limit and control the amount of information that we collect to those items that are necessary to accomplish the research objectives.
Porter Novelli Public Services will be responsible for recruiting and screening participants, preparing screener and moderator guides, moderating the groups, and compiling and summarizing the results. The interviews will be conducted at professional testing facilities in both cities. We will protect the privacy of the participants by not using names in the report. We will also control official access to the information and will not allow public access to the information.
We will apply fair information and record-keeping practices to ensure protection of all participants. The criterion for disclosure laid out in the Privacy Act, the Freedom of Information Act, and section 6103 of the Internal Revenue Code provides for the protection of information as well as its release to authorized recipients.
The estimated time to complete the participant screening is 15 minutes and the estimated time for each focus group attendee is 2.5 hours (i.e., 1.5 hrs. to attend and 1 hr. of travel to/from focus group). We will assume a 10% percent success rate in soliciting qualified participants.
Screening Burden |
||
Total number of potential participants screened (to include 2 alternates per focus group for a total of 10 participants recruited per group) for 12 groups is: (100 screenings per focus group X 12 focus groups = 1200 people)
|
1200 |
People |
Estimated time to complete screening |
15 |
Minutes |
Estimated participant screening burden (1200 X 15 minutes = 18,000 minutes / 60 = 300 hours) |
300 |
Hours |
Focus Group Participation Burden |
||
Estimated number of participants: There will be 2 locations and 12 groups of 8 participants conducted overall (6 in Richmond; 6 in Chicago) for a total of 96 participants. |
96 |
People |
Time to conduct the focus group (1.5 hours) |
1.5 |
Hours |
Time to travel to/from focus group (1 hour) |
1.0 |
Hours |
|
|
|
Estimated focus group participant burden (96 X 2.5 =) plus the 24 participants who are alternates (24 X 1) [use only travel time]
|
240 + 24 264 |
Hours |
Total burden (screening and focus group participation (screening burden of 300 hours + focus group burden of 264 hours = 564) |
564 |
Hours |
The following information will be provided within 60 days after the close of the focus group data collection operations:
Findings: a brief summary of significant (important) findings that were evidenced in the results (including the final versions of tested materials).
Lessons learned: a brief summary of lessons learned because of the findings.
Number of participants screened.
Number of focus group participants.
Date the data collection began.
Date the data collection ended.
Cost: Vendor-related costs, travel, stipend payments, and any other costs incurred as direct result of the focus groups.
Burden hours.
Attachments
Appendix A: Screening Questionnaire – Financially Distressed
Appendix B: Screening Questionnaire – Low-Income Families
Appendix C: Screening Questionnaire – Middle/High-Income Families
Appendix D: Moderator’s Guide
Appendix E: Demographic Sheets
Appendix F: Participant Handouts
Appendix G: Script for follow-up phone call (i.e., for those taxpayers who accept the initial invitation to participate)
1 The formative phase of research involved qualitative sessions in Baltimore, MD, and Los Angeles, CA.
2 Given the limited income differences that distinguish Middle-income Families and Affluent Families, these groups are combined. This will also reduce the overall burden associated with the project and allow for more efficient data collection.
3 It is important to note that at the time of OMB package submission, TAS and Porter Novelli Public Services are in the process of developing messages and the creative materials to be employed in testing. Placeholders for these materials and discussion questions to test them have been included in the moderator’s guide in this package. We will submit to OMB both materials and guides once they are finalized.
| File Type | application/msword |
| File Title | Level 2 Plan 1-04-20-053N |
| Subject | Standards for IRS Notices: Language and Layout |
| Author | ecschu74 |
| Last Modified By | XHFNB |
| File Modified | 2010-01-21 |
| File Created | 2010-01-21 |