Cognitive Interviewing for Q Tax, the Quarterly Survey of Local Government Taxes (F-70), Round 3
Prepared for:
Carma Ray Hogue, GOVS
Brian Zamperini, GOVS
Cheryl Lee, GOVS
Prepared by:
Kristin Stettler, RIRS
Office of Economic Planning & Innovation
Economic Programs Directorate
U.S. Census Bureau
January 30, 2013 draft
Findings and Recommendations 6
About the Response Improvement Research Staff (RIRS) 133
Appendix A: Round 3 F-70 Protocol 144
Appendix A: Round 3 F-70 Draft From 20
Three rounds of cognitive interviewing were conducted in July-November, 2012 to assess whether the two quarterly surveys of property and non-property tax collections could be combined into one form (to reduce respondent burden and collection costs). In all three rounds, we found indications of significant data quality issues, particularly with regard to the concept of “taxes imposed by your government”. Many respondents were including taxes that were NOT imposed by their government -- some because they did not understand the concept, some because they were not able to distinguish those taxes imposed by their government from other taxes, and some because they didn’t want to exclude them because they felt the data wouldn’t be representative of their government. From the opposite side, in several cases, we found that governments were not reporting taxes they collected on behalf of other governments (and that the instructions did not clarify whether they should be including those). We recommend that this issue should be studied further and that the base concept of the survey should be reconsidered, to ask for tax revenues that accrue to or benefit the government, regardless of who imposed the tax. This would better match both how respondents think about their tax collections, and what data they have easily available.
On a positive note, it did seem that the vast majority of respondents we met with would have been easily able to report property tax collections as well as non-property tax collections. (In a few cases, if the form did not get to the appropriate respondent, then those respondents would have left the property tax item blank.) In general, respondents were happy with the shorter form (particularly the 5-item form tested in Rounds 1 and 2), though some respondents did prefer the longer, current form as it seemed to better represent their government’s tax collections.
More specific findings and recommendations are detailed below.
Currently data on quarterly taxes by governments are collected on two separate forms. The Quarterly Survey of Property Tax Collections has one data item (property taxes). The Quarterly Survey of Non-Property Taxes asks for eleven data items, including sales taxes, license and permit fees, and income taxes. The Governments division proposed combining the two surveys into one shorter survey. The move to a single questionnaire and the collection of fewer taxes will reduce respondent burden and costs of collection. If it is possible to combine these collections, a single sample would significantly decrease the sample size. A smaller sample size would allow analysts more time to focus on the sample units, thus leading to better quality data that can be released earlier.
Cognitive testing was proposed to gauge respondents’ reactions to combining the two surveys into one form. The new survey form is going to those who have previously completed the Non-Property Tax form, so we were looking for their reaction to the addition of the Property Tax question and the reduced length. Additionally, we tried to learn about how respondents interpreted the questions, what records they used to answer the questions, how burdensome the form was and who the best respondent is.
In Round 1, we tested a short draft form (5 data items) with 9 respondents in the Kansas City metropolitan area. Based on findings from that round, we conducted 17 cognitive interviews in Round 2 with a slightly revised short form. In Round 3, several additional items from the Non-Property form were added back to the draft form tested, for a total of 8 data items.
In Round 3, RIRS conducted 9 cognitive interviews from November 13-16, 2012, in the Atlanta, GA and Montgomery, AL areas. (Several non-response follow-up (NRFU) cases were also completed.) Interviews were in-person and typically lasted about one hour. Interviews included the respondent(s), the interviewer from Census (RIRS) and an observer from Census (GOVS). All of the interviews were audio-recorded with permission. Cases were selected purposively, primarily with regard to size and type of the government.
Table 1: Characteristics of Governments Visited
Type of Government |
|
|||
County |
City/Town |
|
|
|
4 |
5 |
|
|
|
In all three rounds of testing, there have been indications that respondents are not consistently interpreting the data items, and thus are not reporting consistently. The biggest issue is how respondents interpret “imposed”. Although most respondents are able to give reasonable definitions for “imposed”, many of them later indicate that they are including taxes that are NOT imposed by their government. Some respondents are not able to distinguish whether certain of their taxes are imposed by their government or not. One respondent indicated she would need to call at least two other people to determine which taxes were imposed by their government. Many respondents felt that even though the form said “imposed,” that we couldn’t possibly mean that, since it would exclude a number of their larger taxes, in terms of revenue. Others thought it was odd, but were willing to exclude those taxes not specifically collected on the form. Thus, we get inconsistent reporting. Also, in no case was a government reporting taxes they collected on behalf of another government, that they passed onto them.
Recommendation: Consider re-thinking the base concept of the survey, to ask for tax revenues that accrue to or benefit the government, regardless of who imposed the tax. Alternatively, consider whether the core data items could be collected from the state, perhaps as part of the current F-72 data collection.
Most respondents felt the cover page was what they expected. Most scanned it quickly and then moved to page 2. Several did read it in detail. Again, several respondents didn’t see the information they needed to report online. A couple noticed the Census website address listed on the top left under Visit. Others saw the Report Online paragraph, but missed the website address.
Recommendation: Consider moving the website address to above the UserID and Password boxes. Consider indenting the REPORT ONLINE section (could make the arrow shorter), maybe adding a computer icon in front, and adding space above and below if possible. Also, add “www” or “http:// in front of the website address.
Generally, respondents understood the question, though several were not sure what selected taxes we wanted. Often, respondents thought we wanted “what taxes we collect.” One suggested we add “the following” to before “selected taxes”. Respondents did like the phrase “selected taxes” since it made it clearer to them that not all their tax revenues would be asked for. (They also liked “Selected” in the title of the form.)
As mentioned in Finding #1, when asked what “imposed” meant, respondents provided reasonable definitions, usually relating to levies their government has authority to collect. However, that does not mean that all respondents were willing/able to exclude those taxes that did not meet that definition. Most respondents thought the instruction to “Report zero. . . “ was adequate, though some had either not noticed it or forgotten it by the time they were entering their data. One respondent also noted that he thought the grammar of “Mark the NA box” was odd.
Recommendation: Add “the following” to before “selected taxes”.
Respondents seemed to understand that these Includes/Excludes applied to all of the questions. The Includes made sense to respondents. (One respondent noted that for the current form, F-73, they had only been including their General Fund, since the form did not specify to report for all funds.) The second Exclude, “Shares of taxes imposed by another government,” was not clear to all respondents. Some could not imagine what those taxes might be. Others understood what it might mean, but as mentioned above, weren’t willing to exclude those taxes.
Recommendation: No recommendation needed.
Generally, respondents understood and were able to report this item. Most knew exactly where to pull the figures from their books, and estimated it would take 5-10 minutes or less, on average. The Include, “Special property taxes,” was generally thought to be helpful, as it reminded respondents that they should include motor vehicle ad valorem taxes here. However, one respondent did note that the term “automobiles” was dated and we should consider replacing it with “motor vehicles”. The first Exclude, “Payments in lieu of taxes,” was not often relevant for this population, but they did seem to understand what it meant. The second Exclude, “Special assessments based on area or footage,” was less clear to many respondents. One respondent noted that it would be helpful to have more vertical space between A and B, because the answer space for A is so close to the label for B and it was confusing. Another respondent noted that “real or personal” should be “real and personal”.
Recommendation: Replace “automobiles” with “motor vehicles”. Consider whether the second Exclude is necessary (or could be clarified). Add more vertical space between A and B. Change “real or personal” to “real and personal”.
In general, respondents thought the Include and Exclude statements were clear. When asked about whether they would include the items mentioned in the first Exclude (hotel/motel, car rental, amusements, etc.) they generally said they would not. Although most respondents understood the second Exclude, many did not seem like they were willing to exclude those shares.
Recommendation: Add “the” before “State” in the second Exclude.
Respondents generally understood what this item was asking for, and were able to report it easily. However, quite a number of respondents were confused by the second Include statement, “Taxes applicable with only specified exceptions (e.g., food and prescribed medicines)”. The word “exceptions” made them think they were to back out food and prescribed medicines. This didn’t make any sense to them, and they were hesitant to do it, but at least a couple would have gone to the trouble of backing these out. Even respondents who wouldn’t have backed out those items were confused by the Include. Again, some respondents seemed to have forgotten the instruction to exclude shares of taxes imposed by another government by the time they got here. One respondent noted he would make the assumption Census wanted sales tax data, “even though we didn’t impose”. Another respondent wasn’t certain whether she should include a tax on people who do business in the county, since the Include says taxes on good and SERVICES.
Recommendation: Delete the second Include. Add an Exclude “Shares of taxes imposed by another government.”
It seemed like respondents were interpreting this differently. Some seemed to think only of specific taxes on alcohol sales at state-run stores, some only at private stores, some only on beer sales, some only of per-drink taxes, some on all or some of those. Some respondents thought they would need to break this amount out from what was already reported in Item 2B1. Also, it wasn’t clear if all of the taxes mentioned were “imposed” by the government. Again, some respondents seemed to have forgotten the instruction to exclude shares of taxes imposed by another government by the time they got here.
Recommendation: Add an Exclude “Shares of taxes imposed by another government.”
When applicable, respondents seemed to understand, and were able to report, this item. Again, some respondents seemed to have forgotten the instruction to exclude shares of taxes imposed by another government by the time they got here. One respondent did note that although they have a 4% per gallon business license fee (“gasoline tax”, which they call an excise tax), he wouldn’t include it here. Another respondent mentioned that they have a county-run airport and do collect a commission on fuel, but that he doesn’t think of that as a sales tax, so wouldn’t include.
Recommendation: Add an Exclude “Shares of taxes imposed by another government.”
When applicable, respondents seemed to understand this item. For some respondents, this was a difficult item to get the data for. Also, some respondents noted that this was already included in Item 2B1. Again, some respondents seemed to have forgotten the instruction to exclude shares of taxes imposed by another government by the time they got here.
Recommendation: Add an Exclude “Shares of taxes imposed by another government.”
This seemed to be quite a difficult item for many respondents. Some respondents were not sure how this item differed from what they reported in 2A. (One respondent had been reporting their “motor vehicle ad valorem” here on the current F-73.) Others had to go to separate sources to get the data for this item (often the probate judge). For a number of respondents, this item wasn’t relevant. Both the Includes and Excludes were very confusing to respondents (mostly because they were wordy and several of them were no longer relevant). One respondent commented that this was “an odd one to pick out.”
Recommendation: Delete this item. At a minimum, delete the current Includes and Excludes (perhaps add a very shortened version of the third through fifth Includes, if necessary).
This item was not applicable to the respondents in Alabama and Georgia.
Recommendation: In second Exclude, change “taxes” to “tax” and capitalize “state” in exclude to be consistent.
This item was not applicable to the respondents in Alabama and Georgia. Several respondents did note that the Include is very confusing to them.
Recommendation: No recommendation needed.
Generally, the data requested was available to the respondents themselves, although for some respondents, certain items did require asking another person for data. The majority of respondents mentioned needing to run a report from their computer and pulling the numbers from the one report. The items that were added for this round of testing (2B2, 2B3, 2B4 and 2C) did make this form much more difficult than the versions tested in Rounds 1 and 2. As discussed above, these items were often tricky for respondents to interpret, and sometimes required going elsewhere to get the data.
Recommendation: Delete Items 2B2, 2B3, 2B4 and 2C and go back to shorter version of form tested in Rounds 1 and 2.
As in previous rounds, respondents generally said it could take from 30-60 minutes or more to complete the form the first time, but that it would usually take less (20-45 minutes) after that. (One respondent said 3 minutes total.) Also, many respondents had to add multiple data items and/or funds together. Some would program a report so that after the first time, they could just run that report. Others would just note what they did so they could do it again by hand the next time.
Recommendation: No recommendation needed.
In this third round, we did not always ask respondents about revising their data, since their negative reaction to this was pretty consistent in the first two rounds. Those we did ask again indicated that they didn’t see themselves being willing to revise previous quarters’ data.
Recommendation: No recommendation needed.
Most respondents did not read these in detail. No respondents had any negative reactions.
Recommendation: No recommendation needed.
Suggestions for addressing the survey included: Finance Director, Accounting Manager, Chief Financial Officer, Treasurer, Controller, Internal Auditor, Tax Collector, Tax Department, City Manager, City Clerk, and City Administrator. Several respondents again noted that it is not a good idea to address it to any of the elected officials, like Chairman or Council President. One example of issues that might occur if the form goes to the wrong respondent was when the contact person was the revenue commissioner for a county. He would have only filled out the items that were relevant to him in his job, and entered NA for all other items (even if the county did have them) and return the form. Another revenue manager had a similar situation (he wouldn’t have reported property taxes). Also, a number of respondents noted that they have changed the contact person on the form itself and were frustrated when it didn’t take, and still came to them. They generally did not think they should have to call Census in order to get this changed, especially since the form asks for that information.
Recommendation: Ensure that address/contact information is updated, or at least reviewed, whenever changed.
Most respondents preferred being able to use the Internet, though not all those who said that have been reporting online for current Census surveys. Again, at least one respondent complained about not being able to print her completed survey.
Recommendation: Emphasize the Internet as the preferred data collection method for respondents. Resolve the printing issue.
Again, several respondents wanted to know how the data is used and why it is important. One respondent suggested: “Pay us!”.
Recommendation: Emphasize importance of data.
Most respondents were amenable to receiving notifications/reminders via email, though they noted that errors might occur if the usual respondent leaves the government.
Recommendation: Consider collecting a “backup” email address.
Most respondents preferred the shorter form, saying things like “less is more”. Respondents thought it was slightly easier to respond to. However, several respondents felt that the information provided was not a true representation of their governments. Two respondents mentioned that they are usually looking for a place to report “big-ticket” items. Additionally at least one respondent mentioned that the non-property form breaks out the data in a way they already have.
Recommendation: No recommendation needed.
Respondents felt that quarterly response was a reasonable burden, but wanted to know how the data were used and why such frequent response was necessary.
Recommendation: No recommendation needed.
Not all respondents in Round 3 were given the opportunity to review the revised cover letter. However, those who did responded positively, saying things like “it’s short and to the point”. One respondent noted that the website mentioned in the letter is different from that on the form. Several respondents also reviewed the Internet brochure. One respondent thought we should highlight the sentence about GDP.
Recommendation: Correct typos. In fourth paragraph, move second sentence to new line. In fifth paragraph, “collection” should be “collected”. In footer, “Please not” should be “Please note”. I Add “Selected” to title before “Local”.
Consider alternative methods for collecting this information.
Consider alternative designs for a combined form.
Draft a revised form.
Conduct additional cognitive testing (locally, if possible, or by phone, or on a future QuIP trip).
The Response Improvement Research Staff (RIRS) in the Office of Economic Planning and Innovation (OEPI) assists economic survey program areas and other governmental agencies with research associated with the behavioral aspects of survey response and data collection. The mission of RIRS is to improve data quality in surveys while reducing survey nonresponse and respondent burden. This mission is achieved by:
Conducting expert reviews, cognitive pretesting, site visits and usability testing, along with post-collection evaluation methods, to assess the effectiveness and efficiency of the data collection instruments and associated materials.
Assisting program areas with the development and use of nonresponse reduction methods and contact strategies.
Conducting empirical research to help better understand behavioral aspects of survey response, with the aim of identifying areas for further improvement as well as evaluating the effectiveness of qualitative research.
For more information on how RIRS can assist your economic survey program area or agency, please contact Diane K. Willimack at [email protected].
F-70 Cognitive Interview Protocol
Revised: October 18, 2012
(This protocol is a guide – the questions presented here will not necessarily be asked exactly as worded in the protocol or in this order. It is important to note that not all questions will be asked in every interview.)
Research Questions:
Can the two surveys be combined?
Can the tax imposers provide responses for the new survey?
1. Introduce observers and their background
2. Purpose of visit:
Let me start by telling you a little about what we will be doing today. Currently there are two quarterly tax surveys for local governments collected by the U.S. Census Bureau: Quarterly Survey of Property Tax Collections and Quarterly Survey of Non-Property Taxes.
The Census Bureau is considering combining the two surveys into an eight question survey form. We would like your input and feedback on the possibility of combining the two surveys.
This is not a test of you but the form, and getting information from you about interpretations, impressions, and opinions of the form. There are no right or wrong answers. All feedback is useful.
I will ask you general and specific questions about your government and the form itself. Some questions may seem odd or redundant because we are not accountants, and because we’re meeting with various local governments, that have different organizational structures and ways of keeping records.
Do you have any questions?
3. Permission to audio-tape discussion? Have R sign consent form.
Before we get started: I'd like to audio tape this interview, so I don't have to rely on my memory later. This session is confidential. Only persons connected with this project will have access to your tape. If that is all right with you, please sign this consent form. It also tells you about the confidentiality of this session.
Background Information
1. Respondent Background
Tell me a little about this agency:
What sorts of activities is it involved in?
What is your role in this agency?
Have you filled out government forms in the past? If so, which ones?
Generally speaking, how do government forms get completed at this agency?
Does more than one person fill out a form? If so, what is that process? Do you communicate with other people to complete the form?
When does your fiscal year end? About how long does it take for quarterly figures to be ready?
Now I am going to hand you a copy of the questionnaire. We will go through it page by page. Please write on it, and complete it as if we were not here. I will ask you questions when you have finished the page.
Cognitive Interview
General Probes
In your own words, what was that question asking?
What are you thinking about?
Can you tell me more about that?
You answered, “…” because…?
I want to make sure I understand, can you explain that again?
Can you tell me how you arrived at that answer? What did you include? What did you exclude? What records did you use?
Reflect back on R’s answer and ask R to correct (“I want to make sure I have it right. I think you said, “…?”)
1. Cover page
What is your initial impression upon seeing the form?
In your opinion, what is the most important information on this page?
What is the least important?
Is there any (other) information you would like to see here? What would that be?
What information do you need prior to filling out the form?
Did you notice the UserID and password?
2. Instructions
-Probe on descriptions offered of records, procedures, interpretations, etc.
-Observe whether R appears to read the question and instructions
Did you read any of the information on this page?
(If yes) What did you read?
3. Item 2 (Question Stem and Includes/Excludes)
What do you notice when you look at Question 2?
What does the word “imposed” mean to you?
What is your reaction to the Include and Exclude statements?
(If a government that collects taxes FOR others, ask how quickly they distribute that information?)
4. Item 2A (Property taxes)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
How long would it take to obtain the information?
Would you include taxes collected by others?
What is your reaction to the Include statement?
How easy or difficult would you say this question is?
What does “all taxes on property, real or personal” mean to you?
Would you include receipts from service charges? Interest earnings? Fines?
Would you include levies for debt service? Levies for contributions to pension funds? Levies for other funds or purposes?
5. Item 2B1 (General sales and gross receipts taxes)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include statement?
How easy or difficult would you say this question is?
6. Item 2B2 (Alcoholic beverage sales tax)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include statement?
How easy or difficult would you say this question is?
7. Item 2B3 (Motor fuels sales tax)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include statement?
How easy or difficult would you say this question is?
8. Item 2B4 (Tobacco products tax)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include statement?
How easy or difficult would you say this question is?
9. Item 2C (Motor vehicles licensing and permit taxes)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include and Exclude statements?
How easy or difficult would you say this question is?
10. Item 2D1 (Local individual income taxes)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include and Exclude statements?
How easy or difficult would you say this question is?
11. Item 2D2 (Local corporation net income taxes)
How would you go about answering this question? What records would you need to consult in order to answer this question?
Is this information readily available?
Is this information available here or would you need to go somewhere else to get this information? Where?
What is your reaction to the Include statement?
Would you include commuter or head taxes?
How easy or difficult would you say this question is?
12. Layout
What is your overall impression of the layout of this form?
What is your reaction to the example at the top of page 2?
What do you think the shaded box around the include/exclude statements below Item 2 means?
What do you think about the location of the N/A box?
Wrap-up
Would the information requested on this form generally be available in this department, or would information have to come from some other department/staff member?
How would you go about getting that information?
Would you pass the form over to that department, or just get information from them over the phone/email/in-person?
When would the data be available from others?
How much time would it take you to complete this report (actual time, not elapsed time)?
Is collecting on a quarterly basis reasonable? If no, how often should this information be collected?
If you wanted to revise data for previous quarters, what would you do? How could we tell you that you can revise up to 7 previous quarters?
Overall, how easy or difficult was it to complete this form? Which were the easiest questions to answer? Which were the hardest to answer?
What are you thoughts on completing this form on the internet? Were you aware that was an option?
Who would be the best person to address this form to? What other titles might be applicable at other comparable governments?
(If respondent/government had not returned survey previously) Why have you not returned this survey in the past?
What is your reaction to this shorter version of the form? (show them the previous version if necessary)
How would you prefer to be contacted about this survey initially? If we emailed you, would you find that acceptable?
How would you prefer to be reminded if this survey form was overdue? If we emailed you, would you find that acceptable?
Is there anything else we might do to encourage response to this survey?
Any other comments?
Review cover letter and or Internet brochure.
Thank you for your assistance today.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | pick0002 |
| File Modified | 0000-00-00 |
| File Created | 2021-01-29 |