Download:
pdf |
pdf2021 Annual Integrated
Economic Survey Pilot Phase I
Results
Melissa A. Cidade
Economic Management Division
September 15, 2022
The Census Bureau has reviewed this data product for unauthorized disclosure of confidential information
and has approved the disclosure avoidance practices applied (Approval ID: CDRB-FY24-ESMD001-002).
1
Hi, everyone! I am so excited to be updating you on the findings for the 2021 AIES Pilot
to date. There’s a lot to cover so let’s jump right in.
1
AIES Pilot Overview
Phased Approach:
Phase I: Intensive pretesting
• Goal: Understand response processes and further
instrument refining
Phase II: New Respondents and nonrespondents
• Goal: Induce independent response
Research Modalities:
• Online survey
• Response Analysis Survey (RAS)
• Debriefing interviews
• Contact from the field
Phase III: Data handling and final
testing
• Goal: Troubleshooting and infrastructure building
2
Before I get into our findings, let me take a moment to remind everyone about the
goals of the AIES Pilot. Remember that in the summer and fall of 2021, we had
identified that while we had engaged in some robust formative research around
integrating the in-scope annual surveys into one holistic survey of the company, all of
this research had been theoretical to date. What we needed was a mechanism for
making the theoretical concrete – a test that induced independent response from the
field. We decided that a pilot survey was the best way to start to see how respondents
could actually provide responses to the combined survey.
Remember, too, that we split up the pilot survey into three distinct phases. In brief, the
three phases represented a slow roll-out of the integrated instrument.
Today, we are talking about the results of Phase I. I know your next question so let me
get it out of the way: will there be a phase II and phase III? The answer is that while
yes, we will move forward with additional testing, but no, I do not have details on what
that will look like just yet, and yes, I will come back to this group before moving forward
with a next step.
On the right of the screen are the research modalities that we used for Phase I. This
2
included not only the online survey with the integrated content, but also a Response
Analysis Survey – a short survey of responders focused on perceived and actual burden,
the survey structure, and the response processes. Last, but not least, we conducted
debriefing interviews. These interviews focused on response processes, challenges, and
benefits to the integrated survey instrument. Not in the original plan, but has been an
invaluable source of information about the integrated instrument, we also catalogued
any and all contact from the field – phone calls and emails – to provide additional
information about the response process and how respondents were approaching the
integrated instrument. You’ll see results from these efforts all through this
presentation, so just note that we are looking at a mountain of data.
2
Pilot Recruitment Overview
Pilot Eligibility Criteria:
• Medium-sized (M)
• Mailing address in the
• Multiple in-scope annual
continental United States
surveys
• Not in indicator surveys
• Reported in SY2020 prior • Has valid email address
to June 1
• Not in “take all”
industries
Pilot Recruited Companies’ Units by Manufacturing Status
and Overall
Company
Manufacturing
NonManufacturing
Total
22
56
78
Pilot Recruited Companies’ Units Measures of Central
Tendency by Manufacturing Status and Overall
Manufacturing
NonManufacturing
Overall
Establishments
Mean
5
35
37
Minimum
1
1
2
Maximum
13
374
374
Industries
Mean
2
4
5
Industry
42
151
193
Minimum
1
1
1
Establishment
117
2,746
2,863
Maximum
5
20
17
3
First, let’s talk about who participated in the pilot. We decided in the design phase of
the pilot to focus on “M”- medium – companies. Medium-sized companies fall in
between the comparatively simplistic record keeping practices of smaller businesses
and the complicated and highly structured record keeping practices of larger
businesses, and have historically lower response rates to the Economic Census than
their larger and smaller counterparts. We recruited only from companies that were in
multiple in-scope annual surveys, and reported prior to June in survey year 2020. We
excluded from recruitment M companies in “take all” industries, those with mailing
addresses outside of the United States, those tied to indicator surveys, and those
missing email addresses. We set a minimum of 2 establishments and a maximum of
300 establishments, though two recruited companies had more than 300
establishments. Our colleagues in EWD carefully selected 300 companies meeting
these requirements, and from there, we recruited 78 to participate in Phase I of the
pilot.
This included 22 companies with at least one establishment classified as a six digit
NAICS found in the manufacturing sector. We will call these companies “manufacturing
companies” for this presentation. At the same time, 56 companies had establishments
with six digit NAICS exclusively outside of the manufacturing sector. Note that all
3
manufacturing companies also had at least one establishment with a six digit NAICS
outside of the manufacturing sector. These 78 companies represented a total of 193
unique industries - six digit NAICS – and 2,863 establishments. On average, recruited
companies had 37 establishments and 5 industries.
3
4
This is a representation of the distribution of establishments from recruited companies
for the pilot – we reached pretty far and wide!
I want to pause here and talk about piloting. You will find yourself throughout this
presentation thinking to yourself: but, Melissa, there are so few cases. How does
talking to this number of companies give us insight into anything useful. I want to
remind you that the point of this kind of research is offer representation, not to be
representative. Let me be the first to clearly state: the diversity in size, complexity, and
sophistication of the cases in this pilot give strong representation to the “M” companies
that we encounter in our typical survey interactions. We have not included small
businesses; we have not included the very large businesses. We made design decisions
at the start of this pilot with the intention of understanding the nuances of survey
response through as many lenses as possible – representation, not representative.
That any of these data are being consider in lieu of their current annual survey
response is a testimony to the incredible resourcefulness of our data collection and
processing teams, and not a reflection of the intention of this research.
With that important methodological caveat in place, let’s keep going…
4
AIES Pilot Survey Structure
Module 1: Company-level Data
Module 2: Establishment-level Data
• Manufacturing
• Non-manufacturing
Modules are:
• Order agnostic
• Independent of each other
Module 3: Industry-level Data
• Manufacturing
• Non-manufacturing
Module 4: Additional Establishment(s)
• Classification-dependent
• Each with response-mode choice
5
Before I can talk about responding companies, let me first lay out for you the survey
structure.
Module 1 asks for data at the company level. This includes many of the company-level
questions on the Report of Organization.
Module 2 asks for data at the establishment (or location) level. For those
establishments that are classified within a six digit NAICS within the manufacturing
sector, there were many more questions at the establishment level than those
establishments not classified as manufacturing. This is because we produce more
comprehensive geographic data for manufacturing establishments than we plan to
produce for the other trades. Because of this, module 2 is split into two subsections –
manufacturing and non-manufacturing.
Module 3 asks for data at the industry or kind of activity level, which corresponds to
the six digit NAICS codes that establishments have been classified. In this case, those
companies that have establishments classified as non-manufacturing have more
questions at the industry level than those classified as manufacturing. As such, module
3 is also split into manufacturing and non-manufacturing. Manufacturing companies
5
were asked about their capital expenditures at the industry level, but no other data
were collected at this level for manufacturing industries; collecting capex at the industry
level for manufacturing companies was an experiment, and we will not be reviewing the
results of that specific experiment during our time together today.
Finally, Module 4 allowed respondents to add any establishments not already pre-listed
for their company. Additional information about the added establishments – including
payroll, revenue, and others – was collected in Module 4 at the establishment level.
A few caveats to keep in mind about this structure:
The order of module completion is agnostic – that is to say, respondents could choose
to answer modules out of order. They are also independent of each other – data
entered in Module 1 are not available in Module 2 or 3, and vice-versa. At the same
time, the instrument was programmed so that respondents were only presented with
modules that applied to their company based on the classification of their
establishments. That is, those companies that have establishments classified as
manufacturing would be given access to Modules 2 and 3 for manufacturing response;
those without manufacturing establishments are not even shown the option of
responding to the manufacturing sections of Modules 2 and 3.
Finally, respondents were given the choice of responding by spreadsheet or responding
by page-by-page view at the start of each Module.
5
6
Note: Screenshots represent fictional company.
On screen now are screenshots from the pilot instrument. On the left of the screen is
the survey dashboard. The fictional company represented here had establishments
classified within a six-digit manufacturing NAICS code, and so, both the manufacturing
and non-manufacturing sections of Modules 2 and 3 are available on screen. You can
see from this dashboard that respondents can access a survey preview for that module,
respond to the module, and to share the module with someone else in their company if
they choose.
On the right side of the screen is the first screen of Module 1 for this same fictional
company. In this case, the respondent clicked on “Report Now” for Module 1, and this
is the first screen that populates. Here, the respondent is instructed to select to
respond by online survey – page-by-page – or by spreadsheet. This page was at the
start of each module.
6
Instrument Programming Limitations
• Module 4 – additional establishments
• Submission
• Duplicate responses (especially at the establishment level)
• Edit and content checks
7
Remember that we conducted the pilot using Qualtrics, not the typical online survey
platform we use for production instruments. While we worked closely with Qualtrics
engineers to replicate what we could of the production environment, ultimately there
are important instrument programming limitations that I want to highlight before
getting into results. Many of these issues become moot in production but they
influence the results of the pilot in important ways.
First, all respondents were given access to Module 4 to list out and answer questions
about establishments that were not already pre-listed. We had no way of knowing
beforehand what establishments might be missing, if any, and so, we have no way of
knowing if we have captured all additional establishments for the pilot. We did,
however, have a few companies submit Module 4 blank as a way of indicating that they
had no new establishments.
Next, because each part of the pilot was independent of the others, respondents had
no overarching way of indicating full submission. In fact, we heard from respondents
throughout the field period, inquiring if they had met submission requirements or
letting us know that they believed that they had completed the survey.
7
We also saw a particular issue with duplicate responses. This was particularly prevalent
for the establishment-level collection – the best way that we could program the
establishment-level questions was to have all establishments for a company listed in a
drop-down box that respondents could select from. However, once a respondent
submitted data for that establishment, it was not removed from the dropdown box.
There was no indication on the respondent side that they had met the reporting
obligation for that establishment, and so we ended up with some duplicate response.
This all ties into a larger limitation – edit checks. In our typical surveys, we embed
checks throughout the survey where items must sum, where response is required, and
others. The Qualtrics instrument had no such edit checks within the modules or
between the modules. At the same time, we also often place limitations on the length
and character type of response – again, the Qualtrics instrument had very limited
content checks built in, as well. This lack of within instrument quality control did impact
how clean the resultant data are; this is an important consideration when considering
the quality of the pilot data – we simply did not have the same kinds of guardrails we
usually have in place for our online instruments.
7
Pilot Response Completion Variables
Module 1: Companylevel Data
• Total employees
• Annual payroll
• Q1 payroll
• Total revenue
Module 2:
Establishment-level
Data
• Total employees
• Annual payroll
• Q1 payroll
• Total revenue
Module 3: Industrylevel Data*
• Annual payroll
• Total revenue
• Total operating
expenses
*Non-manufacturing industries only
8
So far, I’ve laid out characteristics of recruited companies and the pilot survey structure.
One key indicator of pilot performance is response rate, but calculating a response rate
for this pilot is more complicated than most. Because each module was completed
independently, companies could submit partial data at the company, establishment,
and/or industry level, for both manufacturing and non-manufacturing establishments
and industries. This creates a conundrum: how complete do data need to be in order
to count as a completed survey?
To understand response patterns, we moved away from a dichotomous understand of
complete – that is, complete or not complete – toward a proportion complete – that is,
how much data did the respondent submit? We looked across the modules and
matched content wherever possible to measure completeness. So, for example, annual
payroll and total revenue are asked across all three modules.
If we think of the proportion complete, then, we can take the number of response
completion variables times the number of units for the module to come up with the
denominator for that company. Then, the total number of non-zero responses for
those key variables becomes the numerator.
8
Total
employees
Annual
payroll
Q1
payroll
Total revenue
Manufacturing Plant 1001: Cat food
Manufacturing Plant 1002: Cat food
Manufacturing Plant 1003: Cat litter
Wholesale Facility 1004: Cat food wholesaler
Wholesale Facility 1005: Pet supply wholesaler
Retail Store 1006: Pet supply store
Retail Store 1007: Pet supply store
Retail Store 1008: Pet supply store
Retail Store 1009: Pet supply store
Retail Store 1010: Pet supply store
Service Location 1011: Pet grooming services
Service Location 1012: Animal shelter services
CCC Module 2
Proportion
Complete Rate
• Manufacturing:
12/12, 100%
• Nonmanufacturing:
23/36, 64%
9
You know we couldn’t go too far without bringing in my favorite fictional company, the
Census Cat Company. Let’s use this company as an example of pilot completion
consideration.
On screen now, you can see the 12 establishments for the CCC. Note that the first
three are manufacturing and the remaining 9 are non-manufacturing. To calculate the
proportion complete for module 2, we take the number of key variables – 4, listed
across the columns – and multiply by the number of units to get the denominator.
Then, we count the number of non-zero responses for the numerator, and divide.
In this case, the CCC provided non-zero responses to 12 of the 12 key items for module
2 manufacturing establishments. That is 100 percent complete.
For module 2 non-manufacturing, though, the CC provided 23 of the total 36 key items.
Note that there is unit non-response – no data are provided for service location 1011,
the pet grooming location – and item non-response – no data are provided for Q1
payroll across the non-manufacturing locations.
9
Percentage of Completed
Key Pilot Variables by
Module
10
Considering this approach of percentage complete, we start to see a picture of
response status for pilot companies. Fully 49 of the 78 companies in the pilot
answered all of the key items in Module 1. This represents 62.8 percent of recruited
companies. Looking across the modules, 12 of 22 companies provided all key items for
Module 2 manufacturing – 15 percent of all companies, but 54.5 percent of
manufacturing companies. And 17 of 78 provided all key items for Module 2 nonmanufacturing – 21.8 percent. Finally 15 of 78 companies provided 100 percent of key
items for Module 3 non-manufacturing – 19.2 percent.
Note that during the field period, six companies revoked consent or otherwise refused
participation in the pilot. These companies are included in the figures on screen so that
we can track response patterns for the totality of recruited companies.
10
Percentage of Completed Key Pilot Variables
Across All Modules
Number of companies by percentage of completed
key pilot variables across all modules*
Percentage of Completed Key
Pilot Variables Across all Modules*
0
Number of
Companies
Percentage of
Companies
16
20.5%
1 – 24.9
4
5.1
25 – 49.9
10
12.8
50 – 74.9
15
19.2
75 – 99.9
28
35.9
5
6.4
100
Total
78
*Note: Includes refusals
100
Number of Companies
Percentage
complete
33
35
30
25
20
15
16
15
10
10
4
5
0
0%
1 - 24.9%
25 - 49.9% 50 - 74.9%
Percentage Complete
75 - 100%
11
I understand that looking at response rates by module can be…unsatisfying for some.
We are used to a single response rate as a measure of “how well we did.” If we went
by our typical metrics, we would say that 5 of 78 companies – 6.4 percent – responded
to the survey in full.
If, however, we shift our view to a proportion complete – we see that 33 of 78
companies – 42.3% of all recruited companies – provided data across at least 75
percent of our key items. In fact, we end up with a bimodal distribution – 16
companies gave us nothing, 33 companies gave us at least three-quarters of what we
asked.
11
Pilot Research Questions
• How are respondents completing the survey?
• Can respondents provide answers to the questions on the
integrated survey?
• Does the order and structure of the integrated survey make
sense?
• Is the survey burdensome?
• Is the integrated survey using appropriate units of analysis?
• What do the resultant data look like?
12
Ok – we know what we did and we know what response looked like. Let’s get into what
the ‘findings’ are!
On screen now are the research questions we outlined at the very start of the pilot.
Some deal with understanding response-processes to the integrated instrument. Some
pertain more to what the data might look like. Let’s take each one in turn and see
where we stand!
12
How are respondents completing the survey?
• Using survey preview materials
• 11 of 15 RAS companies
indicated requiring assistance to
complete pilot survey
Number of companies reporting any data
by survey module by mode of
completion
60
50
39
10
40
41
7
36
30
20
• Order:
10
• Started with Module 1
• Modules 2 and 3 not linear
15
3
11
0
Module 1 Company-level
Module 2 Manufacturing
Establishments
Module 2 - NonManufacturing
Establishments
Spreadsheet
Page-by-page
Module 3 - NonManufacturing
Industries
13
How are respondents completing the survey?
Based on our debriefing interviews, respondents are using a response process that is
similar to the current annual surveys. For example, many respondents mentioned
“clicking through” or “checking out” what the survey is asking before beginning to
gather the information or entering anything into the online survey or the spreadsheet.
This is typical response behavior.
Some talked about parsing out the survey to others on staff to help complete it, either
because they didn’t have access to the necessary data or because they didn’t have time
to complete the survey by themselves. At the same time, asked on the Response
Analysis Survey, 11 out of 15 respondents indicated that they required assistance from
other people or departments to collect relevant information or complete answers to
questions in any of the AIES questionnaire modules, and of those, 8 of 11 said that it
took three or more people to complete the pilot survey.
When asked about the order in which the modules were completed, respondents noted
that they often started with module 1, and could complete it in a single-sitting. But,
Modules 2 and 3 took more than one person to pull all of the data, or there were
13
calculations and data manipulations that needed to happen in order to respond to the
survey. Because of this, those two modules were not answered in a linear fashion –
beginning at question 1 and moving through to the last question – but more were
completed on a flow basis as the data were available or prepared. This bears out in the
pilot response data, too. Note that 69.2 percent of recruited companies provided at
least some data in Module 1, the highest of any of the modules.
Remember earlier when I showed you how respondents were given the explicit choice
of uploading a spreadsheet to respond to the survey or using page-by-page? What we
see in the response data is that respondents chose their mode based on the unit of
analysis. Note that a larger proportion of companies chose to respond using a page-bypage online survey for Module 1, where the unit of analysis is the company. However,
that flips when we change the unit to establishment and industry, where most chose to
respond by spreadsheet. Others talked about depending on their own tools to respond
to our surveys, including their own spreadsheets, custom reports, and even printing out
previews and writing out notes.
13
How are respondents completing the survey?
1. The response process for the AIES Pilot is similar to that of the
current annual surveys.
2. More companies reported data at the company-level, and data
were most complete at the company-level, compared to the
establishment and industry levels.
3. Most companies responded by page-by-page method at the
company-level, and most companies responded by uploadable
spreadsheet at the establishment and industry levels.
14
So, let’s answer the question: how are respondents completing the AIES pilot?
Number one: they are reporting similar response processes as the current annual
surveys and other surveys we currently run.
Number two: Company level data were reported by more companies in general and
were reported more completely than at the establishment and industry levels.
Number three: Respondents used the page-by-page method at the company level
most often, and the uploadable spreadsheet method at the establishment and industry
levels, most often.
14
Can respondents provide answers to the
questions on the integrated survey?
Which of the following, if any, were challenges to completing the AIES questionnaire modules?
Number selecting each*
Had to add, allocate, or otherwise manipulate data to fit questions
15
Had to collect information from more than one database or other source
10
Had to wait to rely on others within my company for the requested data
7
Too many questions
4
Unclear or inadequately defined terms or the online survey was too difficult to use
4
Some other challenge
3
Questions were too complicated
0
I had no challenges completing the AIES questionnaire
0
*Totals more than 15 because respondents could select all that applied.
15
Let’s turn our attention to the next research question: can respondents provide
answers to questions on the integrated survey?
We can start to shed light on this question by looking at the Response Analysis Survey
again. We asked respondents what challenges they faced when completing the AIES
pilot. It seems that the most mentioned of the challenges to response are related to
the content – manipulating company data to fit our questions and collecting the data
from across internal systems and from others within the company.
15
Can respondents provide answers to the
questions on the integrated survey?
• Ambiguous Wording
• Duplication of content
• Duplication of pre-listed information
• Classification
16
In debriefing interviews and feedback from the field, respondents ‘bubbled up’ issues
with some of the content on the survey. Some mentioned ambiguous question
wording or unclear instructions. I’m not going to dwell on this because I know that
there is currently cognitive testing occurring on some of the items on the harmonized
instrument, and that’s outside the scope of the pilot.
One issue that respondents mentioned time and again was the duplication of content
within the survey. They noticed that some questions were asked the company,
establishment, and industry level, and because of the way that the pilot was structured,
these responses did not relate to each other. They voiced frustration for having to
provide the same information in more than one place. We knew that there would be
duplication, but we did not consider the frustration and burden that this duplication
added to completing the survey.
Beyond the content, though, there are two important structural elements to consider
as hindering or helping response. The first is the establishment listings – several
respondents mentioned that establishments were duplicated, outdated, or otherwise
not reflecting the current state of the company. This suggests that respondents may
need additional flexibilities in updating or de-duplicating their establishment listings. At
16
the same time, we also got feedback from the field about how establishments are
classified. For many pilot participants, this is the first time that we have asked for data
from all aspects of their company at one time. Some were simply unaware of the
misclassification because we never asked – for example, one company had been
consistently in sample for ASM, but had not been in sample for AWTS. Seeing the
wholesale companies and corresponding questions listed on the screen, the respondent
requested to change the wholesale locations to manufacturing – where they believe
they should have been all along but about which she had never been asked.
The other side of the coin exists too! One respondent who has only seen his company
represented by industry in SAS wrote in to say that three locations were assigned to
industries that he thought were not a good fit, and that “this has probably happened a
bunch of times but I’ve never seen it laid out like this so I did not notice.” Seeing the
whole picture of the company – with all establishments and corresponding industry
classifications – may lead some respondents to want to change how their
establishments are classified. I’m not necessarily advocating self-classification, but I am
bringing to attention that these requests for changes may be particularly salient
because respondents are for the first time being presented with a holistic view of the
company representing all of the classifications that we have assigned in one place.
16
Can respondents provide answers to the
questions on the integrated survey?
4. Content retention based on industry criteria, and adding NAPCS,
will continue to be an issue.
5. Survey structure and capabilities need to be flexible to meet
respondent needs.
17
Ok, so can respondents provide answers to the integrated questions on the survey?
Well, the pilot demonstrated that respondents are struggling with answering our
questions – in some ways that we already know about, like scope and detail, data being
dispersed throughout the company, and others – and in other ways unique to AIES –
like duplication of content over the multiple collection units. We will need to continue
to revisit and refine content as we move closer to production.
The pilot also demonstrated that establishment listing and classification matters more
than ever in this integrated survey paradigm, and that we have to continue to push for
increased flexibility in structure and capabilities of our online surveys.
17
Does the order and structure of the
integrated survey make sense?
How easy or difficult was completing the
AIES questionnaire modules?
N
Percentage
Very easy or
easy
9
69.2
Difficult
4
30.8
18
Our next research question: Does the order and structure of the integrated survey
make sense?
On the Response Analysis Survey, we asked how easy or difficult it was to complete the
questionnaire modules. Most respondents said it was “easy” or “very easy” – but this
needs to be taken with a grain of salt. All of the RAS respondents completed the pilot
either partially or completely.
What may be more telling, then, is responses to the question of overall, which AIES
module was the most challenging to complete. Note that none of the respondents said
that Module 1 – the company-level – was the most challenging, suggesting again that
these data are the most accessible available. Half of RAS respondents said that Module
3 – Industry-level data for non-manufacturing industries - was the most difficult.
We then followed up with an open-ended question, “what made this the most
challenging module to complete?” For Module 2, respondents mentioned gaining
access to specific data (for example, “getting the payroll info”) and generally collecting
data for a large number of establishments. For Module 3, however, the challenge lies in
the requested data. Responded one, module 3 is challenging because it entails “adding
18
up multiple accounts [and] object codes to fit the questions.” This was echoed by
another respondent, noting that module 3 is challenging because there are “so many
specific questions and so many things from the general ledger [that] needed [to be]
added together to answer the questions, so it took a ton of time.”
18
Does the order and structure of the
integrated survey make sense?
• Topic-based approach
• More directed submission process
• Spreadsheets:
“Once I figured out it was there, I loved having the spreadsheet. A
challenge is that manufacturing is broken out from the other stuff.
[If it was all together] I could consolidate back to my consolidated
financials and then that way I know you have all of the pieces. To
be frank: I think you got all the pieces now, but I don't know if you
got all the pieces right now. I did my best to try to make sure all of
it was there. I was pulling from so many sources of data, that it is
very, very difficult.”
19
We learned a wealth of information about the order and structure of the integrated
survey from debriefing interviews, too. While we ultimately favored a unit-based
collection approach – modules are driven by the unit of response – we asked
respondents what role does topic play in helping them to complete the survey?
Respondents mentioned wanting more organization around a topical approach, either
integrated into the survey – as in, ask me all of my payroll questions at one time, I’m
getting the data from the same place – or in supplemental materials –as in, provide me
a listing of the questions by topic and at what level you’ll be asking them, and just let
me know that this may not be the order that I see them in on the survey.
We also heard from respondents that they wanted a more direct means of letting us
know where they were in the response process, and specifically around submitting
their data. On your screen is a word cloud generated from the interviews and
respondent contact that mentioned submission of data. Respondents most mentioned
wanting to know if they had completed everything and if they could edit submitted
data. Some of this is an artifact of the way that the survey was programmed – there
were no clear indications that a respondent was “done” – but it’s also reflective of the
size and scope of the integrated survey.
19
Finally, we heard some specific feedback on the spreadsheets. Many of those who
responded-by-spreadsheet mentioned how convenient it was to organize their data.
However, one consistent finding with the spreadsheets is that by splitting them into
manufacturing and non-manufacturing components, we have added a layer of
complexity to the response process. On screen is a quote from a respondent who
mentioned: “Once I figured out it was there, I loved having the spreadsheet. A
challenge is that manufacturing is broken out from the other stuff. [If it was all
together] I could consolidate back to my consolidated financials and then that way I
know that you have all of the pieces.” Said another, who only had non-manufacturing
establishments, “Having all the pieces within that one Excel spreadsheet may have
shaved off two to four hours - because everything is one place.”
19
Does the order and structure of the
integrated survey make sense?
6. Company-level data are most accessible and easiest to report.
Industry-level data are most challenging to report.
7. Respondents want content organized by topic, and a direct and
clear submission process.
8. Respondents like respond-by-spreadsheet, but it must take a
holistic approach to the company.
20
So, does the order and structure of the integrated survey make sense?
Well – we know that company-level data are most accessible and easiest to report, and
industry-level data are the most challenging to report. We also know that topic must
play a part in either instrument design or supplemental materials or both, and that
respondents want to be able to clearly identify where they are in the response process.
Finally, we know that while some respondents liked respond-by-spreadsheet, we have
to design the spreadsheet to be more reflective of the company as a whole.
20
Is the integrated survey burdensome?
Number of Companies by Cumulative Total
Number of Minutes to Complete Module 1,
Overall and by Mode of Completion
Approximately how long did it take to
complete the AIES questionnaire modules for
this company, including time spent reviewing
instructions and gathering the necessary data?
N = 15
Low
180 minutes (3 hours)
High
2,385 minutes (≈40 hours)
20
Number of Companies
Mean 970.4 minutes (≈16 hours)
25
20
20
15
14
13
13
10
9
5
6
4
4
5
4
0
Overall
30 minutes or less
Qualtrics
31 to 60 minutes
61 to 120 minutes
Spreadsheet
121 minutes or more
21
Alright, on to the perennial question of burden. Is the integrated survey overly
burdensome?
Remember that we can break burden into two phenomenon: real and perceived
burden. Generally, real burden is the amount of time and resources that a company
must commit to completing a survey, and perceived burden is a respondents
perception of how cumbersome or challenging a survey is. There’s a bunch of literature
to suggest that both are important – that which is perceived as real is real in its
consequence, and burden is no exception.
In terms of actual burden, I will be the first to admit that our measures are crude. On
the left side of your screen are measures of central tendency for respondent-reported
actual burden on the RAS. On average, respondents reported that the survey took
about 16 hours to complete. That ranges from a reported low of 3 hours to a reported
high of 40 hours. We have no way of verifying if these reports are true, and we do have
literature that suggest that respondents are poor reporters of actual burden.
On the right side of the screen is captured paradata from Module 1 of the survey – the
company level data. I’m only looking at Module 1 because it is the most
straightforward – linear – of the modules. Qualtrics captures “the total duration (in
21
seconds) of the survey response. This includes time with the survey open, and time
spent away from the survey if the user closes out of the survey and returns.” Still, I’m
not as confident in the timing capture as reflective of burden – we can’t assume that
respondents are working on their surveys the entire time they have them open. Take
this with a grain of salt – 20 companies took half an hour or less to complete, and
another 20 companies took 2 or more hours to complete.
21
Is the integrated survey burdensome?
Compared to annual surveys you have answered in previous years,
how easy or difficult did you find completing the AIES
questionnaire modules?
N
Percentage
Extremely difficult or
somewhat difficult
6
46.2
Neither easy nor
difficult
4
30.8
Somewhat easy
3
23.1
Extremely easy
0
0
Compared to annual surveys you have answered in previous years,
how much time did it take to complete the AIES questionnaire
modules?
Less time
3
23.1
About the same
amount of time
3
23.1
More time
7
53.9
• 6 revoked consent
• Mixed on time savings
• Estimate vs. true values
22
We did specifically ask about perceived burden in the RAS. No one called the survey
“extremely easy” compared to the current annual surveys. Some respondents noted no
difference from current annuals, while others said it was “somewhat” or “extremely”
difficult comparatively. This is echoed in asking about relative time to complete – 3
said it was less time than usual, and another 3 said it was about the same, but 7 said it
was more time to complete the pilot than to complete the current annual surveys.
Now, the thing about the RAS is that they are all respondents. But, we also have heard
from non-respondents, especially those that dropped out of the pilot. Six companies
dropped out and the main reason was the burden. Said one, “it is too much to enter
information for all of these [250] establishments.” Said another, “each question for
each location is way too time consuming.” It seems that respondents are not used to
being asked for this level of data for each establishment. We may need to consider
scaling back content or socializing respondents to our request.
We asked about burden in debriefing interviews and respondents also made mention of
burden when they contacted us. Some respondents admitted that they didn’t really
spend any more or less time on the pilot than they would have usually spent. These
respondents were also most likely to notice that the content hadn’t really changed all
22
that much, just been put into one survey. A few mentioned that it was taking more time
than usual, and a few mentioned that they liked having everything presented in one
survey at one time so that they could compile the data all at once.
We also specifically asked about estimates vs. real values, and most respondents said
that whenever possible, they use the true values reflected in their accounting software.
Those that did estimate talked about allocating responses based on establishment size
or other variables specific to that industry, for example, proportion of customers in the
region in which the establishment is located.
22
Is the integrated survey burdensome?
9. There is wide dispersion in the amount of time to complete the
pilot overall and Module 1 in particular.
10. Reports on the amount of time to complete relative to the current
annual surveys are mixed.
23
So, is the integrated survey overly burdensome?
It’s hard to say. We know that there is wide dispersion on the perceived and actual
burden of the survey – some say it’s taking less time or about the same amount of time
as the current annual surveys, some say it is taking more or much more time than the
current annual surveys. This is a point where we may need to do some additional
research as we move forward.
23
Is the integrated survey using appropriate
units of analysis?
Sum of Establishments Compared to Company-level Reporting for Four Variables
Number of companies by value match type
80
75
70
65
60
55
50
45
40
35
30
25
20
15
10
5
0
11
15
13
22
13
12
19
22
22
13
7
34
32
Total employees
33
Annual payroll
Missing
Exact match
Within 10 percent
Q1 payroll
9
35
Total revenue
More than 10 percent difference
24
Let’s take a look at the question of the response units on the pilot survey. This is the
first time we have asked companies to provide responses for key variables across
several units of analysis. On screen now is the results of how those units may or may
not be related on the survey. I looked at the values from Module 2 – the establishment
level – for manufacturing and nonmanufacturing establishments, and compared it to
the response from Module 1 – the company level – to see if the parts equaled the
whole.
This analysis is quick and dirty on data that have not gone through our typical edit
checks. Note that for each of the four variables, about half of companies – between 32
and 35 – were missing one or more components to run this comparison. This is the
bringing together of unit and item nonresponse: some did not provide the response to
Module 1, so, nothing to compare to. Some completed Modules 2 for only some
establishments, so some establishments are missing. And some missed individual
questions within a given establishment, so item nonresponse.
But, I also want to draw your attention to the number of cases where we CAN do the
comparison – the other approximately half. Of those, total employees had the highest
perfect match rate – the sum of the establishments was equal to the total company-
24
wide data for 22 of 78 companies. It also had the highest approximate match – the sum
of the establishments and the company total are within ten percent of each other. Note
that we did have cases where the sums were not within the ten percent tolerance.
Some of this is due to measurement error – respondents not understanding that these
things should be summative. But some of this mismatch is also due to rounding error,
where respondents may have entered value rounded to the nearest thousand at the
company level but then entered an exact value at the establishment level. Some may
be due to entry errors, where a respondent “fat fingered” a wrong digit along the way. I
suspect, however, that one of the shortcomings of the pilot programming is coming into
play here. Remember earlier when I mentioned that respondents tended to answer
Module 1 – the company level – in one sitting, but then completed module 2 over
multiple periods. The issue is that once the respondent submitted their answers to
Module 1, they could no longer reference those answers: literally, many respondents
gave us their company totals but then could not see those totals as they went to answer
at the establishment level. I think this flaw proved fatal: respondents weren’t sure
WHAT they were summing up to in the first place. A suggestion might be, then, that the
next iteration of the instrument explicitly build this relationship – such that respondents
can check the sum of their establishments against their reported company totals.
24
Is the integrated survey using appropriate
units of analysis?
• Multiple EINs:
“I guess I thought that since we had different EIN numbers, we
reported completely separate and I assumed they were doing
the same thing too.”
• Headquarters
“The sum of the establishments is not equal to the companywide sums. Income for HQ is an expense at the plant level.”
• Industry and establishment record keeping
25
We do see some conversation around the units in our debriefing interviews, too. None
of this is necessarily “new” – you may be familiar with the reoccurring measurement
issues in economic surveys generally from other surveys you work on. But, the larger
issue with the AIES is that it brings everything together into one collection effort, so
issues that were not as ‘noticeable’ on other surveys may bear additional consideration
for this survey.
The first issue I want to bring up is the inclusion of multiple EINs for a single company.
We had a couple of companies in the pilot who told us that they had more than one
EIN. For some, this is the first time we’ve asked for reporting on all EINs within the
company – so for example, one respondent said that they are four separate companies
with the same parent company – with their own tax id and financial statements. Said
another – “I’ve never reported on behalf of the second company before. I guess I
thought that since we had different EIN numbers, we reported completely separate
*and I assumed they were doing the same thing too.*” This quote highlights that the
respondent has never reported for the “other” EIN, but also that they have never been
ASKED to report for the other EIN.
Next up, we need to talk about headquarters. There are two issues with company
25
headquarters that we need to be aware of. The first is one that we know about from
other surveys: some companies treat data from the HQ as separate from the other units
within their business. The quote on screen sums it up: for some companies, income for
a headquarters is an expense at the plant level, and this intercompany billing can make
reporting out the data challenging. While this may not be new information for us,
again, the challenge is that this integrated survey asks for data at the headquarters
sometimes for the first time. We will need to handle these establishments and the
corresponding industries with careful consideration.
At the same time, the headquarters presented another issue: we did have one – but
only one – case in the pilot that thought that by “company-wide” data in Module 1, we
meant data representing only the headquarters or the ‘administrative duties’. This case
talked about the headquarters as a “mothership” (his word!) containing the liquid
investments, real asset investments, and others, and it “owns” the “main core
operations” entities that then own other entities. As a result, he only reported data for
this ‘mothership’ in Module 1, and so his responses would not sum.
Finally, and I’m not going to linger here because this is a known issue: some companies
keep their records by establishment, and industry is challenging; some companies keep
their records by industry, and establishment is challenging. Some companies keep some
records at the establishment and others at the industry. This mismatch will continue to
be a source of error in responses.
25
Is the integrated survey using appropriate
units of analysis?
11. Summing to the total works for some – but not all – companies, and
that sum needs to be included as a check when asking for subcompany data.
12. We will continue to have unit errors in the integrated survey, but we
can mitigate with flexibility and clear communication.
26
Are we using appropriate units of analysis?
26
What do the resultant data look like?
• Variable name & Survey
Question
• Response variable type &
Restriction
• Remarks/ Notes
Form View
• Different variable name with the
same question
• Different question with the same
variable name
• Different code for categorical data
Spreadsheet
Consolidated Data
Codebook
• A Single-tab Excel Spreadsheet
for each module
• Consolidated variable names &
responses
27
And finally, we wondered what the data from the pilot might look like. Recognizing that
that the production instrument will be structured differently and use different
collection methods than the pilot, we can still learn from the stream of data we
received from the pilot.
On screen now is a diagram that our summer Civic Digital Fellows put together. You can
see that because we offered two modes of completion – spreadsheet and page-bypage, or form view – we had to start analyzing the data by knitting together responses
from these two sources. This included different or overlapping variable names and
response values, as well as some duplicate response. Once the CDFs created a
consolidated datafile with a corresponding codebook, I combed through each of the
responses to identify duplicate entries, and made retention rules for which data to
retain. In total, we had about 60 duplicate establishments and 7 duplicate industries,
but note that this phenomenon is due in part to programming limitations.
It does, however, raise the issues of signaling completion status to the respondent,
allowing respondents to move about freely within the instrument, and bringing
together multiple sources of information into one logical data file.
27
Research Question
How are respondents completing the
survey?
Findings
1. The response process for the AIES Pilot is similar to that of the current annual surveys.
2. More companies reported data at the company-level, and data were most complete at the
company-level, compared to the establishment and industry levels.
3. Most companies responded by page-by-page method at the company-level, and most
companies responded by uploadable spreadsheet at the establishment and industry levels.
4. Content retention based on industry criteria, and adding NAPCS, will continue to be an issue.
Can respondents provide answers to
the questions on the integrated survey? 5. Survey structure and capabilities need to be flexible to meet respondent needs.
Does the order and structure of the
integrated survey make sense?
Is the integrated survey burdensome?
Is the integrated survey using
appropriate units of analysis?
What do the resultant data look like?
6. Company-level data are most accessible and easiest to report. Industry-level data are most
challenging to report.
7. Respondents want content organized by topic, and a direct and clear submission process.
8. Respondents like respond-by-spreadsheet, but it must take a holistic approach to the
company.
9. There is wide dispersion in the amount of time to complete the pilot overall and Module 1 in
particular.
10.Reports on the amount of time to complete relative to the current annual surveys are mixed.
11. Summing to the total works for some – but not all – companies, and that sum needs to be
included as a check when asking for sub-company data.
12. We will continue to have unit errors in the integrated survey, but we can mitigate with
flexibility and clear communication.
13. Pilot data came from several streams of data that had to be combined to analyze.
14. Programming limitations may have led to duplicate responses.
28
Ok, whew. We’ve covered a ton of ground today. On screen now are those research
questions with the corresponding findings.
What we see is that this method of piloting has provided us with a ton of information
that we otherwise would not have about this integrated survey. Circling back to the
beginning, these findings are representative of the issues we will face as we move
toward production of the AIES. They also set a possible agenda for a next phase of pilot
research. This includes expanding our research on response-by-spreadsheet, pushing
our data collection platforms to be flexible to respondent needs, better understanding
the perceived and actual burden of the integrated survey, and beginning to identify
what communication needs respondents may be bringing to the table.
There’s still much work to be done, but this pilot has provided our first glimpses into
concrete response to the AIES instrument. We have proven that we as an organization
can be responsive in new and innovative ways, and I look forward to building on this
momentum and preparing for additional rounds of research.
Thanks!
28
| File Type | application/pdf |
| File Title | Microsoft PowerPoint - Phase I F and R |
| Author | Melissa A Cidade (CENSUS/EWD FED) |
| File Modified | 2024-02-15 |
| File Created | 2024-02-15 |