Title Lorem Ipsum

Status: Original and derived artifacts are available for this document.

Download: pptx | pdf | html

ANNUAL INTEGRATED ECONOMIC SURVEY
_____________________________________________________________________________________________________________

COGNITIVE TESTING RESULTS
ROUND II
Rebecca Keegan, Aryn Hernandez, Kristin Stettler
Economic Statistical Methods Division
1

The Census Bureau has reviewed this presentation for unauthorized disclosure of confidential information and has approved the disclosure avoidance practices
applied. (Approval ID: CBDRB-FY23-ESMD011-001 ) The views expressed in this presentation are those of the presenter and not those of the U.S. Census Bureau.

TABLE OF CONTENTS
□ OVERVIEW

□ WRAP UP

□ DEMOGRAPHICS

▫ DELEGATION

□ RESPONSE OPTIONS

▫ MODULE GROUPINGS

□ MODULE ONE
□ COMPANY SUMMARY
□ PRIMARY BUSINESS ACTIVITY

▫ BURDEN
▫ TIMING AND SUBMISSION
□ KEY TAKE AWAYS

□ MODULE TWO
□ MODULE THREE
2

OVERVIEW
ROUND ONE RECAP
□ Participants were introduced to the organization of their establishments by sector. Initial reactions to grouping this
way was well received
□ Burden was anticipated to be about the same in terms of pulling data, but respondents felt time savings would be
in lack of redundancy with separate surveys &/or less need for coordination of due dates
□ Respondents expressed preference to answering via spreadsheets and being able to utilize pdfs
□ Some feedback as to not being aware of delegation feature in e-corr
□ Participants generally expected certain data (e.g. addresses) to be pre-filled

ROUND TWO RESEARCH GOALS
□ Gather further feedback regarding the content of the modules and how they relate.

▫ Should payroll be separated?
▫ How does a need for delegation play into the organization of the modules?
□ Further explore respondents’ preferences for acquiring data at the establishment versus industry level

▫ How does the level of industry affect this?
□ What are respondents' thoughts about organizing their establishments by NAICS?

3

OVERVIEW
□ Over twenty interviews conducted from 1/20/2022 to 2/17/2022
□ Hour long interviews conducted over MS Teams
□ Participants recruited via Qualtrics.
□ Protocol Structure:

▫ Respondents were shown several “mockups” of certain key screens
▫ They were told these screens were not going to reflect the final design of the instrument but were a tool to
investigate the principle of the content/functionality.

▫ Respondents reviewed a consolidated list of the question topics within each module and were
asked to discuss any potential areas of difficulty and general thoughts about the groupings.

▫ Finally, respondents were asked to reflect on the concept of AIES as a whole, their thoughts about
the overall design, and their thoughts on its affect on their burden.

4

DEMOGRAPHICS: TITLES
Respondents were selected from companies that had responded to at least two annual Census surveys in the past.
The companies were of medium to large size. Several operated in more than one sector, but not all. Others often
operated in more than one NAICS, within one sector.
TITLES:
□Accounting Director
□Associate Vice President of Finance and Risk
□Chief Financial Officer
□Cooperate Controller
□Director of Finance
□Manager of Financial Reporting
□President
□Senior Accountant
□Treasury Manager
5

DEMOGRAPHICS: NAICS
221100

Oil and Gas Extraction

524100

Insurance Carriers

561900

Other Support Services

441000

Motor Vehicle and Parts Dealers

722000

Food Services and Drinking Places.

623000

Nursing and Residential Care Facilities

541300

Architectural, Engineering, and Related
Services

813100

Religious Organizations

312100

Beverage Manufacturing

624000

Social Assistance

445000

Food and Beverage Stores

236000

Specialty Trade Contractors

311000

Food Manufacturing

238000

Specialty Trade Contractors

622100

General Medical and Surgical Hospitals

325900

Structural clay products, nec

6

7

OVERVIEW
FINDINGS
□ The overview made sense to respondents

□It was very common for respondents to report that they would want the ability to see the
full list of questions prior to answering the survey, (list of topics; pdf)
▫ Many respondents gather all the relevant data, then enter it all in as a final step.
▫ “I would probably pull the instructions to see everything that's being collected.”

T E R M I N O L O G Y : “Company Level Data”
□ “Company level” was understandable and generally evoked the correct assumptions, but

there was a strong preference for the word “Consolidated Company Data”.

▫ Some thought “level” was a little vague. Others said “Company Level” referred more to
organizational structural and less about monetary questions
▫ “Consolidated” seems to more appropriately cover both structure and financial information
8

9

REPORTING METHOD
FINDINGS
□Essentially all respondents indicated wanting to use the spreadsheet option

Interactive Spreadsheet:
□ Those who were asked about an interactive spreadsheet mentioned that they would still want a
downloadable template.

□ Many R’s like to print off the template.
□ Some fear losing data by going back and forth in the instrument or leaving their responses to sit
too long/ being erased

□ Difficult to discern how this would be beneficial in terms of time savings, as many R’s want a template
to fill out and/or delegate.

□ A few people might use, but majority wanted to use a template and delegate/share it.
Recommendation: Provide both options (downloadable spreadsheet + online version).

10

11

MODULE ONE
FINDINGS
□In general, the higher the level of the request, the easier the data will be to pull.

▫The information requested in Module One has a greater likelihood to be the easiest to pull.
▫Payroll and Cap ex were mentioned repeatedly as data that would require reaching out.
▫There are cases wherein the data are easier to first pull at the establishment level
▫One person mentioned cap ex in particular would require gathering at the establishment level
before rolling up. There are likely other topics like this as well, (e.g., robotics was also
mentioned)

Other considerations:
□ Some data will be difficult to pull simply because of the volume (e.g., capital lease

agreements) or may just take time to gather (e.g., new lease agreements)

□ R’s may need to reach out to multiple people/departments, regardless of how easy or

difficult the data are to pull

12

*Fictional Data 13

Company Summary
FINDINGS
□This higher level grouping seemed to make sense to respondents

▫Some commented that seeing this breakout would be helpful.
□Many respondents assumed they would only be listed in one sector

▫(Note that R’s do not generally differentiate between our terminology “sector” versus “industry”.

They’re likely assuming that their one primary business line coincides with our definition of a sector.)

□There was some question how warehouses would be treated. Another mentioned e-

commerce shipping centers

□Recommendation: May be helpful to offer an explanation of the groupings/sectors.
14

15

Primary Business Activity
FINDINGS
□ Many found this to be a good way to organize the data. “Straightforward” “Prefiltered is easier”

▫Some R’s did not immediately notice the NAICS description (need to make more prominent)
▫Several R’s mentioned the description of NAICS was helpful
▫“This is how they do it for insurance purposes too. Makes a lot of sense. Easier to understand to be
honest”

□There was variability in respondent’s familiarity with NAICS
□Respondents often don’t understand how/why they’re classified a certain way
UPDATING NAICS
□ The functionality seemed clear for how to update their industry.

▫Some mentioned they likely would use this feature.

16

Primary Business Activity
ESTABLISHMENTS WITH MULTIPLE PBA’S:

□Some R’s mentioned they did have locations wherein more than one PBA might be
relevant. This pre-grouping might make reporting easier, as opposed to breaking out.

▫ “Do have more than one code [line of business]. Picking one, that makes it easier…Won’t paint a picture of
what's going on in that facility.”

▫ Be clear in instructions that respondents are not being asked to exclude data
□ Some R’s assumed this would be where they would update location data (address,
Op. status)

▫ Recommendations:

▫ There should be an area for them to enter in comments regarding any confusion/frustration with their
classification.

▫ PBA should be addressed next to any questions regarding COS (address; operational status) May need
to be on its own page to minimize any confusion and allow respondents to focus on this task alone.

17

Primary Business Activity
RECORD KEEPING AND NAICS
□ Industry questions divided at the 6-digit NAICS, likely more difficult to parse than how R’s are used to

reporting.

▫ For example, ACES is rolled up to four digits not six
▫ This has implications for R’s ability to provide estimates

▫ Respondents will sometimes combine data that we would like them to split (they have one bucket but
we want two)

▫ Respondents often give us what’s easier for them.
▫ “So talking about the buckets earlier, might not be exactly how we have our buckets [organized]. We might combine

them where you want them separate. It gets time consuming and very manual to realign into these buckets. The more
general the bucket you ask for potentially more easy to provide, not needing to get down to a granular level. One broad
general number then I don't need to split it out.“

Recommendation
□Give a heads up for AIES fundamental changes. Especially if there will be a change in industry reporting (i.e., from 4 digit to
more complex) offer guidance and let respondents know ahead of time.

▫ “I go in and use last years guidelines. I try to follow the same process from earlier years.”
□ Note clearly: “For the purposes of this survey, all business lines will be grouped into one industry code per establishment”

18

19

MODULE TWO
What is easier for respondents, obtaining data at the establishment
level, or industry level?
☁☁☁
□ This is a nuanced question because generally a respondent will have to pull establishment

data in order to roll up to industry. Industry is not its own entity.

▫For this reason, sometimes pulling at the establishment level is the likely first step.
□ Rolling up to the industry level is simple for some- but it also depends what we mean by

industry. What level industry? Requiring data at the 6+ level NAICS might essentially be
equivalent to an establishment level for many companies.

▫Flexibility would be ideal
▫Recommendation: Only ask respondents to confirm their establishment data at the NAICS level
that is the broadest necessary level

☁ General rule of thumb: The more detailed the data request is by establishment, the more
difficult it is to collect.☁

20

MODULE TWO
OBTAINING DATA AT THE ESTABLISHMENTS LEVEL
□ Respondents' ability to pull data at the establishment level is dependent on the individual company’s

records
▫“Boss wanted an [item] report. How many were sold at the store, on that date.”
▫“Absolutely can roll up to engineering level. Company model is easiest opposed to by location. By
industry is easier than by location.”
▫Establishment is definitely easier- first then break out, because you'll always have the data based on
that plants [activities]...but we do look by product line. Readily available. I have legal entity drilled in
my end, so start with establishment to look for data.
□ Sometimes the ability to pull data at the establishment level is dependent on the state the

establishment is in. Data may be captured differently and require different ways of looking up the
information.

▫Examples: Those in the utilities industry; Insurance companies/clinics often do not parse
things by location in their records; Contract based services often do not organize records by
establishment.

▫“Financials not broken out by location, but by service lines or business lines”
▫Recommendation: Alert respondents of fundamental collection changes prior to mailout

21

MODULE TWO
OBTAINING DATA AT THE ESTABLISHMENTS LEVEL
□Some R’s will need to reach out to several people/departments. (payroll was frequently mentioned)

▫ “It requires multiple people regardless of module.”
□Some R’s will need to reach out to each establishment if the question is detailed:

▫“But if the info is asked by each operating unit/call center it will take more time. Consolidated
means less time to fill out the report”

FREQUENTLY MENTIONED TOPICS THAT MAY BE DIFFICULT
Some are industry specific
□E-Commerce: “E-commerce is a little tricky- don't have a good way to break that out. What is e-commerce?”
□Recommendation: Group examples of e-commerce by industry

22

23

MODULE THREE
ROLL UP ABILITIES
□ Completely dependent on their record structure. For some will be simple, some companies

have their location data already broken out, so it’s just another step to roll it up. Some
mentioned that rolling up to company level is easiest.

□ One R mentioned this would require manual work because their system cannot run reports on

all their locations at once. Others do not separate their branch data (the utilities person) not
easy at all at the establishment level.

▫“With it currently we can combine 10 or 15 stores and then another 10 or 15 then repeated then
group them all together. manual process by individual store.”

▫“By location just because how our system works.”
▫He was saying they could roll up by industry but because of the way their software is structured, it's
geographical. So location based is actually easier. Rolling up is just an extra step.

24

MODULE THREE
POTENTIAL AREAS OF DIFFICULTY
□ Inventories: Some concern (potentially Manufacturing specific-maybe not all?)
□ Cap Ex: Record organization varies greatly. Brough up several times.
□ Cap Ex for some industries may be particularly difficult to pull, either by establishment or

industry because the equipment moves.

▫E.g., equipment currently in one location may have different uses for different activitiesdifficult to know how to break that out.

▫ Medical industry break out their records by specialty- difficult to do so if the equipment is utilized

by multiple locations.
▫Cap ex may only be associated with locations where property taxes have been paid, even though
the equipment might move from place to place. It was impossible to calculate cap ex if they didn’t
own the location.
▫Recommendation: consider offering guidance for this scenario.

□ Recommendation: If respondents have already provided all relevant establishment figures for a

given NAICS, prefill that data so the respondent is not doing redundant work.

25

DELEGATION
VARIOUS USES
□ Nearly all respondents thought the ability to delegate would be beneficial.
□ Some R’s will have to reach out per topic (i.e., cap ex, payroll, R&D etc.) because each topic is

associated with different departments.

▫“Everyone has their piece of the pie.”
□ Some R’s wanted the ability to see all similar topics grouped together. This was relevant for

delegation wherein you may have to reach out to numerous other departments and it would
be good to have all relevant questions ready to send them.

▫“I think it's just how we collect and report versus your buckets takes time to sort. I don't want to send to
[someone] in payroll fill out page 2, 16, 42, and 26.”

26

DELEGATION OF PAYROLL
DELEGATING PAYROLL:
□ Some respondents were enthused by the idea of breaking this into it’s own section, some were
more neutral to the idea.

▫ “Then for payroll we have to go to HR once, put it in and be done- versus mixing payroll and financials
going in 2 different directions.”

▫ “Not everyone has access to the payroll, so if we wanted to get all the info about cap ex then everyone has
that, but only higher level have payroll.”

□ Payroll can be a sensitive topic, smaller companies may be at a higher risk of exposing salaries;
more of a benefit to have it separated.

▫ “I wouldn't want like the R&D people to see payroll. Don't want that mixed in with operational.”
▫ “not too many people share [payroll access]. portion that one out. fixed assets aren't going to know
payroll. Great one to have maybe sent as a different part. Don’t want people knowing what you make”

□ Payroll is frequently housed by a third party (e.g., ADP), or housed internally in its own department.
27

MODULE GROUPINGS
THOUGHTS ON GROUPINGS

□Some seemed to assume that the topics would be grouped together– but separate
modules have overlapping topics (payroll, cap ex) which may not have been immediately
clear.
▫ “I think the more consolidated in general, the easier to pull the numbers…takes less digging.”
▫ “For me it's all in one spot (similar records) so really going category by category- finding the info.
Going to same spot doing the same thing. I don't have to log into all these different places.”
▫ “To me it's easier to break it out into those categories than mixing them all together. Summary
module level. Just payroll, just expense. Capital expense and payroll expense… those are
independent. Everything payroll related- ALL in one section regardless of what asking for. Anything
for capital specific to expenses. What takes precedent.”

□Recommendation: To mitigate this slightly, clearly lay out the topics within the modules
before the respondent answers questions.
28

BURDEN
THOUGHTS ON BURDEN
□ Generally, the idea of something consolidated seemed good but the number of surveys they

receive, and their company’s structure plays a big role.

□ Respondents who only respond to 1 or 2 surveys may perceive AIES as more work/more

difficult

▫In general, the benefits of AIES are clearer for complex companies
▫Recommendation: In pre-canvas or other initial contacts regarding of changes before mailout,
consider ways to “market” the survey in a way that highlights how these changes may benefit
companies large and small.

□ For companies with more compartmentalized departments, consolidating the surveys may

create more difficulty.

▫Sometimes one individual may be tasked with filling out specific surveys, so this change may seem
overwhelming.

29

BURDEN
THOUGHTS ON BURDEN
□ Some mentioned concerns about saving progress as you go.
□ Several mentioned concerns about the time it would take to complete

▫ “What I'm wondering is how long it would take me to get through this. I don't know about timesaving; it's pulling the data
and confirming it.”

▫ If it was one survey for everything NO thank you. Would be hard to delegate…I want them separately otherwise have to
block off a whole week for this one survey.”

□ As with R1 findings, compiling data is generally most time-consuming
□ Readjusting/reorganizing the data can be time-consuming as well. May be particularly difficult with regard

to breaking out data by detailed NAICS.

▫ “Large part will be how do you structure this. How different are your buckets to ours?”
□ May not really save time but be easier to keep track of one versus multiple, Just ONE due date.

▫ “I hope it would be better, because for me I would be sending to people. All at once and not needing to track and not
filling out the front and back stuff over and over.”

30

SUBMITTING DATA & TIMING
SUBMITTING DATA
□ Majority expressed preference for submitting all the modules at the end.

▫ This has implications for linking the modules within Centurion, versus separating them in respondent
portal

TIMING (MAILOUT) CONCERNS
□ Many R’s mentioned concerns about timing of submission and assumed they would need to extend
the submission date.
▫ “I think I would rather have one (survey), but not during property taxes which are up until March and
April. I already have enough.”
□One mentioned calendar vs fiscal year
▫ “We follow a fiscal year not a calendar year. So, I have to pull it twice. Then I have to clean up the
irrelevant dates.”

□ One respondent mentioned how helpful providing an alternate contact would be for cases when31
the primary contact is temporarily unavailable or switching roles

Key Takeaways
FINDINGS
□Recordkeeping is not uniform- it's organized by what's important to that individual
company.
□AIES may create more difficulty for respondents who will have to reach out to separate
departments for each topic in the survey (cap ex; payroll etc.)

□Respondents want the ability to delegate topics within the survey (often that align with
typical departments titles).
□Respondents do not generally differentiate between industry driven language e.g., sector,
industry, NAICS. Use plain language to describe their NAICS.
□Respondents often do not understand why certain establishments have been classified
into separate industries.
□This has implications for the burden associated with dividing out establishment data that within
their records, is normally combined.

32

Key Takeaways
FINDINGS
□Even for companies with very organized and efficient record systems, AIES will take a long
time to complete. Prefill information where possible (use module 2 data to inform module
3). Have flexible submission timelines.
□Preference for “Consolidated Company Data” over “Company Level Data”
□Essentially all respondents indicated wanting to use the downloadable spreadsheet
option
□Respondents often want to gather data first, then enter in the data all at once.
□There was a general preference for submitting their data (across all modules) all at once,
which has implications for design of the respondent portal.
33

THANK YOU
Q U E S T I O N S?

34