ABS Design and Technology Module Cognitive Testing Results

E. ABS Design and Technology Module Cog Testing F&R.pdf

Annual Business Survey

ABS Design and Technology Module Cognitive Testing Results

OMB: 0607-1004

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 0607-1004 can be found here:

Document [pdf]

Download: pdf | pdf

Findings and Recommendations from
Cognitive Testing for the Annual Business
Survey – Design and Technology Module
Prepared for:
Patrice Hall, Economic Reimbursable Surveys Division (ERD)
Aneta Erdie, ERD
Audrey Kindlon, National Center for Science and Engineering Statistics (NCSES)
John Jankowski, NCSES

Prepared by:
Krysten Mesner, Data Collection Methodology and Research Branch (DCMRB)
Aryn Hernandez, DCMRB

Economic Directorate
U.S. Census Bureau

October 26, 2020

1
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Contents
Research Objectives ................................................................................................................................ 4
Research Methodology............................................................................................................................ 5
Participants ............................................................................................................................................. 6
Findings and Recommendations .............................................................................................................. 6
General Findings and Recommendations ................................................................................................. 6
Finding #1: Estimated time to complete averaged approximately 30 minutes. ..................................... 6
Question Specific Findings and Recommendations .................................................................................. 6
A.

Design ......................................................................................................................................... 6

Finding #2: Design - #1......................................................................................................................... 6
Finding #3: Design - #1a, 1b ................................................................................................................. 7
Finding #4: Design - #2......................................................................................................................... 7
Finding #5: Design - #2a, 2b ................................................................................................................. 7
Finding #6: Design - #3......................................................................................................................... 7
Finding #7: Design - #4......................................................................................................................... 7
Finding #8: Design - #5......................................................................................................................... 8
Finding #9: Design - #6......................................................................................................................... 8
Finding #10: Design - #7....................................................................................................................... 8
Finding #11: Design - #8....................................................................................................................... 8
Finding #12: Design - #9....................................................................................................................... 8
Finding #13: Design - #10..................................................................................................................... 8
B.

Technology .................................................................................................................................. 9

Finding #14: Digital share of business activity ...................................................................................... 9
Finding #15: Cloud services purchases ................................................................................................. 9
Finding #16: Business technologies ...................................................................................................... 9
Finding #17: Purchased services and suppliers - #4, 4a, 4b................................................................... 9
Finding #18: Purchased services and suppliers - #5, 5a, 5b................................................................. 10
Finding #19: Purchased services and suppliers - #6, 6a, 6b................................................................. 10
C.

Innovation ................................................................................................................................. 10

Finding #20: New or improved services.............................................................................................. 10
Finding #21: New or improved goods................................................................................................. 11
Appendix A: Draft Questions, Round 1 ................................................................................................... 12
Appendix B: Interview Protocol, Round 1............................................................................................... 23
2
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Cognitive Interviewing Guide: Draft Content for 2021 ABS (4 th Year) ............................................... 23
Thanks for providing some background information. Now let’s look at the survey questions. ........... 24
Direct participant to look at draft content. ....................................................................................... 24
DESIGN MODULE - ABS (4th Year) ...................................................................................................... 24
Question 1 ........................................................................................................................................ 24
Question 2 ........................................................................................................................................ 24
All Questions ................................................................................................................................. 29
Appendix C: Draft Questions, Round 2 ................................................................................................... 30
Appendix D: Interview Protocol, Round 2 .............................................................................................. 40
Cognitive Interviewing Guide: Draft Content for 2021 ABS (4 th Year) ............................................... 40
Thanks for providing some background information. Now let’s look at the survey questions. ........... 40
Direct participant to look at draft content. ....................................................................................... 40
DESIGN MODULE - ABS (4th Year) ...................................................................................................... 40
Question 1 ........................................................................................................................................ 40
Question 2 ........................................................................................................................................ 41
All Questions ................................................................................................................................. 46
Appendix D: Interview Protocol, Round 2………………………………………………………………………………………………66

3
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Research Objectives
The Annual Business Survey (ABS) asks questions about the business owner(s) as well as various
characteristics about the business. The 2021 ABS cognitive testing included new questions in the design
and technology modules. These proposed questions focused on:

•
•
•

Design – asks about design activities within the business including monetary and personnel
resources.
Technology – asks about which digital technologies the business uses and /or purchases.
Innovation – asks if the business has introduced new or improved goods, services and business
processes. These questions were previously tested during the 2019 ABS content cognitive
testing.

These draft questions were discussed using cognitive interviewing. The objectives of these interviews
were to learn how respondents:
•
•
•
•

understand and interpret the data requests;
understand the terminology used throughout the questions;
keep records related to the data items; and,
perceive the burden that would be associated with answering the questions.

The round one design and technology module draft can be found in appendix A. The round one protocol
can be found in appendix B. The round two design and technology module draft can be found in
appendix C. the round two protocol can be found in appendix D.

Note: The two rounds of testing were conducted in close proximity, time-wise, from each other.
Between the rounds of testing, some changes were made to the draft materials. This report will attempt
to summarize findings from all interviews, but the primary focus will be on the findings from Round 2 of
the interviews, and provides recommendations based on those draft questions.

4
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Research Methodology
Between June and September 2020, Aryn Hernandez and Krysten Mesner from the U.S. Census Bureau’s
Data Collection and Methodology Research Branch (DCRMB) conducted 18 cognitive interviews over
two rounds of testing, regarding proposed new questions on design and technology. Participants in the
interviews were recruited from a list provided by the Federal Reserve Banks of Atlanta and Cleveland.
Cognitive interviews were the testing methodology used for this project. Cognitive interviews are used
to, ‘(a) understand the thought processes used to answer survey items, and (b) to use this knowledge to
find better ways of constructing, formulating, and asking survey questions’ (Forsyth and Lessler, 1991).1
Cognitive interviews traditionally focus on the four steps of Tourangeau’s (1984) cognitive response
model: comprehension, retrieval, judgment, and communication/reporting.2 Comprehension refers to
the respondent’s interpretation and understanding of the question’s language, structure, and grammar.
In order to answer the question, a respondent must understand what information is being requested on
the survey. Retrieval is the step where relevant information is obtained, either from records or from
memory. The next step, judgment, describes the respondent’s evaluation of the completeness or
relevance of the data obtained. It is here that estimates are made based on partial or incomplete data.
The last step, communication or reporting, deals with mapping the response to the answer space
provided and possibly altering the answer.
While Tourangeau’s model is suitable for household and social surveys, the establishment survey setting
presents additional factors that must be considered. First, instead of or in addition to a reliance on
memory, establishment surveys rely heavily on records and the information contained within them.
Second, organizations tend to have distributed knowledge. Some people are experts in one type of
information, while others keep information about something else. Third, competing priorities, both for
the organization and the individual(s) completing the questionnaire, mean that the survey sometimes
does not receive the amount of attention that researchers and data collectors would like. Finally,
organizations regularly authorize only a few individuals to release data. If the data provider is not
authorized to release the data, an additional step must be added to the response process. Tourangeau’s
model was expanded by Sudman et al (2000) to account for these factors. 3

Forsyth, B.H. and Lessler, J.T. (1991). ‘Cognitive Laboratory Methods: A Taxonomy.’ In Measurement Errors in Surveys, P.P. Biemer, R.M.
Groves, L.E. Lyberg, N.A. Mathiowitz, S. Sudman (eds). New York: John Wiley & Sons, Inc.
2

Tourangeau, R. (1984). ‘Cognitive Sciences and Survey Methods.’ In Cognitive Aspects of Survey Methodology, T.B. Jabine, M.L. Straf, J.M.
Tanur, and R. Tourangeau (eds). Washington, DC: National Academy Press.
3

Sudman, S., Willimack, D.K., Nichols, E., and Mesenbourg, T.L. (2000). ‘Exploratory Research at the U.S. Census Bureau on the Survey Response
Process in Large Companies.’ Paper prepared for presentation at the Second International Conference on Establishment Surveys, Buffalo, NY.

5
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Participants
Participants included contacts established by the Federal Reserve banks of Atlanta and Cleveland.
Businesses recruited covered a variety of industries that provide services and goods. Participants were
higher level employees with roles such as owner, president, CFO, CEO, etc. and, given their position in
the company, they were able to answer most of the questions that were tested.
Table 1. Count of businesses by type and round
Count

Round 1

Round 2

Total

Services

Goods

Total

Findings and Recommendations
General Findings and Recommendations
Finding #1: Estimated time to complete averaged approximately 30 minutes.
After reviewing the draft module, most respondents gave a time estimate to complete the module
between 10 minutes and 45 minutes.
Recommendation(s):
•

No recommendations needed.

Question Specific Findings and Recommendations
A. Design
Finding #2: Design - #1
Most participants were able to answer this question. The bulleted lists were helpful. A few participants
mentioned some design activities that they thought were missing from the lists:
•
•
•

Building out IT systems and business processes
Speed of how fast the product can be made
customer service response for design product

6
DRB Clearance Number: CBDRB-FY21-ESMD001-004

One participant asked what was meant by “sustainability or resilience” and wondered if this was
referring to environmental sustainability or the product’s sustainability in the market. Clarification
needs to be provided for this.
Recommendation(s):
•
•

Consider moving the list of service-related activities to above goods related activities to be
consistent with the innovation section.
Clarify the ‘sustainability or resilience’ category.

Finding #3: Design - #1a, 1b
In testing these questions performed well.
Recommendation(s):
•

No recommendations needed.

Finding #4: Design - #2
This question is fine as written, but service-based companies are likely to say no to everything.
Recommendation(s):
•

No recommendations needed.

Finding #5: Design - #2a, 2b
In testing these questions performed well.
Recommendation(s):
•

No recommendations needed.

Finding #6: Design - #3
In testing this question performed well.
Recommendation(s):
•

No recommendations needed.

Finding #7: Design - #4
Nearly all participants were able to answer this question easily. Other positions mentioned by
participants:
•
•
•

Electronic Technicians
Business Analysts
Technicians
7
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Recommendation(s):
•

While question is fine as written, an ‘other’ category could be added.

Finding #8: Design - #5
In testing this question performed well.
Recommendation(s):
•

No recommendations needed.

Finding #9: Design - #6
In testing this question performed well.
Recommendation(s):
•

No recommendations needed.

Finding #10: Design - #7
In testing this question performed well.
Recommendation(s):
•

No recommendations needed.

Finding #11: Design - #8
In testing this question performed well.
Recommendation(s):
•

No recommendations needed.

Finding #12: Design - #9
In testing this question performed well.
Recommendation(s):
•

No recommendations needed.

Finding #13: Design - #10
Most participants were able to answer this question easily. A few participants expressed that ‘trade
secrets’ were a very important means of protecting IP.
8
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Recommendation(s):
•

Consider adding “trade secrets” as an answer category.

B. Technology
Finding #14: Digital share of business activity
Most participants were able to answer this question easily. The column headers were easy to
understand. Participants that reported collecting a single category in both digital and non-digital formats
would select whichever column was the primary method.
Recommendation(s):
•

Consider adding instructions telling respondents how to deal with situations in which they
collect a type of record in both formats.

Finding #15: Cloud services purchases
Most participants were able to answer this question easily. The column headers were easy to
understand. If more than one answer were appropriate, participants would select the predominant one.
A few participants were confused by what ‘software purchases with embedded cloud features’ meant.
Recommendation(s):
•
•

Consider adding instructions telling respondents how to deal with situations in which they
collect a type of record in both formats.
Add help text for ‘software purchases with embedded cloud.’

Finding #16: Business technologies
Most participants were able to answer this question easily. The column headers were easy to
understand. Some participants were unsure of what some of the categories were referring to.
Recommendation(s):
•
•

Consider adding instructions telling respondents how to deal with situations in which they
collect a type of record in both formats.
Include help text for all categories.

Finding #17: Purchased services and suppliers - #4, 4a, 4b
Most participant felt that they could answer this question. All but one participant felt comfortable with
providing vendor names. While most participants felt that the examples were helpful, some participants
felt the formatting was over whelming. These participants would have preferred the examples

9
DRB Clearance Number: CBDRB-FY21-ESMD001-004

formatted into a bulleted list. A few participants wanted to list companies that they contracted that
were not a type of specialized software.
Recommendation(s):
•
•
•

Format examples into a bulleted list for better readability and to be consistent with other
questions.
Consider adding the word ‘software’ to the title of the section.
Provide help text for the phrase ‘specialized software.’

Finding #18: Purchased services and suppliers - #5, 5a, 5b
Most participant felt that they could answer this question. All but one participant felt comfortable with
providing vendor names. While most participants felt that the examples were helpful, some participants
felt the formatting was over whelming. These participants would have preferred the examples
formatted into a bulleted list. A few participants wanted to list companies that they contracted that
were not a type of specialized software.
Recommendation(s):
•
•
•

Finding #19: Purchased services and suppliers - #6, 6a, 6b
Most participant felt that they could answer this question. All but one participant felt comfortable with
providing vendor names. While most participants felt that the examples were helpful, some participants
felt the formatting was over whelming. These participants would have preferred the examples
formatted into a bulleted list.
Recommendation(s):
•
•
•

C. Innovation
Finding #20: New or improved services
Nearly all participants could answer this question.
Recommendation(s):

10
DRB Clearance Number: CBDRB-FY21-ESMD001-004

•

Consider adding an instruction at the beginning of the section stating that services and goods
would be asked about in separate questions.

Finding #21: New or improved goods
Nearly all participants could answer this question.
Recommendation(s):
•

No recommendations needed.

11
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Appendix A: Draft Questions, Round 1
A. DESIGN
1. Does this business conduct design activities?
Design activities are structured creative processes that may include:
• Design of the “look” of manufactured products;
• Design for function;
• Design for ease of manufacturing;
• Design for sustainability or resilience;
• Design for quality;
• Design of business processes, i.e. design thinking;
• Service design for how customers experience the delivery of a service; or
• Design of user experience (UX) that addresses interaction with goods, services,
or digital content by users.
o Yes (Go to Q1a)
o No (Go to Q8)
1a. Does this business provide resources for design activities?
o Yes (Go to Q1b)
o No (Go to Q2)
1b. Do those resources have a designated budget?
o Yes
o No

12
DRB Clearance Number: CBDRB-FY21-ESMD001-004

2. Does this business conduct the following design-related engineering activities?
Yes
•
•
•

Prepare technical specifications including computer-aided drafting
Create first-time use of components or the initial tooling of equipment
Construct prototypes that include all the technical and performance
characteristics of a new good or service

2a. Does this business provide resources for design-related engineering activities?
o Yes (Go to Q2b)
o No (Go to Q3)
2b. Do those resources have a designated budget?
o Yes
o No
3. Are design activities performed by…
Yes
• …employees?
• …contractors?

4. What types of employees and/or contractors routinely perform design activities? Select
all that apply.
 Designers
 Key managers (for example in marketing, product or website development)
 Company leadership
 Related occupations in engineering, craft production, programming, research, or
other fields
5. How many employees and/or contractors routinely perform design activities?
2019: __________
2016: __________

6. How are the employees and/or contractors who routinely perform design activities
organized?
 As one or a few working independently
 As a small team
 As multiple teams and/or centralized design unit/department
13
DRB Clearance Number: CBDRB-FY21-ESMD001-004

7. How would you describe this business’s approach to design? Select ALL that apply.
o Design is used to enhance the appearance and attractiveness of the final good or
service
o Design is integrated into the development of new goods or services
o Design is a central directing element of the business model
o Other __________________________________
8. How much do you expect company resources for design activities to change over the
next 3 years?
o Increase substantially
o Increase slightly
o No change
o Decrease slightly
o Decrease substantially
9. Does this business use the following methods to protect intellectual property?
Yes
No
a. Utility patents (applied for or awarded)
b. Design patents (applied for or awarded)
c. Copyright
d. Trademarks
10. Does this business use the following less formal means of protecting intellectual
property?
Yes
No
a. Design complexity
b. Nondisclosure agreements or non-compete clauses
c. Branding and product image

14
DRB Clearance Number: CBDRB-FY21-ESMD001-004

B. TECHNOLOGY
DIGITAL SHARE OF BUSINESS ACTIVITY
1. In 2020, did this business use a digital format for any of the following types of
information? Mark one for each row

Personnel
Financial
Customer feedback
Marketing
Supply chain
Production
Other (specify)

In use








In testing








Not in use








Don’t know








15
DRB Clearance Number: CBDRB-FY21-ESMD001-004

CLOUD SERVICE PURCHASES
2. In 2020, did this business use cloud services for any of the following IT functions? Mark
one for each row
In use
In testing
Not in use
Don’t know
All IT functions




Security or firewall




Servers




Data storage and management (Examples:




Amazon Web Services, IBM Bluemix,
Microsoft Azure)
Collaboration and file synchronization




(Examples: Dropbox, OneDrive, Google
Drive)
Data analysis




Billing and account management




Customer relationship management




Other (specify)





16
DRB Clearance Number: CBDRB-FY21-ESMD001-004

BUSINESS TECHNOLOGIES
3. In 2020, did this business produce goods or provide services by using or applying any of
the following technologies? Mark one for each row
In use
In testing
Not in use Don’t know
Augmented reality




Automated guided vehicles (AGV) or AGV




systems
Machine learning




Machine vision




Natural language processing




Radio-frequency identification (RFID)




system
Robotics




Touchscreens/kiosks for customer
interface (Examples: self check-out, self




check-in, touchscreen ordering)
Voice recognition software




Additive Manufacturing (3d Printing)





17
DRB Clearance Number: CBDRB-FY21-ESMD001-004

PURCHASED SERVICES AND SUPPLIERS
4. In 2020, did your business purchase outreach, advertising, e-commerce, web analytics,
or customer relationship management (CRM) services from an outside vendor?
Examples of such services include specialized software or services that initiate and
manage customer relationships, both (1) through tailored (profiled/personalized)
advertising and (2) through personalized dialogue and profiling, recommender systems,
monitoring or web/social media, etc. Other examples include focus testing, website
analytics and public profile management, as well as dynamically adaptive product
placement and pricing.
o Yes
o No (Go to Q5)
o Don’t know (Go to Q5)
4.A. List the primary vendor who provided this service.
_________________________________________________________________
o Don’t know
4.B. How important was the purchase of services related to outreach, advertising, ecommerce, web analytics, or customer relationship management (CRM) to your business’s
core operations?
o Very important
o Somewhat important
o Not at all important
o Don’t know

18
DRB Clearance Number: CBDRB-FY21-ESMD001-004

5. In 2020, did your business purchase personnel recruiting and monitoring services from
an outside vendor?
Examples of such services include specialized software or software-enabled services
that recruit new staff (through web-based recruiting websites, tailored job advertising,
social media search, etc.), analyze staff characteristics and activity, monitor staff (health,
office performance, happiness and complaints), and detect and analyze staff web
presence/activity.
o Yes
o No (Go to Q6)
o Don’t know (Go to Q6)
5.A. List the primary vendor who provided this service.
_________________________________________________________________
o Don’t know
5.B. How important was the purchase of personnel recruiting and monitoring services to
your business’s core operations?
o Very important
o Somewhat important
o Not at all important
o Don’t know

19
DRB Clearance Number: CBDRB-FY21-ESMD001-004

6. In 2020, did your business purchase enhanced services related to finances and sales
from an outside vendor?
Examples of such services include software or services dedicated to improving supply
chain management, demand forecasting, problem spotting, sales, and delivery
procedures as well as specialized software or services devoted to managing and
forecasting a business’s finances, including daily adaptive forecasting and situation
monitoring, adaptive office finance software, and adaptive tax management software
(including enhanced tax planning).
o Yes
o No (Go to Q7)
o Don’t know (Go to Q7)
6.A. List the primary vendor who provided this service.
_________________________________________________________________
o Don’t know
6.B. How important was the purchase of enhanced services related to finances and sales to
your business’s core operations?
o Very important
o Somewhat important
o Not at all important
o Don’t know

20
DRB Clearance Number: CBDRB-FY21-ESMD001-004

C. INNOVATION
New and Improved Goods
1. During the three years 2018 to 2020, did this business introduce to the market any new
or improved goods that differed significantly from this business’s previous goods? This
includes the addition of new functions or improvements to existing functions or user
utility. Functions include quality, technical specifications, reliability, durability, economic
efficiency during use, affordability, convenience, usability, and user friendliness. User
utility includes attributes such as affordability and financial convenience.
Goods: usually a tangible object such as a smartphone, furniture, or packaged software, but
also includes digital goods such as downloadable software, music and film. Exclude the simple
resale of new goods or changes of a solely aesthetic nature.
o Yes
o No
New and Improved Services
2. During the three years 2018 to 2020, did this business introduce to the market any new
or improved services that differed significantly from this business’s previous services?
This includes the addition of new functions or improvements to existing functions or user
utility. Functions include quality, technical specifications, reliability, durability, economic
efficiency during use, affordability, convenience, usability, and user friendliness. User
utility includes attributes such as affordability and financial convenience.
Services: intangible activities, such as retailing, insurance, educational courses, air travel,
consulting, etc., also includes digital services. Exclude the simple resale of new services.
o Yes
o No

21
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Novelty of New or Improved Goods or Services
3. During the three years 2018 to 2020, did this business introduce any new or improved
goods or services that differed significantly from goods or services previously offered by
this business that were:
Select one for each row.

Yes

New to the market? This business introduced a new or improved good or
service that was not previously offered by any of your competitors (it may
have already been available in other markets).
New only to this business? This business introduced a new or improved
good or service that was identical or very similar to products already
offered by your competitors.

New and Improved Business Processes
4. During the three years 2018 to 2020, did this business introduce any of the following
types of new or improved business processes that differed significantly (e.g., greater
efficacy, resource efficiency, reliability and resilience, affordability, convenience and
usability) from your previous business processes?
Yes

a. Methods for producing goods or providing services (including methods
related to engineering and related technical testing, analysis and
certification)
b. Logistics, delivery or distribution methods
c. Marketing methods for promotion, packaging, pricing, product
placement or after sales services
d. Information and communication systems (including hardware, software
and data processing)
e. Administration and management activities (including decision-making,
human resource management, and methods for accounting or other
administrative operations)
f. Product and business process development activities (including activities
to identify, develop, or adapt products or processes, or adopting new
methodologies for developing products and processes)

22
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Appendix B: Interview Protocol, Round 1

Cognitive Interviewing Guide: Draft Content for 2021 ABS (4 th Year)
Interviewer notes are in red.
Introduction:
• Explain purpose of meeting: to obtain feedback on reporting to the ABS, to understand the
process of answering questions, and to evaluate how new questions work or don’t work.
• Some questions will ask may seem odd and/or obvious, but we don’t want to assume we know
what you are thinking.
• Explain that we are not testing the respondent – we only want to evaluate the questionnaire
• Structure of meeting: Understand your business and your role in it, then review the
questionnaire and ask you questions as you complete it.
• Ask respondent to sign the consent form.
Research Questions to Address:
• Are respondents able to answer the questions as intended?
• Are there any items that cause the respondents problems?
o What can be done to fix those problems?
• Are there any order/context effects?
Before the Questionnaire:
• What types of goods or services does this business provide?
• What is your role in the company? What kind of responsibilities do you have?
• Are you one of the owners of the business?
• Have you completed other surveys from the Census Bureau or other agencies? If so, which
ones?
• What was your role in the process for responding to this questionnaire? (Gather data? Enter
data? Consult with data providers? Etc.)
• How would you describe your company’s approach to design?

23
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Thanks for providing some background information. Now let’s look at the survey questions.
Direct participant to look at draft content.
DESIGN MODULE - ABS (4th Year)
Question 1
• How would you answer this question?
o IF YES: Can you give me examples of the design activities your company conducts?
• How would you describe ‘design activities’ in your own words?
o Is there another term or phrase that you feel would work better?
• Can you think of any other design activities that we do not have listed?
• Would you need to reach out to other people or departments to answer questions about this
companies design activities?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 1a + 1b
• How would you answer this question?
o IF NO: Does anyone in the company conduct design activities?
• What do you consider to be resources?
• Is this something you would have in your records?
Question 2
• How would you answer this question?
o IF YES: Can you give me examples?
• How would you describe ‘design-related engineering activities’ in your own words?
o Is there another term or phrase that you feel would work better?
• Can you think of any other activities that we do not have listed?
• Would you need to reach out to other people or departments to answer questions about this
companies design activities?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 2a + 2b
• How would you answer this question?
o IF NO: Does anyone in the company conduct design activities?
24
DRB Clearance Number: CBDRB-FY21-ESMD001-004

•
•

What do you consider to be resources?
Is this something you would have in your records?

Question 3
• How would you answer this question?
• How would you define employees?
• How would you define contractors?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 4
• How would you answer this question?
• Can you think of any other types of employees that might perform design activities that we do
not have listed here?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 5
• How would you answer this question?
• How would you define the term ‘routine?’
• How easy or difficult is it to come up with a number for 2016? If R is having difficulty
remembering, ask if the number of individuals increased or decreased between 2016 and 2019.
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 6
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 7
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
25
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Question 8
• How would you answer this question?
• What types of resources were you considering when answering this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 9
• How would you answer this question?
• IF YES: Can you give me any examples?
• When answering this question were you considering all of your company’s IP, or just what
pertained to design?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 10
• How would you answer this question?
• IF YES: Can you give me any examples?
• When answering this question were you considering all of your company’s IP, or just what
pertained to design?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
• Having gone through this set of design questions, do you feel that you are still the most
appropriate person to answer these questions?
o If not, who do you feel would be responsible for this section?
• How long do you think it would take you to complete these questions?

TECHNOLOGY MODULE - ABS (4th Year)
BEFORE STARTING THIS MODULE:
• What does digital mean to you?
26
DRB Clearance Number: CBDRB-FY21-ESMD001-004

•
•
•
•
•
•

What does digital mean to you in terms of record keeping?
Does your company keep any records digitally?
o Can you give me examples?
How would you define cloud services?
Does your company use any cloud services?
o IF YES: What types of cloud services do you use?
Does your company purchase digital or technology-related services from other companies?
Does your company use artificial intelligence in any of its business processes?
o IF YES: How would you define artificial intelligence?

Question 1
• How would you answer this question?
o IF R ANSWERS DK: Do you know if any records are kept digitally?
• Can you think of any other types of records that your company maintains digitally?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 2
• How would you answer this question?
o IF NOT OBVIOUS: When answering this question were you thinking about only cloud
services?
• Are these categories clear or unclear?
• Can you think of any other types of records that your company maintains digitally?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 3
• How would you answer this question?
• Are these categories clear or unclear?
• FOR ANY DK: Did you answer ‘don’t know’ because you don’t know if the company is using this
technology or don’t know what this technology is?
• If we asked you to estimate a percentage of how incorporated that technology is into your
company’s production or service, would you be able to?
o How would you estimate that?
o What are you taking into consideration? Example: outcome, effort, resources, process,
etc…
• Can you think of any other types of technologies that your company uses that we don’t have
listed here?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 4
• How would you answer this question?
27
DRB Clearance Number: CBDRB-FY21-ESMD001-004

•
•

•
•
•

o IF YES: Can you give me any examples?
Is the example helpful or not helpful?
4.A: How would you answer this question?
o Note if R seems apprehensive about providing the vendor name.
o IF DK: Is this information you are comfortable with providing?
▪ IF YES: Is this something you would have in your records?
▪ IF NO: Why?
4.B: How would you answer this question?
Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
Is this something you would have in your records?

Question 5
• How would you answer this question?
o IF YES: Can you give me any examples?
• Is the example helpful or not helpful?
• 5.A: How would you answer this question?
o Note if R seems apprehensive about providing the vendor name.
o IF DK: Is this information you are comfortable with providing?
▪ IF YES: Is this something you would have in your records?
▪ IF NO: Why?
• 5.B: How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 6
• How would you answer this question?
o IF YES: Can you give me any examples?
• Is the example helpful or not helpful?
• 6.A: How would you answer this question?
o Note if R seems apprehensive about providing the vendor name.
o IF DK: Is this information you are comfortable with providing?
▪ IF YES: Is this something you would have in your records?
▪ IF NO: Why?
• 6.B: How would you answer this question?
• IF R PURCHASED: Is there an advantage to purchasing these types of financial services?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

•
•

Having gone through this set of technology questions, do you feel that you are still the most
appropriate person to answer these questions?
o If not, who do you feel would be responsible for this section?
How long do you think it would take you to complete these questions?
28
DRB Clearance Number: CBDRB-FY21-ESMD001-004

INNOVATION QUESTIONS - ABS (4th Year)
Skip innovation section if running low on time.
All Questions
• How would you answer this question?
• Is this something you would have in your records?

WRAP UP
• Do you have any other comments of suggestions about anything we have discussed today?
THANK YOU FOR YOUR TIME!

29
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Appendix C: Draft Questions, Round 2
A. DESIGN
The following set of questions ask about the role that design plays in your business. Design
defined broadly is everywhere—decisions in the making of any good or service contribute to
design. We are interested in collecting information on clear design activities defined as part of
structured creative processes.
1. Does this business take a structured approach to conducting design activities?
Design of goods can include:
• The “look” of manufactured products;
• Function;
• Ease of manufacturing;
• Sustainability or resilience;
• Quality; or
• Architectural design.
Design of services can include:
• Business processes, i.e. design thinking;
• Customers experience;
• Delivery of a service; or
• User experience (UX) that addresses interaction with goods, services, or digital content
by users.
o Yes (Go to Q1a)
o No (Go to Q8)
1a. Does this business provide resources for design activities?
o Yes (Go to Q1b)
o No (Go to Q2)
1b. Do those resources have a budget?
o Yes
o No

30
DRB Clearance Number: CBDRB-FY21-ESMD001-004

3. Does this business conduct the following design-related engineering activities?
Yes
•
•
•

2a. Does this business provide resources for design-related engineering activities?
o Yes (Go to Q2b)
o No (Go to Q3)
2b. Do those resources have a budget?
o Yes
o No
11. Are design activities performed by…
Yes
• …employees?
• …contractors?

12. What types of employees and/or contractors routinely perform design activities? Select
all that apply.
 Designers
 Key managers (for example in marketing, product or website development)
 Senior company leadership
 Engineers, craft production, programmers, researcher, or other related positions

13. How many employees and/or contractors routinely perform design activities?
2019: __________
2016: __________

14. How are the employees and/or contractors who routinely perform design activities
organized? Select ALL that apply.
 As one or a few working independently
 As a small team
 As multiple teams and/or centralized design unit/department
31
DRB Clearance Number: CBDRB-FY21-ESMD001-004

15. How would you describe this business’s approach to design? Select ALL that apply.
o Design is used to enhance the appearance and attractiveness of the final good or
service
o Design is integrated into the development of new goods or services
o Design is a central directing element of the business model
o Other __________________________________
16. How much do you expect this business’s resources for design activities to change over
the next 3 years?
o Increase substantially
o Increase slightly
o No change
o Decrease slightly
o Decrease substantially
17. Does this business use the following methods to protect intellectual property?
Yes
No
e. Utility patents (patents for inventions, applied
for or awarded)
f. Design patents (patents for appearance,
applied for or awarded)
g. Copyright
h. Trademarks
18. Does this business use the following less formal means of protecting intellectual
property?
Yes
No
a. Design complexity
b. Nondisclosure agreements or non-compete clauses
c. Branding and product image

32
DRB Clearance Number: CBDRB-FY21-ESMD001-004

B. TECHNOLOGY
DIGITAL SHARE OF BUSINESS ACTIVITY
7. In 2020, did this business use a digital format for any of the following types of
information? Mark one for each row
Collect in digital Collect, but not
format
digitally
Do not collect
Don’t know
Personnel




Financial




Customer feedback




Marketing




Supply chain




Production




Other (specify)





33
DRB Clearance Number: CBDRB-FY21-ESMD001-004

CLOUD SERVICE PURCHASES
8. In 2020, did this business use cloud services for any of the following IT functions? Do not
include software purchases with embedded cloud features (e.g. Microsoft office,
Adobe). Mark one for each row
In use,
In use, NOT
cloud-based cloud-based
Not in use
Don’t know
Security or firewall




Servers




Data storage and management




(Examples: Amazon Web Services, IBM
Bluemix, Microsoft Azure)
Collaboration and file synchronization




(Examples: Dropbox, OneDrive, Google
Drive)
Data analysis




Billing and account management




Customer relationship management




Other (specify)





34
DRB Clearance Number: CBDRB-FY21-ESMD001-004

BUSINESS TECHNOLOGIES
9. In 2020, did this business produce goods or provide services by using or applying any of
the following technologies? Mark one for each row
In use
In testing
Not in use Don’t know
Augmented reality




Automated guided vehicles (AGV) or AGV




systems
Machine learning




Machine vision




Natural language processing




Radio-frequency identification (RFID)




system
Robotics




Touchscreens/kiosks for customer
interface (Examples: self check-out, self




check-in, touchscreen ordering)
Voice recognition software




Additive Manufacturing (3D Printing),




including prototyping

35
DRB Clearance Number: CBDRB-FY21-ESMD001-004

PURCHASED SERVICES AND SUPPLIERS
10. In 2020, did your business purchase outreach, advertising, e-commerce, web analytics,
or customer relationship management (CRM) services from an outside vendor?
Examples of such services include specialized software or services that initiate and
manage customer relationships, both (1) through tailored (profiled/personalized)
advertising and (2) through personalized dialogue and profiling, recommender systems,
monitoring or web/social media, etc. Other examples include focus testing, website
analytics and public profile management, as well as dynamically adaptive product
placement and pricing.
o Yes
o No (Go to Q5)
o Don’t know (Go to Q5)
4.A. List the primary vendor who provided this service.
_________________________________________________________________
o Don’t know
o If applicable, list other vendors (not primary)
__________________________________________________________________________

4.B. How important was the purchase of services related to outreach, advertising, ecommerce, web analytics, or customer relationship management (CRM) to your business’s
core operations?
o Very important
o Somewhat important
o Not at all important
o Don’t know

36
DRB Clearance Number: CBDRB-FY21-ESMD001-004

11. In 2020, did your business purchase personnel recruiting and monitoring services from
an outside vendor?
Examples of such services include specialized software or software-enabled services
that recruit new staff (through web-based recruiting websites, tailored job advertising,
social media search, etc.), analyze staff characteristics and activity, monitor staff (health,
office performance, happiness and complaints), and detect and analyze staff web
presence/activity.
o Yes
o No (Go to Q6)
o Don’t know (Go to Q6)
5.A. List the primary vendor who provided this service.
_________________________________________________________________
o Don’t know
o If applicable, list other vendors (not primary)
__________________________________________________________________________

5.B. How important was the purchase of personnel recruiting and monitoring services to
your business’s core operations?
o Very important
o Somewhat important
o Not at all important
o Don’t know

37
DRB Clearance Number: CBDRB-FY21-ESMD001-004

12. In 2020, did your business purchase enhanced services related to finances and sales
from an outside vendor?
Examples of such services include software or services dedicated to improving supply
chain management, demand forecasting, problem spotting, sales, and delivery
procedures as well as specialized software or services devoted to managing and
forecasting a business’s finances, including daily adaptive forecasting and situation
monitoring, and adaptive office finance software.
o Yes
o No (Go to Q7)
o Don’t know (Go to Q7)
6.A. List the primary vendor who provided this service.
_________________________________________________________________
o Don’t know
o If applicable, list other vendors (not primary)
__________________________________________________________________________

6.B. How important was the purchase of enhanced services related to finances and sales to
your business’s core operations?
o Very important
o Somewhat important
o Not at all important
o Don’t know

38
DRB Clearance Number: CBDRB-FY21-ESMD001-004

C. INNOVATION
New or Improved Services
5. During the three years 2018 to 2020, did this business introduce to the market any new
or improved services that differed significantly from this business’s previous services?
This includes the addition of new functions or improvements to existing functions or user
utility. Functions include quality, technical specifications, reliability, durability, economic
efficiency during use, affordability, convenience, usability, and user friendliness. User
utility includes attributes such as affordability and financial convenience.
Services: intangible activities, such as retailing, insurance, educational courses, air travel,
consulting, etc., also includes digital services. Exclude the simple resale of new services.
o Yes
o No
New or Improved Goods
6. During the three years 2018 to 2020, did this business introduce to the market any new
or improved goods that differed significantly from this business’s previous goods? This
includes the addition of new functions or improvements to existing functions or user
utility. Functions include quality, technical specifications, reliability, durability, economic
efficiency during use, affordability, convenience, usability, and user friendliness. User
utility includes attributes such as affordability and financial convenience.
Goods: usually a tangible object such as a smartphone, furniture, or packaged software, but
also includes digital goods such as downloadable software, music and film. Exclude the simple
resale of new goods or changes of a solely aesthetic nature.
o Yes
o No

39
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Appendix D: Interview Protocol, Round 2
Cognitive Interviewing Guide: Draft Content for 2021 ABS (4 th Year)
Interviewer notes are in red.
Introduction:
• Explain purpose of meeting: to obtain feedback on reporting to the ABS, to understand the
process of answering questions, and to evaluate how new questions work or don’t work.
• Some questions will ask may seem odd and/or obvious, but we don’t want to assume we know
what you are thinking.
• Explain that we are not testing the respondent – we only want to evaluate the questionnaire
• Structure of meeting: Understand your business and your role in it, then review the
questionnaire and ask you questions as you complete it.
• Ask respondent to sign the consent form.
Research Questions to Address:
• Are respondents able to answer the questions as intended?
• Are there any items that cause the respondents problems?
o What can be done to fix those problems?
• Are there any order/context effects?
Before the Questionnaire:
• What types of goods or services does this business provide?
• What is your role in the company? What kind of responsibilities do you have?
• Are you one of the owners of the business?
• Have you completed other surveys from the Census Bureau or other agencies? If so, which
ones?
• What was your role in the process for responding to this questionnaire? (Gather data? Enter
data? Consult with data providers? Etc.)
• How would you describe your company’s approach to design?

•
•
•
•

How would you describe ‘design activities’ in your own words?
o Is there another term or phrase that you feel would work better?
Can you think of any other design activities that we do not have listed?
Would you need to reach out to other people or departments to answer questions about this
companies design activities?
o IF YES: What departments/positions would you reach out to?
Is this something you would have in your records?

Question 1a + 1b
• How would you answer this question?
o IF NO: Does anyone in the company conduct design activities?
• What do you consider to be resources?
• Is this something you would have in your records?

Question 2
• How would you answer this question?
o IF YES: Can you give me examples?
• How would you describe ‘design-related engineering activities’ in your own words?
o Is there another term or phrase that you feel would work better?
• Can you think of any other activities that we do not have listed?
• Would you need to reach out to other people or departments to answer questions about this
companies design activities?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 2a + 2b
• How would you answer this question?
o IF NO: Does anyone in the company conduct design activities?
• What do you consider to be resources?
• Is this something you would have in your records?

41
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Question 6
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 7
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 9
• How would you answer this question?
42
DRB Clearance Number: CBDRB-FY21-ESMD001-004

•
•
•
•

IF YES: Can you give me any examples?
When answering this question were you considering all of your company’s IP, or just what
pertained to design?
Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
Is this something you would have in your records?

TECHNOLOGY MODULE - ABS (4th Year)
BEFORE STARTING THIS MODULE:
• What does digital mean to you?
• What does digital mean to you in terms of record keeping?
• Does your company keep any records digitally?
o Can you give me examples?
• How would you define cloud services?
• Does your company use any cloud services?
o IF YES: What types of cloud services do you use?
• Does your company purchase digital or technology-related services from other companies?
• Does your company use artificial intelligence in any of its business processes?
o IF YES: How would you define artificial intelligence?

•

o IF YES: What departments/positions would you reach out to?
Is this something you would have in your records?

Question 2
• How would you answer this question?
o IF NOT OBVIOUS: When answering this question were you thinking about only cloud
services?
• Are these categories clear or unclear?
• Can you think of any other types of records that your company maintains digitally?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 3
• How would you answer this question?
• Are these categories clear or unclear?
• FOR ANY DK: Did you answer ‘don’t know’ because you don’t know if the company is using this
technology or don’t know what this technology is?
• If we asked you to estimate a percentage of how incorporated that technology is into your
company’s production or service, would you be able to?
o How would you estimate that?
o What are you taking into consideration? Example: outcome, effort, resources, process,
etc…
• Can you think of any other types of technologies that your company uses that we don’t have
listed here?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 4
• How would you answer this question?
o IF YES: Can you give me any examples?
• Is the example helpful or not helpful?
• 4.A: How would you answer this question?
o Note if R seems apprehensive about providing the vendor name.
o IF DK: Is this information you are comfortable with providing?
▪ IF YES: Is this something you would have in your records?
▪ IF NO: Why?
• 4.B: How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
44
DRB Clearance Number: CBDRB-FY21-ESMD001-004

Question 6
• How would you answer this question?
o IF YES: Can you give me any examples?
• Is the example helpful or not helpful?
• 6.A: How would you answer this question?
o Note if R seems apprehensive about providing the vendor name.
o IF DK: Is this information you are comfortable with providing?
▪ IF YES: Is this something you would have in your records?
▪ IF NO: Why?
• 6.B: How would you answer this question?
• IF R PURCHASED: Is there an advantage to purchasing these types of financial services?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

•
•

45
DRB Clearance Number: CBDRB-FY21-ESMD001-004

INNOVATION QUESTIONS - ABS (4th Year)
Skip innovation section if running low on time.
All Questions
• How would you answer this question?
• Is this something you would have in your records?

WRAP UP
• Do you have any other comments of suggestions about anything we have discussed today?
THANK YOU FOR YOUR TIME!

46
DRB Clearance Number: CBDRB-FY21-ESMD001-004

File Type	application/pdf
Author	Kenneth Herrell (CENSUS/ESMD FED)
File Modified	2020-11-30
File Created	2020-11-30