ABS Cognitive Testing Results

Findings and Recommendations for
Cognitive Testing of the 2019 Annual
Business Survey

Contents
Executive Summary....................................................................................................................................... 4
Research Objectives ...................................................................................................................................... 5
Research Methodology ................................................................................................................................. 6
Findings and Recommendations ................................................................................................................... 9
General Findings ........................................................................................................................................... 9
Finding #1: Difficulty with use of multiple time frames within the survey............................................... 9
Finding #2: Inconsistency with reporting instructions .............................................................................. 9
Finding #3: Inconsistency with providing a reference period in each question ....................................... 9
Finding #4: Missing definitions for key terms ......................................................................................... 10
Finding #5: Inconsistency with use of terminology for those working for or paid by the business ....... 10
Finding #6: Question order could be improved ...................................................................................... 10
Finding #7: Question headers/titles sometimes created confusion ....................................................... 10
Item Specific Findings ................................................................................................................................. 11
Company Overview Module ....................................................................................................................... 11
Research & Development Module .............................................................................................................. 13
Technology Module .................................................................................................................................... 16
Appendix A: About the Data Collection Methodology and Research Branch (DCMRB) ................................. 19
Appendix B: Cognitive Interview Protocol – 2019 Annual Business Survey ................................................... 20
Introduction: ............................................................................................................................................... 21
Appendix C: 2019 Annual Business Survey (ABS) Questions ......................................................................... 29
Appendix D: Research and Development Module for 2019 ABS ................................................................... 31
Research and Development Module for 2019 ABS..................................................................................... 31
1. R&D ACTIVITIES ....................................................................................................................................... 31
During 2017, did this business do any of the following R&D activities? MARK ALL THAT APPLY ............... 31
2.

R&D COSTS .......................................................................................................................................... 32

Total costs for ‘R&D activities’ reported in the R&D Activities question for 2017 ..................................... 32
3.

TYPES OF R&D COSTS .......................................................................................................................... 32

4. FUNDING SOURCES FOR R&D ACTIVITIES: ............................................................................................. 33
5. PERCENT PAID FOR R&D CONDUCTED BY OTHERS:................................................................................ 33
6. R&D CATEGORIES: ................................................................................................................................... 33

7. R&D EMPLOYEES ..................................................................................................................................... 34
8. R&D EMPLOYEE OCCUPATIONS .............................................................................................................. 34
a. Researchers (including R&D scientists, engineers, ___________ ___________ ___________ ............. 34
Number of Researchers with PhD (excluding MD, ___________ ___________ ___________ ................. 34
9. BUSINESS SALES ...................................................................................................................................... 35
Appendix E: Technology Module for 2019 ABS............................................................................................. 36

Executive Summary
In September 2018, Krysten Mesner and Struther Van Horn from the U.S. Census Bureau’s Data
Collection and Methodology Research Branch (DCRMB) conducted 18 cognitive interviews on an existing
survey: The Annual Business Survey (ABS). The ABS asks questions about the business owner(s) as well
as various characteristics about the business. The cognitive testing included testing general questions
about company information, research and development (R&D) questions, as well as a new technology
module. The first round of testing showcased areas of strength for the survey, as well as some areas
that can be improved upon. A primary strength of the survey is that many of the respondents
understood the terminology used throughout the survey. Additionally, many participants indicated that
they were able to complete the survey in the allotted amount of time, which is always a concern. The
main area of concern for the survey which will require improvement is defining and providing
consistency with both the terminology used and the directions provided.

Research Objectives
The ABS asks questions about the business owner(s) as well as various characteristics about the
business. The 2019 ABS cognitive testing will include some general questions from section A and
research and development (R&D) questions from Section D of the ABS as well as a new technology
module. The types of questions in the R&D and technology module are listed below:







Intensity of Use asks about the intensity of usage of different technologies during the recall
period (three years including 2016-2018).
Motivation for Technology Adoption and Utilization asks about the reasons behind adoption and
utilization of the technologies.
Impact of Technology on Workforce is designed to understand how each technology helped
reorganize the responding company’s workforce during the three years 2016 to 2018.
Impact of Technology on Worker Types assesses the impact of each technology for different
worker types.
Factors Adversely Affecting Technology Adoption and Utilization asks about how different
factors adversely affected the adoption or utilization of specific technologies from 2016 to 2018.
Research and Development asks about R&D activities. These data are used to compare R&D
costs across industries, determine where R&D activity is conducted geographically, and identify
the types of businesses with R&D.

The objectives of these interviews were to learn the following from participants:





how they understand and interpret the questions that they are being asked to respond to;
if the answer choices that they are provided are appropriate for their respective question;
if there are any questions that cause the respondents difficulty;
if there are any order or context effects that impact respondents’ ability to answer the
questions

Results of this testing will be used to refine the questions as the survey enters the next stage of
development.

Research Methodology
To address the research objectives, 18 interviews were conducted in total, between September 10th and
September 21st, 2018. Potential participants for the interviews were found using a list of respondents
from the 2017 ABS. During the recruitment phase, participants were asked screening questions to
ensure that they meet the criteria for testing the technology module questions on artificial intelligence,
cloud-based computing systems and applications, specialized software, robotics, and specialized
equipment. The persons of interest for these interviews were the respondents who had previously
completed the 2017 ABS, which included business owners, presidents, chief executive officers, chief
financial officers and managing partners.
Potential participants were contacted via email and asked if they would like to help review draft survey
questions and provide feedback on their experience completing the 2017 ABS. Interviews were
conducted in-person. Table 1 summarizes the number of interviews conducted in each metropolitan
statistical area:
Table 1. Number of interviews per metropolitan statistical area (MSA)
MSA
Los Angeles, CA
New York, NY
TOTAL

# of Interviews
9
9
18

Given the different modules being testing on both R & D and technology, efforts were made during the
recruiting process to get participants from a range of different industries. Table 2 summarizes the
number of businesses that participated in each industry sector, based upon classification by the North
American Industry Classification System (NAICS):
Table 2. Business Industries
Industry Sector

# of Businesses

31- Manufacturing
44- Retail/Trade
51- Information
54- Professional, Scientific, and Technical Services
62- Health Care and Social Assistance
TOTAL

1
4
1
11
1
18

Efforts were also made to get variety in the size of businesses recruited. Table 4 provides a summary of
the number of businesses that participated, based upon business size by the number of employees:
Table 3. Size of Businesses
Business Size
1 to 4 Employees
5 to 9 Employees
20 to 49 Employees
50 to 99 Employees
100 to 249 Employees
TOTAL

Count
8
5
2
2
1
18

Efforts in recruitment were made to get a variety of businesses based upon their responses to the 2017
ABS on technology and R&D questions. Table 4 provides a summary of the businesses that participated
based upon their responses in the 2017 ABS to technology and R & D questions. Additionally, this
information also helped determine which modules would be tested with the respondent:
Table 4. 2017 ABS Responses
R&D*
Robotics*
Artificial Intelligence*
Other Technology*

Count
11
7
7
4

*Note, a business could indicate they used multiple technologies (robotics, artificial intelligence, or other) and R & D.

Table 5 summarizes the number of businesses that completed cognitive testing for each module:
Table 5. Modules Tested
Module
2017 ABS Review
Company Information Module
R & D Module*
Technology Module*

Partial Complete Count
0
0
8
3

Complete Count
18
18
9
15

TOTAL
18
18
17
18

*Note, some businesses were only able to complete a module partially, due to either time constraints or applicability of questions. Partial
completes indicate a business was only able to answer some questions in a module, complete count indicates a business answered all questions
applicable in a module.

Cognitive interviews were the testing methodology used for this project. Cognitive interviews are used
to, “(a) understand the thought processes used to answer survey items, and (b) to use this knowledge to
find better ways of constructing, formulating, and asking survey questions” (Forsyth and Lessler, 1991).1
Cognitive interviews traditionally focus on the four steps of Tourangeau’s (1984) cognitive response
model: comprehension, retrieval, judgment, and communication/reporting.2 Comprehension refers to
the participant’s interpretation and understanding of the question’s language, structure, and grammar.
In order to answer the question, a participant must understand what information is being requested on
the survey. Retrieval is the step where relevant information is obtained, either from records or from
memory. The next step, judgment, describes the participant’s evaluation of the completeness or
relevance of the data obtained. It is here that estimates are made based on partial or incomplete data.
The last step, communication or reporting, deals with mapping the response to the answer space
provided and possibly altering the answer.
While Tourangeau’s model is suitable for household and social surveys, the establishment survey setting
presents additional factors that must be considered. First, instead of or in addition to a reliance on
memory, establishment surveys rely heavily on records and the information contained within them.
Second, organizations tend to have distributed knowledge. Some people are experts in one type of
information, while others keep information about something else. Third, competing priorities, both for
the organization and the individual(s) completing the questionnaire, mean that the survey sometimes
does not receive the amount of attention that researchers and data collectors would like. Finally,
organizations regularly authorize only a few individuals to release data. If the data provider is not
authorized to release the data, an additional step must be added to the response process. Tourangeau’s
model was expanded by Sudman et al (2000) to account for these factors. 3

1

Forsyth, B.H. and Lessler, J.T. (1991). “Cognitive Laboratory Methods: A Taxonomy.” In Measurement Errors in Surveys, P.P. Biemer, R.M.
Groves, L.E. Lyberg, N.A. Mathiowitz, S. Sudman (eds). New York: John Wiley & Sons, Inc.
2

Tourangeau, R. (1984). “Cognitive Sciences and Survey Methods.” In Cognitive Aspects of Survey Methodology, T.B. Jabine, M.L. Straf, J.M.
Tanur, and R. Tourangeau (eds). Washington, DC: National Academy Press.
3 Sudman,

S., Willimack, D.K., Nichols, E., and Mesenbourg, T.L. (2000). “Exploratory Research at the U.S. Census Bureau on the Survey Response
Process in Large Companies.” Paper prepared for presentation at the Second International Conference on Establishment Surveys, Buffalo, NY.

Findings and Recommendations
General Findings
Finding #1: Difficulty with use of multiple time frames within the survey
The use of several different reference years and dates throughout the survey made it difficult for
respondents to clearly answer many of the questions. Some questions ask for information from a single
year while others ask for a recall period of three years. While respondents did not have difficulty
answering questions that called for a three-year response period, they did often indicated that they
would be thinking of the current year when answering these questions. Additionally, respondents had
the most difficulty with the questions that asked for information for the specific date of March 12 th.
Every respondent interviewed questioned the usefulness or need for the March 12th date. The use of
several different reference periods creates additional cognitive burden on the respondents as they go
through the survey.
Recommendations:





For ease of burden and consistency, consider using just one reference frame throughout
the survey. The easiest for recall and burden may be the prior year or the current year.
If it is determined that multiple dates should stay, consider not including the March 12th
date, unless there is an administrative/data editing need. More details on this in the
item-specific findings.
Additionally, if multiple reference dates are used throughout the survey, we
recommend insuring that there is a clear indication of the change of the reference
period in the instructional text or question(s).

Finding #2: Inconsistency with reporting instructions
Several questions in the Research and Development module were missing clear reporting instructions on
whether the respondents should be providing dollar amounts or percentages. Questions 4, 5, and 6 used
ambiguous instructions such as “the amount” or “how much was paid”. Respondents indicated that
they would be able to provide either dollar amounts or percentages in cognitive testing.
Recommendation: Ensure that there is consistency in asking respondents to report in either
dollar amounts or percentages. The reporting instructions should be consistent for all applicable
questions (e.g. questions 4 through 6 in the R & D module) and clearly state if dollar amounts of
percentages should be reported.

Finding #3: Inconsistency with providing a reference period in each question
Several questions were missing a reference period, which made the questions difficult to respond to.
Consistent use of a reference period should be used throughout the survey. Respondents indicated that
they would either have to guess at which reference period to report for or use proceeding questions to
help guide them in determining a reference period.
Recommendation: For the following questions, specify a reference period or date in the
question text: question 3 in the Company Information Module and questions 3, 4, 5, and 6 in the
R & D Module.

Finding #4: Missing definitions for key terms
Several of the questions throughout the different modules were missing definitions for key terminology
that would help respondents answer the questions and ensure they were including the correct
information. These terms include, owner, employee, and worker.
Recommendation: provide definitions of the following key terms:





In the Company Information Module, given the way that the questions ask
about ownership, provide a definition of owner with question 2.
In the Company Information Module, R &D Module, and Technology module,
provide a definition of the term employee and worker the first time the term is
used in each module.
Additionally, these terms could also be defined with the overview information
of the survey or individual modules.

Finding #5: Inconsistency with use of terminology for those working for or paid by the
business
In the Business Overview Module, the term employee is used to discuss people who worked for the
particular business. In the R&D Module, the term employee is also used. In the Technology Module, the
terms worker, production worker, non-production worker, supervisory worker, non-supervisory worker,
and skilled worker are used. While most of the participants understood what was meant by the terms
employee and worker, it was difficult for respondents to think of multiple terms for the same concept.
Recommendation:



Use of either the term employee or worker consistently throughout the modules on the
survey.
If both the terms employee and worker are to be used, clearly define the terms when
the terms are first used. This appears to be particularly important given the inclusion or
exclusion of owners as employees/workers. It may be beneficial to provide a clear
include/exclude list for these questions to help clarification for the respondents.

Finding #6: Question order could be improved
We recommend reviewing the order of questions in the survey to see if question order could be
improved upon, to ease cognitive burden on respondents. The placement of the Business Sales question
within the Research and Development Module did not to fit with the rest of the questions in the
module. Additionally, it may be beneficial to review the question ordering within the Technology
Module, as it may be more beneficial to move question 9 up with the other questions asking about
motivation for use of technologies.

Finding #7: Question headers/titles sometimes created confusion
For some of the questions, the question headers or titles created confusion for participants on what the
question was asking and how they should be answering the question. For example, question 4 in the
Company Information Module, has a header/title of “Number of Employees” but then asks about the
number of non-owner employees, which created confusion for respondents.

Recommendation:


Remove question headers/titles so that participants can focus more on the question instead of
the question header when answering questions.

If question headers/titles are to be used, consider reviewing the information to ensure that it matches
with the information being asked for in the question, to reduce confusion and cognitive burden.

Item Specific Findings
Company Overview Module
Number of Owners

Finding #8: Difficulty in separating spouses as owners
Some participants brought up difficulty in separating spouses as owners for this question. A participant
specifically questioned why married partners were to be counted separately while other agencies (e.g.,
SBIR grant system) counts married couples that invest jointly as individuals.

Recommendation: We should consider not separating out spouses necessarily as separate
owners as spouses are often specifically only considered as 1 owner when investing,
particularly for when looking at percentage of owners as women. We need to ensure that
we are measuring the correct concept with respect to married couples that jointly invest.
Number of Paid Owners

Finding #9: The reference period/date is missing from this question
For the Number of Paid Owners, the reference period/date is missing from the question. Some
participants asked what time period this question was asking about.

Recommendation: Specify the reference period (2017) that is being asked about in this
question.
Number of Employees

Finding #10: Participants found it burdensome to report for March 12, 2017 reference
date
The March 12, 2017 date poses difficulty in that there’s extra burden to look up payroll
information for that pay period. Participants didn’t understand why this date as it doesn’t
coincide with year-end or quarterly reports. Most employees stated that they have a year end
1099 report that they could look up and easily reference.

Recommendation: Although the March 12, 2017 date coincides with the IRS payroll
date, we should consider reducing burden on the respondent and changing date to 2017
as opposed to a specific date.

Finding #11: Reference date of March 12, 2017 may not be representative of actual
number of employees
Given this reference date of March 12, 2017, categories A and B may not be representative of
the actual number of employees that they have (several respondents mentioned this wouldn’t
account for seasonal employees such as summer interns).
Recommendation: Consider changing the reference date to 2017.

Finding #12: Difficulty answering for employees who are owners
Respondents found it confusing to have the title of the question “Number of Employees.” The
instructions were unclear when instructing respondents to “Include both full-time and parttime workers as well as yourself.” This was problematic for owners completing the survey.
Recommendation: If the sponsor wants the question to ask about non-owners, the
instructions need to change. The header could be changed to “Number of Non-Owner
Employees.” The instructions should be reworded to say “include yourself only if you
are not an owner.” Or, just remove header for “Non-Owners Number of People.”

Finding #13: Response categories for employee types are not exhaustive to reflect state
regulations
One respondent questioned the legality of having unpaid employees; NYC prohibits unpaid
employees. They are, however, allowed to have volunteers. Several firms did have volunteers
and one specifically mentioned the pay period of March 12 would miss all of their volunteers.

Recommendation: Add the term “volunteer” as some companies have volunteers
instead of “unpaid interns” per state laws. Keep 1099 and Unpaid workers in separate
question. Call them “workers” and not “employees.”
Total Sales and Revenues

Finding #14: Question and instructions unclear for Total Sales and Revenues question
Some participants experienced confusion and burden when responding to this question. One
participant wondered if this included their venture funding.
Recommendation: Provide examples of the types of things to include and exclude.

Finding #15: Reporting for a calendar year instead of a fiscally driven year can create
added burden for a respondent
A participant indicated that they were not easily able to report this because their company is
fiscally driven with an August 31st fiscal year, but Census wants a calendar year which required
them to pull 12 months of revenue.
Recommendation: None

Domestic Sales and Revenues

Finding #16: Question header for Domestic Sales and Revenues question created
confusion
Most considered that they had foreign sales due to selling products outside of the US. It was
not until they looked at example/definition did they change response. It really seems to be an
issue of production vs sales. This question really seems to be looking at production
sales/revenue as opposed to just domestic sales. Almost all respondents initially said yes to
foreign sales until looking at example provided. “Where the operations took place is
confusing.”
Recommendation: Perhaps change title of question or remove question title per the
recommendation for Finding #7. We could define domestic/foreign sales first or
even switch order of questions 5 (Total Sales and Revenue) and 6 (Domestic Sales
and Revenue), could combine questions 5 and 6 into one question, or screen
respondents out of foreign sales/revenue prior to getting questions 5 and 6.

Research & Development Module
Finding #17: No definition is provided for R&D
There is no definition of R&D provided for participants in the content testing.
Recommendation: R&D should be written out and defined at least once in the module.
It is provided in the web survey instrument. It should also be provided on the paper
instrument for participants in future content testing.

R&D Costs

Finding #18: Expand on what should and should not be included here
One participant struggled whether patent costs should be included or not here.
Recommendation: Provide guidance on whether or not to include or not include patent
costs here.

Funding Sources for R&D Activities

Finding #19: The response category “This U.S. Business” is odd
“This U.S. Business” was a little odd and participants had to reread as it seemed strange when
they first read it.
Recommendation: Revise response category a. to “Your U.S. business.”

Finding #20: More guidance on how to answer for investors is missing from R&D funding
sources question
Participant unsure of how to answer for investors as their largest portion of funding comes
from investors.
Recommendation: Provide more guidance for respondents.

Finding #21: Reference period/year missing for some questions
Questions 4-6 (Funding Sources for R&D Activities, Percent Paid for R&D Conducted by Others,
and R&D Categories) are missing the year we’d like respondents to report for.
Recommendation: Provide the year that we would like them to report for in the
directions.

Finding #22: Respondents have a preference to report in dollar amounts instead of
percentages
Participants reported that they could report in either dollar amounts or percentages for
Questions 4-6 (Funding Sources for R&D Activities, Percent Paid for R&D Conducted by Others,
and R&D Categories). They indicated that dollar amounts would be relatively easy to obtain
from their records. Some participants said that they would give a ballpark estimate if asked for
a percentage or if they wanted to give a more exact percentage that it would require an extra
step of calculating from the dollar amount.
Recommendation: Instruct respondents to report in dollar amounts for these 3
questions.

R&D Employees

Finding #23: It is unclear whether owners should be included in the R&D Employees
question
There are no instructions provided on whether owners should be included in the response for
this question.
Recommendation: Specify in the instructions whether owners should be included. Are
we interested in capturing all W-2 and 1099 employees? We should be defining
employee here.

R&D Employee Occupations

Finding #24: Reporting education level for each R&D employee would create burden for
larger companies
Smaller companies had no issues responding to this question. However, larger companies
mentioned that they wouldn’t have the ability or time to ask each employee about their
education level.
Recommendation: None

Finding #25: It is unclear whether owners should be included in the R&D Employee
Occupations question
The question instructions do not specify if owners should be included in the response or not.
Recommendation: Specify in the instructions whether owners should be included. Are
we interested in capturing all W-2 and 1099 employees? We should be defining
employee here.
BUSINESS SALES

Finding #26: Nearly all participants unable to respond to the Business Sales question
This question created significant difficulty for participants for several reasons. The grid as
written is not clear and while most understood the concepts being addressed there was
cognitive burden associated with the layout. None of the respondents we met with would be
able to respond to the question how it’s currently written.

Recommendation: This question should be dropped or reworked completely.

Finding #27: Some participants unable to provide the information since they don’t mark
sales in this way
Most respondents reported that they could not quantitatively answer this question and any
information provided would be a guess.
Recommendation:

Finding #28: Dates in question are inconsistent
The question asks for total sales in 2018 but then asks about 2016-2018, which is confusing
Recommendation: The time period in question should be consistent.

Finding #29: Not clear what question is asking about
One respondent said they thought the question was asking about new versus older products,
which could be responded to, however, the multiple years and aspect of “between products”
and identical/similar products to competitors is unanswerable.
Recommendation:

Finding #30: Question not applicable for companies that don’t have sales
Some participants were especially confused and said that they don’t have sales. They were
unsure of how to answer given this fact.
Recommendation: Consider adding a check box at the top for Not applicable for
companies that don’t have sales.

Technology Module
Finding #31: There is an overlap between specialized software and cloud-based
computing systems and applications
Many respondents had difficulties separating out responses for specialized software and cloudbased computing systems and applications as they can be both.
Recommendation: Provide an instruction for respondents not to include cloud-based
computing systems and applications as specialized software.

Impact of Technology on Workforce (Process and Methods)

Finding #32: Unclear whether the question is asking about hiring employees or current
employees
It’s unclear whether we are looking at hiring vs. current employees impact on skill level.
Recommendation: We should decide what we’re trying to ask about here and reword
the question to make sure that we can capture it.

Impact of Technology on Worker Types (Process and Methods)

Finding #33: There is no definition of “employee” for this question
The definition of employee is missing here. There is an issue of different workers, particularly
production workers and non-production. Respondent would have to read through all the
options to better understand where each type of worker should be included. Do we want W-2
or 1099 or both?
Recommendation: Provide a definition for employee.

Technology Based Goods and Services

Finding #34: Question wording is confusing, particularly the part about
included/embedded
The question wording is confusing as it isn’t clear what is meant by “included/embedded” the
technology given the proceeding qualification.
Recommendation: Simplify question and broaden scope by dropping “or embedded.”
Motivation for Technology Adoption and Utilization (Goods and Services)

Finding #35: Question wording is confusing, particularly the part about
included/embedded
The question wording is confusing as it isn’t clear what is meant by “included/embedded” the
technology given the proceeding qualification.
Recommendation: Simplify question and broaden scope by dropping “or embedded.”

Factors Adversely Affecting Technology Adoption and Utilization

Finding #36: Participants experienced difficulties understanding that question is asking
about reasons for not adapting
Many participants had difficulties understanding that the question is asking about reasons for
not adopting. However, many respondents responded for technologies that they have already
adapted.
Recommendation: Determine what we’re actually trying to capture here.

Finding #37: Some participants struggled with the language of this question
Some respondents struggled with the language – specifically, having to read the question as a
negative and then apply the negative to the categories. One expressed preference for negating
the categories directly (e.g., “Costs” becomes “Too expensive”, “Access to talent” becomes
“Difficulty hiring talent” etc.). Others struggled with the time frame. Others struggled with
how to respond when nothing adversely affected their technology adoption. Some struggled
with how to respond when they hadn’t considered the technology for their business or it was
not applicable to their business during this time frame.
Recommendation: If possible, consider making category headers “negative” to reduce
cognitive burden.

Finding #38: Question categories could be improved for this question
Categories were typically accepted. Many could not uncover the meaning of “Standards and
Accreditation”; some thought they meant generically industry standards or standard business
practices. Others mentioned laws such as HIPAA yet didn’t associate that with reasons for not
using the technology (cloud.) Many mentioned privacy concerns yet not all selected “security”
category for reporting. Many reported that “Access to capital” is the primary reason for
(small) businesses not adopting more technology.
Recommendation: Remove the “Standards and Accreditation” category as it is poorly
understood. Add “Access to capital” category.

Finding #39: Modify “None” category
The “none” category seemed to catch both people who did not have things adversely affecting
their technology and those who did not adopt the technology for a variety of reasons.

Recommendation: Change the “None” category to something along the lines of “No
factors adversely affected adoption” for those that adopted technology. Add something
like “N/A” or “Did not consider this technology for my business” for those that did not
adopt or consider adoption.

Finding #40: Confusion over whether question was referring to their product, not the
process
Both respondents that produced and used technology thought that this question was referring
to their product, not process.
Recommendation: Move question 9 after questions 1-4 as it relates to process not
products.

Appendix A: About the Data Collection Methodology and Research Branch (DCMRB)

The Data Collection Methodology and Research Branch in the Economic Statistical Methods Division
(ESMD) assists economic survey program areas and other governmental agencies with research
associated with the behavioral aspects of survey response and data collection. The mission of DCMRB is
to improve data quality in surveys while reducing survey nonresponse and respondent burden. This
mission is achieved by:


Conducting expert reviews, cognitive pretesting, site visits and usability testing, along
with post-collection evaluation methods, to assess the effectiveness and efficiency of
the data collection instruments and associated materials.
 Conducting early stage scoping interviews to assist with the development of survey
content (concepts, specifications, question wording and instructions, etc.) by getting
early feedback on it from respondents.
 Assisting program areas with the development and use of nonresponse reduction
methods and contact strategies.
 Conducting empirical research to help better understand behavioral aspects of survey
response, with the aim of identifying areas for further improvement as well as
evaluating the effectiveness of qualitative research.
For more information on how DCMRB can assist your economic survey program area or agency,
please contact the branch chief, Amy Anderson Riemer.

Appendix B: Cognitive Interview Protocol – 2019 Annual Business Survey
DRAFT Cognitive Interview Protocol - 2019 Annual Business Survey

Research Questions to Address:
 Are respondents able to answer the questions as intended?
 Are there any items that cause the respondents problems?
o What can be done to fix those problems?
 Are there any order/context effects?
Materials:
 Copies of questionnaire testing draft
 Recorder, batteries
 Consent form
Procedure:
 Go through the Introduction and Before the Questionnaire questions.
 Then provide R with the draft questions (Attachments B, C, and D) and ask R to fill it out.
Once R completes each question, ask the general probes and any item-specific follow-up
probes.
 Note how R navigates the questionnaire and if they appear to be having any difficulty
with the questions. If R appears to have any problems (utterances like “Hmm”, confused
looks, etc.), probe on this.
 Go through probes about Wrap-up Questions

Introduction:






Explain purpose of meeting: to obtain feedback on reporting to the 2017 ABS, to
understand the process of answering questions, and to evaluate how new questions
work or don’t work.
Some questions I will ask may seem odd and/or obvious, but I don’t want to assume I
know what you are thinking.
Explain that we are not testing the respondent – we only want to evaluate the
questionnaire
Structure of meeting: Understand your business and your role in it, then review the
questionnaire and ask you questions as you complete it.
Permission to record discussion for note taking purposes? This study is being conducted
under the authority of Title 13 USC. We plan to use your feedback to improve the
design and layout of the form for future data collections. Only staff involved in this
product design research will have access to the recording. Have R sign consent form.

Before the Questionnaire:
 What types of goods or services does this business provide?
 What is your role in the company? What kind of responsibilities do you have?
 Are you one of the owners of the business?




Have you completed other surveys from the Census Bureau or other agencies? If so,
which ones?
What was your role in the process for responding to this questionnaire? (Gather data?
Enter data? Consult with data providers? Etc.)

Questionnaire test
Now let’s take a look at the draft questions. I would like you to go through and answer the
questions as if you were filling out an actual form. If you don’t have answers to any particular
questions, you can just make a best guess. I will ask you some questions as we go through
them.
These questions are intended for [name of business as listed in recruiting file].
General Probes:
 What does this question mean to you?
 How did you go about answering this question?
 Would you be able to answer this question yourself? (If no) Who would be the
appropriate person to answer it?
 (If R appears to have problem/confusion) What are you thinking about?
 Reflect R’s specific answer: You said ___. Can you tell me more about that?
 If R notices skip instructions: The electronic system would skip automatically, but we will
go over every question to make sure they are all clear.
 How do you feel about the order or flow of the questions being asked in the survey?
Item-Specific Probes:
Company Information
10% or More Ownership
 What do you think this question is asking?
 Are the instructions clear?
Number of Owners
 For respondents that selected No to the 10% or more ownership question, how did you
come up with your response?
 Does the March 12, 2017 date affect your ability to answer the question? If yes, what
are the reasons?
Number of Paid Owners
 What’s the difference between this question and the previous (number of owners)
question?
 Are you easily able to report this?



If response to number of owners and this question is the same, how would you respond?

Number of Employees
 What’s the difference between the previous (number of paid owners) question and this
question?

Total Sales and Revenues
 Are you easily able to report this value for the entire business?
 Do you believe that the question is asking about worldwide sales and revenue (outside
US and domestic) or only domestic sales and revenue?
 Does your company have foreign sales and revenues?
 In what dollar unit did you report?
 How do you feel about being asked to report dollar the amount in thousands?

Domestic Sales and Revenues
 How did you come up with your response?
 In what dollar unit did you report?
 What do you think this question is asking about? Do you think it is asking about the
domestic portion of the value reported in Total Sales and Revenue question?
 If response to Total Sales and Revenues and this question is the same (meaning only
domestic sales), how would you respond?
 How do you interpret the phrase “attributable to or originated from domestic operations”?


Does that help you understand that this question is asking for only domestic sales and revenue?
Please consider the phrase “Include sales and operating revenues to foreign customers,
including foreign subsidiaries,” Does that help you understand that this question is asking for
only domestic sales and revenue?

Research and Development Module
R&D Activities
 Did all of the response categories make sense to you? If not, which categories did you
experience difficulty with and why?
R&D Costs
 How did you come up with your response?
 Were the directions clear on what to include and not to include?
 In what dollar unit did you report?
 If respondent has costs for R&D undertaken outside of the U.S., ask the following
questions:

o Did the respondent include any foreign R&D expenses in this question?
 If yes, can the respondent split out the foreign vs. domestic R&D
expense?
 If no, what is the reason the foreign R&D expense was not included?
o Does your business’ foreign R&D expenses include any of the following:
 Salaries, wages, fringe benefits
 Plant, machinery, and equipment, except that which was capitalized
because it had an alternative future use Materials, supplies, software
 Rent, utilities
 Consultants, contractors
 Depreciation expense from plant, machinery, and equipment that was
capitalized because it had an alternative future use

Types of R&D Costs
 How did you determine your response for each option?
 Do you need to check records?
 Do you prefer providing percentages or dollar amounts?
 Does the request for percentages or dollar amounts impact how the respondent
prepares their response?
 Note if R provides a percentage and R’s numbers don’t add up to 100 percent. If so,
probe as to why.
Funding Sources for R&D Activities
 How did you determine your response for each option?
 Did you need to check records?
 Do you prefer providing percentages or dollar amounts?
 Does the request for percentages or dollar amounts impact how the respondent
prepares their response?
 Note if R provides a percentage and R’s numbers don’t add up to 100 percent. If so,
probe as to why

Percent Paid For R&D Conducted by Others
 How did you determine your response for each option?
 Did you need to check records?
 Do you prefer providing percentages or dollar amounts?
 Does the request for percentages or dollar amounts impact how the respondent
prepares their response?
 Note if R reports a percentage and R’s numbers don’t add up to 100 percent. If so, probe
as to why.
R&D Categories (*to be answered in dollar amounts instead of percentages)







How did you determine your response for each option?
Do you need to check records?
Do you prefer providing percentages or dollar amounts?
Does the request for percentages or dollar amounts impact how the respondent
prepares their response?
In what dollar unit did you report?

BUSINESS SALES
 What do you think this question is asking about?
 How did you come up with your answers?
 Did you experience any confusion when answering this question? If so, what could we
do to improve how we are asking about this item?
 Do you have any suggestions on how we could ask these questions in order to make
them easier for you to answer?
R&D Employees
 What do you think this question is asking about?
 How did you come up with your response?
 Did you experience any confusion when answering this question? If so, what could we
do to improve how we are asking about this item?
 What time period did you have in mind when answering this question?
 Do you have any suggestions on how we could ask these questions in order to make
them easier for you to answer?

R&D Employee Occupations
 What do you think this question is asking about?
 How did you come up with your response?
 Did you experience any confusion when answering this question? If so, what could we
do to improve how we are asking about this item?
 What time period did you have in mind when answering this question?
 Do you have any suggestions on how we could ask these questions in order to make
them easier for you to answer?

Technology Module
Production Technology for Goods and Services
 Can you tell me a little more about how you answered this question?
 What time period were you referencing when answering this question?
 Did the response categories make sense to you?













o In your own words, please define what the following mean to you:
 Testing, but not in production or service
 Low Use
 Moderate Use
 High Use
In your own words, can please define what the following mean to you:
o Artificial Intelligence
o Cloud-based Computing Systems and Applications
o Specialized Software
o Robotics
o Specialized Equipment
(Show them definitions) Now that you see the definitions, can you tell me if there are
any that didn’t match your definition? Why?
What are the types of artificial intelligence (AI) that you use or are familiar with? Are
there examples of AI that are not provided in the definition that you would recommend
adding? Does your company do R&D that applies artificial intelligence to a problem?
Does your company do R&D that is intended to advance the state of artificial
intelligence?
What are the types of specialized software that you use at your company?
In your own words, what is the difference between AI and specialized software?
(If R marked Robotics) Tell me about the robotics that you have at your company.
What does the phrase “carrying out a complex set of actions” mean to you?
What are the types of specialized equipment that you have at your company? Tell me
about them. Are there any examples of specialized equipment that aren’t listed that
you would recommend that we add?
In your own words, what is the difference between robotics and specialized equipment?

Motivation for Technology Adoption and Utilization (Process and Methods)
 What time period were you referencing when answering this question?
 Did the response categories make sense to you?
 Can you tell me a little more about how you answered this question for each of the
categories listed?
 Are there other reasons why you used the technologies in this question other than the
ones listed?

Impact of Technology on Workforce (Process and Methods)
 A. How did you decide on a response?
 B. How did you come up with your answer?
 (If R marked yes for Q3-B) How did the skill level of your workers change?
Impact of Technology on Worker Types (Process and Methods)







A. How did you chose an answer for this question? What do you think is meant by the
term “production workers?”
B. How did you come up with your answer for this question? What types of workers did
you think about when thinking about “non-production workers?”
C. How did you come up with your response?
D. What were you thinking about when answering this question?
E. What do you think is meant by “skilled workers?” How did you come up with your
response?

Technology Based Goods and Services
 How did you decide on a response?
 How did you come up with your answer?
Motivation for Technology Adoption and Utilization (Goods and Services)
 What time period were you referencing when answering this question?
 Did the response categories make sense to you?
 Can you tell me a little more about how you answered this question for each of the
categories listed?
Impact of Technology on Workforce (Goods and Services)
 A. How did you decide on a response?
 B. How did you come up with your answer?
Impact of Technology on Worker Types (Goods and Services)
 A. How did you choose an answer for this question? What do you think is meant by the
term “production workers?”
 B. How did you come up with your answer for this question? What types of workers did
you think about when thinking about “non-production workers?”
 C. How did you come up with your response?
 D. What were you thinking about when answering this question?
 E. What do you think is meant by “skilled workers?” How did you come up with your
response?
Factors Adversely Affecting Technology Adoption and Utilization
 What is this question asking about?
 How did you come up with your answers?
 Probe R about each of the column headers.

Wrap-up
For businesses with a proxy respondent during the interview:






Now that you have seen this survey, would you typically be the one who would be
responsible for answering this survey?
If so, would you involve the owner(s) in answering these questions? Which questions?
How would you go about asking the owner(s) to answer these questions? Would you
provide them with a copy of the question/survey?
Would you have the owner(s) review the completed form before it was submitted?
Would you have any concerns or hesitancy about answering any of the questions about
the owners without their input?

All businesses:


Are there any other comments or suggestions you would like to make about anything
we have gone over today?

Debriefing Questions
 What were your thoughts on completing the 2017 Annual Business Survey?
 About how long did it take you to complete the survey? Were others involved in
completing the survey? If yes, was their time included in your estimate?
 How many owners are there at the company?
o (if more than 1) How did you go about completing the question for the other
owners? Did they review the questions and provide answers? What was your
process for doing this?
 The survey asked questions about Research and Development activities. Did you have
any issues answering questions in this section?
 Was there anything on the survey that stood out to you as difficult to answer or
confusing?
 Do you have any recommendations for improving the survey?
Thank the respondent for their time.

Appendix C: 2019 Annual Business Survey (ABS) Questions

2019 Annual Business Survey
Company Information

1. BUSINESS - 10% or MORE OWNERSHIP In 2017, did at least one person own 10% or more of
this business? (Do not count parent companies, estates, trusts or other entities.)
 Yes
 No - Select “No” ONLY if no person owned 10% or more of this business.

2. NUMBER OF OWNERS
For the pay period including March 12, 2017, how many people owned this business?
•Do not combine two or more owners to create one owner.
•Count spouses and partners as separate owners.









1 person
2 people
3 people
4 people
5-10 people
11 or more people
Don’t know

3. NUMBER OF PAID OWNERS Of the owners reported in the ‘NUMBER OF OWNERS’
question, how many received a W-2 issued by this business for salary or wages? If none,
report zero.
____________

4. NUMBER OF EMPLOYEES For the pay period including March 12, 2017, how many people
worked for this business, including those paid through grants? Include both full-time and
part-time workers as well as yourself. Count each person only once. If none, report zero.
Non-Owners Number of People
a. Employees who received a W-2 issued by this business for salary or wages
_________
b. Individuals who received payment in other ways
(for example, contractors/ consultants/temporary workers
who received a 1099 or payment from another business)

_________

c. Unpaid individuals who worked for the business
(for example, interns, friends, family members)

_________

5. TOTAL SALES AND REVENUES
What was the amount of this business’s sales and revenues, including grants, during 2017?
Report dollar amount in thousands. If none, report zero.
$Bil. Mil. Thou.
2017 sales, revenues, and grants __________,000
6. DOMESTIC SALES AND REVENUES
How much of the ‘TOTAL SALES AND REVENUES’ in 2017 sales, revenue, and grants was
attributable to or originated from domestic operations? Include sales and operating revenues
to foreign customers, including foreign subsidiaries. For example, a U.S. manufacturing
corporation sells parts to customers around the world, however, because all of its operations
are located inside the United States it reports 100% of its sales in this question. Report dollar
amount in thousands. If none, report zero.
$Bil. Mil. Thou.
__________,000

Appendix D: Research and Development Module for 2019 ABS

Research and Development Module for 2019 ABS

1. R&D ACTIVITIES
During 2017, did this business do any of the following R&D activities? MARK ALL THAT APPLY
Include activities that:
 This business performed
 Others paid this business to do
 This business paid others to do
a. Conducted activities aimed at acquiring new knowledge or understanding
without specific immediate commercial applications or uses
b. Conducted activities aimed at acquiring new knowledge for solving a
specific problem or meeting a specific commercial objective
c. Conducted systematic work, drawing on research and practical
experience and resulting in additional knowledge, which is directed
to producing new products or processes or to improving existing
products or processes
d. Developed and tested goods, services, or processes that were derived
from scientific research or technical findings
e. Developed software that advanced scientific or technological knowledge
f. Produced findings that could be published in academic journals or presented
at scientific conferences
g. Applied scientific or technical knowledge in a way that has never been
done before
h. Created new scientific or technical solutions that can be generalized to
other situations
i. Conducted work to discover previously unknown technological facts, structures,
or relationships
j. Conducted work to extend the understanding of scientific facts, relationships,
or principles in ways that could be useful to others

2.

R&D COSTS

What was the total cost (both direct and indirect) in 2017 for all the R&D activities reported as
“Yes” in the ‘R&D ACTIVITIES’ question? Your best estimate is acceptable.
Include the following costs:
 Salaries, wages, fringe benefits
 Plant, machinery, and equipment, except that which was capitalized because it had an
alternative future use Materials, supplies, software
 Rent, utilities
 Consultants, contractors
 Depreciation expense from plant, machinery, and equipment that was capitalized
because it had an alternative future use
Do not include:
 Costs for routine product testing, quality control, and technical services unless they are
an integral part of an R&D project
 Market research
 Efficiency surveys or management studies
 Literary, artistic, or historical projects, such as films, music, or books and other
publications
 Prospecting or exploration for natural resources
Total costs for ‘R&D activities’ reported in the R&D Activities question for 2017
$___________ 00

3.

TYPES OF R&D COSTS

Of the total R&D amount reported in the ‘R&D COSTS‘ question, what was the amount for the
following types of costs?
a.
b.
c.
d.
e.
f.
g.

Salaries, wages, and fringe benefits
Expensed machinery and equipment (not capitalized)
Materials and supplies
Payments to business partners for collaborative R&D
Purchased R&D services
Depreciation on R&D property and equipment
All other costs (for example, consultants, contractors, travel, rent)

_____
_____
_____
_____
_____
_____
_____

4. FUNDING SOURCES FOR R&D ACTIVITIES:
Of the total R&D amount reported in the ‘R&D COSTS‘ question, how much was paid for by the
following sources?
a.
b.
c.
d.
e.
f.
g.
h.
i.

This U.S. business
Your foreign owner (if the business is foreign owned)
Another U.S. business
Other businesses located outside the U.S.
U.S. university or college
U.S. non-profit organization
U.S. Federal government (including R&D grants)
U.S. State or Local government (not including state universities)
All other organizations outside the U.S.

_____
_____
_____
_____
_____
_____
_____
_____
_____

5. PERCENT PAID FOR R&D CONDUCTED BY OTHERS:
Of the R&D amount this business paid others to perform, what amount went to another U.S.
business, a U.S. university or college, or another source?
a. Another U.S. business
b. U.S. university or college
c. Other (specify)

_____
_____
_____

6. R&D CATEGORIES:
Of the total R&D amount reported in the ‘R&D COSTS‘ question, how much was for the
following categories?
a. Basic Research – activities aimed at acquiring new knowledge or understanding without
specific immediate commercial applications or uses
_____
b. Applied Research – activities aimed at solving a specific problem or meeting a specific
commercial objective
_____
c. Development – systematic work, drawing on research and practical experience and resulting
in additional knowledge, which is directed to producing new products or processes or to
improving existing products or processes
_____

7. R&D EMPLOYEES
For the pay period including March 12, 2017, how many employees from this business’s foreign
and domestic operations, were R&D employees and how many were all other employees?
R&D employees include all employees who work on R&D or who provide direct support to
R&D, such as researchers, R&D managers, technicians, clerical staff, and others assigned to R&D
groups. Exclude employees who provide only indirect support to R&D, such as corporate
personnel, security guards, and cafeteria workers.
(1)Domestic (2)Foreign
(3)Total
Operations Operations
Employees
a. Female R&D employees
b. Male R&D employees
Total employees
c. All other employees
d. Total employees

___________ ___________ ___________
___________ ___________ ___________
___________ ___________ ___________
___________ ___________ ___________
___________ ___________ ___________

8. R&D EMPLOYEE OCCUPATIONS
For the pay period including March 12, 2017, how many of the total R&D Employees reported
in the ‘R&D EMPLOYEES’ question worked in the occupations listed below?
(1)Domestic (2)Foreign
(3)Total
Operations Operations Employees
a. Researchers (including R&D scientists, engineers, ___________ ___________ ___________
and their managers)
Number of Researchers with PhD (excluding MD, ___________ ___________ ___________
JD, and EdD)
b. R&D technicians and equivalent staff
___________ ___________ ___________
c. R&D support staff (clerical and other)
d. Total R&D employees

___________ ___________ ___________
___________ ___________ ___________

9. BUSINESS SALES
Please estimate the percentage of your business's total sales in 2018 from products (goods and
services) that were, in the three years 2016 to 2018
New or Improved
Products

--%
If possible, separate sales from
new or improved products
between products
Not previously offered by any
of your competitors

Unchanged Products (or with
only minor changes)*

+

=
--%
+

Identical or very similar to
products already offered by
your competitors
* Includes the resale of new products purchased from other enterprises.

Total Sales in
2018

=

100%

Appendix E: Technology Module for 2019 ABS
TECHNOLOGY MODULE FOR 2019 ABS
(To be sent to all businesses and industries)

We provide both a short definition and detailed definition for each technology -- Short definition:
Definition that appears near the question, and Detail definition: Definition that appears in pop-up
or in detailed instructions.
Artificial Intelligence:

Short definition: Artificial intelligence is a branch of computer science and engineering
devoted to making machines intelligent. Intelligence is that quality that enables an entity
to perceive, analyze, determine response and act appropriately in its environment.

Detail definition: Artificial intelligence is a branch of computer science and engineering
devoted to making machines intelligent. Intelligence is that quality that enables an entity
to perceive, analyze, determine response and act appropriately in its environment.
Systems with artificial intelligence perform functions including, but not limited to,
speech recognition, machine vision, or machine learning:


Speech recognition transforms human speech into a format useful for computer
applications (for example, a digital assistant)



Machine vision uses sensors and software that allow images to be used as an input for
computer applications (for example, systems that sort or inspect objects or support
navigation in mobile equipment)



Machine learning uses statistical software and data to “learn” and make better
predictions without reprogramming (for example, recommender systems for websites,
or sales and demand forecasting)

Artificial Intelligence technologies also include virtual agents, deep learning platforms,
decision management systems, biometrics, text analytics, and natural language generation
and processing.
Cloud-based Computing Systems and Applications
Short definition: Cloud systems and applications are computing resources available ondemand via the internet.
Detail definition: Cloud systems and applications are computing resources available ondemand via the internet. Cloud computing enables ubiquitous, convenient, on-demand
internet access to a shared pool of configurable computing resources (e.g., networks,

servers, storage, applications, and services) that can be rapidly provisioned and released
with minimal management effort or service provider interaction.
Specialized Software (excluding Artificial Intelligence):
Short definition: Specialized software is software dedicated to performing a particular
business function.
Detail definition: Specialized software is custom or packaged software dedicated to
performing a particular business function. Specialized software includes, but is not
limited to, software applications for accounting, sales, marketing, customer service and
billing, logistics, health care delivery, telemedicine, computer-aided design (CAD),
computer-aided engineering (CAE), or inventory management. Specialized software
excludes general purpose software such as word processing or spreadsheets. Exclude
Artificial Intelligence software reported above.
Robotics:
Short definition: Robotic equipment (or robots) are automatically controlled,
reprogrammable, and multipurpose machines used in automated operations in industrial
and service environments.
Detail definition: Robotic equipment (or robots) are automatically controlled,
reprogrammable, and multipurpose machines used in automated operations in industrial
and service environments. Robots may be mobile, incorporated into stand-alone stations,
or integrated into a production line. A robot may be part of a manufacturing cell or
incorporated into another piece of equipment.
Industrial robots may perform operations such as: palletizing, pick and place, machine
tending, material handling, dispensing, welding, packing/repacking, and cleanroom.
Service robots are commonly used in businesses for such operations as cleaning, delivery,
construction, inspection, and medical services such as dispensing or surgery.
Specialized Equipment (excluding Robotics):
Short definition: Specialized equipment is equipment capable of automatically carrying
out pre-specified task(s).
Detail definition: Specialized equipment refers to equipment capable of automatically
carrying out pre-specified task(s). Specialized equipment includes, but is not limited to,
computer numerically controlled (CNC) machinery, computer-aided manufacturing
(CAM) systems, manufacturing cells, materials working lasers, automated guided
vehicles systems, automated storage and retrieval systems, and automated materials
handling systems. Exclude robotics equipment reported above.

Q1. Production Technology for Goods and Services
During the three years 2016 to 2018, to what extent did this business use the following
technologies in producing goods or providing services?
Mark one for each row.

Did not
use

Testing, but not
using in
production or
service

Low Use

Moderate
Use

High
Use

Artificial Intelligence
Cloud-based Computing
Systems and Applications
Specialized Software
Robotics
Specialized Equipment
SKIP PATTERN
If all answers to Q1 are “Did not use”, “Testing”, or “Don’t know” then proceed to Q5.
As before, if a respondent answers that a technology was used (low, moderate, or high), Q2-Q4
follow for each technology used.

Don’t
know

Q2: Motivation for Technology Adoption and Utilization (Processes and Methods)
For each of the technology categories, please indicate whether any of the following was a reason
for technology adoption and utilization during the three years 2016 to 2018.
Check all that apply for each row.

Automate
tasks
performed
by labor
Artificial Intelligence
Cloud-based Computing
Systems and Applications
Specialized Software
Robotics
Specialized Equipment

Upgrade
outdated
processes or
methods

Improve
quality or
reliability
of
processes
or methods

Expand the
range of
goods or
services

Adopt
standards and
accreditation

Did not
use
Other

Q3: Impact of Technology on Workforce (Processes and Methods)
For {each of the above technology categories checked}, please indicate whether this
technology helped reorganize your workforce during the three years 2016 to 2018.

A. Did the technology allow your business to change its number of workers?
Mark one.
● Yes, we increased the number of workers
● Yes, we decreased the number of workers
● No, we did not change the number of workers

B. Did the technology impact the skill level of workers?
Mark one.
● Yes, worker skills increased overall
● Yes, worker skills decreased overall
● No, worker skills did not change overall

Q4: Impact of Technology on Worker Types (Processes and Methods)
For {each of the above technology categories checked}, please indicate whether this
technology had an impact on the types of workers during the three years 2016 to 2018.
A. Did the technology impact the number of production workers employed by your business?
Mark one.
● We increased the number of production workers
● We decreased the number of production workers
● We did not change the number of production workers
● Not applicable, we did not employ production workers
B. Did the technology impact the number of non-production workers employed by your
business?
Mark one.
● We increased the number of non-production workers
● We decreased the number of non-production workers
● We did not change the number of non-production workers
● Not applicable, we did not employ non-production workers
C. Did the technology impact the number of supervisory workers employed by your business?
Mark one.
● We increased the number of supervisory workers
● We decreased the number of supervisory workers
● We did not change the number of supervisory workers
● Not applicable, we did not employ supervisory workers
D. Did the technology impact the number of non-supervisory workers employed by your
business?
Mark one.
● We increased the number of non-supervisory workers
● We decreased the number of non-supervisory workers
● We did not change the number of non-supervisory workers
● Not applicable, we did not employ non-supervisory workers
E. Did the technology impact the types of skilled workers employed by your business?
Mark one.
● Yes, worker scientific, technological, engineering and mathematical skills increased overall
● Yes, worker scientific, technological, engineering and mathematical skills decreased overall
● No, worker scientific, technological, engineering and mathematical skills did not change
overall
● Not applicable, we did not employ workers with scientific, technological, engineering and
mathematical skills

Q5. Technology-Based Goods and Services
During the three years 2016 to 2018, did this business sell the technology, or goods or services
that included or embedded the technology?
Mark one for each row.

Yes

No

Don’t
know

Artificial Intelligence
Cloud-based Computing Systems and Applications
Specialized Software
Robotics
Specialized Equipment
SKIP PATTERN
If all answers to Q5 are “No” or “Don’t know” then proceed to Q9.
If a respondent answers that a good or service was produced that included/embedded the
technology, Q6 – Q8 follow for each embedded technology.

Q6: Motivation for Technology Adoption and Utilization (Goods and Services)
For each of the technology categories, please indicate whether any of the following was a reason
for your business to produce the technology, or produce goods or services that included or
embedded the technology, during the three years 2016 to 2018.
Check all that apply for each row.

Expand
Upgrade the range
goods or of goods
services or services
Artificial Intelligence
Cloud-based Computing
Systems and Applications
Specialized Software
Robotics
Specialized Equipment

Enter new
markets or
adapt existing
products to
new markets

Increase or
maintain
market
share

Adopt
standards and
accreditation

Other

Did not
produce

Q7: Impact of Technology on Workforce (Goods and Services)
For {each of the above technology categories checked}, please indicate whether producing this
technology, or goods or services that included or embedded this technology, led you to
reorganize your workforce during the three years 2016 to 2018.
A. Did producing the technology or goods or services that included or embedded this
technology lead your business to change its number of workers?
Mark one.
● Yes, we increased the number of workers
● Yes, we decreased the number of workers
● No, we did not change the number of workers

B. Did producing the technology or goods or services that included or embedded this
technology impact the skill level of workers?
Mark one.
● Yes, worker skills increased overall
● Yes, worker skills decreased overall
● No, worker skills did not change overall

Q8: Impact of Technology on Worker Types (Goods and Services)
For {each of the above technology categories checked}, please indicate whether producing this
technology, or goods or services that included or embedded this technology, had an impact on
the types of workers during the three years 2016 to 2018.
A. Did the technology impact the number of production workers employed by your business?
Mark one.
● We increased the number of production workers
● We decreased the number of production workers
● We did not change the number of production workers
● Not applicable, we did not employ production workers
B. Did the technology impact the number of non-production workers employed by your
business?
Mark one.
● We increased the number of non-production workers
● We decreased the number of non-production workers
● We did not change the number of non-production workers
● Not applicable, we did not employ non-production workers
C. Did the technology impact the number of supervisory workers employed by your business?
Mark one.
● We increased the number of supervisory workers
● We decreased the number of supervisory workers
● We did not change the number of supervisory workers
● Not applicable, we did not employ supervisory workers
D. Did the technology impact the number of non-supervisory workers employed by your
business?
Mark one.
● We increased the number of non-supervisory workers
● We decreased the number of non-supervisory workers
● We did not change the number of non-supervisory workers
● Not applicable, we did not employ non-supervisory workers
E. Did the technology impact the types of skilled workers employed by your business?
Mark one.
● Yes, worker scientific, technological, engineering and mathematical skills increased overall
● Yes, worker scientific, technological, engineering and mathematical skills decreased overall
● No, worker scientific, technological, engineering and mathematical skills did not change
overall
● Not applicable, we did not employ workers with scientific, technological, engineering and
mathematical skills

Q9: Factors Adversely Affecting Technology Adoption and Utilization
For each technology, please indicate whether any of the following factors adversely affected the
adoption or utilization of the technology during the three years 2016 to 2018.
Check all that apply for each row.

Cost of
technology

Artificial
Intelligence
Cloud-based
Computing
Systems and
Applications
Specialized
Software
Robotics
Specialized
Equipment

Maturity of
technology

Access and
reliability
of required
data

Access to
required
human
capital and
talent

Laws and
regulations

Concerns
regarding
safety and
security
(physical
and IP)

None

Other

Appendix (Additional Definitions):
Production worker: A worker (up through the line-supervisor level) engaged in
fabricating, processing, assembling, inspecting, receiving, storing, handling, packing,
warehousing, shipping (but not delivering), maintenance, repair, janitorial and guard
services, product development, auxiliary production for the own use of business (e.g. a
power plant), recordkeeping, and other services closely associated with these production
operations in the business covered by the report. Employees above the workingsupervisor level are excluded.
Non-production worker: A worker engaged in the following activities: factory
supervision above the working foreman level, sales (including driver-salesman), sales
delivery (highway truck drivers and their helpers), advertising, credit collection,
installation and servicing of own products, clerical and routine office functions,
executive, purchasing, financing, legal, professional, and technical. Also included are
employees on the payroll of the business engaged in the construction of major additions
or alterations to the plant who are utilized as a separate work force.
Supervisory worker: A worker whose major responsibility is to supervise, plan, or direct
the work of others, such as top executive and managerial positions, officers of
corporations, department heads, and superintendents.
Non-supervisory worker: A worker who does not supervise, plan, or direct the work of
others.