Download:
pdf |
pdfATTACHMENT H
Department of Commerce
United States Census Bureau
Annual Survey of Manufactures (ASM)
OMB Control No. 0607-0449
Robotics Use
Letter of Support from Weatherhead School of Management Case Western Reserve University
Letter of Support from Robotic Industries Association (RIA)
Letter of Support from Manufacturing Institute (MI)
Development of Survey Questions on Robotics Expenditures and Use in the U.S. Manufacturing
Establishments paper from the Center for Economic Studies (CES)
900 Victors Way, Suite 140 ▪ Ann Arbor, Michigan 48108, USA
Telephone: +1 734-994-6088 Fax: +1 734-994-3338
February 15, 2017
Dr. Ron Jarmin
Associate Director
Economic Directorate
U.S. Census Bureau
4700 Silver Hill Road Washington, D.C. 20233
Dear Dr. Jarmin,
The Robotic Industries Association (RIA) is pleased to provide this letter of support for the
Census Bureau's collection of basic robotic use and expenditure data as part of the Annual
Survey of Manufacturers (ASM) and the Economic Census.
Robotics has already changed the world, but more fundamental change is clearly ahead. It is
much easier to see the outline of the eventual new world than to know how soon it will arrive.
We hear a lot these days about things like smart cities, smart mining, and smart farming. Let’s
remember that this all due to smart people. In the robotics industry, we take a perspective that
goes beyond technology for technology’s sake. We strive to understand the impact of our work
on people’s lives, and to make the world better instead of worse.
In order to gain this understanding, we need to have good data. While the RIA and its
international affiliate, the International Federation of Robotics (IFR), collect statistics from
robot manufacturers on the sales of robots into key industries, geographies, and applications, it
is also important to collect information from their customers. Currently, this is an area with
little visibility. The IFR estimates that some 230,000 robots are installed in the U.S. today, and
that number will continue growing. The U.S. Census Bureau’s implementation of the proposed
questions would give us a new level of detail to analyze which type of firms are adopting
robots, which sub-sectors they do business in, and the impact robots are having on
employment.
Several of RIA’s 435 member companies are examples of how when companies improve their
competitiveness through the implementation of advanced robotics, they are saving jobs and
creating ripples of positive change and economic impact in their workplace and communities.
RIA’s parent organization, the Association for Advancing Automation (A3), has created a video
series called “Why I Automate,” which is dedicated to showcasing these companies’ stories and
proliferating their message. With the addition of basic robotic use and expenditure questions to
900 Victors Way, Suite 140 ▪ Ann Arbor, Michigan 48108, USA
Telephone: +1 734-994-6088 Fax: +1 734-994-3338
the ASM and the Economic Census, we could dig deeper into which industries are benefiting
from robots the most, and more effectively drive employment growth in those sectors.
A great example of a company that has become more competitive through robotics is RIA
member, Vickers Engineering of New Troy, Michigan. A medium-sized prototype and
production supplier of CNC machining to automotive and other industries, Vickers had trouble
finding and keeping people to do dull and repetitive jobs. They tried robotics and discovered
that this saved the cost of constant hiring and retraining for positions people didn’t want.
Then, because of lower costs, improved productivity and greater product quality, they were
able to win business that they couldn’t win before. As a result, they hired more people than
they had before they started using robotics. Capturing basic data on robotic investments by
companies like Vickers Engineering would help us strengthen this message, which is why RIA
strongly supports the implementation of this proposal.
In summary, RIA supports the inclusion of the basic robot use and expenditure questions in the
next Annual Survey of Manufacturers and Economic Census. With the growing importance of
robotics and automation in our society today, we believe it is the right time to begin collecting
this type of information. Please feel free to contact me at [email protected] or (734) 9946088 if you have any questions.
Sincerely,
Jeff Burnstein
President
Robotic Industries Association (RIA)
Development of Survey Questions on Robotics Expenditures and Use in U.S.
Manufacturing Establishments
by
Catherine Buffington
U.S. Census Bureau
Javier Miranda
U.S. Census Bureau
Robert Seamans
NYU Stern School of Business
CES 18-44
October, 2018
The research program of the Center for Economic Studies (CES) produces a wide range of
economic analyses to improve the statistical programs of the U.S. Census Bureau. Many of these
analyses take the form of CES research papers. The papers have not undergone the review accorded
Census Bureau publications and no endorsement should be inferred. Any opinions and conclusions
expressed herein are those of the author(s) and do not necessarily represent the views of the U.S.
Census Bureau. All results have been reviewed to ensure that no confidential information is
disclosed. Republication in whole or part must be cleared with the authors.
To obtain information about the series, see www.census.gov/ces or contact Christopher Goetz,
Editor, Discussion Papers, U.S. Census Bureau, Center for Economic Studies 5K028B, 4600 Silver
Hill Road, Washington, DC 20233, [email protected]. To subscribe to the series,
please click here.
The Census Bureau's Disclosure Review Board has reviewed this data product for unauthorized
disclosure of confidential information and has approved the disclosure avoidance practices
applied to this release. (DRB-B0001-ADEP-11092018)
Abstract
The U.S. Census Bureau in partnership with a team of external researchers developed a series of
questions on the use of robotics in U.S. manufacturing establishments. The questions include: (1)
capital expenditures for new and used industrial robotic equipment in 2018, (2) number of
industrial robots in operation in 2018, and (3) number of industrial robots purchased in 2018. These
questions are to be included in the 2018 Annual Survey of Manufactures. This paper documents
the background and cognitive testing process used for the development of these questions.
*
*
Any opinions and conclusions expressed herein are those of the authors and do not necessarily represent the views
of the U.S. Census Bureau. All results have been reviewed to ensure that no confidential information is disclosed.
The Census Bureau developed the ASM robotics capital expenditures content in partnership with an external
research team that includes Rob Seamans (NYU), Sue Helper (Case Western Reserve University) and Erik
Brynjolfsson (MIT). A grant from the National Science Foundation (NSF) (NSF grant #1748045) supported the
cognitive testing of the survey content. Robert Seamans also acknowledges support for this project from the Hewlett
Foundation (Hewlett grant #6324). We thank Kristina McElheran and Kristin Stettler for helpful comments and
review of this paper.
1. Introduction
There have recently been dramatic increases in the technical capabilities of artificial
intelligence (AI) and robotics. For example, according to the AI Index, error rates for image
recognition has dropped from 29 percent to less than 3 percent between 2010 and 2017,
surpassing human performance levels. 3 The Electronic Frontier Foundation (EFF) notes
similarly dramatic improvement in the performance of AI with respect to real-time video
games, abstract strategy games (e.g., Chess, Go), video recognition, reading
comprehension, translation, and other categories. 4 These advancements have led both to
excitement about the capability of AI and robotics to boost economic growth and to
concern about the fate of human workers in a world in which computer algorithms can
perform many of the functions that a human can (e.g., Frey and Osborne 2017, Furman
2016).
Recent academic research, using national level data on worldwide robotics shipments,
suggests that robotics may have been responsible for about a tenth of the increase in gross
domestic product (GDP) between 1993 and 2007 (Graetz and Michaels 2015). Since then,
worldwide demand for robotics has nearly tripled between 2010 and 2016 (Furman and
Seamans 2018), and the number and share of robotics-oriented patents have both also
increased (CEA 2016). Thus, robots may now be contributing even more to GDP growth than
in the past.
However, even as these technologies may be contributing to GDP growth at a national
level, we lack an understanding about how and when robotics, AI and other advanced
technologies contribute to firm level productivity, the conditions under which these
technologies complement or substitute for labor, how these technologies affect new firm
formation, and how they shape regional economies. We lack an understanding of these
issues because, to date, there is a lack of firm-level data on the use of robotics and AI (Raj
and Seamans 2018; McElheran 2018). Indeed, a recent National Academies of Science
Report (NAS 2017) calls for more data collection on the effects of automation, including AI
and robots, on the economy.
In an effort to better understand the effects of robotics on the US economy, a team of
Census employees and university researchers worked to develop questions on robotics
capital expenditures by U.S. manufacturing plants. This paper documents the background
and cognitive testing process used for the development of these questions for the 2018
3
AI Index, November 2017; available: https://aiindex.org/2017-report.pdf
See AI Progress Measurement from Electronic Frontier Foundation for more details, available at
https://www.eff.org/ai/metrics.
4
2
Annual Survey of Manufactures. The team consisted of Erik Brynjolfsson (MIT), Catherine
Buffington (Census), Susan Helper (Case Western), Javier Miranda (Census) and Robert
Seamans (NYU). The questions are to be included in the 2018 Annual Survey of
Manufactures. The questions include: (1) capital expenditures for new and used industrial
robotic equipment in 2018, (2) number of industrial robots in operation in 2018, and (3)
number of industrial robots purchased in 2018. These questions were arrived at following
an extensive cognitive testing process, the details of which are described in the sections
that follow.
The paper proceeds as follows. Section 2 covers historic and current data sources for
robotics equipment in the United States. Section 3 discusses the robotics questions, the
cognitive testing process the questions underwent, and outcome of the testing process.
Section 4 concludes.
2. Historic and Current Data Sources for Robotics
2.1.
Historic Data
Beginning in the late 1980s, the Census Bureau conducted a Survey of Manufacturing
Technology (SMT) in collaboration with the Department of Defense. The purpose of the
SMT was to measure the presence, use, and planned use of advanced technologies in the
manufacturing sector. The Survey was in the field in years 1988, 1991 and 1993 but was
discontinued for funding reasons. The Department of Defense used the data to assess the
diffusion of technology. Other Federal agencies used the data to gauge competitiveness of
the U.S. manufacturing sector. The data were also used by the private sector in market
analysis, competitiveness assessments, and planning. The data were used in multiple
academic studies, including Dunne (1994), McGuckin et al (1996), Doms et al (1997), Lewis
(2005) and Luque and Miranda (2000) to address questions related to productivity growth,
skill based technical change, earnings and capital-labor substitution amongst others.
Beginning in 2003 and discontinued in 2015 due to budgetary reasons, the Census
Bureau collected related expenditures data in the Information and Communication
Technology Survey (ICTS), a supplement to the Annual Capital Expenditures Survey (ACES).
The ICTS collected data on non-capitalized and capitalized business spending for
information and communication technology (ICT) equipment and computer software. The
Census Bureau has also collected data on the establishments’ use of computer networks
and electronic commerce (e-commerce) via a supplement to the Annual Survey of
3
Manufactures (ASM).5 The data from the supplement has been used in research examining
the effect of IT-related expenditures on firm level outcomes (McElheran 2015).
2.2.
Current Data
The Census Bureau does not currently collect expenditures data for robotic equipment.
Capital expenditures data are collected on several survey instruments including the Annual
Survey of Manufactures and ACES, making them candidates for this type of collection.
For academic, practitioner, and policy purposes, current data on the use of robotics are
derived from two sources of data: the International Federation of Robotics (IFR) Robot
Shipment Data and the European Manufacturing Survey (EMS). The IFR has been recording
information regarding worldwide robot stock and shipment figures since 1993. The IFR
collects these data from its members, who are typically large robot manufacturers such as
FANUC, KUKA, and Yaskawa. The data are broken up by country, year, industry and
technological application. The IFR defines an industrial robot as an “automatically
controlled, reprogrammable, multipurpose manipulator, programmable in three or more
axes, which can be either fixed in place or mobile for use in industrial automation
applications.”6 Geographical information in the IFR is often aggregated (e.g., data exist for
the United States, but not an individual state or region within the United States). The IFR
utilizes its own industry classifications when organizing the data, rather than relying on
broadly used identifiers such as the North American Industry Classification System (NAICS).
Mapping IFR data to other datasets (such as BLS or Census data) first requires crossreferencing IFR classifications to other identifiers. While the IFR data are useful for some
purposes, particularly examining the adoption of robotics by industry and country, the
aggregated nature of the data obscures differences occurring within industries and across
regions, making it difficult to uncover when and how robots might serve as substitutes or
complements to labor, their impact on productivity and competitiveness and obscuring the
differential effects of adoption within industries or countries.
The European Manufacturing Survey (EMS) has been organized and executed
periodically by a number of research organizations and universities across Europe since
2001, and is currently one of the only firm-level datasets examining the adoption of
robotics. The overall objective of the EMS is to provide empirical evidence regarding the use
and impact of technological innovation in manufacturing at the firm level. The EMS
accomplishes this via a survey of a random sample of manufacturing firms wi th at least
twenty employees across seven European countries (Austria, France, Germany, Spain,
5
6
https://www.census.gov/content/dam/Census/library/publications/2002/econ/1999-e-stats-mcd/initial-report.pdf
https://ifr.org/standardisation
4
Sweden, Switzerland and the Netherlands). While some aspects of the survey vary across
countries, the core set of questions inquires about whether the firm uses robots, the
intensity of robot usage, and reinvestment in new robot technology. 7 Data currently exists
for five survey rounds: 2001-2002, 2003-2004, 2006-2007, 2009-2010 and 2012-2013, and
have been used in reports created by the European Commission to analyze the use of
robotics and its impact on labor patterns, including wages, productivity and offshoring.
As of now, the EMS appears to be one of the few data sources that are capturing the use
of robots and automation at the firm-level. This provides opportunities to analyze microeffects of robotics technology on firm productivity and labor, and to analyze firm decision making following adoption. However, the survey is performed at the firm rather than
establishment level, and the sample size of 3,000 is quite small. In contrast, the Census’
Annual Survey of Manufactures (ASM) surveys 50,000 establishments annually and 300,000
every five years.
Raj and Seamans (2018) document how data from the IFR and EMS have been used by
researchers to study the effects of robots on productivity growth and employment. The
authors highlight a number of challenges with the data. Notably, the EMS data does not
cover U.S. manufacturing establishments and the IFR data, while covering the U.S., are
aggregated to the industry level, making it impossible to study how robots are affecting
firms and regions.
2.3.
The Annual Survey of Manufactures and Robotics Data Collection
The Annual Survey of Manufactures (ASM) has many characteristics that make it a good
candidate for the collection of capital expenditures data for robotics. The ASM contains a
large representative sample of U.S. manufacturing establishments, a significant share of
which continue across the survey’s sample rotation. The ASM and the Census of
Manufactures (CM) collect detailed measures of establishments’ (versus firms’) inputs and
outputs at the location of production which allows for measuring differences in geographic
variation in production, differences in product mix within large companies, and important
variation within as well as between firms. These, combined with the availability of historic
microdata, make possible studies of the effect of robotics at manufacturing plants and
associated labor outcomes in a way that is not possible on other survey platforms.
The ASM samples and surveys about 50,000 establishments annually from the universe
of establishments with at least one employee that are active and classified in the
manufacturing sector. In years ending in 2 or 7, ASM data are collected as part of the
The EMS defines industrial robots using the ISO definition “An industrial robot is officially defined by ISO
(Standard 8373:1994) as an automatically controlled, reprogrammable, multipurpose manipulator programmable in
three or more axes.” See: http://www.1aufbau.de/isi-wAssets/docs/i/de/publikationen/ems1e.pdf
7
5
Census of Manufactures, which in 2012 included about 290,000 active employer
manufacturing establishments (U.S. Census Bureau 2018a). The ASM microdata are
available back to 1973 for approved research projects in the Federal Statistical Research
Data Centers (FSRDC). The ASM samples using a probability measure proportionate to size,
with establishments meeting certain criteria (e.g., size as measured by value of shipments)
being included in the sample with certainty (U.S. Census Bureau 2018b). These certainty
cases, numbering around 15,600 in 2014, generate a large de facto panel that typically
continues across the five-year sample rotation as many of these establishments continue to
surpass the size threshold.
The ASM currently collects information on capital expenditures for new and used
depreciable assets for the reporting period and the year prior to the reporting period (see
Figure 1). Assets are broken down into new and used buildings and other structures as well
as new and used machinery and equipment. Machinery and equipment are further broken
down into vehicles intended for highway use, computer and other peripheral processing
equipment, and a residual ‘other’ category. The CM includes the same capital expenditures
data items as the ASM but also collects beginning- and end-of-year asset measures and the
gross value of all sold, retired, destroyed, etc. assets, allowing for the construction of
establishment-level annual capital stocks (see Figure 2). Importantly, approximately 75% of
the assets reported by establishment in 2016 fall in the ‘other’ category.
3. Content and the Cognitive Testing Process and Outcome
3.1.
Background
In April 2017, the Census Bureau received a proposal to add robotics questions to the
ASM (see Miranda and Seamans 2017). The proposal included questions that ask
establishments to report their expenditures on robotic arms or other robotic equipment, as
well as expenditures used for the integration of robotics into specific applications such as
assembly and loading or unloading of parts. The proposal suggested these be added as
additional categories in the breakdown of capital expenditures on the ASM.
In addition to the proposal, letters of support were obtained by Seamans and Miranda
from the Robotics Industries Association (RIA) and the National Association of
Manufacturers’ Manufacturing Institute (NAM). The letters express that, in the face of
declining costs of robotics and expectations of dramatic increases in the use of robotics in
U.S. manufacturing, the collection of robotics expenditures data by establishments and
firms is necessary in order to better understand the impact of robotics on U.S. businesses
and workers. Current data collected from the producers of robotic equipment by the RIA
and its international affiliate, the International Federation of Robotics (IFR), are important,
6
but there are no equivalent data collections from the users of robotic equipment; this
proposed collection would fill this data gap.
3.2.
Content Review
When considering survey content proposals, the Census Bureau must ensure the
proposed content is appropriate with respect to the Census Bureau’s mission and position
within the larger Federal Statistical System; that the content is consistent within the survey
instrument on which it would appear; and that the content is optimal when weighing the
benefit of the collection against the burden placed on reporting businesses. Content
proposals undergo internal review to ensure appropriateness and consistency within the
instrument, as well as the benefit of the collection. The process of cognitive testing is used
to ensure that the questions are clear, understandable, and answerable, and to estimate
the reporting burden that the proposed content imposes.
The Survey Director for and other staff who work on the ASM reviewed the proposal to
ensure the appropriateness to the survey instrument and to determine the potential
location of the proposed content. It was determined that, upon successful testing, the
proposed content would be added as a “Special Inquiry” at the end of the ASM. Inclusion
of the robotics content within the Capital Expenditures section of the survey is not possible
at this time given the experimental nature of the collection and the constraints of the
production schedule of the ASM.
The initial content proposal was reviewed internally by subject matter experts working
in technology and capital expenditures measurement as well as former staff who had
worked extensively with the SMT. Subject matter experts also reviewed external data
sources and ensured that no other private entity nor statistical agency was collecting this
information. Changes based on internal review were incorporated into the survey
instrument, including language referring to other one-time costs associated with the
equipment. Multiple definitions of robotic equipment were developed for review and
testing. Reviewers mentioned the need for expenditures data along with a corresponding
stock measure of capital, leading to the addition of an asset question along with the
proposed expenditures question. Reviewers also stated that knowing the value of robotic
equipment might be difficult for respondents, and thus proposed asking the respondent to
estimate how many robots are used at the establishment and their average price in order to
allow for the estimation of the gross value of robotic equipment at the plant. The draft
content resulting from this internal review process was the basis for the first round of
cognitive testing (see Figures 3a and 3b).
7
3.3.
Cognitive Testing
3.3.1. Overview
Census Bureau Statistical Quality Standard A2 requires that all new survey content
undergo cognitive testing prior to inclusion on a Census Bureau survey instrument. The
result of this cognitive testing should be an understanding of the quality of the proposed
data collection as well as the burden imposed on the respondent. Cognitive testing for
business survey instruments is generally comprised of two stages. In the first stage, often
referred to as the exploratory round of testing, interviews are conducted with potential
survey respondents to examine whether the respondent understands the question,
whether the records kept at the business support the data collection, and whether the
measurement concepts embodied in the question reflect the economic activity and/or
record keeping practices of the business. The first round of testing for the ASM robotics
capital expenditures questions included not only a cognitive portion but also an early stage
scoping portion that included probes designed to learn about the use of robotics at the
company, record keeping for the robotics acquisition, use and maintenance of robotics at
the company, and language or terminology used by the respondent when discussing
robotics. After the first round of testing, proposed survey content is revised based on
cognitive, record keeping, or other considerations uncovered during the exploratory phase.
This revised content is the basis for a second stage of cognitive testing, often referred to as
the confirmatory round. In the confirmatory round, changes made to the instrument based
on the first round of cognitive testing are (in-)validated, typically resulting in the final draft
content.
Cognitive testing interviews are scheduled and conducted by a staff member from the
Data Collection Methodology and Research Branch within the Economic Statistical Methods
Division of the Census Bureau. Subject matter experts may attend as observers and are
available if subject matter questions arise from the respondent. Each round of cognitive
testing typically includes about twenty respondents, ten in each of two locations.
Generally, distinct locations are selected in order to generate variability in both geography
and industrial mix. Phone interviews may be used to supplement in-person interviews and
are useful when a willing participant is unavailable duri ng the scheduled testing period or
when additional diversity of geography or industry is required in the face of budget or time
constraints. Materials used in cognitive testing are submitted to and approved in advance
by the Office of Management and Budget (OMB) as is required under the Paperwork
Reduction Act. The number of interviews and the respondent burden, as measured by time
spent recruiting and interviewing, is estimated and reported in the materials submitted to
OMB. Cognitive interviews each last approximately one hour and are confidential under the
8
same law (Title 13 U.S.C.) that governs the Census Bureau’s collection of information from
businesses.
3.3.2. Selection of cases for cognitive testing
The set of establishments in scope for cognitive testing included all active employers
classified as manufacturers in the 2016 ASM. Establishments were also required to have
reported complete contact information including respondent name, business address, and
phone number. In order to maximize the probability of contacting establishments using
robotics for our cognitive testing sample, we used robotics shipments data provided to
Seamans by the RIA to generate a list of 3- and 4-digit NAICS industries that were most likely
to use robotic equipment. Then, using County Business Patterns data we tabulated
establishment counts by core-based statistical area (CBSA) and these targeted industries.
These tabs were used to select CBSAs with a good balance of robot-using industries and the
related set of establishments most likely to use robotics (see Buffington, Miranda and
Seamans, 2017).8 Tables 1 and 2 show the list of top 11 robot intensive industries and the
ranking of top CBSAs respectively based on this analysis.
Based on this analysis, we selected Detroit and Chicago as the locations for our first
round of testing. A day was also spent visiting businesses in the Philadelphia/central New
Jersey area in order to diversify across industries. Based on the same analysis, we selected
Los Angeles, Dallas/Fort Worth and Houston as the locations for the second round of
testing. The first round of cognitive testing was conducted in September 2017 and the
second round over late January into February 2018. Recruiting for these interviews proved
difficult with a high number of refusals as well as difficulty in locating manufacturers with
robotic equipment, limiting the number of cases per location to less than the usual ten.
Including some establishments without robotics equipment was desirable in order to ensure
that manufacturers not using robotics would not mistakenly report e xpenditures, but we
were most interested in interviewing those that did use robotics. Buffington, Helper,
Miranda and Seamans served as observers in many of the cognitive interviews to serve as
subject matter experts while in the field but also to apply subject matter expertise to
revisions that resulted from the testing process.
8
The RIA data provides robot shipment counts and value of shipments by industry for years 2012 through 2017. We
used this information to estimate robot intensity use by industry as well as the likelihood that a random
establishment would use robots in that industry. Our methodology involved the following steps. First we computed
the number of units shipped per establishment by industry and year. Establishment counts for 2016 were
approximated by straight line imputation of CBP by industry based on the 2012-2015 growth trend. We then
estimated the cumulative number of robot units in 2017 for the average establishment by industry and year. We
accounted for differences between the industry codes used by the Census and the industry codes used by RIA.
9
3.3.3. Round 1 Cognitive Testing Recommendations and Findings
The first round of testing took place in September 2017 in Detroit and Chicago, with a
supplemental trip to central New Jersey and the greater Philadelphia area. See Figures 3a
and 3b for the tested content. The content included an extended definition of industrial
robotic equipment based on ISO 8373:2012 used by the RIA and IFR (International
Federation of Robotics 2016). This definition was used to provide clear technical guidance
from an authoritative source as well as to limit the scope of the data reported to that of the
RIA and IFR in order to support future data benchmarking. Two versions of the extended
definition were tested. Figure 3a includes the version of the definition that was preferred
after testing; the other tested version did not include the term ‘industrial’ when referring to
robots.
In total, four questions and two definitions were tested in the first round. One set of
questions used dollars as the unit of measurement and the other set used pieces of robotic
equipment as the unit of measurement. Both pairs of questions sought a capital stock
measure as well as an expenditures or flow measure. Figure 3a presents the dollar-based
questions. The first question (A.) asks about the gross value of robotic equipment at the end
of the year and the second question (B.) asks about expenditures on new and used robotic
equipment. These questions were based on the “ASSETS, CAPITAL EXPENDITURES,
RETIREMENTS, AND DEPRECIATION” section of the 2012 ASM and the “CAPITAL
EXPENDITURES” section of the 2016 ASM, respectively. Figure 3b presents the alternate
question pair. The questions included the number of industrial robots in use at the plant,
the average price, and the number of robots purchased in that year. After testing for these
question pairs and definitions was complete, the cognitive testing staff produced a report
including Findings and Recommendations (See Table 3).
Generally, the response to the proposed content was positive. Respondents on average
reported that the term “industrial robotic equipment” was preferred (see Findings 1 and 11)
to “robotics” or “robotics equipment”. Typically, respondents understood what was meant
by robotic equipment, but many agreed that a list of examples or a list of equipment to
include and exclude would be useful (see Finding 11). Companies typically purchase, not
lease, robotic equipment and most expense the equipment using generally accepted
accounting principles (GAAP) or other guidelines (Findings 5 and 4). Respondents had access
to records that included the information required to answer these questions, but these
records were not identified or flagged as robotic equipment in their asset registers (Finding
2). Because larger establishments and/or companies generally have larger asset registers
and because these questions would require research using the asset register, the burden of
responding to these questions generally would increase along with company size (Finding
12). (Herrell and Stettler (2017)). However, several large respondents indicated that they
10
could add a flag to their registers to identify robotic equipment if they knew the ASM survey
questions would recur, and that this step would reduce their reporting burden. Others
commented that they could call a plant manager, who could easily estimate; thus, providing
the instruction that estimates are acceptable would reduce their reporting burden as well.
The phrase “other one-time expenses” did not create any cognitive issues, and respondents
typically reported that other one-time expenses including installation charges and software
were typically included on the invoices for robotic equipment purchases.
The questions concerning gross value of assets and capital expenditures on industrial
robotic equipment resulted in mixed test results. The term “gross value” had a variety of
interpretations. Some respondents thought the question was asking for current market
value, while others thought the question was asking for net book value or purchase price.
For those respondents who took the term to mean current market value, they stated this
was difficult or impossible to report. Respondents noted that net book value and purchase
price could be easily obtained from records, but many questioned whether net book value
was informative as in many instances depreciation would drive this value to zero before the
end of the useful life of the equipment. Purchase price also had drawbacks, namely the lack
of information about vintage and depreciation (see Finding 8). Based on these findings as
well as additional internal review for consistency with the ASM survey instrument, the
decision was made to drop the gross value question. Apart from the respondent burden
issue, the capital expenditures question tested well (see Findings 2 and 9).
The questions using counts of robotic equipment and average price were generally
understood, and in most cases respondents could answer for both how many were in place
and how many were purchased in the reporting year. Respondents did feel questions about
price were burdensome and questioned the useful ness of average price data. Just as with
respondents with large asset registers, the ability of respondents to answer these questions
and the burden imposed on the respondent increased with the size of the establishment
and/or company. The testing staff recommended that in the second round of testing
specifically stating that individual pieces of robotic equipment should be counted
separately, regardless of whether they were working in conjunction with another piece of
robotic equipment. For example, a robotic welding cell may contain several individual robot
arms (See Finding 10).
3.3.4. Round 2 Cognitive Testing Recommendations and Findings
Figure 4 shows the survey content used in the second round of cognitive testing. In
order to present something closer to what the respondents would see when using the
Census Bureau’s online reporting software, the test instrument was changed to reflect
online formatting and design elements. Material changes to the instrument include the
11
changes to the definition as recommended by the first round of cognitive testing (the
consistent use of the term “industrial robotic equipment” throughout the instrument), the
use of the term “adaptable” instead of the word “multipurpose”, the inclusion of a bulleted
list of examples of robotic equipment considered in-scope, and a specific list of equipment
that should be excluded from reporting. These examples and lists of equipment to exclude
were used to address common questions or issues that arose in the first round of testing
but also to scope the question using the same delineations as the IFR. The gross value
question meant to measure the stock of robotic equipment at the establishment was
dropped, as was the question asking about the average price of robotic equipment. Instead,
the draft survey content included the dollar-based question for capital expenditures and the
count-based questions for capital stock (“how many industrial robots were used”). The
count-based question for expenditures was also retained (“how many industrial robots
were purchased”). Prior to the capital expenditures on robotics question, the cognitive
testing staff also added a reference back to the total value of capital expenditures that
would be reported by the respondent in an earlier survey question. Instructions for how to
count robotic equipment that might be integrated into another piece of equipment or cell
with other robotic equipment was also included. Finally, a “check if none” box was added
for each of the questions as well as a prior year reporting box .
Findings and recommendations from the second round of testing can be found in Table
4. Respondents generally understood the definition, instructions, and questions as written
(see Finding 1). Some respondents believed that Computer Numerically Controlled (CNC)
machining equipment and Automated Guided Vehicles (AGVs) should be included as robotic
equipment despite the exclude instruction (see Finding 2). As in the first round of testing,
respondents’ records did include the information on capital expendi tures and other onetime expenses associated with robotics but reporting this information was found to be not
without burden (see Finding 3). The count questions were not difficult or burdensome for
respondents at establishments that did not use robotics or had a small number of robots,
but burden increased with the increase in the use of robotics at the plant such that the
testing staff believes that data collected from large establishments will be of poor quality , if
reported at all (see Finding 4). Furthermore, some respondents were reluctant to report for
each piece of robotic equipment in the count questions regardless of instructions to do so
(see Finding 5). Respondents reported different guidelines and dollar thresholds used for
depreciating capital equipment, while at least one reported expensing robotic equipment as
maintenance in instances where the robotics were replacing a failed part in a larger system
or integrated equipment (Finding 6). Because of the small sample size for the first two
rounds of cognitive testing, owing in large part due to difficulty in finding establishments
with robotic equipment and willing to participate in cognitive interviews, especially with
regard to large companies, the testing staff recommended additional research including a
12
third round of cognitive testing, as well as debriefing interviews to be conducted after the
2018 ASM is conducted (Finding 7).
3.3.5. Round 3 Cognitive Testing
Although two rounds of cognitive testing are typical, because we had not exhausted the
number of visits or time constraint as approved by OMB for testing, we revised the
instrument based on findings from the second round and participated in a limited third
round of (confirmatory) testing. An additional four interviews were conducted by phone in
late April 2018, with three of these being follow up calls to respondents who had
participated in the earlier rounds of testing and had agreed to review the modified
instrument.
A shaded text box was drawn around the definition in order to cue the respondent that
this was informational; white space was added for readability and to separate concepts
within the information block. A sentence was added to clarify language around robotic cells
and rail systems, and a sentence was added to clarify how semiconductor manufacturers
should treat track systems, as these specific issues arose during the second round of testing
(see Figure 5a).
Research in the cognitive testing field suggests it might be desirable to replace lengthy
instructions in survey instruments with equivalent check box versions formulated in the
form of questions in order to lessen the cognitive burden and to force the respondent to
slow down and pay attention to important concepts (Snijkers et al (2013)). Based on this,
the cognitive testing staff recommended the inclusion of a series of text boxes to capture
the heterogeneity of industrial robotic equipment that a manufacturing plant might use.
The check boxes list the types of robotic equipment included in the second round content
and additional types of robotic equipment not previously listed (see Figure 5b). This format
also allows us to request information about “other” types of robotic equipment not listed
which might be in use at the plant.
The instructions were modified further to reduce confusion and provide more clear and
consistent guidance. Specifically, instructions for the robotics capital expenditures question
making reference to the establishment’s total capital expenditures reported elsewhere in
the survey was removed. The question header was changed to reflect language used
elsewhere in the ASM. Explicit instructions for reporting by question number were added,
as well as instructions to address and recognize the inability to break out the cost of robotic
equipment from integrated equipment purchases reported by some respondents (see
Figure 5c).
13
The cognitive testing staff did not produce a full report given the small sample available
in the third round. However, they did make recommendations based on their experience
and the interviews conducted (See Table 5). First, a simplification was recommended in the
initial description of robotic equipment. The instruction that a robot can be “part of a rail
system” was replaced with “incorporated into another piece of equipment” as further
research indicated that rail systems used in semiconductor manufacturing are
implementations of Automated Materials Handling Systems and should be excluded 9. This
simpler language was also found to be easier to understand. Second, the checkbox
question tested well but the recommendation was that it should not be included if the list
of robotic equipment was not exhaustive. Third, the testing staff recommended that the
terms “new and used”, taken from the ASM capital expenditures section, should not be
included as it created confusion with the concept of counting equipment used at the plant.
Finally, they recommended that because equipment may be capitalized or expensed, it
would be beneficial to clarify the count questions in order to specify whether only
equipment being capitalized should be included in order to align with the concepts in the
first question.
The instrument was finalized in July 2018; see Figures 6a-6c for the content that was
submitted to OMB for clearance in October 2018. The instructions were simplified as
described above. The checkbox question was rejected as it would create additional
reporting burden for ASM respondents. Further white space was added between questions
1, 2, and 3 and instructions just prior to questions 1 and 2 were customized for each type of
question. The language “new and used” was retained in the capital expenditures question
in order to maintain continuity with the earlier ASM question, and the potentially confusing
term “USED” in question 2 was changed to “IN OPERATION”. Last, a new comment box was
added to each of the count questions (e.g., “If you are unable to provide the number of
industrial robots PURCHASED in 2018, please explain.”) in order for respondents to provide
additional information if they are unable to report on the count questions.
4. Conclusion
Robotics will likely have a large effect on our economy and society, but additional data
on the use of robotics is needed. The U.S. Census Bureau does not currently collect any data
on robotics, but it has collected similar data in the past via the Survey of Manufacturing
9
Rail systems in semiconductor manufacturing include autonomous vehicles used to move materials between
locations (Kim 2008). Autonomous vehicles as well as more generally robotic logistical systems are classified as
service robots (International Federation of Robotics 2016).
14
Technology. In Europe, the European Manufacturing Survey collects firm level data on the
use of robotics.
To address the need for data on robotics, our research team, which was comprised of
internal Census employees and external university researchers, developed questions on
robotics for inclusion in the Census’ Annual Survey of Manufactures. The questions include:
(1) capital expenditures for new and used industrial robotic equipment in 2018, (2) number
of industrial robots in operation in 2018, and (3) number of industrial robots purchased in
2018. These questions were arrived at following an extensive cognitive testing process, the
details of which are described within.
15
References
Buffington, Catherine, Javier Miranda and Robert Seamans (2017). “Robotics Industry Analysis.”
Mimeo, U.S. Census Bureau.
Doms, Mark, Timothy Dunne and Kenneth R. Troske (1997). “Workers, Wages and Technology.”
Quarterly Journal of Technology 62(1): February 1997.
Dunne, Timothy (1994). “Plant Age and Technology use in U.S. Manufacturing Industries.” The
RAND Journal of Economics, Vol. 25, No. 3 (Autumn, 1994), pp. 488-499
Furman, Jason (2016) “Is This Time Different? The Opportunities and Challenges of Artificial
Intelligence” Remarks at AI Now: The Social and Economic Implications of Artificial
Intelligence Technologies in the Near Term, New York University, July 7, 2016. Available at:
https://obamawhitehouse.archives.gov/sites/default/files/page/files/20160707_ce a_ai_fur
man.pdf
Furman, Jason and Robert Seamans (2018), “AI and the Economy,” in NBER Innovation Policy
and the Economy, ed Joshua Lerner and Scott Stern.
Frey, Carl B. and Michael A. Osborne (2017) “The Future of Employment: How Susceptible Are
Jobs to Computerization?” Technological Forecasting and Social Change, Vol 114, pp. 254–
280.
Graetz, Georg and Guy Michaels (2015) “Robots at Work,” Centre for Economic Performance
Discussion Paper No. 1335.
Herrell, Kenneth and Kristin Stettler (2017), Findings and Recommendations from the First
Round of Cognitive Testing of Robotics Questions for the Annual Survey of Manufactures ,
internal Census Bureau report.
Herrell, Kenneth and Kristin Stettler (2018a), Findings and Recommendations from the Second
Round of Cognitive Testing of Robotics Questions for the Annual Survey of Manufactures,
internal Census Bureau report.
Herrell, Kenneth and Kristin Stettler (2018b), Findings and Recommendations from additional
cognitive interviews on the revised robotics questions for the Annual Survey of Manufactures
[internal memorandum].
International Federation of Robotics. (2016), World Robotics Industrial Robots 2016.
16
Kim, Dong II "The evolution of automated material handling systems (AMHS) in semiconductor
fabrication facilities," 2008 6th IEEE International Conference on Industrial Informatics,
Daejeon, 2008. doi: 10.1109/INDIN.2008.4618055
Lewis, Ethan (2005). "Immigration, skill mix, and the choice of technique," Working Papers 05-8,
Federal Reserve Bank of Philadelphia.
Luque, Adela & Javier Miranda (2000). "Technology Use and Worker Outcomes: Direct Evidence
from Linked Employee-Employer Data," Working Papers 00-13, Center for Economic Studies,
U.S. Census Bureau
McElheran, Kristina (2015). “Do market leaders lead in business process innovation? The case
(s) of e-business adoption.” Management Science, 61(6), 1197-1216.
McElheran, Kristina (2018). “Economic Measurement of AI.” University of Toronto working
paper.
McGuckin, Robert H., Mary L. Streitwieser and Mark Doms (1998). “The Effect of Technology
Use On Productivity Growth.” Economics of Innovation and New Technology, Vol. 7, Num.1, pg.
1-26.
Miranda, Javier and Robert Seamans (2017). “ASM Questionnaire Content Change Proposal
Robotics”. U.S. Census Bureau.
National Academy of Sciences (NAS) (2017) “Information Technology and the U.S. Workforce:
Where Are We and Where Do We Go from Here?” Available at:
https://www.nap.edu/catalog/24649/information-technology-and-the-us-workforcewhere-are-we-and
Raj, Manav and Robert Seamans. 2018. “AI, Labor, Productivity, and the Need for Firm-Level
Data.” In The Economics of Artificial Intelligence, eds. Ajay Agrawal, Joshua S. Gans and Avi
Goldfarb, University of Chicago Press: Chicago. Forthcoming
Snijkers, Ger, Gustav Haraldsen, Jacqui Jones, and Diane K. Willimack (2013). Designing and
Conducting Business Surveys, Wiley: Hoboken, NJ.
U.S. Census Bureau. 2018. “About the Annual Survey of Manufactures.” Accessed August 3,
2018. https://www.census.gov/programs-surveys/asm/about.html
U.S. Census Bureau. 2018. “Annual Survey of Manufactures Methodology.” Accessed August 3,
2018. https://www.census.gov/programs-surveys/asm/technicaldocumentation/methodology.html
17
Table 1. List of Top 11 Robot Intensive Industries
Beverages
Chemical products, pharmaceuticals, cosmetics
Basic metals (e.g. iron, steel, aluminum, copper, chrome etc.)
Industrial machinery
Household appliances
Electronic components/devices
Semiconductors, LCD, LED
Computers and peripheral equipment
Information communication equipment domestic and professional
Motor vehicles, motor vehicles engines and bodies
Parts and accessories for motor vehicles
Source: Authors’ calculations based on RIA data.
Table 2. Ranking CBSA areas based on Robot Intensity Use
CBSA code CBSA title
14460
31080
16980
19100
33460
35620
38060
40140
41740
41860
12060
12420
19740
19820
26420
33100
37980
38900
41940
42660
45300
12580
26900
Number of
Robot
Intensive
Industries
Boston-Cambridge-Newton, MA-NH
Los Angeles-Long Beach-Anaheim, CA
Chicago-Naperville-Elgin, IL-IN-WI
Dallas-Fort Worth-Arlington, TX
Minneapolis-St. Paul-Bloomington, MN-WI
New York-Newark-Jersey City, NY-NJ-PA
Phoenix-Mesa-Scottsdale, AZ
Riverside-San Bernardino-Ontario, CA
San Diego-Carlsbad, CA
San Francisco-Oakland-Hayward, CA
Atlanta-Sandy Springs-Roswell, GA
Austin-Round Rock, TX
Denver-Aurora-Lakewood, CO
Detroit-Warren-Dearborn, MI
Houston-The Woodlands-Sugar Land, TX
Miami-Fort Lauderdale-West Palm Beach, FL
Philadelphia-Camden-Wilmington, PA-NJ-DE-MD
Portland-Vancouver-Hillsboro, OR-WA
San Jose-Sunnyvale-Santa Clara, CA
Seattle-Tacoma-Bellevue, WA
Tampa-St. Petersburg-Clearwater, FL
Baltimore-Columbia-Towson, MD
Indianapolis-Carmel-Anderson, IN
18
16
16
15
15
15
15
15
15
15
15
14
14
14
14
14
14
14
14
14
14
14
13
13
Number of
Top 11
Robot
Intensive
Industries
11
11
10
10
10
10
10
10
10
10
9
9
9
9
9
9
9
9
9
9
9
8
8
15380 Buffalo-Cheektowaga-Niagara Falls, NY
16740 Charlotte-Concord-Gastonia, NC-SC
17140 Cincinnati, OH-KY-IN
17460 Cleveland-Elyria, OH
28140 Kansas City, MO-KS
40900 Sacramento--Roseville--Arden-Arcade, CA
41180 St. Louis, MO-IL
41620 Salt Lake City, UT
18140 Columbus, OH
24340 Grand Rapids-Wyoming, MI
33340 Milwaukee-Waukesha-West Allis, WI
34980 Nashville-Davidson--Murfreesboro--Franklin, TN
35300 New Haven-Milford, CT
36740 Orlando-Kissimmee-Sanford, FL
46140 Tulsa, OK
49340 Worcester, MA-CT
10420 Akron, OH
13820 Birmingham-Hoover, AL
14500 Boulder, CO
14860 Bridgeport-Stamford-Norwalk, CT
19380 Dayton, OH
24860 Greenville-Anderson-Mauldin, SC
25540 Hartford-West Hartford-East Hartford, CT
29820 Las Vegas-Henderson-Paradise, NV
31140 Louisville/Jefferson County, KY-IN
37100 Oxnard-Thousand Oaks-Ventura, CA
38300 Pittsburgh, PA
47900 Washington-Arlington-Alexandria, DC-VA-MD-WV
Source: Authors’ calculations based on County Business Patterns data.
19
12
12
12
12
12
12
12
12
10
11
11
11
11
11
11
10
10
10
10
10
10
10
10
9
9
10
10
9
7
7
7
7
7
7
7
7
6
6
6
6
6
6
6
6
5
5
5
5
5
5
5
5
5
5
5
5
Table 3. Findings and Recommendations from Round 1 of Cognitive Testing
Findings
Recommendations
Accepted
1) Respondents had definitional
differences between Robots, Robotic
Equipment, and Industrial Robotic
Equipment.
We recommend using the term “robotic equipment” or
“industrial robotic equipment” throughout the survey,
although when a question asks about a singular unit of
robotic equipment, such as asking for the number on hand
at a location, it is unclear whether “robot” should be used,
or something like “piece of robotic equipment.” Another
alternative would be using “robotic equipment” as the
main term, but defining “robot” as an individual unit and
using that for count questions.
No recommendation. “Estimates are acceptable” may be
useful for respondents who feel that going through records
would be too burdensome.
Y
No recommendation.
NA
No recommendation.
NA
Census should be aware that respondents may treat leased
equipment differently than purchased based on what’s
available in their records, and that responses may differ as
a result.
No recommendation.
NA
2) Records have most of the
information we need, but cannot sort
robotic equipment from other
equipment.
3) Servicing of the equipment varies by
company.
4) Most companies capitalize robotic
equipment, based on GAAP or other
requirements.
5) Most companies purchase rather
than lease; leased equipment may pose
a problem for reporting costs.
6) Robotics equipment is currently
reported as “Other” capital
expenditures on the ASM by all
respondents we spoke with.
20
NA
NA
Findings
7) Determining what an individual
“robot” is could be problematic.
Recommendations
Definitions should clarify what exactly an individual “robot”
or piece of “robotic equipment” is and how to count it,
regarding either of the above situations. We make the
following recommendations, based on our understanding
of what can be commonly understood and applied across
respondents:
Accepted
Y
Each individual robot or piece of robotic equipment that
was purchased should count as an individual robot,
regardless of whether it was working in conjunction with
another robot on a specific task.
If a piece of robotic equipment was affixed to a piece of
non-robotic equipment, only the value of the robotic
equipment should be counted.
8) Gross Value version of question had
varying interpretations.
9) Total capital expenditures somewhat
clear, but some slight confusion.
We recommend not asking the gross value of the robotic
equipment, due to the difficulty respondents had
interpreting the question and the questionable usefulness
of the data.
In the second round of testing, probe respondents on what
they may include as “other one-time costs.”
It may be helpful to conduct interviews with a handful of
robotics manufacturers and system integrators, with the
purpose of learning what they included in the sales price of
robotic equipment.
21
Y
Y and N
Findings
10) Count and average purchase price
were generally understood;
questionable usefulness of average
purchase price.
Recommendations
The count question generally tested well, and could be
reported for both the number on hand and the number
purchased in 2017. Census should determine whether it is
more useful to know the number on hand or the number
currently in use, or both, and phrase the question(s)
accordingly.
Accepted
Y
Add specific instructions on this issue to the question, and
probe respondents in Round 2 on whether these
instructions are clear and/or appropriate. For the sake of
ease of reporting and creating consistency between
respondents, we suggest the following:
--Each individual robot or piece of robotic equipment that
was purchased should count as an individual robot,
regardless of whether it was working in conjunction with
another robot on a specific task.
--If a piece of robotic equipment was affixed to a piece of
non-robotic equipment, only the value of the robotic
equipment should be counted.
Average purchase price had issues, with some respondents
feeling that it added burden (having to calculate the
average), that it was not asked elsewhere on the ASM and
is thus an unusual task for them, and some questioned the
usefulness of the data. For those reasons, we recommend
not asking about the average purchase price.
11) Definition with “industrial” in it was
preferable to most respondents.
The version of the definitions/instructions using
“industrial” should be used.
Y
Ensure that terminology is kept consistent between the
instructions/definition and the question itself.
Remove the term “multipurpose” completely, or replace it
with something such as “physically adaptable to different
applications”
Consider adding bulleted lists of include, exclude, and
examples.
12) Estimated difficulty and burden of
questions.
Continue to probe in Round 2 about the estimated burden
and difficulty of these questions, and any estimation
strategies, particularly for larger companies.
[Provided revised questions and definition]
13) Neither of the questions or
definitions tested perfectly; a revised
version should be used in another
round of testing.
NOTE: This table was developed based on Herrell and Stettler (2017).
22
NA
Y
Table 4. Findings and Recommendations from Round 2 of Cognitive Testing
Findings
Recommendations
Accepted
1) Respondents generally
understood what the questions
were asking for, and for the most
part, did not have trouble with the
instructions or definitions, but
changes could be made.
2) Several respondents took issue
with Computer Numerical Control
(CNC) machinery and Automated
Guided Vehicles (AGVs) being
excluded.
Reformat the instructions into a short series of instructions,
followed by a question [that turns instructions into a series of
check boxes].
Y
Although we did not talk to any respondents who would do so,
it is possible that other respondents would feel compelled to
report CNC machinery as robotics. The recommendation from
Question 1 would allow such respondents to select Other, and
use the specify line to include what they deem to be robotic
equipment
In addition to the note that says “estimates are acceptable,”
we recommend providing a checkbox next to the answer field
allowing respondents to indicate that their answer is an
estimate, thus providing further assurances that we are
accepting of estimates for this particular question.
N
Our first recommendation would be to not collect the data on
the count of respondents. Based on the interviews we have
conducted, we think that large companies will have
tremendous issues, if not outright inability, to provide accurate
numbers. Even with the limited number of large companies
that we have interviewed, we have not found any company of
any size that flags purchases by whether or not they are
robotic, and we therefore think that it is unlikely that any
company would have an automated method to pull data
specifically in regards to industrial robotic equipment.
N
3) As in Round 1, most
respondents have the capital
expenditures for robotics included
in their capital expenditures, but
cannot identity them as robotics
easily.
4) Count of robots is feasible for
companies with few or no robots,
but may be too burdensome for
respondents in larger companies
and extremely difficult to get
accurate figures.
N
We recognize that there are some benefits to starting to collect
the data, and under the intended plan to ask the questions only
as a special inquiry, we do not strongly object. However, we do
not believe that the data from the initial collection should be
published.
5) Some respondents would
provide a count that differed from
our instructions, even though they
understood what they were being
instructed to do.
See recommendation for Finding #4 above.
23
N
Findings
Recommendations
6) Companies have different
No recommendation.
requirements for what meets
capital expenditures
requirements; not all companies
follow GAAP.
7) Because of the small sample
We strongly recommend conducting debriefing interviews after
size from these rounds of testing, these questions are fielded, to learn more about respondents’
particularly in regards to large
behaviors when they are actually required to go through their
companies, more research should records or contact plant managers to provide the data that is
be done on the topic.
being asked for.
8) The new tax bill will likely not
No recommendation.
impact how respondents maintain
their accounting records nor how
they report on Census surveys;
respondents did not know how it
would impact their investment in
robotics.
NOTE: This table was developed based on Herrell and Stettler (2018a).
24
Accepted
NA
Y
NA
Table 5. Recommendations from Round 3 of Cognitive Testing
Topics
Recommendations
Accepted
Instructions
Consider changing the line about robotic cells to read “An
industrial robot may be incorporated into another piece of
equipment.”
Y
Checkbox question
Leave the checkbox question as is, but ensure the list of
options is mostly comprehensive. If it is not feasible to have a
comprehensive list, exclude this question.
Remove “new and used” from the capital expenditures
question.
N
Capital expenditures and number
of purchased/used questions
Add clarifying instructions to line 3, regarding whether only
capitalized purchases should be included.
NOTE: This table was developed based on Herrell and Stettler (2018b).
25
N
Figure 1. Annual Survey of Manufactures Capital Expenditures Content
26
Figure 2. 2017 Census of Manufactures Capital Expenditures Content
27
Figure 3a. Survey Content Draft for Round 1 of Cognitive Testing
28
Figure 3b. Survey Content Draft for Round 1 of Cognitive Testing
29
Figure 4. Survey Content Draft for Round 2 of Cognitive Testing
30
Figure 5a. Survey Content Draft for Round 3 of Cognitive Testing
31
Figure 5b. Survey Content Draft for Round 3 of Cognitive Testing
32
Figure 5c. Survey Content Draft for Round 3 of Cognitive Testing
33
Figure 6a. Final Proposed Content
34
Figure 6b. Final Proposed Content
35
Figure 6c. Final Proposed Content
36
| File Type | application/pdf |
| Author | Blynda K Metcalf (CENSUS/EWD FED) |
| File Modified | 2018-11-13 |
| File Created | 2018-11-13 |