Findings and Recommendations from Cognitive Testing for the Annual Business Survey – Innovation Module

Attachment F

Findings and Recommendations from
Cognitive Testing for the Annual Business
Survey – Innovation Module
Prepared for:
Patrice Hall, Economic Reimbursable Surveys Division (ERD)
Aneta Erdie, ERD
Audrey Kindlon, National Center for Science and Engineering Statistics (NCSES)
John Jankowski, NCSES

Prepared by:
Aryn Hernandez, Data Collection Methodology and Research Branch (DCMRB)
Krysten Mesner, DCMRB

Economic Directorate
U.S. Census Bureau

December 13, 2019

DRB Clearance Number: CBDRB-FY20-ESMD003-002

1

Attachment F

Contents
Research Objectives ...................................................................................................................................... 4
Research Methodology ................................................................................................................................. 5
Participants ................................................................................................................................................... 6
Findings and Recommendations ................................................................................................................... 6
General Findings and Recommendations ..................................................................................................... 6
Finding #1: Estimated time to complete averaged approximately one to two hours. ............................. 6
Finding #2: Module was more difficult for service-based industries........................................................ 6
Question Specific Findings and Recommendations ...................................................................................... 6
Finding #3: Introduction............................................................................................................................ 6
Finding #4: New or improved goods or services ....................................................................................... 7
Finding #5: Novelty of new or improved goods or services...................................................................... 8
Finding #6: New or improved goods or services as percent of total sales ............................................... 9
Finding #7: Expectations ........................................................................................................................... 9
Finding #8: Sources of new or improved goods or services ................................................................... 10
Finding #9: Product cooperation partners .............................................................................................. 10
Finding #10: Where were these businesses or organization located ..................................................... 10
Finding #11: Most important innovation ................................................................................................ 11
Finding #12: Most important innovation and sales ................................................................................ 11
Finding #13: New or improved business processes ................................................................................ 12
Finding #14: Improved business process developers ............................................................................. 12
Finding #15: Activities for new or improved products or business processes ....................................... 12
Finding #16: Activity costs ...................................................................................................................... 13
Finding #17: R&D for activity costs ......................................................................................................... 13
Finding #18: Government support .......................................................................................................... 13
Finding #19: Barriers of innovation......................................................................................................... 14
Finding #20: Business strategies ............................................................................................................. 15
Finding #21: Use of digital technologies ................................................................................................. 15
Finding #22: Capabilities for using digital technologies .......................................................................... 16
Finding #23: Technology capabilities ...................................................................................................... 16
Appendix A: Draft Questions, Round 1 ....................................................................................................... 18
Appendix B: Interview Protocol, Round 1 ................................................................................................... 35
Appendix C: Draft Questions, Round 2 ....................................................................................................... 48
DRB Clearance Number: CBDRB-FY20-ESMD003-002

2

Attachment F

Appendix D: Interview Protocol, Round 2………………………………………………………………………………………………66

DRB Clearance Number: CBDRB-FY20-ESMD003-002

3

Attachment F

Research Objectives
The Annual Business Survey (ABS) asks questions about the business owner(s) as well as various
characteristics about the business. The 2020 ABS cognitive testing will include new questions in the
innovation module. These proposed questions focused on:
•

•
•
•
•
•

New or Improved Goods and Services asks if the business has introduced new or improved
goods or services, how much revenue those goods or services account for, how those goods and
services are performed, and some information about the development (three years including
2017-2019).
New or Improved Business Processes asks about the activities the business conducted to
improve its processes and their associated costs.
Government Support asks about which programs and grants the business took advantage of to
aid innovation.
Drivers and Barriers of Innovation assesses the impact markets, public policy, and society had
on the decision of whether or not to conduct innovation activities.
Business Strategies asks about a business’ focus on a multitude of specific innovation activities.
Use of Digital Technologies asks about which digital technologies the business uses in its
innovation activities and the business’ capability to acquire and use new technologies.

These draft questions were discussed using cognitive interviewing. The objectives of these interviews
were to learn how respondents:
•
•
•
•

understand and interpret the data requests;
understand the terminology used throughout the questions;
keep records related to the data items; and,
perceive the burden that would be associated with answering the questions.

The round one innovation module draft can be found in appendix A. The round one protocol can be
found in appendix B. The round two innovation module draft can be found in appendix C. the round two
protocol can be found in appendix D.

Note: The two rounds of testing were conducted in close proximity, time-wise, from each other.
Between the rounds of testing, some changes were made to the draft materials. This report will attempt
to summarize findings from all interviews, but the primary focus will be on the findings from Round 2 of
the interviews, and provides recommendations based on those draft questions.

DRB Clearance Number: CBDRB-FY20-ESMD003-002

4

Attachment F

Research Methodology
Between June and August 2019, Aryn Hernandez and Krysten Mesner from the U.S. Census Bureau’s
Data Collection and Methodology Research Branch (DCRMB) conducted 36 cognitive interviews over
two rounds of testing, regarding proposed new questions. Participants in the interviews were recruited
from a list of recent respondents to the ABS. Generally, those recent respondents were the primary
participants in the interviews, but on occasion these respondents brought one or more colleagues to the
interviews, because they were integral to the typical survey data collection process. On occasion, these
people who accompanied the original participant turned into the primary participant in these
interviews. Interviews were conducted in-person.
Cognitive interviews were the testing methodology used for this project. Cognitive interviews are used
to, ‘(a) understand the thought processes used to answer survey items, and (b) to use this knowledge to
find better ways of constructing, formulating, and asking survey questions’ (Forsyth and Lessler, 1991). 1
Cognitive interviews traditionally focus on the four steps of Tourangeau’s (1984) cognitive response
model: comprehension, retrieval, judgment, and communication/reporting. 2 Comprehension refers to
the respondent’s interpretation and understanding of the question’s language, structure, and grammar.
In order to answer the question, a respondent must understand what information is being requested on
the survey. Retrieval is the step where relevant information is obtained, either from records or from
memory. The next step, judgment, describes the respondent’s evaluation of the completeness or
relevance of the data obtained. It is here that estimates are made based on partial or incomplete data.
The last step, communication or reporting, deals with mapping the response to the answer space
provided and possibly altering the answer.
While Tourangeau’s model is suitable for household and social surveys, the establishment survey setting
presents additional factors that must be considered. First, instead of or in addition to a reliance on
memory, establishment surveys rely heavily on records and the information contained within them.
Second, organizations tend to have distributed knowledge. Some people are experts in one type of
information, while others keep information about something else. Third, competing priorities, both for
the organization and the individual(s) completing the questionnaire, mean that the survey sometimes
does not receive the amount of attention that researchers and data collectors would like. Finally,
organizations regularly authorize only a few individuals to release data. If the data provider is not
authorized to release the data, an additional step must be added to the response process. Tourangeau’s
model was expanded by Sudman et al (2000) to account for these factors. 3

Forsyth, B.H. and Lessler, J.T. (1991). ‘Cognitive Laboratory Methods: A Taxonomy.’ In Measurement Errors in Surveys, P.P. Biemer, R.M.
Groves, L.E. Lyberg, N.A. Mathiowitz, S. Sudman (eds). New York: John Wiley & Sons, Inc.
1

Tourangeau, R. (1984). ‘Cognitive Sciences and Survey Methods.’ In Cognitive Aspects of Survey Methodology, T.B. Jabine, M.L. Straf, J.M.
Tanur, and R. Tourangeau (eds). Washington, DC: National Academy Press.
2

Sudman, S., Willimack, D.K., Nichols, E., and Mesenbourg, T.L. (2000). ‘Exploratory Research at the U.S. Census Bureau on the Survey Response
Process in Large Companies.’ Paper prepared for presentation at the Second International Conference on Establishment Surveys, Buffalo, NY.
3

DRB Clearance Number: CBDRB-FY20-ESMD003-002

5

Attachment F

Participants
Participants included sample members from the 2018 ABS who responded yes to one or more of the
existing innovation questions. Businesses were recruited from across the nation and include a range of
sizes based on employee count (in particular small and medium) and covered a variety of industries.
Table 1. Count of businesses by employee count and round.
Employee Count

Round 1

Round 2

Total

< 50

6

8

14

50 – 499

6

7

13

500+

4

5

9

16

20

36

Total

Findings and Recommendations
General Findings and Recommendations
Finding #1: Estimated time to complete averaged approximately one to two hours.
After reviewing the draft module, most respondents gave a time estimate to complete the module
between 15 minutes and 5 hours. A few respondents gave an estimate that exceeded 5 hours, up to two
days. Respondents that gave the 15 minute estimates believed that the module did not apply to them,
and thus would skip most of the questions. Those that gave higher estimates cited obtaining the
financial data as the main burden.
Recommendation(s):
•

No recommendations needed.

Finding #2: Module was more difficult for service-based industries.
Overall, this module was more difficult for service-based industries. One respondent noted that
‘services’ were not included in this question and as her/his company primarily provided services, s/he
would skip this section.
Recommendation(s):
•

See Finding #3.

Question Specific Findings and Recommendations
Finding #3: Introduction
Upon initially reading the introduction, most respondents stated that they would be the appropriate
DRB Clearance Number: CBDRB-FY20-ESMD003-002

6

Attachment F

person to answer this module based on the instructions, and that their companies did have new or
improved goods or services. Some respondents stated that they may seek input from the following types
of areas:
• Marketing
• Research and Development (R&D)
• Technology
• Products
• COO
• CIO
A couple of respondents felt this section did not apply to their companies as they primarily provided
services. The word ‘products’ seemed to imply tangible goods only to these respondents.
Recommendation(s):
•

Replace all instances of “products (goods or services)” with just “goods or services.”

•

Consider revising title of section as well (e.g. ‘Innovative Goods, Services, and
Business Processes’)
Use Oxford commas.
o

Finding #4: New or improved goods or services
•

Nearly all respondents were able to answer this question initially. However, when asked what
‘differed significantly’ meant to them, respondents gave varying answers. Examples given by
respondents include:
o ‘Upgrade that changes the way people use [product]’
o ‘going to roll out a new product’
o ‘have a dramatic change, not small enhancements, adding features’
o Changes to ‘content of product’
o ‘changes structure of the way you do business, number of employees needed, cost
savings, streamlining’
o Change in ‘how product is made’
o ‘different models, function change, performing a new or better function … create new
item number for form or fit, but could also just be for new color’
o ‘different type of product, had to purchase additional machinery’
o Percentage of difference between old and new products (50 – 75 % difference is
significant)
o
o
o
o

Enough to call it a new revenue stream, line of business, or new skill set
New line of business, building new tools, new tech, total dollar amount spent on
development
Implement new process or deliverable that changes value to customer, anything that
would change the process in how you bring the product to the customer
“Know it when you see it … Professional judgement.”

DRB Clearance Number: CBDRB-FY20-ESMD003-002

7

Attachment F

‘Differed significantly’ requires further explanation or examples. Although there is a brief description in
the opening instructions, respondents are unlikely to read it initially or refer to it after starting the form.
Recommendation(s):
• Split into two questions as the examples for goods and services differ. Questions should remain
on the same screen in the electronic instrument. See example below:
New or improved goods or services
During the three years 2016 to 2018, did this business introduce to the market any new or improved
goods that differed significantly (e.g. in form, function, profit, market share, skills, materials, or
machinery required) from this business’s previous goods or services?

Goods: usually a tangible object such as a smartphone, furniture, or packaged software, but also
includes digital goods such as downloadable software, music and film

Exclude the simple resale of new goods and changes of a solely aesthetic nature.
o
o

Yes
No

During the three years 2016 to 2018, did this business introduce to the market any new or improved
services that differed significantly (e.g. in function, profit, market share, or skills and materials needed)
from the business’s previous goods or services?

Services: usually intangible, such as retailing, insurance, educational courses, air travel, consulting, etc.,
also includes digital services
Exclude the simple resale of new services.
o
o

Yes
No

Finding #5: Novelty of new or improved goods or services
Most respondents were able to answer this question. A couple of respondents were unsure of what was
meant by ‘market’ and mentioned specific local retail locations in their respective areas.
Recommendation(s):
DRB Clearance Number: CBDRB-FY20-ESMD003-002

8

Attachment F

•

No recommendation needed.

Finding #6: New or improved goods or services as percent of total sales
Some respondents believed these figures would be easy to obtain. However, a respondent from a
goods-based industry stated that the revenues from new products were not separated in her/his
records, but that s/he could estimate the revenue based on the sales number. One company stated that
they do get ‘revenues’ but were a nonprofit cooperative. This question was also very difficult for servicebased industries. ‘This is difficult for [service industry], it’s more about decreasing expenses.’ This
respondent went on to say that it would be difficult or impossible to attribute revenue to any specific
change in their services.
Recommendation(s):
•

This item is not formatted as a question. Reword item into a proper question and move
reference year to the beginning. See example below:

In 2018, what percentage of this business’s ‘A.11 DOMESTIC SALES AND REVENUES’ sales and revenues
were attributable to or originated from domestic operations in the following categories?
Details must sum to total dollar amount of Domestic Sales and Revenues.

Finding #7: Expectations
Most respondents felt confident they could answer this question. Respondents gave the following
examples of metrics that may be used to determine their answers:
•
•
•
•
•
•
•
•

Number of new sales
Churn
Revenue
Return on investment
Profit
Decreased expenses
Guest satisfaction scores
Customer acceptance

Some respondents stated that they would reach out to the following areas to get a more official
company opinion:
•
•
•
•
•

Marketing
‘Management’
VP
IT team
Operations Team
DRB Clearance Number: CBDRB-FY20-ESMD003-002

9

Attachment F

When asked to consider a scenario in which two new goods or services were introduced, one doing well
and one failing, and asked how they would answer, respondents gave the following answers:
•
•
•
•
•

‘I would say ‘not met’ because they didn’t meet expectations on both.’
Would answer ‘not met’ if one failed.
‘Would take an average’
‘Would probably say ‘not met,’ but would like a ‘somewhat’ option.’
‘Would still say ‘met’ since it was a combination of success.’

Recommendation(s):
• No recommendation needed.

Finding #8: Sources of new or improved goods or services
Most respondents felt confident they could answer this question. Some respondents mentioned that
they would consult with others in their companies such as managers. Most respondents would consider
parent companies, affiliates, or subsidiaries as part of ‘this business’ when answering.
One service-based respondent stated that s/he would only know if there was an invoice that came
across her/his desk and that it would take a lot of time to reach out to others to determine if other in
her/his company were working with outside sources. One respondent complained that this question lack
personal pronouns, because ‘businesses don’t innovate, people do.’
Recommendation(s):
•

No recommendation needed.

Finding #9: Product cooperation partners
Most respondents could answer this question.
Recommendation(s):
•

No recommendation needed.

Finding #10: Where were these businesses or organization located
Most respondents were able to answer this question. However, when asked how they would report
working with the domestic location of a foreign-owned company, respondents gave the following
rationales:
•
•
•
•
•

‘If I knew, I would still report U.S’
‘Where work was done’
‘Would have to ask, but would probably say U.S.’
‘Where ever I was sending the check to. May not know and may not know who to ask, so would
use billing address.’
‘If I knew, I would choose ‘rest of world.’’
10
DRB Clearance Number: CBDRB-FY20-ESMD003-002

Attachment F

•

‘Where the work was conducted and location of people doing the work.’

Many respondents stated that they likely would not know if the companies they worked with are owned
by foreign parents. Respondents are most likely to answer this question with the billing address of the
companies.
Recommendation(s):
•

If the billing address is what is required to answer this question, then no recommendation
needed. If NSF needs to know where the work is being done, or where the parent company is
located, that should be specified in the question. However, respondents may not know if the
companies they are dealing with have parent companies and are likely to answer if the billing
address regardless.

Finding #11: Most important innovation
Most respondents were able to answer this question. Respondents gave the following examples of the
meaning of ‘important’:
•
•
•
•
•
•
•
•
•
•
•
•

Most efficiencies
Improvements
Cost savings
Long term life of product
Impact of market
Sustainability
Revenue
New to market
Recognition
New products
New sales
Difference from competitors

Most respondents stated that if the question was changed to ‘the product that accounted for the
most revenue,’ their answers would change.
Recommendation(s):
•

No recommendation needed.

Finding #12: Most important innovation and sales
Most respondents stated that this number would be difficult to obtain and would likely need to be
estimated. A few of these respondents stated that they would need to reach out to others in their
companies such as finance or marketing. Some respondents did state that they could provide this
number. One respondent stated that they would prefer a ‘Do not know’ option, so that they would not
be forced to provide a number that could be incorrect.
Recommendation(s):
DRB Clearance Number: CBDRB-FY20-ESMD003-002

11

Attachment F

•

No recommendation needed, however, data will likely consist mostly of rough estimates.

Finding #13: New or improved business processes
Most respondents stated that they could answer this question. Some respondents stated that they
would need to reach out to other areas such as marketing or HR. Most respondents said that the
categories were clear, however, some respondents were confused by item ‘f’.
A service-based respondent stated that it was difficult to parse out what was ‘customer-facing vs
backend processes.’ A few respondents would say yes to all items because they are ‘always trying to
innovate.’ This question could be prone to false positives.
Recommendation(s):
•
•

Consider adding examples of ‘differed significantly,’ similar to the first question.
o Cost or production time savings, streamlining, etc…
Add examples to or clarify item ‘f’.

Finding #14: Improved business process developers
Most respondents stated that they could answer this.
Recommendation(s):
•

No recommendation needed.

Finding #15: Activities for new or improved products or business processes
Most respondents were able to answer this question. Some respondents stated that this question would
be difficult due to the need of reaching out to others in their companies, and that they ‘may not know
what everyone is doing’. Respondents mentioned reaching out to the following areas for this
information:
•
•
•
•

R&D
‘Energy management’
Communications
‘Someone higher up’

Most respondents stated that all of the items were clear. Again, a couple of respondents wanted
examples of ‘differed significantly.’ One respondent stated that it was hard to tell if these categories
differed significantly when the product did not change much or at all.
Recommendation(s):
•

Consider adding examples of ‘differed significantly,’ similar to the first question.
o Cost or production time savings, streamlining, etc…

DRB Clearance Number: CBDRB-FY20-ESMD003-002

12

Attachment F

Finding #16: Activity costs
Respondents did not have difficulty understanding the meaning of this question. However, most
respondents stated that this figure would be ‘hard to pull together.’ Respondents indicated that this
question would require extra research to find all categories, which could include calculating percentages
of salaries for individuals. Overall, this question would be time consuming and difficult. This figure would
be an estimate at best for most respondents.
One respondent stated that s/he may be able to estimate it, since they have to keep track of costs.’
Another respondent stated that they would have to look at individuals’ salaries, as well as the cost of
contractors. This respondent stated that they would reach out to their VP to find out what people were
working on. A few respondents stated that their answer to this question would be the same as their
answer to ‘R&D FOR ACTIVITY COSTS.’
Recommendation(s):
•

This question would be difficult to answer for most respondents due to the data being spread
out amongst different records, if available at all, consider dropping this question.

Finding #17: R&D for activity costs
Most respondents considered this question easier and more obtainable than the previous question.
Some respondents stated that their answer to this question would be the same as their answer to
‘ACTIVITY COSTS.’
Recommendation(s):
•

If NSF would prefer to retain the ‘ACTIVITY COSTS’ question, auto populate the amount from the
previous question into this question:

Of the $XXX,XXX,XXX ‘ACTIVITY COSTS’ reported, how much was for research and development (creative
and systematic work undertaken in order to increase the stock of knowledge and to devise new
applications of available knowledge) in 2018?

$Mil. Thou. Dol.

,000

Finding #18: Government support
Most respondents felt that they could answer this question; however, most stated that they did not
receive government support. Most support came in the form of state tax incentives.
Recommendation(s):
•

No recommendation needed.
DRB Clearance Number: CBDRB-FY20-ESMD003-002

13

Attachment F

Finding #19: Barriers of innovation
As most of the respondents that researchers spoke to were not part of their companies’ upper
management, most respondents stated that they would either take their best guess or reach out to the
leadership of their companies. One respondent stated that all of these factors were ‘important,’
however, not all of them had a huge impact on their business. This respondent suggested we ask about
the ‘level of discouragement’ (i.e. ‘Very discouraging,’ ‘Somewhat discouraging,’ ‘A little discouraging,’
‘Not at all discouraging’).
Items respondents felt were missing:
•
•
•
•
•
•
•
•
•

Ability to obtain equipment
Lack of access to raw materials (could be considered ‘Costs too high’)
Overall economic conditions
Current administration (trade wars)
Public opinion
Health trends
Increasing minimum wage
Cost of healthcare
Insufficient time

Recommendation(s):
•

Change the measure from level of importance to level of discouragement:

During the three years 2016 to 2018, how discouraging were each of the following factors to this
business in conducting the following innovation activities?
‘Very discouraging,’ ‘Somewhat discouraging,’ ‘A little discouraging,’ ‘Not at all discouraging’
•

Consider adding extra categories:
o Ability to obtain equipment
o Lack of access to raw materials (could be considered ‘Costs too high’)
o Overall economic conditions
o Current administration (trade wars)
o Public opinion
o Health trends
o Increasing minimum wage
o Cost of healthcare
o Insufficient time

DRB Clearance Number: CBDRB-FY20-ESMD003-002

14

Attachment F

Finding #20: Business strategies
Most respondents felt that this was an easy question. Most respondents stated that they would also
reach out to the leadership of their companies. Most respondents preferred ‘extent’ to ‘importance.’
Most respondents had a reasonable definition of ‘extent’:
•
•
•
•
•
•
•
•
•

Impact
Effort
Resources used
Time spent
Staffing
Investment
Consideration
Trying hard
Degree

Recommendation(s):
• No recommendation needed.

Finding #21: Use of digital technologies
Most respondent understood the categories. However, some respondents found item ‘j’ to be
confusing. Some respondents stated that they would reach out to others in their companies such as
engineering or management. Most respondents did not know what ‘blockchain’ was. Respondents gave
varying definitions of ‘automation’:
•
•
•
•

Processes that don’t require a human to interact, something that runs in the background.
Creating a process or code to do task automatically.
A change from a manual calculation to an automatic one
Robotics, machine innovation, manufacturing facilities

When considering AI, one respondent stated, “Machine learning or actual robots? I would think about
machine learning.”
Recommendation(s):
•

•
•
•
•

Similar categories should be placed next to each other.
o Items ‘a’ and ‘f’ should be side by side, or combined into one category. If kept separate,
consider adding examples.
o Items ‘b’ and ‘d.’
o Items ‘c’ and ‘e.’
Remove ‘and value exchange’ from item ‘h’ as it was confusing.
Consider adding examples to items ‘i’ as it seems out of place.
Remove ‘blockchain’ from item ‘j,’ it was more confusing than informative.
See example of revisions below:

DRB Clearance Number: CBDRB-FY20-ESMD003-002

15

Attachment F

a) Computer infrastructure (server technologies)
b) Cloud computing
c) Artificial Intelligence
d) Automation
e) Internet-connected devices
f)

Mobile communication technologies

g) The use of digital technologies for collaboration
h) Communication (e.g. through social media)
i) Standardized goods and services (e.g., goods and
services are the same regardless of market or
customer type)
j) Digital technologies for planning and management
(e.g. enterprise resource planning, customer
relationship management) or distributed
ledgers

Finding #22: Capabilities for using digital technologies
Most respondents believed that they could answer this question and that all of the categories were
clear. Items ‘c’ and ‘e’ was confusing to some respondents. The term ‘appropriate’ made some
respondents wonder who would determine what was ‘appropriate.’
Recommendation(s):
•

Consider removing or changing the word ‘appropriate’ in items ‘c’ and ‘e.’

Finding #23: Technology capabilities
Most respondents could answer this question, and nearly all respondents said ‘yes’ to all items except
for ‘d’. One respondent wondered if trademarks would be included in item ‘b.’ One respondent was
unsure if item ‘d’ included software or not. Respondents gave the following definitions of technical
expertise:
•
•
•
•
•
•

Whether you know someone that knows or know how to do something yourself. Subject matter
experts that can actually move forward.
Have employees that understand necessary technology and can apply it to what they do.
Individuals who have the knowledge and capability to meet technology needs and can
implement.
Combination of internal and outsourced IT.
Know-how
Knowledge of how to take advantage of technology offerings.

DRB Clearance Number: CBDRB-FY20-ESMD003-002

16

Attachment F

Recommendation(s):
•
•
•
•

‘Embodied in objects’ is awkward and confusing in item ‘a.’ Change to ‘Acquired technology
products (machinery, equipment, software) from other firms or organizations.
Consider specifying both hardware and software in item ‘d.’
Consider specify trademarks in item ‘b.’
‘In house’ should be written as ‘in-house’ in item ‘d.’

DRB Clearance Number: CBDRB-FY20-ESMD003-002

17

Attachment F

Appendix A: Draft Questions, Round 1
SECTION C: PRODUCTS AND BUSINESS PROCESSES

The following section collects information on the business’s introduction of a new or
improved product (goods or services) or business process that differed significantly from the
business’s previous products or processes.
The products (goods or services) or business processes must have characteristics or intended
uses that are new or which provide a significant improvement over what was previously
used or sold by the business. However, they can fail or take time to prove themselves.
The products or business process need only be new or improved for the business. It could
have been originally developed or used by other businesses or organizations.
The following section asks about the three previous years including the calendar year 2018
instead of one year as in other sections of this questionnaire.

New or improved goods or services

During the three years 2016 to 2018, did this business introduce to the market any new or improved
goods or services that differed significantly from the business’s previous goods or services?

Select one for each row.

Yes

No

a. Goods. (Exclude the simple resale of new goods and changes
of a solely aesthetic nature.) A good is usually a tangible
object such as a smartphone, furniture, or packaged
software, but also includes digital goods such as
downloadable software, music and film.
b. Services. (Exclude the simple resale of new services.) A

service is usually intangible, such as retailing,
insurance, educational courses, air travel, consulting,
etc., and also includes digital services.

If ‘No’ is selected for a. and b., skip to ‘PROCESS INNOVATION’

DRB Clearance Number: CBDRB-FY20-ESMD003-002

18

Attachment F

NOVELTY OF New or improved goods or services
During the three years 2016 to 2018, did this business introduce any new or improved goods or services
that were:

Select one for each row.

Yes

No

New to the market? This business introduced a new or improved
product (good or service) that was not previously offered by any of
your competitors (it may have already been available in other
markets)
New only to this business? This business introduced a new or
improved product (good or service) that was identical or very similar
to products already offered by your competitors
New or improved goods or services AS PERCENT OF TOTAL SALES
Using the definitions in the previous question, what percentage of this business’s ‘A.11 DOMESTIC SALES
AND REVENUES’ in 2018 sales and revenues were attributable to or originated from domestic operations:

a. New or improved goods and services introduced during 2016 to 2018 that were

%

new to your market.
b. New or improved goods and services introduced during 2016 to 2018 that were

%

new only to this business.
c. Goods and services that were unchanged or only marginally modified during 2016

%

to 2018 (include the resale of new goods or services purchased from other companies

Total sales in 2018
100%

DRB Clearance Number: CBDRB-FY20-ESMD003-002

19

Attachment F

EXPECTATIONS

As of December 2018, did the new or improved products introduced during 2016 to 2018 meet your
business’s expectations?

o
o
o
o

Yes, expectations were exceeded
Yes, expectations were met
No, Expectations were not met
Too early to tell

SOURCES OF New or improved goods or services
Who developed these new or improved products (goods and services)? Select all that apply

This business by itself
This business together with other businesses, universities, research
institutes, nonprofits, or other organizations
This business by adapting or modifying products originally developed by
other businesses or organizations
Other businesses or organizations

DRB Clearance Number: CBDRB-FY20-ESMD003-002

20

Attachment F

PRODUCT COOPERATION PARTNERS

(This question should be asked only of respondents who selected option 2 or 4 of the above question.)

During the three years 2016 to 2018, did this business cooperate with any of the following partners in
developing new or improved goods or services?

Yes

No

a) Parent, affiliated, or subsidiary business
b) Suppliers of equipment, materials, components, or software
c) Clients or customers from the private sector
d) Clients or customers from the public sector
e) Competitors or other businesses in your industry
f)

Consultants and commercial laboratories

g) Universities, colleges or other higher education institutions
h) Government, public or private research institutes
i)

Non-profit organizations

j)

Households or individuals

DRB Clearance Number: CBDRB-FY20-ESMD003-002

21

Attachment F

Where were these businesses or organizations located? Select all that apply

[NOTE: Question below is populated only with yes responses from above]
United

Canada

Rest of

States

or Mexico

World

a) Parent, affiliated, or subsidiary business
b) Suppliers of equipment, materials, components, or software
c) Clients or customers from the private sector
d) Clients or customers from the public sector
e) Competitors or other businesses in your industry
f)

Consultants and commercial laboratories

g) Universities, colleges or other higher education institutions
h) Government, public or private research institutes
i)

Non-profit organizations

j)

Households or individuals

MOST IMPORTANT INNOVATION

Think about the most important good or service your company introduced during 2016 to 2018, was
that good or service new to the market or new only to your business?

Yes

No

New to the market - This business introduced a new or improved
product (good or service) that was not previously offered by any of
your competitors (it may have already been available in other
markets)
New only to this business - This business introduced a new or
improved product (good or service) that was identical or very similar
to products already offered by your competitors
DRB Clearance Number: CBDRB-FY20-ESMD003-002

22

Attachment F

MOST IMPORTANT INNOVATION AND SALES

What percentage of this business’s domestic sales and revenues in 2018 were attributed to your
company’s most important innovation?

%

NEW OR IMPROVED BUSINESS ProcessES
During the three years 2016 to 2018, did this business introduce any of the following types of new or
improved business processes that differ significantly from your previous business processes?

Yes

No

a. Methods for producing goods or providing services
(including methods for developing goods or services)
b. Logistics, delivery or distribution methods
c. Marketing methods for promotion, packaging, pricing, product
placement or after sales services
d. Information and communication systems
(including hardware, software and data processing)
e. Administration and management activities (including decision-making,
human resource management, and methods for
accounting or other administrative operations)
f.

Product and business process development activities (including
activities to identify, develop or adapt products or processes)

If ‘No’ is selected for a. – f., SKIP to ‘INNOVATION ACTIVITIES’

DRB Clearance Number: CBDRB-FY20-ESMD003-002

23

Attachment F

Improved BUSINESS Process Developers
Who developed these new or improved business processes? Select all that apply

This business by itself
This business together with other businesses, universities, research
institutes, nonprofits, or other organizations
This business by adapting or modifying business processes originally
developed by other businesses or organizations
Other businesses or organizations

DRB Clearance Number: CBDRB-FY20-ESMD003-002

24

Attachment F

Activities for new or improved products or business processes
During the three years 2016 to 2018, did this business have any of the following types of innovation
activities? Include all developmental, financial and commercial activities that were intended to result in
a new or improved product or business process that differed significantly from your previous products
or processes.
Yes

No

a. Research and development: creative and systematic work
undertaken in order to increase the stock of knowledge and to devise
new applications of available knowledge
b. Engineering and design activities: planning of technical
specifications, testing, evaluation, setup and pre-production for
goods, services, processes or systems; installing equipment, toolingup, testing, trials and user demonstrations; and activities to extract
knowledge or design information from existing products or process
equipment. Also include activities to develop a new or modified
function, form or appearance for goods, services, or processes
c. Marketing and brand equity activities: include market research,
market testing, methods for pricing, product placement and product
promotion; product advertising, the promotion of products at trade
fairs or exhibitions, and the development of marketing strategies
d. Intellectual Property (IP) related activities: protection or exploitation
of knowledge, often created through R&D, software development,
and engineering, design and other creative work, including all related
administrative and legal work
e. Employee training: activities that are paid for or subsidized by the
firm to develop knowledge and skills required for the specific trade,
occupation or vocation of a firm’s employees
f.

Software development and database activities: in-house
development and purchase of computer software; collection and
analysis of data in proprietary computer databases and data obtained
from publicly-available reports or the Internet; and activities to
upgrade or expand the functions of IT systems, including computer
program and databases

g. Acquisition of machinery, equipment and other tangible assets
h. Management related to innovation: Activities to plan, govern and
control internal and external resources
DRB Clearance Number: CBDRB-FY20-ESMD003-002

25

Attachment F

If ‘No’ is selected for a.-h. AND total NUMBER OF W-2 PAID EMPLOYEES OR EMPLOYEE/OWNERS from
question ‘NUMBER OF W-2 PAID EMPLOYEES OR EMPLOYEE/OWNERS’ is between 1 and 9, skip to
SECTION D: RESEARCH AND DEVELOPMENT.

If ‘No’ is selected for a.-h. AND total NUMBER OF W-2 PAID EMPLOYEES OR EMPLOYEE/OWNERS from
question ‘NUMBER OF W-2 PAID EMPLOYEES OR EMPLOYEE/OWNERS’ is greater than or equal to 10,
skip to SECTION E: TECHNOLOGY AND INTELLECTUAL PROPERTY.
ACTIVITY COSTS
How much did this business spend on the innovation activities selected in the previous question in
2018? Note that this question refers only to the year 2018 and not 2016 – 2018.

$Mil. Thou. Dol.

,000

If yes is selected for a. from question ‘INNOVATION ACTIVITIES’ AND total NUMBER OF W-2 PAID
EMPLOYEES OR EMPLOYEE/OWNERS from question ‘NUMBER OF W-2 PAID EMPLOYEES OR
EMPLOYEE/OWNERS’ is greater than or equal to 10 AND response to ‘INNOVATION COST’ does not
equal zero, continue to ‘R&D INNOVATION COSTS’.

If response to ‘INNOVATION COST’ equals zero AND total NUMBER OF W-2 PAID EMPLOYEES OR
EMPLOYEE/OWNERS from question ‘NUMBER OF W-2 PAID EMPLOYEES OR EMPLOYEE/OWNERS’ is
between 1 and 9, skip to Section D: Research and Development.
If response to ‘INNOVATION COST’ equals zero AND total NUMBER OF W-2 PAID EMPLOYEES OR
EMPLOYEE/OWNERS from question ‘NUMBER OF W-2 PAID EMPLOYEES OR EMPLOYEE/OWNERS’ is
greater than or equal to 10, skip to SECTION E: TECHNOLOGY AND INTELLECTUAL PROPERTY.

R&D FOR ACTIVITY COSTS
Of the ‘ACTIVITY COSTS’ reported, how much was for research and development (creative and
systematic work undertaken in order to increase the stock of knowledge and to devise new applications
of available knowledge) in 2018?

$Mil. Thou. Dol.

,000
DRB Clearance Number: CBDRB-FY20-ESMD003-002

26

Attachment F

GOVERNMENT SUPPORT

During the three years 2016 to 2018, did this business use any of the following types of government
programs (federal, state, or local) to aid the following innovation activities?

•
•
•
•
•
•
•
•

Research and development
Engineering and design activities
Marketing and brand equity activities
Intellectual Property (IP) related activities
Employee training
Software development and database activities
Acquisition of machinery, equipment and other tangible assets
Management related to innovation
Yes

No

a. Government tax incentive or tax credit programs: Include programs intended
to support innovation activities such as research and development or capital
expenditures
b. Government grants and contributions programs: Include programs intended
to support innovation activities such as research and development, business
development or commercialization of intellectual property
c. Government training and hiring programs: Include programs intended to
support innovation activities such as hiring and training of researchers, interns
or other personnel
d. Government procurement: Include programs intended to support innovation
activities such as the procurement of new or improved goods, services or
businesses processes
e. Other government programs: Include programs and activities not included
elsewhere such as access to facilities, export incentives, technical assistance,
market information or loans

DRB Clearance Number: CBDRB-FY20-ESMD003-002

27

Attachment F

DRIVERS OF INNOVATION

During the three years 2016 to 2018, how important were each of the following factors in encouraging
this business to conduct the following innovation activities:

•
•
•
•
•
•
•
•

Research and development
Engineering and design activities
Marketing and brand equity activities
Intellectual Property (IP) related activities
Employee training
Software development and database activities
Acquisition of machinery, equipment and other tangible assets
Management related to innovation

Select one for each row
Very
important

Somewhat
important

A little
important

Not at all
important

MARKETS
Domestic customers
Access to international markets
Suppliers and value chains
Availability or cost of skills
Availability or cost of finance
Competitors
Technical or Voluntary Consensus Standards
Markets for knowledge
Digital platforms
PUBLIC POLICY
Regulations
Functioning of courts and rules enforcement
Taxation
Public spending (level and priorities)

DRB Clearance Number: CBDRB-FY20-ESMD003-002

28

Attachment F

Government support for innovation
Government demand for innovation
Public infrastructure
General policy stability
SOCIETY
Consumer responsiveness to innovation
Favorable public opinion towards innovation
Level of trust among economic actors

DRB Clearance Number: CBDRB-FY20-ESMD003-002

29

Attachment F

BARRIERS OF INNOVATION

During the three years 2016 to 2018, how important were each of the following factors in discouraging
this business to conduct the following innovation activities:

•
•
•
•
•
•
•
•

Research and development
Engineering and design activities
Marketing and brand equity activities
Intellectual Property (IP) related activities
Employee training
Software development and database activities
Acquisition of machinery, equipment and other tangible assets
Management related to innovation

Select one for each row
Very
important

Somewhat
important

A little
important

Not at all
important

Lack of internal finance for innovation
Lack of credit or private equity
Difficulties in obtaining public grants or
subsidies
Costs too high
Lack of skilled employees within your
enterprise
Lack of collaboration partners
Lack of access to external knowledge
Uncertain market demand for your ideas
Too much competition in your market
Different priorities within your enterprise

DRB Clearance Number: CBDRB-FY20-ESMD003-002

30

Attachment F

BUSINESS STRATEGIES

During 2016 to 2018, to what extent did the business focus on the following business strategies:
Select one for each row
A great
extent

To some
extent

To a small
extent

Not at all

Focusing on improving existing goods or services
Focusing on introducing new goods or services
Focusing on reducing prices for its goods and services
Focusing on increasing the quality of its goods and services
Focusing on broad range of products
Focusing on one or a small number of key products
Focusing on satisfying established customer groups
Focusing on reaching out to new customer groups
Focusing on standardized products
Focusing on customer-specific solutions

DRB Clearance Number: CBDRB-FY20-ESMD003-002

31

Attachment F

USE OF DIGITAL TECHNOLOGIES

To what extent does your company use the following Digital Technologies for innovation activities?
Select one for each row
A great
extent

To some
extent

To a small
extent

Not at all

Computer infrastructure (server technologies)
Artificial Intelligence
Internet-connected devices
Automation
Mobile communication technologies
Cloud computing
The use of digital technologies for collaboration
Communication and value exchange (e.g. through social
media)
Digital technologies for planning and management (e.g.
enterprise resource planning, customer relationship
management) or distributed ledgers (blockchain)

DRB Clearance Number: CBDRB-FY20-ESMD003-002

32

Attachment F

CAPABILITES FOR USING DIGITAL TECHNOLOGIES

To what extent does your company use any of the following for innovation activities?
Select one for each row
To a great
extent

To some
extent

To a small
extent

Not at all

Digital integration within and across different business
functions
Access to and ability to use data analytics to design,
develop, commercialize and improve products, including
data about the users of the firm’s products and their
interactions with such products
Access to networks and the use of appropriate solutions
and architectures (hardware and software)
Effective management of privacy and cybersecurity risks
Adoption of appropriate business models for digital
environments, such as e-commerce, participative
platforms, etc.

DRB Clearance Number: CBDRB-FY20-ESMD003-002

33

Attachment F

TECHNOLOGY CAPABILITIES
(NOTE: This section currently consists of concepts that have not been translated into questions yet.
Feedback from the cognitive interviews will inform the question structure and wording)
Technical expertise consists of a firm’s knowledge of and ability to use technology. This knowledge is
derived from the skills and qualifications of its employees, including its engineering and technical
workforce, accumulated experience in using the technology, the use of capital goods containing the
technology, and control over the relevant IP.
•
•
•
•

acquiring technology embodied in objects (machinery, equipment, software) from other firms or
organizations
acquiring IP rights that give ownership, exclusion rights or rights to use technical knowledge
modifying or adapting existing technology to the firm’s specific needs
developing new technology in house

Design capabilities are difficult to define in a way that is consistently understood by all types of firms
across different countries. For the purposes of this form, design is defined as an innovation activity
“aimed at planning and designing procedures, technical specifications and other user and functional
characteristics for new products and business processes.”
•
•
•

engineering design, including technical specifications, tooling up and prototype construction
product design that determines the shape, color or pattern of objects, the interface between
software and users, or the user experience of services
design thinking, which is a systematic methodology for approaching the design of a good,
service or system

Capabilities related to digital technologies and data analytics are part of a firm's technical expertise.
These are specifically singled out because of the enabling, general purpose nature of digital technologies
and data analytics.

•
•
•
•
•
•
•

divergent idea generation or brainstorming
techniques to develop an understanding of the customer experience, particularly ethnographic
field research methods (observing how people use a product in real-world environments,
developing an empathetic understanding of what users want in a product, etc.)
co-design or co-creation (involvement of potential users in generating design concepts)
prototyping and testing
feedback from sales or marketing personnel
evaluation of user-initiated reports of their experiences with a product (social media, online
reviews and comments, etc.)
structured data collection (feedback forms, dedicated user surveys, focus groups)

DRB Clearance Number: CBDRB-FY20-ESMD003-002

34

Attachment F

Appendix B: Interview Protocol, Round 1
Cognitive Interviewing Guide: INNOVATION MODULE for ABS (3rd Year)
Respondent’s company should have been identified as an innovator and confirmed.
Boxes indicate questions being evaluated by following probes.
Interviewer notes are in red.
Introduction:
•
•
•
•
•

Explain purpose of meeting: to obtain feedback on reporting to the ABS, to understand the
process of answering questions, and to evaluate how new questions work or don’t work.
Some questions will ask may seem odd and/or obvious, but we don’t want to assume we know
what you are thinking.
Explain that we are not testing the respondent – we only want to evaluate the questionnaire
Structure of meeting: Understand your business and your role in it, then review the
questionnaire and ask you questions as you complete it.
Permission to record discussion for note taking purposes? This study is being conducted under
the authority of Title 13 USC. We plan to use your feedback to improve the design and layout of
the form for future data collections. Only staff involved in this product design research will have
access to the recording. Have R sign consent form.

Research Questions to Address:
•
•
•

Are respondents able to answer the questions as intended?
Are there any items that cause the respondents problems?
o What can be done to fix those problems?
Are there any order/context effects?

Before the Questionnaire:
•
•
•
•
•

What types of goods or services does this business provide?
What is your role in the company? What kind of responsibilities do you have?
Are you one of the owners of the business?
Have you completed other surveys from the Census Bureau or other agencies? If so, which
ones?
What was your role in the process for responding to this questionnaire? (Gather data? Enter
data? Consult with data providers? Etc.)

DRB Clearance Number: CBDRB-FY20-ESMD003-002

35

Attachment F

REVISED INNOVATION MODULE for ABS (3rd Year)
Provide ‘Innovation Handout 1’ to participant.
SECTION C: PRODUCTS AND BUSINESS PROCESSES

The following section collects information on the business’s introduction of a new or
improved product (goods or services) or business process that differed significantly from
the business’s previous products or processes.
The products (goods or services) or business processes must have characteristics or
intended uses that are new or which provide a significant improvement over what was
previously used or sold by the business. However, they can fail or take time to prove
themselves.
The products or business process need only be new or improved for the business. It could
have been originally developed or used by other businesses or organizations.
The following section asks about the three previous years including the calendar year 2018 instead of
one year as in other sections of this questionnaire.

•

•

Looking at the title and instructions, does this look like a section that you would
answer?
o If no: Who would be the most appropriate person to answer these questions?
Based on the instructions you just read, did your company have new or improved
goods or services?
o Were these goods or services new to the firm? Were they new to the
industry?
o Please describe those goods or services.

•

What does ‘differed significantly’ mean to you in this context?

•

Now, we’re going to skip past some of these beginning questions in the interest of
time. They may look familiar as they are already on the form, and we want to focus
on the new draft questions.

DRB Clearance Number: CBDRB-FY20-ESMD003-002

36

Attachment F

EXPECTATIONS

As of December 2018, did the new or improved products introduced during 2016 to 2018 meet your
business’s expectations?

o
o
o
o
•
•
•

•
•

•

Yes, expectations were exceeded
Yes, expectations were met
No, Expectations were not met
Too early to tell

How would you answer this question?
How easy or difficult is it to recall the 2016-2018 time frame?
What does “meet your business’s expectations” mean to you? How would you or your company
determine if a product had met expectations?
o Examples if needed: increased revenue, increased efficiency, increased brand
recognition, etc…
Would someone else in your company need to make this determination? If so, who?
If you had more than one new or improved products or services, how would you determine your
answer?
o What if those products/services had performed differently, such as one failing and one
succeeding?
Do you have documentation to track expectations for your new or improved products/services?
o What information is included in that documentation?
o Is this documentation something you would have access to?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

37

Attachment F

SOURCES OF New or improved goods or services
Who developed these new or improved products (goods and services)? Select all that apply

This business by itself
This business together with other businesses, universities, research
institutes, nonprofits, or other organizations
This business by adapting or modifying products originally developed by
other businesses or organizations
Other businesses or organizations

•
•
•

How would you answer this question?
What does ‘other businesses or organizations’ mean to you?
o Can you give me an example?
If respondent answers with option 2:
o Can you describe that development process you went through?
o What type of organization did you work with?
o Were you the leading partner?
o Can you describe how your organizations cooperated? How was the workload
distributed?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

38

Attachment F

PRODUCT COOPERATION PARTNERS

(This question should be asked only of respondents who selected option 2 or 4 of the above question.)

During the three years 2016 to 2018, did this business cooperate with any of the following partners in
developing new or improved goods or services?

Yes

No

k) Parent, affiliated, or subsidiary business
l)

Suppliers of equipment, materials, components, or software

m) Clients or customers from the private sector
n) Clients or customers from the public sector
o) Competitors or other businesses in your industry
p) Consultants and commercial laboratories
q) Universities, colleges or other higher education institutions
r) Government, public or private research institutes
s) Non-profit organizations
t)

Households or individuals

•

How would you answer this question?
o If participant answered yes to any categories: Could you give some examples?
o If participant answered no to all categories: What kinds of businesses or organizations
were you thinking of when you answered the previous question?
 How would you describe them?
Were there any categories that were unclear?
Is this something you would have in your records?

•
•

DRB Clearance Number: CBDRB-FY20-ESMD003-002

39

Attachment F

Where were these businesses or organizations located? Select all that apply

[NOTE: Question below is populated only with yes responses from above]
a) Parent, affiliated, or
subsidiary business
b) Suppliers of equipment,
materials, components, or
software
c) Clients or customers from
the private sector
d) Clients or customers from
the public sector
e) Competitors or other
businesses in your industry
f) Consultants and
commercial laboratories
g) Universities, colleges or
other higher education
institutions
h) Government, public or
private research institutes
i) Non-profit organizations
j) Households or individuals
•
•
•
•

United States

Canada or Mexico

Rest of World

How would you answer this question?
If you worked with the domestic location of a foreign-owned company, how would you report
that?
What does “rest of world” mean to you? Is there some other phrase that you think would be
better?
Is this something you would have in your records?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

40

Attachment F

MOST IMPORTANT INNOVATION

Think about the most important good or service your company introduced during 2016 to 2018, was
that good or service new to the market or new only to your business?

Yes

No

New to the market - This business introduced a new or improved
product (good or service) that was not previously offered by any of
your competitors (it may have already been available in other
markets)
New only to this business - This business introduced a new or
improved product (good or service) that was identical or very similar
to products already offered by your competitors

•
•
•

•

How would you answer this question?
Did your company have multiple innovations during this time frame?
What does ‘most important’ mean to you or your company?
o What if the question asked you to ‘think about the good or service your company
introduced during 2016 to 2018 that accounts for the most revenue’?
o Would your answer change?
Is this something that would be formally documented somewhere?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

41

Attachment F

MOST IMPORTANT INNOVATION AND SALES

What percentage of this business’s domestic sales and revenues in 2018 were attributed to your
company’s most important innovation?

%

•
•
•

How would you answer this question?
o How would you calculate that answer?
Is this something you would have in your records?
Up to now, we’ve asked you to think about an innovation that was introduced between
2016 and 2018, but this question asks only about the revenue reported in 2018. Did you
consider this when formulating your answer?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

42

Attachment F

GOVERNMENT SUPPORT

During the three years 2016 to 2018, did this business use any of the following types of government
programs (federal, state, or local) to aid the following innovation activities?

•
•
•
•
•
•
•
•

Research and development
Engineering and design activities
Marketing and brand equity activities
Intellectual Property (IP) related activities
Employee training
Software development and database activities
Acquisition of machinery, equipment and other tangible assets
Management related to innovation

Government tax incentive or tax credit programs

Yes

No

Include programs intended to support innovation activities such as research and development or
capital expenditures.

Government grants and contributions programs

Include programs intended to support innovation activities such as research and development,
business development or commercialization of intellectual property.

Government training and hiring programs

Include programs intended to support innovation activities such as hiring and training of researchers,
interns or other personnel.

Government procurement

Include programs intended to support innovation activities such as the procurement of new or
improved goods, services or businesses processes.

Other government programs

Include programs and activities not included elsewhere such as access to facilities, export incentives,
technical assistance, market information or loans.

•
•
•
•
•
•

How did you decide on your answers to these items?
Did you read the descriptions of the various categories? Were they helpful or unhelpful in
determining your answer?
Who would know whether or not your company took part in any of these programs?
How easy of difficult was it to answer this question thinking only of your company’s innovation
activities?
If yes to any: can you tell me about the process of applying for that [support]?
o Did you have to submit a proposal? Do you know which agency you had to work with?
o Can you describe the type/amount of support provided?
Is this something you would have in your records?
DRB Clearance Number: CBDRB-FY20-ESMD003-002

43

Attachment F

BARRIERS and DRIVERS

•
•
•
•
•

Thinking about the innovation activities we’ve been discussing, what would you say are the most
important things that encourage (or allow) your company to innovate?
What are the things that may have discouraged your company from innovating?
Are these factors formally listed or documented anywhere?
If we provided you with a list of encouraging and discouraging factors, would you be able to rate
each one based on importance?
Provide ‘Drivers and Barriers of Innovation Handout’ to participant.
o Looking at this list, did any of these factors affect your company’s innovation activities?
o Do the headings for the drivers mean anything to you?
 What does mean ‘market factors’ mean to you?
 How about Public policy factors? Societal factors?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

44

Attachment F

BUSINESS STRATEGIES

During 2016 to 2018, to what extent did the business focus on the following business strategies:

Focusing on improving existing goods or services
Focusing on introducing new goods or services
Focusing on reducing prices for its goods and services
Focusing on increasing the quality of its goods and services
Focusing on broad range of products
Focusing on one or a small number of key products
Focusing on satisfying established customer groups
Focusing on reaching out to new customer groups
Focusing on standardized products
Focusing on customer-specific solutions

•
•
•
•
•
•
•
•
•
•

A great
extent

To some
extent

To a small
extent

No

Provide the ‘Business Strategies’ questions version A or B (alternating versions for each other
participant)
How would you answer this question?
Would you need to consult with anyone else in your company to answer these questions?
Is this something you would have documented somewhere?
What does ‘extent’ mean to you in this context?
What if the questions were asking you to rate the ‘importance’ of each of these focuses to your
business? Would that be easier or more difficult to answer?
What if these were phrased as ‘yes or no’ questions? For example:
o During 2016 to 2018, did this business have a strategy in place to improve its existing
goods and services? Yes or No
Do you prefer this format of [individual questions/a table] or would you prefer this format?
Why? Provide the other version for comparison.
In general, when answering a series of questions like this, do you prefer individual questions or a
table?
Thinking about all of the questions we’ve gone over so far today, how long do you think it would
take to complete these questions?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

45

Attachment F

We’re interested in how companies are digitizing, or utilizing digital technologies.
•
•
•
•

Can you tell me what digitalization means to you?
Does your company use any digital technologies? Can you give me some examples?
Does your company have any documentation about how these digital technologies are used?
Does your company have any documented policies regarding digital technologies?

Now, I’d like to do an activity. I have here a stack of cards. Each card has an item related to
digital technology on it. These cards are in no particular order. Please take a few minutes to
read the cards and put them in groups you feel make the most sense. Please talk through your
thought process as you arrange the cards. There are no right or wrong answers. (Give the
participant the ‘Digital Technologies Cards.’ Note the order in which the respondent lists the
cards.)
• For each group the participant creates: How would you describe this group? How would
you label it?
• Do you feel like there are any technologies or concepts missing from these cards?
• Are there any cards that you feel do not belong?
• Probe on any cards participant had difficulty placing.
• Looking at the items on the cards, would you or someone in your company know which
of these technologies this company used for innovation purposes?
• Is this something that would be in your records?
•
•
•

Can you think of any other digital technologies your company uses for innovation activities?
Would you or someone in your company be able to determine to what extent your company
utilized these technologies for innovation activities?
Would you or someone in your company be able to determine how important these
technologies were for your company’s innovation activities?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

46

Attachment F

I’m going to show you a few lists of concepts and ask you some questions about them
•
•
•
•
•
•
•
•
•
•

Give participant the first ‘Technologies Capabilities List.’
Take your time to read these and let me know when you’re done.
Please describe this list of activities in your own words.
o If needed: how would you label this list?
Does your company engage in any of these activities?
Would this be in your records?
Repeat for lists two and three.
After participant has gone over all three lists:
Would you be able to determine the importance of each of these activities to your company’s
ability to innovate separately?
How would you define ‘technical expertise’ as it applies to your company? Can you give me
examples?
How would you define ‘design capabilities’ as it applies to you company? Can you give
examples?

WRAP UP
•
•

In general, are the innovation activities we covered today something that your company
formally documents?
Do you have any other comments of suggestions about anything we have discussed today?

THANK YOU FOR YOUR TIME!

DRB Clearance Number: CBDRB-FY20-ESMD003-002

47

Attachment F

Appendix C: Draft Questions, Round 2
SECTION C: GOODS, SERVICES, AND BUSINESS PROCESSES

The following section collects information on the business’s introduction of a new or
improved goods, services, or business process that differed significantly from the business’s
previous goods, services, or processes.
The goods, services, or business processes must have characteristics or intended uses that
are new or which provide a significant improvement over what was previously used or sold
by the business. However, they can fail or take time to prove themselves.
The goods, services, or business process need only be new or improved for the business. It
could have been originally developed or used by other businesses or organizations.
The following section asks about the three previous years including the calendar year 2018
instead of one year as in other sections of this questionnaire.

New or improved goods or services

During the three years 2016 to 2018, did this business introduce to the market any new or improved
goods or services that differed significantly from the business’s previous goods or services?

Select one for each row.

Yes

No

c. Goods. (Exclude the simple resale of new goods and changes
of a solely aesthetic nature.) A good is usually a tangible
object such as a smartphone, furniture, or packaged
software, but also includes digital goods such as
downloadable software, music and film.
d. Services. (Exclude the simple resale of new services.) A

service is usually intangible, such as retailing,
insurance, educational courses, air travel, consulting,
etc., and also includes digital services.

If ‘No’ is selected for a. and b., skip to ‘PROCESS INNOVATION’

DRB Clearance Number: CBDRB-FY20-ESMD003-002

48

Attachment F

NOVELTY OF New or improved goods or services
During the three years 2016 to 2018, did this business introduce any new or improved goods or services
that were:

Select one for each row.

Yes

No

New to the market? This business introduced a new or improved
product (good or service) that was not previously offered by any of
your competitors (it may have already been available in other
markets)
New only to this business? This business introduced a new or
improved product (good or service) that was identical or very similar
to products already offered by your competitors

New or improved goods or services AS PERCENT OF TOTAL SALES
Using the definitions in the previous question, what percentage of this business’s ‘A.11 DOMESTIC SALES
AND REVENUES’ in 2018 sales and revenues were attributable to or originated from domestic operations:

a. New or improved goods and services introduced during 2016 to 2018 that were

%

new to your market.
b. New or improved goods and services introduced during 2016 to 2018 that were

%

new only to this business.
c. Goods and services that were unchanged or only marginally modified during 2016

%

to 2018 (include the resale of new goods or services purchased from other companies

Total sales in 2018
100%

DRB Clearance Number: CBDRB-FY20-ESMD003-002

49

Attachment F

EXPECTATIONS

Did the new or improved goods or services introduced during 2016 to 2018 meet this business’s
expectations as of December 2018?

o
o
o
o

Yes, expectations were exceeded
Yes, expectations were met
No, Expectations were not met
Too early to tell

SOURCES OF New or improved goods or services
Who developed these new or improved goods and services? Select all that apply

This business by itself
This business together with other businesses, universities, research
institutes, nonprofits, or other organizations
This business by adapting or modifying products originally developed by
other businesses or organizations
Other businesses or organizations

DRB Clearance Number: CBDRB-FY20-ESMD003-002

50

Attachment F

PRODUCT COOPERATION PARTNERS

During the three years 2016 to 2018, did this business cooperate with any of the following partners in
developing new or improved goods or services?

Yes

No

u) Parent, affiliated, or subsidiary business
v) Suppliers of equipment, materials, components, or software
w) Clients or customers from the private sector
x) Clients or customers from the public sector
y) Competitors or other businesses in your industry
z) Consultants and commercial laboratories
aa) Universities, colleges or other higher education institutions
bb) Government, public or private research institutes
cc) Non-profit organizations
dd) Households or individuals

DRB Clearance Number: CBDRB-FY20-ESMD003-002

51

Attachment F

Where were these businesses or organizations located? Select all that apply

[NOTE: Question below is populated only with yes responses from above]
United

Canada

Rest of

States

or Mexico

World

k) Parent, affiliated, or subsidiary business
l)

Suppliers of equipment, materials, components, or software

m) Clients or customers from the private sector
n) Clients or customers from the public sector
o) Competitors or other businesses in your industry
p) Consultants and commercial laboratories
q) Universities, colleges or other higher education institutions
r) Government, public or private research institutes
s) Non-profit organizations
t)

Households or individuals

MOST IMPORTANT INNOVATION

Think about the most important good or service this business introduced during 2016 to 2018, was that
good or service new to the market or new only to this business?

o New to the market - This business introduced a new or improved
good or service that was not previously offered by any of your
competitors (it may have already been available in other markets)
o New only to this business - This business introduced a new or
improved good or service that was identical or very similar to
products already offered by your competitors

DRB Clearance Number: CBDRB-FY20-ESMD003-002

52

Attachment F

MOST IMPORTANT INNOVATION AND SALES

In 2018, what percentage of this business’s domestic sales and revenues were attributed to this
business’s most important innovation?

%

NEW OR IMPROVED BUSINESS ProcessES
During the three years 2016 to 2018, did this business introduce any of the following types of new or
improved business processes that differ significantly from your previous business processes?

Yes

No

g. Methods for producing goods or providing services
(including methods for developing goods or services)
h. Logistics, delivery or distribution methods
i. Marketing methods for promotion, packaging, pricing, product
placement or after sales services
j.

Information and communication systems
(including hardware, software and data processing)

k. Administration and management activities (including decision-making,
human resource management, and methods for
accounting or other administrative operations)
l.

Product and business process development activities (including
activities to identify, develop or adapt products or processes)

If ‘No’ is selected for a. – f., SKIP to ‘INNOVATION ACTIVITIES’

DRB Clearance Number: CBDRB-FY20-ESMD003-002

53

Attachment F

Improved BUSINESS Process Developers
Who developed these new or improved business processes? Select all that apply

This business by itself
This business together with other businesses, universities, research
institutes, nonprofits, or other organizations
This business by adapting or modifying business processes originally
developed by other businesses or organizations
Other businesses or organizations

DRB Clearance Number: CBDRB-FY20-ESMD003-002

54

Attachment F

Activities for new or improved GOODS, SERVICES, or business processes
During the three years 2016 to 2018, did this business have any of the following types of innovation
activities? Include all developmental, financial and commercial activities that were intended to result in
a new or improved product or business process that differed significantly from your previous products
or processes.
Yes

No

a. Research and development: creative and systematic work
undertaken in order to increase the stock of knowledge and to devise
new applications of available knowledge
b. Engineering and design activities: planning of technical
specifications, testing, evaluation, setup and pre-production for
goods, services, processes or systems; installing equipment, toolingup, testing, trials and user demonstrations; and activities to extract
knowledge or design information from existing products or process
equipment. Also include activities to develop a new or modified
function, form or appearance for goods, services, or processes
c. Marketing and brand equity activities: include market research,
market testing, methods for pricing, product placement and product
promotion; product advertising, the promotion of products at trade
fairs or exhibitions, and the development of marketing strategies
d. Intellectual Property (IP) related activities: protection or exploitation
of knowledge, often created through R&D, software development,
and engineering, design and other creative work, including all related
administrative and legal work
e. Employee training: activities that are paid for or subsidized by the
firm to develop knowledge and skills required for the specific trade,
occupation or vocation of a firm’s employees
f.

Software development and database activities: in-house
development and purchase of computer software; collection and
analysis of data in proprietary computer databases and data obtained
from publicly-available reports or the Internet; and activities to
upgrade or expand the functions of IT systems, including computer
program and databases

g. Acquisition of machinery, equipment and other tangible assets
h. Management related to innovation: Activities to plan, govern and
control internal and external resources
DRB Clearance Number: CBDRB-FY20-ESMD003-002

55

Attachment F

ACTIVITY COSTS

How much did this business spend on the innovation activities selected in the previous question in
2018? Note that this question refers only to the year 2018 and not 2016 – 2018.

$Mil. Thou. Dol.

,000

Answer the question below only if option ‘a’ was selected in the activities question.

R&D FOR ACTIVITY COSTS
Of the ‘ACTIVITY COSTS’ reported, how much was for research and development (creative and
systematic work undertaken in order to increase the stock of knowledge and to devise new applications
of available knowledge) in 2018?

$Mil. Thou. Dol.

,000

DRB Clearance Number: CBDRB-FY20-ESMD003-002

56

Attachment F

GOVERNMENT SUPPORT

During the three years 2016 to 2018, did this business use any of the following types of government
programs (federal, state, or local) to aid the following innovation activities?

•
•
•
•
•
•
•
•

Research and development
Engineering and design activities
Marketing and brand equity activities
Intellectual Property (IP) related activities
Employee training
Software development and database activities
Acquisition of machinery, equipment and other tangible assets
Management related to innovation
Yes

No

Government tax incentive or tax credit programs: Include programs intended
to support innovation activities such as research and development or capital
expenditures
g. Government grants and contributions programs: Include programs intended
to support innovation activities such as research and development, business
development or commercialization of intellectual property
h. Government training and hiring programs: Include programs intended to
support innovation activities such as hiring and training of researchers, interns
or other personnel
i. Government procurement: Include programs intended to support innovation
activities such as the procurement of new or improved goods, services or
businesses processes
j. Other government programs: Include programs and activities not included
elsewhere such as access to facilities, export incentives, technical assistance,
market information or loans
f.

DRB Clearance Number: CBDRB-FY20-ESMD003-002

57

Attachment F

BARRIERS OF INNOVATION

During the three years 2016 to 2018, how important were each of the following factors in discouraging
this business to conduct the following innovation activities:

•
•
•
•
•
•
•
•

Research and development
Engineering and design activities
Marketing and brand equity activities
Intellectual Property (IP) related activities
Employee training
Software development and database activities
Acquisition of machinery, equipment and other tangible assets
Management related to innovation

Select one for each row.

Very
important

Somewhat
important

A little
important

Not at all
important

a. Lack of internal finance for innovation
b. Lack of credit or private equity
c. Difficulties in obtaining public grants or subsidies
d. Costs too high
e. Lack of skilled employees within this business
f.

Lack of collaboration partners

g. Lack of access to external knowledge
h. Uncertain market demand for your ideas
i.

Too much competition in your market

j.

Different priorities within this business

k. Government Regulations

DRB Clearance Number: CBDRB-FY20-ESMD003-002

58

Attachment F

BUSINESS STRATEGIES

During 2016 to 2018, to what extent did the business focus on the following business strategies?

Select one for each row

a. Improve existing goods or services

A great
extent

To some
extent

To a small
extent

Not at all

b. Introduce new goods or services
c. Reduce prices for goods and services
d. Increase the quality of goods and services
e. Develop a broad range of products
f.

Develop one or a small number of key products

g. Satisfy established customers
h. Reach out to new customers
i.
j.

Develop standardized goods and services (e.g., goods and
services are the same regardless of market or
customer type)
Develop customer-specific solutions
(e.g., customized goods and services)

DRB Clearance Number: CBDRB-FY20-ESMD003-002

59

Attachment F

USE OF DIGITAL TECHNOLOGIES

During 2016 to 2018, to what extent does this business use the following digital technologies for
innovation activities?

Select one for each row

a. Computer infrastructure (server technologies)

A great
extent

To some
extent

To a small
extent

Not at all

b. Artificial Intelligence
c. Internet-connected devices
d. Automation
e. Mobile communication technologies
f.

Cloud computing

g. The use of digital technologies for collaboration
h. Communication and value exchange
(e.g. through social media)
i. Standardized goods and services (e.g., goods and
services are the same regardless of market or
customer type)
j. Digital technologies for planning and management
(e.g. enterprise resource planning, customer
relationship management) or distributed
ledgers (blockchain)

DRB Clearance Number: CBDRB-FY20-ESMD003-002

60

Attachment F

CAPABILITES FOR USING DIGITAL TECHNOLOGIES

During 2016 to 2018, to what extent does this business use any of the following for innovation
activities?

Select one for each row

a. Digital integration within and across different business
functions
b. Access to and ability to use data analytics to design,
develop, commercialize and improve products, including
data about the users of the firm’s products and their
interactions with such products
c. Access to networks and the use of appropriate solutions
and architectures (hardware and software)
d. Effective management of privacy and cybersecurity risks
e. Adoption of appropriate business models for digital
environments, such as e-commerce, participative
platforms, etc.

A great
extent

To some
extent

DRB Clearance Number: CBDRB-FY20-ESMD003-002

To a small
extent

Not at all

61

Attachment F

TECHNOLOGY CAPABILITIES

During 2016 to 2018, did this business obtain the necessary technical expertise for its innovation
activities through any of the following means?

Select one for each row.

a. Acquired technology embodied in objects (machinery,
equipment, software) from other firms or organizations
b. Acquired intellectual property (IP) rights that give
ownership, exclusion rights or rights to use technical
knowledge
c. Modified or adapted existing technology to the firm’s
specific needs
d. Developed new technology in house

Yes

DRB Clearance Number: CBDRB-FY20-ESMD003-002

No

62

Attachment F

Appendix D: Interview Protocol, Round 2
Cognitive Interviewing Guide: INNOVATION MODULE for ABS (3rd Year)
Respondent’s company should have been identified as an innovator and confirmed.
Boxes indicate questions being evaluated by following probes.
Interviewer notes are in red.
Introduction:
•
•
•
•
•

Explain purpose of meeting: to obtain feedback on reporting to the ABS, to understand the
process of answering questions, and to evaluate how new questions work or don’t work.
Some questions will ask may seem odd and/or obvious, but we don’t want to assume we know
what you are thinking.
Explain that we are not testing the respondent – we only want to evaluate the questionnaire
Structure of meeting: Understand your business and your role in it, then review the
questionnaire and ask you questions as you complete it.
Permission to record discussion for note taking purposes? This study is being conducted under
the authority of Title 13 USC. We plan to use your feedback to improve the design and layout of
the form for future data collections. Only staff involved in this product design research will have
access to the recording. Have R sign consent form.

Research Questions to Address:
•
•
•

Are respondents able to answer the questions as intended?
Are there any items that cause the respondents problems?
o What can be done to fix those problems?
Are there any order/context effects?

Before the Questionnaire:
•
•
•
•
•

What types of goods or services does this business provide?
What is your role in the company? What kind of responsibilities do you have?
Are you one of the owners of the business?
Have you completed other surveys from the Census Bureau or other agencies? If so, which
ones?
What was your role in the process for responding to this questionnaire? (Gather data? Enter
data? Consult with data providers? Etc.)

DRB Clearance Number: CBDRB-FY20-ESMD003-002

63

Attachment F

REVISED INNOVATION MODULE for ABS (3rd Year)
Provide ‘Innovation Handout ’ to participant.

INSTRUCTIONS
•

Looking at the title and instructions, does this look like a section that you would
answer?

•

o If no: Who would be the most appropriate person to answer these questions?
Based on the instructions you just read, did your company have new or improved
goods or services?

•

What does ‘differed significantly’ mean to you in this context?

NEW OR IMPROVED GOODS OR SERVICES
•

How would you answer this question?

•
•

How easy or difficult was it to recall the 2016 – 2018 timeframe?
When answering this question, were you thinking only of customer-facing goods or services, or
did you also consider back-end processes?
Is this something you would have in your records?

•

o Please describe those goods or services.

NOVELTY OF NEW OR IMPROVED GOODS OR SERVICES
•

How would you answer this question?

•

Is this something you would have in your records?

o Were these goods or services new to the firm? Were they new to the industry?

NEW OR IMPROVED GOODS OR SERVICES AS PERCENT OF TOTAL SALES
•
•
•

How would you answer this question?
o How would you calculate that answer?
Is this something you would have in your records?
Up to now, we’ve asked you to think about an innovation that was introduced between 2016
and 2018, but this question asks only about the revenue reported in 2018. Did you consider this
when formulating your answer?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

64

Attachment F

EXPECTATIONS
•
•

•
•

•

How would you answer this question?
What does “meet your business’s expectations” mean to you? How would you or your company
determine if a product had met expectations?
o Examples if needed: increased revenue, increased efficiency, increased brand
recognition, etc…
Would someone else in your company need to make this determination? If so, who?
If you had more than one new or improved products or services, how would you determine your
answer?
o What if those products/services had performed differently, such as one failing and one
succeeding?
Do you have documentation to track expectations for your new or improved products/services?
o What information is included in that documentation?
o Is this documentation something you would have access to?

SOURCES OF NEW OR IMPROVED GOODS OR SERVICES
•
•
•
•

How would you answer this question?
When answering this question, were you considering development services provided by parent,
affiliate, or subsidiary companies?
o If so, do you consider those to be separate businesses or as a part of this business?
What does ‘other businesses or organizations’ mean to you?
o Can you give me an example?
If respondent answers with option 2:
o Can you describe that development process you went through?
o What type of organization did you work with?
o Were you the leading partner?
o Can you describe how your organizations cooperated? How was the workload
distributed?

PRODUCT COOPERATION PARTNERS
•

•
•
•

How would you answer this question?
o If participant answered yes to any categories: Could you give some examples?
o If participant answered no to all categories, but answered with choices 3 or 4 in the
previous question: What kinds of businesses or organizations were you thinking of when
you answered the previous question?
 How would you describe them?
Were there any categories that were unclear?
How would you define the term ‘partner’ in this context?
Is this something you would have in your records?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

65

Attachment F

WHERE WERE THESE BUSINESSES OR ORGANIZATIONS LOCATED
•
•
•
•

How would you answer this question?
When answering this question, were you thinking of the location where the work was
conducted, the location of the parent company, the billing address, or something else?
If you worked with the domestic location of a foreign-owned company, how would you report
that?
Is this something you would have in your records?
o How easy or difficult would it be to look this up?

MOST IMPORTANT INNOVATION
•
•
•

•

How would you answer this question?
Did your company have multiple innovations during this time frame?
What does ‘most important’ mean to you or your company?
o What if the question asked you to ‘think about the good or service your company
introduced during 2016 to 2018 that accounts for the most revenue’?
o Would your answer change?
Is this something that would be formally documented somewhere?

MOST IMPORTANT INNOVATION AND SALES
•
•

How would you answer this question?
o How would you calculate that answer?
Is this something you would have in your records?

NEW OR IMPROVED BUSINESS PROCESSES
•
•
•

How would you answer this question?
o If participant answered yes to any categories: Could you give some examples?
Were there any categories that were unclear?
Is this something you would have in your records?

IMPROVED BUSINESS PROCESS DEVELOPERS
•
•

How would you answer this question?
If respondent answers with option 2:
o Can you describe that development process you went through?
o What type of organization did you work with?
o Were you the leading partner?
o Can you describe how your organizations cooperated? How was the workload
distributed?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

66

Attachment F

ACTIVITIES FOR NEW OR IMPROVED PRODUCTS OR BUSINESS PROCESSES
•
•
•
•
•

How would you answer this question?
Did you read the descriptions of the various categories? Were they helpful or unhelpful in
determining your answer?
If participant is unsure: Who would know whether or not your company conducted any of these
activities?
If yes to any: can you give me an example?
Is this something you would have in your records?

ACTIVITY COSTS
•
•
•
•

How would you answer this question?
o How would you calculate that answer?
When answering, were you thinking only of the activities you selected ‘yes’ for in the previous
question?
Where would this information come from? Is this something you would have in your records?
Did your business have activity costs in 2016 or 2017?

R&D FOR ACTIVITY COSTS
•
•
•
•

How would you answer this question?
o How would you calculate that answer?
Can you give me an example of specific R&D activities you were thinking about when answering
this question?
Where would this information come from? Is this something you would have in your records?
Did your business have any of these costs in 2016 or 2017?

GOVERNMENT SUPPORT
•
•
•
•
•
•

How did you decide on your answers to these items?
Did you read the descriptions of the various categories? Were they helpful or unhelpful in
determining your answer?
Who would know whether or not your company took part in any of these programs?
How easy of difficult was it to answer this question thinking only of your company’s innovation
activities from the ‘ACTIVITIES FOR NEW OR IMPROVED PRODUCTS OR BUSINESS PROCESSES’
question?
If yes to any: can you tell me about the process of applying for that [support]?
o Did you have to submit a proposal? Do you know which agency you had to work with?
o Can you describe the type/amount of support provided?
Is this something you would have in your records?

DRB Clearance Number: CBDRB-FY20-ESMD003-002

67

Attachment F

BARRIERS OF INNOVATION
•
•
•
•
•
•

How would you answer this question?
Are the categories clear or unclear?
How easy of difficult was it to answer this question thinking only of your company’s innovation
activities from the ‘ACTIVITIES FOR NEW OR IMPROVED PRODUCTS OR BUSINESS PROCESSES’
question?
Would you have to consult anyone else in the company to answer this question?
o If so, who?
Do you feel like there are any barriers missing from this list?
o If so, what?
Are these factors formally listed or documented anywhere?

BUSINESS STRATEGIES
•
•
•
•
•

How would you answer this question?
Would you need to consult with anyone else in your company to answer these questions?
Is this something you would have documented somewhere?
What does ‘extent’ mean to you in this context?
What if the questions were asking you to rate the ‘importance’ of each of these focuses to your
business? Would that be easier or more difficult to answer?

USE OF DIGITAL TECHNOLOGIES
•
•
•

How would you answer this question?
o If participant answered yes to any categories: Could you give some examples?
Were there any categories that were unclear? (probe specifically on automation)
Is this something you would have in your records?

CAPABILITES FOR USING DIGITAL TECHNOLOGIES
•
•
•

How would you answer this question?
o If participant answered yes to any categories: Could you give some examples?
Were there any categories that were unclear?
Is this something you would have in your records?

TECHNOLOGY CAPABILITIES
How would you answer this question?
o If participant answered yes to any categories: Could you give some examples?
• Were there any categories that were unclear?
• Is this something you would have in your records?
• How would you define ‘technical expertise’ as it applies to your company? Can you give me
examples?
•

DRB Clearance Number: CBDRB-FY20-ESMD003-002

68

Attachment F

WRAP UP
•

•
•

Having gone through the entire set of questions, do you feel that you are still the most
appropriate person to answer these questions?
o If not, who do you feel would be responsible for this section?
Thinking about all of the questions in this handout, how long do you think it would take you to
complete these questions?
Do you have any other comments of suggestions about anything we have discussed today?

THANK YOU FOR YOUR TIME!

DRB Clearance Number: CBDRB-FY20-ESMD003-002

69