Download:
pdf |
pdfAttachment F
Cognitive Interviews with Small and Medium-Sized
Businesses on Innovation
Phase 1 Report
November 2, 2017
Prepared for:
National Science Foundation
Arlington, VA
Prepared by:
Westat
An Employee-Owned Research Corporation®
1600 Research Boulevard
Rockville, Maryland 20850-3129
(301) 251-1500
Attachment F
Table of Contents
Page
Introduction ........................................................................................................................... 2
Detailed Findings and Recommendations......................................................................... 6
2.1 Q1. Importance of Business Strategies to Company .................................... 6
2.2. Q2. New or Significantly Improved Products and Processes.................... 10
2.3. Q3. Product Innovations ................................................................................ 15
2.4. Q4 Percentage of Total Sales.......................................................................... 21
2.5. Q5. Engagement in Innovation Activities & Investment .......................... 26
2.6. Q6. Innovation Activities – Abandoned/Suspended or Ongoing............ 32
2.7. Q7. Public Financial Support for Innovation Activities ............................. 35
2.8. Q8. Innovation Cooperation .......................................................................... 37
2.9. Q9. Innovation Cooperation Partner and Location.................................... 40
2.10. Q10. New Organizational and Marketing Innovations .............................. 45
2.11. Q12. Factors that Hinder/Prevent Innovation ........................................... 50
2.12. Q13. Effects of Legislation or Regulations on Innovation Activities ...... 54
2.13. Q14. Intellectual Property ............................................................................... 57
E-1
Attachment F
Introduction
As part of its mandate to provide a central clearinghouse for the collection, interpretation, and
analysis of data on scientific and engineering resources, the National Science Foundation (NSF)’s
National Center for Science and Engineering Statistics (NCSES) is engaged in several projects to
measure innovation. NSF currently asks questions related to innovation on the Business R&D and
Innovation Survey (BRDIS). With assistance from Westat, NSF developed a series of questions that
asked about product, process, organizational, and marketing innovations that have been
administered on the Microbusiness, Innovation, Science and Technology (MIST) survey and the
MBRDIS survey.
In continuing to develop the innovation questions NSF engaged Westat to conduct exploratory
interviews of small businesses in March 2017. The exploratory interviews addressed what types of
innovations businesses perform, how they plan for and accommodate innovation, and what types of
questions might be required to develop a complete picture of their innovation activities.
This report is a summary of cognitive interviews conducted with 20 businesses during August and
September of 2017 to test a draft module of 13 innovation questions. Several of the questions have
been tested and used previously on BRDIS, MIST, and the Community Innovation Survey (CIS).
The objective was to assess respondent interpretation of the questions, relevance of the content,
feasibility of response, and appropriateness of the response options. Westat gauged whether there
was any need to change the question order; change question wording; add, remove, or revise the
wording of response options; and provide or modify an introduction, definitions, and/or
instructions. The intended result of the testing was to identify any refinements needed to make the
questions appropriate for U.S. businesses and the Annual Business Survey (ABS).
Interview Procedures
Eight in-person interviews were conducted at the respondent’s place of business in Chicago and in
the Washington, D.C. metropolitan area. For the other 12, sessions were conducted remotely via
WebEx online meeting software, which allowed the interviewer to share a computer screen with the
respondent and show a PDF version of the questionnaire.
E-2
Attachment F
As an ice-breaker and to help establish some understanding of the innovation context for the
company, each respondent was asked to briefly describe the type of work the company conducts.
Then, the interviewer handed (for in-person) or showed (for WebEx) the assigned questionnaire
version and asked the respondent to read the first question to himself or herself and answer it. For
the in-person interviews, the respondent marked answers on a hard copy of the survey. For WebEx
sessions, the interviewer allowed the respondent to control the shared screen and mark answers on a
PDF version created to capture responses for the company. After the respondent completed the
first question, the interviewer administered several scripted probes (e.g., how the respondent
interpreted specific phrases, such as “new or significantly improved;” how the respondent decided
on answer(s), and whether the company tracked that type of information). The interviewer and/or
any NSF or Westat observers asked any follow-up questions needed for clarification, based on the
respondent’s answers.
This same approach was used to cover the remaining questions in the assigned questionnaire
version. If the respondent was not able to immediately answer a question, he or she was asked to
describe what steps would be required, who might help to supply the data, and what level of effort
would be involved.
At the end of the interview session time, the NSF survey manager and the interviewer thanked the
respondent for their participation. As approved in the OMB clearance package, for in-person
interviews, the respondent was given a $40 cash incentive. Those who completed a WebEx session
sere sent a $40 check. All those who participated were sent a thank-you letter after the interview.
E-3
Attachment F
Recruiting
The following criteria were established for recruiting businesses to the first phase of cognitive
interviews:
•
The firms should have between 5 and 500 employees, and
•
The firms should be involved in innovation.
In addition, to maximize the chances for contacting firms that were eligible, the sample focused on
those industries that were most likely to be involved in innovation. Since self-reports of being
involved in innovation were relatively rare, and all recruiting and interviews had to be conducted
within a relatively short time, this approach helped to identify eligible respondents both quickly and
efficiently. NSF provided a list of the 20 industries with the intention of selecting one firm from
each industry, but it was soon decided that those industries were not sufficiently diverse. For
example, the top 20 included 6 industries that were wholesalers, possibly all sharing relatively similar
perspectives, while none of the top 20 were retailers. One firm each from the following industries
were selected (Table 1).
Table 1.
NAICS
5112
5417
4541
5191
3261
4242
3399
5182
5111
3121
3363
4248
3329
3391
5415
3118
3335
4532
4539
4251
Industries selected for cognitive interviews
Industry
Software Publishers
Scientific Research and Development Services
Electronic Shopping and Mail-Order Houses
Other Information Service
Plastics Product Manufacturing
Drugs and Druggists' Sundries Merchant Wholesalers
Other Miscellaneous Manufacturing
Data Processing, Hosting, and Related Services
Newspaper, Periodical, Book, and Directory Publishers
Beverage Manufacturing
Motor Vehicle Parts Manufacturing
Beer, Wine, and Distilled Alcoholic Beverage Merchant Wholesalers
Other Fabricated Metal Product Manufacturing
Medical Equipment and Supplies Manufacturing
Computer Systems Design and Related Services
Bakeries and Tortilla Manufacturing
Metalworking Machinery Manufacturing
Office Supplies, Stationery, and Gift Stores
Other Miscellaneous Store Retailers
Wholesale Electronic Markets and Agents and Brokers
E-4
Attachment F
The businesses were selected using a database maintained by the Small Business Administration
(found at http://dsbs.sba.gov/dsbs/search/dsp_dsbs.cfm). Anticipating that it might be difficult to
recruit eligible businesses to participate, a sample of 25 firms per industry was created. When finding
eligible and cooperative participants provided difficult, the sample was supplemented with additional
firms in the targeted industries, and in some cases multiple participants per industry were allowed.
As part of the sampling process, the interviews were divided into three sets: we targeted five for inperson interviews in Chicago, there for in-person interviews within the Washington, D.C.
metropolitan area, and 12 for online interviews using a service (WebEx) that allowed the interviewer
and business to share the same computer screen. (Ultimately, four were conducted in Chicago, four
in the Washington, D.C. metropolitan area, and 12 online.)
E-5
Attachment F
Detailed Findings and
Recommendations
This chapter presents summary-level findings from the cognitive interviews for each of the 13
questions tested. 1
2.1
Q1. Importance of Business Strategies to Company
Question Comprehension
This question worked well as an opening question; respondents all found it relatively easy to answer.
Respondents generally seemed to understand the intent of this question. For example, one
respondent paraphrased the question as: “What are your business’ priorities?” Others said the
question asked what the strategies are for day-to-day business, what the organization is focused on,
what the business plan is, or what the company’s business model is.
The questions were numbered 1 – 14, but Question 11 was not included within the questionnaire, so the total number
of questions tested was 13.
1
E-6
Attachment F
There were two minor comprehension issues with specific parts of the question. One respondent
said that item D (focus on customer-specific solutions) and item G (focus on satisfying key clients)
seem to “fall into the same group.” That company’s business is mostly generated through word-ofmouth and repeat business, so the respondent felt that both parts seemed to touch on selling to their
existing clientele. Another respondent voiced a bit of confusion in response to item E (Focus on
Low Price). She asked: “Low price of what?” and assumed it meant a focus on lowering her prices,
or being “price competitive.” She found this part of the question to be worded in a grammatically
awkward way and suggested revising it to “Focus on lowering price.”
Relevance of Question
Based on the fact that all 20 respondents marked “High” and/or “Medium” for more than one part
of this question, it seemed relevant for the wide variety of companies that participated in this round
of testing.
Feasibility of Reporting for 3-Year Reference Period
None of the respondents expressed any difficulty with answering this question for a 3-year period.
However, several respondents commented on the time period in relation to the company’s current
status or stage of development. One respondent who worked for a company in the start-up stage
said that some of the strategies listed are not as relevant now, but will increase in importance as the
business grows and starts to sell products. Another said that if he currently had more long-term
contracts, his answers to the parts of the question would be different.
Response Accuracy and Consistency
All respondents found it generally easy to mark answers when they worked on their own.
Respondents who were asked to describe their response thought process were able to easily explain
and differentiate ratings for High and Medium for different parts of the question.
Respondents were also asked if any strategies they consider important for their company were
missing from the list. Only four respondents said the list of strategies was complete. Some of the 16
other respondents suggested the following as strategies that were missing:
E-7
Attachment F
•
Fundraising or financing (important to a small business)
•
Advertising
•
Expanding to existing clients
•
Customer satisfaction
•
Market dynamics
•
The customers’ position or receptiveness to innovation
•
Developing and patenting technology
•
Setting up streams of recurring revenue.
Respondents also said that the question asked only about strategies that were external. One
explained that the parts of the question are all very “outside the building” and that the question is
missing some “inside the building” topics, such as finding a way to borrow more money or finding a
new advertising tool or technique. Suggestions from these respondents for strategies of improving
internal processes or factors include the following:
•
“Taking care of employees” to reduce turnover and avoid the costs associated with hiring
and training new employees
•
Improving the workforce
•
Improving employee benefits
•
Strategic planning
•
Improving internal processes to increase cost efficiency
•
Improving internal systems through increased use of technology
•
Focusing on technological challenges
•
Keeping current with technology changes and improvements (in processing).
Several respondents suggested asking this question in a different format, to collect priorities or
rankings of the set of strategies vs. a rating for each strategy. These respondents had rated multiple
parts of the question as High and/or Medium. They seemed to want to indicate their highest
priority, but it would not be evident from their ratings. One respondent commented that he read
E-8
Attachment F
through all of the subparts first to determine his priorities, and then chose responses that would
reflect his priorities.
Q1 Recommendations
Consider broadening the question to include strategies with an internal focus. Multiple
respondents pointed out that the parts of the tested question are primarily focused on external
factors. Given that this is the first question respondents will answer in the new innovation module,
leaving out internally-focused strategies might have two implications: (1) making the question appear
incomplete, and (2) possibly communicating to respondents that NSF defines innovation in a limited
way—in terms of only the external market. Also of consideration is whether it is important for the
parts of this first question to explicitly relate to innovation. For example, a focus on employee
satisfaction might have little to do with innovation.
Through discussion of the initial findings with NSF, a decision was reached to add three parts to the
question. The first, listed below, has an external focus. The other two reflect an internal focus, to
ensure survey respondents will think about innovation more broadly, rather than limited to external
factors. Although NSF may wish to revise the wording slightly, these additional parts to consider
adding include:
•
Focus on understanding customer needs
•
Focus on improving employee skills
•
Focus on improving internal processes.
Based on the respondent suggestions, an item related to the use of technology would also be a
welcome addition. However, OECD’s current stance is to avoid using the term “technology”
because it can refer to almost everything, so NSF decided against that.
We do not recommend changing the question to ask for priorities. Aside from lessening
comparability with international data, such a format might be harder to answer, and respondents’
reported priorities could change depending on what items are listed.
E-9
Attachment F
2.2.
Q2. New or Significantly Improved Products and Processes
Question Comprehension
A key issue for this question is respondents’ interpretations of the phrase “new or significantly
improved” within the question stem.
Respondents seemed to differ in their assumptions of the level of newness of the “new or
significantly improved” product or process. Two of the respondents thought the question asked
about what was new to the world or to the marketplace.
•
One respondent said he did not know whether “new or significantly improved” meant
new/improved for the company or the world. He assumed the latter since the questions
were about innovation. Their company niche is to develop things that other people are not
doing. But he went on to say that perhaps the phrase could mean a new product for the
company, and suggested that a clarification (new for the company or the world) was
needed.
•
Another respondent said the phrase meant a significant effort to bring something different
to the world, to improve a product or service.
More of the respondents seemed to interpret the phrase as referring to what was new to their own
company. For example:
E-10
Attachment F
•
One said that he considered “new or significantly improved” as a product that they have
not offered before or a new purchase for their company.
•
One said the phrase meant making major changes to processes in their facility.
•
One interpreted the phrase to mean revamping the way you had previously done business.
•
One respondent defined “new or significantly improved” as new to his company, not
necessarily new to the marketplace globally.
•
One interpreted the phrase as asking if they did something—internal and/or external—
that greatly improved their business.
•
Another said that “new or significantly improved” means to improve upon the way of
doing business, whether it be the design or actual services offered to customers, or the
way of producing it.
Respondents differed in their assumption about the degree to which something would have to
change to qualify as “new or significantly improved.” One narrowly interpreted the phrase as
meaning “a simple facelift of an existing service.” Another respondent used the phrase “a
generational product change” and said that a product is either brand new, or the same technology
but a new product with a new marketing push. He gave the example of an iPhone 5 vs. an iPhone 6.
Some of the respondents went into more detail in explaining what the concept of “new or
significantly improved” meant to them. For instance:
•
One respondent said that there are two parts to this: (1) incremental continuous
improvement and (2) transformational. While he said the process for both might be
similar, he considers most innovation taking place today to be incremental (introducing
new features, new processes, etc.).
•
One respondent said: “We’re always tweaking what we do, but we’ve always done it well.
How do you significantly improve an investigation?” The respondent seemed to want
some way to quantify change, as in improving something by X percent or more.
•
One said that the term “significant” is subjective; what he might think of as significantly
improved, someone else might not think so. He elaborated that this question was easier to
answer because there is “more leeway.”
•
One respondent said during probing that “new or significantly improved” means
“upgrade.” However, he did not consider that meaning until after he had read through the
separate parts of the question. He said that the question made more sense to him and
helped him to define the meaning after he finished reading items A–E. In particular, item
E (Supporting activities) made him think about upgrades to the accounting system.
E-11
Attachment F
Two other key terms were “goods” and “services.” Most respondents found the explanations to
align with how they operate or what they produce. However, several expressed some confusion
about how to classify certain types of products, as described below.
•
One respondent was confused by the definition of “Goods” because her company
primarily resells rather than manufactures items. However, they do purchase certain items
(for instance, herbs) and combine those ingredients into products that they sell. She
wondered whether such products qualify as goods, and ultimately decided to count them.
•
One respondent described goods as the curriculum they have developed and regularly
improve. She elaborated that it was difficult to separate this good from the service of
training.
•
One respondent was confused by the wording “but downloadable software… are also
goods” of the explanation for item A. He thought that phrase implied there is some other
type of software that is not included.
•
About item B (Services), one respondent said: “I’m not quite sure what you mean by
intangible…in our industry, if you perform a service, you give someone the result.” His
company does lab testing, so customers send him samples and he sends back results. He
viewed the reports as tangible, but also as a service: “That would be a service because you
receive something from the customer and you process it and you give them the result.”
•
One respondent did not agree with the item B explanation that services are usually
intangible. He included their customer service as a tangible service, based on his belief that
customer service and long-term relationships are key components of a successful business
and can be more important than pricing.
During probing, one respondent suggested clarifying “processes” in the wording of item E
(Supporting activities for your processes) to more clearly indicate this means administrative
processes. He thought only about manufacturing processes when answering the question.
Relevance of Question
As evidence of the relevance of this question, 18 of the respondents who answered this question all
marked “Yes” for two or more parts of this question. The other two respondents only responded
“Yes” to one part of the question.
However, one respondent said that three items in this question—items A, B, and D—were not
applicable for her company because they do not make or sell products. This response appears to
reflect a definition we have run into elsewhere: businesses that prepare electronic products or
E-12
Attachment F
reports sometimes do not think of these products as either goods or services. This particular firm
served companies and government agencies, conducting investigations to detect financial fraud and
mismanagement; collecting intelligence on “high profile” people and companies to determine legal,
financial, and reputational risks; and performing compliance monitoring for educational service
grants. She suggested adding a “Not applicable” response option for the parts of this question since
she felt that “No” was not an appropriate response for their company.
Feasibility of Reporting for 3-Year Reference Period
We did not explicitly ask respondents about the feasibility of reporting these types of innovations
for the reference period. However, several respondents commented on the reference period
indicating that it was feasible for them to consider that time period in their response. They said they
answered “No” because they might have had that type of innovation at some point, but not within
the 3-year period. For example, one respondent who indicated uncertainty about item C (Methods
of manufacturing) explained that he thought of several software-type products, but he did not think
there were any new or significant improvements in how they manufactured them during the
reference years.
The respondent who works for a new company mentioned that new companies may not have goods
or services at the point in time they respond to the survey.
Response Accuracy and Consistency
For the first interviews we held, we asked respondents to describe how they decided on their
answers—to judge accuracy of their “Yes” and “No” answers. (In later interviews, we probed less
on the responses, to conserve interview time for later questions.) Most of the explanations indicated
respondents were able to answer the different parts of the question.
However, a couple of respondents had trouble answering a part of the question, or said something
that might not align with NSF’s measurement intent. The respondent who considered his reports to
be “tangible” services explained his “No” answer for item B (Services) by saying that it was because
he didn’t charge his clients for the reports he provides, in addition to the fact that the service is not
new or significantly improved. He did reports for free as a courtesy if customers had questions.
When asked, he said he only considered things in this list if his company was paid to do them.
E-13
Attachment F
One respondent explained that his company does not make packaged software, so he had difficulty
choosing between item A (Goods) and item B (Services). Each software package is unique to the
client, so it seems more like a service than a good because they are not doing the same thing across
clients. The company does not create an app (“a good”) and sell it; they create the whole package as
a service. He decided to mark “Yes” for Services, because in the past 3 years, an improved service
has been to gather metrics for clients (e.g., who uses the product, how it is being used, user
information), which they sell as a part of the package.
During probing, one respondent decided that he should change his “No” answer to “Yes” for item
D (Logistics). He thought he probably just “rushed through that a little bit” and did not think it
through enough when he initially answered. His company makes products more accessible to
customers. They don’t have their own trucks or do their own distribution.
Two respondents had trouble answering item E (Supporting activities for your processes); they
wanted to add an example to those mentioned. One considered that “Supporting activities” for his
company included “equipment, maintenance systems, software, and their employees,” and another
suggested adding something related to “human resources” to the wording.
We also asked most of the respondents whether anything they considered innovative within their
company was not captured in this question. The only things respondents mentioned were the
following. We note that the latter two of these perceived missing innovations might be captured in
Q10 (which the respondents had not yet seen), which covers organizational and marketing
innovations.
•
One had purchased large dual monitor computers for company staff, to increase their
efficiency in processing of orders and inventory. She did not think this “technological
update” fit into any of the parts of the question. (She did not seem to realize that it might
fit with item E—“Supporting activities for your processes…”). 2
•
One respondent said his company is now focused on educating employees, to change their
perceptions of their jobs and the company.
•
As noted above, one respondent suggested adding “human resources” to item E.
This example also displays the ambiguity of defining an innovation. As explained later in the discussion of other
questions, some respondents would not consider this to be an innovation because both the technology and the use of
the technology were already established. However, having two screens helped the staff to simultaneously view the orders
and the inventory database.
2
E-14
Attachment F
Q2 Recommendations
We provide recommendations for changing Q2 in the section of recommendations for Q3 (below),
since the results from testing these two questions suggest introducing concepts in a different order
as well as revising the wording.
2.3.
Q3. Product Innovations
Question Comprehension
Most respondents seemed to understand the difference between new to their market vs. new only to
their company. However, one respondent said that she had to read the definition for “Only new to
your company” a couple of times before she really got it. She noted that she had to go back to the
bold text on the left in order to grasp the definition (the text in regular font).
Two respondents had an issue with the definition. They made the point that if a company is bringing
a new product to the market, there may be no competitors. The definition also seemed confusing if
they were looking to introduce a product where there is not a lot of competition.
Another respondent wondered aloud why the question asked about “Only new to your company”
because she considers this as a situation where the competitor has done the innovation, and you are
essentially stealing the idea: “It's theft of intellectual property.”
We probed respondents for their interpretation of the term “product innovations” in the question
stem. Examples include the following.
E-15
Attachment F
•
One said: “Something about your product is different from before you did it. Whether you
were first to the market, or if you make improvements to something that’s already there.”
•
For one respondent, “product innovation” means novel ideas or solutions for clients; the
“product” is the service.
•
One respondent said that since Q3 refers to Q2A and Q2B, he assumed product
innovation and goods were the same thing.
•
Another explained that product innovation is something that came to the marketplace
with some significant difference from what came before.
•
One described innovation as an “aha” moment; something was being done one way
(status quo), but then someone else introduced an easier path. This respondent said the
word “novel” is used in the R&D community.
Where time allowed, respondents were also asked to explain their thoughts on the phrase “product
(good or service)” within the question stem. Probing revealed some differential interpretation across
respondents. Several considered products to be the same as goods (or perhaps somewhat broader),
but not to include services. For example:
•
One respondent thought that the phrase was confusing and unclear, saying that it first
talks about a product, but then the qualification says goods and services. To him, products
are tangible goods and services are intangible.
•
Another respondent said: “Well there's product innovations and then there's service
innovations.” What his company does falls in a grey area between products and services:
“It's a product in that it's tangible, it was basically a report…but to develop that report we
were providing a service. So I basically said we're a product…”
•
One respondent initially had an issue with the term “products,” and said: “This is not
relevant to us because we don't sell products.” She does not think of services as being
products—“Product in my mind in a physical good.” After she noticed and thought more
about the “good or service” in parentheses, she decided it was okay, but suggested that it
would be better to just say “goods or services.”
•
One respondent said that goods and products are the same thing to him.
On the other hand, several respondents indicated that they clearly understood that “product (good
or service)” means the product could be either a good or a service. For example, one respondent
said that a product can be a good or a service. However, he would prefer that the two were broken
out and asked about separately, since to him, they are different.
E-16
Attachment F
Relevance of Question
We judged this question as relevant to a variety of companies based on the finding that the 18
respondents who completed it answered “Yes” to one or both parts of it.
Feasibility of Reporting for Three-Year Reference Period
We did not probe on ease or diffculty of reporting for the 3-year period. However, none of the
respondents explicitly expressed any difficulties in reporting for this reference period.
Response Accuracy and Consistency
Respondents seemed to select the correct response based on their explanations. Examples of “Yes”
responses for “New to your market” include: providing energy via fiber, turning what had been only
an idea into a final product, and developing a new device for doing resuscitation for a different
patient population than they had targeted with a prior product before. One mentioned having done
extensive research on what her competitors do, even getting details on their government contracts
through FOIA requests.
Three participants explicitly mentioned issues related to the order of the questions. One participant
stated that if he had understood Q2 as asking about what was “new to your company,” not new to
the market, he would have answered Q2 differently. Another said that when answering Q2, he was
not really thinking about “innovative new products or services.” Although he did not consider it an
issue to begin using the word “innovation” in Q3, he said he did not immediately apply that term
before reading Q3. A third respondent said that when responding to Q2, he was thinking about
innovations that were new to his company, not to their market.
Two respondents said they needed a different response option. One respondent said she only knows
what products are new to her company, so wished there was a “Don’t know” or an “I’m not sure”
option in addition to “Yes” and “No.” A second respondent said he would like a “Not known”
response to the “New to the market” part of the question because he does not know for sure if
some of the products they are developing exist elsewhere.
E-17
Attachment F
Q2 and Q3 Recommendations
Modify the wording for the reference period to improve clarity and consistency. For these
questions (and all subsequent questions with the same reference period), we also recommend using
the wording “2015 through 2017” to make the 3-year reference period clearer.
Consider adding a “Don’t know” response option. Underlying Q3 is an assumption that
respondents know everything going on in their market. Two respondents made the point that this is
not always the case or possible. Some new companies, and even some companies with longer tenure,
may not have goods or services during the 3-year reference period. As noted above, two
respondents suggested adding a “not applicable” response option. One was not comfortable
responding “No” to multiple parts of Q2 when she felt they were really not applicable. We also note
that respondents may perceive that answering “No” to parts of Q2 or Q3 suggests a lack of interest
in innovation. This is a socially undesirable response, especially for companies in areas of business
that value innovation. When discussing these initial results for these two questions with NSF, we
explored the approach of providing an additional response column for capturing “Not applicable”
responses. NSF decided against this recommendation.
Modify the approach for respondent reporting of product and process innovation. Question 2
is complex, due to having multiple parts as well as a phrase (new or significantly improved) that was
differentially interpreted. There was also evidence that information provided in Q3 (new or
significantly improved goods or services are considered to be product innovations, as well as the
distinction between new to the market vs. only new to the company) was needed at the time
respondents read Q2. Some respondents appeared to think only of innovation that is new to the
market, not both types of innovations. If Q2 is retained as is, survey respondents may under-report
innovation at that point; they need to be instructed or reminded to consider innovation that is new
to the company as well as to the world. Making a change should help to improve data quality.
Based on the Q2 and Q3 comprehension and order issues, we suggest using an unfolding approach,
to ask for innovation data in a different sequence of questions, with the aim of ensuring each
individual question will be easier to understand and answer. Parsing the concepts into simpler
questions should help to improve data quality.
E-18
Attachment F
The sequence could be:
•
First, identify whether the company produced or offered goods and/or services during the
3-year reference period.
•
Ask a follow-up question only of those who produced/offered goods during the reference
period: determine if the company introduced new or significantly improved goods—new
to the market and new to the company.
•
Ask a separate follow-up question only of those who produced/offered services during
the reference period: determine if the company introduced new or significantly improved
services—new to the market and new to the company.
•
In a separate question, ask about remaining aspects of innovation covered in items C–E of
the tested version of Q2 (Methods of manufacturing; Logistics, delivery or distribution
methods; and Supporting activities).
We expect this approach will help respondents to define innovation in a more consistent manner.
Question routing (as noted below) will be need to be handled through programming to ensure that
respondents get only the questions that are appropriate for them in the new question sequence.
Here are the suggested revisions for order and wording of four new questions (to replace the tested
Q2 and Q3 versions).
2. During 2015 through 2017, did your company offer any:
A. Goods? (place the definition here)
B. Services? (place the definition here)
Yes
Yes
E-19
No
No
Attachment F
[If Yes to Q2A]
3. During the three years 2015 through 2017, did your company introduce new or
significantly improved goods that were:
Yes
A. New to
your
market?
Your company introduced a new or
significantly improved good to your market
before your competitors (it may have already
been available in other markets)
B. Only new
to your
company?
Your company introduced a new or
significantly improved good that was already
available from your competitors in your
market
No
[If Yes to Q2B]
4. During the three years 2015 through 2017, did your company introduce new or
significantly improved services that were:
Yes
A. New to
your
market?
Your company introduced a new or
significantly improved service onto your
market before your competitors (it may have
already been available in other markets)
B. Only new
to your
company?
Your company introduced a new or
significantly improved service that was
already available from your competitors in
your market
E-20
No
Attachment F
5. During the three years 2015 through 2017, did your company introduce new or significantly
improved:
Yes
No
A. Methods of manufacturing for producing goods or
services?
B. Logistics, delivery or distribution methods for your
inputs, goods or services?
C. Supporting activities for your processes, such as
maintenance systems or operations for purchasing,
accounting, or computing?
2.4.
Q4 Percentage of Total Sales
Question Comprehension
Based on their responses, the nine respondents understood the overall purpose of this question.
However, almost none of the respondents understood which time period applied to this question.
E-21
Attachment F
Q4 asks for the total number of sales in 2017, based on various categories defined by a 2015−2017
period. None of the respondents understood that they should only report 2017 sales (with the
possible exception of one respondent whose response was unclear). 3
Relevance of Question
A majority of respondents did not express concerns about the relevance of this question to their
company. However, two respondents had no sales in 2017 and did not know how to respond, since
the question assumes that the three categories will sum to 100% of sales. One respondent decided to
put 100% under “products… that were marginally modified.” The other respondent with no sales
put 100% under “new to market,” using the rationale that the products would eventually be new to
market when sold in 2018. Both of these respondents expressed misgivings about their answer.
Reactions to Definitions
Q4 contained no definitions. However, one respondent expressed concerns about the wording of
the “only new to your company” option. This respondent felt that “only new to your company”
sounded negative (e.g., the company missed something common in the marketplace).
Feasibility of Reporting for Reference Period
A few respondents volunteered that reporting for 3 years was easier than reporting for 1, partly
because they felt freer to provide estimates when asked about a 3-year time period, and in one case
because the most recent year was atypical for that firm. In a few cases, respondents stated that they
did not read the question carefully since it did not apply to their company (see below); however,
even respondents who read and answered the question did not notice the 2017 time period.
Response Accuracy and Consistency
Many respondents expressed doubt about their ability to respond to this question. Of the seven
respondents whose companies had sales:
•
3
Four stated that they guessed or used very rough approximation
Throughout this report, not every respondent provided information on every topic.
E-22
Attachment F
•
Two estimated using a specific process
•
One refused to answer due to the difficulty of estimating.
The four respondents who guessed expressed uncertainty about their estimates. For example, one
respondent stated that he did not deal with sales numbers, so he was uncertain. Another respondent
was uncertain about which products were new to the market, versus just to her company. She
partially answered the question, indicating a “Don’t know” response for “new to market.” One
respondent was reasonably confident in his estimate due to his company role, but he pointed out
that answering this question would be very difficult unless the survey carefully screened for
knowledgeable respondents. He expressed concern that estimates would vary widely; he felt he
could give the survey to six different engineers in his company and get six different answers.
Two respondents provided estimates and also provided their rationale for the estimates. One of
these respondents was the company comptroller and reported having a deep knowledge of the
company’s finances.
One respondent refused to answer the question, stating that it was too difficult given his company’s
business model, which sold bundles of innovative and established products. He said that
disentangling how much income came from the new products would not be worth his time: “This is
a level of granularity that I would say is a pain in the b*** and I wouldn’t do it.”
Finally, responses to this question should be consistent with Q3. However, one respondent’s
responses were inconsistent: in Q3, he indicated no innovative products or services to the market,
but in Q4, he indicated that these were 10% of sales.
Q4 Recommendations
Consider deleting Q4 or screening for knowledgeable respondents. Most respondents did not
feel confident in their estimates of sales; one said she did not know how to answer part of the
question, and one refused to answer due to the difficulty. Another respondent suggested screening
for survey respondents who would be in a position to provide reasonable estimates. Due to these
concerns, we recommend weighing the value added of this question against the possibility of
inaccurate data, and/or consider screening respondents for this question.
E-23
Attachment F
Add the word “approximate.” If this question is retained, consider rephrasing to make
respondents more comfortable providing ballpark estimates. For example, instead of “please give
the percent of your total sales in 2017,” consider the phrasing: “please estimate the approximate
percent of your total sales in 2017.”
Consider possibilities to fix the time period issue. All of the respondents incorrectly reported
for the 2015–2017 period, as described above for the reference period. This issue could lead to
incorrect estimates: as one respondent stated, small businesses can have huge swings in sales, with a
contract in one year swinging sales dramatically. If Q4 is retained, consider the following options to
address this issue:
•
Change question to focus on 2015−2017 instead of 2017 only. A disadvantage of this
approach is that this question will no longer be internationally comparable. However,
since none of the respondents answered this question accurately, the data may not
currently be useful for comparisons regardless. As a benefit, focusing on 2015−2017
would be consistent with the rest of the survey, as respondents who are in the mind frame
of answering for 2015−2017 would not have to switch gears at this question.
•
Continue to report percentages only for 2017, but reinforce this timeframe. For
example, consider adding an introductory sentence saying, “Unlike other questions on this
survey, this question focuses on 2017 only. Please consider your company’s sales in 2017
only when answering this question.” Consider also underlining “2017” throughout and
increasing the font size of instructions related to the 2017 time frame. However, one of
the confusing elements of the question is that in different places it refers both to 2017 and
to 2015 through 2017. For maximal clarity, it would be best to refer to only one time
period throughout the question.
Ensure consistency between Q3 and Q4. Respondents who did not check “new to market” in
Q3 should report zero sales for this category in Q4. One respondent provided inconsistent
responses, saying in Q3 that he did not have anything “new to market,” but reported 10% of sales in
Q4 for that category. Consistency between Q3 and Q4 could be achieved via several options. For
example:
•
Programming to guide presentation of appropriate parts of the question (e.g., assuming
web administration, provide only those options that were checked in Q3); or
•
Cleaning on the back end (e.g., permit respondents to provide inconsistent responses on
Q3 and Q4 and clean their data for these items if necessary).
E-24
Attachment F
Clarify the distinction between products, goods, and services. Respondents often viewed
products as a larger category than goods, but without including services within products. One solution is
to avoid the word products; another is to use all three terms.
Permit companies to enter zero sales. A few companies had zero sales during the time frame
(e.g., new companies). Consider adding language to permit companies to report no sales. Some
options include:
•
Add another question before Q4 to screen for companies with sales; or
Add an option to Q4 to indicate “no sales,” perhaps below the question options.
E-25
Attachment F
2.5.
Q5. Engagement in Innovation Activities & Investment
E-26
Attachment F
Half of the respondents were given version A of Q5, which included the column for total
investment, and half were given version B, which did not include the column for total investment.
However, not all respondents were administered the question because after initial testing we began
skipping Q5 in order to allow time to test other questions later in the survey. Interviewers returned
to Q5 if time allowed. Seventeen respondents discussed Q5, and fourteen provided answers to the
questionnaire items.
Question Comprehension
Respondents were asked what they thought the question asked, and what “innovation activities” and
“engage in innovation activities” mean. Their responses were used to assess their general question
comprehension.
Most respondents generally understood that the question focused on innovation, but several did not
articulate an understanding that the question was specifically about innovation activities. Two
respondents said that the question was about any materials, products, or services that were new and
different for the company. One of those two said “change” and “innovation” are almost
synonomous. Another respondent said “innovation activites” made him think about the acquisition
of machinery, knowledge, and training, and talked about obtaining the knowledge needed to develop
a new product. One respondent said innovation activities referred to efforts made in order to bring
products or services to market. Another said the question looks at where they invest their resources
in the development of the company.
Relevance of Question
None of the respondents expressed concerns about the relevance of this question to their company.
Almost all respondents who answered the question provided an answer for all items A through I.
Reactions to Definitions
When asked if they read the information in the middle column of Q5A, eight of the respondents
said they read or at least skimmed the definitions. Most said the definitions were clear and helped
them better understand the term in the column to the left. One respondent said she changed some
of her answers after reading the information in the middle column. Another said he read the
definitions a few times before answering. Another said he read the middle column first and then the
associated term in the left column.
E-27
Attachment F
Only two respondents said they did not read the middle column carefully. One said he did not spend
much time on the middle column (20%), compared to the first column (80%). He said that was due
to the amount of time it would take to read it. He read the middle column only when he needed
clarification or didn’t understand the term on the left. During probing, this respondent read the
definition for F. Market introduction of innovations. After reading it he changed his answer from
“No” to “Yes.” The other respondent said he only skimmed the middle column definitions.
One respondent felt that the first column was difficult to read because the question items were too
close together, especially items C, D, and E.
A−H Definitions: Some respondents commented on specific defnitions.
•
A and B (in-house and external R&D). One respondent at a small software and
hardware design company struggled with both of these items because he did not know
what would be considered an investment. Another respondent asked “How do you define
R&D?” He defines research as the fact that they are prototyping, testing, and finding the
solutions and applications “on the fly.” He feels the more traditional definition of R&D
implies a more sophisticated process and is not sure if his company’s process would
qualify. He felt the question assumes that all innovation occurs in R&D, but that is not
always true. Sometimes “someone has an idea in the shower and that is not part of R&D.”
•
C (Acquisition of machinery, equipment, software & buildings). One respondent
noted that property and machinery are often leased, rather than acquired. She answered C
“Yes” based on leasing two new offices but wondered if her answer was correct.
•
D (Acquisition of existing knowledge from other companies or organizations). One
respondent felt that the middle column definition was too wordy.
•
E (Training for innovative activities). One respondent wondered if one of his
engineers taking a class or course to learn better use of a software program on design
projects should be included in E.
•
F (Market introduction of innovations). One respondent did not understand why “inhouse” was specified and felt that the definition was not useful. This same respondent
found it difficult to discrimate between F, G, and H. He felt that G overlapped with both
F and H. Another respondent asked if F should include email blasts to customers.
Another respondent said F should include email campaigns, product annoucements on the
website and LinkedIn, and going to research and scientific meetings.
•
G (Brand building). See F and H.
•
H (Design). One respondent was thinking about design work on the website and training
manuals and put it under H, but then moved it to G since she decided H was only relevant
E-28
Attachment F
to goods. Another respondent wasn’t sure if H included web design, since the respondent
considers web design an innovative activity since if affects the appearance of the good to
the market.
•
I (Other). One respondent found this item confusing. Another respondent wanted to put
“contracting out software development” in I, but was not sure. She noticed references to
software in A and C and she was further confused. She decided to put it in A for software
developed by your own employees rather than C, which she thought was about purchasing
software. One respondent wondered if “tooling up” in I was redundant with “acquiring
new equipment” in C. One respondent answered “Yes” to I because he had to lease some
equipment from other companies. He wasn’t sure if his answer was correct since he was
only assuming that this is “tooling up.”
Feasibility of Reporting for 3-Year Reference Period
We did not probe on this topic. Only three respondents mentioned the 3-year reference period. All
three noted that they were referring specifically to the reference period.
Feasibility of Reporting Investment Totals
Many respondents’ first reaction was to the total investment portion of the question. Respondents
felt the question was primarily about the investment totals rather than determining innovation
activities, and had a lot to say about it. Nine of the eleven respondents who commented on the total
investment column responded negatively. They stated that filling out the total investment column
would require going over numbers, consulting the finance person, or would be a tedious, timeconsuming process. They found it difficult to enter rough estimates, and were not very confident.
One respondent said he probably undercounted and to take his answers “with a grain of salt.”
Another respondent said he was frustrated with the question and was unable to provide percentage
estimates. He said it was “off-putting.” Respondents felt that this portion of the question was asking
for too much detail.
One respondent was neutral in his response; he stated that it wasn’t too difficult but that it might be
for a larger company. One other respondent said he retains a lot of information and was easily able
to answer the 5A Total Investment.
Response Accuracy and Consistency
Four respondents expressed a clear preference for considering the two options under 5A
(Continuously and Occasionally) as requiring a choice between the two, rather than answering yes or
E-29
Attachment F
no for each option. The CIS also formats the question to require a choice between the two, and
NSF expressed a preference for returning to the CIS format.
One respondent had difficulty answering 5A because her staff are not specifically R&D staff, but
contribute to R&D as part of their regular responsibilities. She considered this “continuous” but did
not think that it met the survey definiton of continuously. She suggested using “daily” or “weekly”
instead of “continuously.”
When asked if there were any additional innovation activities not shown in question 5 that were
relevant to their company, four respondents made suggestions. One said that legal activities related
to innovation (intellectural property) should be included. Another said his company goes on
research or inspirational trips. For example they spent time and money to go to Disneyland to see
the newest attractions. One respondent mentioned “open management,” management that is open
to new ideas and input, and encourages employees to try new things. He provided the example of
Google giving employees time to just think. Another employee said Q5 is missing “recruiting
suitable employees who fit well with innovation.”
Q5 Recommendations:
Keep middle definitions. Respondents find them useful.
Add spacing between rows to improve readability.
Revise definition of A. In-house R&D. Add “formal or informal” to the definition of A (Formal
and informal research and development activities undertaken...).
Revise definition of C. Acquisition of machinery, equipment, software buildings. Add
“acquired or rented” to the definition.
Simplify definition of D. Acquisition of existing knowledge from other companies or
organizations. “Acquisition of existing know-how or inventions from other companies or
organizaitons for the development of new or significantly improved products or processes.”
Edit definition of E. Training for innovative activities to include coursework. “In-house or
contracted out training or coursework for your personnel for the development or introduction of
new or significantly improved products or processes.”
E-30
Attachment F
Improve definitions of F, G, and H.
•
Provide more detail in the definition for G to differentiate it from F and H.
•
Add “email and social media announcements” to F.
•
Add website design to a definition and consider proper placement in F, G, or H.
Consider revisions to the definition of I. Other.
•
Eliminate “tooling up.”
•
During later cognitive testing, allow a space for respondents to write in an example.
Qualitative data will inform the potential need for additional items.
Consider adding additional innovation activities. Consider “legal activities related to
innovation” and “research or personal development trips” as new items or examples within existing
definitions.
Eliminate the Total Investment column.
Within item A, reformat the sub-items.
•
Use bold instructions for A. follow-up questions. Revise 5A so respondents understand to
go to the follow-up questins if they answer yes to 5A. Bold “If you answered YES,...”
•
Revise the follow-up questions so that respondents can select “Yes” for just one or the
other. Add an instruction to check just one of the two boxes.
If you answered YES to A. In-house R&D,
did your company perform R&D during the
three years 2015 to 2017:
□ Continuously (your company had
permanent R&D staff in-house)
OR
□ Occasionally (as needed only)
E-31
Attachment F
2.6.
Q6. Innovation Activities – Abandoned/Suspended or Ongoing
Sixteen respondents answered this question and seventeen responded to probes about the question.
Reactions to Definitions
Some of the respondents did not appear to understand what “product or process innovation” within
the question stem meant, or appeared to be thinking of only “product” or “process,” but not both.
One said he was thinking of the things listed in Q2 but went on to explain that he thought it
referred to innovation activities, trying to get different things to market. Another found the question
a little unclear and went on to explain that when she read “process innovation” she thought about
how her company is always trying to improve their processes. Another said “process” can be in
reference to either product or service. Another was thinking of new products or new equipment to
make new products. Another thought of innovation overall, and did not separate between product
or process innovation.
However, several other respondents appeared to understand the question stem. When directly asked
what product or process innovation he was thinking of, one respondent explained he was thinking
back to items listed in the previous quesitons and of new and novel solutions for things the
company is currently working on. Another respondent felt the question was clear and that it asked if
something resulted in a product or process improvement. One respondent said he was thinking of
products and services, but more about service, the way his company executes the delivery of the
good.
Question Comprehension
We probed about the meaning of the phrase “Abandoned or suspended before completion.” Some
respondents were not sure if something qualified as “abandoned” or “suspended.” For example, one
asked if it is abandonment or suspension if you have an idea and you change it because of the
E-32
Attachment F
market or funding. This respondent had a difficult time answering the question since “innovation is
a perpetual process. You don’t abandon innovation,” because you have to adjust to the market.
Another respondent seemed unsure of his “No” response. The company had developed some
products that they might have introduced to the market, but decided instead to abandon them due
to not being completely happy with them. The respondent decided that perhaps in “their corner of
the world” these might have been innovation, but not “in the big scheme.”
Another respondent answered “Yes” even though she does not think of her innovation activities as
“abandoned or suspended.” She said they “evolve” or are “put on the back burner.”
One respondent confidently answered “No” to this part and explained that they cannot abandon
projects because they produce made-to-order products according to specs. The respondent
considered this customer service and problem solving, not process innovation.
We also probed about the phrase “Still ongoing at the end of 2017.” One respondent who easily
answered “Yes” to the first item also answered “Yes” to the second item, but said he did not know
how to answer it. This respondent explained how the company has a solid product development
process but that they do not put money into products they cannot complete. Another respondent
similarly answered “Yes” to both items but struggled more with the second item and explained that
because of the engineering and design focus of the company they always have a number of things
going on.
Another respondent answered “No” but asked “Is that innovative?” after explaining that they have
looked at expanding their market internationally.
Two respondents had no problem answering “No” to the first item and “Yes” to the second item,
explaining that they have not abandoned anything, and that their innovation is ongoing.
Some tended to think of innovation only in terms of the final release or implementation, so that
abandoning an innovation meant abandoning it after it had been released (e.g., because it failed to
develop a market) rather than abandoning something during the development process.
Some did not consciously seek to innovate, but rather sought to solve problems (their own problems
or customers’ problems), which may or may not involve innovation. For them, the term “innovation
activities” was not fully meaningful.
E-33
Attachment F
Relevance of Question
We did not probe directly about relevance for this question. However, based on the finding that the
majority of respondents answered “Yes” to at least one item, we judged that it is a relevant question.
Feasibility of Reporting for Three-Year Reference Period
We did not probe about the reference period for this question.
Response Accuracy and Consistency
For two respondents, it was unclear if their “Yes” responses were correct. One who answered “Yes”
to both items of the question explained that his first company was abandoned before completion
because they ran out of investor funding and his second company is still ongoing. Another
respondent answered “Yes” to the first item because he created a product but suspended it because
nobody was buying it.
While some respondents were unsure of their answers to the two items (as described above), other
respondents appeared to answer correctly and when probed, provided an explanation.
For the first item (Abandoned or suspended before completion), one respondent answered “Yes”
and explained that projects frequently fall apart or budgets fail. Another answered “Yes” and said
the company suspended the development of a product because of the lack of engineering resources
to develop it, but that it is “still on the table.” Another respondent appeared to accurately report
“Yes” based on some ideas the company started to develop, but then stopped after evaluating the
prototype. One respondent appeared to correctly answer “Yes” to this item based on his explanation
that they tested several concepts that were eventually abandoned because they were not funded. Still
another answered “Yes” and explained how the company just “looked at” creating something.
For the second item (Still ongoing at the end of 2017), one respondent answered “Yes” and
explained that some innovation is ongoing and related to multi-year projects. Another appeared to
answer “Yes” correctly based on his explanation that they started something this year but may not
complete and release it until next year, so it is “still ongoing.”
E-34
Attachment F
Q6 Recommendations
This question presented multiple issues that are likely to result in collecting inconsistent data: people
sometimes thought of innovation activities as final products rather than development activities, they
did not know how to handle activities that evolved or went into temporary abeyance, there was a
general reluctance to say they had stopped innovating, and they did not know how to report on
continuous improvement efforts. However, we understand that NSF wishes to minimize changes in
order to promote comparability with CIS data. One partial solution is to provide an introduction or
text box that defines innovation activities in more detail. Similarly, it may help to provide
supplemental definitions for both items.
NSF might consider the possibility of adding a question where respondents would be able to report
that they have continuous innovation due to the nature of their business and/or an R&D
department that is always innovating. Some respondents answered “Yes” to the second item because
of continuous innovation, not necessarily because there was something specific started during the
reference period that has yet to be completed.
2.7.
Q7. Public Financial Support for Innovation Activities
Seven respondents answered this questions and eight responded to some of the probes about the
question.
Question Comprehension
Of those respondents asked, most appeared comfortable when answering the question, but some
may have interpreted the question more broadly than intended, counting any kind of tax credit or
E-35
Attachment F
deduction regardless of the relevance to innovation. Two respondents said they understood it to be
asking if any money came from the government, federal or state. Two others said they could easily
answer it and that the answers were obvious to them. One respondent stated that she was hesitant to
answer since she does not file the company’s taxes. She answered “Yes” to both items of the
question because she knew that they receive tax credits for certain purchases and for installing solar
panels.
Relevance of Question
When asked if the question was easy or difficult to answer, three respondents said “easy” or “very
easy.” Another respondent felt it was difficult because she is not knowledgeable about the
company’s taxes. Another respondent did not say whether the question was easy or difficult to
answer, but said that it was a good question. One other respondent suggested separating out “local”
from “state.” (However, this change would probably make the question harder to answer for some
companies.)
Feasibility of Reporting for 3-Year Reference Period
We did not explicitly probe about the reference period for this question.
Response Accuracy and Consistency
Only three respondents explained their “Yes” answer(s). One answered “Yes” for Local or State
Government because they receive a tax increment financing for where their building is located.
Another answered “Yes” and provided the example of a tax credit for installing solar panels.
However, one respondent answered “Yes” for Federal Government because they have a subcontract
on a federal grant, which appeared to be an incorrect answer based on the instruction to “Exclude
R&D and other innovation activities conducted entirely for the public sector under contract.” This
respondent eventually explained that they also get federal funding in the form of a tax deduction for
R&D.
Q7 Recommendations
Consider adding a “Don’t know” response option. Some of the respondents may not know or
have access to this information.
E-36
Attachment F
Consider reformatting the instructions to make them more salient. To make the include and
exclude text more likely to be noticed and read, consider using bullet points to set them off from the
bold text. Also change “subsidised” to “subsidized” for the U.S. business population. For example:
7. During the 3 years 2015 to 2017, did your company receive any public financial
support for innovation activities from the following levels of government?
•
•
2.8.
Include financial support via tax credits or deductions, grants, subsidized loans,
and loan guarantees.
Exclude R&D and other innovation activities conducted entirely for the public
sector under contract.
Q8. Innovation Cooperation
Question Comprehension
Generally, the eight respondents who answered this question understood the question as intended
and provided relevant responses. An exception was one respondent: her organization shared office
space with a community-based organization (CBO). She answered “Yes” because she considered
this arrangement to be an innovative partnership in their town, which may not reflect the intention
of this question.
Despite understanding the overall purpose of the question, we suspect that many respondents did
not fully grasp the definition of partnership embedded in this question, which has implications for
question comprehension (see Reactions to Definitions).
E-37
Attachment F
Relevance of Question
Most respondents did not express concerns about the relevance of this question to their company,
and almost all answered “Yes” to Q8. These examples illustrate the relevance of cooperative
innovation activities to respondents’ companies:
•
A respondent’s company was hired to design a car dashboard for another company.
Although the other company decided not to sell the product, the respondent felt that the
dashboard was innovative and that the two companies collaborated.
•
Another respondent’s company asked undergraduates in a class to develop an app, which
the company then hoped to market. No money exchanged hands, and the app did not
work out. The respondent considered this a partnership with a university.
•
Another respondent’s company developed a product to track football speed and
collaborated with another company that makes similar equipment to manufacture it. They
are still trying to figure out how to work together to merge their skillsets to make a final
product.
The only potentially irrelevant response was, again, the respondent who described office-sharing
with a CBO (see above). Finally, one respondent answered “No” to Q8 but grasped the potential
relevance of the question to his company: he did not form partnerships as he did not want to share
intellectual property with other companies.
Reactions to Definitions 4
Innovation cooperation. No one expressed confusion about this definition, and a few volunteered that
the definition helped them to understand the overall question.
Both partners do not need to commercially benefit. Respondents did not express confusion about this
sentence.
Exclude pure contracting out of work with no active co-operation. Respondents expressed confusion about this
sentence. One suggested removing the clause “out of work.” Another disagreed with the premise of
the definition: he felt that contracting relationships necessarily involved cooperation (e.g., providing
the vendor with specifications). Finally, one respondent expressed no confusion with this sentence,
Due to time constraints, our data is limited on some of these definitions. For example, only 4 out of 8 respondents
were probed on “Exclude pure contracting out of work with no active co-operation.”
4
E-38
Attachment F
but interpreted it incorrectly to mean he should exclude all “for hire” partnerships (versus just
excluding “for hire” partnerships with no active cooperation).
Partner. Q8 excludes “for hire” partnerships with no active cooperation, but includes “for hire”
partnerships with meaningful collaboration on innovation activities. With this goal in mind, some
respondents’ definitions of partnership aligned with the sense of Q8, but a few of respondents’
definitions did not or were unclear. For example, a majority of respondents described “for hir”’
partnerships such as clients or contractors, but in a few cases, it was not clear whether these
relationships included active cooperation on innovation activities.
Respondents’ intuitive understanding of the word “partner” also varied. Again, one respondent
considered all contractors as partners because he felt that contracting fundamentally required
cooperation to conduct the work. On the other hand, other respondents stated that they would not
consider anyone that worked for hire as a partner (e.g., clients or contractors). In addition, one
respondent had a definition of partner that seemed overly broad, including another business that
merely shared office space with her organization. Another respondent narrowly defined a partner as
a company that sold another company’s products.
Finally, two respondents volunteered that they considered informal business relationships to be
partnerships, with one specifically stating that signed paperwork was not necessary to constitute a
partnership. In fact, one respondent found it challenging to answer this question because his
business had many informal connections to other businesses. For each relationship, he asked
himself, “Do they meet the threshold of 'partner' or are they an acquaintance?”
Feasibility of Reporting for 3-Year Reference Period
We did not specifically probe about the reference period for this question, but most respondents did
not volunteer any issues related to the 3-year reporting period.
Response Accuracy and Consistency
In the majority of responses to this question, we suspected inconsistencies related to the
respondents’ definition of “partner” (see above).
E-39
Attachment F
Q8 Recommendations
Please see the recommendations that follow Q9.
2.9.
Q9. Innovation Cooperation Partner and Location
Question Comprehension
Almost all of the seven respondents correctly identified the overall purpose of this question when
asked. Only one respondent indicated a possible misunderstanding, stating the purpose of the
question was to learn how free the company was to work with domestic or foreign entities.
Despite almost all respondents understanding the general purpose of this question, specific aspects
of this question created comprehension challenges.
Innovation cooperation partners. Q9 contains instructions to focus on “innovation cooperation partners”
(i.e., according to Q8, only partners that actively participated in innovation, not all partners). About
half of respondents correctly described partnerships that included an innovation component.
However, in several cases, it was unclear whether respondents understood they should include only
innovation-specific partnerships. For instance, one respondent may have been reporting any client in
the United States or abroad and not just those partners in innovation.
E-40
Attachment F
In addition, one respondent understood that he should include only partners actively involved with
innovation, but was uncertain whether one partner met this definition. His company made
innovative products and hired a partner to manufacture one piece in another country. Although the
overall product was innovative, the piece manufactured in the other country was not.
Location of partner. A majority of respondents correctly indicated the location of their partner. One
respondent stated the location reflected where the innovation occurred, as well as the benefactors of
the innovation (e.g., potential customers). Another respondent expressed doubt about which
location to indicate: he worked with a foreign company with locations in the United States, but he
only worked with its employees in the United States. He decided to put “U.S.” as location.
Categories. Almost all of the seven respondents expressed confusion about this question’s categories.
Broadly, several categories overlapped, were unclear, or provoked questions from respondents.
Below, we list the categories and the various issues identified by respondents.
Type of Cooperation Partner
Respondents’ Issues and Questions
A. Other affiliated companies
(legal entities under common
ownership)
A respondent did not understand how a co-owned company could be
a partner; he would consider it part of the company.
B. Suppliers of equipment,
materials, components, or
software
One respondent asked if this category included vendors, or if vendors
should go under C.
C. Clients or customers from the
private sector
One respondent pointed out that a company could fall under both C
and E (e.g., he worked with a partner sometimes as a client and
sometimes as a competitor).
D. Clients or customers from the
public sector
One respondent said that this option overlapped with Option H.
Another respondent asked what public sector meant: "Does that
mean government?"
One respondent did not like the word “competitor” here. By definition,
E. Competitors or other companies
he would not cooperate with a competitor. He suggested changing to
in your sector
“other companies in your sector.”
F. Consultants or commercial labs
None.
G. Universities or other higher
education institutes
One respondent suggested adding “K-12 schools” as well.
H. Government or public research
institutes
See comment above on D.
I. Private research institutes
None.
E-41
Attachment F
Finally, one respondent expressed concern that her partner did not fit neatly into the list (a
community-based organization); however, NSF would likely not consider this organization would to
be an innovation partner.
Relevance of Question
None of the respondents expressed concerns about the relevance of this question to their company,
and all respondents checked at least one box in Q9.
Feasibility of Reporting for 3-Year Reference Period
Q9 does not contain a reference period, but the preceding lead-in question does. We did not probe
on this topic, but no respondent volunteered concerns about the 3-year reference period. One
respondent worked at two companies during the 3-year period and picked one company for this
question.
However, one respondent described a possible challenge that could be exacerbated by the 3-year
reference period. Although his company did not have many partners, he predicted that large
companies with multiple partnerships might struggle to remember all their partners. (See also Q8,
where one respondent struggled to remember all his partners).
Response Accuracy and Consistency
The various issues noted above may have implications for respondents’ ability to consistently report
information into the categories that NSF has listed.
Q8 and Q9 Recommendations
Consider combining Q8 and Q9. Q8 is a filter for Question 9 and contains a skip pattern for Q9
if answered “No.” However, some respondents might not spontaneously think of all the entities
listed in Q9, yielding inconsistent responses between the two items. For example, one respondent
reported that his company co-owned another company on Q9, but stated that this company did not
occur to him when responding to Q8. Also, some respondents were confused about what
constituted a partner (e.g., if a client counted). Having a list of specific examples would be helpful to
resolve this confusion. Placing Q8’s detailed instructions immediately before Q9 will also reinforce
E-42
Attachment F
the definition of active innovation partner in Q9. For these reasons, we suggest combining Q8 and
Q9, adding an option for “no innovation cooperation partners.”
Consider adding an “other” option. One respondent did not know how to indicate a partner that
did not fit on the list (a community-based organization; CBO). In this particular case, the CBO may
not have met NSF’s definition of “partner.” However, this instance points to a larger issue: some
innovation partners may not be captured if they do not fit into the categories in this list. For this
reason, NSF may wish to consider adding an “other partner” option to the list. If NSF is open to
coding text fields, NSF could also consider allowing an open text response (“Other, please specify:
__________”) to obtain more information about these other partners.
Consider clarifying whether “for hire” contracting should be included in Q8 (and by
extension, in Q9). Currently the question states, “Innovation cooperation is active participation
with other companies or organizations on innovation activities,” but also instructs to “exclude pure
contracting out of work with no active cooperation.” The directions imply that contractors can be
included if they actively cooperated on innovation, but a few respondents assumed that all
contractors should be excluded.
To address this issue, we recommend explicitly stating that contracting with cooperation on
innovation should be included. An alternative phrasing is:
Include subcontractors who work actively with you on an innovation activity. Exclude work
that is contracted out with no active cooperation.
Consider clarifying that partnerships can be either formal or informal. Some respondents
included informal relationships, but others described only formal relationships. For this reason,
responses across respondents may be inconsistent without explicit instructions. If informal
partnerships should be included, some phrasing options are:
•
Both partners do not need to commercially benefit, and partnerships can be formal or
informal in nature.
•
Both partners do not need to commercially benefit. Include both formal and informal
partnerships.
Consider changing the “Type of cooperation partner” heading. A few respondents reported all
partners, not just those participating in innovation. Combining Q8 and Q9 (see above) should help
to reinforce the definition of innovation partnership. However, we also recommend adding the
E-43
Attachment F
word “innovation” to the left-side column heading as a reminder, or as a heading above the two
response columns. Some suggestions include:
•
Type of partner in innovation (above the left column); or
•
Location of partner in innovation (above response columns).
Consider clarifying the definition of “location of partner.” One respondent collaborated with a
foreign company, but only encountered their U.S.-based employees within their U.S.-based entity.
He expressed uncertainty about marking “U.S.” or “all other countries” (he decided on “U.S.” for
this company). Although this issue did not frequently occur in testing, it could point to a larger
concern: many companies have multiple offices around the world. For companies that have both
U.S. and non-U.S. offices, respondents may not select “U.S.” or “non-U.S.” consistently without
instruction. 5 We recommend adding instructions to clarify how to classify such companies (e.g.,
consider only the company headquarters, or consider the office with which you work most
frequently). NSF may wish to consider utilizing a similar approach to other NSF surveys, such as
HERD, when asking respondents to identify company locations.
Consider adding “Location of partner” as a column heading above “United States” and “All
other countries.” The majority of respondents thought that they should mark the location of their
partner. However, one person thought he should mark the location of the innovation. Adding a
column heading above the location columns will reduce the potential for confusion. If desired,
reinforce the definition of partner location within this heading (see above; e.g., “Location of partner
headquarters.”)
Consider reworking the response categories to improve clarity. Currently, the list of categories
contains two overlapping concepts: 1) the type of organization (e.g., university, company,
government; A, C, D, E, F, G, H, I), and 2) the nature of the partnership (e.g., a client, a vendor; B,
C, D, E). These are not mutually exclusive concepts. As one respondent pointed out, the same entity
can fall in multiple categories (e.g., a university that is also a client). In addition, two categories also
contain both the type of entity and the nature of the partnership (C and D), which is more specific
than the other categories. Together, these overlapping categories have the potential for confusion. If
desired, Westat would be happy to develop alternative constructions of this question (e.g., revised
non-overlapping categories). Alternatively, if NSF’s priority is to maintain comparability with other
The problem was exacerbated by a mistake in the design of the questionnaire, which did not allow respondents to
check both columns for a single line. Several wanted to check both columns because they had multiple partners, and
those that had a partner in both locations might have checked both columns.
5
E-44
Attachment F
questionnaires used internationally, then analysts should be advised that the same partner may be
listed in multiple categories. For clarity, it would be best to instruct respondents how to handle such
situations—i.e., whether to pick all applicable categories or to pick the category that best applies.
Otherwise, the meaning of the responses is likely to vary across respondents.
2.10.
Q10. New Organizational and Marketing Innovations
Question Comprehension
This question differed from previous questions by focusing on “new” rather than “new or
significantly improved.” The respondents differed in whether they noticed this difference and how
they responded. Following are examples of their interpretations.
•
One said the phrase meant “Brand new.” This respondent mentioned that variations or
upgrades could count as new if the company had purchased new software or tooling in the
process of making those changes. The respondent considered this part of keeping up with
their competition.
E-45
Attachment F
•
One respondent noted that he answered based on what was not necessarily new, but
basically a change from what they have done in the past.
A few of the respondents had a slightly broader interpretation of “new.” Three respondents seemed
to think along the lines of earlier questions that used explicit wording of “new or significantly
improved.” Their interpretations were as follows.
•
One respondent said that he was thinking about improvement and went on to say: “New
does not necessarily mean improvement, but that is the hope.”
•
One who works for a new company said he would consider a variation or an upgrade as
new, but he did not have to think of it at that level since they are still in a rapid growth
phase and are doing something new all the time.
•
One respondent said that for his company, a variation or an upgrade would in some
instances count as something new. For example, when they upgrade software on their
device with a new component, he would consider that a new software platform, even
though it is not a completely new system.
Another respondent was initially confused about whether the question was focused on new to the
company or new to the market. He explained that the practice may be new to them, but he would
not know for sure if it is new to others. He initially reasoned that the question seemed to be asking
about innovation in business practices, but he would not know this information. After further
thought, he reasoned the question might be about whether the business practice is new for the
company. He mentioned that part a (business practices for organizing procedures) of the question
did not identify whether the new practice is new for the company or new to the world.
Relevance of Question
None of the respondents said this question was irrelevant for their company. All answered “Yes” for
one or more parts of the question, providing further evidence that at least parts of the questions
were relevant for every company. One said that he was not the appropriate person to answer about
business practices for his company. His expertise and involvement are with product and process
innovations. This may be a general problem in larger companies: one person may not be qualified to
answer questions concerning all types of innovation.
E-46
Attachment F
Reactions to Definitions
When probed about their attention to and need for the definitions, several said the provided
examples were helpful. For example, one said the definitions “absolutely” clarified the questions,
especially for item g (Methods of pricing goods or services). Another said the examples helped
because they “narrowed in on choices.”
One said he did not read the text in the middle column; he only read the text in the left column that
was bold. He elaborated that if he had not understood the lead-in text for each item, he would have
continued on to read the “no-bold text.” He explained a preference to see text like this in bullet
points rather than full sentences; for him, that would make it easier to read and process the
information.
One participant suggested using bold font for the word “exclude” in item d (Aesthetic design or
packaging of a good or service), based on his thinking that some people would not notice that.
Feasibility of Reporting for 3-Year Reference Period
The reference period seemed to work well for this question; no respondents mentioned difficulty in
reporting for the 3-year period. One reapondent who works for a new company mentioned that it
was easy for him to report for 3 years, but it would have been harder to report for a short period,
such as 6 months.
Response Accuracy and Consistency
As time allowed, respondents were also asked to give examples for their “Yes” responses, to check
for reporting accuracy. Their examples seemed to qualify; respondents mentioned new ways of
partnering, new ways of training, new ways of branding, reorganizing the management structure or
management model, and changing a website.
Respondents seemed to correctly explain “No” answers, as well. They reasoned that a change did
not qualify because it was not new within the time period or was not really new. One decided that a
“tweak” to an existing method rather than a completely new method did not apply; for item C
(Methods of organizing external relations…), she said that the question was asking “about new
systems rather than just new partners.”
E-47
Attachment F
The respondent, who works for a new company and interpreted “new” more broadly (to mean a
variation or an upgrade), thought he was able to report acurately since they are still in a rapid growth
phase and are doing something new all the time. If the reference period had been shorter (e.g., the
last 6 months), he may have answered “No” to a few items of the question, but with a 3-year period
he was easily able to answer “Yes.”
One respondent was going to answer “Yes” for the first item of the question, then read the example
in parentheses and decided instead to answer “No.” Then for the remaining items of the question,
she answered in a limited way, only marking “Yes” if they had introduced the specific examples in
parentheses.
One respondent who answered “Yes” to all items of the question expressed the opinion that the
question items are too broad: “If any one of those areas is a yes, then the whole question [item] is a
yes.”
Q10 Recommendations
To help reduce the opportunity for misinterpretation and to further clarify that the focus of this
question is new for the company, consider making a slight modification to the wording of the
question stem. This is important since this question is placed later in the module than several
questions that differentiate new to the company vs. new to the market. Without explicit direction,
respondents may differ in their basis for answering this question. Three possibilities include the
following:
During the 3 years 2015 through 2017, did your company introduce new (new to your
company):
During the 3 years 2015 through 2017, were each of the following new (to your
company)?
During the 3 years 2015 through 2017, were the following introduced as new methods in your
company?
E-48
Attachment F
A larger issue is respondents’ inclusion of significant improvements in their responses, rather than
limiting themselves to what is new, even though every item but 10d includes the phrase “first time
use.” It may be helpful to add a note, such as an attention-grabbing box above the question items,
providing an instruction such as the following:
NOTE: This question asks you to consider only innovations that were new. Please do not count
those things that were significantly improved.
Consider using bold font for the word “exclude” in item d (Aesthetic design or packaging of a good
or service) of the question. It is important to draw attention to this key wording since it is in
parentheses, where those who typically avoid reading parenthetical information may overlook it.
This should help clarify what this item is asking for and improve reporting accuracy.
For consistency with questions that precede and follow this one, use capital letters for the individual
items of the questions.
E-49
Attachment F
2.11.
Q12. Factors that Hinder/Prevent Innovation
Question Comprehension
Of the five respondents probed specifically on what “hampering or preventing your company from
innovating” means, two were confused by a high rating for something that has a negative
connotation. One of the two asked how the question about preventing innovation is different from
preventing your company from performing. The respondent also said that the question was
confusing because, “It’s kinda like there’s a triple negative.” The respondent felt the scale was
confusing if “High” means that it hampered a lot. The other respondent who was slightly confused
said that “hampering or preventing your company from innovation” refers to things that get in the
E-50
Attachment F
way of running the business, and factors that keep you from running the company. This respondent
tended to think about the business overall, and not just the innovative aspect of the business.
The three other respondents who responded to a probe on the phrase (specifically the word
“hampering”) appeared to understand it. One said it refers to external factors interfering with her
progress. Another said “hampering” suggests slowing down, not stopping, the innovation process.
However, this respondent also said the word was not descriptive enough. Another respondent
explained her answers by using the words “hinder,” “prevent,” and “prohibit.”
Respondents found some parts of the question confusing. One did not understand item A (Lack of
internal finance for innovation); and said that innovation is not something you sell, but rather an
inspiration when you encounter a problem and come up with a solution. Another respondent was
confused by the parts of the question (A, B, D, E, and O) that included the phrase “lack of.” To
him, that signifies that he is not concerned at all; therefore, it is of low importance.
Relevance of Question
Respondents were asked which part(s) of the question were not relevant for their company. Three
respondents noted specific irrelevant parts. One respondent said H, L, M, N, and O did not apply to
her company or market. She answered “Not important” for all of them. She would have chosen
“Not applicable” if it were an option. She said, because there is a high demand for innovations in
her market, rating the importance of the lack of demand was difficult for her. Another respondent
also said items M, N, and O are not relevant to her business because innovation and competition are
the key components of her industry.
Another respondent said item B (Lack of credit or private equity) was not relevant to his business.
They are internally financed to maintain control of their scientific direction and do not seek out
private equity.
Feasibility of Reporting for 3-Year Reference Period
We did not probe on the reference period for this question, but none of the respondents mentioned
issues associated with reporting for a 3-year period.
E-51
Attachment F
Response Accuracy and Consistency
There were several instances in which the respondent’s explanation of the company’s situation did
not seem to match the response option chosen. For example, the respondent who said item B (Lack
of credit or private equity) was not relevant to his business gave the response “Low” rather than
“Not important.”
Six respondents provided examples of parts of the question that were most difficult for them to
answer. These tended to be the parts that they cited as hard to understand and/or included a
negative word. One respondent specifically mentioned that items A, B, D, E, and O were confusing;
these all included the wording “lack of.” To him, that signified that he is not concerned at all;
therefore, it would be of low importance.
Another respondent said item K (Legislation/regulation that lacked consistency across the United
States) was difficult to answer because the company has not had any exposure to anything that is
regulated differently from another state. Another respondent said she did not know how to answer
item H (Too much competition in your market) because it is not something that hinders you from
innovating, but rather is the reason you have to innovate. Two other respondents also had some
difficulty with item H; they were focused on whether or not the factor is important rather than
whether it is a factor in preventing innovation. One needed to re-read the question stem. The other
did not know what to check because the item did not seem applicable; they did not have too much
competition, so he could not say having too much competition was hampering or preventing
innovation. (This respondent answered “Low” for item H). This last respondent generally had a
problem with Q12 because he preferred to think about what is important to further innovation
rather than what is preventing innovation. He also said item E was difficult because they did not lack
collaboration partners, and answered with “Low” to the item.
One respondent understood the question but said items A, B, and C were the most difficult to
answer because his company only does work when the client wants it. As a business that does
contract work, it is up to the client, not the company. One other respondent voiced some confusion
over items L and N, although his answers appeared to match his situation. For item L he said,
“What’s low demand for innovations in your market?” He explained that any market can benefit
from innovations and that you usually don’t know what those innovations will be until they are
created. He answered “Low.” For item N (No need to innovate due to very little competition in
your company’s market) he was confused and said, “If your company doesn’t have competition, you
don’t have a business.” He answered “Medium.”
E-52
Attachment F
When probed about other factors that affect their company’s ability to innovate but were not
included on the list, five respondents offered the following comments and suggestions.
•
One said that having to train employees because of their low skill set is a hindrance to
their ability to innovate. Higher skill sets would improve their ability to innovate. This
respondent also said lacking “local suppliers and support entities” could be a hindrance to
innovation.
•
Another respondent suggested adding the lack of access to tools, such as software.
•
One respondent said a major obstacle is not the regulation itself, but rather the “bizarre
interpretations on the part of the regulators and people inspecting the company premises.”
•
Another respondent said what hampers their innovation is when another company claims
to sell their product, but actually does not sell it, and then captures the market and figures
out how to make the product.
•
One respondent explained that innovation is hindered when “big players” in the aerospace
market can afford to make mistakes, and smaller companies cannot. Another obstacle is
that some countries pay international companies to innovate.
Q12 Recommendations
Modify the wording of the question stem, to clarify the concept and make the reporting task
easier. We note that when NSF and Westat met to review preliminary results of the interviews, NSF
explained that for the testing, they had merged two questions into Q12, to help minimize the
number of questions within the innovation module. NSF suggested separating items L–O again.
After meeting with NSF, Westat offered several alternative ways to word the question stem and the
anchors, to eliminate the confusion about rating importance of negative factors. We also suggest
using scale anchors that range from low (“Not at all”) to high (“A great deal”). Consider using the
version shown below.
The recommended wording for the question stem should also work well with items L–O, so a
separate question will not be needed.
12. During the 3 years from 2015 through 2017, how much did each of the following
factors hinder or prevent your company from innovating?
How much the factor hindered innovation
E-53
Attachment F
Not at all
A little
A
moderate
amount
A great
deal
Another option to consider for the question is:
12. During the 3 years from 2015 through 2017, to what extent did each of the
following factors hinder or prevent your company from innovating?
Extent to which the factor hindered
innovation
Did not
hinder or
prevent
innovation
2.12.
A slight
extent
A
moderate
extent
A great
extent
Q13. Effects of Legislation or Regulations on Innovation Activities
E-54
Attachment F
Question Comprehension
Of the nine respondents who responded to probing about the ease or difficulty of the question, six
provided insight into their thought process while answering the question. One of those six
respondents appeared to answer according to what “could be” and not the actual effect on his
company. The respondent felt it was a very difficult question because most of the listed legislations
and regulations could have either a positive or a negative impact. They could have either viewed it as
a constraint or as an opportunity to work around the constraint. The respondent suggested adding a
column to capture a response that the effect could go either way. This respondent answered
“Stimulated innovation” for all items and seemed to think in general terms about what could be. For
example, he said that since they are a software company, “Environmental” would not affect them,
but yet could be an opportunity to be more lean and efficient.
Four of the six respondents sometimes mentioned difficulties. Several said it was because items did
not apply to their companies. In those cases, respondents answered appropriately with “Not
relevant.” Specifically, one respondent said he had difficulty with “Operational work safety” because
that is not a factor in his field. He answered appropriately with “Not relevant.” Another said
“Environmental” was difficult to answer because it does not apply to her company. She appeared to
understand and answer appropriately with “Not relevant.” Another respondent thought, at first
glance, the question was not relevant to his company, but then recalled a proposal related to product
safety. The respondent appeared to understand and answered either “Stimulated innovation” or
“Not relevant” for each item. One respondent answered “Created uncertainty” and “Generated
excessive burden” to the item “Product safety/consumer protection” but said she had some
difficulty answering because she did not want to appear as if she did not care about the topic.
Probing revealed that one respondent clearly understood the question and provided varying answers
for different items. He noted that none of these types of legislation or regulation caused his
company to innovate, so he did not check “Stimulated” for any.
Relevance of Question
Two respondents wrote in a response for the “Other, specify” row. One wrote in “Enforcement of
existing regulations” and “Licensing” and said that there is nothing about building regulations.
Another respondent said to add “Competition” because she was thinking about the list as categories
of things that stimulate innovation, not necessarily as part of regulations, but removed
“competition” when she realized the listed items were specifically related to legislation.
E-55
Attachment F
Number of Responses to Allow
Of the nine respondents who answered this question, seven of them selected one response per row.
Two respondents selected two or more responses per row for some parts of the question. One of
these two respondents selected two per row for three of the six items. She said she was unable to
narrow it down on those three (A, B, and D) and selected “Created Uncertainty” and “Generated
Excessive Burden” because they go hand-in-hand.
Of those who selected one response per row, their response did not necessarily mean that the factor
worked in only one direction. For example, when probed, one respondent said it would be possible
to select more than one answer, but usually one outweighed the others, so he selected one answer
per row. Another respondent did not have a problem selecting just one per row and felt the choices
were appropriate. One respondent said she did not think about selecting more than one for each
row and just selected the best one that applied. However, for some she was thinking about
stimulating innovation and lost sight of the legislative side. In hindsight, she felt that for some items
it would be better to select more than one.
One other respondent, who did not realize that the response format allowed for selecting options
representing both stimulation and burden, suggested adding a column that shows legislation or
regulations could go either way.
Q13 Recommendations
Change the question stem word order to put the reference period first. To be consistent with
all other questions with the same reference period, start the question stem with “During the years
2015 through 2017,…”
Revise the question text to help clarify the intent of the question and facilitate response.
•
Emphasize the actual effect on the respondent’s company, not the possible effect. Change
the wording of the question stem from “...assess the effect....” to:
During the years 2015 through 2017, what was the effect of the following types of
legislation or regulations on your company’s innovation activities?
•
For consistency with other questions in the module, letter each row of the matrix.
E-56
Attachment F
•
If relevant for measurement purposes, consider adding two additional types of regulations:
building and licensing (as shown in the example below).
•
Consider whether or not the definitions of “Licensing regulations” and “Building
regulations” overlap with existing items, and if not add them to the list of Q13 items.
•
Add the explicit instruction “Check all that apply for each row.”
Example:
Q13. During the 3 years from 2015 through 2017, what was the effect of the following types of
legislation and regulations on your company’s innovation activities?
(Check all that apply for each row)
2.13.
Q14. Intellectual Property
Question Comprehension
Of the six respondents who answered this question, three did not have difficulty and the other three
either had to change an answer or commented on parts of it that were unclear. One of the three
respondents who had some difficulty asked if “trade secrets” is about licensing someone’s process or
formula, and described a chemistry-based techology product they developed. Another respondent
said he would not have known what a “utility patent” is without the description provided in
parentheses, even though his company applied for patents a lot in the past.
E-57
Attachment F
Relevance of Question
Although we did not administer probes about the relevance of the question and response options,
two respondents provided insight into the relevance of the question. One respondent felt the
question was “limited” because it does not ask if the compnay was awarded any kind of patent or
trademark, only if it applied or registered. He understood that the question was asking about
intellectual property, but felt it left out material transfer agreements, MOUs, business-related
contracts, CRADAS (cooperative research and development agreements, which are formal
documents to do research with a government agency). Another respondent felt that the bulk of his
company’s work was not covered in this question since everything his company does is public
domain under government contacts.
Feasibility of Reporting for 3-Year Reference Period
Although probes about the reference period were not administered, one respondent explicitly
mentioned and understood it. The respondent whose company developed a chemistry-based
technology product explained that he answered “No” to “trade secrets” because it was developed
before the 2015−2017 window.
Response Accuracy and Consistency
One respondent had a reporting issue: he changed his answer from “Yes” to “No” for item D
(Claim copyright). He said everything his company does is public domain with government dollars,
and suggested that it should be added as a response option.
Q14 Recommendations
Consider providing a brief definition of “trade secrets.” This may help improve reporting
accuracy.
Consider adding more types of intellectual property agreements. If appropriate for
measurement purposes, consider adding material transfer agreements, MOUs, business-related
contracts, and CRADAS (cooperative research and development agreement).
E-58
Attachment F
E-59
| File Type | application/pdf |
| Author | Jennifer Crafts |
| File Modified | 2017-11-14 |
| File Created | 2017-11-14 |