Annual SBO Findings & Recommendations

Attachment D - ASE Cognitive Testing Report

Findings and Recommendations from
Cognitive Testing of the Proposed Annual
Survey of Business Owners
FINAL REPORT
Prepared for:
Patrice Norman (EWD)

Prepared by:
Jennifer Beck
Kristin Stettler

Data Collection Methodology and Research Branch
Economic Statistics and Methodology Division
U.S. Census Bureau

April 13, 2015

Attachment D - ASE Cognitive Testing Report

Table of Contents
Background .................................................................................................................................................................4
Research Methodology ..............................................................................................................................................4
Participants .........................................................................................................................................................4
Materials .............................................................................................................................................................4
Procedures..........................................................................................................................................................4
Financing and Credit Market Experiences Findings and Recommendations .............................................................5
Finding #1: Money put into the business (Qs 1-5) .............................................................................................5
Finding #2: Attempt to gain funding (Q6) ..........................................................................................................7
Finding #3: Not applying for funding (Q7) ..........................................................................................................8
Finding #4: Customer location (Q8)....................................................................................................................9
Finding #5: Importance of owning company (Q9) ........................................................................................... 10
Finding #6: Profits or losses (Q10) ................................................................................................................... 10
Finding #7: Five-year vision (Q11) ................................................................................................................... 11
Finding #8: Negative effects on profitability (Q12) ......................................................................................... 11
R&D and Innovation Supplement Findings and Recommendations ....................................................................... 12
Finding #9: Research and Development and Improvement activities (Qs 13, 19, and 20) ............................. 12
Finding #10: Cost and purchase of R&D services (Qs 14 and 15) .................................................................... 13
Finding #11: Itemized R&D purchases and services paid by others (Qs 16 and 17) ....................................... 14
Finding #12: R&D workers (Q18) ..................................................................................................................... 16
Management Practices Supplement Findings and Recommendations ................................................................... 16
Finding #13: Addressing problems (Q2) .......................................................................................................... 16
Finding #14: Performance indicators (Qs 3 and 4) .......................................................................................... 17
Finding #15: Productions/service targets (Qs 5 and 5a) ................................................................................. 18
Finding #16: Employee performance (Qs 6 and 7) .......................................................................................... 18
Overall Findings and Recommendations ................................................................................................................. 19
Finding #17: Response time............................................................................................................................ 19
Finding #18: Web response ............................................................................................................................. 19
Finding #19: New title ..................................................................................................................................... 19
Appendix A: Cognitive Research Methodology ....................................................................................................... 21
Appendix B: Cognitive Interview Protocol............................................................................................................... 22
Appendix C: Questionnaire Used During Cognitive Testing .................................................................................... 27
Annual SBO Final F&R
Page 2 of 48

Attachment D - ASE Cognitive Testing Report
Appendix D: Final Revised Questions ...................................................................................................................... 39

Annual SBO Final F&R
Page 3 of 48

Attachment D - ASE Cognitive Testing Report

Background
The Survey of Business Owners (SBO) staff, in conjunction with the Kauffman Foundation, has proposed
conducting the Survey of Business Owners on an annual basis. Currently, the SBO is fielded every five years. The
proposed annual survey would include the current SBO questions along with some additional questions on
business activities and potential supplements. The Data Collection Methodology and Research Branch (DCMR)
conducted cognitive testing on the proposed additional questions and the proposed supplements. This report
contains preliminary findings based on this initial round of testing.

Research Methodology
Participants
In February and March 2015, we conducted fifteen interviews with business respondents in two U.S. cities.
While some of the respondents were the owners of the businesses, others were not. The latter respondents
tended to be in charge of accounting and financial reporting at the business. Table 1 indicates the number of
respondents we interviewed in each city.
Table 1: Number of respondents in each location.
Location

Number of
Respondents

New York, New York

7

Seattle, Washington

8

Total

15

Materials
The proposed Annual SBO will have additional core questions that all respondents will answer. In addition, there
will also be periodic supplements that focus on specific business activities. For this round of testing, there were
two proposed supplements. The Research & Development and Innovation supplement contained questions
about R&D activities, the cost associated with those activities, and any other improvements to the business’
processes or services. The Management Practices supplement contained questions about handling problems
and how the business monitored the business and its employees. Appendix B contains the protocol for testing,
Appendix C contains the questions we tested, and Appendix D contains the final, revised questions.
Procedures
We conducted our interviews using concurrent and retrospective think-aloud techniques with follow-up probes.
We gave respondents copies of the proposed questions, along with verbal instructions to complete it, but not to
report actual data if doing so would be inconvenient or time-consuming. Most respondents were able to answer
the majority of the questions without consulting their records. For the few items that required calculations, we
had respondents indicate if they had that information available in the records and how they would go about
deriving their answers from those records.
Annual SBO Final F&R
Page 4 of 48

Attachment D - ASE Cognitive Testing Report
As respondents went through the questions, we observed how they navigated from question to question and
page to page. We asked non-leading questions about their impressions of the layout and instructions. We nondirectively probed their responses until we had a thorough and accurate understanding of their perceptions.
Appendix A contains a more detailed discussion of cognitive interview methodology,
In this report, we present our findings and recommendations and the resolutions for the core questions
(Financing and Credit Market Experiences), the R&D and Innovation supplement, and the Management Practices
supplement, separately.

Financing and Credit Market Experiences Findings and Recommendations
Overall, the proposed questions were not overly problematic for respondents. However, respondents did have
problems with some of the questions. In this section, we discuss our findings for the individual questions.
Finding #1: Money put into the business (Qs 1-5)
For the most part, respondents understood the intent of these questions. Most respondent also were familiar
with the different sources of funding mentioned in the questions. However, some respondents were not familiar
with the term “angel investors” in Q5. The information was also available in the respondents’ records.
However, despite not finding any comprehension problems with the questions, respondents did feel that the
questions were not entirely clear. In particular, the lengthy instructions after Q1, which inform the respondent
of the content of the next four questions, confused some of the respondents (and some respondents did not
read them). The instructions were not helpful, nor did respondents feel that they were necessary. Likewise,
respondents ignored the repeated, “Your best estimate is fine. Enter 0 if no financial capital was received from
xx.”
Q1 asks for a total of the subsequent items. For respondents, it made more sense to ask for the specific sources
of funding first, and then ask for the total. (One respondent noted that Q1 was so long that it was off-putting to
her. She noted that if the survey were optional, she would put it aside if she saw Q1 first.)
Respondents discussed different ways that their businesses handle finances. Most of our respondents did not
indicate that any of the owners put any money into the business. However, some inclusions may have been
incorrect. One respondent, whose business recently became a public company with an IPO, included the amount
of the IPO, as they now “own” the company. It is unclear if publically traded funds should be included in this
item.
One respondent included money from an account set up to contribute to employee retirement that he
“borrowed.” The money provided funding for company operations, and the business replaced it once more
funds came in. It is unclear if this money, which still belongs to the company, is an “outside source.
Two respondents entered amounts in Q1, and then, when they had completed Qs 2-4, realized that their Q1
amount was incorrect. One of these entered the same amount in both Qs 2 and 4, as it was $10,000 of her own
savings, but through a bank. The other respondent originally thought that he should report as of December 31,
so he entered $200,000 ($350,000 line of credit, but had paid back $150,000). After he answered Qs 2-4, and I
pointed out the discrepancy, he realized that he should have included the full $350,000 he reported in Q4, plus
Annual SBO Final F&R
Page 5 of 48

Attachment D - ASE Cognitive Testing Report
the $25,000 he and his wife had put into the business, plus the $55,000 they had borrowed from friends (paid
back), for a total of $430,000.
Several respondents asked about reinvesting into the company. One respondent was not sure if $0 was an ok
answer because she put the profits back into the business. She suggested using “Exclude profits reinvested in
the business,” in the question. Another respondent suggested clarifying “not earned from the business”.
Respondents also mentioned that they have credit cards or lines of credit that the business used for expenses.
However, because the business always paid off the cards and did not carry a balance, respondents omitted these
expenses in their answers. One respondent indicated that she sometimes owed $100,000 on a business line of
credit, sometimes $0. She ended up writing in a range, “$50,000 – 150,000” and writing in “Outsider debt.”
Respondents seemed to interpret this question to be referring to more “long-term” financing through credit
card balances, and not just using credit for expenses.
In addition to some confusion about what to report, respondents also noted that the reference period was
confusing. Respondents were unsure if they should report what the business owed on December 31, excluding
the money that the business paid back before the reference date, or the total amount borrowed earlier in the
year.
Finally, some of these questions were worded slightly differently. One respondent noted this difference. In Q4
and Q5, the examples are part of the question text, rather than part of a parenthetical reference after the
question. In Q5, the item wording is different from Qs 2 and 3. Q5 refers to “total amount of financial capital”
instead of the “total amount of money” in the other questions.
Recommendations:
 Consider changing the order of Qs 1-5, having respondents provide money from specific
sources, and then having respondents either verify an automatically calculated total, or
calculate the total themselves. The latter recommendation would avoid confusion if there were
sources of money not accounted for in the detailed questions.
 Clarify how to report for money borrowed during the year, but paid back before the end of the
year.
 Clarify how respondents should handle credit cards and other revolving account balances.
 In Q3, add “this business” before “received.”
 In Q4, change “the business” to “this business” (be consistent with this throughout the
questionnaire).
 In Q5, replace “financial capital” with “money.”
 Consider adding a definition for “angel investors.”
Resolutions:
 Q1 (total funding) was deleted because the sponsors determined that having a total was not
important.
 Instructions were added to Q3 to clarify how to report on business lines of credit and credit
cards.
 A definition was added for “angel investors.”
Annual SBO Final F&R
Page 6 of 48

Attachment D - ASE Cognitive Testing Report



The sponsors adopted the wording changes that maintain consistency among these questions.
The sponsors also decided to add a new question on government grants. The new question
reads:
For 2014, what was the total amount of money this business received from government grants (such
as the Small Business Innovation Research (SBIR) and/or Small Business Technology Transfer (STTR)
programs)?
Finding #2: Attempt to gain funding (Q6)
Although familiar with the funding sources in the question, respondents had some difficulties with this question.
Because of its complexity, respondents spent some time reading it and attempting to figure out how to answer
it. Respondents also noted the redundancy between this question and the preceding questions.
Only two respondents reported attempts to get funding from any of the sources in the question. For those who
reported they did not attempt to get funding, these responses seemed inconsistent with the answers that they
provided in Qs 1-5. Some respondents seemed to view this question as concerning the active seeking of funding,
rather than passively transferring money from different accounts (as in the case of using money set aside in a
retirement fund).
One respondent pointed out that there could be situations where the company could receive more money than
originally requested, especially for grants and research funding. The question does not allow respondents to
account for receiving more money.
Respondents also noted that their business might not have records of the amounts of funding requested, only
the amount of funding received. In these cases, the respondents would report the amount received, giving an
inaccurate picture of the gap between needed resources and acquired resources.
Some respondents were confused about how to complete the table. The question does not specifically state
what they are to do. One respondent entered $10,000 in the Yes column, but nothing in the Total amount
column. Another respondent thought that she should enter dollar amounts in all of the columns.
The sources of funding were also problematic. Several respondents noted that grants from sources such as NSF,
CDC or NIH were missing from this list. Some respondents were unfamiliar with “trade credit” and
“crowdfunding platform”. Other respondents suggested adding “Indigogo” as an additional example for
crowdfunding. There were also questions about whether the two Federal government programs listed were the
only ones of interest or just examples. If the latter is true, additional programs should be listed as examples.
Finally, there was some confusion about credit cards. Respondents interpreted the term “credit card”
differently. Some respondents thought it meant only business credit cards, while others thought it could refer to
personal credit cards used for the business. In addition, some respondents thought they should only include
newly opened business credit card accounts, while others thought they should also include existing cards that
had their credit limits raised during the year. Is the intent to collect information on business credit cards only? If
not, then the type of credit cards respondents should consider should be clarified.

Annual SBO Final F&R
Page 7 of 48

Attachment D - ASE Cognitive Testing Report
Recommendations:
 If this question is redundant with Qs 1-5, consider deleting either this question or Qs 1-5.
 If keeping this question, break it down into two questions: a filter questions asking respondents
if they attempted to get any of the sources of funding and a follow-up question that asks for the
outcome of the attempt only for respondents that attempted to get a source of funding.
 Reword “Federal government” category to clarify if the two programs are just examples or the
only ones of interest.
 Consider collecting information on exceeding the requested funding amount.
 Consider adding grants as a category (perhaps with NSF and NIH as examples).
Resolutions:
 This question was not intended to be redundant with the previous questions. The sponsors were
interested in collecting information on new funding relationships. Accordingly, the wording of
the question was clarified to reflect new funding relationships:
In 2014, did this business attempt to establish any new funding relationship (e.g., loans, investments,
or gifts) with any of the following sources?




Due to programming issues that limit the number of questions that can be added to this survey
during the 2015 cycle, this question was unable to be broken down into two less complex
questions. The sponsors indicated that there will be opportunity for content changes in future
waves of the survey. We strongly recommend implementing this change after the 2015 survey
cycle.
The “Federal Government” category was reworded and included examples of the programs
respondents should be considering.

Finding #3: Not applying for funding (Q7)
Respondents understood this question. However, as noted in comments on an earlier draft of these questions,
this question is complex. Using a filter question to determine if the respondent applied for any funding would
reduce the amount of information respondents need to process in order to answer the question, especially
when the business did not attempt to garner funding.
Additionally, one respondent raised the concern that this question does not allow him to indicate that he was
able to use alternate forms of financial support for his company. He did not feel that “additional funding was no
longer needed” adequately covered his situation of using retirement funding to temporarily cover company
finances. His company needed funding but it came from within the company. This respondent ultimately
selected the “other” response option and specified that it was “owner funded.”

Annual SBO Final F&R
Page 8 of 48

Attachment D - ASE Cognitive Testing Report
Recommendations:
 Add a filter question about needing but choosing not to apply for funding. If the respondent
answers “yes,” to the filter question The filter question and follow up could read:
At any time during calendar year 2014, did this business need more financing, but the owner(s) chose not
to apply?
Yes
No
Why did this business choose not to apply for more financing?
Resolutions:
 Due to programming issues that limit the number of questions that can be added to this survey
during the 2015 cycle, this question was unable to be broken down into two less complex
questions. The sponsors indicated that there will be opportunity for content changes in future
waves of the survey. We strongly recommend implementing this change after the 2015 survey
cycle.
Finding #4: Customer location (Q8)
Respondents had problems with this question. They found the response categories to be vague and not mutually
exclusive. Respondents were not sure what “region” meant and often included their city and the surrounding
cities, counties, and states as part of the region. Because these response options are vague and over-lapping,
respondents are unlikely to provide quality data. Respondents thought that “region” was too vague of a term,
leading to them perceiving these response options as not being orthogonal.
In addition, one respondent may have answered this question incorrectly. Because her business is in the food
service industry in a high-tourist area, she indicated that she had international customers. Through credit card
sales, the respondent is able to track information on the locations from which customers come. However, all of
these sales are within her business. It is unclear if she should be considering these sales as “international” if they
are taking place within her store.
For some respondents, answering this question would be fairly time consuming. One respondent actually
looked up all her clients on her computer, and counted them all up, in order, getting out a calculator to calculate
the exact percentages. It took her about five minutes. Another respondent indicated he would have gone into
Excel and make some calculations in order to answer this question.
The question wording was also awkward, asking respondents to “estimate the percentages” before they see the
response task. Several service business owners also noted that they do not have “customers,” they have
“clients.”

Annual SBO Final F&R
Page 9 of 48

Attachment D - ASE Cognitive Testing Report
Recommendations:
 Revise the response options to be mutually exclusive. The new options could be:
Same region as the business
Outside the region but within the U.S. (domestic)
Outside the U.S. (international)
 Reword the question to eliminate the awkward phrasing. The new wording could read:
During 2014, where were this business’ customers or clients located? Please estimate the
percentage in each category, totaling to 100%.
Resolutions: The sponsors adopted the recommended wording and response-option changes.
Finding #5: Importance of owning company (Q9)
This question was problematic because it assumes that the respondent is not only the owner, but that there is
only one owner. For respondents that are not the owners of the company or in situations where there is more
than one owner, this question is difficult to answer. Two respondents indicated that because they were not the
owners of the company, they would leave this question blank. Two other respondents, for whom there was
more than one owner at the company, indicated that they would photocopy this question and solicit answers
from the owners. However, one respondent pointed out that the different owners could have different answers
for this question. The respondent would have no way of accounting for those differences. The respondents who
were owners, however, did not have any problems answering this question. One of these respondents did
double check some categories because she did not think the column headings were discrete enough. Two
respondents noted that “balance work and family” and “flexible hours” were related categories, and should
occur in sequence in the response options.
Recommendations:
 Ask this question only of respondents who are also the owners of the business.
 Revise the question and the response options so that they do not address a specific owner.
 Move “balance work and family” up to after “flexible hours.”
 Change “your company” to “this business”.
Resolutions:
 Because this question is only applicable to owners, it was moved to the owner-specific question
sets in the core SBO.
 “Balance work and family” now follows “flexible hours” in the question.
 “Your company” was changed to “this business.”
Finding #6: Profits or losses (Q10)
For the most part, this question was not problematic. Respondents understood the question and were able to
answer it, although some respondents commented that “just break even” seemed like an awkward phrase.
However, one respondent did struggle with his answer. He would answer the question differently based on
whether he was considering the question from an accounting perspective, rather than a business development
standpoint. Because his company used funding from their employee retirement account, from an accounting
standpoint, the company operated at a loss. However, in terms of business development and the revenue that
Annual SBO Final F&R
Page 10 of 48

Attachment D - ASE Cognitive Testing Report
the projects coming in will generate, the company shows a profit. Ultimately, the respondent decided that he
should answer from an accounting perspective, to make sure that his responses were consistent with his
reporting to the IRS.
Recommendation:
 Consider changing the wording to “For 2014, did this business have profits, losses, or break
even?”
Resolution: The sponsors adopted the recommended wording change.
Finding #7: Five-year vision (Q11)
Respondents understood this question and were able to answer it. However, respondents expressed the desire
to be able to select more than one response option. They did not view the response options as mutually
exclusive. For example, the company could be both larger in sales, but also sold to another company. This
question also assumes the owner is the respondent. It also refers to “your company” rather than using the term
“this business,” as is used in all other questions.
Finally, one respondent indicated that the terms in this question might be too limiting. He pointed out that sales
could increase, but the overall profitability of the business could decrease, especially if the sales are for products
that are reduced in price to remain competitive. Although he was able to answer that he wanted the company
to increase in sales, he qualified it by saying that he wanted the company to increase in profitability.
In addition, the answer categories were wordy, repeating “Company will” at the beginning of each response
option.
Recommendations:
 To capture the increase in profitability that an increase in sales implies, revise the question and
the response options to refer to “profitability.”
 Consider making this question “check all that apply.”
 Consider rewording to “Where would the owner(s) like this business to be in five years?”
 Delete “Company will be” in front of answer categories.
Resolutions: This question was eliminated from the final questionnaire.
Finding #8: Negative effects on profitability (Q12)
This question was highly problematic for respondents. Although they could understand the question and the
events that they were evaluating, the rating scale was inconsistent with the question, something multiple
respondents pointed out when struggling to read and answer it. The question stem asks about the occurrence of
events that may have affected the business. However, the Likert scale refers to the importance of the events. As
a result, respondents’ answers seemed to be focused on how important they viewed these events to
profitability, rather than if the events actually affected their business’ profitability. This inconsistency between
the question and the response scale is likely to lead to the over-reporting of negative events, as respondents
tended to view many of the events as “somewhat” or “very” important.

Annual SBO Final F&R
Page 11 of 48

Attachment D - ASE Cognitive Testing Report
Additionally, two respondents pointed out that the “lost or slow sales” item does not necessarily apply to some
service-industry businesses. For example, consulting firms do not rely on sales, but instead on bids for projects.
Revising this category to be more general would make it more applicable to different types of businesses.

Recommendations:
 Revise the Likert scale to be consistent with the question. The scale would include: “No impact,”
“Some impact,” and “A lot of impact,” or something similar.
 If it is not important to capture the degree of affect the negative experiences might have had on
the business, consider making this a Yes/No question where respondents indicate only if an
event occurred. The wording could read:
In 2014, did each of the following negatively impact the profitability of this business?


Consider revising “slow or lost sales” to encompass more business activities. The new option
could read: “slow business or lost sales.”

Resolutions:
 This question was changed to a “yes/no” format, adopting the recommended wording change.
 The response option was revised to reflect the recommended wording change.

R&D and Innovation Supplement Findings and Recommendations
In this section of the report, we discuss the findings from the R&D supplement questions. Because only six
respondents screened into the supplement, we were unable to thoroughly test the questions. However, we did
observe some issues. We discuss the issues with the questions, individually.
Finding #9: Research and Development and Improvement activities (Qs 13, 19, and 20)
Question 13 was highly problematic. Although respondents understood the intent of the question and were able
to answer it, the question screened out all but five respondents from the R&D and Innovation questions.
However, walking the other respondents though Q19 and Q20 revealed that their businesses did engage in
improvements to products, services, and processes that would go unreported because of the skip pattern. For
example, one business, a bakery, sends workers to France to learn new techniques for making pastries. This
assignment often leads to the development of new products to sell in the store. The business also had
purchased new machinery to increase the efficiency with which they can produce their products. However,
because the screener question was limited to patents and scientific research, this respondent would not have
answered subsequent questions that collected this information.
In addition, one respondent, who worked at an architecture firm, screened into the R&D supplement because
her business garnered a patent for a piece of furniture. However, as she read the remaining questions in the
supplement, she noted that the remaining questions seemed inapplicable because the patent was not for
“scientific” activity.
Is the intent of this supplement only to collect information on pure R&D activities or is the intent to capture
other innovations and process improvements? Based on the results of this testing, limiting some of the
Annual SBO Final F&R
Page 12 of 48

Attachment D - ASE Cognitive Testing Report
questions to businesses that only engage in pure R&D activity would not capture other business activities
focused on offering new goods and services or introducing new ways of conducting business and producing
products.
We also observed one respondent reporting R&D activities that took place outside of the reference period. In
previous years, her business had applied for patents, although it had not during the calendar year referenced in
the questions. This respondent indicated that she would have initially checked “None of these R&D activities.”
However, she then decided to report on some R&D her other company had done last year, seemingly in an
effort to be helpful during the cognitive interview.
There were also some problems with the wording of the questions. In Q19, the response category, “No, we
didn’t offer any goods or services for sale,” seemed unnecessary and was confusing to some respondents. The
last category, “did not make any innovations to the goods or services sold by this business,” refers to
“innovations” rather than “improvements,” which is used in the question stem and in Q20.
For Q20, respondents noted that the categories seemed very production- and product-based and did not seem
to apply to their service businesses. Respondents thought about this question before answering because it
seemed to apply to only production-based businesses.
Recommendations:
 If the intent of this supplement is to capture both pure R&D activities and process
improvements and innovation, then this question should not contain a skip pattern, or the
questions should be reordered to include Q19 and Q20 as part of the screener.
 Turning all of the response options for these three questions into Yes/No questions might make
it easier to ensure that respondents read each answer category. If revising the questions in this
way, reword the questions. For example, Q13 would read: In 2014, did this business do each of
the following research and development (R&D) activities?
Resolutions:
 The sponsors adopted the recommended re-ordering, placing Q19 and Q20 before Q13.
 The questions were revised to the “yes/no” format.
Finding #10: Cost and purchase of R&D services (Qs 14 and 15)
With the exception of the one respondent who screened into the supplement because her business garnered a
patent for a piece of furniture, these questions were not problematic for the respondents that answered them.
Overall, respondents were able to understand and answer these questions. However, respondents did note that
the wording of these questions was awkward. The question refers to the total cost “paid” for R&D activities, and
most respondents thought of the total cost “spent on” R&D activities.
Recommendations:
 Because so few respondents answered this question, we recommend further testing.
 To address the awkward wording, reword both questions. The new questions could read:
In 2014, what was the total amount this business spent on R&D activities?
and
Annual SBO Final F&R
Page 13 of 48

Attachment D - ASE Cognitive Testing Report
Of the total amount this business spent on R&D activities in 2014 (fill from Q14), what was the amount
spent to purchase R&D services from others?
Resolution: The sponsors adopted slightly different wording for these two questions. The revised
wording reads:
In 2014, what was this business’ total cost for R&D?
and
In 2014, of the total ($XXXXXX) R&D costs, what was the amount used to purchase R&D services from
others?
Finding #11: Itemized R&D purchases and services paid by others (Qs 16 and 17)
We observed some problems with both of these questions. For Q16, one respondent questioned why he should
include software and equipment in this question. He wondered why capital expenditures should be included in
R&D expenses, but would provide the information.
The biggest problem with this question was respondents not understanding what information to include and
exclude in their calculations. These two questions instruct the respondent to exclude the “costs of services
performed by others.” Based on this instruction, respondents should be basing their calculation on the total they
report in Q14 minus the total they report in Q15. Because this is an arbitrary number that respondents neither
have in their records nor have been asked to provide, respondents were incorrectly basing their calculation on
the total R&D costs reported in Q14, rather than excluding the cost of services paid by others. For example, one
respondent originally put 20% for “employee payroll” in Q16, which corresponded to the total R&D costs she
reported in Q14. When the interviewer pointed out that the responses should add up to 100%, she corrected it
to “100%.” However, she was basing her calculations on the total costs of R&D, including costs paid to others.
Respondents who noticed the instruction and attempted to follow it found these questions confusing.
Finally, several respondents noted that asking for tenths of a percent was burdensome and unnecessary, as
these would likely be estimates rounded to the nearest whole number.
Recommendations:
 Because so few respondents answered these questions, we recommend further testing.
 Because respondents are unlikely to “back out” costs paid to others from their total R&D costs
when making calculations in Q16 and 17, we recommend basing these two questions on the
total R&D costs. Reword the questions to reflect this change. The questions could read:
Of the (fill from Q14) this business spent on R&D activities in 2014, what percent was spent for each
of the following?
And
Of the (fill from Q14, this business spent on R&D activities in 2014, what percent was paid by each of
the following?
Annual SBO Final F&R
Page 14 of 48

Attachment D - ASE Cognitive Testing Report



Delete the tenths in the answer categories.

Annual SBO Final F&R
Page 15 of 48

Attachment D - ASE Cognitive Testing Report
Resolutions:
 The sponsors decided to keep both questions. Respondents will still need to make the
calculations.
 Respondents will round their answers to the nearest percent.
Finding #12: R&D workers (Q18)
This question is potentially problematic. First, the response task is unclear to respondents. Some respondents
were confused about how to complete the table. The format of this question is inconsistent with the response.
Respondents are asked to check a box to indicate a number, rather than provide a number. One respondent
pointed this out, saying, “‘how many’ is a number.” Because of this format, one respondent wrote in the number
of employees in each box, rather than checking each box. A second respondent initially thought of doing the
same. One respondent noted that it would be easier to provide a number.
Two respondents indicated that they would be unable to include any information on contractors or
subcontractors. Because these workers are not paid employees, businesses do not track the number of workers
the contractor uses to complete the work in the agreement. This finding is consistent with similar proposed
questions on other surveys. Contracts are not handled through payroll, but instead through invoices. A business
pays the contract fee, but does not have information on the number of employees working.
Finally, respondents found the March 12 reference date surprising, as it seemed arbitrary and was inconsistent
with the “calendar year 2014” reference period in the other questions. It seemed as though most respondents
were not thinking of that actual pay period when they answered the question, but just providing an average
number of workers.
Recommendations:
 Delete the inclusion of contractors and subcontractors.
 To eliminate respondent confusion on how to answer this question, revise the question to make
the response open-ended for each category. Responses can be coded into ranges during postcollection processing.
Resolutions: The sponsors adopted the recommended changes to the question.

Management Practices Supplement Findings and Recommendations
In this section, we present our findings from the Management Practices supplement. Because the focus of this
round of testing was the R&D supplement, in cases where the respondent had limited time, we did not
administer these questions. Only 11 of our 15 respondents answered these questions. Several respondents
worked at companies with fewer than five employees, and screened out of the remaining questions. For Q1,
one respondent suggested clarifying whether this only refers to W2 employees or to 1099s as well.
Finding #13: Addressing problems (Q2)
This question was problematic for some respondents. For some respondents, the response options were not
adequate for describing their business’ approach to handling problems. In particular, respondents felt that the
“fixed it and took action to make sure it did not happen again” was too narrow to fit situations that are out of
Annual SBO Final F&R
Page 16 of 48

Attachment D - ASE Cognitive Testing Report
the business’ control. For example, a piece of machinery could break down and action could be taken to fix it,
but a similar problem could arise again on the same or different piece of equipment. In this situation, it is
impossible to guarantee that the problem will not happen again. Respondents did not feel that the “fixed and
took no further action” described the situation, as there would always be some effort to mitigate the problem in
the future. Respondents wanted to see a response option that reflected efforts to mitigate but not entirely
prevent some problems from occurring again in the future.
In addition, the types of problems respondents were thinking about and including in their responses varied. One
respondent noted she was thinking of “handling unhappy clients.” Other respondents noted that they were not
thinking of a particular incident, were not thinking of 2014 specifically, or were just reporting their general
policies. Several respondents noted that the examples were very production-oriented and thought that it would
be helpful to have a service or project management example as well.
Recommendations:
 Revise the “We fixed it and took action to make sure that it did not happen again” to reflect
efforts to mitigate, but not eliminate the occurrence of a problem. The new response option
could read:
“We fixed it and took action to reduce to possibility of it happening in the future.”
 Correct the sentence structure of the third response option. It should read:
“We fixed it, took action to make sure that it did not happen again, and had a continuous improvement
process to anticipate problems like these in advance.”


Consider adding a service or project management example.

Resolution:
 At the time this final report was issued, revisions to these questions were still on-going.
Finding #14: Performance indicators (Qs 3 and 4)
There were some problems with these questions. Respondents’ answers to Q4 differed based on their
interpretation of “monitoring” performance indicators. For example, one respondent interpreted Q3 to be
asking about formal monitoring of performance indicators, answering that her business monitored performance
on a yearly basis for the follow-up, Q4. However, she indicated that the owners of the company informally
monitor the business on at least a monthly basis. Is it the intent of this question to capture both formal and
informal monitoring? If respondents are interpreting “monitoring” to refer to a formal process, respondents will
underreport the frequency of such activity. Some respondents also did not realize they could mark all that apply
and struggled to select just one category.
In addition, there should be a skip pattern if respondents indicate that their business does not monitor key
performance indicators in Q3. They should not answer a question about the frequency with which those
indicators are monitored if they do not monitor them.

Annual SBO Final F&R
Page 17 of 48

Attachment D - ASE Cognitive Testing Report
Recommendations:
 Include a skip pattern in Q3 for respondents who indicate that their business does not monitor
performance.
 If the intent of Q3 is to capture both formal and informal monitoring, change the wording of the
question to indicate that respondents should be considering both processes. The new question
could read:
“In 2014, how many key indicators were monitored, either formally or informally, at this business?”
Resolution:
 At the time this final report was issued, revisions to these questions were still on-going.
Finding #15: Productions/service targets (Qs 5 and 5a)
For the most part, these questions were not problematic for respondents. However, some respondents
indicated that “short term” and “long term” were vague. These respondents were still able to answer the
questions despite the uncertainty, and did not offer any clear feedback on how to clarify the vague reference
period.
Some respondents seemed unclear about the intent of this question and how these targets differed from key
performance indicators. Testing this question on more respondents could help determine how to clarify them.
One respondent noted that this should not be a “Mark (x) all that apply” question, because a business “can’t
have two main focuses."
Recommendations:
 Because some respondents were uncertain about the time-frame references in the question,
we recommend further testing for this question in future rounds of testing.
 This question should be “Check (x) only one.”
Resolution:
 At the time this final report was issued, revisions to these questions were still on-going.
Finding #16: Employee performance (Qs 6 and 7)
We observed some problems with this question. One respondent did note that it might be difficult to check the
third category, because “family connections” seems very negative. He initially was “worried about checking that
one,” but did end up selecting that response. Because this response option may be uncomfortable for some
respondents, they may be reluctant to select it. One respondent noted that “tenure” is a very academic term
and several suggested using the term “loyalty” or “length of service” instead of “tenure”.
Several respondents thought Q7 was odd because it assumes that a business had an under-performing
employee in 2014. Many respondents were thinking about the entire time they have owned their business, not
just 2014. In addition, there seemed to be variation in what was meant by “under-performing”. Several
respondents noted they had only ever fired employees due to theft or other major issues, not merely “underperformance”.

Annual SBO Final F&R
Page 18 of 48

Attachment D - ASE Cognitive Testing Report
Recommendation: Consider adding a category “Did not have any underperforming employees” or
adding a screener Yes/No question to determine if there was an underperforming employee before
asking this follow-up. If choosing the latter change, the new question could read:
In 2014, did this business have any under-performing employees?
Yes
No
Resolution:
 At the time this final report was issued, revisions to these questions were still on-going.

Overall Findings and Recommendations
In this section, we discuss general findings about the survey and the response process.
Finding #17: Response time
Overall, respondents did not think that these questions would be difficult to fill out. However, they did feel that
some items that required calculation would make the response process longer. Estimates to fill out the entire
survey ranged from twenty minutes to one hour. In particular, respondents gave estimates that ranged from five
to forty-five minutes to calculate their answer to the customer location questions. If respondents screen into
the R&D questions, the questions that require calculations will also significantly increase the response time.
Recommendation: None needed.
Finding #18: Web response
In 2012, SBO was fielded as a “Web-only” survey. We asked our respondents how they would feel about the
Web-only response mode in future survey cycles. Respondents did not express any objections to answering the
questionnaire on the Web. However, respondents did indicate that having access to a paper copy of the
questionnaire was important to help record their responses and maintain a copy of what they reported for their
records.
Recommendation: Provide online access to a copy of the questions, instructions, and the answers that
respondents provide.
Finding #19: New title
The sponsors were considering renaming the survey. Accordingly, we asked respondents which of the four
alternatives they preferred. Twelve of our Fifteen respondents preferred “Annual Survey of Business Owners
(ASBO),” and two respondents preferred “Annual Survey of Business Characteristics and Activities (ASBCA).” One
respondent preferred both ASBO and ASBCA equally. Both of these titles made the survey seem relevant to the
respondents, and they felt it described the content well. Respondents expressed dislike of the other two
possible names. Respondents did not like the mention of “entrepreneur” or “entrepreneurial,” as they did not
feel like they were entrepreneurs. Seeing this title made the survey seem irrelevant.

Annual SBO Final F&R
Page 19 of 48

Attachment D - ASE Cognitive Testing Report
Recommendation: We recommend choosing either “Annual Survey of Business Owners (ASBO)” or
“Annual Survey of Business Characteristics and Activities (ASBCA)” as the new survey name.
Resolution: Despite respondents' dislike of the use of “entrepreneur,” the survey will be
renamed, “Annual Survey of Entrepreneurs.”

Annual SBO Final F&R
Page 20 of 48

Attachment D - ASE Cognitive Testing Report

Appendix A: Cognitive Research Methodology
Cognitive research is used in survey methodology “(a) to understand the thought processes used to answer
survey items, and (b) to use this knowledge to find better ways of constructing, formulating, and asking survey
questions” (Forsyth and Lessler, 1991).1 Cognitive interviews traditionally focus on the four steps of
Tourangeau’s (1984)2 cognitive response model: comprehension, retrieval, judgment, and
communication/reporting. Comprehension refers to the respondent’s interpretation and understanding of the
question’s language, structure, and grammar. In order to answer the question, a respondent must understand
what information is being requested on the survey. Retrieval is the step where relevant information is obtained,
either from records or from memory. The next step, judgment, describes the respondent’s evaluation of the
completeness or relevance of the data obtained. It is here that estimates are made based on partial or
incomplete data. The last step, communication or reporting, deals with mapping the response to the answer
space provided and possibly altering the answer.
While Tourangeau’s model is suitable for household and social surveys, the establishment survey setting
presents additional factors that must be considered. First, instead of or in addition to a reliance on memory,
establishment surveys rely heavily on records and the information contained within them. Second,
organizations tend to have distributed knowledge. Some people are experts in one type of information, while
others keep information about something else. Third, competing priorities, both for the organization and the
individual(s) completing the questionnaire, mean that the survey sometimes does not receive the amount of
attention that researchers and data collectors would like. Finally, organizations regularly authorize only a few
individuals to release data. If the data provider is not authorized to release the data, an additional step must be
added to the response process. Tourangeau’s model was expanded by Sudman et al (2000)3 to account for
these factors.

1

Forsyth, B.H. and Lessler, J.T. (1991). “Cognitive Laboratory Methods: A Taxonomy.” In Measurement Errors in Surveys, P.P. Biemer,
R.M. Groves, L.E. Lyberg, N.A. Mathiowitz, S. Sudman (eds). New York: John Wiley & Sons, Inc.
2
Tourangeau, R. (1984). “Cognitive Sciences and Survey Methods.” In Cognitive Aspects of Survey Methodology, T.B. Jabine, M.L. Straf,
J.M. Tanur, and R. Tourangeau (eds). Washington, DC: National Academy Press.
3
Sudman, S., Willimack, D.K., Nichols, E., and Mesenbourg, T.L. (2000). “Exploratory Research at the U.S. Census Bureau on the Survey
Response Process in Large Companies.” Paper prepared for presentation at the Second International Conference on Establishment
Surveys, Buffalo, NY.
Annual SBO Final F&R
Page 21 of 48

Attachment D - ASE Cognitive Testing Report

Appendix B: Cognitive Interview Protocol
Annual Survey of Business Owners
Questionnaire Testing Protocol

Research Questions to Address:
 Are the questions straightforward and answerable?
 Are there any items that cause the respondents problems? What can be done to fix those problems?

Introduction:






Explain purpose of meeting: to understand the process used to answer questions, to get feedback on
proposed changes
Meeting with many businesses of different types and of different sizes. Not everybody keeps their
records the same way, so some questions will seem odd and/or obvious.
Explain that we are not testing the respondent – we only want information
Structure of meeting: general and specific questions about the form itself, the reporting organization
and also how R reported data
Permission to record discussion? Have R sign consent form.

Before the Questionnaire:
 What types of services does this business provide?
 What is your role in the company? What kind of responsibilities do you have?
 What would be your role in the process for responding to this questionnaire? (Gather data? Check
responses? Fill out?)
Annual Survey of Business Owners
Have the respondent begin filling out the questionnaire. Pay attention to how respondent navigates the
questionnaire and if they appear to be having any difficulty with the questions.
General Probes:
 Can you tell me what you are thinking?
 What does that term mean to you? (If R is not familiar with the term) Is there another term you would
use?
 Can you say more about that?
For questions that include the “None” instruction, pay attention to how respondents answer them.
Finance and Credit Market Experiences
Capital funding from owners:
 In your own words, what is this question asking?
 What does “personally put into this business” mean to you?
 How would you answer this question?
 What kinds of thing are you including in your answer?
Annual SBO Final F&R
Page 22 of 48

Attachment D - ASE Cognitive Testing Report
Family, friends, and employees:




In your own words, what is this question asking?
How would you answer this question?
What kinds of thing are you including in your answer?

Formal debt financing:
 In your own words, what is this question asking?
 How would you answer this question?
 What kinds of thing are you including in your answer?
Outside investors:
 In your own words, what is this question asking?
 What does “angel investor” mean to you?
 What does “venture capitalist” mean to you?
 How would you answer this question?
 What kinds of thing are you including in your answer?
Attempts to obtain capital funding:
Pay careful attention to how the respondent navigates the matrix in this question.
 In your own words, what is this question asking?
 Is it clear or not clear how you are supposed to answer it?
 How would you answer this question? Probe the categories the respondent does select.
 Are there any of these sources of funding that are unfamiliar to you? (If yes) Which ones?
 (If R chooses “other”) Can you tell me about what made you pick “other?
Need but did not apply for financing:
Pay careful attention to how the respondent handles the complexity of this question.
 In your own words, what is this question asking?
 Is it clear or not clear how you are supposed to answer it?
 How would you answer this question? Probe the categories the respondent does select.
 (If R chooses “other”) Can you tell me about what made you pick “other?
Customer location:
 In your own words, what is this question asking?
 Is it clear or not clear how you are supposed to answer it?
 How would you answer this question?
 What kinds of customers are you thinking about for each category? Can you give me some examples?
 How would you come up with these numbers? Is the information available in your records?
Reasons for owning company:
Pay careful attention to how the respondent handles the complexity of this question.
 In your own words, what is this question asking?
 Is it clear or not clear how you are supposed to answer it?
 How would you answer this question?
 (If R chooses “other”) Can you tell me about what made you pick “other?
Impacts on profitability:
Pay careful attention to how the respondent handles the complexity of this question.
Annual SBO Final F&R
Page 23 of 48

Attachment D - ASE Cognitive Testing Report





In your own words, what is this question asking?
Is it clear or not clear how you are supposed to answer it?
How would you answer this question?
(If R chooses “other”) Can you tell me about what made you pick “other?

Innovation and Research & Development Module
Research & Development:
 In your own words, what is this question asking?
 What does “research and development” mean to you? Can you give me some examples of the kinds of
activities that you think would be research and development?
 How would you answer this question?
 Are all of the things this question asks about familiar or not familiar to you? Are there any you are not
sure about?
 What kinds of thing are you thinking about when you are answering?
Total R&D costs and Cost of purchasing:





In your own words, what is this question asking?
How would you answer this question?
What kinds of thing are you including in your answer?
Where would you find the information to answer this question? Is it readily available?

Type of R&D costs:
 In your own words, what is this question asking?
 How did you arrive at your answer?
 Where would you find the information to answer this question? Is it readily available?
Funding of research and development:
 In your own words, what is this question asking?
 How did you arrive at your answer?
 (if R selects a source) Can you tell me about these different sources of funding?
 Where would you find the information to answer this question? Is it readily available?
R&D employment:
 In your own words, what is this question asking?
 How did you arrive at your answer?
 Where would you find the information to answer this question? Is it readily available?

Annual SBO Final F&R
Page 24 of 48

Attachment D - ASE Cognitive Testing Report
Goods and services improvements:
Pay careful attention to how the respondent handles the complexity of this question.
 In your own words, what is this question asking?
 Is it clear or not clear how you are supposed to answer it?
 How would you answer this question?
 (If R chooses “other”) Can you tell me about what made you pick “other?
Production improvements:
 In your own words, what is this question asking?
 How would you answer this question?
 What kinds of thing are you including in your answer? Probe the categories the respondent does select.
 Are there any of these activities that are unfamiliar to you? (If yes) Which ones?
Improved production processes:
 In your own words, what is this question asking?
 How would you answer this question?
 What kinds of thing are you including in your answer? Probe the categories the respondent does select.
 Are there any of these activities that are unfamiliar to you? (If yes) Which ones?

IF THERE IS TIME, GO THROUGH THESE QUESTIONS
Management-Practices Module
Continuous improvement:
 In your own words, what is this question asking?
 How would you answer this question?
 What does “service or production problem” mean to you? Can you give me an example of something
like that?
 What kinds of things were you thinking of when you were answering it? Can you tell me about the
problem that arose?
Performance indicators and frequency of monitoring:
 In your own words, what is this question asking?
 What does “performance indicator” mean to you?
 (If indicates that they monitored them) Can you tell me about the performance indicators you
monitored?
Employee promotion, performance, and monitoring:
 In your own words, what is this question asking?
 Can you tell me how you came up with your answering? What kinds of things were you thinking about?

Annual SBO Final F&R
Page 25 of 48

Attachment D - ASE Cognitive Testing Report
ASK OF ALL RESPONDENTS
Experiences with Web surveys
My next questions are about the SBO in general.









SBO only offered Web response in 2012. How did you feel about that?
Can you tell me how you handled not getting a paper copy? Did it have an effect or not have an effect
on how you handled filling this out?
Thinking about the questions we have been going over today, how do you feel about not getting a paper
copy of the form?
Do you want to keep copies of what you filled out and submitted for your records? (If yes) In what
format would you want these copies to be?
What has been your experience with using either the Web to gather and report your information?
What have been some of the positive aspects of reporting on the Web?
What have been some of the negative aspects of reporting on the Web?
Are there things that could be done to address the problems you encountered with these Web surveys?

Wrap-up Questions
 About how long did it take you to complete the SBO before?
 How long do you think it would take you to complete this one?
 Any other feedback you would like to give us on these questions or your experiences?

Alternative Names
 Which of the following names would you prefer?
Annual Survey of Business Owners (ASBO)
Annual Survey of Entrepreneurs (ASE)
Annual Survey of Business Characteristics and Activities (ASBCA)
Entrepreneurial Dynamics Survey (EDS)

Thank respondent(s) for his/her (their) assistance.

Annual SBO Final F&R
Page 26 of 48

Attachment D - ASE Cognitive Testing Report

Appendix C: Questionnaire Used During Cognitive Testing

Annual Survey of Business Owners
Financing and Credit Market Experiences
1. For calendar year 2014, what was the total amount of money put into this business?
(Include private investments, loans, owner equity, and personally borrowed funds. Your best estimate is
fine. Enter zero if no financial capital was put into the business in calendar year 2014). You will be asked
to separate the amount into 4 categories: Owner financing (debt or equity), Insider financing (funding
from friends and family, debt or equity), Outsider debt (bank loans, credit card balances, business lines of
credit), and Outsider equity (Venture Capital, Angel Investment, Business investment).
$_________________

2. For calendar year 2014, what was the total amount of money that the owner(s) personally put into
this business?
(Include investments from personal savings, home equity, and personally borrowed funds.) Your best
estimate is fine. Enter zero if no financial capital was received from owner(s).
$_________________

3. For calendar year 2014, what was the total amount of money received from family, friends, and
employees?
(Include investments from personal savings, home equity, and personally borrowed funds.] Your best
estimate is fine. Enter zero if no financial capital was received from any of these sources.
$_________________

4. For calendar year 2014, what was the total amount of money that the business borrowed from a bank
or other financial institution, including business loans, a business credit card carrying a balance, or a
business line of credit?
Your best estimate is fine. Enter zero if no financial capital was received from any of these sources.
$_________________

5. For calendar year 2014, what was the total amount of financial capital received from angel investors,
venture capitalists, or other businesses in return for a share of ownership in this business?
Your best estimate is fine. Enter zero if no financial capital was received from any of these sources.
$_________________

Annual SBO Final F&R
Page 27 of 48

Attachment D - ASE Cognitive Testing Report
6. During calendar year 2014, did this business attempt to get funding (e.g., loans, investments, or gifts)
from one or more of the following sources?
Yes,
Total
Yes,
attempted
amount
received
Yes, but
but did
sought
total
received
not
from each
amount of only part
receive
source
the
of the
any of the
funding
amount
funding
No requested requested requested
Other owner(s) (if applicable)
Family, friends, or employees
Banks, credit unions, or other financial institutions
Home equity loans in name of business owners
Credit cards
Trade credit
Angel investors
Venture capitalists
Other investor businesses
Crowdfunding platform (Prosper, Kickstarter, etc.)
Federal government's Small Business Technology
Transfer Program (STTR) or Small Business Innovation
Research Program (SBIR)
Other (Specify:
)

Annual SBO Final F&R
Page 28 of 48

Attachment D - ASE Cognitive Testing Report
7. At any time during calendar year 2014, did this business need more financing, but the owner(s) chose
not to apply for any of the following reasons? Mark X all that apply.
__ No, never needed additional financing and chose not to apply
__Yes, needed additional financing but chose not to apply because:
___Did not think business would be approved by lender
__ Did not want to accrue debt
__ Decided the financing costs would be too high
__ Preferred to reinvest the business profits instead
__ Felt the loan search/application process would be too time consuming
__ Decided the additional financing was no longer needed
___Decided to wait until funding conditions would improve
___Decided to wait until company hit milestones to be in stronger position to raise funds
__ Other (Specify: _______________________________________________)

8. During calendar year 2014, please estimate the percentages where [NAME BUSINESS]’s customers
were located.

________Same city or county as the business
________Same region (such as nearby counties or states) as the business
________Nationwide (Domestic)
________Outside the United States (International)

100%

Annual SBO Final F&R
Page 29 of 48

Attachment D - ASE Cognitive Testing Report

9. How important to you are each of the following reasons for owning your company?

Not
Important

Somewhat
Important

Very
Important

Wanted to be my own boss

()

()

()

Flexible hours

()

()

()

Opportunity for greater
income/Wanted to build
wealth

()

()

()

Best avenue for my
ideas/goods/services

()

()

()

Couldn't find a job/Unable
to find employment

()

()

()

Working for someone else
didn't appeal to me

()

()

()

Always wanted to start my
own business

()

()

()

Balance work and family

()

()

()

An entrepreneurial friend or
family member was a role
model

()

()

()

Other
(Specify:______________)

()

()

()

Annual SBO Final F&R
Page 30 of 48

Attachment D - ASE Cognitive Testing Report
10. For calendar year 2014, did the business have profits or losses or just break even? (check one)
( ) Profits
( ) Losses
( ) Break even

11. Where would you like your company to be in five years?
( ) Company will be larger in terms of sales
( ) Company will have about the same amount of sales
( ) Company will be smaller in terms of sales
( ) Company will be purchased by another company
( ) Uncertain
( ) Other (Specify): _________________________________________________

Annual SBO Final F&R
Page 31 of 48

Attachment D - ASE Cognitive Testing Report
12. In 2014, which of the following have negatively impacted the profitability of your business?

Not
Important

Somewhat
Important

Very Important

Access to financial capital

()

()

()

Cost of financial capital

()

()

()

Finding qualified labor

()

()

()

Taxes

()

()

()

Slow or lost sales

()

()

()

Customer or clients not making
payments or paying late

()

()

()

The unpredictability of business
conditions

()

()

()

Other
(Specify:____________________
)

()

()

()

Annual SBO Final F&R
Page 32 of 48

Attachment D - ASE Cognitive Testing Report
Innovation and Research and Development

13. In 2014, did this business do any of the following research and development (R&D) activities? Mark X
all that apply.
(Include R&D activities that this business performed, others paid this business to do, or this business paid
others to do.)
__ Conducted work that might lead to a patent
__ Developed and tested prototypes that were derived from scientific research or technical findings
__ Produced findings that could be published in academic journals or presented at scientific conferences
__ Applied scientific or technical knowledge in a way that has never been done before
__ Created new scientific research or technical solutions that can be generalized to other situations
__ Conducted work to discover previously unknown scientific facts, structures, or relationships
__ Conducted work to extend the understanding of scientific facts, relationships or principles in way that
could be useful to others
__ None of these R&D activities -> SKIP TO END

14. In 2014, what was the total cost paid for R&D activities?
(Include labor paid for employees, temporary staffing, contractors, subcontractors, independent
contractors, or outside consultants; materials, equipment, software, or other supplies purchased; and
money spent for rent, utilities or other overhead. Reasonable estimates are acceptable.)
$_________________

15. Of the total cost paid for R&D activities in 2014, what was the total amount used to purchase R&D
services from others?
(Include labor paid for contractors, subcontractors, independent contractors, or outside consultants.)
$_________________

Annual SBO Final F&R
Page 33 of 48

Attachment D - ASE Cognitive Testing Report
16. In 2014, what percent of this business’s total cost of R&D activities consisted of the following
purchases? (Include only costs of R&D services performed by this business, do not include costs of
services performed by others)
Employee Payroll
___ __ ___ . ___ %
__ None
___ Don’t know
Equipment Purchases

___ __ ___ . ___ %

__ None

___ Don’t know

Software and Licensing Purchases

___ __ ___ . ___ %

__ None

___ Don’t know

Other R & D Expenses

___ __ ___ . ___ %

__ None

___ Don’t know

100%
17. In 2014, what percent of the costs of R&D services performed by this business was paid by the
following sources (do not include costs of services performed by others)?
This business
___ __ ___ . ___ %
__ None
___ Don’t know
Another U.S. business

___ __ ___ . ___ %

__ None

___ Don’t know

U.S. college or university

___ __ ___ . ___ %

__ None

___ Don’t know

U.S. nonprofit organization

___ __ ___ . ___ %

__ None

___ Don’t know

U.S. federal government
(Include R&D grants)

___ __ ___ . ___ %

__ None

___ Don’t know

U.S. state or local government
(Do not include public schools.)

___ __ ___ . ___ %

__ None

___ Don’t know

___ __ ___ . ___ %

__ None

___ Don’t know

Other (Specify:
_______________________)

__________________________________________________________________________

100%

Annual SBO Final F&R
Page 34 of 48

Attachment D - ASE Cognitive Testing Report
18. During the pay period that included March 12, 2014, how many of each type of the following workers
worked on R&D activities?

None 1-4

5-9

1019

2049

5099

100Don't
499 500+ Know

Owner(s)
Paid Employees
Other Paid Workers - Include
labor paid for temporary
staffing, contractors,
subcontractors, independent
contractors, or outside
consultants.
Unpaid Workers/Interns

19. Over the past three years (2012,2013,2014) were any of the following improvements made to the
goods and/or services sold by this business? Mark X all that apply.
(Do not include adaptation or customization of a good or service for a specific customer’s needs.)
__ No, we didn’t offer any goods or services for sale in the last three years  SKIP TO 20
Yes, improvements were made such that:
___ Sold a new good or service that no other business has ever offered before
__ Sold a new good or service that this business has never offered before
__ Improved a good’s performance by making changes in materials, equipment, software or other
components
__ Developed a new use for a good or service
__ Added a new feature to a good or service
__ Made it easier for customers to use a good or service
__ Don’t know
__ Did not make any innovations to the goods and/or services sold by this business

Annual SBO Final F&R
Page 35 of 48

Attachment D - ASE Cognitive Testing Report
20. Over the past three years (2012, 2013, 2014), were any of the following improvements made to the
production processes used by this business? Mark X all that apply.
__ Applied a new way of purchasing, accounting, computing, maintenance, inventory control, or other
support activity
__ Reduced costs by changing the way a good or service was distributed
__ Upgraded a technique, equipment, or software to significantly improved a good or service
__ Made a significant improvement in a technique or process by increasing automation, decreasing
energy consumption, or using better software
__ Decreased production costs by improving the materials, software, or other components
__ Changed a delivery method to be faster or more reliable
__ Don’t know
__ Did not make any improvements to the production processes used by this business

Annual SBO Final F&R
Page 36 of 48

Attachment D - ASE Cognitive Testing Report
Management Practices
1. Are there 5 or more employees working at this business?
( ) Yes
( ) No  SKIP TO END
2. In 2014, what best describes what happened at this business when a service or production problem
arose? Mark X for answer that applies.
(Examples: Finding a quality defect in a product or piece of machinery breaking down.)
__ We fixed it but did not take further action
__ We fixed it and too action to make sure that it did not happen again
__ We fixed it and took action to make sure that it did not happen again, and had a continuous
improvement process to anticipate problems like these in advance
__ No action was taken

3. In 2014, how many key performance indicators were monitored at this business? Mark X all that apply.
Examples: Metrics on production, cost, waste, quality, inventory, energy, absenteeism and deliveries on
time.
__ 1-2 key performance indicators
__ 3-9 key performance indicators
__ 10 or more key performance indicators
__ No key performance indicators
4. During 2014, how frequently were the key performance indicators reviewed by managers at this
business? Mark X all that apply.
__ Yearly
__ Quarterly
__ Monthly
__ Weekly
__ Daily
__ Hourly or more frequently
__ Never
Annual SBO Final F&R
Page 37 of 48

Attachment D - ASE Cognitive Testing Report

5. In 2014, what best describes the time frame of service or production targets at this business? Mark X
all that apply.
Examples of service or production targets are: production, quality, efficiency, waste, on-time delivery.
__ Main focus was on short term (less than one year) service or production targets
__ Main focus was on long term (more than one year) service or production targets
__ Combination of short-term and long-term service or production targets
__ No service or production targets  SKIP TO 6

5a. In 2014, how easy or difficult was it for this business to achieve its service or production
targets?
__ Possible to achieve without much effort
__ Possible to achieve with some effort
__ Possible to achieve with normal amount of effort
__ Possible to achieve with more than normal effort
__ Possible to achieve with extraordinary effort
6. In 2014, what was the primary way employees were promoted at this business? Mark X all that apply.
__ Promotions were based solely on performance and ability
__ Promotions were based partly on performance and ability and partly on other factors (for example,
tenure or family connections)
__ Promotions were based mainly on factors other than performance and ability (for example, tenure or
family connections)
__ Employees are not normally promoted
7. In 2014, when was an under-performing employee reassigned or dismissed? Mark X all that apply.
__ Within 6 months of identifying employee under-performance
__ After 6 months of identifying employee under-performance
__ Rarely or never

Annual SBO Final F&R
Page 38 of 48

Attachment D - ASE Cognitive Testing Report

Appendix D: Final Revised Questions

Finalized Additional Questions for the
Annual Survey of Entrepreneurs
Owner Specific Questions – This additional question will be asked of each owner (up to 4) entered for the
percent ownership question in the electronic collection instrument. This will be the last question in the Owner
Section prior to sending the respondent to the business specific section.
Item: Reasons for Owning the Business (Note: New question, not from form SBO-1)

How important to Owner 1 are each of the following reasons for owning this business?” (Select one for
each row)
Not
Important

Somewhat
Important

Very
Important

Wanted to be my own boss







Flexible hours







Balance work and family







Opportunity for greater income/Wanted to build wealth







Best avenue for my ideas/goods/services







Couldn't find a job/Unable to find employment







Working for someone else didn't appeal to me







Always wanted to start my own business







An entrepreneurial friend or family member was a role
model







Other (Specify):







Business Specific Questions – These additional questions will apply to the entire business and only require one
response from the respondent regardless of how many owners are entered in the percent ownership question.
Annual SBO Final F&R
Page 39 of 48

Attachment D - ASE Cognitive Testing Report
Item: Business Aspirations (Note: New question, not from form SBO-1)
Where would the owner(s) like this business to be in five years? (Select one)

o

Larger in terms of sales or profits

o

About the same amount of sale or profitss

o

Smaller in terms of sales or profits

o

Other (Specify):

Item: Funding from Owner(s) (Note: New Question, not from form SBO-1)
For 2014, what was the total amount of money that the owner(s) personally put into the business? Include:





Investments from personal savings
Personal retirement accounts
Home equity loans
Personally borrowed funds.

$_________,000

Item: Funding from Insiders (Note: New Question, not from form SBO-1)
For 2014, what was the total amount of money this business received from family, friends, and employees?

$_________,000

Item: Funding from Outsiders (Note: New Question, not from form SBO-1)
For 2014, what was the total amount of money this business borrowed from a bank or other financial institution,
including business loans, a business credit card carrying a balance, or a business line of credit?


Include all draws on business lines of credit, even if paid off during the year.

$_________,000

Annual SBO Final F&R
Page 40 of 48

Attachment D - ASE Cognitive Testing Report
Item: Funding from Outside Investors (Note: New Question, not from form SBO-1)
For 2014, what was the total amount of money this business received from angel investors, venture capitalists,
or other businesses in return for a share of ownership in this business? (An “angel investor” is an affluent
individual who provides capital for a business start-up, usually in exchange for convertible debt or ownership
equity.)

$_________,000

Item: Funding from Government Grants (Note: New Question, not from form SBO-1)
For 2014, what was the total amount of money this business received from government grants (such as the
Small Business Innovation Research (SBIR) and/or Small Business Technology Transfer (STTR) programs).

$_________,000

Item: New Funding Relationships (Note: New Question, not from form SBO-1)
In 2014, did this business attempt to establish any new funding relationship (e.g., loans, investments, or gifts)
with any of the following sources? (Select one for each row)
No

Yes, received
total amount
of the funding
requested

Yes, but did not receive
the total amount
requested

Other owner(s) (if applicable)







Family, friends, or employees







Banks, credit unions, or other financial institutions







Home equity loans in name of business owners







Credit cards







Trade credit (buy now, pay later)







Angel investors







Venture capitalists







Annual SBO Final F&R
Page 41 of 48

Attachment D - ASE Cognitive Testing Report
Other investor businesses







Crowdfunding platform (e.g., Prosper, Kickstarter,
etc.)







Grants (e.g., Federal government's Small Business
Technology Transfer Program (STTR) or Small
Business Innovation Research Program (SBIR))







Other (Specify):

Item: Avoidance of Additional Financing (Note: New Question, not from form SBO-1)
At any time during 2014, did this business need additional financing and the owner(s) chose not to apply for any
of the following reasons?

o
o

No
Yes, needed additional financing but chose not to apply because (Select all that apply)
 Did not think business would be approved by lender
 Did not want to accrue debt
 Decided the financing costs would be too high
 Preferred to reinvest the business profits instead
 Felt the loan search/application process would be too time consuming
 Decided the additional financing was no longer needed
 Decided to wait until funding conditions improved
 Decided to wait until company hit milestones to be in stronger position to raise funds
 Other (Specify):

Item: Profitability (Note: New Question, not from form SBO-1)
For 2014, did this business have profits, losses, or break even? (Select one)”
Annual SBO Final F&R
Page 42 of 48

Attachment D - ASE Cognitive Testing Report

o

Profits

o

Losses

o

Break even

Item: Negative Impact on Profitability (Note: New Question, not from form SBO-1)
For 2014, did each of the following negatively impact the profitability of this business? (Select one in each row)
Yes

No

Access to financial capital





Cost of financial capital





Finding qualified labor





Taxes





Slow business or lost sales





Customers or clients not making payments or paying late





The unpredictability of business conditions





Changes or updates in technology





Other (Specify below)





Item: Customer Locations (Note: New Question, not from form SBO-1)
During 2014, where were this business’ customers or clients located?
Round to the nearest whole percent. Your best estimate is fine. If none, report “0.”
Same region as the business

_____%

Outside of the region but within the U.S. (domestic)

_____%

Outside the United States (international)

_____%

Total

100%

Annual SBO Final F&R
Page 43 of 48

Attachment D - ASE Cognitive Testing Report
Innovation and Research and Development Module - This starts the module selected for 2014. All respondents
will be asked about product and process innovation. For the research and development related questions, only
respondents that indicate they have research and development activities for 2014 will be asked the remaining
questions.
Item: Product Innovation (Note: New Question, not from form SBO-1)
During the last three years (2012, 2013, 2014), did this business do each of the following regarding the goods or
services it offers? (Select one for each row)


Do not include adaptation or customization of a good or service for a specific customer’s needs.
Yes
No
Sold a new good or service that no other business has ever offered before





Sold a new good or service that this business has never offered before





Improved a good’s performance by making changes in materials,
equipment, software or other components
Developed a new use for a good or service









Added a new feature to a good or service





Made it easier for customers to use a good or service





Item: Process Improvement (Note: New Question, not from form SBO-1)
During the last three years (2012, 2013, 2014), did this business do each of the following regarding the goods or
services it offers? (Select one for each row)
Yes

No

Applied a new way of purchasing, accounting, computing, maintenance, inventory
control, or other support activity
Reduced costs by changing the way a good or service was distributed









Upgraded a technique, equipment, or software to significantly improve a good or
service





Made a significant improvement in a technique or process by increasing
automation, decreasing energy consumption, or using better software





Decreased production costs by improving the materials, software, or other
components





Annual SBO Final F&R
Page 44 of 48

Attachment D - ASE Cognitive Testing Report


Changed a delivery method to be faster or more reliable



Item: Research and Development Activity (Note: If respondent answers NO to ALL questions, then SKIP to
“Currently Operating Screen”) (Note: New Question, not from form SBO-1)
In 2014, did this business do any of the following research and development (R&D) activities? (Select one for
each row).


Include R&D activities that this business performed, others paid this business to do, or this business
paid others to do.
Yes
No
Conducted work that might lead to a patent





Developed and tested prototypes that were derived from scientific research
or technical findings





Produced findings that could be published in academic journals or
presented at scientific conferences





Applied scientific or technical knowledge in a way that has never been done
before





Created new scientific research or technical solutions that can be
generalized to other situations





Conducted work to discover previously unknown scientific facts, structures,
or relationships





Conducted work to extend the understanding of scientific facts,
relationships or principles in ways that could be useful to others





Item: Total R&D Cost (Note: New Question, not from form SBO-1)
In 2014, what was this business’ total cost for R&D activities?
Include:




Labor paid for employees, temporary staffing, contractors, independent contractors, or outside
consultants
Materials, equipment, software, or other supplies purchased
Money spent for rent, utilities or other overhead

Annual SBO Final F&R
Page 45 of 48

Attachment D - ASE Cognitive Testing Report
$______________,000

Item: Purchased R&D Costs (Note: XXXXXX will carryover from the previous total R&D screen) (Note: New
Question, not from form SBO-1)
In 2014, of the total ($XXXXXX) of R&D costs, what was the amount used to purchase R&D services from
others?
Include labor paid for employees, temporary staffing, contractors, independent contractors, or outside
consultants
$_________,000

Item: Business R&D Costs (Note: The XXXXX cost should be a calculation of the entries from the items: Total
R&D Costs – Purchased R&D Costs) (Note: New Question, not from form SBO-1)
In 2014, what percent of the costs of R&D services performed by this business ($XXXX, Item XX – Item YY)
consisted of the following purchases? If none, report “0.”
Employee payroll

____%

Equipment purchases

____%

Software and licensing purchases

____%

Other R & D expenses

____%

TOTAL 100%

Item: Funding Sources for R&D Activities (Note: The XXXXX cost should be a calculation of the entries from the
items: Total R&D Costs – Purchased R&D Costs) (Note: New Question, not from form SBO-1)
Annual SBO Final F&R
Page 46 of 48

Attachment D - ASE Cognitive Testing Report
In 2014, what percent of the costs of R&D services performed by this business ($XXXX, Item XX – Item YY) was
paid by the following sources? If none, report “0.”

This business

____%

Another U.S. business

____%

U.S. college or university

____%

U.S. nonprofit organization

____%

U.S. federal government (Include R&D grants)

____%

U.S. state or local government (Do not include public schools)

____%

Other (Specify):

____%

Total

100%

Item: Number of R&D Employees (Note: New Question, not from form SBO-1)
For the pay period including March 12, 2014, how many of each type of the following workers worked on R&D
activities?
Number of workers
Owner(s)

_____

Paid Employees

_____

Other Paid Workers - Include l temporary staffers, contractors, independent
contractors, and outside consultants.

_____

Unpaid Workers/Interns

_____

Annual SBO Final F&R
Page 47 of 48

Attachment D - ASE Cognitive Testing Report

Annual SBO Final F&R
Page 48 of 48