Findings and Recommendations from the 2017 Annual Business Survey Cognitive Testing

Attachment E

Findings and Recommendations from the
2017 Annual Business Survey Cognitive
Testing
Prepared for:
Patrice Norman, Elizabeth Goldberg, and Naomi Blackman
Economy-Wide Statistics Division

Prepared by:
Krysten Mesner and Rebecca Keegan
Economic Statistical Methods Division
September 8, 2017

Attachment E

Table of Contents
I.

Background ........................................................................................................................................... 3

II.

Research ................................................................................................................................................ 4

III.

Research Methodology ..................................................................................................................... 5

IV.

Section Findings ................................................................................................................................ 6
Business Structure Findings .................................................................................................................. 6
Owner Characteristics Findings............................................................................................................. 6
Other Business Characteristics Findings ............................................................................................... 7
Business Financing Findings .................................................................................................................. 8
Business Performance Findings .......................................................................................................... 10

V.

Research Findings ............................................................................................................................... 10

Appendix A: About the Data Collection Methodology and Research Branch (DCMRB) ............................. 12
Appendix B: Cognitive Interview Protocol – 2017 Annual Business Survey ............................................... 13

Attachment E

I.

Background

Staff from the Data Collection Methodology and Research Branch (DCMRB, see Appendix A for more
information) in the Census Bureau conducted nineteen interviews to evaluate several new questions
that have been added to the Annual Business Survey (ABS). The ABS is a comprehensive survey that
provides the only detailed, regularly collected sources of data on the status, nature, and scope of
women-, minority-, and veteran-owned businesses. ABS provides estimates for the number of employer
and nonemployer firms, sales and receipts, annual payroll, and employment by owners’ gender,
ethnicity, race, and veteran status. Estimates will be available for the U.S., states, metropolitan and
metropolitan statistical areas, counties, and places. Among other uses, the Small Business
Administration (SBA) and the Minority Business Development Agency (MBDA) use results from the SBO
to assess business assistance needs and allocate available program resources. More information on the
ABS can be found at https://www.census.gov/programs-surveys/sbo.html.

Attachment E

II.
•
•
•

Research Objectives
Identify areas of the survey that are problematic for respondents
Identify instructions/features that are difficult for users to understand
Provide recommendations for improvements to be implemented during the development of the
instrument.

Research Questions


Does placement of the sensitive questions on sex, race, ethnicity, disability, and veteran status
at the beginning of the owners section bias responses or promote nonresponse to these
questions?



After taking the survey, participants were presented with a list of five options for potential
survey names and were asked their preference.



Did participants experience any cognitive difficulties in answering any of the survey questions?

Attachment E

III.

Research Methodology

From July 2017 through August 2017, researchers from DCMRB conducted cognitive interviews inperson at businesses in Denver, Colorado and Nashville, Tennessee. A total of 19 participants were
tested and were recruited via database of past survey respondents. The protocol for this testing can be
found in Appendix B. During the interviews, respondents answered self-administered questionnaires.
Researchers used concurrent and retrospective probes to assess respondents’ understanding of the
questions and the flow of the questionnaire.
Half of cognitive participants received version A protocol and the other half received version B protocol.
The only difference being the placement of the sensitive questions (see V. Research Findings).
About half of respondents were owners of the business they were filling out the survey on, while the
other half were proxy respondents. Participants included in testing worked in a wide range of company
sectors, and a range of company sizes. The distribution of participants by company size and sector can
be found in the tables below.

Distribution of Sample Companies
Number of Employees
1 to 4 employees
6
5 to 9 employees
4
10 to 19 employees
3
20 to 49 employees
1
50 to 99 employees
4
100 to 249 employees
1
Total
19
Distribution of Sample Company Sector
Sector
23 (Construction)
31 (Manufacturing)
42 (Wholesale Trade)
45 (Retail Trade)
48-49 (Transportation and Warehousing)
52 (Finance and Insurance)

2
1
3
2
2
2

54 (Professional, Scientific, and Technical Services)

6

56 (Admin Support & Waste Management & Remediation Services)

1

Attachment E

IV.

Section Findings

Business Structure Findings
The business structure section of the Annual Business Survey (ABS) includes questions that gather
information about the ownership of the business, and the percentage of the business held by the
owners.
In general, respondents had little trouble answering questions in this section. These questions were
considered to be straightforward and understandable. Only one participant expressed hesitation listing
the owners name, and instead wrote down the name of the company that owned their company. The
participant expressed that the owners information is considered sensitive.
Number of Owners
Several participants wrote the owners name in the column specified for percent ownership, and percent
of ownership in the column specified for the name of the owner. Consider switching the columns in the
chart so that the respondent lists the owners name first, and then the percentage owned. This will
better emulate general documents that request ones name first, and potentially reduce error.
Recommendation: Put the column designated for a name before the column for percent
ownership.
Family Ownership
In 2017, did two or more members of one family own the majority of this business?
For Question 24, respondents felt that the definition of family was all encompassing. There was general
support for the addition of unmarried partners to this definition. However, one respondent commented
that “step” relationships might be considered excluded by the definition provided.
Recommendation: Consider including “step” relationships to this definition.
Did spouses/unmarried partners jointly own this business; Was this business operated equally by
both spouses/unmarried partners?
Some participants expressed confusion with these questions because a time frame was not specified.
Including the specific year of interest may reduce confusion.
Recommendation: Consider specifying what time frame is relevant to the questions.

Owner Characteristics Findings
The owner characteristics section of the ABS includes questions regarding demographic details about
the owner of the business, such as the owners race, age, sex, educational background.
Certain questions in this section including those regarding the owners sex, race, ethnicity, disability, and
veterans status, are questions that are considered sensitive and were of interest to researchers. Of
particular interest was the question of whether altering the placement of these questions would bias
responses, or promote nonresponse to these questions.

Attachment E

Among the participants who struggled somewhat were those who had complex business structure, in
which the owners of the business were too far removed to even be considered moderately involved in
the businesses day-to-day operations.
Education Prior to Owning the Business
What was the highest degree or level of school Owner X completed prior to establishing,
purchasing, or acquiring this business?
One participant had an Associate’s degree in Architecture and selected both “Architecture and Building”
and “No Bachelor’s, Master’s, Doctorate, or Professional Degree”. The lack of explicit instructions
regarding this participants particular degree led him to commit an error.
Some participants were confused by the construction of the question and expressed that they would
prefer to see the term “major” as a better indicator of the questions intent.
Recommendation: Consider changing or adding to “field of highest degree completed” to
include “major” or some other simplifying term. Additionally, for individuals who have received
an Associate degree, it may be helpful to explicitly state that Associate degrees are to be
excluded.
Reasons for Owning the Business
Many multigenerational family owned business respondents reported a desire for “taking over the
family business” or “carrying on the family business” as an overwhelming reason for running this
business. One participant stated that bettering the community and becoming more involved in her
community has been the most rewarding part of her job.
Recommendation: Consider additional response categories to include: “carrying on the family
business”, and “Helping and/or becoming more involved in my community”.
Hearing, Vision, Cognitive or Mobility Difficulty
Does Owner X have any of the following difficulties?
Participants who didn’t have any disabilities to report were looking for a “None” or “None of the Above”
category and were confused as to how to answer when they didn’t find one. Not selecting any response
category made respondents feel uneasy as if they were just skipping the question. Also, participants
wondered whether or not the use of a hearing aid would qualify for a hearing disability or not.
Recommendations: Add response category for “None of the Above” to differentiate between
having no disability and item nonresponse. Additionally, specifically mention “even when using a
hearing aid” for the hearing difficulty response category.

Other Business Characteristics Findings
The Other Business Characteristics Findings section includes projections for the businesses future (e.g
where would the owner(s) like this business to be in five years?), year in which business was established,

Attachment E

types of customers, types of workers used by the business, use of the internet to promote or conduct
business, business operating status, etc.

Types of Workers
In 2017, which of the following types of workers were used by this business? Select all that apply.
Businesses using workers who receive a commission, seasonal employees, workers who are on
call/demand, etc. were unsure of how to classify these types of workers.
Recommendation: Consider adding category for workers who receive a commission, seasonal
employees, workers who are on call/demand, etc.
Company Website
In 2017, did this business use any of the following to promote or conduct business?
Participants almost always selected the option for “company website” even if they didn’t use it to
promote or conduct business. This finding was uncovered while we were probing respondents.
Recommendation: Since participants selected the option for “company website” even if they
didn’t use it to promote or conduct business, perhaps the words “promote” and “conduct”
should be in bold print or underlined to emphasize what this question is asking.

Business Financing Findings
The Business Financing Section includes questions regarding the specific sources and amount of capital
used to start or initially acquire the business. Within this section, many of the proxy respondents
reported that they would need to defer to the owners for certain questions, particularly those that
involved initial acquisition actions.
Funding from Owners
For 2017, what was the total amount of money that the owner(s) personally put into the
business? Your best estimate is fine.
Proxy respondents who worked in law firms reported that there was a set amount required for partners
to invest. There was some confusion in determining whether to answer the set amount that each
partner pays, or to sum those amounts and report the total. For example, one participant commented
that the question was “tricky” because 4 new owners had just joined the firm. Each new partner paid
$75,000 to join, and the respondent marked $50,000 - $99,999 as her answer, but had she totaled these
amounts, her answer would have been $250,000 or more.
Additionally, some participants felt that the maximum amount category of $250,000 or more, doesn’t
capture what would best describe their company’s financial situation in which, “tens of millions” would
be an accurate response. In this case, the company’s ownership the respondent was reporting for was
that of a long line of parent companies.

Attachment E

Recommendation: Due to the fact that there was some confusion regarding whether to report
the amount that each owner put in to the company, or to sum those amounts, it may be
beneficial to bold and/or italicize the word “total” in the question.
New Funding Relationship
In 2017, did this business attempt to establish any new funding relationships (for example, loans,
investments, or gifts) with any of the following sources?
This question was generally well understood, but some respondents felt unsure whether opening a new
account/loan with an existing bank should be included.
Recommendation: Specify whether new financing with a bank the owner already has a
relationship with would still qualify as an affirmative response.
Funding from Outside Investors
For 2017, what was the total amount of money this business received from angel investors,
venture capitalists, or other businesses in return for a share of ownership in this business?
The vast majority of respondents were unfamiliar with the term angel investor, and while venture
capitalist was a more familiar term, it was still widely misunderstood. Conversely, nearly all respondents
knew about crowdsourcing, although age appeared to be a factor in understanding this term, where
older participants reported less familiarity with the term.
Recommendation: Consider adding definitions to questions containing esoteric language.
Specifically, respondents did not understand the terms, “angel investors”, “crowdfunding” and
“trade credit”.
Funding from Banks or Other Financial Institutions
For 2017, what was the total amount of money this business borrowed from a bank or other
financial institutions, including business loans, a business credit card carrying a balance, or a
business line of credit? Include all draws on a business line of credit, even if paid off during the
year.
This question generated some confusion from respondents primarily due to the portion of the question
about line of credit. Respondents were uncertain of what exactly this question is asking. For example, if
credit cards are paid in full at the end of each month respondents questioned if they were expected to
include them in their response.
Recommendation: This question may benefit by having a list for respondents to reference of
specifically what to include and exclude in their response.
Funding from Family, Friends, and Employees
For 2017, what was the total amount of money this business received from family, friends, and
employees?
Some respondents were confused if they had done business for/with family members (sales/revenue).
Others were confused if employees had e.g., paid into health insurance premiums.

Attachment E

Recommendation: Instead of “total amount of money” consider using “capital or investment
funds” from family etc.
Avoidance of Additional Financing
At any time during 2017, did this business need additional financing?
What is meant by “additional financing?” Participants expressed that it is not clear whether or not this a
continuation of the prior question.
Recommendation: Perhaps consider moving this question sooner if including the previous
question or specify not to include what was already covered by the previous question if that is
the intent of this question.

Business Performance Findings
The Business Performance Findings section contains questions regarding whether the business was
profitable for the year, and what sorts of environmental or internal conditions affected those profits.
Negative Impact on Profitability
For 2017, which of the following negatively impacted the profitability of this business?
Respondents reported that this question is somewhat subjective, because although some things listed as
response options were experienced (i.e in healthcare, unpredictability of market), that it doesn't
necessarily mean there was an impact on profit. Some respondents said this would be, “hard to
measure” in an objective way.
Some participants expressed that this question felt like it was making a value judgement, which was in
contrast to the majority of other questions on the survey that were more quantifiable or factual.
Additionally, several participants commented that the phrasing felt somewhat loaded because it
assumes negative consequences occurred at all.
Recommendation: Prior to asking this question, consider asking respondents first a yes/no
question regarding whether they experienced any negative impacts on their profit. Regarding
the subjectivity of the question, it may help to include a qualifier stating something akin to
“please only include factors that impacted profit”.

V.

Research Findings

Researchers were also interested in interviewing participants to get their thoughts and opinion
regarding two separate research questions. The first point of interest was regarding the placement of
sensitive questions. Secondly, researchers asked the respondents about their preference for the name
of the survey. Both of these research questions were asked once participants had completed the survey.

Attachment E

Firstly, sensitive questions are those questions within the survey that ask about the owner’s sex, race,
veteran status, and disability status. One concern in asking questions such as these is that respondents
may feel uncomfortable answering and choose to not answer them, or to discontinue the survey
entirely. Researchers were interested in finding out whether placing these questions further back into
the survey may help mitigate discomfort on the part of the respondent.
Participants were given one of two versions of the survey. In version A, the sensitive questions were at
the very beginning of the Owner Characteristics section. In version B, the sensitive questions were
further towards the back of that section.
The results were that regardless of the placement of the questions, there was a general preference for
version A, where the sensitive questions were presented first in the owners section. Survey participants
expressed that receiving the demographic questions upfront flowed better, and followed a familiar
pattern more in line with most surveys they’ve encountered.
Secondly, each participant was presented with several options of potential survey names including
Annual Business Survey, Annual Company Survey, Annual Enterprise Survey, Annual Firm Survey, and
Business Leadership and Innovation Survey.
There was an overwhelming preference for Annual Business Survey. Respondents reported that the
term “Business” was a more encompassing term than the other options.
While company was generally regarded as the second favorite, many respondents commented that the
term company implies that more than one person works there, and therefore was not an applicable
term across all businesses. There was a strong negative reaction to the option of “Enterprise”, primarily
because of the term’s negative connotations. For example, some participants associated the term with
loan sharks and money-making schemes. Additionally, several participants mentioned that they were
reminded of Star Trek, which contributed to the sentiment that the term “Enterprise” sounded less
legitimate than the other options. There was a general consensus that the term “Firm” applied more to
specific practices such a law, and investment firms. Finally, participants felt that the option of Business
Leadership and Innovation Survey was too long.

Attachment E

Appendix A: About the Data Collection Methodology and Research Branch (DCMRB)
The Data Collection Methodology and Research Branch in the Economic Statistical Methods Division
(ESMD) assists economic survey program areas and other governmental agencies with research
associated with the behavioral aspects of survey response and data collection. The mission of DCMRB is
to improve data quality in surveys while reducing survey nonresponse and respondent burden. This
mission is achieved by:
•

•

•
•

Conducting expert reviews, cognitive pretesting, site visits and usability testing, along
with post-collection evaluation methods, to assess the effectiveness and efficiency of
the data collection instruments and associated materials.
Conducting early stage scoping interviews to assist with the development of survey
content (concepts, specifications, question wording and instructions, etc.) by getting
early feedback on it from respondents.
Assisting program areas with the development and use of nonresponse reduction
methods and contact strategies.
Conducting empirical research to help better understand behavioral aspects of survey
response, with the aim of identifying areas for further improvement as well as
evaluating the effectiveness of qualitative research.

For more information on how DCMRB can assist your economic survey program area or agency,
please contact the branch chief, Amy Anderson Riemer.

Attachment E

Appendix B: Cognitive Interview Protocol – 2017 Annual Business Survey
Cognitive Interview Protocol - 2017 Annual Business Survey
Research Questions to Address:
• Are respondents able to answer the questions as intended?
• Are there any items that cause the respondents problems?
o What can be done to fix those problems?
• Are there any order/context effects?
Materials:
• Copies of questionnaire testing draft
• Recorder, batteries
• Consent form
Procedure:
• Go through the Introduction and Before the Questionnaire questions.
• Then provide R with the draft questionnaire and ask R to fill it out. Once R completes
each question, ask the general probes and any item-specific follow-up probes.
• Note how R navigates the questionnaire and if they appear to be having any difficulty
with the questions. If R appears to have any problems (utterances like “Hmm”, confused
looks, etc.), probe on this.
• Go through probes about Wrap-up Questions
Intro:
• Explain purpose of meeting: to understand the process of answering questions, and to
evaluate how new questions work or don’t work.
• Some questions I will ask may seem odd and/or obvious, but I don’t want to assume I
know what you are thinking.
• Explain that we are not testing the respondent – we only want to evaluate the
questionnaire
• Structure of meeting: Understand your business and your role in it, then review the
questionnaire and ask you questions as you complete it.
• Permission to record discussion? Solely for note-taking purposes. Have R sign consent
form.
Before the Questionnaire:
• What types of goods or services does this business provide?
• What is your role in the company? What kind of responsibilities do you have?
• Are you one of the owners of the business?
• Have you completed other surveys from the Census Bureau or other agencies? If so,
which ones?

Attachment E

•

What was your role in the process for responding to this questionnaire? (Gather data?
Enter data? Consult with data providers? Etc.)

Questionnaire test
Now let’s take a look at the draft questions. I would like you to go through and answer the
questions as if you were filling out an actual form. If you don’t have answers to any particular
questions, you can just make a best guess. I will ask you some questions as we go through
them.
These questions are intended for [name of business as listed in recruiting file].
General Probes:
• What does this question mean to you?
• How did you go about answering this question?
• Would you be able to answer this question yourself? (If no) Who would be the
appropriate person to answer it?
• (If R appears to have problem/confusion) What are you thinking about?
• Reflect R’s specific answer: You said ___. Can you tell me more about that?
• If R notices skip instructions: The electronic system would skip automatically, but we will
go over every question to make sure they are all clear.
Item-Specific Probes:
Number of Owners
• Note respondents selection and which question they move to. Who do you consider to
be an owner? What criteria does an “owner” meet, in your words?
10% or More Ownership
• How did you decide which option to select?
• Would you need to consult any records to answer this question?
10 Percent or More Ownership
• Are there any arrangement such as parent companies, trusts, estates or otherwise part
of the ownership of this company?
• If yes, Please explain.
• How would you treat those entities for the purposes of this survey?
Percent Ownership
• For each percentage: How did you arrive at this figure?
• If more than one owner: How did you select the order of Owners you provided?

Attachment E

•

How do you feel about providing the Owners’ names?

Service-Connected Disability
• How do you feel about providing this information?
• If answering for an owner that isn’t present – How would you answer this question on
behalf of another owner? Would you consult the owner?
Prior Business Ownership
• How did you decide which option to select?
Education Prior to Owning the Business
• How did you decide which option to select?
Field of Degree
• Are these response categories clear?
• Are there any important categories that are missing in your opinion?
• Note how the respondent would respond if they answered < bachelor’s degree in
previous question
• If answering for an owner that isn’t present – How would you answer this question on
behalf of another owner? Would you consult the owner?
Citizenship
• How do you feel about providing this information?
Place of Birth
• How do you feel about providing this information?
• If answering for an owner that isn’t present – How would you answer this question on
behalf of another owner? Would you consult the owner?
Hearing, Vision, Cognitive or Mobility Difficulty
• What do you think that this question is asking?
• What do you consider to be a serious hearing difficulty?
• What do you consider serious difficulty walking or climbing stairs to be?
• How do you feel about providing this information?
• If answering for an owner that isn’t present – How would you answer this question on
behalf of another owner? Would you consult the owner?
Reasons for Owning the Business
• What does each response category mean to you?
• If other (specify), can you tell me more about that?
• Are there any other main reasons that does apply to you or could apply to you that
should be listed?

Attachment E

One Family Majority Ownership
• How did you decide which option to select?
• Did you notice the addition of “unmarried partners?”
• What do you think is meant by spouses/unmarried partners?
• How does this definition of family match or differ from what you consider to be family?
• Is there anything else you would include or exclude from this definition?
Joint Ownership
• How did you decide which option to select?
Equal Operation
• What do you think is meant by “operated equally?”
• How did you decide which option to select?
• In your opinion, how does operation differ from ownership?
Business Aspirations
• What is this question asking?
• What does each answer category mean?
• How did you decide which option to select?
• If you chose Other (specify), please tell me more about your answer.
Funding From Owner(s)
• How did you come up with your answer?
• Where did the money come from that the owner put into the business?
• What time frame where you thinking of?
• If answering for an owner that isn’t present – How would you answer this question on
behalf of another owner? Would you consult the owner?
Funding From Family, Friends, and Employees
• How did you come up with your answer?
• If answering for an owner that isn’t present – How would you answer this question on
behalf of another owner? Would you consult the owner?
Funding From Banks Or Other Financial Institutions
• How did you come up with your answer?
• What types of funding were you including in your response?
Funding From Outside Investors
• How did you come up with your answer?
• Prior to reading this question, had you ever heard of the term “angel investor?”
• In your own words, what is a venture capitalist?
Funding From Government Grants

Attachment E

•
•

How did you come up with your answer?
Which government grants did you include in your answer?

New Funding Relationships
• How did you come up with your answer?
• What do you think is meant by “new funding relationship?”
• Have you ever heard of “trade credit?”
• What do you think is meant by “trade credit?”
• What is a “Crowdfunding platform?”
• If selected Other (Specify), please explain.
Avoidance of Additional Financing
• How did you come up with your answer?
Reasons for Avoidance of Additional Financing
• How did you come up with your answer?
• Are there other options that are not available that would be applicable to your current
situation or even a similar situation in the past?
Profitability
• How did you come up with your answer?
Negative Impact On Profitability
• How did you come up with an answer for each of the categories?
• If selected Other (Specify), please explain.
Types of Customers
• How did you decide which option(s) to select?
• If selected Other businesses, please explain.
• If selected Other organizations, please explain.
Types of Workers
• Are you able to differentiate which full and part-time employees receive a W-2?
• Are you able to differentiate between contractors, subcontractors or outside
consultants who did and did not receive a 1099 or payment from another company?
• Are there any other types of workers that are listed as options that you used in 2017?
Website
• Can you provide more background about the types of activities that you do over the
Web?
• Which activities are for promotion?
• Which activities are for conducting business?

Attachment E

Wrap-up
For businesses with a proxy respondent during the interview:
•
•
•
•
•

Now that you have seen this survey, would you typically be the one who would be
responsible for answering this survey?
If so, would you involve the owner(s) in answering these questions? Which questions?
How would you go about asking the owner(s) to answer these questions? Would you
provide them with a copy of the question/survey?
Would you have the owner(s) review the completed form before it was submitted?
Would you have any concerns or hesitancy about answering any of the questions about
the owners without their input?

All businesses:
•

Are there any other comments or suggestions you would like to make about anything
we have gone over today?

Placement of Sensitive Questions – Sex/Ethnicity/Race/Veterans Status
•

•

Point out the positioning of the questions on sex/ethnicity/race/veterans status to the
participant. How did you feel about the placement of the questions on
Sex/Ethnicity/Race and Veterans Status?
Show the participant the alternate version of the survey where these questions come in a
different order. Where do you think that these questions should be placed? Why?

Name of Survey
•

We are considering changing the name of this survey. After seeing the questions on this
survey, which of the following names do you think is most appropriate for this survey?
Why?
o
o
o
o
o

Annual Business Survey
Annual Company Survey
Annual Enterprise Survey
Annual Firm Survey
Annual Business Leadership and Innovation Survey

Thank the respondent for their time.