Download:
pdf |
pdfFindings and Recommendations from
Cognitive Interviews for Current and
Proposed Content for the Annual Business
Survey (ABS)
Prepared for:
Gail White, Economic Reimbursable Surveys Division (ERD)
Patrice Hall, Economic Reimbursable Surveys Division (ERD)
Audrey Kindlon, National Science Foundation (NSF)
Prepared by:
Hillary Steinberg, Data Collection and Methodology Research Staff (DCMRB), Economic Statistical
Methods Division (ESMD)
Krysten Mesner, DCMRB, ESMD
Economic Programs Directorate
U.S. Census Bureau
11.14.23 final
The Census Bureau has reviewed this data product to ensure appropriate access, use, and disclosure
avoidance protection of the confidential source data (Project No. P-7504866, Disclosure Review Board
(DRB) approval number: CBDRB-FY24-ESMD010-002).
Table of Contents
Contents
Table of Contents ....................................................................................... Error! Bookmark not defined.
Tables ........................................................................................................ Error! Bookmark not defined.
Executive Summary ................................................................................................................................ 3
Research Objectives ................................................................................................................................ 4
Research Methodology ............................................................................................................................ 5
Findings and Recommendations .............................................................................................................. 9
Finding #1: There is too much content on the ABS, and especially on longer sections. ..................... 9
Finding #2: Participants were often coordinating for sections, and sometimes did not know who to
send the survey to within their company........................................................................................... 9
Finding #3: Participants did not understand what ABS was used for in general. .............................. 11
Finding #4: Participants would like to report a section was not applicable to them. ......................... 12
Finding #5: Some topics would benefit from further testing using more open-ended approaches..... 13
Findings and Recommendations for Sections ......................................................................................... 14
Research and Development ............................................................................................................ 14
Sustainability ................................................................................................................................. 17
Technology Transfer...................................................................................................................... 20
Reshoring ...................................................................................................................................... 26
Labor............................................................................................................................................. 27
Entrepreneurship............................................................................................................................ 28
Demographics................................................................................................................................ 29
About the Data Collection Methodology and Research Branch (DCMRB) ............................................. 44
Appendix A –Moderated Protocol (Round 1) ......................................................................................... 45
Appendix B – Moderated Instrument (Round 1)..................................................................................... 56
Technology Transfer.......................................................................................................................... 62
Labor................................................................................................................................................. 65
Reshoring .......................................................................................................................................... 65
Research and Development................................................................................................................ 66
Entrepreneurship ............................................................................................................................... 71
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 2 of 88
Demographics ................................................................................................................................... 71
Appendix C –Unmoderated Instruments ................................................................................................ 75
Appendix D – Changes to the Round 2 Moderated Protocol ................................................................... 84
Research and Development................................................................................................................ 84
Entrepreneurship ............................................................................................................................... 88
Demographics ................................................................................................................................... 88
Tables
Table 1. The components of each round of testing.................................................................................... 5
Table 2. Frequency of each module throughout moderated testing............................................................ 7
Table 3. Number of Round 2 participants who identified as owners, proxies, or representing companies
with no owner.......................................................................................................................................... 8
Table 4. Item nonresponse and survey drop off for sensitive demographic questions in the unmoderated
survey (n = 149), ................................................................................................................................... 37
Figures
Figure 1. Updated question on ownership that allows for the identification of proxy reporting.
31
Figure 2. Round 1 Disability Status Question, adapted from the American Community Survey .............. 40
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 3 of 88
Executive Summary
The Data Collection Methodology and Research Branch (DCMRB) of the Economic Survey
Methods Division (ESMD) utilized moderated cognitive interviewing and unmoderated
cognitive testing to garner feedback on current and proposed questions for the Annual Business
Survey (ABS). Representatives of 233 companies, both single- and multi-firm units, took part in
this evaluation.
Finding #1: There is too much content on the ABS, and especially on longer sections.
Recommendation: Streamline content. Do not include all proposed content.
Finding #2: Participants were often coordinating gathering information for sections, and
sometimes did not know who to send the survey to within their company.
Recommendations: Include delegation mechanisms, especially by topic. Consider
instructions on the appropriate respondent (e.g., the general professional title of a best
respondent).
Finding #3: Participants did not understand the purpose and use of ABS generally.
Recommendation: Include explanations of the purpose of the survey and topics,
especially those related to sexual orientation and gender identity (SOGI).
Finding #4: Participants would like to report when topics and questions are not
applicable.
Recommendation: Add “not applicable” response options to items, and only display
applicable content.
Finding #5: Some topics would benefit from further testing using more open-ended
approaches.
Recommendation: Expand research to target specific populations and consider other
methodological approaches.
Research Questions
This research was guided by four key research questions, including:
•
•
•
•
How do respondents comprehend specific questions?
How do respondents use records and/or estimation strategies for answering specific
questions?
Are respondents able to answer the questions?
What difficulties might respondents encounter when reporting?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 4 of 88
Research Methodology
1.1 Rounds of data collection
Researchers in ESMD engaged in two rounds of data collection to test the Annual Business
Survey (ABS). Each round included both moderated and unmoderated cognitive testing. The
first round took place late summer, and the second round early fall. Table 1 outlines the
components of each round.
Table 1. The components of each round of testing.
Timing
Round 1
August 1- August 16
August 7 – September 11
Round 2
August 22 – September 13
August 30 – September 14
September 15 – September
27
*
Applicable responses are
those responses answering yes
to the relevant screener
questions on the topic.
Mode
Content
Moderated
All Modules
Unmoderated Labor and
Reshoring
Number
Number of applicable
of
responses*
Responses
22
56
Moderated
All modules
17
Unmoderated Demographics 164
Unmoderated Collaborative 82
research
N/a
23
n/a
n/a
7
1.2 Recruitment
The ABS survey staff provided to ESMD researchers a sample file containing contact
information for 48,000 businesses that had previously reported to the ABS.
1.2.1
Moderated Cognitive Testing Recruitment
We conducted two rounds of cognitive testing. For each moderated recruitment list, we
purposively sampled a quarter of respondents from zip codes that were identified as
“Gayborhoods” according to Wikipedia and real estate websites. A “Gayborhood” a
neighborhood historically home to LGBTQ residents and businesses serving them. Each sample
drew purposively on multi-unit companies and those whose respondent names matched that of
the business’s owner in the ABS. In round one of recruiting, we purposively sampled for
businesses with at least one location classified in a North American Industry Classification
(NAICS) of 31 through 33 (manufacturing) to better get a sense of entrepreneurs building
tangible products. We sent email invitations to 754 companies to participate in a moderated
interview; 704 of those companies received reminder emails urging them to select a day and time
to participate in an interview. Most companies participating in this evaluation signed up for
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 5 of 88
moderated interviews within a few days of receiving the invitation to participate. The research
team conducted 39 moderated interviews total.
1.2.2
Unmoderated Cognitive Testing Recruitment
Unmoderated survey participants were contacted separately from moderated participants. In the
round 1 unmoderated testing, we selected companies with both Labor flags (those companies
flagged as more likely to have organized labor in some capacity at the business) and Reshoring
flags (those companies identified as having international activities) for recruitment. This
accounted for 409 email invitations to participate in unmoderated testing. Of the 56 participants
that completed the unmoderated survey, 21 indicated experience with labor unions, and two were
at companies that had engaged in reshoring in the past year.
In round 2, the ABS survey staff built recruitment files based on the same criteria used for the
moderated interviews, described above. The focus of the round 2 unmoderated testing was
demographic items on the survey. This accounted for 1,492 email invitations to participate in
unmoderated testing. Of these, 275 emails resulted in an opened survey, and 164 surveys were
counted as “completed” because respondents provided an answer to at least the first item and
prompt. Additionally, we sent a short unmoderated survey (4 questions) to those companies with
the Technology flag (companies identified as having a specific relationship to technology, like
patents) to ask about collaborative research. This accounted for an additional 601 email
invitations, resulting in 82 participants completing this unmoderated testing.
1.2.3
Participant Overview
In general, the participants were owners of the business or were non-owner proxy respondents in
finance, accounting, or human resources roles. We interviewed a few government reporters at
larger companies whose role in the business is exclusively or largely to complete requests for
data from the government.
We note that recruitment was complicated by the ABS still being actively fielded concurrent
with this pretesting research. Some participants did reference their current ABS answers during
interviewing. After the first few recruitment email blasts, we also used more informal language
with our reminder emails than typical as a means of distinguishing from the active production
collection. While we believe this change was effective, it may have led to more questions about
the ABS survey request(s). This should be noted for future ABS content testing evaluations.
Respondent research should consider testing at different periods than production, or creating
distinctions from production.
1.3 Procedure
This cognitive testing focused on existing and new content within the survey. Survey staff in
consultation with the survey sponsors developed this content based on existing questions on the
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 6 of 88
ABS supplemented with new questions. Survey staff developed one particular set of questions
pertaining to sexual orientation and gender identity (SOGI) relying on best practices as outlined
by the White House’s memo Recommendations on the Best Practices For the Collection of
Sexual Orientation And Gender Identity Data On Federal Statistical Surveys
(https://www.whitehouse.gov/wp-content/uploads/2023/01/SOGI-Best-Practices.pdf),
promulgated in January 2023.
In consultation with researchers, survey staff updated some questions when they did not test well
or a need occurred. For example, in round 2, interviewers began asking if participants were
reporting for themselves, on behalf of an owner, or a combination of the two, as a result of the
round 1 testing. Similarly, we had copied language from the Census Bureau’s American
Community Survey (ACS) questions on disability status, but the phrasing invoked compliance
with the Americans with Disabilities Act (A.D.A.), which was not the purpose of the item. We
changed the wording after round 1 because it did not perform as expected in a business setting.
1.3.1
Round One
For moderated interviews, companies received emails asking them to participate in a 45 minute
interview, and inviting them to schedule their interview session during days and times when
researchers were available to conduct the interview. We completed 22 cognitive interviews in the
first round of moderated testing. Interviews lasted about 45 minutes, with participants joining a
Microsoft Teams meeting, and clicking through a Qualtrics survey instrument (see Appendix B)
with the draft questions, and the interviewer taking notes in the interview protocol (found in
Appendix A).
Moderated interviews took a modular approach, meaning that different sets of questions were
asked of different companies. In round one, this was largely based on flags provided in the ABS
recruitment file. All participants in the first round of moderated cognitive interviews answered
questions about owner demographics, with the exception of one respondent who only had time
for one module. Table 2 includes how many interviews featured which modules for the duration
of testing. Please note that some modules were shorter than others, so some interviews featured
testing of multiple modules.
Table 2. Frequency of each module throughout moderated testing.
Module
Owner Demographics
Entrepreneurship
Reshoring
Labor
Research and Development
Technology Transfer
Sustainability
Number of
interviews*
22
18
14
11
10
10
9
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 7 of 88
*
Interviews are counted in each module completed,
and so the total for this table is greater than the
overall total number of interviews (n = 39)
Researchers used the same instrument in the unmoderated interviews. The foundation of the
probes was the same but updated slightly to match the needs of an unmoderated administration.
Specifically, we added multiple choice options for close-ended questions so that respondents
could select the response that best matched their impressions. In round one, we tested the labor
and reshoring modules using unmoderated methods. Half of participants received a survey with
the labor module first, and the other half received the reshoring module first, using random
assignment. The unmoderated instruments can be found in Appendix C. While some participants
opened the survey or completed the consent form, we only included responses where any part of
the survey was filled out. This accounted for 302 responses.
1.3.2
Round Two
Between rounds of research, in collaboration with ABS survey staff, we made changes to the
substantive content of questions that did not perform as expected in Round 1. These changes are
reflected in [Appendix D]. First, we reordered the questions such that questions about Research
and Development were placed first, and we asked all participants if the definition is sufficiently
captured in question one. We also began allowing respondents to report if there was no owner
who owned 10% or more of the company based on feedback from Round 1. From there, we
implemented the remainder of the Round 1 protocol. We completed 17 moderated interviews in
this round. Note that for Round 2, we asked only the demographic questions of unmoderated
participants. Table 3 represents participants identification as owner or proxy.
Table 3. Number of Round 2 participants who identified as owners, proxies, or representing companies with no owner.
Ownership status of
participant
Owner
Owner/Proxy
Proxy
No owner
Total
Frequency
(moderated)
9
8
19
5
41
Frequency
(unmoderated)
65
n/a
90
9
164
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 8 of 88
General Findings and Recommendations
Throughout this section, we describe the general findings and recommendations from both
modalities of testing (moderated and unmoderated) across both rounds of research. Later in this
report, we lay out findings about specific items that we tested with respondents.
Finding #1: There is too much content on the ABS, and especially on longer sections.
During the evaluation, many participants in our testing reported that the ABS is too long. Many
commented that the length might be a result of content on the survey not reflecting what their
business does. One participant of a holding company said, “One of the frustrating things about
this survey it that it’s not very applicable to this business. It’s mostly not applicable.” A
participant representing a manufacturing single-unit said, “You get a lot of these surveys and
then they're burdensome because you're a small business and a lot [of the content] doesn't apply.”
Of the modules tested in this evaluation, participants reported that the Sustainability and the
Technology Transfer modules are particularly long. Indeed, these often took the longest with
testing, and participants seemed to get bored of fatigued with repetition. See Finding X from the
Specific Findings section below for more information on this feedback.
Recommendation: Streamline or cut content to reduce respondent burden and fatigue.
Do not include all proposed material.
Finding #2: Participants were often coordinating gathering information for sections of the
survey from across their businesses, and sometimes did not know who to send the survey to
within their company.
Most participants have access to specific records based on their role and the size of their
company. For example, while owners of small businesses may have access to all records, at
larger firms, financial reporting staff may not have access to payroll or human resources records.
Participants spent time coordinating the collection and aggregation of information from across
their businesses for specific sections of the survey. This issue was magnified for the ABS due to
the specific nature of certain sections. As such, some participants expressed uncertainty as to the
person or persons within their company that would be best suited for answering certain sections
of the survey.
Many participants knew what sections of the survey they would have a hard time finding
reporters for their business from their experiences with previous ABS iterations, although new
sections could pose challenges too. A participant at a small biotech manufacturer said, “Some of
the answers for ABS are hard because I'm not typically in a lot of the R&D planning sessions,
[and I] had to coordinate answers with them.” Indeed, R&D was not a topic most participants
knew about, and from experience, had to find someone within their company to answer
questions. One participant said, “The R&D questions mainly around patents was a little hard, just
because I don't know that about it. It's not something I regularly know about or have access to.”
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 9 of 88
Another said, “The ones that would be hard are the ones that I would have to bring people from
the parent company overseas, which they’re 7 hours ahead. There’s only 2-3 hours we can get to
them. The R&D stuff goes through them.” While the data to answer questions about this topic
were not accessible to participants, most by now knew where to look within their company to
find answers to the questions.
Other questions or sections had less clear pathways to answers. A participant of a multi-unit
manufacturing company said that he would kick a question about sustainability “around from the
environmental guys to whoever is tracking transportation providers.” He said similarly of
emissions tracking, “If I got this as a separate survey, I would talk to facilities guy. Carbon
would be purchasing. In my job, I’m reaching out, putting together what I can from the previous
years. In a meeting I would ask if it's changed and determine what answers need to changed.”
Labor questions, which were new to this testing, were an example of a topic that might not have
a clear reporter. An unmoderated participant said, “I do not interact with shop employees much
at all in my role therefore any knowledge I would have would likely be speculative.” In the
technology transfer section, one participant was unsure of our definition of design complexity
and said she would, “Maybe need to consult with a colleague” in order to provide response. One
participant summed up the issue with information gathering in response to the question about the
number of patents, saying, “I would not know the answer to this question and would need help
and would need to pawn it off on someone else. I'm not even sure who I would read out to as it's
a very big company and I don't even know who I would call.” Another participant said, “The
patent questions were hard because that is not my area of knowledge.”
One particular area of high data dispersion throughout the company is in the owner demographic
questions (more information on this in the Specific Findings). For these demographic questions,
proxies struggled to identify who they could go to for answers, especially if they felt their
owners themselves were not accessible. Generally, said one, “most hard questions were about the
owner because I am a proxy.” Regarding the gender identity questions, a participant from a law
firm said, “This would be something that I would probably write don't know on. I don't want to
assume and/or talk to our firm manager” about this topic. Though she felt the disability status
“would have to be tracked by HR manager or director,” she did think, “I could find this out.” In
response to which questions would be the hardest to answer, one participant said, “definitely the
demographics. I would be trying to figure as best an answer I could.”
Participants did have specific recommendations on the issue of data dispersion and response
coordination throughout the company. One participant at a large, manufacturing multi-unit
company said that he would want, “delegation for specific parts. I used the sharing [function] for
this month on a survey, that was really straightforward. It was easy to use and really great.” This
is feedback we often get on surveys with varied content like the ABS. An unmoderated
respondent argued to “include ‘Do Not Know’ in the survey” as a valid response option. They
explained, in response to reshoring questions, that “large corporations have many departments
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 10 of 88
handling the financials and operations. Information always needs to be confirmed by appropriate
personnel.”
Recommendation: We recommend adding a delegation feature as respondents to the
ABS could benefit from the option to delegate specific sections of the web survey to the
most appropriate party within their business. Furthermore, providing examples of the
types of roles/positions most appropriate to answer certain sections may be helpful for
respondents.
Finding #3: Participants did not understand what ABS is used for in general.
During this testing, respondents expressed a lack of clarity regarding the purpose of the survey
and intent of its questions. One participant from a larger single-unit company said that while she
“appreciate[s] knowing that the [Decennial] Census is a head count,” she was unclear on the
purpose of the ABS, concluding that materials “need to communicate what this survey actually
does.” Another participant representing a very large medical multi-unit company listened
patiently to the NSF representative explain what the ABS does at the end of the session, and then
retorted:
Love the passion and spirit, in the future feel free to include me. I’m the point person for
all the Census and government surveys. Whenever you need feedback, I’d love to have
the discussion on how make survey better in the spirit of collaborate. I think you guys
challenged me for thinking outside the box, ‘why are they asking me this? Take a step
back, what does this info mean for the people receiving this.’ That it will be much more
accurate and comprehensive means it’s being used for great things.
In this case, the respondent found the expanded context of the survey coming directly from
experts in a conversational setting to be motivating for response, not just on the survey but in
future exploratory and confirmatory research, suggesting that if more respondents better
understood the purpose of the collection, they may be more inclined to respond.
Sometimes a lack of context or understanding of the purpose of the ABS meant that participants
might find questions to be less applicable to their companies, or try to skip pertinent questions
altogether. A participant from a small manufacturing single-unit company said of the Research
and Development section, “I shouldn't be getting this question because I won't put thought into
doing it. I’m not applicable because of the threshold. It's pointless for somebody like me” to try
to answer these questions.
Participants seemed especially confused about how the ABS is used when reviewing the new
SOGI measures (see the Specific Findings Section below for more information on this set of
questions specifically). Sometimes this prompted general distrust of the survey. One participant
said, “I still don't understand the point of all of these surveys anyway. I think I should work on
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 11 of 88
the productivity of my company instead of helping you do your job.” One moderated interview
participant flatly stated that “You should not ask these questions. Why do you want to capture it
in the survey? Surveys for the business or the U.S. government should be neutral - how does the
sex and gender impact your survey? These questions aren't relevant to me.” Some participants
outright said they would like an explanation for these questions. One participant said, “I would
be wondering why are you asking this and what you are getting at - if you gave notes as to why,
then you may get a better” response from companies. Another speculated, “Maybe [the
government] want[s] to understand how many businesses are owned by LGBT [identifying
people], but I don't understand why they want to know.” Another owner told us, “It's helpful to
have a short explanation as to why [this set of questions is] there.” Another participant
recommended, “A blurb wouldn’t hurt,” again suggesting that more information on the point and
purpose of the SOGI questions specifically might lead to higher item response rates.
Recommendation: Considering including a short introduction about the purpose of the
survey, and at the start of each section of the survey. This is especially critical for SOGI
measures, and is in alignment with guidelines promulgated by the White House.
Finding #4: Participants would like to report when topics and questions were not applicable.
Respondents were mixed at identifying if a section or question was applicable their business
given a description of the section. Participants that saw the R&D description and said it would be
applicable, then read over first question, and affirmed that it was applicable to their business. All
participants were correct in their assessment. This was not always the case for all sections. A
participant from a manufacturing multi-unit company said, “Sustainability is not our purview.”
However, the questions were not outside the purview of any given company, given that
sustainability questions were not company-specific. Another large multi-unit company involved
in restaurants said, “A lot of not applicable [content] like R&D and robotics doesn't apply.” This
was not the case, as later in the interview, many of their described activities around menu
development would fall into the R&D category.
Many participants who correctly identified questions that were not applicable to their company
endorsed more “not applicable” response options. For example, a participant from a law firm
found two of the questions around sustainability did not apply to her company. She said, “My
understanding of this is that it presents like it's going to be asked to manufacturing companies
and their sustainability efforts, like materials and things of that nature. It’s less applicable to us.”
With a question about supply chains, she said, “Supply chain feels like a manufacturing thing. It
isn't applicable.” Another participant from a manufacturing multi-unit company said of a
sustainability question (Q4), “If there was a ‘not applicable’ [option] I totally would have picked
that.” Another participant spoke about how a question in the entrepreneurship section (Q2) “was
not really applicable, so I would put don’t know…but I would want a ‘not applicable’” option. It
is important to respondents’ trust, and to get better quality data, to expose respondents to
applicable content and allow them to communicate when an item is not applicable.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 12 of 88
Recommendation: Ensure items have “not applicable” response options in addition to
“don’t know” response options. Create skip logic or pathways to show applicable
sections to respondents, possibly with screener questions.
Finding #5: Some topics would benefit from further testing using more open-ended approaches.
Some topics would benefit from a different approach to gathering information to consider how to
ask questions, like open-ended interviews, focus groups, usability testing, or respondent
debriefing interviews, in order to ensure the research question matches the method. Of the
modules and items tested in this evaluation, our findings suggest that reshoring, labor,
entrepreneurship, SOGI, and disability are all topics that could benefit from additional testing
using a variety of methods previously mentioned. Targeted recruitment to ensure that questions
are being tested appropriately with companies for which modules/sections are relevant as well as
representation of respondents, is crucial when testing questions on certain topics.
One limitation of our evaluation is that no one identified an owner - by self-report or proxy - as
LGBTQ. As a result, for the SOGI questions, we were not able to assess the perceived
sensitivity, reaction (whether positive or negative) to the potential addition of these questions on
the ABS, impression of the questions and response options, level of willingness to report this
data, and other considerations for businesses having owners who identify as LGBTQ.
Recommendation: We recommend additional testing to incorporate open-ended
interviews, focus groups, usability testing, respondent debriefing interviews, and others
for modules/items including reshoring, labor, entrepreneurship, SOGI, and disability.
Furthermore, targeted recruitment is imperative for additional testing efforts on these
modules/items.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 13 of 88
Findings and Recommendations for Sections
Research and Development
Despite pointing out that the first question was long and wordy, participants could correctly
identify whether the initial definition of R&D on the first screen applied to their company. We
tested this module in the second round, moving forward with companies it applied to, and were
able to get more responses.
As for the R&D question requesting a dollar amount, participants claimed that these data are
least accessible. Furthermore, participants recommended providing an instruction about
estimates being acceptable and some indicated a preference for percentages rather than dollar
figures.
•
•
•
“The R&D [was the hardest to answer]. When they talk about how much we spend, we
aren't measuring all that info. Of course, at some point we need to check, but usually
when we are trying to develop a new product, we spend whatever exists, we don't have a
budget for that. We know it's not going to be really expensive the new product, we don't
care how much we're going to spend. It's not going to be an extreme amount of money,
more a matter of time. The way they invest isn't in those ways.” – biotech SU, owner who
runs R&D activities
“Dollar amount on R&D is hard. I have to look at payroll, costs, and then guess at my
best percentages. Takes some time.” – manufacturing medical MU
“Dollar amount [on R&D was the hardest part]. We have different categories on our
records and how we think about R&D.” – MU restaurant franchise
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 14 of 88
Q1:
Participants found this question wordy and hard to get through. Some had trouble making sure
they were understanding each answer option correctly:
•
•
“3rd [answer option] is too wordy, I wouldn't answer. The more relevant question is does
your business do R&D, yes or no? If you're doing it below a dollar threshold, like what
I'm doing, is it beneficial?” – manufacturing SU
“First one unclear. I would have left it blank.” – manufacturing MU
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 15 of 88
Q2:
Most participants, especially owners, were comfortable estimating for this question. Most
participants did not remember they could estimate, and wanted that explicitly stated in the
question. Some preferred answering as a percentage rather than estimated dollar amount. Most
had this information in records or had dedicated personnel for whom they could attribute salary
under this category. Others struggled to attribute salary if certain employees worked on multiple
projects:
•
•
•
•
•
“I would have to talk to accountant and my wife running R&D. I'm doing part of that job.
I would have to decide how much of that time, and it’s difficult to put in a number. I have
it for tax purposes. I am doing R&D. But it's difficult to disclose that information, it's not
a dedicated department or full time. Percentage of time would good [as another way to
capture]. Salary and wages, I put one salary, the person that is doing that. Expenses and
equipment, no I'm not using anything, I'm not buying anything to do what I'm doing.
Materials and supplies it's another product but we're going to invest but it's almost
nothing, not significant.” – biotech SU
“I would have this in records. But I’d leave it blank, if it allowed me to leave it blank. But
I would rather have a ‘don't know’ option.” – restaurant MU
“I have all this in records. I can pull directly from this system.” - manufacturing MU
“Salary is a real estimate, I can look at our wages and see who does the most of that type
of work. A large amount of time quantifying and I'm doing a percentage. Machinery and
equipment, I wouldn't put materials because it's minimal. Salaries, I don't break out by
R&D.” – manufacturing medical MU
“I can estimate anything. I’d rather percentages rather than dollar amounts. You should
say estimate your R&D costs. People might worry this is for the IRS.” – manufacturing
SU
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 16 of 88
•
“I would have to go to my books and be able to split it up, and we're talking about this
survey, the value of this and my time to go to books. Time is valuable as well. You
should put that you’re okay with estimates in the question.” – manufacturing SU
Q4:
Most participants were willing to categorize certain activities as research or development, but felt
this could be hard to give an exact amount for. Most companies had internal ways of having
funds fit these categories. While there wasn’t reported issues with interpretation, they were often
idiosyncratic, meaning the answers may not be precise or consistent.
•
•
•
“I would put on applied. Because actually right now I'm trying to solve a problem that we
see is coming. I’m troubleshooting an existing problem. We know we're going to see a
problem so we're trying to research that problem.” – biotech SU
“Individualized patient [research] would applied, development would be a bigger
portion.” – manufacturing medical MU
“If I can estimate it’s pretty easy. Maybe round to the nearest thousand.” – manufacturing
SU
Recommendation: Consider testing a definition for R&D that is less wordy. The addition of
an instruction for “estimates are acceptable” should be considered for individual questions.
Allow some answers by a percentage rather than dollar amount.
Sustainability
We tested 21 questions on environmental sustainability. Overall, we found that respondents
struggled with some of the terminology in the questions, that they were not always sure what to
include or exclude from sustainability questions, and that they are not maintaining records on all
of the topics within this section.
General Terminological Issues: Some respondents struggled with the wording of some of the
sustainability questions. This included specific wording choices, but also generally the topic of
sustainability. However, most of these issues were remedied in round two, when all questions
were narrowed to “environmental sustainability” specifically. Below are representative
comments from Round 1:
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 17 of 88
•
•
•
“I need better definitions on the question itself. Is the sustainability recruiting or trying to
keep customers? Sustainability as keeping the business going or green impact?” –
manufacturing MU
“[I didn’t expect questions] about environmental. Questions don't match what you're
asking or what they say you go to.” – SU law firm
“Sustainability [questions are] difficult, [and] definitely the hardest section of questions
that were tested.” – manufacturing SU
Definition Issues: Some participants struggled with what to include or to exclude when
considering various sustainability efforts:
•
•
•
•
“We are going for green policies, things to do around our office. Things we help our
vendor implement. We make it more attractive to customers and green movement.” –
manufacturing MU
“The only thing that's applicable, attempting to move from paper based to electronic
based, so we're stopping the bills being sent out. Reducing paper usage. We are doing
that. Some of the directors might be actively thinking that, but probably mostly pushed
that direction because PDF bill book is easier to look at. Like a green office initiative.” –
Law firm MU
“Elimination of waste that was a new one on me. I don't know what circular economy,
sounds like it's taking something like packing material to use ship other things out.” – in
response to Q14, manufacturing SU
“We do click on the hyperlinks, more definition are better. Sigh. we say we want to
greener. You might want to define what a strategic objective would be: mission
statement, driving rationale?” – in response to Q15, manufacturing MU
Two participants were not sure if the questions referred to a specific product or team, rather than
the whole company.
•
•
“You’re going to get different answers based on the person taking it. You won't get
consistency even within the firm…I have to get approval through partners and more
stakeholders” – law firm SU in response to Q14
“This one is difficult for us, would say no, however we have a practice group that is
environment law, so if that specific practice area is not looking at things to help their own
law firm. 2 of 20 partners do care a lot. Add a box a manually type in box, definitely is
easier for yes/no. Feedback for if there are exceptions.” – law firm MU in response to
Q15
Tracking: Many participants felt specific questions about utilities and carbon emissions were out
of the scope of their knowledge because the company did not track them. This was especially
relevant for Q4, discussed below.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 18 of 88
•
•
“We do not track emissions. We track things such as we use gas in our production
tracked based on a meter, energy use based off bills. We don't track other things. We use
basic HVAC and we use transportation like Fedex and UPS.” – Manufacturing MU
“We rent. We’re tracking because you have to pay for utilities, not part of sustainability
initiatives” – Law firm MU
Sensitivity: The topic, rather than the questions themselves, could be sensitive. A few
participants felt they should be working on environmental sustainability, and that the questions
made it seem as though they were not prioritizing this topic. Further, participants were not
always privy to discussions or decisions around environmental sustainability.
•
•
•
“I think there could be a ‘hasn't been discussed.’ We're not developing plans but not
against it. Hopefully everybody is working to reduce carbon emissions so it's oof [noise
of embarrassment] when you say no.” – in response to Q9, biotech manufacturing SU
“We don't have strategic objectives about the environment. We have green program in the
office, but I don't go out and mention that. It's just an internal [orientation], we're trying
to be good stewards of the environment.” – in response to Q15, law firm SU
“We a small building and I'm not going to have solar or wind or geothermal [energy];
we're tiny.” - In response to Q14, biotech manufacturing SU
Q3: SUSTAINABILITY OBJECTIVES
Has this business realized any of the following benefits from pursuing its sustainability goals? Select one
for each row.
Yes
No
a.
Improved efficiency
b.
Improved regulatory compliance
c.
Improved employee recruitment and/or retention
d.
Reduced waste
e.
Improved brand reputation
f.
Decreased costs
g.
Improved customer acquisition and/or loyalty
h.
Increased revenue
i.
Other (Specify)
____________________
Participants did not often struggle with this question, but sometimes they were not sure how to
attribute the phenomena described in the options to sustainability practices.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 19 of 88
•
“Just have the cost gone down, but I’m not sure if it's sustainability parts. We haven't had
a brand reputation shift. Might have more revenue and people due to sustainability.” –
manufacturing MU
Q4: ENERGY OR CARBON EMISSIONS TRACKING
Does this business track energy use or carbon emissions from the following sources?
No, does not
track either
Yes, tracks
Energy
Expenditures
Yes, tracks
Carbon
Emissions
Source is not
applicable to
this business or
is inaccessible
a.
Heating of buildings
b.
Cooling of buildings
c.
Fleet transportation
d.
Electricity (used for
lighting, computing,
HVAC, production, etc.)
Thermal energy in
production (for example,
from natural gas,
petroleum, coal, biomass,
etc.)
e.
This question was particularly challenging for companies that rented offices, especially service
companies.
•
•
•
“Canned expenses that go to rent, we don't track it. It’s tracked by a third party or outside
person.” – law firm SU
“Expenditures is misleading to me. Energy usage (carbon emissions) would be better.
Expenditures implies amount of spent.” – law firm MU
“We keep a track on kilowatt usage… We track expenditures, not fleet. We track it,
because we track price and kw hours, we don't [do] much about it, but switched to green
[energy] sources. But that wouldn't be captured here in this question.” – manufacturing
SU
Recommendation: Ensure every question reflects the topic of environmental sustainability.
Consider alternative options or questions for those who rent or have utilities through a 3rd
party.
Technology Transfer
Ten companies answered questions in the technology transfer module during moderated
interviewing. Overall, most participants understood if the questions applied to them, and did not
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 20 of 88
have much trouble answering. There were no moderated interviewees that had collaborations,
and so we sent an unmoderated survey targeted to companies suspected of engaging in
collaborations. We screened out 75 respondents as out of scope. Seven companies had
collaborations, although only one participant gave written feedback on this question.
In general, these questions performed as expected. The questions about the number of patents (3
and 4) were more difficult if the participant had patents and needed to reach out to others,
although they could estimate. Participants struggled most with question 5. In general, we
recommend adding “don’t know” options and some hover over definitions. We will clarify what
of that applies to each question below.
Most participants said this section would take about 5-15 minutes to complete, especially once
they discerned what of this section was applicable to their business. However, if they were
reaching out to others in the company, this section would take much longer:
•
•
“10-15 min. for things I knew but would have had someone else verify other questions,
[like the] comptroller, head of R&D, VP of emerging technologies. This could take a few
hours to a few days [to get the data].” – biotech manufacturing SU
“It would take me 2 hours just because I have to reach out to others - I'd do copies of the
page and send off to whoever and wait for them to respond.” – food manufacturing MU
Q1: Formal Intellectual Property Protection
Does this business use the following methods to protect intellectual property?
a. Utility patents (patents for inventions, applied for or awarded)
Yes
No
b. Design patents (patents for appearance, applied for or awarded)
c. Copyright
d. Trademarks
Respondents indicated that they easily knew if this question was applicable to their company.
One person was unsure if a logo would count as a trademark. This was a question someone
unfamiliar with these practices would have to reach out to another person at their company for
and answer.
•
•
“A company that we work for does [these things], but the problem with the system is - in
order to get a patent or copyright, you must publish your program. And anyone can make
a few changes and have all of your thought process. So we password protect and are
always changing and moving. If someone gets in our firewall, it's already old
information.” – manufacturing SU
“Some of the answers for ABS because I'm not typically in a lot of the R&D planning
sessions, had to coordinate answers with them.” – biotech manufacturing SU
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 21 of 88
•
•
•
“Our patents are international patents and through [another country] and registered in
each country [in which] we do business. We would probably write it in because our
situation is unique.” – manufacturing SU
“All items make sense. Not missing formal IP protections from this list - not concerning
us.” – Food manufacturing MU
“All items here make sense. We don't invent or design anything. We don't copyright or
trademark yet. Company does not have any intellectual property - wait, what's intellectual
property? Is a recipe intellectual property? We want to keep a recipe a secret and want to
keep to the mind of the brewer.” – Food MU
Q2: Informal Intellectual Property Protection
Does this business use the following less formal means of protecting intellectual property?
a. Design complexity
Yes
No
b. Nondisclosure agreements or non-compete clauses
c. Branding and product image
d. Trade secrets
Most participants felt they were able to answer this, although one said they would need to
consult with a colleague. Another, which had international patents, said they would write in a
response. There was some interest and use for an “other/specify” textbox capture option.
Several participants were unsure what design complexity meant in this context:
•
•
•
“Design complexity is a little tough because everyone who builds deconstructs what they
build so I would say yes to that. All items here make sense. Not missing any informal IP
protections from this list.” – manufacturing SU
“I don't understand what design complexity - does that mean you've made it so complex
that it's hard to do?” – manufacturing SU
“I think this is a little vague. Complex design and trade secrets are vague - what does that
mean specifically? What is your meaning of intellectual property?” – manufacturing SU
Q3: Patents Pending
How many U.S. patent applications, if any, did this business have pending as of the end of 2022? If
none, report zero.
_______________
This question was clear to respondents in instances where their company held no patents. If a
participant did not have access to this information, they indicated that they could guess a
response or they could have to reach out to others within the business.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 22 of 88
•
•
•
•
“I don't know we had any US patents. Answer would be 0 because how our company
works.” – manufacturing SU
“I think we have 3 but I would have to check with our team.” – biotech SU
“I am not sure. Several passed, 5 or 6 pending from first half of the year. I would need to
contact head of R&D.” – manufacturing SU
“This I'm not going to know, because we have several approved at the end of 22, 5 or 6, 5
or 6 patents. Do we still have some pending or not? I would need to contact our head of
R&D.” – biotech manufacturing SU
Q4: Patents Owned
How many U.S. patents did this business own as of the end of 2022? If none, enter zero.
________________
This question performed as expected especially if it was not applicable to the business or the
participant indicated that they had access to this information.
•
•
“Knew off the top of my head, easy answer for a small business. I'm involved in
everything.” – manufacturing SU
“I know we have patents, not sure how many, I don't review. The controller reviews our
patent law or head of R&D. I have access, but I don't review it.” – biotech SU
Q5: Intellectual Property Activities
During the three years 2020 to 2022, did your company exchange intellectual property (IP) rights,
including patents, copyrights, and trademarks that give ownership, exclusion rights or rights to use
technical knowledge using any of the following mechanisms. Select one for each row.
a. Licensed or purchased IP from companies that are not affiliated
Yes
No
b. Licensed or sold intellectual property (IP) to companies that are not affiliated
c. Transferred intellectual property (IP) to companies that are not affiliated through participation in
technical assistance or "know how" agreements
d. Received IP from companies that are not affiliated through participation in technical assistance or
"know how" agreements
e. Participated in cross-licensing agreements with companies that are not affiliated
f.
Acquired or merged with companies that are not affiliated to acquire their IP
g. Transferred IP to a spin-off or spin-out of your company
h. Received IP from a parent company as part of a spin-off or spin-out
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 23 of 88
i.
Allowed free use of patents or other IP owned by this company (for example, allowing free use of
software patents by the open-source community)
j.
Made use of open-source patents or other freely available IP not owned by this company
Generally, participants found this question to be too wordy, with participants taking long pauses
before providing an answer. Many said they were not sure about specific options, such as option
e. Just three participants said that this response option made sense, and they did not have
anything applicable for this response option. One wanted joint venture and commercial license
definitions included in the question. Another wanted ‘don’t know’ or ‘not applicable’ response
options, especially about the open source option. This question would need to be simplified to be
more accessible for respondents.
•
“Don’t know #5 so I would rather have option than say no. For open source, not
applicable would be a better answer choice. I don't know answer last question.” –
manufacturing SU
Q6: Collaboration with Universities
During the three years 2020 to 2022, did this business engage in any of the following collaborative
activities with universities? Select one for each row.
a. Performed collaborative research between company and university researchers
Yes
No
b. Funded university research
c. Funded PhD students and postdoctoral researchers at universities
d. Licensed or purchased IP from a university
e. Signed non-disclosure agreements, material transfer agreements, or other agreements with
universities governing usage of IP
f.
Hired science and engineering graduates
g. Engaged in formal or informal personnel exchanges with universities
Most participants said this item was clear. One participant mentioned he might have to talk to
HR about hiring for the graduate student question.
Q7, Q8, and Q9: Collaboration with Government Laboratories
During the three years 2020 to 2022, did this business engage in any of the following collaborative
activities with federal laboratories including Federally Funded Research and Development
Centers? Select one for each row.
Yes
No
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 24 of 88
a. Conducted collaborative research using agreements such as Cooperative Research and
Development Agreements, Space Act Agreements, Material Transfer Agreements, Confidential
Disclosure Agreements, or other transaction authorities
b. Conducted research at government user facilities
c. Funded research at government laboratory
d. Licensed IP from government laboratory
e. Engaged in formal or informal personnel exchanges with government laboratories
f.
Attended workshops or seminars located at or sponsored by government laboratories
Collaboration with Government Laboratories
During the three years 2020 to 2022, did this business engage collaborative activities with federal
laboratories in the following departments? Select one for each row.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Department of Agriculture
Department of Commerce
Department of Defense
Department of Energy
Department of Health and Human Services
Department of Homeland Security
Department of Transportation
Department of Veteran Affairs
National Aeronautics and Space Administration
Other federal department
Collaboration with Nonprofits
During the three years 2020 to 2022, did this business engage in any of the following collaborative
activities with nonprofit organizations? Select one for each row.
a. Performed collaborative research between company and nonprofit researchers
Yes
No
b. Funded research at nonprofit organizations
c. Licensed or purchased IP from a nonprofit organization
d. Signed non-disclosure agreements, material transfer agreements, or other agreements with
nonprofit organizations governing usage of IP
e. Engaged in formal or informal personnel exchanges with nonprofit research organizations
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 25 of 88
These questions performed as expected, although they could be grouped together or screened to
reduce response burden. One unmoderated respondent who wrote in said that the “survey was
well-written, simple, and short. If/when you implement the survey, share the results with the
Federal Lab Consortium.” Note that no participants from the unmoderated testing indicated that
their company was doing IP.
Recommendation: Shorten section. Add “don’t know” options and hover over
definitions where appropriate.
Reshoring
The reshoring questions were mostly not applicable to companies in this testing – two
unmoderated participants answered “yes” to the company reshoring in the past year, and no one
said they had plans to engage in reshoring in the future. During the moderated testing, two
respondents contemplated whether these questions applied to them but ultimately selected no.
For one of these participants, she thought reshoring might apply to her company, but responded
‘no’ because the company was opening a factory rather than moving it. For the other participant,
the part of the question specifying international trade felt unclear to him, so he selected ‘no.’ Of
note, the reshoring questions had the most participants break-off from the survey altogether
during the unmoderated testing, suggesting that this item may lead to survey break-offs in
production.
While most individuals seemed to understand the intent of the questions, there was some
confusion about what activities should be included, especially those to do with foreign
manufacturing. There was not consensus on if using foreign suppliers would count as reshoring.
•
•
•
•
•
•
“It's actually more cost effective to build it here, than to build it in China and ship it.” –
manufacturing SU
“I am trying to think of the best way to answer. We follow our customers, and some
moved in 2023 because of new laws like inflation reduction. We're reopening a US
location...If I read this and answer it exactly as it's asking, we're not shutting down a
facility in Mexico to move to the US... I’m comfortable reporting to you. It's not
something that happens too often. It's pretty infrequent for us, with the changes here with
the laws.” – manufacturing MU
“I do not know onshoring and reshoring definition but would need to find out. Based on
my knowledge, I'm not aware of new productions and services that occurred in 2022
without confirming with various internal departments.” – unmoderated, answered “do not
know” to Q1
“Would not include for 3rd party suppliers, only if switched from intercompany supplier
to in house. Hard to track all the suppliers and the physical location of the production at
my position.” – unmoderated, answered “no” to if they would include foreign supplier
“I wouldn't typically consider onshoring or offshoring from a supply chain perspective.”
– unmoderated, answered “no” to if they would include foreign supplier
“I consider internal sources only - not where supplies are sourced from” – unmoderated,
answered “no” to if they would include foreign supplier
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 26 of 88
This topic might be considered sensitive from our interviews: the previous round of testing on
reshoring questions on behalf of the Business Trends and Outlook Survey (BTOS) in 2023
resulted in high sensitivity from respondents. We also note the number of “don’t know” answers
to the question if a company will reshore on the unmoderated testing (25% of answers),
suggesting that if this information is known, respondents may be unwilling to share it with us.
•
•
“I would say no [to being told in advance of reshoring] because I am retired. In my prior
position, I would be involved. At some point, usually senior management, and then I
would get involved so that I would a new plant and setting up the accounting. 6 months
after the decision had been made I would know.” – MU with some overseas
manufacturing
“How far in advance [would I be told]? Maybe a year, maybe longer. Major facility
overhaul. Strategic thing to tell us and hold it close to chest.” - manufacturing MU
Recommendation: Additional testing with targeted recruitment is recommended for
reshoring questions, though we note that given the apparent low incidence of this activity
and the possibility of sensitivity of response, a survey may not be the most appropriate
method for collecting information about firms’ intent to engage in reshoring activities.
Labor
We tested a series of questions on organized labor. The labor questions were mostly not
applicable to companies in our testing. Although no moderated participants represented
companies with collective bargaining agreements, about 28% of the unmoderated participants
did represent companies with collective bargaining agreements in place. Participants felt it was
clear that these questions did not apply to their businesses. They were confident they would
know if there was unionizing activity in their companies through a variety of mechanisms.
•
•
•
•
“During the past 3 years, no. There were two attempts, and the employees voted 2/3 not
to unionize.” – MU manufacturing, unionized outside of US
“I would definitely know if they were unionizing” – MU restaurants and hotels
“We are a non-union shop” – unmoderated, no union
“I am not aware of any labor union. Labor Unions do not come in discussions about our
labor cost.” – unmoderated, no union
Most with collective bargaining agreements found the first question very easy, with some saying
it was neither easy nor difficult, and a few saying it was difficult. One person wanted a “do not
know” response option. Another said they liked being able to report approximate percentages
under the collective bargaining agreement. Three participants did say they would need to reach
out to HR or payroll to answer this question.
•
•
“I am new on this position and do not have the details already with me. For this survey I
just guessed based on what was discussed in the past.” – unmoderated
“We have some disclosures in our annual report in regards to collective bargaining.” unmoderated
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 27 of 88
•
“I participate in routine business updates for all divisions, which include discussion
regarding employee-related matters (headcount, hiring challenges, turnover, etc.) in
which matters like this would be discussed.” - unmoderated
Recommendation: Additional testing with targeted recruitment is recommended for
labor questions.
Entrepreneurship
We found varying amounts of accessibility around the information of the origins of a business.
Owners and those who inherited family businesses were the most ready to answer, with built
narratives of their company. However, some participants were not sure if their company would
fit the response options, even if they knew the reasoning behind the founding.
•
•
•
•
“From what I've read about the family history and from the memory of what my dad used
to say. He was in a family business and didn't agree with how it was run so he left and
started the [restaurant].” – Family business SU
“My grandfather started the business because he lost his leg in WWII, and then bought an
existing [medical] business. It’s kind of hard to answer, it wasn't exactly, all products are
customized. I’d probably say yes.” – manufacturing medical MU
“It would be that my history and passion for making it safer for [workers].” –
manufacturing SU
“I saw a need in the marketplace, and was frustrated over that need. It took a while to
come up with the solution, but we did. Ever been in a [hardware] store? [Customer
service invention]. We invented it in the late 80s. They're in a lot of stores now, and
chains. That was the first product. My personal idea.”- SU
Based on feedback from Round, we revised the question. Participants responded positively to
the edits in Round Two, and liked answering question three open-ended, but their answers did
not fit neatly into the options. There is a diversity of reasons to start a business, and these were
hard to capture, especially if the company provided services rather than a specific product. These
answers often were beholden to emotional narratives, like those of the American dream and
independence.
•
•
“You see the opportunity, you have right idea at the right moment. We realized we did
something great, now I understand what it takes. The opportunity, the thing happened
because the market, the product that existed was only overseas, there was no product in
the US. From another country, it was developed in other countries, but we don't have it in
my country. We developed a new product, but only for my country. It was very expensive
to buy the product, so we make domestic/local, we saw in the US there was no product
like that. We started selling that product in US. In US we created another company, all
the rights, start making the product.” – biotech SU
“In the early 80s. I've heard stories on why the founder founded it, but it was 40 years
ago. There’s a little history in the FCC filings.” – service MU
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 28 of 88
One participant found Q2 wordy and hard to answer. Participants did seem to struggle most to
answer this question, given the options.
•
“Again, wordy. I'm trying so hard to imagine what any bureaucrat, what are you getting
at?” – manufacturing SU
Recommendation: We recommend that this question series focus on owners rather than
asking proxies. Consider changes to wording and more cognitive testing to further refine the
question wording. There may be benefit from asking for open-ended answers and backcoding them into set categories, but this practice is time and resource intensive and should be
used with caution.
Demographics
Collecting demographic information on business owners poses a unique challenge on an
economic survey. There are two major issues we address here: first, many respondents are filling
out the questionnaire for owners (the proxy issue); second, new content may be sensitive,
especially for proxies (new content issue, specifically SOGI and disability). We divide our report
and recommendations by these two issues, which are related.
Reporting by Proxies
Most participants did not have trouble with identifying if they had owners, and who those owners
were. The instruction “someone owning 10% or more of the company” was helpful to orienting
participants.
There were a few participants who struggled to answer this question. Holding companies, for
example, were unsure who counted as owners. Similarly, law firms were not sure if equity
partners counted, and did not always know what percent ownership each had, especially if they
were unsure of the number of equity partners. Publicly traded companies had unique challenges
with the concept of ‘owner’ – for example, a participant representing a larger MU knew there
was an owner of over 15% of the company because it was publicly traded, but did not know
anything about this person or how to access them. This participant went on to describe:
“My recollection is I struggled with answer the question because it was looking for a person,
partnership. An equity finance company had purchased the company. I would suspect there
are people considered owners, but I'm really confused because they have different titles for
folks that represent the private equity company, and I don't have details about their personal
lives. It would have been helpful for something in the survey to communicate that. I only
know a couple of individuals names who meet with our senior individuals to review each
month, but I'm not in those meetings, and those people can change.” – participant at SU,
moderated
Even if participants could identify owners, proxies did not always know owners well enough to
answer demographic questions about them. Few of the demographic data asked on ABS are
available in records, and many respondents do not have access to HR information even if they
were held in records. This was especially true at large MUs, where respondents have not
interacted with the owners.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 29 of 88
•
•
•
•
•
“You might not have access to owners. I can answer about 80% of this section.” – proxy
at SU
“It would be more appropriate for someone there to have answer these.” – a proxy who
does not know the owner
“Definitely need a don't know [response option]. This is one of the ones that I struggled
with.” – proxy at a MU
“For age, I would look in HR records or age they finished law school.” – proxy at a law
firm
“There is an owner, but the ownership is an LLC, so there's one LLC that owns 51% and
it's made up of an individual and his family. I know the owner. I would be able to
answer.” – manufacturing SU
Q1: BUSINESS OWNERSHIP
Not including this business, what is the status of the previous business Owner 1 started most recently?
This is the owner’s first business
Business is still operating, and Owner 1 still owns it
Business is no longer in operation
Business was purchased by another company
Business was purchased by another individual
Other, specify: [textbox]
Q9: REASONS FOR OWNING A BUSINESS
How important to Owner 1 is each of the following reasons for owning this business?
Select one for each row.
Wanted to by my own boss
Flexible hours
Balance work and family
Opportunity for greater income
Best avenue for my ideas/goods/services
Unable to find employment
Working for someone else didn’t appeal to me
Always wanted to start my own business
Wanted to carry on the family business
Wanted to help an/or become more involved in my community
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 30 of 88
Other, specify: [textbox]
As such, Q1 and Q9 were the most difficult for proxy respondents. They were more subjective,
and had to do with the opinion of owners. Participants found Q1 wordy, and one wanted a “select
all that applied” response option.
•
•
•
•
“I would have to ask. It’s very subjective. I would be pretty annoyed if you gave this to
me. This is not something that the equity partners find a great deal of value from. It’s not
a billable experience.” – proxy at a law firm in response to Q9, which she rated as the
most difficult question to complete
“This would be a challenging one to get, to be honest with you. Even who I would think
of on the executive, getting them to answer these questions. Only they could answer and
[they are] not super accessible. [It would result in an] email sitting in an inbox for a long
time.” – proxy at a manufacturing MU
“When we receive these survey requests, it's not like [the] owner is coming into here and
filling out report. It’s a point person in corporate accounting. Would I know why the
owner wanted to start the business? It’s probably a portion of all these different reasons,
more personalized for an individual specifically. More likely than not it would be
delegated on the staff.” – proxy at MU
“I can answer because I've had discussion, but if you're not in a small organization you
wouldn’t know.” – proxy at a SU
Q9 was received the most negative response in the demographic section from both owners and
proxies. While participants liked the “don’t know” and “not applicable” response options,
proxies felt they could not answer the question, and owners did not always feel the options
captured their experiences. One participant pointed out that there was a difference between desire
and reality the options did not capture. For example, options on work/life balance and hours did
not always have intent and practicality aligned.
•
•
“’Not applicable’ is different than ‘don't know,’ like for family business.” – proxy at
MU who identified Q9 as most difficult question to answer
“Most of it doesn't apply. I would put other, specify: very important is the passion, is
helping others. For community, I'm involved being on boards. That's the community
where I live. Helping others in my industry.” – Owner of manufacturing SU
As a result of the Round 1 testing, we added the following question to the unmoderated survey
on demographics, and it had improved performance in the field.
Figure 1. Updated question on ownership that allows for the identification of proxy reporting.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 31 of 88
Recommendation: We used the question in Figure 1 for the unmoderated demographic
section. We asked during moderated testing if a participant was responding for both
themselves and other owners, and it performed appropriately during testing. As such, we
suggest adding that response option here. It is vital to analyze the difference in answers
between proxies and owners, and what kinds of companies have owners answer rather
than proxies. Consider rewording Q1 and Q9, only showing them to proxies, eliminating,
or adding “don’t know” and “not applicable” options.
Sensitive Topics
Several of the measure of demographics may be considered sensitive: gender identity, sexual
orientation, and disability. LGBTQ and disabled individuals are stigmatized, and often face
challenges to disclose these identities safely in the workplace. Additionally, sexual orientation is
rarely collected in employee records, meaning that there is no universal official source of owner
sexual orientation that a survey respondent can use to provide response to the survey questions.
This stigma, and the mechanism of proxy reporting, adds complexity to reporting these
demographics. These questions were asked in a battery of existing demographic questions to
gauge the participants’ responses to being asked about demographics in general.
We want to emphasize that, despite our efforts to recruit LGBTQ individuals to answer about
SOGI questions, participants only answers “male” or “female” for the gender identity question.
Two participants gave any response other than “straight” for the sexual orientation question, and
both were given by proxy. There were a few owner-participants who thought they may be
disabled, including a Deaf participant who did not identify as disabled but answered yes to the
question, and another who spoke about his brother’s experiences being disabled.
We first found that owners were much more comfortable than proxies reporting demographic
answers. This was true of all three of the sensitive questions, and also true of the question on
owner age. Owners in general had less negative reactions to being asked these questions as a
pattern, although some owners did feel explicitly negatively about answering these topics. Both
groups endorsed the ‘don’t know’ option on all of these questions, and many endorsed adding it
to the age question.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 32 of 88
•
•
•
•
•
•
•
•
•
“Was transgender on the question before? I’m trying to be inclusive.” – Owner for
holding company on the GI question
“Looks like it aligns with what I'm thinking, much more inclusive.” – Owner on GI
question
“Pretty straightforward, being of the older generation, it was either male or female [for all
the owners].” – Owner with family business on GI
“It would be weird for an employee to answer on behalf of the owner and there are
probably instances where employees don't feel comfortable asking the owner - so it's
good you have a ‘don't know’.” – owner on SO
“I know for other owners. The ‘don’t know’ is good if you don't want to be asking other
owners that. But all my family, my parents and my brother is buying the business from
them.” – Owner about SOGI in general
“I don't know, I guess I think that most people would know how they identify. It's
different to say ‘I don't know’ than ‘I'm choosing not to in this survey’.” – Owner for
SOGI
“I would feel comfortable answering for the two other owners because they are my sons”
– Owner who felt negatively about SOGI
“I recognize that there are minority issues for people - it's the same as asking racial
questions. For example, a woman owned business is given some more opportunities than
a male owned business because of history, and the same thing is true for sexual
orientation.” – owner after SO
“For me, I'm not owner, I shouldn't answer this question. I have an idea, what people do
on their personal time, I have no idea. I might be guessing on these.” – proxy
Proxies were concerned about revealing information about owners, being unsure how to collect
such information, or being able to find it in records. Two proxy participants did think it was
illegal to disclose this information, or ask about it.
•
“Company does not have records on the gender identity of its owners, in fact, it's
prohibited in [our state].” - proxy
Many participants did not understand how the ABS worked in general. They mentioned,
especially in unmoderated testing, that this made them wary of answering the SOGI and
disability measures. They were not sure how the information would be used, did not understand
the relevance, and had some privacy concerns. There were a few that asked for a blurb or more
information on the survey to help them determine the appropriateness of responding. One
unmoderated participant said that the questions “seem invasive, would prefer to understand why
it is relevant to business ownership/Census matters.”
In this case, the issue with the sensitive questions is not comprehension of the question but rather
reluctance to disclose a response. Participants did not have an issue with the options presented,
and most understood what the questions were asking. In moderated interviews, 12 participants
positive responded about SOGI questions, 16 neutrally, 10 negatively.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 33 of 88
•
•
•
•
“We do have LGBTQ [owners]. These questions would capture that.” – proxy from law
firm after SO
“We now, all public companies on the stock market, proxy put a table in for diversity of
gender. We chart it, including nonbinary [gender identity]. We have tracking for public
companies. I would know, I can answer. Certain people I know are gay. We have some
board members. We have a matrix for proxy filing, and we email with a questionnaire.
We say this is required and they might say I'm not answering. We have two board
members, where one is gay and one is a lesbian.” – proxy from publicly traded MU after
SO
“I don't feel like the order is discriminatory but someone else may so maybe it's easier to
put in alphabetical order to be politically correct or maybe separate instead of having a
list - like two on right and two on left - so someone wouldn't see any discrimination or
anything.” – owner of restaurant MU after GI
After the sex question: “This is a loaded question these days. The world has gone crazy
over sex and gender these days. I never see sex on questionnaires these days. I don't care
this much about it - but I don't feel comfortable with sex as opposed to using gender. It
shows indifference and insensitivity to the way things are today - you will have a
negative consequence if you leave the question like this.” After the gender identity
question: “I'm offended by the sex question. The gender identity question is better.” –
owner of telecom SU
Individuals who responded negatively to the SOGI questions often held negatively
preconceived notions about LGBTQ people and used language from political talking points.
This is reflective of the charged nature of this topic, and may lead to data quality issues in
collection.
•
•
•
•
•
“I understand that human beings have changed a lot, but these kinds of question are too
much. I don’t have problem with lesbian and bisexual but they push it’s normal. It’s not
normal. To push information for the children is not normal.”
“I know where this shit is going. I know it’s the world we live in. I’m not saying it like a
Trumper. I get it, I know it, we have to deal with it. It’s just so meaningless. To
somebody who is grumpy old dude like me, why are we spending time [asking this]?” –
owner of SU
“I would say female but then get pissed off about it. I don’t see how this matters. You are
who you are. I’m familiar with the answer categories. Our company does not have
records on gender identity of the owners. I’ll do it if I have to. I’m an old lady and I think
it’s a private thing. I think it shouldn’t matter with business. Who cares? I’m libertarian
in this regard.”
“Transgender people want a lot of attention and this gives them a lot of attention”
After the gender identity question: “This makes me crazy. I’m a Republican. Ask me if
I’m male or female and go on. I would answer this for my husband and myself.” After the
sexual orientation question: “You are overstepping your boundaries…My husband and I
would be really hating the government to see this – this is a waste of time. I understand
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 34 of 88
•
the leanings of our country is more inclusive but sexual orientation does not matter for
business ownership and doesn’t matter what you do in your own time.” – owner.
After the SO question “Oh my goodness – now this – now I do have a problem. … I don’t
know why you would need to ask this. This has absolutely nothing to do with being a
business owner. This would irritate me because this would be someone searching for
money to give for special preference, or a way to ... In my economy, people need to be
responsible for their actions. Personally I don’t look at someone different – come to
work, let’s get it done. I don’t understand why the Census Bureau would be asking this.
When I come across stuff like this, I would say ‘don’t know’ or ‘do not apply’ – as it’s
not the Census Bureau’s business.”
The responses from the unmoderated survey were notably more negative, and these negative
responses were more charged, political, and potent. This is in line with methodological literature
that suggests that self-administered collection reduces social desirability bias – respondents are
more forthcoming because they feel less social pressure to give answers that are socially
acceptable. Note, though, that there were also positive and mixed or neutral responses to the
unmoderated testing for these questions, too.
•
•
•
“I think it is great that you are asking and I think all companies should be required to do a
self-identification survey that will remain anonymous if the employee chooses, but will
help create transparency around the diversity of the organization and roles.”
“I am the sole owner and well aware of my status in all regards, but I do find the
questions to be intrusive. I can see that it is an important issue in general but I don’t see
that these are things that a business should have to know about their employees.”
“Personally, I do not believe it is relevant as a lot of these questions are not known by
anyone other than the owners. In most scenarios, there are multiple owners of a business
and it would be difficult to know how to answer this clearly. I understand the Census
Bureau is attempting to compile/categorize business ownership demographics by age,
sex, gender identity, and sexuality, but I think most owners may not feel comfortable
sharing this information. That’s my opinion. Thanks!”
We also received negative responses in our emails to the survey. It is important to note the types
of negative responses, and how they can be organized. Below are the categories the free
responses can be assigned to, with at least one example quote. There were 97 open responses in
cognitive testing, and a quote could be categorized multiple times:
•
•
•
37 comments were about the question being irrelevant to business
o “What does any of this have to do with owning or running a business?”
15 comments were about not being government’s purview
o “It is intrusive and has no bearing on the economic health of the business and no
bearing on the US economy as a whole.”
12 comments expressed that no one cares/it does not matter
o “I think it shouldn't matter what gender someone is or what their sexual
orientation is. The only thing that should matter is if the business has honesty and
integrity and provides the services or products that you are interested in.”
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 35 of 88
•
•
•
•
o “TOTAL WASTE OF TIME - WHO CARES?”
19 comments noted that the questions are none of your business/prying/personal/intrusive
o “A little too prying.”
o “It is information that we would not retain in a corporate record, but I have
personal knowledge. I also really felt my skin crawl answering those highly
personal questions.”
12 comments gave strongly negative
o “NEGATIVE, INTRUSIVE, DISCRIMINATORY, HATEFUL, NOT
APPROPRIATE OR IMPORTANT INFORMATION, SHAMEFUL!!!!!”
o “Stupid government shit”
o “NONE OF YOUR GOD DAMN BUSINESS”
5 comments noted privacy concerns
o “How would this information be used? Would the data be anonymous? Who
would have access to the details if these questions were included?”
7 comments were overtly political/homophobic
o “I think there is too much concern lately about sex, gender identity, and sexual
orientation. There are ONLY two sexes/genders. Male & Female. And NO, men
cannot have babies - at least not real men. Why sex, gender identity, and sexual
orientation is relevant to your surveys, is a mystery to me. Who cares? What's
the difference? Gender is determined at birth. As I said, there are only two
genders. Gender identity is something that goes hand in hand with wokeness, and
trust me the majority of the population is tired of hearing about it. We need to
stop the brainwashing of our children. As for sexual orientation, that's something
that is a personal choice and a personal issue. I, and most people I know, don't
care who anyone sleeps with; that's their private business. Why do we have to
keep publicizing it? And why do we have to keep hearing about who isn't
heterosexual? Heterosexuals don't require or demand special treatment, special
recognition, or special days or months - why do others with different sexual
orientations? Sexual orientation, like skin color, is irrelevant. We're all people.
Sexual orientation and skin color don't define us. It's time to stop this nonsense of
labeling people.”
o “Please understand that not everyone wants to know that information about you,
and it bothers some people. As a business owner I believe there needs to be
separation between what goes on at work and in your personal life.”
o “Sex, gender, and sexual orientation are totally irrelevant. These are not relevant
in determining who is qualified to running this, or any business. Identity politics
are creating more problems by moving away from decision making based on
merit.”
o “This gender identity issue is a bunch of BS. This is a complete waste of time and
brings no value or understanding of our population at large and business in
general. Only in a country as affluent as ours can a person look at their genitalia
and believe it to represent the opposite sex. Any hiring requirements predicated
on sex, sexual identity, race, ethnicity or any other consideration except their
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 36 of 88
ability to contribute to the company's profitability is misplaced and frankly,
tyranny. Any more questions?”
o “I think that stuff is a bunch of crap. Not into that woke stuff. They are
brainwashing people.”
More participants were willing to answer the gender identity question that the sexual orientation
question. This is most firmly demonstrated in item nonresponse and drop off rates in the
unmoderated survey, as show in table 5.
Table 4 charts item nonresponse and drop off for these measures. Nonresponse means that a
participant skipped answering that question but continued with the testing. Drop off means the
participant did not answer this question, and did not answer any subsequent questions in the
testing.
Table 4. Item nonresponse and survey drop off for sensitive demographic questions in the unmoderated survey (n = 149), Any
answers with less than 5 companies are labeled n/a.
Item nonresponse
Drop off
Sex
18
11
Gender Identity
28
n/a
Sexual Orientation
60
12
This pattern held true for moderated testing as well. In moderated testing, 10 participants would
not answer the sexual orientation, while three would not answer both gender identity and sexual
orientation. Some participants had a stronger reaction to the sexual orientation question than the
gender identity question. Others were less comfortable reporting sexual orientation than gender
identity.
•
•
•
•
•
“I understand asking about someone's sex but gender identity and/or orientation is not
relevant.” - unmoderated
After the GI question: “We have [diversity equity inclusion] surveys. We track that
information internally, and some of it I know. I don't go around and ask someone how
they identify.” After the SO question “A bigger firm has an official tracking. I think you
need a track for single or group. To me it's clear based on my firm. If I don't know [if a
person is] gay or lesbian and [I will] fall back to heterosexual. I won't make the
assumption on what I know and has been shared.” – Law firm SU
After the GI question: “Here are the options I would have expected on the previous
question.” After the SO question: “I understand why all of this is here but it doesn't mean
much to me personally…I’m fine with answering this for the owners.” – retail medical
SU
After the SO question “I would never imagine that this would be in a Census. I would
feel like I don't want to answer because I don't know and I don't want to ask the owners. I
would say ‘Don't Know’.” – restaurant MU
After the SO question “Comfortable answering and providing a response to this, but I
don't see the relevance. My sexual preferences have nothing to do with how I run a
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 37 of 88
•
•
•
business. It's which of these things does not belong - I don't have a problem answering,
but it does belong with the others.” – manufacturing SU
After the GI Question: “It's awesome that it's inclusive, it might be challenging to get an
accurate answer.” After the SO question: “In the spirit of inclusivity, I don't know how
much value it adds.” – medical service MU
After the SO question: “If I didn't know the client personally, I would say ‘Don't Know’
because I'm not going to call to ask them.” – service SU
After the GI question: “I don’t mind question for, it seems appropriate.” After the SO
question: “I wouldn’t ask the owner.” – service SU
No participants said they preferred to have two questions to measure gender identity. However,
some preferred to have just one question, and had a preference for the gender identity question
over the sex question if just one gender identity question was presented.
•
•
•
•
•
“I don't know why they need to know the gender identity question. Fill out both the
same.” – restaurant MU
“Just be one. I thought I just clicked. I didn't even read that seemed redundant. Didn't like
clicking back. Don't mind answering both, but it's confusing. I clicked back
immediately.” – biotech manufacturing MU
“Seems like we should honor gender identity. If we go back to three, sex, seems like you
could just have question 4.” – manufacturing SU
“Questions 3 and 4 seem to be the same.” – service SU
“Just go with question four, going to go to question four anyway, just go there.” –SU
We note the particular demographics of reports for the ABS and how this may play into answers.
Many mentioned their ages as a factor as to why they might be uncomfortable with SOGI
measures. These individuals who select into their professions may also be more risk adverse or
conservative. Company culture also seemed to dictate levels of comfort. Companies that were
more open about personal lives knew more personal information about each other. Other
companies, where it was expected to come to work and leave without engaging around personal
lives, were more uncomfortable with what they considered “personal” questions. This also may
mean the owners at their companies may not disclose LGBTQ status. This, combined geographic
variation in acceptance of LGBTQ populations, could impact responses. Many such participants
spoke about how they did not care about LGBTQ identities and how they should not matter,
echoing a colorbind narrative.
•
•
“I think it's a big joke, all the fuss over people's gender…Well, I think it's intrusive
asking about sexual orientation and you would not know. I'm showing my age, and I don't
think it matters either way, so who cares? Like I said, it would be my educated guess
based on a photo and that's all I can tell you. I don't usually discuss people's anything like
this with someone I work with.” – participant from a MU who was older. He also
confused sexual orientation and gender identity.
“Don't know, don't understand why that's important. What does the sexual orientation of
the business owner have to do with the business? That's my personal opinion. Not
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 38 of 88
•
•
•
•
•
•
•
something I would know, because I would feel like they would feel that to themselves.
Not something we ask or keep on file. This is something that I don't talk to people about
at work, that isn't something that is important to me. They are a person, so I don't feel like
that's my business what they do outside of my job. Some people are fine sharing it, that's
not something that's important to me. they're a person. I don't ask unless they specifically
told me.” – SU, conservative area, older
“I am older so I am not as familiar with the terminology but it's not something in my line
of work I deal with.” – manufacturing SU
“I don't talk to people at work about this and not my business. If I knew I would be
comfortable reporting it, but I don't know and we do not ask.” – manufacturer SU
“This is something that I don't talk to people about at work. That isn't something that is
important to me. They are a person, so I don't feel like that's my business what they do
outside of my job. Some people are fine sharing it, that's not something that's important to
me. they're a person. I don't ask unless they specifically told me.” – biotech SU
“I'm showing my age, and I don't think it matters either way, so who cares?” – law firm
MU
“Not important to me. You just have to be a decent human being. I don’t care about your
private life. I’m not paid to hang out.”- participant described the geographic location as
“not too much anti-woke in the area” – manufacturing SU
“Our company is pretty open, and people I think feel pretty comfortable.” – participant at
MU in liberal area
“We do have people, so I understand the answer to the question and what all the other
things mean.” – Proxy at SU
We note that during testing, several moderated interviewing participants noted they were more
comfortable answering SOGI questions on the aggregate level, where the question might ask if
any owners, instead of a specific owner, identifies as LGBT:
•
•
•
•
“People get huffy with individual level. Group more okay with it.” – manufacturer SU
After the GI question: “If you want to give trans nonbinary, then you should establish
grants. You can say “we have 57% trans owners than the year prior.” If somebody is trans
or nonbinary then they can try to get grant money, but this is survey business.” After the
SO question: “I can answer it about myself, even the question, this is a classic rubber hits
the road. If you have to know answer, then I would just say, do you have anyone under
the definition of LGBTQ? Okay there are 4 business owners, so somebody did. More
than 25% then you extrapolate.” – manufacturing SU
“For age, I’d rather aggregate across the 5 owners for age. Same with disability, yes/no.”
– owner/proxy SU
After the sex question: “I could do percentage.” After the gender identity question;
“Percentage of is better. This would be something that I would probably write don't know
on, I don't want to assume and/or talk to our firm manager.” – law firm SU
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 39 of 88
Like members of the LGBTQIA+ community, the disabled business owner population is a
stigmatized population that is not well defined and can be challenging to survey. Disability status
had the added complexity of no set definition of disability, and the American with Disabilities
Act (ADA) dictates legal accommodations in the work place, so businesses have a legal
obligation to accommodate disability status. As such, the first version of this question was
adapted from language in the ACS, as shown in Figure 2. This was the question used in Round
One of testing:
Figure 2. Round 1 Disability Status Question, adapted from the American Community Survey
Most participants thought this was asking if disabled owners could complete their jobs, or about
the business’ efforts to be ADA compliant. This was not the intention of the question (to collect
information on owner disability status):
•
•
•
•
•
•
“We have not reached the threshold to have this information. I would know because of
payroll, may see. It might be confidential.” – biotech SU
“I’m kidney dialysis patient, but it doesn't interfere with performing my duties. I consider
myself to have a chronic health condition, but it has not caused any disability for me.”
“When I hear physical impairment, I think of difficulty getting to the office. Mental is
like Alzheimer’s. This could use descriptive [instructions].”
“Disability wouldn’t be in my records, not even accommodations.” – Law firm SU
“I could only guess. I wouldn't be certain. A psychological mental impairment, if
someone has depression, I don't know if that fits into that category, nor should they have
to. If someone is physical disabled, it's a little bit easier. It’s confusing. Are you talking
about the correct angle of a wheelchair ramp? Are we facilitating space or do people who
are invisibly self identify and then let management know? If the US Census Bureau asked
us to track that and a requirement we have to keep confidential. Probably for a lot of it, if
people are getting their work done perfectly fine, unless they tell me they identify, I'm not
going to presume that they do.” - Law firm MU
“So what are you asking me on this question again? It’s not clear what this question is
asking. It's clear what it's asking, what is the why? Owners, they have little impact on
what is produced…Major impairment isn't the right. Impairment makes me feel like
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 40 of 88
Coolidge had a stroke theoretically and is the one running the country. Impairment as
capacity for business stuff.” – manufacturing SU
In response to the feedback we received in Round 1 of testing, we altered the question wording
for Round 2.
7. DISABILITY
Does Owner 1 have a chronic physical or mental condition that they manage in daily life?
The Round 2 version of the question was better received. This measure was not interpreted as
asking about ADA accommodations, or the ability to perform job tasks. However, there was still
little consensus on what counts as a disability or condition. This may be another example where
proxy reporters might have a poor understanding of who would identify as disabled if they are
not familiar with disability, or may not know this about coworkers, as these data are not present
in company records.
•
•
•
•
•
•
•
•
•
“For me disability is that you cannot walk, do not have arms, but you can what you're
doing. If your mental condition isn't good, you're not doing the survey. For me, it's like
mobility things.” – biotech manufacturing SU
“I would think it’s any handicap where, anything you would need the sticker for your car.
You have to be active in a restaurant. I wouldn't think it's asking about ADA or can't do
their job.” – restaurant MU
“I don't know have anything like this. HR did this have this, they would be willing to give
out that information. Probably share the survey and have HR do it, not sure if they record
it.” – food manufacturing MU
“I’m surprised to see it on a Census [survey]. I don't know why I'm providing health
information to the government. It’s harder to answer when you're not the person.
Disabled carries a lot more of a negative health connotation. Disability and condition
don't mean the same things at all. Diabetic, I wouldn't look at that as a disability, for
example.” – biotech manufacturing SU
“Personal information would not have line of sight to…I feel like this is important, to be
a voice for my brother right now. I think it's very very important, but the way is question
is reading it comes off as a negative thing. I don't know how to word, more interested in
what they are, rather than limitation.” – service MU
“It doesn’t smell like ADA.” – manufacturing SU
“If you go into it, I manage pain in my hip on a daily basis, but it wasn't, by the question,
I wasn't thinking that. Maybe some examples what you're looking for there [would help].
Daily pain, headaches. We all manage all kinds of things in daily life.” – owner
“Unless they share that with me, I wouldn't know…It’s a personal and confusing
question. A proxy wouldn’t know that.” – SU proxy
“Chronic, to me, almost a permeant thing, not temporary. I have pretty serious lung
cancer, which is a physical condition limits me, but it's hopefully not chronic. It’s only
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 41 of 88
for two months and three or four would be passed that. Guess most people would read it
that way…I like that mental is on there.” - Owner
The second version of the question performed more consistently in testing. However, it is still a
sensitive question. It may be easier for owners than proxies to give accurate response. Further,
we cannot offer a concrete definition of disability to respondents because of the lack of concrete
definition in mainstream understandings of disability, and how marginalized and stigmatized the
population is.
Recommendations: We strongly recommend the inclusion of an owner/proxy reporting
question, like that of the one tested here. It is important to know the differences in
answers, data quality, and characteristics of companies with proxy reporters. This has
bearing on sensitive questions too. We suspect there will be differences in how owners
and proxies report these questions. It is also possible some demographic questions are not
appropriate to ask proxy reporters.
Sensitive questions would benefit from other types of testing outside of cognitive
interviews. This is particularly true because participants did not have a problem with the
measures themselves, but with the topics. Methods like focus groups, targeted in depth
interviews, usability testing, and respondent debriefing interviews might better get at
motivations and behavior. This would shape approaches to asking about these topics.
Similarly, we are concerned by the lack of participants identified as LGBTQ and disabled
in testing sensitive questions. We propose identifying pools of participants, especially
owners, from this population for additional testing. Community based methods, such as
focus groups organized by LGBTQ Chambers of Commerce or snowball sampling,
would better address if these questions are appropriate for counting these populations.
We recommend exploring other means of measuring sensitive demographic information
(including age). Some questions might only be shown to owners, and not proxies. Some
could be given as yes/no (such as “Do any owners identify as part of the LGBTQIA+
community?”) rather than a list of identities. Delegation functionality could be used to
send sensitive questions directly to owners.
We recommend testing a 1-2 sentence explanation about why we are asking a question,
what the information will be used for, and that it is confidential before each sensitive
question. For SOGI questions, this is in line with White House recommendations.
For this round of the ABS, we recommend eliminating the sex question and including the
gender identity question. The use of one question, rather than two, is in line with White
House guidelines for business surveys. If the sex question is included, we do not
recommend asking about the sex “assigned at birth,” “on a birth certificate,” or
“biological sex,” given that a proxy reporter would not know this information. We
recommend more testing, especially with LGBTQ participants, to include the sexual
orientation question on the production survey.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 42 of 88
Similarly, we recommend more testing with disabled participants for the disability
question. However, version two is more in line with the intent of the question as it relates
to business surveys. In the future, we suggest testing a list of applicable conditions as
examples. This may be especially helpful for populations where there is a debate if they
culturally consider themselves disabled, like the Deaf, diabetic, and autistic communities.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 43 of 88
About the Data Collection Methodology and Research Branch (DCMRB)
The Data Collection Methodology and Research Branch (DCMRB) in the Economic Statistical
Methods Division (ESMD) assists economic survey program areas and other governmental
agencies with research associated with the behavioral aspects of survey response and data
collection. The mission of DCMRB is to improve data quality in surveys while reducing survey
nonresponse and respondent burden. This mission is achieved by:
•
•
•
•
Conducting expert reviews, cognitive pretesting, site visits and usability testing, along
with post-collection evaluation methods, to assess the effectiveness and efficiency of the
data collection instruments and associated materials;
Conducting early-stage scoping interviews to assist with the development of survey
content (concepts, specifications, question wording and instructions, etc.) by getting early
feedback from respondents;
Assisting program areas with the development and use of nonresponse reduction methods
and contact strategies;
And conducting empirical research to help better understand behavioral aspects of survey
response, with the aim of identifying areas for further improvement as well as evaluating
the effectiveness of qualitative research.
For more information on how DCMRB can assist your economic survey program area or agency,
please visit the DCMRB net site or contact the branch chief, Amy Anderson Riemer.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 44 of 88
Appendix A –Moderated Protocol (Round 1)
2024 ABS Content Testing Protocol Draft – 07/2023
General Research Questions:
• Do respondents understand what information they were asked to provide with this question?
• Do respondents understand the terminology used in the question?
• Are respondents able to answer the question that is being asked?
• Is the question layout clear and understandable?
• Is the requested information available in respondent’s records?
• Are the lists/categories organized effectively?
• Are the instructions helpful to respondents?
Informed Consent: Respondents will be asked to complete a consent form electronically before the time
of the interview.
Materials Needed:
• Electronically signed consent form
• Copy of questionnaire
Introduction
• Introduce everyone on the call
• Thank respondent for completing the survey
• Brief overview of why survey exists and role of NSF and Census
o The Annual Business Survey (ABS) is conducted jointly by the U.S. Census Bureau and
the National Center for Science and Engineering Statistics within the National Science
Foundation. ABS provides information on selected economic and demographic
characteristics for businesses and business owners. Additionally, the survey measures
research and development, innovation and technology, as well as other business
characteristics
• Tell respondent why we asked to speak with them
• Suggestions for improvement
• Refine questions and make them easier to answer
Recording the interview
• As mentioned in the consent form that you signed, we would like to record this interview so that
we have an accurate record of your feedback. We plan to use your feedback to improve the design
and layout of the form for future data collections. Only staff involved in this data collection will
have access to the recording. Would that be ok? If respondent says no, move on.
About the Respondent
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 45 of 88
•
•
Can you tell us a little bit about your business -- what types of goods or services does this
business provide?
What is your role in the company? What kind of responsibilities do you have?
General probes that may be used for any question or module throughout the interview:
• In your own words, what is this question asking?
• Reflect back on respondent’s answer: “you said…”
• How did you arrive at this number/answer this question?
• What records (if any) would you look at?
• Specifically, what would you include in this answer? What would you exclude?
• Would you consult other people to obtain this answer?
• How easy or difficult is it to answer this question?
Additional Probes Specific to Modules and Questions
Sustainability Management Module
Throughout this module, remain neutral on climate change, but note if the R has strong opinions or
objections to these questions.
Intro:
•
•
•
Please take a minute to read the introduction above the first question.
What sort of questions are you expecting to see in this module?
What does sustainability mean to you?
o In relation to your company? Can you give me some examples?
Questions:
1. How would you define ‘business priority’ in the context of this question?
a. Do all of the items make sense, are there any you are unfamiliar with?
b. Would you put anything under other?
c. When thinking about how you would answer this question, were you thinking about the
time frame from 2020 to 2022?
2. Note how respondents react to item C, as it is similar to item A.
a. How would you determine importance?
b. Do you have any thoughts about the answer categories/column headers?
c. Do all of the items make sense, are there any you are unfamiliar with?
d. Would you put anything under other?
3. What does ‘realized’ mean to you in the context of this question?
a. If ‘realized’ is confusing: is there another term or phrase that would make more sense
here?
b. Do you think anything is missing with the response categories?
c. Do all of the items make sense, are any there any you are unfamiliar with?
d. Would you put anything under other?
4. Note how respondents react to column headers and if there is any confusion. Note that item E is
mainly relevant to manufacturing.
a. If R answers yes to any items: Can you give me examples of what you track and how you
track it?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 46 of 88
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
b. Do you have any outside or consulting companies track these for you?
c. Do you have any thoughts about the answer categories/column headers?
d. Do all of the items make sense, are there any you are unfamiliar with?
i. Is the possible double counting of 4a. and 4b. with 4d. (HVAC) confusing?
If R answers yes: Can you give me examples or the name of the software you use?
If R answers yes: Can you give some examples of what your company tracks?
a. If R answers limited tracking and estimates: Can you tell me more about that?
i. Can you give me examples of the types of tracking or evaluation your company
is doing?
b. Do you find the text in parentheses and what is in italics helpful or unhelpful?
How do you define ‘value chains’?
a. What does this term mean to you in the context of this question?
b. Do all of the items make sense, are there any you are unfamiliar with?
c. Are there any other challenges that are missing from our list?
If R answers yes: can you tell me more about that? Can you give me any examples?
a. How do you define “develop a plan”?
If R answers yes: can you tell me more about that? Can you give me any examples?
Do all of the items make sense, are there any you are unfamiliar with?
a. Particularly, can you tell me what item D is asking in your own words?
b. Are there any other strategies that are missing from our list?
c. When answering this question, what time frame were you thinking about?
Note if R has difficultly in choosing an answer category. Note if R asks if the target date pertains
to only Scope 1 & 2 or to Scope 1, 2 & 3.
a. Do these answer categories make sense to you, or are they confusing?
Note if R mentions negative carbon emissions (highly unlikely). Note if R asks if the target goal
pertains to only Scope 1 & 2 or to Scope 1, 2 & 3.
a. Do these answer categories make sense to you, or are they confusing?
Do all of the items make sense, are there any you are unfamiliar with?
a. Are there any other effects that are missing from our list?
If R answers yes to any items, ask for examples.
a. Do all of the items make sense, are there any you are unfamiliar with?
i. In your own words, what is the difference between items E and F?
ii. We’re particularly interested in Item G, can you tell me what Item G means to
you in your own words? Note if R includes recycling.
b. Are there any other activities that are missing from our list?
Starting from question 15, the questions may be redundant, note if R mentions it.
a. What do ‘strategic objectives’ mean to you in context of this question?
b. If R answers yes: Can you give me any examples?
General probes
If R answers yes to any items, ask for examples.
If R answers yes: can you give me any examples?
a. When answering this question, what time period were you thinking of?
Were the includes helpful or unhelpful?
a. If R answers yes: can you give me any examples?
b. When answering this question, what time period were you thinking of?
If R answers yes: can you give me any examples?
a. When answering this question, what time period were you thinking of?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 47 of 88
21. After general probes, and if it was not obvious: When answering this question, what time period
were you thinking of?
•
•
Having gone through this entire section, about how long do you think it would take you to
complete?
Would you be able fill out this section, or would you need help from someone else in your
company fill it out?
Technology Transfer Module:
1. Are the includes helpful or unhelpful?
a. Do all of the items make sense, are there any you are unfamiliar with?
b. Are there other R&D activities that we are missing from this list?
2. General probes
a. How easy or difficult would it be to provide this dollar amount?
3. Does your company have any intellectual property? Can you give me any examples?
a. Do all of the items make sense, are there any you are unfamiliar with?
b. Are there other formal IP protections that we are missing from this list?
4. Do all of the items make sense, are there any you are unfamiliar with?
a. Are there other informal IP protections that we are missing from this list?
5. General probes
6. General probes
7. Do all of the items make sense, are there any you are unfamiliar with?
a. What does a ‘companies that are not affiliated’ mean to you?
b. Are there other mechanisms of transfer that we are missing from this list?
8. Do all of the items make sense, are there any you are unfamiliar with?
a. Are there other collaborative activities or agreements that we are missing from this list?
9. Do all of the items make sense, are there any you are unfamiliar with?
a. Are there other collaborative activities or agreements that we are missing from this list?
b. Is it always clear if the organizations you work with are federal government laboratories?
10. Note that this question has the same title, and this may confuse R (this will be changed).
a. If you work with federal government laboratories, is it obvious which parent agency the
lab operates under? How would you go about answering this question if it was not
obvious?
11. If R answers yes to any item, ask for examples. Note if R includes charitiable donations in any
item.
a. Do all of the items make sense, are there any you are unfamiliar with?
b. Are there other collaborative activities or agreements that we are missing from this list?
c. Is it always clear if the organizations you work nonprofits?
•
•
Having gone through this entire section, about how long do you think it would take you to
complete?
Would you fill out this section, or would someone else in your company fill it out?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 48 of 88
Labor Related Questions
These questions will also be tested online via Qualtrics.
1: General Probes
• How would you determine your answer to this question?
• Are supervisory and non-supervisory workers distinct categories to you?
2: General Probes
3: General Probes
• What does ‘certified’ mean to you in the context of this question?
Reshoring Questions
These questions will also be tested online via Qualtrics.
1: General Probes
• In your own words, what would you say this question is asking?
• How did you come up with your answer to this question? Would you need refer to records or
reach out to anyone else in your business to answer this question? If yes, Who/what?
• How helpful or unhelpful was the clarification after the question – very helpful, somewhat
helpful, neither helpful nor unhelpful, somewhat unhelpful or very unhelpful?
o Very helpful
o Somewhat helpful
o Neither helpful nor unhelpful
o Somewhat unhelpful (Do you have any suggestions for improving this clarification?)
o Very unhelpful (Do you have any suggestions for improving this clarification?)
• Are there other terms you would use to describe this activity (other than ‘onshoring’ or
‘reshoring’)?
• If you switched from a foreign supplier to a domestic supplier, would you include that here?
• How did you determine which parts of your business to answer for? Was there anything tricky
about answering for “this business”?
•
•
•
What time period were you thinking about? Was there anything tricky about answering for
“During 2022”?
How easy or difficult was it to answer this question – very easy, somewhat easy, neither easy nor
difficult, somewhat difficult or very difficult?
o Very easy
o Somewhat easy
o Neither easy nor difficult
o Somewhat difficult (Can you tell me more about that?)
o Very difficult (Can you tell me more about that?)
Do you have any suggestions on how to improve this question?
2: General Probes
•
In your own words, what would you say this question is asking?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 49 of 88
•
•
•
•
•
•
•
How did you come up with your answer to this question? Would you need refer to records or
reach out to anyone else in your business to answer this question? If yes, Who/what?
What do each of the answer choices mean to you?
If R selects an answer choice: You selected “xx” -- can you give me any examples of what you
were thinking about?
Do the answer choices make sense to you?
o Yes
o No (Which did not?)
o Don’t know
Are there any answer choices potentially missing from this list?
How easy or difficult was it to answer this question – very easy, somewhat easy, neither easy nor
difficult, somewhat difficult or very difficult?
o Very easy
o Somewhat easy
o Neither easy nor difficult
o Somewhat difficult (Can you tell me more about that?)
o Very difficult (Can you tell me more about that?)
Do you have any suggestions on how to improve this question?
3: General Probes
•
•
•
•
•
•
•
In your own words, what would you say this question is asking?
How did you come up with your answer to this question? Would you need refer to records or
reach out to anyone else in your business to answer this question? If yes, Who/what?
What time period were you answering for? Was there anything tricky about answering for
“During 2023”? [Probe about how confident they are about the remaining part of 2023.] Would
your answer have been the same or different if they question asked “In the next 12 months…”?
Do you have a preference for “During 2023” v. “In the next 12 months”?
How confident were you in answering this question for 2023?
o
o
o
o
Very confident
Somewhat confident
Not very confident
Not at all confident
If your company was planning to move production to the United States, how far in advance do
you think you would be aware of this?
How easy or difficult was it to answer this question – very easy, somewhat easy, neither easy nor
difficult, somewhat difficult or very difficult?
o Very easy
o Somewhat easy
o Neither easy nor difficult
o Somewhat difficult (Can you tell me more about that?)
o Very difficult (Can you tell me more about that?)
Do you have any suggestions on how to improve this question?
4: General Probes
•
In your own words, what would you say this question is asking?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 50 of 88
•
•
•
•
How did you come up with your answer to this question? Would you need refer to records or
reach out to anyone else in your business to answer this question? If yes, Who/what?
If R selects an answer choice: You selected “xx” -- can you give me any examples of what you
were thinking about?
How easy or difficult was it to answer this question – very easy, somewhat easy, neither easy nor
difficult, somewhat difficult or very difficult?
o Very easy
o Somewhat easy
o Neither easy nor difficult
o Somewhat difficult (Can you tell me more about that?)
o Very difficult (Can you tell me more about that?)
Do you have any suggestions on how to improve this question?
Now that we have looked at these draft questions, what is your overall impression of these questions?
How comfortable were you in answering these questions for your company – very comfortable, somewhat
comfortable, not very comfortable or not at all comfortable?
o
o
o
o
Very comfortable
Somewhat comfortable
Not very comfortable (Can you say more about this?)
Not at all comfortable (Can you say more about this?)
How confident are you in your answers to these questions – very confident, somewhat confident, not very
confident or not at all confident?
o
o
o
o
Very confident
Somewhat confident
Not very confident (Can you say more about this?)
Not at all confident (Can you say more about this?)
If you had been answering these questions without being on the phone with me, would you likely have
consulted records or others within your company?
o
o
o
Yes, records (What records would you consult?)
Yes, others within company (Who would you consult with?)
No
About how long do you think it would have taken you to complete this survey (if we hadn’t been on the
phone)?
_____ minutes
Thinking back through all the questions you answered for your overall company, do you feel you were the
best person to answer these questions or would someone else at the company have been a better fit?
o
Yes, I was the best person
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 51 of 88
o
No, someone else would have been a better fit (What is that person’s job title?)
Overall, how easy or difficult was it to answer these questions– very easy, somewhat easy, neither easy
nor difficult, somewhat difficult or very difficult?
o
o
o
o
o
Very easy
Somewhat easy
Neither easy nor difficult
Somewhat difficult (Can you say more about what was difficult?)
Very difficult (Can you say more about what was difficult?)
Research and Development Questions
D1. General Probes
D5. General Probes
• Are you able to break down the R&D costs for each of the different types of costs this question
asks about?
D7. General Probes
• Are you able to break down the funding sources for R&D activities?
• In your own words, how would you define venture capital?
• Has your business used venture capital as a funding source for R&D activities? If yes, could you
please tell us more about that?
• In your own words, what is angel financing?
• Has your business used angel financing as a funding source for R&D activities? If yes, could you
please tell us more about that?
D8. General Probes
Entrepreneurship Questions
1. General Probes
2. General Probes
a. What do you think this question is asking?
b. In your own words, what do you think is meant by “originally founded” in the context of
this question?
c. If the word “originally” was removed from the wording of this question, would it change
your interpretation or understanding of the question?
d. How do you interpret “new or customized product or service” in the context of this
question?
3. General Probes
Owner Characteristics: Sexual Orientation and Gender Identity Questions
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 52 of 88
These questions will also be tested online via Qualtrics.
Throughout this module, remain neutral, but note if the R has strong opinions or objections to any these
questions.
Note whether respondent is an owner of this business, an owner/proxy respondent, or a proxy respondent.
B.1.9. Prior Business Ownership:
• General Probes
• In your own words, what is this question asking? (Interviewer note: Does the respondent notice
the instruction for ‘Not including this business’ when responding to this question?)
B.1.12. Age
• General Probes
B.1.1 Sex
• General Probes
• In your own words, what is this question asking?
• How do you define Sex in the context of this question?
• Do you think there are any answer categories missing from this question? If yes, please describe.
• Does your company have records on the Sex of the owner(s)? If yes, what are the categories in
your records or systems for Sex of the owner(s)? And would you need to consult the records in
order to be able to answer this question?
• How comfortable are you in answering this question for the owner(s) of this business?
• Do you have any other thoughts on this question that we have not already discussed?
Gender Identity
• General Probes
• In your own words, what is this question asking?
• How would you go about answering this question if it appeared on a future Annual Business
Survey?
• How do you define Gender in the context of this question?
• In your own words, what is the difference between Sex and Gender Identity?
• Are you familiar with the answer categories in this question? If not, would definitions be helpful?
• Do you think there are any answer categories missing from this question? If yes, please describe.
• What, if any, are your thoughts on the order in which the categories are arranged in this question?
• Does your company have records on the Gender of the owner(s)? If yes, what are the categories
in your records or systems for Gender of the owner(s)? And would you need to consult the
records in order to be able to answer this question?
• How comfortable are you in answering this question for the owner(s) of this business?
• Do you have any thoughts on a two question approach to asking about Sex and Gender Identity?
• Do you have any other thoughts on this question that we have not already discussed?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 53 of 88
Sexual Orientation
• General Probes
• In your own words, what is this question asking?
• How would you go about answering this question if it appeared on a future Annual Business
Survey?
• How do you define Sexual Orientation?
• Are you familiar with the answer categories in this question? If not, would definitions be helpful?
• Do you think there are any answer categories missing from this question? If yes, please describe.
• What, if any, are your thoughts on the order in which the categories are arranged in this question?
• Does your company have records on the Sexual Orientation of the owner(s)? If yes, what are the
categories in your records or systems for the Sexual Orientation of the owner(s)? And would you
need to consult the records in order to be able to answer this question?
• How comfortable are you in answering this question for the owner(s) of this business?
• Do you have any other thoughts on this question that we have not already discussed?
B.1.14 U.S. Citizenship
• General probes
B.1.15 Owner Disability
• General probes
• In your own words, what is this question asking?
• What do you consider to be a physical impairment? Would it be helpful to list examples of
physical impairments with this question?
• How would you define a mental impairment? Would it be helpful to list examples of mental
impairments with this question?
• In your own words, how do you interpret “substantially limits” in the context of this question?
• What does “major life activities” mean to you in the context of this question?
• Does your company have records on the Disability status of the owner(s)? If yes, would you need
to consult the records in order to be able to answer this question?
• How comfortable are you in answering this question for the owner(s) of this business?
• Do you have any other thoughts on this question that we have not already discussed?
B.1.5 Military Service Disability
• General probes
B.1.16 Reasons for Owning the Business
• General probes
• How would you go about answering this question for the owner(s) of the business?
• Are there any reasons not listed here that might be an important reason for owning the business?
• Do you have any other thoughts on this question that we have not already discussed?
Wrap up
• Overall, which questions did you feel like were the most time consuming or difficult to complete?
• Do you have any other comments or additional feedback?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 54 of 88
Thank you for your time!
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 55 of 88
Appendix B – Moderated Instrument (Round 1)
Draft of Sustainability Management Module (07/2023)
2024 Annual Business Survey
Sustainability management addresses how the operation of your business is affected by demands from
consumers, other businesses, and the government to meet the needs of present and future generations and
the environment. These questions collect information on what your business is doing in response to or in
anticipation of these demands. All responses are anonymized and will only be used for statistical
purposes.
BUSINESS PRIORITY
Over the past 3 years, from 2020 through 2022, have any of the following been this business’s priority?
Select one for each row.
Yes
No
a.
Improving profitability
b.
Increasing revenue
c.
Accelerating digital transformation activities/improving cybersecurity
d.
Implementing or maturing sustainability initiatives
e.
Improving the customer experience
f.
Improving resilience to competitive pressures
g.
Other (Specify)
________________
If you reported ‘Yes’ for item d. ‘Implementing or maturing sustainability initiatives’, continue to ‘2.
Sustainability Prioritization’.
All others skip to ‘4. Energy or Carbon Emissions Tracking’.
2. SUSTAINABILITY PRIORITIZATION
How important to this business were each of the following factors for prioritizing sustainability?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 56 of 88
Very
Important
Somewhat
Important
Not
Important
e.
Attracting customers who make purchasing decisions
based on sustainable products or sustainable business
values
Integrating sustainable value proposition into the
company’s brand
Addressing stakeholder demand for climate risk
disclosures and better sustainability management
Employee recruitment/retention tied to sustainable
business values
Green procurement requiring life-cycle assessment
f.
Some other (Specify)
a.
b.
c.
d.
_________________________________
3. SUSTAINABILITY OBJECTIVES
Has this business realized any of the following benefits from pursuing its sustainability goals? Select one
for each row.
Yes
No
j.
Improved efficiency
k.
Improved regulatory compliance
l.
Improved employee recruitment and/or retention
m. Reduced waste
n.
Improved brand reputation
o.
Decreased costs
p.
Improved customer acquisition and/or loyalty
q.
Increased revenue
r.
Other (Specify)
____________________
4. ENERGY OR CARBON EMISSIONS TRACKING
Does this business track energy use or carbon emissions from the following sources?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 57 of 88
No, does not
track either
Yes, tracks
Energy
Expenditures
Yes, tracks
Carbon
Emissions
Source is not
applicable to
this business or
is inaccessible
f.
Heating of buildings
g.
Cooling of buildings
h.
Fleet transportation
i.
Electricity (used for
lighting, computing,
HVAC, production, etc.)
Thermal energy in
production (for example,
from natural gas,
petroleum, coal, biomass,
etc.)
j.
5. CARBON EMISSIONS SOFTWARE USE
Does this business use any carbon emissions reporting software?
Yes
No
Don’t Know
6. INDIRECT CARBON EMISSIONS TRACKING
Does this business track indirect carbon emissions that are a consequence of the activities of the business
but from assets not owned or controlled by the business? For example, emissions from employee
commuting, transporation and distribution, use of sold products, etc. Also known as ‘value chain’ or
‘Scope 3.’
No, does not track - Skip to ‘8. Carbon Emissions Reduction Plan’
Not yet, but some evaluation of major carbon hotspots
Yes, major carbon hotspots tracked
7. INDIRECT CARBON EMISSIONS POTENTIAL CHALLENGES
Does this business expect to face any of the following challenges to measuring indirect carbon emission
in its value chain (Scope 3)? Select one for each row.
Yes
No
a.
The company is not aligned internally on what to track and how.
b.
The company lacks data from suppliers or end users.
c.
d.
The company lacks in-house expertise.
The company lacks financial resources.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 58 of 88
8. CARBON EMISSIONS REDUCTION PLAN
Has this business developed plans for reducing carbon emissions?
Yes - Skip to ’10. Carbon Emissions Strategies’
No
9. FUTURE PLANS FOR CARBON EMISSIONS
Is this business considering developing plans in the future for reducing carbon emissions?
Yes
No - Skip to ’13. Sustainability Alternatives’
10. CARBON EMISSIONS STRATEGIES
Will this business implement any of the following strategies to reduce carbon emissions? Select one for
each row.
a.
b.
c.
d.
Eliminating products or activities reliant on fossil fuel use by changing
the company’s product or service mix
Replacing fossil fuel using activities with zero or low emissions
alternatives (for example, adding electric vehicles to the company’s fleet)
Reducing carbon emissions by increasing the efficiency of activities
using fossil fuels
Offsetting carbon emissions that cannot be abated by supporting projects
that reduce carbon emissions elsewhere (e.g., providing funds for
reforestation or capping abandoned oil wells)
Yes
No
11. CARBON EMISSIONS REDUCTION GOAL DATE
What is the target date for achieving this business’s carbon emissions reduction goals?
No target date set
No later than 2030
No later than 2040
No later than 2050
After 2050
12. CARBON EMISSIONS REDUCTION GOAL OBJECTIVE
How much of a reduction carbon emissions is this business planning?
No reduction goal set
Less than 25% reduction
25-50% reduction
More than 50%, but less than 100% reduction
100% reduction
13. SUSTAINABILITY ALTERNATIVES
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 59 of 88
If this business is not successful at improving sustainability, does the business foresee any of the
following effects? Select one for each row.
Yes
No
a.
Fines for lack of compliance with regulations
b.
Inability to raise capital or secure credit
c.
A loss of customer trust
d.
A loss of brand reputation
e.
Hiring difficulties or lower employee retention
f.
Financial risks associated with future policy changes such as a
carbon tax
Lost sales due to inability to provide emissions or other
environmental information to potential customers
g.
14. SUSTAINABILITY INVESTMENTS
Is this business making any of the following investments to improve sustainability? Select one for each
row.
Yes
No
Not
applicable
a.
Purchase power agreements for renewable energy
b.
Renewable energy generation on-site (e.g., solar, wind,
geothermal)
Battery storage or other means of saving renewable
energy generation for later use
Improved energy efficiency and management
Engineering innovation to improve sustainability of our
materials (e.g., innovating to produce goods with lower
CO2 content)
Redesigning processes to make similar products with
less environmental impacts
c.
d.
e.
f.
g.
h.
Elimination of waste through circular economy or
design for re-use strategies
Recycling initiatives going beyond municipal mandates
15. STRATEGIC OBJECTIVES AND THE ENVIRONMENT
Does this business have strategic objectives that mention environmental or climate change issues?
Yes
No
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 60 of 88
Don’t Know
16. ENVIRONMENTAL MANAGER
Does this business have a manager responsible for environmental and climate change issues?
Yes
No
Don’t Know
17. ENVIRONMENTAL CERTIFICATIONS
Do any of this business’s customers require environmental certifications or adherence to certain
environmental standards as a condition to do business with them?
Yes
No
Don’t Know
18. CO2 EMISSIONS MONITORING
Over the last 3 years, from 2020 through 2022, has this business monitored CO2 emissions along its
supply chain?
Yes
No
Don’t Know
19. EXTREME WEATHER EVENTS AND IMPACT
Over the last 3 years, from 2020 through 2022, has this business experienced monetary losses due to
extreme weather events? For example, hurricanes, floods, droughts, or heat waves.
Include:
• Incurred costs
• Losses of sales or assets.
Yes
No
Don’t Know
20. INVESTMENTS TO MITIGATE RISK
Over the last 3 years, from 2020 through 2022, has this business undertaken investments to reduce the
risk of damages caused by extreme weather events? For example, hurricanes, floods, droughts, wildfires,
or heat waves.
Yes
No
Don’t Know
21. FUTURE IMPACTS DUE TO EXTREME WEATHER
Over the next decade, how likely is this business to experience negative impacts from extreme weather
events?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 61 of 88
For example, hurricanes, floods, droughts, wildfires, or heat waves.
Not at all likely
A little likely
Somewhat likely
Very likely
Extremely likely
Technology Transfer
Formal Intellectual Property Protection
Does this business use the following methods to protect intellectual property?
a. Utility patents (patents for inventions, applied for or awarded)
Yes
No
b. Design patents (patents for appearance, applied for or awarded)
c. Copyright
d. Trademarks
Informal Intellectual Property Protection
Does this business use the following less formal means of protecting intellectual property?
a. Design complexity
Yes
No
b. Nondisclosure agreements or non-compete clauses
c. Branding and product image
d. Trade secrets
Patents Pending
How many U.S. patent applications, if any, did this business have pending as of the end of 2022? If
none, report zero.
_______________
Patents Owned
How many U.S. patents did this business own as of the end of 2022? If none, enter zero.
________________
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 62 of 88
Intellectual Property Activities
During the three years 2020 to 2022, did your company exchange intellectual property (IP) rights,
including patents, copyrights, and trademarks that give ownership, exclusion rights or rights to use
technical knowledge using any of the following mechanisms. Select one for each row.
a. Licensed or purchased IP from companies that are not affiliated
Yes
No
b. Licensed or sold intellectual property (IP) to companies that are not affiliated
c. Transferred intellectual property (IP) to companies that are not affiliated through participation in
technical assistance or "know how" agreements
d. Received IP from companies that are not affiliated through participation in technical assistance or
"know how" agreements
e. Participated in cross-licensing agreements with companies that are not affiliated
f.
Acquired or merged with companies that are not affiliated to acquire their IP
g. Transferred IP to a spin-off or spin-out of your company
h. Received IP from a parent company as part of a spin-off or spin-out
i.
Allowed free use of patents or other IP owned by this company (for example, allowing free use of
software patents by the open-source community)
j.
Made use of open-source patents or other freely available IP not owned by this company
Collaboration with Universities
During the three years 2020 to 2022, did this business engage in any of the following collaborative
activities with universities? Select one for each row.
a. Performed collaborative research between company and university researchers
Yes
No
b. Funded university research
c. Funded PhD students and postdoctoral researchers at universities
d. Licensed or purchased IP from a university
e. Signed non-disclosure agreements, material transfer agreements, or other agreements with
universities governing usage of IP
f.
Hired science and engineering graduates
g. Engaged in formal or informal personnel exchanges with universities
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 63 of 88
Collaboration with Government Laboratories
During the three years 2020 to 2022, did this business engage in any of the following collaborative
activities with federal laboratories including Federally Funded Research and Development
Centers? Select one for each row.
Yes
No
a. Conducted collaborative research using agreements such as Cooperative Research and
Development Agreements, Space Act Agreements, Material Transfer Agreements, Confidential
Disclosure Agreements, or other transaction authorities
b. Conducted research at government user facilities
c. Funded research at government laboratory
d. Licensed IP from government laboratory
e. Engaged in formal or informal personnel exchanges with government laboratories
f.
Attended workshops or seminars located at or sponsored by government laboratories
Collaboration with Government Laboratories
During the three years 2020 to 2022, did this business engage collaborative activities with federal
laboratories in the following departments? Select one for each row.
11.
12.
13.
14.
15.
16.
17.
18.
19.
20.
Department of Agriculture
Department of Commerce
Department of Defense
Department of Energy
Department of Health and Human Services
Department of Homeland Security
Department of Transportation
Department of Veteran Affairs
National Aeronautics and Space Administration
Other federal department
Collaboration with Nonprofits
During the three years 2020 to 2022, did this business engage in any of the following collaborative
activities with nonprofit organizations? Select one for each row.
a. Performed collaborative research between company and nonprofit researchers
Yes
No
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 64 of 88
b. Funded research at nonprofit organizations
c. Licensed or purchased IP from a nonprofit organization
d. Signed non-disclosure agreements, material transfer agreements, or other agreements with
nonprofit organizations governing usage of IP
e. Engaged in formal or informal personnel exchanges with nonprofit research organizations
Labor
1. As far as you know, what percentage of non-supervisory employees at your business are covered by
a collective bargaining with a labor union? (Select one.)
o
o
o
o
o
No collective bargaining agreements present at your business
Collective bargaining agreement present, covers less than 25% of non-supervisory employees at
your business
Collective bargaining agreement present, covers between 25% to 49% of non-supervisory
employees at your business
Collective bargaining agreement present, covers between 50% to 74% of non-supervisory
employees at your business
Collective bargaining agreement present, covers 75% or more of non-supervisory employees at
your business
Recent union organizing
2. As far as you know, did any non-supervisory employees at your business attempt to organize a union
over the past three years?
o
o
Yes
No
Recent union certification (if recent union organizing drive)
3. Was the union sought by your non-supervisory employees certified?
o
o
Yes
No
Reshoring
MOVING OPERATIONS TO U.S. IN 2022
1. During 2022, did this business move any of its operations from another country to the
United States?
• Yes
• No (skip to 3)
• Not applicable, this business does not have operations outside of the United States
(skip to end)
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 65 of 88
•
Do not know (skip to 3)
REASONS FOR MOVING OPERATIONS TO U.S. IN 2022
2. For what reasons did this business move any of its operations from another country to the
United States in 2022? Select all that apply.
• Costs (for transportation, labor, materials, etc.)
• Quality concerns
• Supply chain issues
• Global current events
• Government incentives
• Access to American workforce
• More control over operations
• Other (describe) -- _____________________________
• Do not know
MOVING OPERATIONS TO U.S. IN 2023
3. During 2023, does this business plan to move any of its operations from another country
to the United States?
• Yes
• No (skip to end)
• Do not know (skip to end)
REASONS FOR MOVING OPERATIONS TO U.S. IN 2023
4. For what reasons does this business plan to move any of its operations from another
country to the United States in 2023? Select all that apply.
• Costs (for transportation, labor, materials, etc.)
• Quality concerns
• Supply chain issues
• Global current events
• Government incentives
• Access to American workforce
• More control over operations
• Other (describe) -- _____________________________
• Do not know
Research and Development
[Note: Skip Questions 1 and 2 for businesses with W-2 employment between 1 and 9.]
What is Research and Development (R&D)?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 66 of 88
Research and development (R&D) comprise creative and systematic work undertaken in order to increase
the stock of knowledge and to devise new applications of available knowledge.
The term R&D does NOT include expenditures for:
•
•
•
•
•
Costs for routine product testing, quality control, and technical services unless they are an integral
part of an R&D project
Market research
Efficiency surveys or management studies
Literary, artistic, or historical projects, such as films, music, or books and other publications
Prospecting or exploration for natural resources
R&D activity in software INCLUDES:
•
•
Software development or improvement activities that expand scientific or technological knowledge
Construction of new theories and algorithms in the field of computer science
R&D activity in software EXCLUDES:
•
•
•
•
Software development that does not depend on a scientific or technological advance, such as
• supporting or adapting existing systems
• adding functionality to existing application programs, and
• routine debugging of existing systems and software
Creation of new software based on known methods and applications
Conversion or translation of existing software and software languages
Adaptation of a product to a specific client, unless knowledge that significantly improved the base
program was added in that process
Reporting unit
The reporting unit is this business, including all subsidiaries and divisions. Include subsidiary companies
where there is more than 50 percent ownership.
Reporting period
Report data for the calendar year 2022, if possible, or for this business’s fiscal year ending between April
2022 and March 2023.
Estimates are acceptable.
Report all items to the best of your ability.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 67 of 88
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 68 of 88
c
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 69 of 88
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 70 of 88
Entrepreneurship
1. Do you have a good understanding of the decisions that led to the founding of this business?
Yes
No – (End)
2. If yes, was the business originally founded around a new or customized product or service that
was created by one of the founders of the business?
Yes
No (End)
Don’t Know (End)
3. If yes, thinking about this new or customized product or service, why was it originally
developed?
a. One of the founders created it for personal use
b. One of the founders created it for use at a previous job or business
c. One of the founders identified a business opportunity
Demographics
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 71 of 88
OWNER GENDER IDENTITY
What is Owner X’s gender? Select all that apply.
Woman
Man
Transgender
Nonbinary
They use a different term: (specify)
_____________________________________________________
Don’t Know
OWNER SEXUAL ORIENTATION
Which of the following does Owner X consider themselves to be? Select all that apply.
Straight or heterosexual
Gay
Lesbian
Bisexual
They use a different term: (specify) ____________________________________________________
Don’t Know
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 72 of 88
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 73 of 88
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 74 of 88
Appendix C –Unmoderated Instruments
Labor and Reshoring
Hello. Thank you for your time today. We are looking to obtain feedback on some potential new
questions for a Census Bureau survey.
We plan to use your feedback to improve the survey questions and make sure they make sense to
respondents like you.
We are not testing you– we only want to evaluate the questions.
Today, we will have you answer these questions, then you will be asked some questions to get your
feedback.
As far as you know, what percentage of non-supervisory employees at your business are covered by
a collective bargaining with a labor union? (Select one.)
o
o
o
o
o
No collective bargaining agreements present at your business
Collective bargaining agreement present, covers less than 25% of non-supervisory employees at
your business
Collective bargaining agreement present, covers between 25% to 49% of non-supervisory
employees at your business
Collective bargaining agreement present, covers between 50% to 74% of non-supervisory
employees at your business
Collective bargaining agreement present, covers 75% or more of non-supervisory employees at
your business
On a scale of 1 to 5 with 1 being very easy and 5 being very difficult, how easy or difficult was it to
answer the question above?
1 very easy
2 somewhat easy
3 neither easy nor difficult
4 somewhat difficult
5 very difficult
How did you come up with your response to the question?
Would you have to consult any records or person to answer the question? If yes, describe who or what
you would consult?
Yes, describe [textbox]
No
Recent union organizing
As far as you know, did any non-supervisory employees at your business attempt to organize a union
over the past three years?
o
o
Yes
No
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 75 of 88
On a scale of 1 to 5 with 1 being very easy and 5 being very difficult, how easy or difficult was it to
answer the question above?
1 very easy
2 somewhat easy
3 neither easy nor difficult
4 somewhat difficult
5 very difficult
How did you come up with your response to the question?
Would you have to consult any records or person to answer the question? If yes, describe who or what
you would consult?
Yes, describe [textbox]
No
Recent union certification (if recent union organizing drive)
Was the union sought by your non-supervisory employees certified?
o
o
Yes
No
On a scale of 1 to 5 with 1 being very easy and 5 being very difficult, how easy or difficult was it to
answer the question above?
1 very easy
2 somewhat easy
3 neither easy nor difficult
4 somewhat difficult
5 very difficult
How did you come up with your response to the question?
Would you have to consult any records or person to answer the question? If yes, describe who or what
you would consult?
Yes, describe [textbox]
No
MOVING OPERATIONS TO U.S. IN 2022
5. During 2022, did this business move any of its operations from another country to the
United States?
• Yes
• No (skip to 3)
• Not applicable, this business does not have operations outside of the United States
(skip to end)
• Do not know (skip to 3)
What activities would you count as onshoring or reshoring?
If you switched from a foreign supplier to a domestic supplier, would you include that here?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 76 of 88
Yes (can you say more?) [textbox]
No
REASONS FOR MOVING OPERATIONS TO U.S. IN 2022
For what reasons did this business move any of its operations from another country to the
United States in 2022? Select all that apply.
• Costs (for transportation, labor, materials, etc.)
• Quality concerns
• Supply chain issues
• Global current events
• Government incentives
• Access to American workforce
• More control over operations
• Other (describe) -- _____________________________
• Do not know
Do the answer choices make sense?
Yes
No (which do not?) [textbox]
Don’t know
MOVING OPERATIONS TO U.S. IN 2023
During 2023, does this business plan to move any of its operations from another country
to the United States?
• Yes
• No (skip to end)
• Do not know (skip to end)
If your company was planning to move production to the United States, how far in advance do
you think you would be aware of this?
REASONS FOR MOVING OPERATIONS TO U.S. IN 2023
For what reasons does this business plan to move any of its operations from another
country to the United States in 2023? Select all that apply.
• Costs (for transportation, labor, materials, etc.)
• Quality concerns
• Supply chain issues
• Global current events
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 77 of 88
•
•
•
•
•
Government incentives
Access to American workforce
More control over operations
Other (describe) -- _____________________________
Do not know
Are there any answer choices potentially missing from this list?
Yes (which answer choices are missing?) [textbox]
No
Would you consult records or others within your company to answer these questions about
reshoring/onshoring?
Yes, records [textbox]
Yes, others [textbox]
No
Thinking back through all the questions you answered for your overall company, do you feel you were the
best person to answer these questions or would someone else at the company have been a better fit?
Yes, I would have been the best person
No, someone else would have been a better fit [what is this person’s job title?) [textbox]
Do you have any comments or suggestions about either topic or any question asked on this survey?
Finally, please tell us a bit about you and your business. What types of goods or services does this
business provide?
What is your role in this business?
Thank you very much for your time today. We really appreciate your help.
Demographics
Hello. Thank you for your time today. We are looking to obtain feedback on some potential new
questions for a Census Bureau survey.
We plan to use your feedback to improve the survey questions and make sure they make sense to
respondents like you.
We are not testing you– we only want to evaluate the questions.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 78 of 88
Today, we will have you answer these questions, then you will be asked some questions to get your
feedback.
Are you an owner of the company:
An owner here is defined as someone who owns 10 or more percent of the business.
Yes
No
No one owns 10% or more of this company
If the last option is selected:
Note: Although your business does not have someone who would qualify as an owner here, please review
the following questions and provide any feedback.
1. AGE
What is the age of Owner 1 as of December 31, 2022?
If you are an owner, please respond for yourself.
Under 25
25-34
35-44
45-54
55-64
65 and over
2. Education Prior to Owning the Business
Prior to establishing, purchasing, or acquiring this business, what was the highest degree or level of
school Owner 1 completed?
Less than high school/secondary school
High school/secondary school graduate – diploma or GED
Technical, trade, or vocational school
Some college but no degree
Associates degree (for example, AA, AS)
Bachelor’s degree (for example, BA, BS)
Master’s degree (for example, MA, Meng, Med, MSW, MBA)
Doctorate degree (for example, PhD, EdD)
Professional Degree, beyond a Bachelor’s degree (for example, MD, DDS, DVM, LLB, JD)
3. SEX
What is the sex of Owner 1?
Male
Female
4. GENDER IDENTITY
What is Owner 1’s gender?
Select all that apply.
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 79 of 88
Male
Female
Transgender
Nonbinary
They use a different term, specify [textbox]
Don’t know
Is this information something you would have in a record for this business?
Yes
No
Don’t know
Is this information something you would know about an owner?
Yes
No
Don’t know
5. SEXUAL ORIENTATION
Which of the following does Owner 1 consider themselves to be?
Select all that apply.
Straight or heterosexual
Gay
Lesbian
Bisexual
They use a different term, specify [textbox]
Don’t know
Is this information something you would have in a record for this business?
Yes
No
Don’t know
Is this information something you would know about an owner?
Yes
No
Don’t know
What are your thoughts about answering questions 3, 4, and 5 (sex, gender identity, and sexual
orientation)?
6. Disability
Does Owner 1 have a chronic physical or mental condition that they manage in daily life?
Yes
No
Don’t know
What kinds of conditions might fall under this category?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 80 of 88
Are there more owners in this business?
Yes
No
Don’t know
If yes: Would it be easier or more difficult to answer about the owner(s)?
A lot more difficult
A little more difficult
Neither easier nor more difficult
A little easier
A lot easier
If yes: How would your approach be different or the same for other owner(s)?
Overall, which questions did you feel like were the most time consuming or difficult to complete?
Do you have any other comments or suggestions?
Finally, please tell us a bit about you and your business. What types of goods or services does this
business provide?
What is your role in this business?
Thank you very much for your time today. We really appreciate your help.
Collaborative Research
Hello. Thank you for your time today. We are looking to obtain feedback on some potential new
questions for a Census Bureau survey.
We plan to use your feedback to improve the survey questions and make sure they make sense to
respondents like you.
We are not testing you– we only want to evaluate the questions.
Today, we will have you answer these questions, then you will be asked some questions to get your
feedback.
1. Research Collaboration
Did this business do any collaborative research with any of the following – universities, government labs,
or nonprofit organizations?
Yes
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 81 of 88
No
Don’t know
Not applicable
(Participants were skipped to the end of the survey if yes was not selected)
8. Collaboration with Universities
During the three years 2020 to 2022, did this business engage in any of the following collaborative
activities with universities? Select one for each row.
a. Performed collaborative research between company and university researchers
Yes
No
b. Funded university research
c. Funded PhD students and postdoctoral researchers at universities
d. Licensed or purchased IP from a university
e. Signed non-disclosure agreements, material transfer agreements, or other agreements with
universities governing usage of IP
f.
Hired science and engineering graduates
g. Engaged in formal or informal personnel exchanges with universities
9. Collaboration with Government Laboratories
During the three years 2020 to 2022, did this business engage in any of the following collaborative
activities with federal laboratories including Federally Funded Research and Development
Centers? Select one for each row.
Yes
No
a. Conducted collaborative research using agreements such as Cooperative Research and
Development Agreements, Space Act Agreements, Material Transfer Agreements, Confidential
Disclosure Agreements, or other transaction authorities
b. Conducted research at government user facilities
c. Funded research at government laboratory
d. Licensed IP from government laboratory
e. Engaged in formal or informal personnel exchanges with government laboratories
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 82 of 88
f.
Attended workshops or seminars located at or sponsored by government laboratories
10. Collaboration with Government Laboratories
During the three years 2020 to 2022, did this business engage collaborative activities with federal
laboratories in the following departments? Select one for each row.
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
Department of Agriculture
Department of Commerce
Department of Defense
Department of Energy
Department of Health and Human Services
Department of Homeland Security
Department of Transportation
Department of Veteran Affairs
National Aeronautics and Space Administration
Other federal department
11. Collaboration with Nonprofits
During the three years 2020 to 2022, did this business engage in any of the following collaborative
activities with nonprofit organizations? Select one for each row.
a. Performed collaborative research between company and nonprofit researchers
Yes
No
b. Funded research at nonprofit organizations
c. Licensed or purchased IP from a nonprofit organization
d. Signed non-disclosure agreements, material transfer agreements, or other agreements with
nonprofit organizations governing usage of IP
e. Engaged in formal or informal personnel exchanges with nonprofit research organizations
Was it clear what questions, if any, applied to your business?
Yes
No
Other [textbox]
Do you have any feedback about the questions?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 83 of 88
Appendix D – Changes to the Round 2 Moderated Protocol
Research and Development
Asked of everyone first in round 2
What is Research and Development (R&D)?
Research and development (R&D) comprise creative and systematic work undertaken in order to increase
the stock of knowledge and to devise new applications of available knowledge.
The term R&D does NOT include expenditures for:
•
•
•
•
•
Costs for routine product testing, quality control, and technical services unless they are an integral
part of an R&D project
Market research
Efficiency surveys or management studies
Literary, artistic, or historical projects, such as films, music, or books and other publications
Prospecting or exploration for natural resources
R&D activity in software INCLUDES:
•
•
Software development or improvement activities that expand scientific or technological knowledge
Construction of new theories and algorithms in the field of computer science
R&D activity in software EXCLUDES:
•
•
•
•
Software development that does not depend on a scientific or technological advance, such as
• supporting or adapting existing systems
• adding functionality to existing application programs, and
• routine debugging of existing systems and software
Creation of new software based on known methods and applications
Conversion or translation of existing software and software languages
Adaptation of a product to a specific client, unless knowledge that significantly improved the base
program was added in that process
Reporting unit
The reporting unit is this business, including all subsidiaries and divisions. Include subsidiary companies
where there is more than 50 percent ownership.
Reporting period
Report data for the calendar year 2022, if possible, or for this business’s fiscal year ending between April
2022 and March 2023.
Estimates are acceptable.
Report all items to the best of your ability.
Does your company do R&D?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 84 of 88
1. Are the includes helpful or unhelpful?
a. Do all of the items make sense, are there any you are unfamiliar with?
b. Are there other R&D activities that we are missing from this list?
If answers to the definition and question one do not match, please explain why.
If yes is not selected on question 1, skip to end of block
Sustainability Management
2. SUSTAINABILITY PRIORITIZATION
How important to this business were each of the following factors for prioritizing environmental
sustainability?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 85 of 88
Very
Important
Somewhat
Important
Not
Important
Attracting customers who make purchasing decisions based on
sustainable products or sustainable business values
Integrating sustainable value proposition into the company’s brand
Addressing stakeholder demand for climate risk disclosures and
better sustainability management
Employee recruitment/retention tied to sustainable business values
Green procurement requiring life-cycle assessment
Some other (Specify)
_________________________________
3. SUSTAINABILITY OBJECTIVES
Has this business realized any of the following benefits from pursuing its environmental sustainability
goals? Select one for each row.
Yes
No
s.
Improved efficiency
t.
Improved regulatory compliance
u.
Improved employee recruitment and/or retention
v.
Reduced waste
w. Improved brand reputation
x.
Decreased costs
y.
Improved customer acquisition and/or loyalty
z.
Increased revenue
aa. Other (Specify)
____________________
13. SUSTAINABILITY ALTERNATIVES
If this business is not successful at improving environmental sustainability, does the business foresee any
of the following effects? Select one for each row.
Yes
No
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 86 of 88
Fines for lack of compliance with regulations
Inability to raise capital or secure credit
A loss of customer trust
A loss of brand reputation
Hiring difficulties or lower employee retention
Financial risks associated with future policy changes such as a
carbon tax
Lost sales due to inability to provide emissions or other
environmental information to potential customers
14. SUSTAINABILITY INVESTMENTS
Is this business making any of the following investments to improve environmental sustainability? Select
one for each row.
Yes
No
Not
applicable
Purchase power agreements for renewable energy
Renewable energy generation on-site (e.g., solar, wind,
geothermal)
Battery storage or other means of saving renewable
energy generation for later use
Improved energy efficiency and management
Engineering innovation to improve sustainability of our
materials (e.g., innovating to produce goods with lower
CO2 content)
Redesigning processes to make similar products with
less environmental impacts
Elimination of waste through circular economy or
design for re-use strategies
Recycling initiatives going beyond municipal mandates
15. STRATEGIC OBJECTIVES AND THE ENVIRONMENT
Does this business have strategic objectives that mention environmental sustainability or climate change
issues?
Yes
No
Don’t Know
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 87 of 88
16. ENVIRONMENTAL MANAGER
Does this business have a manager responsible for environmental sustainability and climate change
issues?
Yes
No
Don’t Know
Entrepreneurship
3. What was the motivation for developing that product?
Demographics
Who is the respondent?
An owner here is defined as someone who owns 10 or more percent of the business.
Owner
Owner/proxy respondents
Proxy respondent
There are no owners
7. DISABILITY
Does Owner 1 have a chronic physical or mental condition that they manage in daily life?
Findings and Recommendations from Cognitive Interviews on Current and New Content for ABS – Multi-Units and Single-Units
10.16.23
Page 88 of 88
| File Type | application/pdf |
| Author | Hillary Steinberg (CENSUS/ESMD FED) |
| File Modified | 2023-11-14 |
| File Created | 2023-11-14 |