2024 ABS Content Testing Protocol Draft – 07/2023
General Research Questions:
Do respondents understand what information they were asked to provide with this question?
Do respondents understand the terminology used in the question?
Are respondents able to answer the question that is being asked?
Is the question layout clear and understandable?
Is the requested information available in respondent’s records?
Are the lists/categories organized effectively?
Are the instructions helpful to respondents?
Informed Consent: Respondents will be asked to complete a consent form electronically before the time of the interview.
Materials Needed:
Electronically signed consent form
Copy of questionnaire
Introduction
Introduce everyone on the call
Thank respondent for completing the survey
Brief overview of why survey exists and role of NSF and Census
The Annual Business Survey (ABS) is conducted jointly by the U.S. Census Bureau and the National Center for Science and Engineering Statistics within the National Science Foundation. ABS provides information on selected economic and demographic characteristics for businesses and business owners. Additionally, the survey measures research and development, innovation and technology, as well as other business characteristics
Tell respondent why we asked to speak with them
Suggestions for improvement
Refine questions and make them easier to answer
Recording the interview
As mentioned in the consent form that you signed, we would like to record this interview so that we have an accurate record of your feedback. We plan to use your feedback to improve the design and layout of the form for future data collections. Only staff involved in this data collection will have access to the recording. Would that be ok? If respondent says no, move on.
About the Respondent
Can you tell us a little bit about your business -- what types of goods or services does this business provide?
What is your role in the company? What kind of responsibilities do you have?
General probes that may be used for any question or module throughout the interview:
In your own words, what is this question asking?
Reflect back on respondent’s answer: “you said…”
How did you arrive at this number/answer this question?
What records (if any) would you look at?
Specifically, what would you include in this answer? What would you exclude?
Would you consult other people to obtain this answer?
How easy or difficult is it to answer this question?
Additional Probes Specific to Modules and Questions
Sustainability Management Module
Throughout this module, remain neutral on climate change, but note if the R has strong opinions or objections to these questions.
Intro:
Please take a minute to read the introduction above the first question.
What sort of questions are you expecting to see in this module?
What does sustainability mean to you?
In relation to your company? Can you give me some examples?
Questions:
How would you define ‘business priority’ in the context of this question?
Do all of the items make sense, are there any you are unfamiliar with?
Would you put anything under other?
When thinking about how you would answer this question, were you thinking about the time frame from 2020 to 2022?
Note how respondents react to item C, as it is similar to item A.
How would you determine importance?
Do you have any thoughts about the answer categories/column headers?
Do all of the items make sense, are there any you are unfamiliar with?
Would you put anything under other?
What does ‘realized’ mean to you in the context of this question?
If ‘realized’ is confusing: is there another term or phrase that would make more sense here?
Do you think anything is missing with the response categories?
Do all of the items make sense, are any there any you are unfamiliar with?
Would you put anything under other?
Note how respondents react to column headers and if there is any confusion. Note that item E is mainly relevant to manufacturing.
If R answers yes to any items: Can you give me examples of what you track and how you track it?
Do you have any outside or consulting companies track these for you?
Do you have any thoughts about the answer categories/column headers?
Do all of the items make sense, are there any you are unfamiliar with?
If R answers yes: Can you give me examples or the name of the software you use?
If R answers yes: Can you give some examples of what your company tracks?
If R answers limited tracking and estimates: Can you tell me more about that?
Can you give me examples of the types of tracking or evaluation your company is doing?
Do you find the text in parentheses and what is in italics helpful or unhelpful?
How do you define ‘value chains’?
What does this term mean to you in the context of this question?
Do all of the items make sense, are there any you are unfamiliar with?
Are there any other challenges that are missing from our list?
If R answers yes: can you tell me more about that? Can you give me any examples?
How do you define “develop a plan”?
If R answers yes: can you tell me more about that? Can you give me any examples?
Do all of the items make sense, are there any you are unfamiliar with?
Particularly, can you tell me what item D is asking in your own words?
Are there any other strategies that are missing from our list?
When answering this question, what time frame were you thinking about?
Note if R has difficultly in choosing an answer category. Note if R asks if the target date pertains to only Scope 1 & 2 or to Scope 1, 2 & 3.
Do these answer categories make sense to you, or are they confusing?
Note if R mentions negative carbon emissions (highly unlikely). Note if R asks if the target goal pertains to only Scope 1 & 2 or to Scope 1, 2 & 3.
Do these answer categories make sense to you, or are they confusing?
Do all of the items make sense, are there any you are unfamiliar with?
Are there any other effects that are missing from our list?
If R answers yes to any items, ask for examples.
Do all of the items make sense, are there any you are unfamiliar with?
In your own words, what is the difference between items E and F?
We’re particularly interested in Item G, can you tell me what Item G means to you in your own words? Note if R includes recycling.
Are there any other activities that are missing from our list?
Starting from question 15, the questions may be redundant, note if R mentions it.
What do ‘strategic objectives’ mean to you in context of this question?
If R answers yes: Can you give me any examples?
General probes
If R answers yes to any items, ask for examples.
If R answers yes: can you give me any examples?
When answering this question, what time period were you thinking of?
Were the includes helpful or unhelpful?
If R answers yes: can you give me any examples?
When answering this question, what time period were you thinking of?
If R answers yes: can you give me any examples?
When answering this question, what time period were you thinking of?
After general probes, and if it was not obvious: When answering this question, what time period were you thinking of?
Having gone through this entire section, about how long do you think it would take you to complete?
Would you be able fill out this section, or would you need help from someone else in your company fill it out?
Technology Transfer Module:
Are the includes helpful or unhelpful?
Do all of the items make sense, are there any you are unfamiliar with?
Are there other R&D activities that we are missing from this list?
General probes
How easy or difficult would it be to provide this dollar amount?
Does your company have any intellectual property? Can you give me any examples?
Do all of the items make sense, are there any you are unfamiliar with?
Are there other formal IP protections that we are missing from this list?
Do all of the items make sense, are there any you are unfamiliar with?
Are there other informal IP protections that we are missing from this list?
General probes
General probes
Do all of the items make sense, are there any you are unfamiliar with?
What does a ‘companies that are not affiliated’ mean to you?
Are there other mechanisms of transfer that we are missing from this list?
Do all of the items make sense, are there any you are unfamiliar with?
Are there other collaborative activities or agreements that we are missing from this list?
Do all of the items make sense, are there any you are unfamiliar with?
Are there other collaborative activities or agreements that we are missing from this list?
Is it always clear if the organizations you work with are federal government laboratories?
Note that this question has the same title, and this may confuse R (this will be changed).
If you work with federal government laboratories, is it obvious which parent agency the lab operates under? How would you go about answering this question if it was not obvious?
If R answers yes to any item, ask for examples. Note if R includes charitiable donations in any item.
Do all of the items make sense, are there any you are unfamiliar with?
Are there other collaborative activities or agreements that we are missing from this list?
Is it always clear if the organizations you work nonprofits?
Having gone through this entire section, about how long do you think it would take you to complete?
Would you fill out this section, or would someone else in your company fill it out?
Labor Related Questions
These questions will also be tested online via Qualtrics.
1: General Probes
How would you determine your answer to this question?
Are supervisory and non-supervisory workers distinct categories to you?
2: General Probes
3: General Probes
What does ‘certified’ mean to you in the context of this question?
Reshoring Questions
These questions will also be tested online via Qualtrics.
1: General Probes
In your own words, what would you say this question is asking?
How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?
How helpful or unhelpful was the clarification after the question – very helpful, somewhat helpful, neither helpful nor unhelpful, somewhat unhelpful or very unhelpful?
Very helpful
Somewhat helpful
Neither helpful nor unhelpful
Somewhat unhelpful (Do you have any suggestions for improving this clarification?)
Very unhelpful (Do you have any suggestions for improving this clarification?)
Are there other terms you would use to describe this activity (other than ‘onshoring’ or ‘reshoring’)?
If you switched from a foreign supplier to a domestic supplier, would you include that here?
How did you determine which parts of your business to answer for? Was there anything tricky about answering for “this business”?
What time period were you thinking about? Was there anything tricky about answering for “During 2022”?
How easy or difficult was it to answer this question – very easy, somewhat easy, neither easy nor difficult, somewhat difficult or very difficult?
Very easy
Somewhat easy
Neither easy nor difficult
Somewhat difficult (Can you tell me more about that?)
Very difficult (Can you tell me more about that?)
Do you have any suggestions on how to improve this question?
2: General Probes
In your own words, what would you say this question is asking?
How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?
What do each of the answer choices mean to you?
If R selects an answer choice: You selected “xx” -- can you give me any examples of what you were thinking about?
Do the answer choices make sense to you?
Yes
No (Which did not?)
Don’t know
Are there any answer choices potentially missing from this list?
How easy or difficult was it to answer this question – very easy, somewhat easy, neither easy nor difficult, somewhat difficult or very difficult?
Very easy
Somewhat easy
Neither easy nor difficult
Somewhat difficult (Can you tell me more about that?)
Very difficult (Can you tell me more about that?)
Do you have any suggestions on how to improve this question?
3: General Probes
In your own words, what would you say this question is asking?
How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?
What time period were you answering for? Was there anything tricky about answering for “During 2023”? [Probe about how confident they are about the remaining part of 2023.] Would your answer have been the same or different if they question asked “In the next 12 months…”? Do you have a preference for “During 2023” v. “In the next 12 months”?
How confident were you in answering this question for 2023?
Very confident
Somewhat confident
Not very confident
Not at all confident
If your company was planning to move production to the United States, how far in advance do you think you would be aware of this?
How easy or difficult was it to answer this question – very easy, somewhat easy, neither easy nor difficult, somewhat difficult or very difficult?
Very easy
Somewhat easy
Neither easy nor difficult
Somewhat difficult (Can you tell me more about that?)
Very difficult (Can you tell me more about that?)
Do you have any suggestions on how to improve this question?
4: General Probes
In your own words, what would you say this question is asking?
How did you come up with your answer to this question? Would you need refer to records or reach out to anyone else in your business to answer this question? If yes, Who/what?
If R selects an answer choice: You selected “xx” -- can you give me any examples of what you were thinking about?
How easy or difficult was it to answer this question – very easy, somewhat easy, neither easy nor difficult, somewhat difficult or very difficult?
Very easy
Somewhat easy
Neither easy nor difficult
Somewhat difficult (Can you tell me more about that?)
Very difficult (Can you tell me more about that?)
Do you have any suggestions on how to improve this question?
Now that we have looked at these draft questions, what is your overall impression of these questions?
How comfortable were you in answering these questions for your company – very comfortable, somewhat comfortable, not very comfortable or not at all comfortable?
Very comfortable
Somewhat comfortable
Not very comfortable (Can you say more about this?)
Not at all comfortable (Can you say more about this?)
How confident are you in your answers to these questions – very confident, somewhat confident, not very confident or not at all confident?
Very confident
Somewhat confident
Not very confident (Can you say more about this?)
Not at all confident (Can you say more about this?)
If you had been answering these questions without being on the phone with me, would you likely have consulted records or others within your company?
Yes, records (What records would you consult?)
Yes, others within company (Who would you consult with?)
No
About how long do you think it would have taken you to complete this survey (if we hadn’t been on the phone)?
_____ minutes
Thinking back through all the questions you answered for your overall company, do you feel you were the best person to answer these questions or would someone else at the company have been a better fit?
Yes, I was the best person
No, someone else would have been a better fit (What is that person’s job title?)
Overall, how easy or difficult was it to answer these questions– very easy, somewhat easy, neither easy nor difficult, somewhat difficult or very difficult?
Very easy
Somewhat easy
Neither easy nor difficult
Somewhat difficult (Can you say more about what was difficult?)
Very difficult (Can you say more about what was difficult?)
Research and Development Questions
D1. General Probes
D5. General Probes
Are you able to break down the R&D costs for each of the different types of costs this question asks about?
D7. General Probes
Are you able to break down the funding sources for R&D activities?
In your own words, how would you define venture capital?
Has your business used venture capital as a funding source for R&D activities? If yes, could you please tell us more about that?
In your own words, what is angel financing?
Has your business used angel financing as a funding source for R&D activities? If yes, could you please tell us more about that?
D8. General Probes
Entrepreneurship Questions
General Probes
General Probes
What do you think this question is asking?
In your own words, what do you think is meant by “originally founded” in the context of this question?
If the word “originally” was removed from the wording of this question, would it change your interpretation or understanding of the question?
How do you interpret “new or customized product or service” in the context of this question?
General Probes
Owner Characteristics: Sexual Orientation and Gender Identity Questions
These questions will also be tested online via Qualtrics.
Throughout this module, remain neutral, but note if the R has strong opinions or objections to any these questions.
Note whether respondent is an owner of this business, an owner/proxy respondent, or a proxy respondent.
B.1.9. Prior Business Ownership:
General Probes
In your own words, what is this question asking? (Interviewer note: Does the respondent notice the instruction for ‘Not including this business’ when responding to this question?)
B.1.12. Age
General Probes
B.1.1 Sex
General Probes
In your own words, what is this question asking?
How do you define Sex in the context of this question?
Do you think there are any answer categories missing from this question? If yes, please describe.
Does your company have records on the Sex of the owner(s)? If yes, what are the categories in your records or systems for Sex of the owner(s)? And would you need to consult the records in order to be able to answer this question?
How comfortable are you in answering this question for the owner(s) of this business?
Do you have any other thoughts on this question that we have not already discussed?
Gender Identity
General Probes
In your own words, what is this question asking?
How would you go about answering this question if it appeared on a future Annual Business Survey?
How do you define Gender in the context of this question?
In your own words, what is the difference between Sex and Gender Identity?
Are you familiar with the answer categories in this question? If not, would definitions be helpful?
Do you think there are any answer categories missing from this question? If yes, please describe.
What, if any, are your thoughts on the order in which the categories are arranged in this question?
Does your company have records on the Gender of the owner(s)? If yes, what are the categories in your records or systems for Gender of the owner(s)? And would you need to consult the records in order to be able to answer this question?
How comfortable are you in answering this question for the owner(s) of this business?
Do you have any thoughts on a two question approach to asking about Sex and Gender Identity?
Do you have any other thoughts on this question that we have not already discussed?
Sexual Orientation
General Probes
In your own words, what is this question asking?
How would you go about answering this question if it appeared on a future Annual Business Survey?
How do you define Sexual Orientation?
Are you familiar with the answer categories in this question? If not, would definitions be helpful?
Do you think there are any answer categories missing from this question? If yes, please describe.
What, if any, are your thoughts on the order in which the categories are arranged in this question?
Does your company have records on the Sexual Orientation of the owner(s)? If yes, what are the categories in your records or systems for the Sexual Orientation of the owner(s)? And would you need to consult the records in order to be able to answer this question?
How comfortable are you in answering this question for the owner(s) of this business?
Do you have any other thoughts on this question that we have not already discussed?
B.1.14 U.S. Citizenship
General probes
B.1.15 Owner Disability
General probes
In your own words, what is this question asking?
What do you consider to be a physical impairment? Would it be helpful to list examples of physical impairments with this question?
How would you define a mental impairment? Would it be helpful to list examples of mental impairments with this question?
In your own words, how do you interpret “substantially limits” in the context of this question?
What does “major life activities” mean to you in the context of this question?
Does your company have records on the Disability status of the owner(s)? If yes, would you need to consult the records in order to be able to answer this question?
How comfortable are you in answering this question for the owner(s) of this business?
Do you have any other thoughts on this question that we have not already discussed?
B.1.5 Military Service Disability
General probes
B.1.16 Reasons for Owning the Business
General probes
How would you go about answering this question for the owner(s) of the business?
Are there any reasons not listed here that might be an important reason for owning the business?
Do you have any other thoughts on this question that we have not already discussed?
Wrap up
Overall, which questions did you feel like were the most time consuming or difficult to complete?
Do you have any other comments or additional feedback?
Thank you for your time!
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Jessica Nickel (CENSUS/ESMD FED) |
| File Modified | 0000-00-00 |
| File Created | 2025-08-12 |