Protocol

Cognitive Interviewing Guide: Draft Content for 2021 ABS (4th Year)
Interviewer notes are in red.
Introduction:
• Explain purpose of meeting: to obtain feedback on reporting to the ABS, to understand the
process of answering questions, and to evaluate how new questions work or don’t work.
• Some questions will ask may seem odd and/or obvious, but we don’t want to assume we know
what you are thinking.
• Explain that we are not testing the respondent – we only want to evaluate the questionnaire
• Structure of meeting: Understand your business and your role in it, then review the
questionnaire and ask you questions as you complete it.
• Ask respondent to sign the consent form.
Research Questions to Address:
• Are respondents able to answer the questions as intended?
• Are there any items that cause the respondents problems?
o What can be done to fix those problems?
• Are there any order/context effects?
Before the Questionnaire:
• What types of goods or services does this business provide?
• What is your role in the company? What kind of responsibilities do you have?
• Are you one of the owners of the business?
• Have you completed other surveys from the Census Bureau or other agencies? If so, which
ones?
• What was your role in the process for responding to this questionnaire? (Gather data? Enter
data? Consult with data providers? Etc.)
• How would you describe your company’s approach to design?
Thanks for providing some background information. Now let’s look at the survey questions.
Direct participant to look at draft content.
DESIGN MODULE - ABS (4th Year)
Question 1
• How would you answer this question?
o IF YES: Can you give me examples of the design activities your company conducts?
• How would you describe ‘design activities’ in your own words?
o Is there another term or phrase that you feel would work better?
• Can you think of any other design activities that we do not have listed?
• Would you need to reach out to other people or departments to answer questions about this
companies design activities?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

1

Question 1a + 1b
• How would you answer this question?
o IF NO: Does anyone in the company conduct design activities?
• What do you consider to be resources?
• Is this something you would have in your records?
Question 2
• How would you answer this question?
o IF YES: Can you give me examples?
• How would you describe ‘design-related engineering activities’ in your own words?
o Is there another term or phrase that you feel would work better?
• Can you think of any other activities that we do not have listed?
• Would you need to reach out to other people or departments to answer questions about this
companies design activities?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 2a + 2b
• How would you answer this question?
o IF NO: Does anyone in the company conduct design activities?
• What do you consider to be resources?
• Is this something you would have in your records?
Question 3
• How would you answer this question?
• How would you define employees?
• How would you define contractors?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 4
• How would you answer this question?
• Can you think of any other types of employees that might perform design activities that we do
not have listed here?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

2

Question 5
• How would you answer this question?
• How would you define the term ‘routine?’
• How easy or difficult is it to come up with a number for 2016? If R is having difficulty
remembering, ask if the number of individuals increased or decreased between 2016 and 2019.
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 6
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 7
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 8
• How would you answer this question?
• What types of resources were you considering when answering this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 9
• How would you answer this question?
• IF YES: Can you give me any examples?
• When answering this question were you considering all of your company’s IP, or just what
pertained to design?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

3

Question 10
• How would you answer this question?
• IF YES: Can you give me any examples?
• When answering this question were you considering all of your company’s IP, or just what
pertained to design?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
• Having gone through this set of design questions, do you feel that you are still the most
appropriate person to answer these questions?
o If not, who do you feel would be responsible for this section?
• How long do you think it would take you to complete these questions?

FINANCING MODULE - ABS (4th Year)
Question 1
• How would you answer this question?
• Are there any categories we are missing?
• Did the current owner(s) start this business?
• How long ago was this business started?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 2
• How would you answer this question?
• Are you able to fit your answer into the provided categories?
• If this question asked for an estimate of the dollar amount of capital needed to start this
business would you be able to provide that figure?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 3
• How would you answer this question?
• Are you able to fit your answer into the provided categories?
• If this question asked for the dollar amount would you be able to provide that figure?
• Are there other sources of money you would include in your answer?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

4

Question 4
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 5
• How would you answer this question? Note how R answered previous question.
• Are the categories clear or unclear?
• Can you think of anything else that impacted the profitability of this company that is not listed
here?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 6
• How would you answer this question?
• What types of impacts were you considering when answering this question?
• In your opinion, what is the difference between ‘significant’ and ‘somewhat?’
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 7
• How would you answer this question?
• Is the list of includes helpful or unhelpful?
• Can you think of any other loans or grants you would include when answering? (Note if R
includes other kinds of government programs that may not be specifically COVID-19 related)
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 8
• How would you answer this question?
• How easy or difficult is it to provide this figure?
• Would you prefer to provide an open-ended dollar amount field or a range for this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 9
• How would you answer this question?
• How easy or difficult is it to provide this figure?
• If R has not received any money: Why?

5

•
•

Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
Is this something you would have in your records?

Question 10
• How would you answer this question?
• How easy or difficult is it to answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 11
• How would you answer this question?
• How easy or difficult is it to answer this question?
• What factors were you considering when answering this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 12
• How would you answer this question?
• How easy or difficult is it to answer this question?
• What factors were you considering when answering this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 13
• How would you answer this question?
• If this question asked for the dollar amount would you be able to provide that figure?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 14
• How would you answer this question?
• What does “work from home” mean to you?
• Could you tell me a little bit about your company’s work from home programs or policies?

Question 15
• How would you answer this question?
• How did you come up with your answer?
• Would you need to reach out to anyone else in your company to answer this question? If YES,
what is the role/position of the person(s) you’d need to reach out to?
• Is this something that you would have in your records?

6

Question 16
• How would you answer this question?
• In your own words, what is this question asking?
• What factors were you considering when answering this question?
• Were you thinking only of negative effects? Or were you also thinking of any positive effects?
• Is this something that you have in your records?
• If R selected Other, what other factors were you thinking about?
Question 17
• How would you answer this question?
• How would you define ‘new credit?’
• Are the examples helpful or unhelpful?
• Up until now, we have been asking you about this company’s activities in 2019, but this
question asks about the ‘last 12 months.’ Did you take that into account when answering this
question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

Question 18
• How would you answer this question?
• Are the categories clear or unclear?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 19
• How would you answer this question?
• If this question asked for the dollar amount would you be able to provide that figure?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 20
• How would you answer this question?
o IF OTHER: What would put in other?
• Are the categories clear or unclear?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

7

TECHNOLOGY MODULE - ABS (4th Year)
BEFORE STARTING THIS MODULE:
• What does digital mean to you?
• What does digital mean to you in terms of record keeping?
• Does your company keep any records digitally?
o Can you give me examples?
• How would you define cloud services?
• Does your company use any cloud services?
o IF YES: What types of cloud services do you use?
• Does your company purchase digital or technology-related services from other companies?
• Does your company use artificial intelligence in any of its business processes?
o IF YES: How would you define artificial intelligence?
Question 1
• How would you answer this question?
o IF R ANSWERS DK: Do you know if any records are kept digitally?
• Can you think of any other types of records that your company maintains digitally?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 2
• How would you answer this question?
o IF NOT OBVIOUS: When answering this question were you thinking about only cloud
services?
• Are these categories clear or unclear?
• Can you think of any other types of records that your company maintains digitally?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 3
• How would you answer this question?
• Are these categories clear or unclear?
• FOR ANY DK: Did you answer ‘don’t know’ because you don’t know if the company is using this
technology or don’t know what this technology is?
• If we asked you to estimate a percentage of how incorporated that technology is into your
company’s production or service, would you be able to?
o How would you estimate that?
o What are you taking into consideration? Example: outcome, effort, resources, process,
etc…
• Can you think of any other types of technologies that your company uses that we don’t have
listed here?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?

8

•

Is this something you would have in your records?

Question 4
• How would you answer this question?
o IF YES: Can you give me any examples?
• Is the example helpful or not helpful?
• 4.A: How would you answer this question?
o Note if R seems apprehensive about providing the vendor name.
o IF DK: Is this information you are comfortable with providing?
 IF YES: Is this something you would have in your records?
 IF NO: Why?
• 4.B: How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 5
• How would you answer this question?
o IF YES: Can you give me any examples?
• Is the example helpful or not helpful?
• 5.A: How would you answer this question?
o Note if R seems apprehensive about providing the vendor name.
o IF DK: Is this information you are comfortable with providing?
 IF YES: Is this something you would have in your records?
 IF NO: Why?
• 5.B: How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 6
• How would you answer this question?
o IF YES: Can you give me any examples?
• Is the example helpful or not helpful?
• 6.A: How would you answer this question?
o Note if R seems apprehensive about providing the vendor name.
o IF DK: Is this information you are comfortable with providing?
 IF YES: Is this something you would have in your records?
 IF NO: Why?
• 6.B: How would you answer this question?
• IF R PURCHASED: Is there an advantage to purchasing these types of financial services?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

9

•
•

Having gone through this set of technology questions, do you feel that you are still the most
appropriate person to answer these questions?
o If not, who do you feel would be responsible for this section?
How long do you think it would take you to complete these questions?

MANAGEMENT PRACTICES MODULE - ABS (4th Year)
Skip management practices section if running low on time.
Question 1
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 2
• How would you answer this question? Note if R has issues/concerns with “key performance
indicators”.
o IF YES: Can you give me any examples?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
• Are you familiar with the term “key performance indicator?” Is this a term that your business
uses?
Question 3
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 4
• How would you answer this question?
o IF YES: Can you give me any examples of metrics and/or websites that you monitor?
 Is this monitoring a formal process?
o IF NO: Does your company monitor customer satisfaction in any informal way?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 5
• How would you answer this question?
o IF YES: Can you give me examples of business targets?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

10

Question 6
• How would you answer this question?
• How would you define ‘effort?’
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 7
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 8
• How would you answer this question?
• How would your company define ‘underperforming?’
• How would you define ‘reassigned?’
• Does your company have procedures in place for dealing with an underperformer?
o IF YES: Can you give me examples of the types of actions you might take?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 9
• How would you answer this question?
• How does your company define ‘formal training programs?’
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 10
• How would you answer this question?
• Do you think it would be easier or more difficult to answer this question with a count as
opposed to a percentage?
• When answering this questions were you thinking only of W-2 employees, or other types of
employees as well?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
Question 11
• How would you answer this question?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?

11

Question 12
• How would you answer this question?
• Can you think of other types of training that we do not have listed?
• Would you need to reach out to other people or departments to answer this question?
o IF YES: What departments/positions would you reach out to?
• Is this something you would have in your records?
•
•

Having gone through this set of management questions, do you feel that you are still the most
appropriate person to answer these questions?
o If not, who do you feel would be responsible for this section?
How long do you think it would take you to complete these questions?

INNOVATION QUESTIONS - ABS (4th Year)
Skip innovation section if running low on time.
All Questions
• How would you answer this question?
• Is this something you would have in your records?
WRAP UP
• Do you have any other comments of suggestions about anything we have discussed today?
THANK YOU FOR YOUR TIME!

12