Protocol for C and D Versions

Appendix F – Protocol for ASM Robotics Draft Questions, Versions C and D

ASM Robotics Question

Early Stage Scoping and Cognitive Interviews

Kenny Herrell and Rebecca Keegan

Protocol for Versions C and D

Research Questions:

• Do respondents understand what information they are being asked to provide with these questions?

• Do respondents understand the terminology used in the questions?

• Are respondents able to answer the questions that are being asked?

• Is the requested information available in respondent’s records?

• Are the instructions helpful to respondents?

Expected Length of Interview: 30-60 minutes

Materials Needed:

• Version C or D of the Draft Questions.

• Consent forms.

• Digital recorder.

General probes that may be used throughout the interview:

• Reflect back on respondent’s answer: “you said…”

• In your own words, what is this question asking?

• How did you arrive at this number/answer this question?

• What records (if any) did you look at? What line or lines were of interest?

• Specifically, what did you include in this number? What did you exclude?

• Would you consult other people to obtain this answer?

INTRODUCTION

• Explain purpose of meeting: get feedback on potential new questions for the Annual Survey of Manufactures on a new topic.

• There may be times throughout this interview that this seems like a test of you, I assure you that it is not. The purpose of these interviews is to ensure that we are communicating the data requests clearly with you, so in that way, these interviews are actually a test of us. On that note, we want your honest feedback throughout, and if it’s not clear to you how to answer, please do your best. There will be ample opportunities for you to let us know how to make improvements so that you could do a better job answering.

• As we go through the form, we will ask both general and specific questions. If at any time a question seems odd to you, please let us know. We encourage all feedback.

• Before we start, I have a consent form that goes over the authority that we have to conduct these interviews. There’s also a piece in here where we ask for permission to record this interview, which is strictly for our note taking purposes. These interviews will only be heard by people directly involved in the development of the survey. Do we have permission to record our conversation for research purposes?

[Have respondent sign consent form. If they decline recording, note on the consent form that they declined recording and have them sign.]

ABOUT THE RESPONDENT

First, I’d like to learn a bit about you, and your role here at the company.

• What is your job title?

• What is your role in completing government surveys?

• How much experience do you have completing the Annual Survey of Manufactures?

• Do you work with anyone else in your company to get the data for the ASM?

  • How are those other people involved?

  • How many people are involved?

Now I’d like to learn a little about your company.

• Can you give us a brief description of what your company does?

• What kind of products do you manufacture?

• How many domestic manufacturing locations do you have?

• Does your company have any international manufacturing locations?

• What sort of manufacturing equipment do you have?

SCOPING QUESTIONS

Thank you very much for that background information! Now, I’d like to transition to the main topic of today’s interview.

Terminology

Within the context of the manufacturing that your company does, I’d like to introduce you to some terms. For each of these terms, please define them in your own words. It may also be helpful for you to provide us with examples, if any come to mind.

Interviewer:

• Allow respondent to define each term individually before moving on to the next.

• Randomize the order of these terms between each interview.

1. Robots

2. Robotic equipment

3. Industrial robotic equipment

• In your mind, are there major differences between these terms? Can you explain them to us?

• Are there any other terms that you may use that you consider synonyms of robots or robotic equipment?

Record Keeping

• In your financial records, are you able to identify:

  • costs and values of robots/robotic equipment from the costs and values of other manufacturing equipment?

  • counts of robots/robotics equipment?

  • purchase price of robots/robotics equipment?

  • the year of purchase?

• Does your company keep these records on robots/robotics at the establishment level?

• If you are not able to access these records, does someone else have access to them, or are they simply not kept?

• Would someone, such as a plant manager, have access to other information or records that could be used to answer questions about equipment?

  • What types of records? Financial, maintenance, etc.?

In Practice

• What products/goods does your company manufacture with robots/robotic equipment?

• In how many manufacturing locations does your company have robots/robotic equipment?

• Are robots expensed differently than other assets like software or other related expenses?

• Are they acquired or leased?

• What other costs are associated with robots? (e.g., training, installation, maintenance, programming, safety precautions, etc.)

• Are you able to separate out these costs?

• Are you able to break down the costs between the robot, its software, and any other installation costs?

• Are robots/robotic equipment capitalized or expensed?

• Are the robots themselves expensed differently than their related software or other expenses?

• Where do you purchase the robots/robotic equipment from?

  • If needed: A robot manufacturer, a system integrator, both?

• Who services the robots/robotic equipment?

  • If needed: The manufacturer, an integrator, in-house, something else?

• Do the robots/robotic equipment physically interact with humans in a shared workspace, or are they fully autonomous?

COGNITIVE INTERVIEW QUESTIONS

Now I’d like to introduce you to the questions that we have drafted.

First, I’d like you to review this page.

Interviewer: Allow them to read the instructions if they choose, but do not explicitly direct them to do so.

• In your own words, what is this question asking you?

• Do you have this information in your records?

• Do you have a ballpark figure for what these answers might be?

• What do you think is intended when the survey asks for “industrial robots”?

  • What types of equipment are you including?

  • Excluding?

• I’d like to go line by line, and ask if you would be able to retrieving the data that’s being asked for in that line, by establishment. Also, please tell me if there would be anything difficult about retrieving the data.

  • The number of industrial robots used at the plant?

  • The average purchase price of industrial robots used at the plant?

    • Is it possible that the number of robots at the plant is different from the number of robots USED at the plant? In other words, are there any robots that are going unused at the plant?

  • Number of industrial robots purchased for this plant in 2018?

  • Average purchase price of industrial robots purchased for this plant in 2018?

• What is the difference between what you’re being asked in Line B, versus Line D?

Now I’d like to show you an alternate version of this question.

• In your own words, what is this question asking you?

• Do you have this information in your records?

• Do you have a ballpark figure for what this amount might be?

  • How would you answer this question if you had no equipment of this type for a particular plant?

• What do you think is intended when the survey asks for “industrial robotic equipment”?

  • What types of equipment are you including?

  • Excluding?

• What does the term “gross value” mean to you?

• What are you including in “capital expenditures”?

  • Is there anything discretionary about what’s included in capital expenditures, or does this follow strict accounting guidelines?

• If this question were included in the actual survey, would you be able to retrieve the data that’s being asked for, by establishment?

  • Is there anything difficult about retrieving this data?

Reviewing both versions of this question:

• Did you review the introductory text before answering the question?

• Does this match the definition you provided earlier? If not, how is it different?

  • Does your company have any machinery that you would not personally consider to be a “robot” or “robotic equipment,” but you would be inclined to include because of the definition? What are they?

  • Are there any types of “robots” or “robotic equipment” that your company has, according to how you defined it earlier, that you believe this definition is telling you to exclude? What are they?

• If you didn’t have anything to report in one of the lines in these questions, would you enter a zero, or would you leave it blank?

  • Would a Check if None box be helpful, or do you think that is unnecessary?

Which version of the question do you prefer, the first one you reviewed, or the second one? Why?

Alternate Definition

The last thing I’d like to show you today is an alternate version of the introductory section and definitions.

• Do you notice any differences between this version and the version you reviewed previously?

  • What are they?

• Is this passage telling you anything substantially different than the version you’ve previously reviewed?

• Which version of the definition do you prefer? Why?

Wrap-up

We are almost done, and I just have a few wrap-up questions.

• Do you think they would be easy to answer, difficult to answer, or somewhere in between? Why?

• The capital expenditures question on the ASM currently has three sections: automobiles, computing equipment, and other. In which of these three sections have you been reporting expenses related to your robotics equipment?

• Is there anything else we haven’t touched on today, that you think is relevant?

• Do you have any questions for us?

Thank you very much for your time today!