Protocol

Protocol for BRDS Draft Questions

BRDS Questions

Early Stage Scoping / Cognitive Interviews

Amy Anderson Riemer

Draft

June 28, 2018

Revised: August 14, 2018

Research Questions:

• Do respondents understand what information they are being asked to provide on these questions?

• Do respondents understand the terminology used in the questions?

• Is the requested information available in respondents’ records?

• Are the instructions helpful to respondents?

Expected Length of Interview: 30-60 minutes

Materials Needed:

• Draft questions and Artificial Intelligence definition sheet

• Consent forms

• Digital recorder (for in person interviews)

General probes that may be used throughout the interview:

• Reflect back on respondent’s answer: “you said…”

• In your own words, what is this question asking?

• How did you arrive at this number/answer this question?

• What records (if any) did you look at? What line or lines were of interest?

• Specifically, what did you include in this number? What did you exclude?

• Would you consult other people to obtain this answer?

INTRODUCTION

• Explain purpose of meeting: get feedback on potential new questions for the Business Research and Development Survey on artificial intelligence and leased or temporary employees, including on-site consultants.

• There may be times throughout this interview that this seems like a test of you, I assure you that it is not. The purpose of these interviews is to ensure that we are communicating the data requests clearly with you, so in that way, these interviews are actually a test of us. On that note, we want your honest feedback throughout, and if it’s not clear to you how to answer, please do your best. There will be ample opportunities for you to let us know how to make improvements so that you could do a better job answering.

• As we go through the interview, we will ask both general and specific questions. If at any time a question seems odd to you, please let us know. We encourage all feedback.

• (For in person interviews) Before we start, I have a consent form that goes over the authority that we have to conduct these interviews. There’s also a piece in here where we ask for permission to record this interview, which is strictly for our note taking purposes. These interviews will only be heard by people directly involved in the development of the survey. Do we have permission to record our conversation for research purposes?

[Have respondent sign consent form. If they decline recording, note on the consent form that they declined recording and have them sign.]

ABOUT THE RESPONDENT

First, I’d like to learn a bit about you, and your role here at the company.

• What is your job title?

• What is your role in completing government surveys?

• How familiar are you with your company’s research and development activities?

• How much experience do you have completing the Business Research and Development Survey?

• What has your experience completing the Business Research and Development Survey been like?

• Do you work with anyone else in your company to gather the data for the Business Research and Development Survey?

  • How are those other people involved?

  • How many people are involved?

Now I’d like to learn a little about your company.

• Can you give us a brief description of what your company does?

• What kind of products do you sell/manufacture or services do you provide?

• [If manufacturing] Do you have any domestic or international manufacturing locations? How many?

SCOPING QUESTIONS

Thank you very much for that background information! Now, I’d like to transition to the main topic of today’s interview.

Terminology

I’d like to introduce you to some terms. For each of these terms, please define them in your own words, within the context of your company. It may also be helpful for you to provide us with examples, if any come to mind.

Interviewer:

• Allow respondent to define each term individually before moving on to the next.

1. Artificial Intelligence

• Are there any other terms that you may use that you consider synonyms for these terms?

• Can you give me some examples of research and development performed by your company that was for artificial intelligence?

Record Keeping

• In your financial records, are you able to identify the amount of R&D work for artificial intelligence?

• If you are not able to access these records, does someone else have access to them, or are they simply not kept?

COGNITIVE INTERVIEW QUESTIONS

Now I’d like to introduce you to the questions that we have drafted.

First, I’d like you to review this page.

Interviewer: Allow them to read the instructions if they choose, but do not explicitly direct them to do so.

Artificial Intelligence 5-13 AND Artificial Intelligence Definition

• In your own words, what is question 5-13 asking you?

• Is this the right location for this question within this section of the form?

• Do you have this information in your records?

• What are you including in “artificial intelligence?”

  • Does your company do R&D that applies artificial intelligence to a problem?

  • Does your company do R&D that is intended to advance the state of artificial intelligence?

  • Which of those (applying artificial intelligence or advancing artificial intelligence) would you include in your answer?

  • If we were only interested in R&D to advance artificial intelligence, how could we make that clear in the question?

    • (It is important to probe whether the respondent sees a distinction between R&D that applies artificial intelligence to a problem versus R&D intended to advance the state of the art of artificial intelligence itself.)

• If this question were included in the actual survey, would you be able to retrieve the data that’s being asked for? Is there anything difficult about retrieving this data?

• Would you need to provide an estimate?

  • How would you go about doing so?

  • Would you be comfortable providing an estimate?

• If you didn’t have anything to report in this question, would you enter a zero or would you leave it blank?

• How long would it take you to answer this question?

• Did you review the definition before answering the question?

• Does this match the definition you provided earlier? If not, how is it different?

  • Are there any types of artificial intelligence R&D that your company does, according to how you defined it earlier, that you believe this definition is telling you to exclude? What are they?

• If you haven’t looked at the definition provided, please do so.

  • Is there anything missing from our definition of artificial intelligence?

  • Are all of the capabilities of artificial intelligence that you are aware of or research at your company included in the definition? If not, what is missing?

  • Are all of the product and service applications of artificial intelligence that you are aware of or research at your company included in the definition? If not, what is missing?

  • Are there any terms provided in the definition that could cause potential confusion?

  • Do you have any other recommendations for how we could improve the definition that we have provided?

• What do you think is intended when the artificial intelligence definition includes “intelligent behavior in computers? “

• Would you like any additional clarifications/definitions in the introductory text?

• Did you notice the note that “You may report the same R&D in multiple areas”?

  • How would you handle that?

  • Do you think it is helpful, or do you think it is unnecessary?

• Do you have any artificial intelligence expenses that are not related to R&D work/would not be included in R&D? Can you tell me a little more about that? What are those expenses? Did you include any of that in this question?

Artificial Intelligence 5-27

• In your own words, what is question 5-27 asking you?

• Is this the right location for this question within this section of the form?

• In your own words, can you describe the difference between this question and 5-13?

• Would you include anything different in your definition of artificial intelligence in this question compared to 5-13? Why or why not?

• Do you have this information in your records?

• If this question were included in the actual survey, would you be able to retrieve the data that’s being asked? Is there anything difficult about retrieving this data?

• Would you need to provide an estimate?

  • How would you go about doing so?

  • Would you be comfortable providing an estimate?

• If you didn’t have anything to report in this question, would you enter a zero or would you leave it blank?

• How long would it take you to answer this question?

• Did you notice the note that “You may report the same R&D in multiple areas”?

  • How would you handle that?

  • Do you think it is helpful, or do you think it is unnecessary?

SCOPING QUESTIONS

Terminology

I’d like to introduce you to some terms. For each of these terms, please define them in your own words, within the context of your company. It may also be helpful for you to provide us with examples, if any come to mind.

Interviewer:

• Allow respondent to define each term individually before moving on to the next.

1. Leased Employees

2. Temporary Employees

3. On-site consultants/Off site consultants

• Are there any other terms that you may use that you consider synonyms for these terms?

• In your mind, are there major differences between the terms leased employees and temporary employees? Can you explain them to us?

• In your mind, how does the term “on-site consultant” relate to the terms “leased employees” and “temporary employees”?

• Can you give me some examples of employees that might be classified as temporary or leased?

  • What types of jobs do these employees perform at your organization?

  • Are any of these leased or temporary employees associated with research and development? Can you tell me more about them?

  • What types of R&D activities do they participate in?

Record Keeping

• In your financial records, are you able to identify:

  • The number of R&D workers that would be classified as temporary or leased employees? As on-site consultants?

  • The number of R&D work performed by temporary or leased employees or on-site consultants?

• If you are not able to access these records, does someone else have access to them, or are they simply not kept?

Leased or temporary R&D employees 6-8

• In your own words, what is question 6-8 asking you?

• Is this the right location for this question within this section of the form?

• Do you have this information in your records?

• If this question were included in the actual survey, would you be able to retrieve the data that’s being asked for? Is there anything difficult about retrieving this data?

• Would you need to provide an estimate?

  • How would you go about doing so?

  • Would you be comfortable providing an estimate? Why or why not?

• If you didn’t have anything to report in this question, would you enter a zero or would you leave it blank?

• Is the phrase ‘including on-site consultants’ meaningful to you? How?

• What are the types of occupations that you would report for?

  • Do you track job titles or assigned roles for:

    • leased employees?

    • Temporary employees?

    • On-site consultants?

    • Off-site consultants?

      • For the purposes of this question, are you able to differentiate On-site consultants from Off-site consultants? If so, how do they differ?

      • Can you tell me more about the on-site consultants that you would include in this question? How integrated are they in R&D activities?

      • Are tasks different for on-site versus off-site consultants who work on R&D activities? If different, how so?

    • To what extent are the R&D activities conducted by On-site and Off-site consultants managed by your company? Please describe.

    • How easy or difficult would it be to include totals for On-site consultants to this question?Describe. How are these data maintained for these types of employees/staff?

    • Are you able to identify on-site consultants involved in R&D activities who fall into each of the categories outlined in the question? Are there any that would be difficult to account for? Describe.

      • If possible, can you describe the types of consultants that may be included in each of the categories.

      • Would you be able to differentiate between domestic and foreign operations for these on-site consultants?

• How long would it take you to answer this question?

Wrap-up

We are almost done, and I just have a few wrap-up questions.

• Do you think these questions would be easy to answer, difficult to answer, or somewhere in between? Why?

• Is there anything else we haven’t touched on today, that you think is relevant?

• Do you have any questions for us?

Thank you very much for your time today!

10