Protocol

ABS 2019 Respondent Debriefing Interviews – Fall 2019-Winter 2020

Required Materials

• Copy of questionnaire

• Information on Respondent’s answers (these can be screen shots or some other form of record of the participants’ responses)

• Recording device

Introduction

• Introduce all attendees and include brief overview of why the survey exists and role of NSF and Census:

The Annual Business Survey (ABS) is conducted jointly by the U.S. Census Bureau and the National Center for Science and Engineering Statistics within the National Science Foundation. ABS provides information on selected economic and demographic characteristics for businesses and business owners. Additionally, the survey measures research and development, innovation and technology, as well as other business characteristics.

• Thank respondent for completing the survey

Tell respondent why we asked to speak with them

• Understand how they developed their answers

• Refine questions and make them easier to answer

Ask about recording the interview: We would like to record this interview so that we have a record of an accurate record of your feedback. We plan to use your feedback to improve the design and layout of the form for future data collections. Only staff involved in this data collection will have access to the recording. Would that be ok? If respondent says no, move on.

Respondent will be sent screenshots of questions from the web survey instrument (blank).

General Questions

Introduction:

• Can you tell us a little bit about your business, what types of goods or services does this business provide?

• What is your role in the company? What kind of responsibilities do you have?

• Have you completed other surveys for the Census Bureau or other agencies? If so, which ones?

• What was your role in the process for responding to this questionnaire? (Gathering data? Entering data? Consulting with data providers? Etc.)

• About how long did it take for you to complete the survey? Were others involved in completing the survey? If yes, approximately how many? If yes, was their time included in your estimate?

General questions about the process of completing the survey:

• Can you tell me about your process for filling out the Annual Business Survey? What did you do once you receive the survey invitation in the mail?

• Did you have any comments of concerns about the letter you initially received about the survey?

• Can you tell me about how you accessed the survey? How easy or difficult was that process?

• Tell me about your experience with the online website to complete the survey? How easy or difficult was it for you to enter in your data online? How easy or difficult was it for you to navigate through the survey?

Now I would like to ask you about your responses to some specific questions from the ABS.

General Probes:

• What did this question mean to you?

• How did you go about answering this question?

• Were you able to answer this question yourself? (If no) Who did you consult to help you answer this question?

• (If R appears to have had a problem/confusion with a specific question) What are you thinking about?

• Reflect R’s specific answer: You said ___. Can you tell me more about that?

Item specific Probes:

Section A: Company Information

A.8 Number of W-2 Paid Domestic Employees or Employee/Owners

• How did you go about answering this question?

• Who did you include in your response and what is their role in this business?

• When answering this question, were you thinking only of domestic employees?

• Where did you get the data in order to answer this question?

• Did you need to reference a W-2 to help you understand which owners to include in your response?

• Is there anything else we should know about the answer you provided?

A.9 Number of Domestic Workers Who Did Not Receive a W-2

• Who did you include in your response to this question?

• Where did you/would you get the data needed to answer this question? How did you distinguish between (a) Individuals whose work was directed by this business who received payment in other ways and (b) Unpaid individuals who worked for the business?

• What did you think of the reference to the 1099 form? Are you familiar or unfamiliar with this form? (If familiar, did you utilize this in your response to this question?)

A.13 Types of Workers

• How did you come up with your response to this question?

• Are the types of workers outlined in the response categories clear to you? Did you find any of the categories confusing? Are there categories of workers that you think should be on this list?

• Did you include workers who did not receive a W-2 in your response to this question?

Section C: Innovation

C.1 New or Improved Goods or Services

• How did you go about answering this question?

• Do you consider your business to offer goods, services, or both to its customers?

• How do you define goods? (If respondent said yes to business offering goods, how do you define goods in the context of your business?)

• How do you define services? (If respondent said yes to business offering services, how do you define services in the context of your business?)

• (If participant said Yes to Goods or Services) What were you thinking of when you said Yes to this question? Please provide examples of Goods and/or Services.

• In what ways did those Goods and/or Services you referenced differ significantly from the business’s previous goods or services?

C3. New or Improved Goods or Services as Percent of Total Sales

• How did you come up with your answer to this question?

• This question references ‘A.11 Domestic Sales and Revenues,’ was it helpful to you when answering this question?

• Did the reference to A.11 change your answer to question C.3? Did this reference in C.3 change your answer to question A.11? How did you handle these two questions in relation to each other?

• How did you come up with your figure for this question? Was your response based on records or an estimate?

  • If your response was based on records, what kind of records did you use to answer this question?

  • If your response was based on an estimate, how did you go about estimating? (Spontaneously probe on this, based on what respondents tell you.)

C5. New or Improved Business Processes

• Please tell me how you came up with your response to this question.

• (If participant said Yes to any of the answer categories) What were you thinking of when you said Yes to this item? Please provide examples of the new or improved business processes. How did these new or improved business processes differ significantly from your previous business processes?

• When answering this question, were you thinking about the three years 2016 to 2018, or some other time period? If some other time period, would you answer change if you had been thinking about the three years 2016 to 2018?

C7. Activities for New or Improved Products or Business Processes

• How did you come up with your response to this question?

• When answering this question, were you thinking about only things that related to innovation or were you thinking about other things?

• Some people have told us that they read only some of the text on the page, and other people read all of it. Do you happen to remember what you read on this page? Headings only? Headings and text? Probe to find out exactly how much of the material on the page they actually read.

C8. Activity Costs

• How did you determine your answer to this question? How easy or difficult was it to come up with an answer?

• What types of costs were you thinking of when you answered to this question?

• Was the figure that you provided based on records or an estimate?

  • If the amount was based on records, what kinds of records did you use to formulate your response?

  • If the amount you provided was based on an estimate, how did you come up with that estimate?

C9. R&D for Activity Costs

• How did you determine your response to this question? Could you please give me some examples of activities you were thinking about when answering this question?

• What costs were you thinking of when you answered this question?

• Was the figure you provided for R&D costs based on records or an estimate?

  • If the amount was based on records, what kinds of records did you use to formulate your response?

  • If the amount you provided was based on an estimate, how did you come up with that estimate?

• How easy or difficult was it to come up with an answer to this question?

Section D: Research & Development (*only asking R&D questions for companies with 1 to 9 employees)

D1. R&D Activities

• How did you determine your response to this question?

• (If the respondent answered yes for any R & D activities): I see that you responded yes to the following research and development activity: ______, can you provide me an example of the specific activities you were thinking of in your response?

D9. R&D Employees

• How did you go about answering this question?

• What were you thinking about when answering this question?

• Did you use records to answer this question, or did you estimate? Tell me more about that.

• When responding to this question, did you include only your business’s employees, or did you include some other business’s employees, too?

• Did you include only domestic employees, or did you include foreign employees, too?

D10. R&D Employee Occupations

• How did you come up with your response to this question?

• Does your company identify employees as researchers, technicians, or R&D support staff, or not? If yes, how are they identified (in your company records)?

• Can you tell me a little about your business’s R&D employees and their occupations? What sort of work do the R&D employees do?

• What types of records, if any, did you reference to formulate your response?

D11. Domestic R&D Employees Full-Time Equivalent

• How did you come up with your answer to this question?

• Does your business have any R&D employees who are not domestic?

• Are your R&D employees full-time or are they doing other things for the business?

• What types of records, if any, did you reference to formulate your response?

• What does “domestic employee” mean to you? Who gets counted as a “domestic employee” and who does not? For example, how would you count someone who teleworks from Canada?

D12. Domestic Researchers Full-Time Equivalent

• How did you come up with your response to this question?

• In your own words, what is the difference between questions D11 and D12? Is this distinction meaningful to you?

• What types of records, if any, did you reference to formulate your response?

Section E: Technology and Intellectual Property

• When responding to this survey, do you recall using any of our hyperlinks? If yes, were they helpful or not? If you didn’t use them, did you notice that hyperlinks were available?

E3. Production Technology for Goods and Services

• For each technology that the respondent reported to use, ask them to define that technology in their own words or to give examples of how their business uses that technology.

• We’ve been hearing that some people interpret “low use,” “moderate use,” and “high use” in different ways. What do these terms mean to you, for your company?

• How did you decide which box was most appropriate for your business for each of the technologies?

• Did you feel that the answer categories provided allowed you to adequately describe your business’s use of the various technologies, or not? If not, please tell me more about that. What do you think would be a better way of asking this question?

E5. Impact of Artificial Intelligence Technology on Workforce

• If company had AI:

• How did you come up with your response to this question?

• What types of records, if any, did you reference to formulate your response?

E6. Impact of Artificial Intelligence Technology on Workforce

• If company had AI:

• How did you come up with your response to this question?

• What types of records, if any, did you reference to formulate your response?

E8. Impact of Cloud-based Computing Systems and Application Technology on Workforce

• If company had Cloud-based systems:

• How did you come up with your response to this question?

• What types of records, if any, did you reference to formulate your response?

Wrap Up

• When completing the survey, did you refer to the definitions? Did you click on the definition links provided in the web instrument? If yes, were the definitions helpful to you?

• I’ve only asked you about some of the questions on the survey. Were there any other questions you wanted to tell me about?

• Do you have any other recommendations for improving the survey that we didn’t get a chance to talk about?

That is all the questions I have for you today. Thank you very much for taking the time to participate in this evaluation and provide valuable feedback.