Protocol

ACES Machine Learning Respondent Debriefing Interviews – Spring/Summer 2019

May 2, 2019

Research Questions:

• How would respondents describe their overall experience with the survey? Item 3A and 3B?

  • Is it clear how reported values for expenditures are associated with the write-in descriptions?

• Do respondents understand what information they are being asked to provide with the survey question?

• Is the Other, specify useful?

• What type of responses did they provide in the description field?

• Did respondents report any expenditures that prompted the machine learning component?

  • If so, what were their initial thoughts on the functionality?

• Did respondents encounter any error messages (regarding classification of expenditures)?

• Are respondents able to successfully resolve errors associated with the classification of expenditures?

  • Did respondents remove entries in both the value and description fields when reclassifying?

• Are the instructions helpful to respondents?

  • Do respondents understand the terminology used?

• Did respondents view definitions?

  • Were the definitions helpful?

  • Do respondents understand the terminology used?

  • Did the definitions prompt respondents to reclassify expenditures?

• How are expenditures and associated values maintained in records?

• Did the respondent agree with the reclassification of an expenditure by the machine learning component?

  • If no, how would they resolve this discrepancy?

• Respondent thoughts on the machine learning design/functionality

  • Automatic reclassification version confirmation of reclassification

Expected Length of Debriefing Interview: 30 minutes

Required Materials

• Screenshot of Item 3

Introduction

• Introduce all attendees and include brief overview about the survey

• Tell respondent why we asked to speak with them

• Understand how they developed their answers

• Refine the survey instrument and make it easier to report

• Explain purpose of the interview: Assess the functionality of the write-in field for reporting expenditures not listed

• Send visual illustrating the machine learning functionality via email (e.g., images/screenshots, video clip illustrating functionality

• Administer debriefing probes to address research questions

General probes that may be used throughout the interview:

• Reflect back on respondent’s answer: “you said…”

• In your own words, what is this question asking?

• How did you arrive at this number/answer this question?

• What records (if any) did you look at? What line or lines were of interest?

• Specifically, what did you include in this number? What did you exclude?

• Did you consult other people to obtain this answer?

General Questions

ABOUT THE RESPONDENT

First, I’d like to learn a bit about you, and your role here at the company.

• What is your job title?

• How much experience do you have completing the Annual Capital Expenditures Survey?

• What has your experience completing the Annual Capital Expenditures Survey been like?

• Do you work with anyone else in your company to get the data for the Annual Capital Expenditures Survey?

  • How are those other people involved?

  • How many people are involved?

Now I’d like to learn a little about your company.

• Can you give us a brief description of what your company does?

  • What kind of products do you sell/manufacture or services do you provide?

Debriefing Probes/ Questions

In order to refresh your memory, I’d like for you to review the screenshots and video clip that I e-mailed to you. (Note: Test administrator explains machine learning functionality to the respondent.)

• How would you describe your overall experience with completing the Annual Capital Expenditures Survey?

• Can you tell me about your experience responding to item 3?

  • In your own words, what is being asked in this item?

  • Is it clear how reported values for expenditures are associated with the write-in descriptions above?

    • Can you tell me a little more about that?

• What information do you think that this survey question is asking for you to provide?

• Did you find the Other, specify category useful?

  • If no, why wasn’t the Other, Specify category useful for you?

• Do you recall what type of responses you provided in the description field?

  • Can you tell me more about that?­

• Did you encounter any error messages (regarding classification of expenditures)?

  • If so, what were your initial thoughts on the functionality of this?

  • How did you resolve the errors associated with the classification of expenditures?

    • Did you remove entries in both the value and description fields when reclassifying?

    • Did you contact the Census Bureau for assistance? If no, can you tell me about that?

  • Did you agree with the reclassification of an expenditure by the machine learning component?

    • If yes, can you tell me more about that?

    • If no, how did you resolve this discrepancy?

  • If the survey were to automatically reclassify your entry without the use of the message what are your thoughts on this?

    • Would you prefer this reclassification to be confirmed by you? Why or why not?

• Did you find the instructions to be helpful?

  • Did you understand the terminology used?

    • If no, what terminology did you not understand?

• Did you view definitions for Structures and/or Equipment?

  • If yes, which? Were the definitions helpful?

  • Did you understand the terminology used in the definitions?

  • Did the definitions prompt you to reclassify expenditures?

• How are expenditures and associated values maintained in your company’s records?

  • Can you tell me more about that?

  • Are the records easily accessible to you?

  • Did you have to reach out to others to answer the survey questions?

    • If yes, about how many other people? What are their role(s) in your company?

Wrap-up

We are almost done, and I just have a few wrap-up questions.

• Do you think this specific question (Item 3) was easy to answer, difficult to answer, or somewhere in between? Why?

• How long did it take to answer items 3A and 3B?

• How much time did it take to complete the entire survey, including gathering and entering the data?

• Is there anything else we haven’t touched on today, that you think is relevant?

• Do you have any questions for us?

Thank you very much for your time today!