Protocol

2020 MOPS Cognitive testing protocol

• Before beginning

• Introductions

• Audio taping – would only be done if interviews take place in a private room.

  • Permission to audio- record discussion?

  • Before we get started: I'd like to audio record this interview, so I don't have to rely on my memory later. This session is confidential. Only people connected with this project will have direct access to your recording. Is this all right with you?

  • Ask respondent to sign the consent form

• Participant Background

• How long have you been with the company?

• What is your title/role?

• What are your major responsibilities?

• What are your government reporting responsibilities?

• What other government surveys or filings, if any, do you also handle?

• General Probes (Use these whenever necessary)

• In your own words, what would you say is the purpose of this question?

• In your opinion, are the instructions helpful or not helpful?

• Can you tell me what you are thinking about?

• Can you tell me more about that?

• Reflect back on R’s answer (“I want to make sure I have it right. I think you said, “…?”)

• Survey Background

• Can you tell me about your process for filling out the Annual Survey of Manufactures in general, and specifically the Management and Organizational Practices Survey, MOPS, which was last conducted in 2015?

  • Additional probes (if necessary):

    • Was this your first time filling out this survey?

    • How many locations did you report for?

    • What is your role? How many people are involved?

    • How do you typically gather the data?

    • Did someone else have to review the data before you submitted it?

• Section B: Organization

This first section of the form asks questions about your establishment.

• Question 23 and Question 25

  1. In your own words, what is this question asking?

     1. How easy or difficult would it be to report? Describe

• Section C: Outsourcing

I want to ask you about the next proposed section of the MOPS survey. This section of the form asks question about outsourcing products or services.

• Question 26A:

  1. In your own words, what is this question asking?

     1. Is it clear how you should report?

     2. How does your establishment define each category?

        1. Own employees?

        2. Independent contractors?

        3. Temp Staff and Leased workers?

        4. Other Purchased services?

     3. How does your establishment define the services outlined? What types of activities may they include?

     4. Review the definitions for each, how do these definitions compare to how you define these categories?

        1. Anything missing?

        2. Anything confusing?

     5. How easy or difficult would it be to report the requested information by each of the categories? Describe.

        1. What records (if any) would you consult?

  2. Are there any other services that should be considered in this listing that account for a large proportion of labor costs/purchased services?

• Question 26B

  1. In your own words, what is this question asking?

     1. Is it clear how you should report?

        1. What records (if any) would you consult?

  2. What do you think is meant by full-year, full time units in the context of this question?

     1. Is the instruction “If your establishment engaged one full time independent contractor for six months, enter 0.5 under “Independent Contractors.” helpful? If yes, how so?

• Question 27:

  1. In your own words, what is this question asking?

     1. Do you have access to the requested information?

        1. What records (if any) would you consult?

     2. Is it clear that we are requesting information from 2015? Any thoughts on this?

        1. How easy or difficult would it be to report the requested information? Describe.

• Question 28

  1. In your own words, what is this question asking?

     1. How easy or difficult would it be to report the requested information?

     2. Is it clear that we are requesting information for 2022? Any thoughts on this?

        1. Describe the steps you would take to report the requested information.

• Question 29

  1. In your own words, what is this question asking?

  2. How easy or difficult would it be to report the requested information?

  3. What other factors, that aren’t listed, would be important in driving the decision?

• Section D: Data, Decision Making and AI

The next series of questions asks about your establishment’s use of artificial intelligence and other data products

• Question 30:

  1. Is it clear what is being asked? Tell me more about that.

     • Is it clear what is meant by stored in a digital format? What could this include?

  2. How easy or difficult would it be to report the requested information? Describe.

     • Thoughts on the answer categories?

       • Understanding of each category- None vs Function not performed at this Establishment

       • Understanding of functions listed

• Question 31

  1. Is it clear what is being asked? Tell me more about that.

     • Is it clear what is meant by descriptive summaries or graphics? What could this include?

     • Is the definition provided helpful or unhelpful? Explain

       • How does it compare to your interpretation?

         • Anything missing? Confusing?

  2. How easy or difficult would it be to report the requested information? Describe.

• Question 32

  1. Is it clear what is being asked? Tell me more about that.

     • Is it clear what is meant by predictive analytics? What could this include?

     • Is the definition provided helpful or unhelpful? Explain

       • How does it compare to your interpretation?

         • Anything missing? Confusing?

  2. How easy or difficult would it be to report the requested information? Describe.

• Question 33

  1. Is it clear what is being asked? Tell me more about that.

     • Is it clear what is meant by artificial intelligence? How do you define this?

     • Is the definition provided helpful or unhelpful? Explain

       • How does it compare to your interpretation?

         • Anything missing? Confusing?

  2. How easy or difficult would it be to report the requested information? Describe.

• Question 34

  1. Is it clear what is being asked? Tell me more about that.

     • Understanding of applications listed

       • Any missing?

  2. How easy or difficult would it be to report the requested information? Describe.

• Question 35

  1. How easy or difficult would it be to report the requested information for 2015? Describe.

• Question 36

  1. Is it clear what is being asked? Tell me more about that.

     • Understanding of applications listed

       • Any missing?

  2. How easy or difficult would it be to report the requested information? Describe.

     • Understanding of available response options

       • Prevented Use, Delayed use, Made use less effective/more difficult, no effect, accelerated use, don’t know

• Section F: Background characteristics-Auditing

This next section asks about auditing services

• Question 50:

1. How easy or difficult would it be to report the requested information? Describe.

2. How would you define the following

   1. Audit?

   2. Review?

   3. Compilation?

3. What are the differences between these terms?

4. Are there alternative terms that your establishment uses for these concepts?

5. Have you interacted with an auditor in your current position?

6. For multi-unit businesses, does HQ consult with your establishment about end-of-year financial audits?

• (If applicable) Question 51:

1. How easy or difficult would it be to report the requested information? Describe.

• (If applicable) Question 52:

1. How easy or difficult would it be to report the requested information? Describe.

   1. What records (if any) would you consult?

   2. Do you observe the activities of the auditor when she visits your establishment?

2. Is it clear what each of the categories are referencing?

   1. How do they compare to the way your establishment describe activities conducted by the auditor?

   2. Any missing?

   3. Any confusing?

• Are there any other comments or suggestions you would like to make about anything we have gone over today?

Thank the respondent for their time.