Attachment A_ASM Unfilled Orders Protocol 10.15.20

Annual Survey of Manufactures (ASM)

Unfilled Orders Question (from M3 UFO)

Discussion Guide for Company Visits

10/15/20

---

Introduction:

• Thank you for your time today. My name is XXXXXX and I work for the United States Census Bureau. My research team evaluates how easy or difficult Census products are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it. Before we start, I wanted to review the pdf I emailed you, to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to record the session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Any information you provide will be kept confidential. Are you ok with being recorded? Great, I will start recording now.

Thank you.

• I am going to give you a little background about what we will be working on today. We are considering adding a question related to Unfilled Orders to the Annual Survey of Manufactures. Today, we’d like to have you answer a few questions relating to the value of products shipped and unfilled orders. We’d also like to get your feedback on what the level of burden associated with retrieving that data might be.

• To do this, we will have you answer several questions to the best of your ability, as they pertain to your establishment. There are no right or wrong answers, we are mainly interested in your impressions both good and bad about your experience.

• While you are going through the questions, I would like for you to think aloud. Essentially what this means is to verbally express what you are doing and experiencing as you navigate the questionnaire.

Demographics:

• Titles/Roles of Interviewees:

• Years in Position:

• Experience completing the ASM:

Great, thank you. Now I’d like for you to take a look at the draft questions. I’ll have you navigate through the questions and answer them as they pertain to your establishment to the best of your ability. Remember to think aloud as you go.

[Have them open the draft questionnaire.]

Question Specific Probes:

Question 5a: What was the value of products shipped and other receipts for this establishment?

• In your own words, what would you say this question is asking?

• How did you come up with your answer?

• Would you consider this the same as “Annual Sales”?

• Do you find the Includes and Excludes helpful or not helpful?

• Would you need to get assistance from others to answer this question?

  • Who would you need assistance from?

  • How would you contact them?

  • How easy or difficult would it be to get assistance from them?

• Do you have any suggestions to improve this question?

Question 5b: What percent of the $xxx,000.00 reported in Item 5, line A was for goods that were ordered or whose movement was controlled or coordinated over electronic networks?

• In your own words, what would you say this question is asking?

• How did you come up with your answer?

• Do you find the definition and Includes helpful or not helpful?

• Would you need to get assistance from others to answer this question?

  • Who would you need assistance from?

  • How would you contact them?

  • How easy or difficult would it be to get assistance from them?

• Do you have any suggestions to improve this question?

Question 5c: What was the value of this establishment’s unfilled orders (order backlog) as of December 31?

• In your own words, what would you say this question is asking?

• How did you come up with your answer?

• What does “unfilled orders” mean to you? How about “order backlog”?

• Would you need to get assistance from others to answer this question?

  • Who would you need assistance from?

  • How would you contact them?

  • How easy or difficult would it be to get assistance from them?

• What records do you keep related to this topic? In what format are these records kept?

  • Are they available electronically, or paper files? Both?

• How easy or difficult would you say this question is for you to answer? Could you estimate how much extra time it would take you to provide the answer to this question?

• If you didn’t have unfilled orders, or you did not maintain records for them, how would you answer this question?

• What does “Check if None” mean to you?

• Would you like additional definitions or Includes/Excludes related to this question? What would you most want to know?

• Do you have any suggestions on how to improve this question?

Instructions:

• Now I would like you to scroll down to see the Instructions related to this question that will be in the Annual Survey of Manufactures User Manual.

• What is your first impression of these instructions?

• Could you please read through them and let me know what your reaction is?

• Is there anything else you would like to see added to these instructions? Anything that could be deleted?

Industry Probes:

• Thank you for your input on those questions and the instructions.

• Now, I would like to ask you a bit about the manufacturing activities that occur at this establishment.

• We have your NAICS code listed as xxxxxx: xxxxxxxx xxxxxxx xxxxx. Is that how you think of this establishment?

• Does that cover all the manufacturing activity that occurs at this establishment?

• Does this establishment manufacture any other types of products?

• (If Yes) Could you please open the last attachment, labeled Manufacturing Activities List?

• Please look over the codes and tell me which ones are relevant to your company.

• [Probe regarding how/whether they could report Unfilled Orders for multiple codes.]

Respondents who also reported for M3UFO

• What was your experience completing the Manufacturers’ Unfilled Orders Survey (M3UFO)?

• Do you think it would be easier or harder for you to report this on the ASM, versus on the Manufacturers’ Unfilled Orders Survey (M3UFO)?

• Which would you prefer?

• Do you have any concerns about reporting this item on the online-only Annual Survey of Manufactures, versus the current Manufacturers’ Unfilled Orders Survey (M3UFO) which is a paper form sent in the mail?

• If Yes, please describe.

Respondents that report for multiple estabs

• Are you able to provide this data for every establishment you report for on the Annual Survey of Manufactures?

• Would you need to contact anyone else to answer these questions?

Wrap Up:

• We are almost finished. Do you have any other comments or suggestions for us?

• Thank you so much for your time today.

4