Attachment A_CFS Pilot Debriefing Protocol 10152020

CFS Pilot Test Debriefing Protocol

Draft 10/15/20

Introduction

Thank you for your time today. My name is XX and I work for the United States Census Bureau. My research team evaluates how easy or difficult Census surveys are to complete. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it.

Before we start, I wanted to review the pdf I emailed you, to read and sign. It explains the purpose of today’s session and your rights as a participant. It also informs you that we would like to record the session to get an accurate record of your feedback. Only those of us connected with the project will review the recording and it will be used solely for research purposes. Your name will not be associated with the recording or any of the other data collected during the session. Any information you provide will be kept confidential. Are you ok with being recorded? Great, I will start recording now.

About the Respondent

First, I’d like to learn a bit about you, and your role here at the company.

• What is your job title?

• What is your role in completing government surveys?

• How much experience do you have completing the Commodity Flow Survey?

• Do you work with anyone else in your company to get the data for the Commodity Flow Survey?

• How are those other people involved?

• How many people are involved?

• Do you use a particular software or third party source to obtain any of the information requested? If so, which one(s)?

Now I’d like to learn a little about your company.

• Can you give us a brief description of what your company does and how it is organized?

  • Is it an independent business, or part of larger establishment?

  • Does your company prefer that individual locations report to Census? Why or why not?

Overall Impressions of CFS

• In reflecting on your experience filling out the Commodity Flow Survey for the pilot, what were your overall reactions or impressions?

  • Did anything stand out to you as being easier compared to the original method?

  • Were any of the sections particularly difficult or time consuming to complete?

    • Any ideas on how we can make completing this portion of the survey easier?

Thanks for that feedback, what I’ll have us do now, is walk through the instrument and discuss the separate sections/screens. Just to refresh your memory, I’ve sent you a link to the website that hosts the CFS; we can walk through it together.

Login Process

• Did you have any difficulty logging into the survey?

Verification of Name and Shipping Address

• One of the first tasks for you was to choose between uploading a document with your shipping locations or verifying a pre-existing list of your locations that Census has on file.

  • Which did you choose? Why?

  • Did you have any difficulty with this method?

  • [If verify path] Was anything missing from the list?

    • Was it clear how one should make a correction if the address shown is incorrect?

    • Was it clear how to change operating status if needed?

  • [If upload path] Was it clear how to upload your file?

    • Did you have to make any formatting changes or edits for our system to accept the file?

    • Did you find this process easy or difficult?

Timing of Data Submission

• Did you have any thoughts about this screen?

• What did you select?

Outbound Shipment Activity

• Did these questions make sense to you?

• Was there anything difficult about these questions?

• If you needed help on this screen, what would you do? (If they don’t notice the Help icon, ask them to click on it.)

• Was the information here helpful? What else would you have wanted to see here?

Modes of Transportation

• Did you have any difficulty selecting the modes of transportation your company uses?

  • Was anything missing?

Upload Your Shipments

• [If 2017 respondent] How did this process differ from the original reporting method?

• What were your overall impressions of this new method of providing your shipment data via an export from your existing records?

• Was the format requested for your upload compatible with your files, or was any conversion required?

• Were the instructions clear?

• Was this method easier for you? More difficult?

• [Error Screen] Do you remember encountering any errors after trying to upload your file?

  • How easy or difficult was it to fix the errors you encountered?

    • Was it clear what action was needed?

• The survey requires about 8 core variables describing the commodities you ship (listed below), were any of these not available in your records, or very burdensome to obtain?

  • Shipment ID

  • Shipment Date

  • Shipment Value

  • Shipment Weight

    • Requested in pounds, was this difficult?

  • Product Description

    • How easy or difficult was it for you to identify the commodity descriptions? How are product descriptions maintained in your records? Multiple columns? Do you have any thoughts about not needing to provide SCTG codes using this method?

  • HAZMAT Code

• Shipment Destination

• Shipment Mode of Transportation

• Any suggestions for how we can make this process easier?

Shipment Validation

[Validate Record Screen] It’s possible you would have seen this screen if it wasn’t clear to our system which SCTG code matched the product description you provided (describe SCTG codes). Do you remember encountering this screen?

• How did it work for you?

• Were the descriptions provided clear or unclear?

• Were you able to select a code that accurately described the product?

• Do you have any thoughts on how we could make this screen more clear?

Contact Information

• Is it clear how to fill out the contact information?

• Are you the right person in your company to reach out to, to fill out the CFS?

CFS Pilot Review Your Responses

• How did the submission process go for you? Any difficulties?

  • Did you review your data before submitting, or just click “Submit”?

  • Did you feel like the data being sent to Census was secure?

  • Did you have any privacy/confidentiality concerns?

Wrap-Up

• How long did it take you to complete this pilot CFS?

• Was this an improvement compared to your previous reporting for CFS?

• After reviewing all sections of the survey, which section of the survey took you the most time to complete? Tell me more about that.

  • Any ideas on how the Census Bureau may be able to make reporting for this section easier/quicker?

• Would you be willing to provide data more frequently than every five years?  Would your data compilation process be easier if there weren’t time gaps? Why or why not?

• Would you be willing to fill out more than one week of data at a time? Why or why not?

• Do you have any other comments or suggestions for us?

• Any questions?

[Provide observer time to ask questions, if applicable.]

This concludes our session. Thank you for your time and valuable feedback.

6