Form ASM-RO Debriefing ASM-RO Debriefing ASM-RO Debriefing

Revised 1-13-2016

2015 ASM/RO Respondent Debriefings

Draft Interview Protocol

Research questions

• How did SU and MU respondents use the worksheet provided in the 2015 ASM/RO instruments? E.g., previewing questions, reviewing question requirements during course of gathering data, writing down responses, writing down procedural notes, conferring with colleagues, sending to data providers, receiving data from providers, etc.

• What specific parts of the worksheet did respondents use? E.g., global instructions, item/section headers, item-specific instructions, response fields, response field labels, margins (procedural notes), etc.

• What do respondents perceive to be the advantages and disadvantages of the worksheet?

• What additional features or information do respondents want in the worksheet?

• Did respondents use the worksheet as the primary survey instrument – i.e., to review questions/instructions, write down responses, etc. – before entering the data online?

• How did respondents use the worksheet in tandem with the electronic reporting instrument, if at all?

• If they used the worksheet in tandem with the electronic instrument, did respondents have any problems doing so? If so, what kinds of problems did they encounter?

• Did respondents rely solely on the electronic instrument to gather data and not use the worksheet at all?

• Did respondents try to obtain additional reporting instructions from BHS? If so, what information did they seek?

• Did respondents use the worksheet to capture procedural notes?

• Did respondents keep the worksheet for their records?

• Were any respondents unaware of the availability of the worksheet, and did they desire such a document?

• Were any respondents aware of the worksheet but decide not to use it? If so, why?

• Are there any other formats we should consider for presenting sufficient reporting instructions?

• How do respondents react to the format and content of our downloadable questions/instructions document mockup?

Procedure

• Before the meeting:

• Have an email ready to send attachments – prototype worksheet, and current worksheet if it is needed (i.e., if R did not retain a copy). Write “Worksheets” in the subject line. Include no text in the body of the email, except email signature.

• Note whether R responded to the RO, ASM, or both. Frame questions appropriately.

• Note whether R is SU or MU.

• Go through the Introduction.

• Go through each section of questions, ask general probes and any item-specific follow-up probes.

• When presenting worksheet prototypes, alternate which version R opens first from interview to interview, to achieve equal number of cases seeing each version before the other.

Introduction

• Reasons for visit/call:

  • To get a better understanding of how companies use the worksheet to complete the Report of Organization / Annual Survey of Manufactures.

  • To inform the design of our worksheets, which will be available for the RO/ASM and 2017 Economic Census

• Introduction of Census employees

• Introduction of company employee(s)

• Audio taping (Only ask this if recording is possible, e.g., over speaker phone)

  • Before we get started: With your permission, I'd like to audio tape this interview, so I don’t have to rely solely on taking notes.

  • This session is confidential, and only Census Bureau employees directly involved in this research project would have access to the recording.

  • Permission to audio-tape discussion? Obtain verbal consent.

• What is your role in the company and your primary responsibilities?

• Is this your first time completing the RO/ASM?

• How many locations did you report for?

Now I would like to ask you some questions about your experiences with completing the RO/ASM. If possible, please look over the resources you used to gather data in case you need to refresh your memory of using it.

Process for completing the RO/ASM

• Please briefly describe the steps you took to complete the RO/ASM.

Use of Worksheet and preferences

Now I would like to ask you questions specifically about the worksheet that was available as a download with the survey.

• Did you use the worksheet provided in the online system to help you complete the RO/ASM in any way?

  • If R did NOT use the worksheet:

    • Did you know there was a worksheet available?

      • (If yes) What made you decide not to use the worksheet?

      • (If no) Did you want a document such as a worksheet to use when filling out the survey?

        • If you had known the worksheet was available, do you think you would have used it?

• For our discussion today, I would like to review the current worksheet, and a prototype worksheet we will offer in the future. May I send it to you as attachments in an email?

  • (If consent is given) What is your email address?

  • Verify that the email arrived and ask R to open the “Current Worksheet” file before proceeding. Ask R not to open the “Worksheet Prototype” file yet.

  • Go to Worksheet Prototype section.

• (If yes) Did you save a copy of this worksheet, either paper or electronic?

• (If yes) Can you please open or retrieve your copy?

• (If no) May I send it to you it as an attachment in an email?

  • (If consent is given) What is your email address?

  • Verify that the email arrived and ask R to open the “Current Worksheet” file before proceeding. Ask R not to open the “Worksheet Prototype” file yet.

• Continue with closed-ended questions on next page.

I have a short series of questions about how you may have used the worksheet. For each question, please answer yes or no.

	Yes	No
1. Did you print the worksheet?
2. Did you preview the questions before getting started with gathering the data?
3. Did you review questions during the course of gathering the data?
4. Did you write your answers on the worksheet before entering data in the online system?
5. Did you write down notes about the process you went through to gather data, such as where certain data came from, etc.?
6. Did you use it to discuss the survey request in meetings?
7. Did you send the worksheet to other people in the company who provided you with data?
8. (If 7=Y) How did you send it to them?
Paper copy?
Email attachment?
Other? Specify:
9. (If 7=y) Did you use the worksheet to receive data from those people?
10. Did you provide the worksheet to someone else to review the data before submitting?
11. Did you save a paper copy of the worksheet?
12. Did you save an electronic copy of the worksheet?

What specific parts of the worksheet did you use?

What parts did you not use?

Overall, what is your opinion of the worksheet? Did you find it helpful or not helpful?

What about the worksheet was helpful/not helpful?

Are there any parts you found confusing? If so, what?

In your opinion, is there any missing information that should be included? If so, what?

(If 4=Y) You said that you wrote your responses on the worksheet before entering them into the online system.

• Please tell me about this.

• Did you have any problems in transferring data from the worksheet to the online system?

(If 5=Y) You said that you wrote some notes about the process of gathering data on the worksheet.

• Can you tell me about the notes you made?

(If 9=Y) You said that the other people in your company who provided data sent it to you using the worksheet.

• Can you describe that process for me?

• Did they write on the worksheet?

• How did they send it back to you (sent the paper copy, scanned and sent as email attachment, etc.)?

(If 10=Y) You said you provided a copy of the worksheet to someone else to review before submitting.

• Can you describe that process for me?

(If 11/12=Y) Did you use any support documents from the prior year to complete this year’s RO/ASM, or have you ever done that in the past?

• If so, what did you use?

• What information did they contain?

• Did you create them, or were they made by someone else?

Did you go to the Census Bureau’s website to look for any information about this survey?

• If so, what web page did you go to?

• What types of information did you look for?

• Did you find information that met your needs?

We are in the process of designing a new worksheet. Do you have any suggestions for us to create the most effective and user-friendly worksheet?

• What would it look like?

• What information would it contain?

• Is the PDF format acceptable, or would you prefer a different kind of document? If so, what?

Worksheet Prototypes

Now I would like to show you an alternative design we are considering.

Have R open either version 1 or 2.

Please take a moment to skim the document. (Allow R up to 30 seconds to skim)

• How would you use this document?

• What are your overall impressions of this document?

• What particular features do you notice?

Front page

What is your opinion of the information presented on the front page?

• What information would you want to see on the front page of a document like this?

• What is your opinion about the appearance of this page? Do you have any suggestions for changing the way this page looks?

• Do you recall seeing the text “Do not mail this worksheet”? What is your opinion of this message?

Overview section

How would you describe the purpose of this section?

• In your opinion, does this information need to be in this document?

  • (If yes) Where do you think it should be placed, here or somewhere else?

• Is there any other information you would want in this section? If so, what?

Item-specific instructions

Do you recall seeing the paragraph under the “Item-Specific Instructions” header? (Text about “some questions that may not apply…”)

• Please tell me in your own words what this statement means to you?

• In your opinion, is it clear or not clear?

• (If not clear) How would you rephrase this to make it clearer?

• What do you think about the way the questions and instructions are laid out?

If presented with Version 1 first:

In your opinion, what do the gray boxes mean?

Would you write in them?

Do you recall seeing the “Check if none” boxes?

• What does that mean to you?

• Have you ever marked that box in a Census Bureau survey?

Content types and order

In your opinion, what are the important parts of this document, if anything?

What parts are not important?

If you were designing this document, what parts would you keep, and what would you discard, if anything?

What would be the most logical order for these types of information?

What other information would you include? Where would you place it?

Wrap-up

What would you call this kind of document? For example, if you asked your co-worker, “Please send me the BLANK”?

Have you used other similar documents?

• What were they for? How did you use them?

• Have you seen other examples that have features we should consider? If so, what do you like about others you have used?

Thank respondent(s) for his/her (their) assistance.

11