Debriefing Interviews Protocol

AIES Pilot Study

Respondent Debriefing Interview Protocol

Phase II

Winter/Spring 2023

This protocol is a guide – the questions here will not necessarily be asked exactly as worded in the protocol or in this order. Not all questions will be asked in every interview. As much information about interview participants and enterprises/establishments should be obtained prior to the interview as possible. In some cases, probing may need to be adjusted based on the background research that has been conducted and/or in response to participants’ insights.

These interviews will be guided by four research questions, each with sub-questions:

Research Question 1: How are respondents completing the survey?

• What is the process of gathering the data necessary to complete the survey?

• What aspects of survey completion were the most time consuming or involve the most people?

Research Question 2: Is the new survey overly burdensome?

• For production cases: How, if at all, has the existing survey response procedures been the same and different using the new survey, as compared to the regular annual surveys?

• How many and what kind of staff are involved in gathering the information for this survey?

Research Question 3: Is the new survey using appropriate units of analysis?

• How do respondents evaluate the new “top down” approach to collecting these data?

• For which questions did respondents have to allocate their responses? For which are they estimating? For which are they aggregating?

Research Question 4: For total or partial non-responders: Are unit response patterns the same or different as annual surveys?

• What role does actual burden or perceived burden play in the decision to complete the survey?

• What roles does perceived redundancy play in the decision to complete the survey?

• What aspects of the survey instrument may be impeding the response process?

Materials: Protocol, consent form, survey responses, respondent instrument (overview of responses)

Method: We will conduct the interviews by phone, or we will conduct the interviews in person, depending on availability and federal employee travel restrictions.

Expected length of interview: 1 hour (60 minutes) maximum

General probes that may be used throughout the interview:

• How did you arrive at this number/answer this question?

• Were these data easy to access?

• What else can you tell me about this?

1. Introduction

If necessary: I sent you a consent form in an email today – did you get a chance to sign that? If not, please do so now, and then we’ll get started.

Thank you so much for agreeing to talk with me today!

As part of the pilot program for the new Annual Integrated Economic Survey, we are following up with some companies to learn more about the processes you may or may not have used to complete the survey.

I am part of a group within the Census Bureau that makes sure that our surveys are performing as expected, and provides feedback to other parts of the Bureau about ways to improve the performance of our instruments. I’m talking with you today because of your unique role in testing out the new AIES survey instrument.

Remember, my job is to improve the surveys. Please be candid and frank in your responses. Our interview is being conducted under the authority of Title 13, which means that your responses are confidential, and neither your name nor the name or identifying information about your company will be included in any of our findings.

Do you have any questions before we get started?

I’d like to record our session today so that when I go to analyze the results of these interviews, I can use the recording to pick up on anything I may have missed in my notes. Do I have your permission to record our session today?

<<Turn on recorder>>

2. Firm Background (5 mins)

• What is your role in the company? How long have you been with the company?

• What is your role in the process for responding to Census Bureau surveys?

  • Examples: gathering data, entering data, consulting with data providers, etc.

• Tell me a little bit about your business. What types of goods or services does this business provide?

  • How many locations does your business currently operate? In how many states? In how many countries?

  • Is this a foreign or domestic company?

  • Are there related companies?

  • Did you have any confusion about which parts of the company to include when answering the survey?

3. Research Questions 1 and 2: Response Process and Burden (20 mins)

• Tell me about your company’s process for responding to Census Bureau requests. Once you get a letter or email informing you of the request, what happens next? And then?

  • What staff are responsible for responding to Census Bureau requests? What are the duties of these positions?

  • Sometimes companies tell us about internal templates, spreadsheets, or other documents they use to complete our surveys, whether for tracking response or for manipulating data. Do you have anything like that? Please tell me more about that.

    • For production cases: How was completing the AIES pilot the same as completing the typical annual survey(s)?

    • For production cases: How was completing the AIES pilot different than completing the typical annual survey(s)?

• The pilot survey is comprised of three modules or parts.

  • Which of these three parts was easiest to complete, and why?

  • Which of these three parts was most difficult to complete, and why?

  • How did you decide whether to respond by uploading a spreadsheet or screen-by-screen?

  • Did you answer the modules in order, or did you respond another way, and why?

4. Research Question 3: (For responders) Units of Analysis (25 minutes)

• Looking at Step 1 – how easy or difficult was it to answer questions on organizational change? Why?

  • What about other organizational questions – how easy or difficult was it for you to answer these questions?

  • How clear was it that we were asking about the overall company at this point?

• Looking at Step 2 –

  • How easy or difficult was it to answer questions by establishment? Why?

    • Are certain data more accessible than others at this level? Which?

    • What did you do for questions where you did not have the data at this level, if any?

  • If applicable: How about employment data – how easy or difficult was it to get the production and non-production breakdowns? Why?

• Thinking about the survey overall, do you remember which questions you:

  • Allocated responses? (if necessary: This happens when you have one large number that you need to split up among several business units.)

  • Estimated responses? (if necessary: This happens if the data are unavailable or incomplete.)

  • Aggregated responses? (if necessary: This happens when you have data for specific business units but we are asking at a higher level in the company.)

• Overall, which question(s) would you say was the most time-consuming or difficult to complete?

5. Research Question 4: Barriers to Responding (25 minutes)

• I noticed that you did not complete the AIES pilot survey. Can you tell me about that?

  • What, if anything, could we at the Census Bureau do to support you in responding to this survey?

  • What are the reasons why you did not fully complete (or did not complete at all) the AIES pilot survey?

  • Do you have any suggestions for ways the Census Bureau could encourage companies to respond to this survey?

6. Wrap-up

• That’s all the questions I have for you today! Do you have any other comments, questions, or suggestions for us?

Thank you so much for your time today.