OMB Letter

Annual Integrated Economic Survey (AIES) Phase III Pilot Research Program

Submitted Under Generic Clearance for Field Tests and Evaluations

Request:

The U.S. Census Bureau plans to conduct additional research under the Generic Clearance for Census Bureau Field Tests and Evaluations (OMB Control Number 0607-0971). In support of the 2022 Annual Integrated Economic Survey (AIES) (OMB Control Number 0607-1024) the Census Bureau requests to field a Response Analysis Survey of respondents to further explore response burden to the AIES.

Background on the AIES:

The Census Bureau’s Economic Directorate asked the National Academy of Sciences (NAS) to convene an expert panel to review their appropriated annual economic surveys and recommend improved methodologies for conducting and processing them. The panel started work in July 2015 and the final report was released in May 2018 (Reengineering the Census Bureau's Annual Economic Surveys). From these recommendations, the Economic Directorate is conducting research towards the goal of harmonizing and simplifying the design and production process for these surveys and the Economic Census. Previous research in support of this survey has been conducted under the generic clearance for questionnaire pretesting research¹

Most recently, the Census Bureau has completed two multi-method pilots to evaluate the AIES instrument performance. The goal of the two pilots were to conduct further pilot testing of the harmonized annual survey for AIES through deployment of the updated instrument, followed by qualitative follow-up interviews and a short survey to understand response processes, gauge burden, and identify barriers to survey completion.

In September, the Census Bureau will conduct the final larger-scale test of the instrument – the 2022 AIES. Concurrent with that collection, Census Bureau researchers will conduct additional debriefing interviews with respondents. This request provides for the fielding of a Response Analysis Survey of respondents to the 2022 AIES.

Purpose:

Building on the success of the Phase I and Phase II pilots, the research laid out in this request is a continuation of the Response Analysis Survey methodology for collecting more information about respondent-reported actual and perceived burden. The results of this survey will be used to address research questions of the pilot including:

• Can respondents provide answers to the questions on the survey?

• Does the order and structure of the survey make sense?

• Compared to the current annual surveys, is the AIES overly burdensome?

Population of Interest:

The Response Analysis Survey will be sent to all respondents of the 2022 AIES. In previous iterations of the RAS, we note about a 33 percent response rate to the RAS request. We are sending the RAS to all respondents in expectation of similar performance as to gather enough response from enough types of businesses (size, industry, and others) to analyze results by these subgroups.

Methods:

Within seven days of completing the 2022 AIES survey, we will send respondents a Response Analysis Survey (RAS) titled “2022 Annual Integrated Economic Survey Respondent Feedback Survey,” and estimated to take no more than 10 minutes to complete. The survey will be hosted using Qualtrics online survey platform. At the beginning of the survey, respondents will be presented with the purpose of the RAS, the OMB clearance statement, and the privacy and Title 13 statement. Please see Appendix A for the survey instrument. One week later, we will send a reminder email to those who had not yet responded encouraging response to the RAS.

A RAS is a retrospective structured questionnaire that can “generate quantitative data about how respondents answered questions, about the records available for answering those questions, and about the real burden imposed by a survey” (Goldenburg 1993: 2). We have relied on a RAS at each of the iterations of this research because it is a methodology that lends itself to “identify[ing] response errors addressed in subsequent questionnaire design activities for the survey” (Phipps et al. 1995: 340) – a RAS can be administered throughout the iterations of a survey to track the impact of between round changes (O’Neill et al. 2004). It is administered quickly after survey completion to minimize recall bias (Mokovak and Stang 2012: 21).” Phipps et al. 1995 identify that while the RAS as a methodology does have limitations (including increased burden on respondents to the production survey, the risk of recall bias, and the small, nonrandom samples), the benefit is that it can provide information on actual and perceived burden that would otherwise be uncollected.

The AIES RAS will focus on real and perceived burden as well as response processes to better refine the survey instrument. See Attachment A for the RAS instrument. See Appendix B for the recruitment emails.

Timeline:

We anticipate responses to the 2022 AIES to begin on September 7, 2023, and the first RAS surveys to be deployed no later than September 14, 2023. Data collection will run concurrent with the 2022 AIES response through December 31, 2023.

Length of Interview:

We estimate that the survey will take an average of 10 minutes to complete. None of the RAS questions require consulting records or are required in order to submit a response. Given the performance of similarly structured surveys (like the Small Business Pulse Survey), we can anticipate this being a relatively short, low burden survey.

We expect to send one initial email and one follow-up reminder email to encourage response, each taking no more than five minutes to read. The follow-up reminder email will only be sent to nonresponding RAS companies.

Table 1: Burden Estimate for 2022 AIES Response Analysis Survey

Category of Response	Number of Respondents	Participation Time (in minutes)	Burden (in hours)
Initial email	3,000	5	250
Reminder email	2,100	5	175
Response Analysis Survey	1,000	10	167
Total			592

The AIES Phase II Pilot achieved a 35.7 percent response rate at the close-out of the research field period.² We expect, given the similar length of the field period between Phases II and III, a similar response rate for the close of the research field period in Phase III (8,500 x 33% ≈ 3,000). The RAS in Phase II was sent to all responding companies, and had a response rate of 33 percent. We expect that the Phase III RAS will perform similarly (3,000 x 33% = 1,000).

Language:

Because all correspondences and materials are English-only for the AIES Pilot Phase III, all correspondences and materials will be in English only for the RAS.

Incentives:

No monetary incentives are included in this research program.

Timeline:

Activities for this pilot research will run from September 2023 (contingent upon approval) through the delivery of findings in January 2024.

Table 2: Overall AIES 2022 RAS Schedule

Initial RAS deployment	September 21, 2023
Survey closeout	October 17, 2023
RAS close out	October 31, 2024
Findings due	December 1, 2024

Works Cited:

Goldenberg, Karen L., Shail Butani, and Polly A. Phipps. 1993. Response Analysis Surveys for Assessing Response Errors in Establishment Surveys. Washington, DC: Bureau of Labor Statistics.

Mockovak, William, and Sharon Stang. 2012. Using Iterative Pretesting and Response Analysis Surveys to Develop a Questionnaire for a Multi-Mode Survey. Washington, DC: Bureau of Labor Statistics.

O’Neill, Grace E., and Jessica R. Sincavage. 2004. Response Analysis Survey:  A Qualitative Look at Response and Nonresponse in the American Time Use Survey. Statistical Survey Paper. Bureau of Labor Statistics.

Phipps, Polly A., Shall J. Butani, and Young I. Chun. 1995. “Research on Establishment-Survey Questionnaire Design.” Journal of Business and Economic Statistics 13(3):337–46.

Contact:

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Melissa A. Cidade, Ph.D.

Survey Methodologist

Economic Management Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-8325

[email protected]

cc:
Nick Orsini (ADEP) with attachments

Lisa Donaldson (EMD) “ ”

Stephanie Studds (EWD) “ ”

Blynda Metcalf (ADEP “ “

William Davie (ESMD) “ ”

Thomas Smith (EMD) “ ”

Michelle Karlsson (EMD) “ ”

Jenna Morse (EMD) “ ”

Jasmine Luck (ADRM) “ ”

Aliea Yvonne Clark Fobia (ADRM) “ ”

Mary Lenaiyasa (PCO) “ ”

Danielle Norman (PCO) “ ”

1 Previous research projects approved through OMB Control Number 0607-0725 include:

• In-depth Exploratory Interviewing to Study Record-Keeping Practices, July 2019

• Respondent Debriefings for the Coordinated Contact Pilot Experiment, March 2020

• Cognitive Interviewing for the Content Harmonization and Collection Unit Determination Instrument, October 2020

• Non-respondent Debriefings for the Coordinated Contact Pilot Experiment, August 2021

• Cognitive Interviews for the Annual Integrated Economic Survey, Phase I, October 2021

• Cognitive Interviews for the Annual Integrated Economic Survey, Phase II, October 2021

2 Please note: the 35.7 percent response rate was as of April 17, 2023, the end of the research field period. We continued to encourage and collect response throughout the entirety of the legacy annual surveys’ field period. As of August 18, 2023, the AIES Pilot Phase II collection has a 58.8 percent response rate.