OMB Memo

Generic Information Collection Request:
Cognitive Testing for the Annual Integrated Economic Survey

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). Researchers will be conducting cognitive testing with participants to assess the respondent communications for the new Annual Integrated Economic Survey (AIES), and how these communications would be expected to affect respondents’ understanding of the AIES and the process for completing it.

Background and Previous Research: The Census Bureau’s Economic Directorate asked the National Academy of Sciences (NAS) to convene an expert panel to review their appropriated annual economic surveys and recommend improved methodologies for conducting and processing them. The panel started work in July 2015 and the final report, with recommendations, was released in May 2018. To address these recommendations, the Economic Directorate has been conducting research into harmonizing and simplifying the design and production process for these surveys and the Economic Census.

This research program began with a series of focus groups and interviews to determine the record-keeping practices of medium-sized multi-unit companies, beginning in July 2019. The findings from this research informed a further exploration of the accessibility of various data by topic (including revenue, expenditures, and others) by unit of analysis (including company, establishment, state, and others), beginning in October 2020.

At the same time, survey operations staff in the Economic Directorate began the process of consolidating contact information for the largest firms, leading to a multi-year pilot focused on coordinated contact for the standing annual surveys. The first two rounds of contact consolidation were supported by two rounds of debriefing interviews: the first, beginning in March 2020, examined the impact of the contact strategies used in the pilot, while the second, beginning in August 2021, focused on non-respondents and barriers to completion.

The results of all this research have been incorporated to compile a single harmonized survey instrument designed to be administered as one survey cycle throughout the economy, regardless of firm size, industry, or other characteristics.

In December 2021, we received approval to conduct the first phase of pilot research in support of the new instrument and have additional requests for approval to conduct other research for this survey throughout 2023.

In total, six rounds of respondent research have been conducted to support the development and design of the AIES product.¹

For more information about the AIES: https://www.census.gov/programs-surveys/aies.html

Purpose: Cognitive interviews will be conducted to assess the content of AIES letters, emails, and other potential modes of communication. Researchers will assess whether respondents comprehend the survey communications and the effectiveness of those communications.

Objectives for the evaluation of the online AIES communications include the following:

• How do respondents comprehend our communications?

• Do respondents understand that the AIES is mandatory?

• Do respondents understand that the AIES is replacing some annual surveys?

• Do respondents understand how to access their surveys based on instructions embedded in the communications?

• Do respondents have preferences regarding the mode of communications?

• How are respondents currently receiving our communications?

• What component of the messaging, if any, motivates respondents to start the survey?

Population of Interest: Respondents representing companies selected in at least two of the following surveys: Annual Retail Trade Survey, Annual Wholesale Trade Survey, Services Annual Survey, Annual Survey of Manufacturers, and/or Annual Capital Expenditures Survey. Respondents participating in the communications cognitive testing will not have taken part in previous AIES pilot studies.

Timeline: Testing will be conducted over three rounds from May through June 2023.

Language: Testing will be conducted in English only.

Method: We will conduct cognitive interviews with up to 30 respondents in order to evaluate and refine the AIES communications. These interviews will be conducted via Microsoft Teams. Drafts of the target communications will be sent to scheduled respondents prior to the interviews (see Attachment B). Interviews will be conducted by researchers from the Data Collection Methodology and Research Branch. Other Economic Directorate staff within the Census Bureau may observe if available. We will use probes (see Attachment A) to assess respondents’ understanding of the AIES communications.

All participants will be informed that their response is voluntary and that the information they provide is confidential and will be seen only by Census Bureau employees (see Attachment C).

Sample: We plan to conduct a maximum of 30 interviews with a variety of sizes and types (i.e., industries) of businesses. The sample size necessary for this test was determined by qualitative research experience. This sample will yield a suitable, broad representation of U.S. businesses for this cognitive testing, and it should be large enough to provide reactions to the communications to identify meaningful findings.

Recruitment: Participants will be recruited using the sample file from the coordinated collection sample frame which included respondents representing companies selected in at least two of the following surveys: ARTS, AWTS, SAS, ASM, and ACES. The recruitment sample will not include those who participated in the pilot testing. Before beginning the interviews, we will provide participants a consent form (Attachment C, informing them that their response is voluntary and that the information they provide is confidential under Title 13.

Protocol: A copy of a draft interview protocol is attached (Attachment A).

Use of incentive: Monetary incentives for participation will not be offered.

Length of interview: For the cognitive interviews, we expect that each interview will last no more than 60 minutes (30 cases x 60 minutes per case = 30 hours). Additionally, to recruit respondents we expect to reach out via email and to make up to 3 phone contacts per completed case. The recruiting calls are expected to last on average 3 minutes per call (3 attempts per phone call per completed case x 30 cases x 3 minutes per case = 4.5 hours). 

Thus, the estimated burden for this project is 34.5 hours (30 hours for interviews + 4.5 hours for recruiting).

Below is a list of supporting documents referenced herein:

1. Attachment A: Protocol used to outline how the research study will be conducted

2. Attachment B: Draft Communications

3. Attachment C: Consent Form to obtain participant consent for participation and recording of the cognitive interview session

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Aryn Hernandez

Data Collection Methodology and Research Branch

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-7982

[email protected]

cc:
Nick Orsini (ADEP)

Amy Anderson Riemer (ESMD) “ ”

Lisa Donaldson (EMD)

Jenna Morse (EMD)

James Burton (EMD)

Kimberly Moore (ESMD)

Blynda Metcalf (ADEP)

Melissa Cidade (EMD)

1 The full suite of respondent research approved through OMB Control Number 0607-0725 is listed below:

• In-depth Exploratory Interviewing to Study Record-Keeping Practices, July 2019

• Respondent Debriefings for the Coordinated Contact Pilot Experiment, March 2020

• Cognitive Interviewing for the Content Harmonization and Collection Unit Determination Instrument, October 2020

• Non-respondent Debriefings for the Coordinated Contact Pilot Experiment, August 2021

• Cognitive Interviews for the Annual Integrated Economic Survey, Phase I, October 2021

• Cognitive Interviews for the Annual Integrated Economic Survey, Phase II, October 2021

3