OMB Memo

Generic Information Collection Request:
Cognitive Interviewing for the Business Trends and Outlook Survey (BTOS)

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). We will be conducting cognitive interviews to test a proposed one-time supplement on the topic of Work from Home for the Business Trends and Outlook Survey (BTOS), along with an online survey with cognitive probes (and short debriefing interviews after for a subset of online respondents).

The Business Trends and Outlook Survey (BTOS) collects qualitative survey responses every two weeks from a representative sample of all single location, non-farm businesses in the United States, Puerto Rico, and Washington D.C. This high frequency program produces timely, granular data products that provide measures of business conditions and outlook and capture how extensive the impact on businesses events such as economic downturns or adverse weather events are. As economic and other conditions change, businesses are faced with new challenges and opportunities; if successfully tested, newly proposed content may help address data gaps for these emerging issues. For more information about the Business Trends and Outlook Survey, see: Business Trends and Outlook Survey (BTOS) (census.gov)

Purpose: While the pandemic propelled Work from Home (WFH), WFH continues post-pandemic to be an important mode of work. Just how prevalent is not clear since measures of WFH have varied across surveys and other collections (see for example, Abraham et al. 2024¹, Barrero et al. 2023², and Brynjolfsson et al 2023³). The Census Bureau has long collected information on WFH on demographic surveys but only recently began collecting this information on business surveys. The Census Bureau and an external team of experts from Stanford University and the Instituto Tecnologico Autonomo de Mexicoon propose adding a one-time supplement for fall 2024 4. In the last six months, did this business experience any monetary losses due to an extreme weather event (for example, hurricane, flood, drought, or heat wave)? • Yes • Noto the Business Trends and Outlook Survey as a way to measure WFH. The WFH BTOS questions were developed with some of the same team of experts who developed the original 2015 MOPS question and who have since gone on to develop the U.S. Survey of Working Arrangement and Attitidues (SWAA). Thus, the team of experts has experience with both demographic and business surveys concerning WFH. It is hoped that this data, combined with currently collected demographic and business data on WFH, will help to produce a more comprehensive and complete picture of WFH and WFH policies as businesses still work to find a new norm. Staff from the Census Bureau along with the external experts plan to write a working paper as well as discuss findings at National Bureau of Economic Research’s CRIW 2025 Conference on "The Changing Nature of Work." The planned research paper will provide results from the BTOS WFH collection at national, state, and large MSA level and by NAICS sectors (such as retail trade, manufacturing, and so on). The planned paper will also provide comparisons to results from the relevant WFH questions from the Annual Business Survey where possible. With this focus on businesses’ perspectives, we hope to be able to understand differences between the ability to work from home occasionally in a hybrid mode and fully remote work.

The purpose of cognitively testing newly proposed supplemental questions for the BTOS is to assess whether the questions are measuring the underlying constructs of interest and to better understand the accessibility of the requested data and the burden of compiling responses to the questions. The feedback from these interviews will be used to refine question wording and decide whether (or not) to include these new questions on Work from Home as a one-time supplement to the 2024 Business Trends and Outlook Survey.

A number of the core questions will be asked of participants, to add context for the Work from Home supplement that will come after all the core questions. We will do minimal probing on these core questions, though there are some questions where we would like to learn more about how participants are interpreting them, particularly the newish AI questions. Additionally, one new core question has been proposed by a data user and will be tested. This question asks about the type of extreme weather event that resulted in monetary losses (for those who selected Yes to the previous question).

Results: The results from the cognitive testing will be documented in a report, outlining the findings of the pretest and suggesting recommendations for improvement to questions. This report will include:

• Understanding how respondents comprehend the proposed Work from Home supplement questions

• Identifying respondents’ use of records and/or estimation strategies for answering these questions

• Assessing respondents’ ability to answer these questions

• Identifying difficulties in completing the questionnaire (where applicable)

• Recommended changes to questions, or the addition of new questions, and response options to be implemented in the BTOS Work from Home supplement

Population of Interest: Respondents from single and multi-unit businesses across all sectors.

Timeline: Testing will be conducted during June – August, 2024.

Language: Testing will be conducted in English only.

Method: The purpose of cognitively testing the proposed BTOS Work from Home supplement questions is to minimize measurement error and maximize the validity of these questions by assessing whether the questions accurately measure the underlying construct of interest. Cognitive interviewing is a method of pretesting instruments that involves in-depth interviewing, paying particular attention to the mental processes respondents use to respond to questions⁴. Cognitive interviewing uses a framework dependent on evaluating questions against their outcome objectives, including accurately eliciting the underlying construct of interest, and to what level of accuracy respondents can provide data in response.

Staff from the Data Collection and Methodology Branch plan to conduct up to 40 moderated interviews over Microsoft Teams/telephone, over two rounds of interviewing. The interviewers will follow a semi-structured interview protocol (Attachment A). Interviewers will send a link to the survey online to respondents prior to calling so that respondents can work through the survey while on the phone. The survey will be hosted through the Qualtrics online survey platform. Interviews will be recorded if the respondent consents.

In the first round, to test the new content, we will target 20 to 25 respondents of single and multi-unit businesses across all industries to complete a cognitive interview. The research staff will then review the resultant data and make adjustments to the new supplemental content questions and to the interviewing protocol if necessary. We will then conduct a second round of moderated cognitive interviews targeting 15 to 20 single and multi-unit business respondents, again across all industries.

In addition, in the second round of testing, we will collect up to 100 online survey responses with cognitive probes. We will send a self-administered version of the questions via Qualtrics, which will include a smaller number of cognitive probes similar to those used during telephone interviews to assess respondents’ interpretation and experiences with reporting to the survey items.

Sample: We plan to conduct a maximum of 40 moderated cognitive interviews total over two rounds of data collection, and 100 online unmoderated interviews in the second round using the same sample (described below).

This number of interviews is targeted because it is a manageable number of interviews for the time period allotted, it should adequately cover targeted businesses, and it should be large enough to provide reactions to the questions in order to identify meaningful findings.

The sampling universe for the BTOS cognitive testing sample consists of single-unit and multi-unit firms in the 2023 Annual Business Survey universe for which the Census Bureau has email addresses. Those companies that indicated they had employees who worked from home will be prioritized, though a small number of companies that did not indicate they had employees who worked from home will also be recruited to ensure that the supplement works well for those companies as well.

Recruitment: Participants will be recruited using a sample file from the 2023 Annual Business Survey universe. First, we will send an email to the contact. This email will include instructions for respondents to schedule an interview time and date. We will verify the appointment time, and respond by email to the respondent with confirmation of scheduling; in that email, we will also verify the best number to reach the respondent. About 30 minutes before their appointment time, we will email the respondent again reminding of the upcoming appointment and including a link to the survey. The first screen of the online survey will be a Paperwork Reduction Action (PRA) and Privacy Act (PA) statement, informing participants that their response is voluntary and that the information they provide is confidential under Title 13 and asking for consent to be interviewed. Respondents will need to click a checkbox indicating that they understand these rights and agree to be interviewed. If email recruitment is not meeting our recruitment goals, we will move to telephone calls.

Protocols: A copy of a draft interview protocol for the moderated interviews is enclosed.

Use of incentive: Monetary incentives for participation will not be offered.

Below is a list of materials to be used in the current study:

Attachment A: Draft instrument for the BTOS including the Work from Home Supplement

Attachment B: Draft protocol outlining intended questions to guide the moderated debriefings for the BTOS Work from Home Supplement

Attachment C: Consent form example, including PRA/PA statements

Burden Estimate: For moderated cognitive interviews, we expect that each interview will last no more than 45 minutes (40 cases x 45 minutes per case = 30 hours).

Additionally, to recruit respondents testing, we expect to make up to 3 email contacts per completed case. The recruiting emails are expected to take on average 3 minutes to read [(3 attempted emails per completed case x 40 cases x 3 minutes per case) / 60 minutes] = 6 hours).

Thus, the estimated burden for the moderated cognitive interviewsis 36 hours.

The online survey with cognitive probes will be administered to up to 100 respondents in the second round and will take approximately 20 minutes to complete. This results in a total burden of 33.34 hours.

The total burden for all research is 69.34 hours.

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Kristin Stettler

Data Collection Methodology and Research Branch

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-7596

[email protected]

cc:
Nick Orsini (ADEP) with enclosures

Kathryn Bonney (EMD) “ “

Catherine Buffington (EID)     “          ”

Lucia Foster (CES)     “          ”

Katie Genadek (ESMD)     “          ”

Temika Holland   (ESMD)     “          ”

Rebecca Keegan   (ESMD)     “          ”

Aleia Clark Fobia (ADRM)     “          ”

Jasmine Luck (ADRM)     “          ”

Danielle Norman (PCO)     “          ”

Mary Lenaiyasa (PCO)     “          ”

1 Abraham Katherine G, Mohammad Ashoori, Aref Darzi, Nathalie Gonzalez-Prieto, John C. Haltiwanger, Aliakbar Kabiri, and Erkut Y. Ozbay (2024). “Local Variation in Onsite Work during the Pandemic and its Aftermath,” NBER Working Paper 32042.

2 Barrero José María, Nicholas Bloom, and Steven J. Davis (2023). “The Evolution of Work from Home,” Journal of Economic Perspectives 37(4), 23–50.

3 Brynjolfsson, Erik, John J. Horton, Christos Makridis, Alexandre Mas, Adam Ozimek, Daniel Rock, and Hong-Yi TuYe (2023). “How Many Americans Work Remotely? A Survey of Surveys and Their Measurement Issues,” NBER Working Paper 31193.

4 Campanelli, P. 2007. “Methods for Testing Survey Instruments.” Short Course, Joint Program in Survey Methodology (JPSM). Arlington, VA.