Original OMB Memo

Generic Information Collection Request:
Usability Testing for the Annual Integrated Economic Survey

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). Researchers will be conducting usability testing with participants to understand whether respondents can navigate through and understand the proposed structure of the new Annual Integrated Economic Survey (AIES), and how this structure would be expected to affect procedures respondents use to gather and enter the data for it.

Background and Previous Research: The Census Bureau’s Economic Directorate asked the National Academy of Sciences (NAS) to convene an expert panel to review their appropriated annual economic surveys and recommend improved methodologies for conducting and processing them. The panel started work in July 2015 and the final report, with recommendations, was released in May 2018.

To address these recommendations, the Economic Directorate has been conducting research into harmonizing and simplifying the design and production process for these economic surveys and the Economic Census. The results of several years of this research have culminated into a single harmonized survey instrument designed to be administered as one survey cycle throughout the economy, regardless of company size, industry, or other characteristics.

The full suite of research conducted in support of the AIES development is documented in the recent OMB submission, Census Bureau Participant Debriefing Research for the 2023 Annual Integrated Economic Survey (under OMB clearance number 0607-0725).

Usability testing assessing the functionality of the prototype instrument occurred at the end of 2023 in support of the 2024 production survey launch. This protype represented the most up-to-date iteration of the AIES instrument and was tested with 28 participants. Since the conclusion of that testing, several changes have been incorporated into the survey website instrument. Further usability testing is required to examine the effect of these changes on the response process.

Purpose: Usability interviews will be conducted to assess functionality of the production instrument, by examining whether respondents can successfully complete tasks that are designed to mimic those they would need to complete when filling out the AIES. Researchers will assess whether the instrument is user friendly by assessing the ability of respondents to navigate through the instrument in an efficient way, and to test the respondents’ ability to successfully provide data to the AIES.

Objectives for the evaluation of the online AIES instrument include the following:

• Evaluate the instrument’s performance in terms of accuracy, and user satisfaction.

• Identify areas of the instrument that are problematic for users.

• Identify instructions/features that are difficult for users to understand.

• Evaluate the ability of respondents to complete the basic data collection steps.

• Understand how respondents navigate and use the spreadsheet.

• Identify if respondents demonstrate an understanding of the establishment versus industry reporting.

• Evaluate if respondents can access instrument support documents.

• Evaluate how respondents resolve errors.

• Evaluate if respondents understand how to submit their data.

• Provide recommendations for improvements to the design of the instrument that will enhance its usability.

Staff from ESMD’s Data Collection Methodology & Research Branch will be conducting the interviews for this testing.

Population of Interest: Respondents representing companies not selected for the initial mailout of the production AIES. Most respondents will represent “complex” multi-unit companies, whose operations span at least two industries. A small number of single unit companies will also be included in the testing.

Timeline: Testing will be conducted from May through July 2024.

Language: Testing will be conducted in English only.

Method: The method of research will be usability interviews. Usability testing is a method of testing the usability of an instrument by providing users with specific tasks designed to mimic the actions they would need to do when interacting with the instrument outside of the testing environment. The success or failure of the tasks allow researchers to assess the functionality, effectiveness of the instrument. For the purposes of this research, the usability tasks will be focused on the participant’s ability to complete basic tasks such as data entry, site navigation, and accessing or interacting with new features built into the instrument (e.g., auto-summing; optional reporting features). Researchers will also assess the flow of the instrument and examine whether any difficulties arise with reporting at various units (company; industry; establishment.

Most interviews will be completed in person. Travel is required for the interviews¹. These interviews will take place in-person at participants’ places of business or another secure private location of their choosing. The interviews will follow a semi-structured interview protocol (Attachment A) that includes a suite of tasks designed to assess the usability of the instrument by having respondents complete actions they would need to complete during actual data collection. Answers to any data entry tasks will be provided to the participant so the focus of the tasks remains on navigation and interacting with the website and not data retrieval.

The testing environment for this usability testing will be preloaded with the respondent’s live data. Specific data, such as establishment addresses and associated NAICS, will be prefilled and accurate, as it is in the actual AIES survey. No other substantive information, such as employee count or any financial data, will be prefilled. The answer key is provided in Attachment B. The interviews may be recorded (with consent), to facilitate summarization. Attachment C contains screenshots of the production instrument that the participants are evaluating.

Sample: We plan to conduct a maximum of 30 interviews with a variety of sizes and types (i.e., industries) of businesses. Usability testing will involve recruiting primarily larger establishments, which will be most affected by the new collection method, followed by a smaller sample of medium and small sized establishments, as determined by annual revenue. A portion of the larger companies will represent those whose business activities span multiple industries. This industry complexity affects the front-end design of the instrument and is an important aspect of the testing. The sample size necessary for this test was determined by qualitative research experience. This sample will yield a suitable, broad representation of U.S. businesses for this testing, and it should be large enough to provide reactions to the questions in order to identify meaningful findings.

Recruitment: Participants will be recruited using a sample file which was developed using the AIES sample frame, created from the Census Bureau's Business register. The recruitment sample will not include those who are selected to participate in the initial mailout of the AIES survey launch. Before beginning the interviews, we will provide participants a consent form (Attachment D), informing them that their response is voluntary and that the information they provide is confidential under Title 13.

Protocol: A copy of the interview protocol including the participant tasks is attached (Attachment A). The answer key for the tasks is in Attachment B.

Use of incentive: Monetary incentives for participation will not be offered.

Length of interview: For the usability interviews, we expect that each interview will last no more than 60 minutes (30 cases x 60 minutes per case = 30 hours). Additionally, to recruit participants we expect to reach out via email up to 2 times per completed case. The emails are expected to take on average 2 minutes to read and act upon (2 email contacts per completed case x 30 cases x 2 minutes per case = 2 hours). Thus, the estimated burden for this project is 32 hours (30 hours for interviews + 2 hours for recruiting).

Below is a list of supporting documents referenced herein:

1. Attachment A: Protocol used to outline how the research study will be conducted

2. Attachment B: Answer Key for respondent data entry tasks

3. Attachment C: Screenshots of the production instrument and reporting spreadsheet.

4. Attachment D: Consent Form to obtain participant consent for participation and recording of the cognitive interview session

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Rebecca Keegan

Data Collection Methodology and Research Branch

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-6003

[email protected]

cc:
Nick Orsini (ADEP)

Stephanie Studds (ADEP)

Blynda Metcalf (ADEP)

Amy Anderson Riemer (ESMD)

Lisa Donaldson (EMD)

Shelley Karlsson (EMD)

Melissa Cidade (EMD)

Edward Watkins (EWD)

Aleia Clark Fobia (ADRM)

Jasmine Luck (ADRM)

1 Testing locations to be determined.

4