OMB Memo

Generic Information Collection Request:

Usability Evaluation for the 2022 Economic Census

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). The Census Bureau plans to conduct usability interviews for the online 2022 Economic Census instrument.

The Economic Census is a mandatory survey conducted by the Census Bureau every five years. The survey collects data electronically from nearly 4 million businesses (including large, medium and small companies representing all U.S. locations and industries) on a range of operational and performance questions. Data from the survey are used as the official five-year measure of American business and the economy. This survey is collected under the authority of Title 13, United States Code (U.S.C.), Section 131. Further information regarding the Economic Census can be found at this website: https://www.census.gov/programs-surveys/economic-census.html

Purpose: The Economic Census was first deployed using online data collection as the sole mode of collection in 2017. The upcoming 2022 Economic Census (EC22) provides an opportunity to further refine the functionality of the online platform. One of the areas targeted for improved usability and lowered cognitive burden are a series of features allowing firms to self-classify. To that end, usability interviews provide a window into the processes establishments use to assign themselves a North American Classification Code System (NAICS) code. Specifically, this research will assess an establishment’s ability to refine their NAICS designation using proposed design features in the online instrument. Historically, the Census Bureau has assigned an establishment’s NAICS code before deploying the corresponding Economic Census form. However, prior research, including debriefing interviews with 2017 Economic Census respondents as well as exploratory research into establishments’ reporting needs, has indicated that the pre-designated code can be misaligned. This can lead to questions that are not applicable to the respondent. Additionally, there is an attempt to lessen the number of Principal Business or Activity write-ins and nine-digit Principal Business Activity codes, so that we can get enough information about an establishment to provide the best survey to the respondent.

During the process of testing the aforementioned functionality of identifying one’s NAICS, the usability interview will also provide additional insight into other features of the online instrument, allowing us to:

• Evaluate the instrument’s performance in terms of efficiency, accuracy, and user satisfaction

• Identify areas of the instrument that are problematic for users

• Identify instructions/features that are difficult for users to understand

• Provide recommendations for improvements to the design of the instrument that will enhance its usability

Results from the interviews will be recorded and a report will be produced that outlines findings and provides recommendations for improvement.

Staff from the Data Collection Methodology & Research Branch within the Economic Statistical Methods Division (ESMD) of the Census Bureau will be conducting usability interviews for this testing, with support from staff from the Economy-wide Statistics Division (EWD), and the Economic Management Division (EMD). For this testing, we will interview up to 100 respondents.

Population of Interest: Large, medium and small companies from various U.S. locations and industries.

Language: Testing will be conducted in English only.

Method: Being sensitive to the current restrictions regarding social distancing throughout the United States, most of these interviews will be conducted via phone and video as allowed (e.g., Skype for Business). Virtual meetings with respondents may be conducted over Skype for Business. Participants will be sent a link to the online instrument via Skype and follow instructions on how to screen-share with researchers. Sessions will be recorded using Camtasia Studio to facilitate post hoc analysis. These recordings will capture the actions taken on the screen as well as audio (participants’ faces will not be seen or captured on recordings). When possible, in-person interviews may be included, but the majority of interviews are expected to happen remotely. Questionnaires including a Computer Use and Internet Experience (Attachment E), and Satisfaction (Attachment D) will be administered using Qualtrics via a survey link. For the purposes of this research, the interviewing questions and probes will be focused on the respondents’ interactions with the data collection instrument. During testing, Census Bureau researchers will provide respondents example screen shots or an interactive mock-up of the proposed designs to test how respondents interact with the interface. The interviews will follow a semi-structured interview protocol (Attachment A). Subject area specialists from the Census Bureau may observe some of the interviews.

Sample: Staff from EMD and EWD will provide DCMRB staff with a list of recent 2017 Economic Census and 2019 Annual Survey of Manufactures/Report of Organization respondents for recruiting. This listing will include contact information, a size indicator for the company, and a listing of their major industry classification.

We plan to conduct up to 100 interviews across two rounds of testing, with revisions in between rounds. We plan to conduct interviews with a variety of company sizes (small, medium, and large) and industry classification (i.e., services, wholesale, retail, etc.). Table 1 displays the targeted distribution for interviews.

Table 1. 2017 Economic Census Usability Interviews: Potential Respondent Distribution:

	Service	Wholesale	Retail	Manufacturing	Mining	Construction	TOTALS
Small	6	6	6	6	3	3	30
Medium	10	6	6	6	4	3	35
Large	10	6	6	6	4	3	35
TOTALS	26	18	18	18	11	9	100

This distribution was developed in collaboration with subject area specialists. This number of interviews was selected because it is a manageable number of interviews for the time period allotted, it should adequately cover target company sizes and classifications, and should be large enough to provide reactions to the instrument in order to identify meaningful findings.

Recruitment: We will contact potential participants via email and phone, explain the nature of our research, and ask them to participate in our study. The sample of participants will be those who are able to be contacted and who agree to participate in the study. Participants will be informed that the interview is voluntary and that the information they provide is confidential and will be seen only by Census Bureau employees involved in the research project

(See Attachment B).

We will not be providing monetary incentives to participants in this study. If an interview is scheduled, respondents will receive a confirmation email as well as a reminder email or phone call prior to the interview.

Enclosures: Below is a list of materials to be used in the current study:

1. Attachment A: Protocol used to outline how the research study will be conducted and tasks/probes that will be administered to asses features of the EC 2022 instrument included in the evaluation

2. Attachment B: PRA/Consent form to inform participant about privacy, confidentiality and burden associated with participation and obtain consent for participation and recording of the session (may be captured via electronic signature)

3. Attachment C: Instrument screenshots to illustrate features of the instrument included in the evaluation.

4. Attachment D: Satisfaction questionnaire to obtain subjective measure of respondents’ interaction with the instrument

5. Attachment E: Computer use and Internet experience to get an understanding of participants' experience using computers and the internet

Timeline: Testing will be conducted from July through September 2020

We expect that each interview will last no more than 60 minutes (100 cases x 60 minutes per case = 100 hours), including completing questionnaires. Additionally, to recruit respondents, we expect to make up to 5 phone contacts per completed case. The recruiting calls are expected to last on average 3 minutes per call (5 attempted phone calls per completed case x 100 cases x 3 minute per case = 25 hours). Thus, the estimated burden for the entirety of this project is approximately 125 hours (100 hours for interviews + 25 hours for recruiting).

Contact: The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Temika Holland

Data Collection Methodology & Research Branch

Economic Statistics and Methodology Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-5241

[email protected]

Cc:

Nick Orsini (ADEP) with enclosure

Carol Caldwell (ESMD) with enclosure

Diane Willimack (ESMD) with enclosure

Amy Anderson Riemer (ESMD) with enclosure

Temika Holland (ESMD) with enclosure

Melissa Cidade (ESMD) with enclosure

Lisa Donaldson (EMD) with enclosure

Kim Moore (EWD) with enclosure

William Samples (EWD) with enclosure

Theresa Riddle (EMD) with enclosure

Michelle Vile Karlsson (EMD) with enclosure

James Jamski (EMD) with enclosure

Jennifer Hunter Childs (ADRM) with enclosure

Jasmine Luck (ADRM) with enclosure

Danielle Norman (PCO) with enclosure

Mary Lenaiyasa (PCO) with enclosure

5