Memo

Generic Information Collection Request:

Instrument Debriefings for the 2017 NAPCS

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). The Census Bureau plans to conduct respondent debriefings for the North American Product Classification (NAPCS) survey items on the online 2017 Economic Census instrument.

The Economic Census is a mandatory survey conducted by the Census Bureau every five years under the authority of Title 13, United States Code (U.S.C.), Section 131. The survey collects data electronically from nearly 4 million establishments (including large, medium and small companies representing all U.S. locations and industries) on a range of operational and financial questions. Data from the survey are used as the official five-year measure of American business and the economy. As part of the Economic Census, the NAPCS section of this survey collects detailed information about the revenue that businesses generated by goods and services they provide. Further information regarding the Economic Census can be found at this website: https://www.census.gov/programs-surveys/economic-census.html. For more information about NAPCS, please visit https://www.census.gov/eos/www/napcs/ .

Purpose: Historically, revenue details were requested mainly for goods and services associated with an establishments’ primary North American Classification System (NAICS) thus focusing primarily on where goods were produced. However, beginning with the 2017 Economic Census, the Census Bureau introduced a new way of collecting and disseminating the product revenue information through the NAPCS framework. The focus of this framework is on the demand-side economy, providing a summary of where goods and services are sold, just in the primary industry in which they are produced.

Under the NAPCS framework, businesses were asked to explicitly report detailed information on all revenue generating goods and services they offer, even if those goods and services are not typical of that business’ industry. With the NAPCS structure, although some goods and services might make up a fraction of the business’ revenue, respondents should still report these products, providing a description of them and their associated revenue.

The NAPCS survey item in the 2017 Economic Census first presents respondents with a list of products typical to their industry. After selecting products, respondents are then asked to report revenues associated with each of the selected products on a subsequent screen. The survey includes additional open-ended response fields where the respondent can write-in other revenue generated goods or services that generates revenue that was not pre-listed. This two-stage design takes advantage of automation offered by electronic delivery of the instrument.

However, this design differs substantively from legacy paper questionnaires where a full list of relevant products and response calls for revenue values were presented together on the same page. While the two-stage electronic design, reflect the cognitive response process observed during pretesting it never the less represents a different reporting strategy from previously and therefore warrants and evaluation to assess its effectiveness.

Data gathered from debriefings will help inform user-centered design decisions for the future and ensure that the NAPCS is easy for respondents to report. Additional objectives outlined for the evaluation of the online NAPCS survey item in the 2017 Economic Census instrument include the following:

• Evaluate the performance of NAPCS in terms of item non response attributed to its design (e.g. ordering), functionality (e.g., two stage approach), and ease of use

• Identify terms (e.g., products, services) that are difficult for respondents to understand and/or are not comparable to records

• Provide recommendations for improvements to the design of the NAPCS survey item that can potentially enhance data quality and respondents’ reporting experience

Results from the debriefings will be recorded and a report will be produced that outlines findings and provides recommendations for improvement.

Staff from the Data Collection Methodology & Research Branch within the Economic Statistical Methods Division (ESMD) of the Census Bureau will be conducting debriefing interviews for this testing, with support from staff from the Economy-wide Statistics Division (EWD), and the Economic Management Division (EMD). For this testing, we will interview up to 100 respondents.

Population of Interest: Large, medium and small companies from various U.S. locations and industries.

Language: Testing will be conducted in English only.

Method: The method of research will be telephone debriefing interviews, which are interviews aimed at understanding how a respondent recently reported to a survey. For the purposes of this research, the debriefing questions will be focused on the respondents’ interactions with the data collection instrument. All interviews will be conducted over the telephone. The goal of this research is to contact respondents within 2-3 weeks of their survey submission. Respondents will be asked to log back into the survey instrument and navigate to Item 22 (using the Go To functionality to jump to screen in the instrument). This will allow them to see the survey item formatted for their industry as well as an account of their responses. Alternatively, respondents will be sent example screen shots during the interview as an aid in helping them remember the web instrument. The interviews will follow a semi-structured interview protocol (attached). Subject area specialists from the Census Bureau will participate in some of the debriefing interviews in order to listen to the interview.

Sample: Staff from EMD and EWD will provide DCMRB staff with a list of recent 2017 Economic Census respondents for recruiting. This listing will include contact information, a size indicator for the company, and a listing of their major industry classification. We will also attempt to obtain responses provided to the NAPCS survey item to reference during the debriefings.

We plan to conduct up to 100 interviews. We plan to conduct interviews with a variety of company sizes (small, medium, and large) and industry classification (i.e., services, wholesale, retail, manufacturing, mining, and construction). Table 1 displays the targeted distribution for interviews.

Table 1. 2017 Economic Census NAPCS Debriefings: Potential Respondent Distribution:

	Service	Wholesale	Retail	Manufacturing	Mining	Construction	TOTALS
Small	6	6	6	6	5	3	32
Medium	10	6	6	7	3	3	35
Large	10	6	6	7	1	3	33
TOTALS	26	18	18	20	9	9	100

This distribution was developed in collaboration with subject area specialists. This number of interviews was selected because it is a manageable number of interviews for the time period allotted. This number should adequately cover target company sizes and classifications, and should be large enough to provide reactions to the instrument in order to identify meaningful findings.

Recruitment: We will contact potential participants who reported to the 2017 Economic Census. The resulting sample of participants will be those who are able to be contacted and who agree to participate in the study. Participants will be informed about the purpose of the study, the confidentiality of their responses, and the voluntary nature of their participation.

We will not be providing monetary incentives to participants in this study. Once an interview is scheduled, respondents will receive a confirmation email as well as a reminder email or phone call prior to the interview if the interview is scheduled for a later time.

Enclosures: Below is a list of materials to be used in the current study:

1. Attachment A: Protocol

2. Attachment B: NAPCS screenshots to illustrate the design of the NAPCS survey items being evaluated. This may be e-mailed to the participant after they agree to participate in the study.

Timeline: Testing will be conducted from October through December 2018.

For the debriefing phone calls, we expect that each interview will last no more than 30 minutes (100 cases x 30 minutes per case = 50 hours). Additionally, to recruit respondents, we expect to make up to 5 phone contacts per completed case. The recruiting calls are expected to last on average 3 minutes per call (5 attempted phone calls per completed case x 100 cases x 3 minute per case = 25 hours). Thus, the estimated burden for the entirety of this project is approximately 75 hours (50 hours for interviews + 25 hours for recruiting).

Contact: The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Temika Holland

Data Collection Methodology & Research Branch

Economic Statistics and Methodology Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-5241

[email protected]

Cc:

Nick Orsini (ADEP) with enclosure

Carol Caldwell (ESMD) with enclosure

Carma Hogue (ESMD) with enclosure

Diane Willimack (ESMD) with enclosure

Amy Anderson Riemer (ESMD) with enclosure

Temika Holland (ESMD) with enclosure

Lisa Donaldson (EMD) with enclosure

Kim Moore (EWD) with enclosure

William Samples (EWD) with enclosure

Theresa Riddle (EMD) with enclosure

Michelle Vile Karlson (EMD) with enclosure

James Jamski (EMD) with enclosure

Jennifer Hunter Childs (ADRM) with enclosure

Jasmine Luck (ADRM) with enclosure

Danielle Norman (PCO) with enclosure

Mary Lenaiyasa (PCO) with enclosure

5