OMB Memo

Generic Information Collection Request:

Cognitive Interviews for the Annual Survey of Manufactures Unfilled Orders Question

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725) for the Annual Survey of Manufactures (ASM). These activities will include cognitive interviews with respondents to the 2019 ASM to learn whether these respondents are able to answer a proposed new question about unfilled orders without significantly increasing respondent burden. Currently, data about unfilled orders is collected on the M3UFO survey.

The Census Bureau has conducted the Annual Survey of Manufactures (ASM) since 1949 to provide key measures of manufacturing activity during intercensal periods. In economic census years ending in “2” and “7”, we do not conduct the ASM.  The ASM furnishes up-to-date estimates of employment and payroll, hours and wages of production workers, value added by manufacture, cost of materials, value of shipments by the North American Product Classification System (NAPCS) product codes, inventories, and expenditures for both plant equipment and structures. The survey provides data at the two-through six-digit North American Industry Classification System (NAICS) levels.  It also provides geographic data by state at a more aggregated industry level.

 

The survey is an integral part of the Government’s statistical program. Its results provide a factual background for decision making by the executive and legislative branches of the Federal Government.  Federal agencies use the annual survey’s input and output data as benchmarks for their statistical programs, including the Federal Reserve Board’s Index of Industrial Production and the Bureau of Economic Analysis’ (BEA) estimates of the gross domestic product. The data also provide the Department of Energy with primary information on the use of energy by the manufacturing sector to produce manufactured products. These data also are used as benchmark data for the Manufacturing Energy Consumption Survey, which is conducted for the Department of Energy by the Census Bureau.  Within the Census Bureau, the ASM data are used to benchmark and reconcile monthly and quarterly data on manufacturing production and inventories.  The ASM is the only source of complete establishment statistics for the programs mentioned above.  The survey also provides valuable information to private companies, research organizations, and trade associations.  Industry makes extensive use of the annual figures on product class shipments at the U.S. level in its market analysis, product planning, and investment planning.  State development/planning agencies rely on the survey as a major source of comprehensive economic data for policymaking, planning, and administration. 

Further information regarding the Annual Survey of Manufactures can be found at this website: https://www.census.gov/programs-surveys/asm.html.

The Manufacturers’ Shipments, Inventories, and Orders (M3) survey collects monthly data on shipments, inventories, new orders, and unfilled orders from manufacturing companies.  The orders, as well as the shipments and inventories data, are widely used and are valuable tools for analysts of business cycle conditions, including members of the Council of Economic Advisers, Bureau of Economic Analysis, Federal Reserve Board, Conference Board, and the business community.  

             

New orders serve as an indicator of future production commitments and the data are direct inputs into the leading economic indicator series.  New orders are derived by adding shipments to the net change in the unfilled orders from the previous month. The ratio of unfilled orders to shipments is an important indicator of pressure on manufacturing capacity. 

 

The monthly M3 estimates are based on a relatively small panel of domestic manufacturers and reflect primarily the month-to-month changes of large companies.  There is a clear need for periodic benchmarking of the M3 estimates to reflect the manufacturing universe.  The Economic Census, which covers the entire manufacturing sector, and the Annual Survey of Manufactures (ASM) provide annual benchmarks for the shipments and inventories data in the monthly M3 survey.  The Manufacturers’ Unfilled Orders Survey (M3UFO), the subject of this request, provides the annual benchmarks for the unfilled orders data.

 

The industries selected for the M3UFO survey are those which the U.S. Census Bureau determined to have considerable unfilled orders.  The survey is necessary to ensure future accuracy of the unfilled orders and new orders data in the M3 survey and to determine which North American Industry Classification System (NAICS) industries continue to maintain unfilled orders.  

 

The Census Bureau conducts this survey under the authority of an Act of Congress, Title 13, U.S.C., Sections 131 and 182. Section 131 defines the Secretary of Commerce’s (Secretary) authority to conduct, compile, and publish the Economic Census.  Section 182 provides authority for the Secretary to conduct surveys deemed necessary to furnish annual and other interim current data on the subjects covered by the Census. This collection is made mandatory under the provisions of Title 13 U.S. C., Sections 224 and 225. 

 

Report forms are mailed to approximately 6,000 companies requesting data for 42 of the M3 Survey’s 92 NAICS defense and nondefense industry categories.  

 

Further information regarding the M3UFO can be found at this website: https://www.census.gov/programs-surveys/m3ufo.html.

If it is determined that respondents are able to provide unfilled order data at the establishment level, Census plans to move the unfilled order content to the ASM.  By collecting this content on the ASM, Census would eliminate the M3UFO survey.  If Census is able to eliminate the M3UFO survey, there would be a significate savings in resources related to sample design, testing, processing, and ratio analysis.  Respondent burden is expected to be reduced since companies would receive one survey, instead of two separate surveys.  Rolling the M3UFO content into the ASM reduces burden, simplifies collection, and provides program alignment.

Purpose: The purpose of this evaluation is to explore whether respondents to the ASM understand a proposed question about unfilled orders and are able to provide the data.

Additionally researchers want to explore how respondents anticipate gathering the relevant information, and what burden may be associated with that retrieval process.

The results from these cognitive interviews will be consolidated into a report that outlines the findings and suggestions for next steps.

Population of Interest: U.S. manufacturing companies, both single and multi-units, across a variety of sizes, industries, and locations, focusing on those industries that are likely to have unfilled orders.

Timeline: Testing will begin in October 2020 and will likely conclude by March 2021.

Language: Testing will be conducted in English only.

Method: We plan to conduct a combination of phone and in-person¹ cognitive interviews with up to 40 respondents, over two rounds. We will be asking cognitive questions which have a broader focus on the respondents’ understanding and familiarity with the concepts and questions being asked, how respondents expect to retrieve the information, respondent’s perception of difficulty of the question and the what level of burden might be associated with data retrieval. In-person interviews will take place at the respondent’s location. The locations for each round have not yet been selected, but they will be determined based on where we expect to find companies who are most likely to have HAZMAT transportation. We will request that the cognitive interviews be audio recorded with the participants’ permission (see attachment B), to aid researchers in accurately summarizing key findings from the interviews. The cognitive interviews will follow a semi-structured interview protocol (Attachment A).

COVID-19 Contingency Methodology: Due to the Coronavirus, it may be impossible to meet with respondents in person. Virtual meetings with respondents may be conducted over Skype for Business. Participants will be sent a link to the online instrument via Skype and follow instructions on how to screen-share with researchers. Sessions will be recorded using Camtasia Studio to facilitate post hoc analysis. These recordings will capture the actions taken on the screen as well as audio (participants’ faces will not be seen or captured on recordings).

Sample: We plan to conduct a maximum of 40 interviews across two rounds of testing (20-25 in the first round, 10-15 in the second round). This number of interviews was selected because it is a manageable number of interviews for the time period allotted, and should be large enough to provide reactions to the questions and definitions that are representative of the survey population. We plan to conduct interviews with a variety of sizes and types (i.e., industries) of manufacturing establishments. Efforts will be made to sample respondents across industries representative of companies that participated in the ASM and with an emphasis on those that might be in multiple industry categories. The sampling frame for these interviews comes from a list of establishments that participated in the 2019 ASM.

Recruitment:

We will contact potential interviewees via phone or email, explain the nature of our research, and ask them to participate in our study. The sample of participants will be those who are able to be contacted and who agree to participate in the study. Participants will be informed that their response is voluntary and that the information they provide is confidential and will be seen only by Census Bureau employees involved in the research project. We will not be providing monetary incentives to participants in this study. Once interviews are scheduled, researchers will send respondents a confirmation via email.

Protocol: The draft protocol for the study is enclosed (see Attachment A). We anticipate that each interview will take 30 minutes to complete.

Use of Incentive: Monetary incentives for participation will not be offered.

Below is a list of materials to be used in the current study:

1. Attachment A: Protocol used to outline how the research study will be conducted

2. Attachment B: Consent form to obtain participant consent for participation and recording of the session

3. Attachment C: Draft question mockup and instructions

4. Attachment D: 2019 ASM survey and 2019 M3 UFO survey to illustrate the current surveys

5. Attachment E: Manufacturing activities list for respondents in multiple industries to review

Length of interview: For the cognitive testing, we expect that each interview will last no more than 30 minutes (40 cases x 30 minutes per case = 20 hours). Additionally, to recruit respondents we expect to make up to 5 phone contacts per completed case. The recruiting calls are expected to last on average 3 minutes per call (5 attempted phone calls per completed case x 40 cases x 3 minute per case = 10 hours). Thus, the estimated burden for this project is 30 hours (20 hours for interviews + 10 hours for recruiting).

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Kristin Stettler

Data Collection Methodology & Research Branch

Economic Statistics and Methodology Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-7596

[email protected]

Cc:

Nick Orsini (ADEP) with enclosure

Carol Caldwell (ESMD) with enclosure

Diane Willimack (ESMD) with enclosure

Amy Anderson Riemer (ESMD) with enclosure

Kristin Stettler (ESMD) with enclosure

Amy Newman Smith (ESMD) with enclosure

Blynda Metcalf (EWD) with enclosure

Marlo Thornton (EWD) with enclosure

Kim Moore (EWD) with enclosure

Carol Aristone (EID) with enclosure

Jennifer Hunter Childs (ADRM) with enclosure

Jasmine Luck (ADRM) with enclosure

Danielle Norman (PCO) with enclosure

Mary Lenaiyasa (PCO) with enclosure

1 Testing locations to be determined.