OMB Memo

Generic Information Collection Request:

Usability Evaluation of the Commodity Flow Survey Portal

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725) for the Commodity Flow Survey (CFS). These activities will include a usability evaluation of the new portal based alternative reporting option via interviews with respondents to the 2017 CFS.

The CFS is administered every 5 years through a partnership between the U.S. Census Bureau and the Bureau of Transportation Statistics (BTS). It provides data on the movement of goods in the United States including commodities shipped, their value, weight, and mode of transportation, as well as the origin and destination of shipments of commodities from manufacturing, mining, wholesale, and selected retail and services establishments. These data are used by policy makers and transportation planners to evaluate the demand for transportation facilities and services, energy use, and safety risk and environmental concerns. Additionally, the data collected from CFS are used to analyze trends in the movement of goods, mapping spatial patterns of commodity and vehicle flows, forecasting demands for the movement of goods, and determining needs for associated infrastructure and equipment.

Respondents to the CFS have traditionally been asked to provide a sample of their shipments four separate times during the reporting year, which occurred every 5 years. Some respondents to the CFS reported this process of creating a sample to be burdensome, and expressed interest in altering this method of data collection. Census Bureau staff created a secure depository (i.e., portal) for respondents to upload large amounts of shipment data from their records on a semi-continuous/automatic basis, thus eliminating the need for respondents to create a quarterly sample. Small scale participation in a trial of this new method of ‘passive data collection’ was found to be successful for some companies, and had promising implications for companies of similar attributes (i.e., size, employee count, resources) who could potentially participate as well.

Anticipated data collection will entail respondents reporting one time, on a yearly basis.

Researchers conducted early stage scoping interviews to assess the feasibility of wide scale implementation of this data collection change and its potential effects on respondent burden. An additional goal was to determine whether the CFS can successfully move to an annual collection, as opposed to one that occurs every 5 years.

Preliminary testing revealed an ability and willingness to provide larger amounts of shipment data, among respondents who had shipping data electronically available. For these respondents, this new method would often reduce burden. Similarly, a move to an annual collection was generally well received, also by those respondents who had good electronic record systems.

Further information regarding the Commodity Flow Survey can be found at this website: https://www.census.gov/econ/cfs/ .

Purpose: The purpose of this evaluation is to test the usability of the new collection instrument by assessing the ability of respondents to navigate through the instrument in an efficient way and to test the respondents’ ability to successfully provide data to the Commodity Flow Survey via this new data transfer process.

Pretesting will help inform user centered design ensuring that key functions are captured within the instrument, making it easy for respondents to understand and report. Additional objectives for the evaluation of the online 2022 Commodity Flow instrument include the following:

• Evaluate the instrument’s performance in terms of efficiency, accuracy, and user satisfaction

• Identify areas of the instrument that are problematic for users

• Identify instructions/features that are difficult for users to understand

• Provide recommendations for improvements to the design of the instrument that will enhance its usability

The results from the usability evaluation interviews will be recorded and a report will be produced that outlines the findings and recommendations for improvement.

Population of Interest: U.S. companies that ship goods, across a variety of sizes, industries, and locations.

Timeline: Testing will begin in May 2020, concluding in August 2020.

Language: Testing will be conducted in English only.

Method: The method of research will be usability interviews for the online platform. Travel is required for the interviews¹. These interviews will take place in-person at participants’ places of business or another location of their choosing. The interviews will follow a semi-structured interview protocol (Attachment A). The interviews may be recorded (with consent), to facilitate summarization.

Eye tracking will be used to assess participants’ visual interaction with the online platform during the usability evaluations. Eye tracking will additionally facilitate post-hoc analysis of the instrument. A consent form will be used to obtain participants’ permission for the use of eye tracking and audio recording.

COVID-19 Contingency Methodology: Due to the Coronavirus, it may be impossible to meet with respondents in person. Virtual meetings with respondents may be conducted over Skype for Business. Participants will be sent a link to the online portal via Skype and follow instructions on how to screen-share with researchers. Sessions will be recorded using Camtasia Studio to facilitate post hoc analysis. These recordings will capture the actions taken on the screen as well as audio (participants’ faces will not be seen or captured on recordings).

Sample: We plan to conduct a total of 40 interviews over 2 rounds of testing, with revisions to the instrument after the first round. The sample size necessary for this test was determined by qualitative research experience. We plan to conduct interviews with a variety of sizes and types (i.e., industries) of business establishments. The sampling frame for these interviews comes from a list of companies that were sampled to participate in the 2017 CFS. Participating companies will most likely have submitted responses to the CFS survey, but not necessarily. We will focus on the small number of companies that used spreadsheet reporting in 2017, but will also meet with other companies to get their input as well.

Recruitment: Respondents will be recruited via email and phone. Respondents will be informed that their participation is voluntary.

Protocol: The protocol for the study is enclosed (see Attachment A). We anticipate that each interview will take 60 minutes to complete.

Use of Incentive: Monetary incentives for participation will not be offered.

Length of interview: For the usability interviews, we expect that each interview will last no more than 60 minutes (40 cases x 60 minutes per case = 40 hours). Additionally, to recruit respondents we expect to make up to 5 phone contacts per completed case. The recruiting calls are expected to last on average 3 minutes per call (5 attempted phone calls per completed case x 40 cases x 3 minute per case = 10 hours). Thus, the estimated burden for this project is 50 hours (40 hours for interviews + 10 hours for recruiting).

Below is a list of materials to be used in the current study:

1. Attachment A: Protocol that will be used during the interviews

2. Attachment B: User Tasks participants will complete to navigate and use the site

3. Attachment C: Demographic questionnaire to obtain participant characteristics and Computer use and Internet experience questionnaire to get an understanding of participants' experience using computers and the internet

4. Attachment D: Satisfaction questionnaire to allow for a subjective satisfaction measure to be gathered based on the participants’ experience with the online form

5. Attachment E: Screenshots of the Commodity Flow instrument that will be tested (instrument still in development)

6. Attachment F: Consent forms for respondents to sign or verbally agree to

7. Attachment G: 2017 CFS survey to illustrate the survey being evaluated

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Rebecca Keegan

Data Collection Methodology & Research Branch

Economic Statistics and Methodology Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-6003

[email protected]

Cc:

Nick Orsini (ADEP) with enclosure

Carol Caldwell (ESMD) with enclosure

Diane Willimack (ESMD) with enclosure

Amy Anderson Riemer (ESMD) with enclosure

Rebecca Keegan (ESMD) with enclosure

Kristin Stettler (ESMD) with enclosure

Demetria Hanna (ESMD) with enclosure

James Hinckley (ERSD) with enclosure

Christian Moscardi (ERSD) with enclosure

Julie Parker (BTS) with enclosure

Cha-Chi Fan (BTS) with enclosure

Jennifer Hunter Childs (ADRM) with enclosure

Jasmine Luck (ADRM) with enclosure

Danielle Norman (PCO) with enclosure

Mary Lenaiyasa (PCO) with enclosure

1 Testing locations to be determined.

4