OMB Memo

Generic Information Collection Request:

Usability evaluation for the 2017 Economic Census

Request: The Census Bureau plans to conduct additional research under the generic clearance for questionnaire pretesting research (OMB number 0607-0725). The Census Bureau plans to conduct pretesting activities for the online 2017 Economic Census instrument. These activities will include debriefings with respondents to the 2016 Annual Survey of Manufactures and Report of Organization, as well as a usability evaluation with the online 2017 Economic Census instrument itself.

The Annual Survey of Manufactures provides data about manufacturing that are crucial to accurate estimates of domestic production and productivity, and to making sound decisions on economic trade policies. Further information regarding the Annual Survey of Manufactures can be found at this website: https://www.census.gov/programs-surveys/asm.html. The Report of Organization is used to update the Census Bureau's list of businesses and provides key source data for County Business Patterns and other statistical series. Further information regarding the Report of Organization can be found at this website: https://www.census.gov/econ/overview/mu0700.html. Both of these surveys are used to update the Census Bureau’s Business Register, the sampling frame that is also used for the Economic Census. Additionally, these surveys are collected electronically via an online instrument. This 2016 version of the ASM/COS instrument is the basis for the design and functionality of the online instrument to be used to collect the 2017 Economic Census. Therefore, debriefings will be used to help inform any necessary design improvements for the 2017 Economic Census instrument.

The Economic Census is an electronic reporting survey conducted by the Census Bureau every five years. The survey is sent out to nearly 4 million businesses (including large, medium and small companies representing all U.S. locations and industries) and collects information on a range of operational and performance data. Data from the survey are used as the official five-year measure of American business and the economy. This survey is collected under the authority of Title 13, United States Code (U.S.C.), Section 161 and 182. Further information regarding the Economic Census can be found at this website: http://www.census.gov/econ/census/ .

Purpose: Historically, data for the Economic Census were collected via paper forms, and a downloadable software application (beginning in 1997). However, in 2017 the Economic Census will be conducted via a self-administered Web only method for all establishments. This change in data collection requires pretesting to evaluate the instrument prior to its release in 2017. In addition, pretesting will help inform user centered design ensuring that key functions are captured within the instrument, making it easy for respondents to understand and report. Additional objectives outlined for the evaluation of the online 2017 Economic Census instrument include the following:

• Evaluate the instruments’ performance in terms of efficiency, accuracy, and user satisfaction

• Identify areas of the instrument that are problematic for users

• Identify instructions/features that are difficult for users to understand

• Provide recommendations for improvements to the design of the instrument that will enhance its usability

The results from the usability evaluation will be recorded and tabulated and a report produced that outlines the findings of the pretest and recommendations for improvement.

Population of Interest: Large, medium and small companies from various U.S. locations and industries.

Timeline: Testing will be conducted from March through April 2017.

Language: Testing will be conducted in English only.

Method: We plan to conduct debriefing and usability interviews with respondents in up to 75 establishments. Travel is required for the usability interviews, whereas debriefing interviews will be conducted over the phone.

Sample: We plan to conduct a total of 75 interviews. The sample size necessary for this test was determined by qualitative research experience. We plan to conduct interviews with a variety of sizes and types (i.e., industries) of business establishments.

Recruitment: Participants will be recruited using a listing of respondents from the 2012 Economic Census and incoming respondents from the 2016 Annual Survey of Manufactures and Report of Organization. Respondents will be informed that their participation is voluntary. The interviews may be recorded (with consent), to facilitate summarization.

Protocol: The protocols for the debriefing study and the usability evaluation are enclosed (see Attachments A_1 and A_2). Debriefing respondents will be contacted via phone and asked to describe their experience with completing the 2016 ASM/COS instrument. Sessions for the usability evaluation will be conducted in person. Participants will be asked to complete several tasks using the online instrument (see Attachment C). Measures of efficiency, accuracy, and user satisfaction will be used to identify usability issues and guide recommendations for improvement. We anticipate all sessions will take 60 minutes to complete.

Use of Incentive: Monetary incentives for participation will not be offered.

Below is a list of materials to be used in the current study:

1. Attachment A_1 (ASM/COS debriefing) & A_2 (Usability study) Protocol used for each study to outline how the research study will be conducted

2. Attachment B: Demographic questionnaire (Usability study) to obtain participant characteristics

3. Attachment C: Consent form (Usability study) to obtain participant consent for participation and recording of the session

4. Attachment D: Computer use and Internet experience questionnaire (Usability study) to get an understanding of participants' experience using computers and the internet

5. Attachment E: Satisfaction questionnaire (Usability study) to allow for a subjective satisfaction measure to be gathered based on the participants’ experience with the online instrument

6. Attachment F: User tasks (Usability study) to allow for an assessment of the instrument features/design and its ease of use

7. Attachment G: Debriefing questions with targeted probes (Usability study) to allow for a conversational exchange between test administrator and the participant about topics related to the instrument that were not yet covered or need further clarification

8. Attachment H: Instrument screenshots to illustrate features of the instrument that are being evaluated (images shown are from a similar 2016 version of the instrument)

Length of interview: For the debriefing phone calls, we expect that each interview will last no more than 60 minutes (35 cases x 60 minutes per case = 35 hours). Additionally, to recruit respondents we expect to make up to 5 phone contacts per completed case. The recruiting calls are expected to last on average 3 minutes per call (5 attempted phone calls per completed case x 35 cases x 3 minute per case = 9 hours). Thus, the estimated burden for this portion of the project is 44 hours (35 hours for interviews + 9 hours for recruiting).

For the in-person usability testing, we expect that each interview will last no more than 60 minutes (40 cases x 60 minutes per case = 40 hours). Additionally, to recruit respondents we expect to make up to 5 phone contacts per completed case. The recruiting calls are expected to last on average 3 minutes per call (5 attempted phone calls per completed case x 40 cases x 3 minute per case = 10 hours). Thus, the estimated burden for this portion of the project is 50 hours (40 hours for interviews + 10 hours for recruiting).

The total estimated burden for the entirety of this project is 94 hours (44 hours for the phone debriefings + 50 hours for the in-person usability testing).

The contact person for questions regarding data collection and statistical aspects of the design of this research is listed below:

Temika Holland

Data Collection Methodology & Research Branch

Economic Statistics and Methodology Division

U.S. Census Bureau

Washington, D.C. 20233

(301) 763-5241

[email protected]

Cc:

Ron Jarmin (ADEP) with enclosure

Nick Orsini (ADEP) with enclosure

Carol Caldwell (ESMD) with enclosure

Carma Hogue (ESMD) with enclosure

Diane Willimack (ESMD) with enclosure

Amy Anderson Riemer (ESMD) with enclosure

Temika Holland (ESMD) with enclosure

Aryn Hernandez (ESMD) with enclosure

Demetria Hanna (ESMD) with enclosure

Jennifer Hunter Childs (ADRM) with enclosure

Jasmine Luck (ADRM) with enclosure

Danielle Norman (PCO) with enclosure

Mary Lenaiyasa (PCO) with enclosure

Lisa Donaldson (EMD) with enclosure

Kim Moore (EWD) with enclosure

William Samples (ERD) with enclosure

Theresa Riddle (EMD)with enclosure

Michelle Vile Karlson (EMD)with enclosure

James Jamski (EMD)with enclosure

4