Instrument Development and Design Research and Testing Highlights

Attachment K.docx

2017 Economic Census

Instrument Development and Design Research and Testing Highlights

OMB: 0607-0998

Document [docx]
Download: docx | pdf

Attachment K


Department of Commerce

United States Census Bureau

OMB Information Collection Request

2017 Economic Census

OMB Control Number 0607-XXXX


Instrument Development and Design Research and Testing Highlights



























Component

Activity

Dates

Number of Cases

OMB Generic Clearance Date

Description, Selected Results and/or Recommendations

Single Unit Businesses

Task analysis

July 2014

8

June 2014

Response process is relatively straightforward for SUs; however, they rely heavily on paper questionnaires for gathering the data.

Usability testing

Round 1

October –November 2014

10

October 2014

Tested a low-fidelity prototype and provided recommendations for a response-driven design for implementation in 2014 ASM. Respondents preferred less information on the screen and skipping questions that were not relevant to them.

Usability testing

Round 2

October 2015

9

October 2014

Tested the 2015 ASM response-driven instrument prior to release in December 2015

Respondent debriefings for 2015 ASM full-scale implementation of response-driven design

February – June 2016

23

January 2016

  • Respondents liked response-driven design, answering questions “one at a time.”

  • Respondents wanted a worksheet preview, but did not see this feature prior to beginning.

Analysis of Web paradata from 2015 ASM full-scale implementation of response-driven design

April 2016

NA

NA

  • Identified troublesome screens/questions; reviewed use of features/functions; evaluated indicators of burden.

  • Prioritized changes for 2016 ASM.






Multi-Unit (MU) Businesses

Respondent debriefings for 2012 Economic Census

February 2014

11

January 2014

Debriefed respondents about their use of the Census Surveyor instrument in order to identify features that should be maintained and/or improved for the 2017 Economic Census electronic data collection instrument. Major findings included:

  • Desire for online reporting because downloading Surveyor software is troublesome

  • Use of spreadsheets is critical to ease burden

  • Questionnaire-like reporting guide and instructions are needed to aid data-gathering if paper forms are not available

  • Tools are needed to aid complex response process that requires gathering data from multiple data sources / areas in the company

  • Many respondents were unaware of existing burden-reducing functionality in Surveyor, and these tools need to be made more evident, along with clear instructions on how to use them

  • Enhance automated features/functions and streamline review of edit failures

Multi-Unit (MU) Businesses, continued

Usability testing

Round 1

November 2014

10

November 2014

Tested a low-fidelity prototype and gathered requirements for a functioning prototype, and tested initial spreadsheet designs.

Usability testing

Round 2

May - June 2015

24

November 2014

  • Performance of SharePoint prototype inadequate and solution abandoned.

  • Feedback on proposed MU instrument interface generally positive for main menu design, introductory screens, and step-by-step instructions for instrument features.

  • Feedback on alternative spreadsheet designs generally positive for incorporating questions’ instructions and improved visual design.

Focus group with NPC Register Analysis Branch analysts

April 2015


NA

Discussed analysts’ experience assisting respondents with Census Surveyor and obtained feedback on plans for revising the MU instrument.

Multi-Unit (MU) Businesses, continued


Debriefing interviews with 2014 COS/ASM respondents

April – June 2015

30

March 2015

Evaluated use of Census Surveyor features / functionality with respondents at small and mid-size multi-unit companies. Corroborated past research:

  • Small MUs preferred within-instrument functions;

  • Mid-size / larger MUs preferred Excel spreadsheet functions;

  • Full capabilities of Census Surveyor overlooked by Rs;

  • Downloaded / printed documents critical for gathering data from other company personnel

Usability testing

Round 3

September 2015

17

June 2015

Evaluated basic functionality of MU instrument prototype in Centurion - spreadsheet download/upload, form view, and database that communicates to spreadsheet and form view; navigation and usability of pages; effectiveness of content, especially instructions about reporting procedures; navigation and usability of spreadsheet template; gathered requirements for worksheet/reporting guide

Multi-Unit (MU) Businesses, continued

Analysis of Census Surveyor paradata from 2015 ASM

July – September 2016

NA

NA

Although the legacy Census Surveyor instrument was a powerful tool with useful features, Rs were not aware of / did not use quite a few of them.






North American Product Classification System (NAPCS)

Record-keeping study

November 2014

9

August 2014

Explored availability of product line data in respondents’ business records.

Usability testing

Round 1

October 2015

9

June 2015

Tested alternative presentations of product lists for SUs; recommended 2-screen design where Rs first select relevant products from a list, then second screen displays only the selected products and requests receipts for each one.

Usability testing

Round 2

October 2015

7

June 2015

Tested with Construction SUs and MUs:

  • Prefer 2-screen design over spreadsheets;

  • Building” and “Non-building” categories helped Rs focus on their construction type

Usability testing

Round 3

April 2015

12

August 2014

Investigated the utility of a search feature for looking up products. Rs preferred “write-in” over search functionality, as tested.

Usability testing

Round 4

November 2015

9

August 2014

Tested orientation of MU NAPCS spreadsheets; decision to replicate Census Surveyor version with products in columns and locations in rows.

North American Product Classification System (NAPCS),

continued

Field test of alternative NAPCS question designs and write-ins

January – March 2017

892

Submitted: December 2016;

Approval expected: January 2017

This will be a field test using an experimental design to examine the effectiveness of two alternative designs and lay-out options for obtaining descriptions of products not explicitly provided in the product lists. Results are expected in late March, and will be used to recommend a design strategy for “write-ins” in the final Economic Census instrument.






MU/SU Centurion Instrument

Usability testing for 2016 ASM/COS Round 1

July 2016

18

June 2016

Tested a limited-functioning prototype. The redesigned instrument performed well overall, but additional instructions and clearer labeling of buttons would be helpful to respondents


Usability testing for 2016 ASM/COS Round 2

October 2016

28

June 2016

Tested a more fully functioning prototype. There were some features and functionality that were not clear to participants, e.g. how to update their locations.

Usability testing for 2017 Economic Census

TBD

TBD

TBD

This will be the first usability testing of complete fully functioning Centurion data collection instrument for 2017 EC.






Web Respondent Portal Design and Functionality

Usability testing

July 2016

18

June 2016

Joint with ASM/COS instrument usability testing. Early results did not show major difficulties with using the new Portal.

Usability testing

October 2016

19

September 2016

Overall, the account creation process and linking a survey to that account was not problematic.






Worksheet / Question Preview

2015 COS/ASM Worksheet Debriefings

February – March 2016

25

January 2016

Evaluated use of form/PDFs in current systems, and obtained feedback on worksheet prototype:

  • Form/PDFs play critical role in data collection

  • Most Rs used them (or would have, had they known about them)

  • Positive feedback on prototype worksheet

New Content: Industry-specific Special Inquiries

Cognitive testing

August – December 2016

159

July-November 2016

21 sets of questions were tested. Much of the new content was not problematic. The majority of the questions required only minor changes. Only one set of questions required revisions that were re-tested.






New Content: Factoryless Goods Production (FGP)

2012 Economic Census respondent debriefings

January – June 2014

97

April 2014

Evaluated effectiveness of “Special Inquiry” questions re: contract manufacturing services (CMS) to aid in identifying FGPs at the establishment level. Results found wide variation in Rs’ interpretation and responses to these questions demonstrated that CMS alone was inadequate for identifying FGP.

2016 COS Enterprise Statistics Program cognitive testing –

Rounds 1&2

June – October 2015

37

May 2015

Cognitive testing of enterprise level questions about FGP activities. Found that respondents had inconsistent interpretation or confusion about some key terminology / characteristics associated with FGP.

New Content: Factoryless Goods Production (FGP),

continued

Exploratory early stage scoping exploratory interviews

Round 1

September 2015

12

October 2014

Interviews with suspected FGPs and Manufacturing Service Providers (MSPs) demonstrated complexity:

  • Misunderstanding and inconsistent interpretation of several key concepts and terms central to FGP concept.

  • Typical Econ Census Rs in HR or Finance depts indicated someone else in headquarters, e.g. the Controller’s or Operations office, would be more knowledgeable about this activity.

  • MSPs suggested alternative types of agreements. They were found to be less knowledgeable about several key attributes and record-keeping practices for FGPs, and so will not be included in subsequent testing.

Exploratory early stage scoping exploratory interviews

Round 2

December 2015

9

July 2015

Interviews with suspected FGPs gleaned additional information about FGP practices, concepts, and terminology. FGPs considered this to be an enterprise-level concept rather than estab-level. Also revealed additional difficulties for MUs associated with complexity of structure / ownership of the company, particularly those with foreign affiliates / ownership.

New Content: Factoryless Goods Production (FGP),

continued

Cognitive testing

Rounds 1-3

June – September

2016

40

April 2016

Multiple iterative rounds of cognitive testing of questions drafted following early stage scoping, in consultation with BLS collaborators. Draft questions were written with the expectation that “typical” respondents would answer, not necessarily the “most knowledgeable,” and this was found to work adequately. Findings for SUs were generally good, and questions required minor revisions between successive rounds. Findings for MUs were more problematic, generally related to complexities in structures and ownership of companies, particularly those with foreign affiliates or owners.

Cognitive testing

Round 4

November- December 2016

Up to 35

April 2016

Questions were modified based on Round 3 results, and the final round of cognitive testing with revised questions will focus on MUs. Will also include non-FGPs, in order to ensure that they can answer appropriately, that is, to guard against propensity for “false positive” responses.





File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
AuthorBlynda K Metcalf (CENSUS/EWD FED)
File Modified0000-00-00
File Created2021-01-22

© 2024 OMB.report | Privacy Policy