Instrument Development and Design Research and Testing Highlights

Attachment G 

Department of Commerce 
United States Census Bureau 
OMB Information Collection Request 
2017‐2019 Report of Organization 
OMB Control Number 0607‐0444 

Instrument Development and Design Research and Testing Highlights 

OMB
Number Generic
of Cases Clearance
Date
8
June 2014

Component

Activity

Dates

Single Unit
Businesses

Task analysis

July 2014

Usability testing
Round 1

October –
November
2014

10

October
2014

Usability testing
Round 2

October
2015

9

October
2014

Respondent
debriefings for
2015 ASM fullscale
implementation of
response-driven
design

February –
June 2016

23

January
2016

Analysis of Web
paradata from
2015 ASM fullscale
implementation of
response-driven
design

April 2016

NA

NA

Description, Selected
Results and/or
Recommendations
Response process is
relatively straightforward
for SUs; however, they
rely heavily on paper
questionnaires for
gathering the data.
Tested a low-fidelity
prototype and provided
recommendations for a
response-driven design
for implementation in
2014 ASM. Respondents
preferred less
information on the screen
and skipping questions
that were not relevant to
them.
Tested the 2015 ASM
response-driven
instrument prior to
release in December
2015
 Respondents liked
response-driven
design, answering
questions “one at a
time.”
 Respondents wanted a
worksheet preview,
but did not see this
feature prior to
beginning.
 Identified
troublesome
screens/questions;
reviewed use of
features/functions;
evaluated indicators
of burden.
 Prioritized changes
for 2016 ASM.

Component

Activity

Dates

OMB
Number Generic
of Cases Clearance
Date

Description, Selected
Results and/or
Recommendations

Multi-Unit (MU)
Businesses

Respondent
debriefings for
2012 Economic
Census

February
2014

11

January
2014

Multi-Unit (MU)
Businesses,
continued

Usability testing
Round 1

November
2014

10

November
2014

Debriefed respondents
about their use of the
Census Surveyor
instrument in order to
identify features that
should be maintained
and/or improved for the
2017 Economic Census
electronic data collection
instrument. Major
findings included:
 Desire for online
reporting because
downloading Surveyor
software is
troublesome
 Use of spreadsheets is
critical to ease burden
 Questionnaire-like
reporting guide and
instructions are needed
to aid data-gathering if
paper forms are not
available
 Tools are needed to aid
complex response
process that requires
gathering data from
multiple data sources /
areas in the company
 Many respondents
were unaware of
existing burdenreducing functionality
in Surveyor, and these
tools need to be made
more evident, along
with clear instructions
on how to use them
 Enhance automated
features/functions and
streamline review of
edit failures
Tested a low-fidelity
prototype and gathered
requirements for a

Component

Activity

Dates

OMB
Number Generic
of Cases Clearance
Date

Usability testing
Round 2

May - June 24
2015

November
2014

Focus group with
NPC Register
Analysis Branch
analysts

April 2015

NA

Description, Selected
Results and/or
Recommendations
functioning prototype,
and tested initial
spreadsheet designs.
 Performance of
SharePoint prototype
inadequate and
solution abandoned.
 Feedback on proposed
MU instrument
interface generally
positive for main menu
design, introductory
screens, and step-bystep instructions for
instrument features.
 Feedback on
alternative spreadsheet
designs generally
positive for
incorporating
questions’ instructions
and improved visual
design.
Discussed analysts’
experience assisting
respondents with Census
Surveyor and obtained
feedback on plans for
revising the MU
instrument.

Component

Activity

Multi-Unit (MU)
Businesses,
continued

Debriefing
April –
interviews with
June 2015
2014 Report of
Organization/ASM
respondents

Usability testing
Round 3

Dates

September
2015

OMB
Number Generic
of Cases Clearance
Date
30
March 2015

17

June 2015

Description, Selected
Results and/or
Recommendations
Evaluated use of Census
Surveyor features /
functionality with
respondents at small and
mid-size multi-unit
companies.
Corroborated past
research:
 Small MUs preferred
within-instrument
functions;
 Mid-size / larger MUs
preferred Excel
spreadsheet functions;
 Full capabilities of
Census Surveyor
overlooked by Rs;
 Downloaded / printed
documents critical for
gathering data from
other company
personnel
Evaluated basic
functionality of MU
instrument prototype in
Centurion - spreadsheet
download/upload, form
view, and database that
communicates to
spreadsheet and form
view; navigation and
usability of pages;
effectiveness of content,
especially instructions
about reporting
procedures; navigation
and usability of
spreadsheet template;
gathered requirements for
worksheet/reporting
guide

OMB
Number Generic
of Cases Clearance
Date
NA
NA

Component

Activity

Dates

Multi-Unit (MU)
Businesses,
continued

Analysis of
Census Surveyor
paradata from
2015 ASM

July –
September
2016

North American
Product
Classification
System (NAPCS)

Record-keeping
study

November
2014

9

August
2014

Usability testing
Round 1

October
2015

9

June 2015

Usability testing
Round 2

October
2015

7

June 2015

Usability testing
Round 3

April 2015

12

August
2014

Description, Selected
Results and/or
Recommendations
Although the legacy
Census Surveyor
instrument was a
powerful tool with useful
features, Rs were not
aware of / did not use
quite a few of them.
Explored availability of
product line data in
respondents’ business
records.
Tested alternative
presentations of product
lists for SUs;
recommended 2-screen
design where Rs first
select relevant products
from a list, then second
screen displays only the
selected products and
requests receipts for each
one.
Tested with Construction
SUs and MUs:
 Prefer 2-screen design
over spreadsheets;
 “Building” and “Nonbuilding” categories
helped Rs focus on
their construction type
Investigated the utility of
a search feature for
looking up products. Rs
preferred “write-in” over
search functionality, as
tested.

Component

Activity

Dates

Usability testing
Round 4

November
2015

North American
Product
Classification
System
(NAPCS),
continued

Field test of
alternative
NAPCS question
designs and writeins

January –
March
2017

MU/SU
Centurion
Instrument

OMB
Number Generic
of Cases Clearance
Date
9
August
2014

Description, Selected
Results and/or
Recommendations
Tested orientation of MU
NAPCS spreadsheets;
decision to replicate
Census Surveyor version
with products in columns
and locations in rows.
This will be a field test
using an experimental
design to examine the
effectiveness of two
alternative designs and
lay-out options for
obtaining descriptions of
products not explicitly
provided in the product
lists. Results are
expected in late March,
and will be used to
recommend a design
strategy for “write-ins” in
the final Economic
Census instrument.

892

Submitted:
December
2016;
Approval
expected:
January
2017

Usability testing
July 2016
for 2016 Report of
Organization/ASM
Round 1

18

June 2016

Tested a limitedfunctioning prototype.
The redesigned
instrument performed
well overall, but
additional instructions
and clearer labeling of
buttons would be helpful
to respondents

Usability testing
October
for 2016 Report of 2016
Organization/ASM
Round 2

28

June 2016

Tested a more fully
functioning prototype.
There were some features
and functionality that
were not clear to
participants, e.g. how to
update their locations.

Component

Web Respondent
Portal Design
and
Functionality

Activity

Dates

Usability testing
for 2017
Economic Census

TBD

OMB
Number Generic
of Cases Clearance
Date
TBD
TBD

Usability testing

July 2016

18

June 2016

Usability testing

October
2016

19

September
2016

25

January
2016

2015 Report of
February –
Worksheet /
Question Preview Organization/ASM March
Worksheet
2016
Debriefings

Description, Selected
Results and/or
Recommendations
This will be the first
usability testing of
complete fully
functioning Centurion
data collection
instrument for 2017 EC.
Joint with Report of
Organization/ASM
instrument usability
testing. Early results did
not show major
difficulties with using the
new Portal.
Overall, the account
creation process and
linking a survey to that
account was not
problematic.
Evaluated use of
form/PDFs in current
systems, and obtained
feedback on worksheet
prototype:
 Form/PDFs play
critical role in data
collection
 Most Rs used them (or
would have, had they
known about them)
 Positive feedback on
prototype worksheet

OMB
Number Generic
of Cases Clearance
Date
159
JulyNovember
2016

Component

Activity

Dates

New Content:
Industry-specific
Special Inquiries

Cognitive testing

August –
December
2016

New Content:
Factoryless
Goods
Production
(FGP)

2012 Economic
Census respondent
debriefings

January –
June 2014

97

April 2014

2016 Report of
Organization
Enterprise
Statistics Program
cognitive testing –
Rounds 1&2

June –
October
2015

37

May 2015

Description, Selected
Results and/or
Recommendations
21 sets of questions were
tested. Much of the new
content was not
problematic. The
majority of the questions
required only minor
changes. Only one set of
questions required
revisions that were retested.
Evaluated effectiveness
of “Special Inquiry”
questions re: contract
manufacturing services
(CMS) to aid in
identifying FGPs at the
establishment level.
Results found wide
variation in Rs’
interpretation and
responses to these
questions demonstrated
that CMS alone was
inadequate for
identifying FGP.
Cognitive testing of
enterprise level questions
about FGP activities.
Found that respondents
had inconsistent
interpretation or
confusion about some
key terminology /
characteristics associated
with FGP.

Component

Activity

Dates

New Content:
Factoryless
Goods
Production
(FGP),
continued

Exploratory early
stage scoping
exploratory
interviews
Round 1

September
2015

OMB
Number Generic
of Cases Clearance
Date
12
October
2014

Description, Selected
Results and/or
Recommendations
Interviews with
suspected FGPs and
Manufacturing Service
Providers (MSPs)
demonstrated
complexity:
 Misunderstanding and
inconsistent
interpretation of
several key concepts
and terms central to
FGP concept.
 Typical Econ Census
Rs in HR or Finance
depts indicated
someone else in
headquarters, e.g. the
Controller’s or
Operations office,
would be more
knowledgeable about
this activity.
 MSPs suggested
alternative types of
agreements. They
were found to be less
knowledgeable about
several key attributes
and record-keeping
practices for FGPs, and
so will not be included
in subsequent testing.

Component

Activity

Dates

Exploratory early
stage scoping
exploratory
interviews
Round 2

December
2015

OMB
Number Generic
of Cases Clearance
Date
9
July 2015

Description, Selected
Results and/or
Recommendations
Interviews with
suspected FGPs gleaned
additional information
about FGP practices,
concepts, and
terminology. FGPs
considered this to be an
enterprise-level concept
rather than estab-level.
Also revealed additional
difficulties for MUs
associated with
complexity of structure /
ownership of the
company, particularly
those with foreign
affiliates / ownership.

OMB
Number Generic
of Cases Clearance
Date
40
April 2016

Component

Activity

Dates

New Content:
Factoryless
Goods
Production
(FGP),
continued

Cognitive testing
Rounds 1-3

June –
September
2016

Cognitive testing
Round 4

November- Up to
December 35
2016

April 2016

Description, Selected
Results and/or
Recommendations
Multiple iterative rounds
of cognitive testing of
questions drafted
following early stage
scoping, in consultation
with BLS collaborators.
Draft questions were
written with the
expectation that “typical”
respondents would
answer, not necessarily
the “most
knowledgeable,” and this
was found to work
adequately. Findings for
SUs were generally good,
and questions required
minor revisions between
successive rounds.
Findings for MUs were
more problematic,
generally related to
complexities in structures
and ownership of
companies, particularly
those with foreign
affiliates or owners.
Questions were modified
based on Round 3 results,
and the final round of
cognitive testing with
revised questions will
focus on MUs. Will also
include non-FGPs, in
order to ensure that they
can answer appropriately,
that is, to guard against
propensity for “false
positive” responses.