Communicating Data Quality and Treatment as Experimental Data Product

Census Bureau pre-decisional

Small Business Pulse Survey: Communicating Data Quality and Treatment as
Experimental Data Product
Communicating Data Quality
It is critical to understand the economic impacts of the response to COVID-19 on the business
community, but in particular small businesses. The SBPS is intended to demonstrate the ability
of the federal statistical system to respond in a timely manner and to produce relevant statistics in
near real time. Communication will occur to both our survey respondents and our data users each
week.
The Census Bureau is providing the results of the SBPS as an experimental data product. In
December of 2019, the Census Bureau launched an experimental data products site allowing for
the dissemination of new, innovative statistical products that may not otherwise meet all of the
Census Bureau Quality Standards and would:
 benefit from feedback from data users and other stakeholders;
 benefit data users and other stakeholders in the absence of other available data;
and/or
 be used to gauge demand before investing more resources in operationalizing.
In the short-term, SBPS data are represented in a user-friendly graphical format and available at
https://www.census.gov/businesspulsedata as a featured experimental data product. The data are
shown as a new product tile on the existing experimental data products page and includes a
description of the data product, data sources, methodology, periodicity, as well as address
relevance and feasibility.
Additionally, the dissemination of the SBPS product includes:
 the experimental statistics;
 methodology and supporting research about the experimental statistics;
 links to longer methodology and/or research papers that support the release of the
experimental statistics; and
 a feedback mechanism for data users and other stakeholders;
 the experimental logo:

1

Census Bureau pre-decisional

Experimental statistics should try to comply with all Census Bureau Statistical Quality Standards
but may not meet some requirements due to their novel approach. Experimental statistics must
meet all Quality Standards related to the protection of information and transparency (S1 –
Protecting Confidentiality; F2 – Providing documentation to support transparency in information
products).
To comply with the Statistical Quality standards, we:


Performed cognitive testing on the questionnaire for Survey Phases 1 and 2.



Publish response rates with each weekly data release. We also publish a source and
accuracy statement, which fully documents the design and results of the survey, including
limitations of the results.



Provide standard errors with each survey estimate and notes about data limitations. Tables
of estimates will include notes that flag low quality estimates with text similar to the
following: The standard error of this estimate yields a coefficient of variation greater than
30 percent, which is an indicator of potential quality issues. Caution should be used when
interpreting this estimate.

2

Guiding principles: Experimental Statistical Products
Draft 20190530

Census Bureau
Statistical Products

Experimental
Statistical
Products

Taxonomy:
Census Bureau Statistical Products
All statistical products produced and released by the Census Bureau, as designated by OMB Statistical
Directives #3 and #4.
Experimental Statistical Products
New or innovative Census Bureau statistical products produced and released that may not meet all of
the Census Bureau Quality Standards and would:
 benefit from feedback from data users and other stakeholders;
 benefit data users and other stakeholders in the absence of other available data; and/or
 be used to gauge demand before investing more resources in operationalizing.
Designation:
Experimental statistical products (or experimental statistics) are statistical products created using new
data sources and/or methodologies. The resulting product may have unproven quality.
Experimental statistics may originate in program areas, the research directorate, and from within the
FSRDC system.
The Associate Director over the program or research area has authority over their experimental
statistics.

Experimental statistics provide an important path for the creation of a new, regularly occurring data
product. Experimental statistical products should be proposed only if there is a reasonable expectation
of producing relevant and useful experimental statistics for release to the public.
Experimental statistics are released through the Census Bureau website as a Census Bureau statistical
product.
A proposal to release experimental statistics through the Census Bureau website should include a
description of new statistics, data sources, methodology, periodicity, etc., as well as address relevance
and feasibility. The proposal should also address shortcomings of the proposed experimental statistics.
The Methodology and Standards Council will review proposals for experimental statistics. The Council
may offer concurrence and/or additional advice on development of the experimental statistics but the
ultimate authority lies with Associate Director proposing the experimental statistics.
Designating statistics as no longer experimental is the responsibility of program areas and their
management up to their Associate Director. For experimental statistics to be advanced to a regularly
occurring statistic without the experimental designation, relevant quality standards must be met.
The cessation of an experimental statistical product does not require further action by the Methodology
and Standards Council.
Statistical products produced for and released in research papers or for peer reviewed journals are not
in scope, although they eventually may lead to experimental statistical products.

Experimental Statistics Quality:
Experimental statistics should try to comply with all Quality Standards but may not meet some
requirements due to their novel approach.
Experimental statistics must meet all Quality Standards related to the protection of information and
transparency (S1 – Protecting Confidentiality; F2 – Providing documentation to support transparency in
information products).
Experimental statistics may be released with (serious) sampling or non-sampling quality issues provided:




affected statistics are clearly identified ;
the statistics are clearly labeled as experimental; and
quality issues with the statistics are described.

Publication and Feedback:
Experimental statistics should be published on as Census Bureau webpage that includes:




the experimental statistics;
methodology and supporting research about the experimental statistics;
links to longer methodology and/or research papers that support the release of the
experimental statistics; and



a feedback mechanism for data users and other stakeholders. Feedback should be solicited for
the experimental statistics sources, methodology, and measures of quality.

Governance outside of Quality Standards:
Some measure of governance should apply to experimental statistics, though not necessarily be codified
in the Quality Standards.
At minimum, Associate Directors or their designees should track experimental statistics. Specifically, a
list of statistics should be maintained as well as their outcome.