Survey Methodology

Census Bureau pre-decisional

Small Business Pulse Survey Methodology
Target Population
The target population is all nonfarm, single-location employer businesses with less than 500
employees and receipts of $1,000 or more in the 50 states, District of Columbia, and Puerto
Rico. The sampling frame was extracted in April 2020 from the final 2018 Business Register.
The following industries were designated as out of scope for the Business Pulse Survey:










Agriculture production (NAICS in (‘110000’, ‘111’, ‘112’)),
Railroads (NAICS = ‘482’),
U.S. Postal Service (NAICS = ‘491’),
Monetary Authorities – Central Bank (NAICS = ‘521’),
Funds, Trusts, and Other Financial Vehicles (NAICS = ‘525’),
Religious grant operations and Religious organizations (NAICS = ‘813’),
Private households (NAICS = ‘814’),
Public administration (NAICS = ‘92’), or
Unclassified with legal form of organization as tax-exempt or unknown

The set of businesses in the target population that responded to the 2017 Economic Census
were identified as the subset of businesses eligible to participate in the Small Business Pulse
Survey (SBPS). The 2017 Economic Census (EC) utilized an all-electronic data collection
strategy. Respondents to the EC were mailed a letter containing an authentication code and
were invited to create an account using the Respondent Portal. To establish their Respondent
Portal account, respondents provided a valid email address, their name, phone number, and
were required to establish a password.
Of the 8 million employer establishments in scope to the EC, about 2 million were classified as
multi-unit and 6 million were classified as single units. Of the 6 million single unit employer
businesses, almost 5 million had between 1 and 500 paid employees. Approximately 1.7 million
single unit businesses received an invitation to respond to the EC, and administrative data were
used for the remaining cases to minimize respondent burden. Approximately 1.1 million of the
1.7 million single unit business invited to participate in the EC had between 1 and 500
employees.
In total, 4 million establishments received an invitation to the 2017 EC. The response rate was
70% for single unit businesses and 79% for multi-unit businesses. Nearly 1.2 million responses
including authenticated email address from the single unit universe were captured as part of
1

Census Bureau pre-decisional

the EC data collection process. Roughly 550K units of the units that were active at the time of
the EC would now be considered deaths due to having no payroll in 2018, 2019, or 2020; the
single-unit deaths were excluded from the SPBS frame.
In April 2020, the Census Bureau extracted from the final 2018 Business Register the singlelocation businesses with e-mail addresses in the scope of the Small Business Pulse Survey. This
resulted in approximately 950K units. We then matched these single location businesses to our
current survey databases to determine if we had a more recent email address. Using date time
stamps associated with updates to the email address field, we found 91K email addresses that
were more recent than what was collected in the EC. An additional 10K emails were removed
due to validity checks, resulting in a target population of about 940K.
An initial comparison of firm age, as calculated for the Business Dynamics Series (BDS), in the
set of businesses in the target population vs. those with e-mail addresses, demonstrates a
slightly smaller percentage of younger firms (0-5 years) and a slightly larger percentage of older
firms (21+ years) in the e-mail population than in the population overall. These differences
might be due in part to different scoping criteria in BDS, which excludes businesses that do not
have positive March 12 employment.
Firm Age
0-5 years
6-10 years
11-20 years
21+ years

BDS Small SUs
33.8
16.0
22.3
27.9

E-mail Sample
25.1
15.7
25.4
33.8

We plan to use firm age, where available, to monitor and compare distributions of responding
firms to the target population to assess potential nonresponse bias in the estimates.
Sample Design
To maximize the use of available e-mail addresses, the Census Bureau initially intended to use
the full set of 940K businesses with e-mail addresses in the group, dividing the 940K into nine
panels of roughly equal size and representation for the weekly e-mail invitations to respond to
the SBPS. To establish the panels, the entire sampling frame (including businesses with and
without e-mail addresses) was stratified by state and by 3-digit NAICS. Within each sampling
stratum, the units were sorted by an indicator for presence of an e-mail address, MSA, and
2018 annual payroll. Each business in the stratum was then systematically assigned to one of
the nine panels. The sample weight (wgt) for each unit is the ratio of the count of the total
2

Census Bureau pre-decisional

number units in the stratum (both with and without email addresses) divided by the number of
selected sample units (with e-mail addresses).
During Week 1 of the Phase 1 collections, a decision was made to drop from collection e-mail
addresses associated with more than two businesses in the sample, in order to reduce
ambiguity over which business was responding. Weights for the remaining businesses were recalculated. Respondents with an e-mail address associated with two businesses were
instructed to report for one of its businesses.
In Phase 2, email addresses linked to more than 10 businesses remain ineligible for data
collection, but all others will be retained in the collection. The business name and last fivedigits of a selected business’ Employer Identification Number will be provided to the
respondent in both the survey invitation and upon authenticated entry to the survey
instrument in order to clarify the intended respondent to the survey.
Tabulation
Weekly visualizations are based on weighed percentages for response categories. Nonresponse adjustments are applied to the current weekly panel as well as to late-responding
units included in the current period’s estimates.
Within any tabulation domain, we define the following:
 R
is the number of respondents in the current weekly panel
 NR
is the non respondents in the currently weekly panel
 wgt
is the sample weight for the current weekly panel
 R’
is the late respondents from the previous weekly panels included in the
current estimate
 wgt’
is the sample weight for the previous weekly panels
Initial on-response adjustments are calculated for each state by 3-digit NAICS.
𝑁𝐴𝐹ℎ =

∑𝑅+𝑁𝑅 𝑤𝑔𝑡
∑𝑅 𝑤𝑔𝑡 + ∑𝑅′ 𝑤𝑔𝑡′

The non-response weight adjustment factor is applied to the current weekly panel and the late
response units. The tabulation weight is calculated as follows:
𝑊𝐺𝑇𝑡𝑎𝑏 = {

𝑁𝐴𝐹ℎ × 𝑤𝑔𝑡, 𝑓𝑜𝑟 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒𝑠 𝑖𝑛 𝑐𝑢𝑟𝑟𝑒𝑛𝑡 𝑤𝑒𝑒𝑘 𝑝𝑎𝑛𝑒𝑙
𝑁𝐴𝐹ℎ × 𝑤𝑔𝑡 ′ , 𝑓𝑜𝑟 𝑙𝑎𝑡𝑒 𝑟𝑒𝑠𝑝𝑜𝑛𝑠𝑒𝑠 𝑓𝑟𝑜𝑚 𝑝𝑟𝑒𝑣𝑖𝑜𝑢𝑠 𝑤𝑒𝑒𝑘𝑠
3

Census Bureau pre-decisional

For a specific question and response category a, the published response percentage is
calculated as:
∑𝑅,𝑅′ ∈𝑎 𝑊𝐺𝑇𝑡𝑎𝑏
𝑃𝑎 =
∗ 100
∑𝑅+𝑅′ 𝑊𝐺𝑇𝑡𝑎𝑏
Quality Measures
The Census Bureau will be transparent about the survey and its limitations. We will publish
quality metrics including response rates, nonresponse bias indicators, and estimates of
variability so that users may assess the quality and suitability of the results for their purposes.
-

Calculation of Response Rate

The unit response rate is calculated as follows:
𝑈𝑅𝑅 =

𝑛𝑢𝑚𝑏𝑒𝑟 𝑖𝑛 𝑅+𝑛𝑢𝑚𝑏𝑒𝑟 𝑖𝑛 𝑅′
𝑛𝑢𝑚𝑏𝑒𝑟 𝑖𝑛 𝑅+𝑛𝑢𝑚𝑏𝑒𝑟 𝑖𝑛 𝑁𝑅

* 100

In this calculation, we are assuming that the late respondents from the previous week's
panel are similar to the non-respondents in this week's panel who will be
delayed respondents for the following week. Thus we do not adjust the denominator for the
late responders; but add them to the numerator.
Response rates and collected data will be monitored internally on a daily basis at the national,
state, and MSA level.
-

Estimates of Sampling Variability

Variance estimates will be calculated using a delete-a-group jackknife estimator with 10
replicates. A description of the method can be found in the paper Using the Delete a Group
Jackknife Estimator in NASS Surveys, by Phil Kott. This method is used in a number of Census
Bureau surveys of businesses, including the Annual Business Survey and the Annual Capital
Expenditures Survey. We will be adapting code used in the Annual Business Survey to do the
calculations.
We will provide standard errors with each survey and notes about data limitations. Tables of
estimates will include notes that flag low quality estimates with text similar to the following:
The standard error of this estimate yields a coefficient of variation greater than 30 percent,
which is an indicator of potential quality issues. Caution should be used when interpreting
this estimate.
4

Census Bureau pre-decisional

-

Indicators of Nonresponse Bias

Census Bureau business surveys generally account for nonresponse through imputation or
non-response adjustment. Following OMB standards, we conduct a non-response bias study
for each program with an expected unit response rate below 80%. Based on the study results,
we attempt to mitigate the nonresponse bias where possible. For example, in ongoing surveys,
extra follow-up efforts for smaller units or particular industries may be implemented to
improve response when bias is indicated in estimates for these domains.
The Census Bureau intends to monitor the potential non-response bias in the Small Business
Pulse Survey. To assess potential nonresponse bias, we will compare distributions of
characteristics, such as employment, annual payroll, and firm age, of survey respondents to
similar distributions of non-respondents and the full in-scope business population from the
Business Register. The Census Bureau will make these comparisons at all published levels,
including the national, state, and published metropolitan area levels as well as by NAICS sector.
To address bias, a post-stratification adjustment by employment size class will be applied at
the state by 2-digit NAICS level. Note that the size adjustments, developed from Business
Register data, do not yet reflect the full impact of the pandemic on businesses, such as
business closures. Data users will need to take into consideration the potential effects of the
pandemic on particular geographic locations and industries in assessing potential nonresponse
bias.
We release standard errors and response rates with each data release.

Disclosure Avoidance
Business Pulse Survey releases will comply with Census Bureau disclosure avoidance practices.
U.S.-level estimates will be suppressed if based on fewer than three contributors. In
compliance with IRS Publication 1075, Tax Information Security Guidelines for Federal, State,
and Local Agencies, state-level and MSA-level estimates with fewer than 10 contributors will be
suppressed.

5