Response Rate Summary

Appendix E: Summary of Response Rate
Experience to Date
Establishment Method
Data collection for the O*NET Data Collection Program began in June 2001 and has
been in continuous operation since then. Appendix Exhibit E-1 shows our cumulative response
rate experience as of December 31, 2020. As indicated, 199,711 establishments and 227,256
employees have responded to the survey request, resulting in an establishment response rate of
72% and an employee response rate of 63%.
Appendix Exhibit E-1.

Establishment Method Data Collection Results

Sampled establishments

360,552

Eligible establishments

277,154

Participating establishments

199,711

Establishment response rate (participating establishments/eligible establishments)

72%

Sampled employees

358,877

Participating employees

227,256

Employee response rate (participating employees/eligible employees)

63%

Comparisons of the O*NET response rates with those of other business surveys are
complicated by several unusual design characteristics of the O*NET survey, including the
following:
•

Voluntary rather than mandatory participation. The literature indicates that
response rates on mandatory surveys are typically higher than those on comparable
voluntary surveys (Navarro, King, & Starsinic, 2011; Tulp, Hoy, Kusch, & Cole,
1991; Worden & Hoy, 1992). Because the O*NET survey is voluntary, its response
rates would be expected to be lower than those for the average federally mandated
survey.

•

No direct personal contact with the survey respondents by the survey organization
conducting the data collection. The O*NET requirement of respondent anonymity
means that participation at the employee level relies exclusively on the interactions
between the establishment’s point of contact (POC) and the employee. The survey
organization is not able to speak to the employee to respond to questions, motivate
responses, or follow up on noncompliance. In their review of establishment mail
survey response rates, Paxon, Dillman, and Tarnai (1995) found that establishment
surveys featuring anonymous mailings typically have lower response rates (by as

Appendix E-1

Appendix E:
Summary of Response Rate Experience to Date

O*NET Data Collection Program
OMB Control No. 1205-0421
February 2021

many as 30 percentage points) than surveys featuring direct personal contact with the
respondents.
•

Participation required at three stages of response—establishment level, point of
contact level, and employee level. The typical establishment survey requires
participation at only one or two levels: the establishment level and, in some cases, the
POC level. By contrast, three often distinct entities must agree to participate in the
O*NET Program: the establishment administration, the POC identified in the
screening interview, and the employee who is asked to complete the questionnaire.
Because very few surveys incorporate such a design, the survey methods literature is
essentially devoid of examples on which to base a reasonable response rate
expectation for the O*NET Data Collection Program. However, it is possible to
compare O*NET response rates at each stage with other establishment surveys that
incorporate these stages either separately or in combination. For example, the O*NET
establishment-level response rate can be compared with other mail establishment
surveys having only one response stage at the establishment level. In addition, the
O*NET employee-level response rate can be compared with the response rate of other
establishments’ self-conducted employee surveys.

The literature indicates that voluntary business surveys typically experience relatively
low response rates. Cycyota and Harrison (2006) analyzed response rate data from 231 surveys
of business executives conducted from 1992 to 2003 and found the overall average rate to be
32%. Tarnai & Paxon (2004) obtained a response rate of 48.6% in their survey of 2,626
businesses on survey mode preference; they concluded that the typical establishment survey
achieves a response rate of between 40% and 50%. Baruch and Holtom (2008) analyzed the
response rates for 463 studies published across 17 first- and second-tier journals in 2000 and
2005. Among the studies examined, 117 of them were organization-level surveys whose average
response rate was 35.7%. The 2001 Survey of Respirator Use in Private Sector Firms, a
voluntary mail survey of 40,002 establishments sponsored by the National Institute of
Occupational Safety and Health, used a two-tiered sampling process similar to the one used on
O*NET and achieved a business-level response rate of 75.5% (U.S. Department of Labor, 2003).
However, the field of survey work has seen a gradual decline in response rate. This was
evidenced by Roger Tourangeau’s 2017 presidential address (Tourangeau, 2017, May) to the
American Association of Public Opinion Research (AAPOR), in which he remarked that
response rates have been falling for more than 30 years and that this trend was the same across
the developed world and, within AAPOR’s overview of response rates (American Association of
Public Opinion Research, n.d.): “Largely due to increasing refusals, response rates across all
modes of survey administration have declined, in some cases precipitously.” In a recent review
of four major, voluntary, establishment-based surveys conducted by BLS, initial unweighted data
collection response rates ranged from 64% to 73% in December of 2013 but had declined to a

Appendix E-2

O*NET Data Collection Program
OMB Control No. 1205-0421
February 2021

Appendix E:
Summary of Response Rate Experience to Date

range 58% to 62.2% by December of 2019. Additionally, by March of 2020, response rates
among three of the surveys had fallen an average of 6.6 additional percentage points (Office of
Survey Methods Research, 2020) with a low response rate of 51%. In addition, from 2012
through 2017, the Census Bureau’s Advanced Monthly Sales for Retail and Food Services
Survey experienced a decline in response rates of more than 20 percentage points; response rates
for the Monthly Retail Trade Survey and the Manufacturers’ Shipments, Inventories, and Orders
Survey declined more than 10 percentage points (National Academies of Sciences, 2017). By
way of comparison, O*NET has also observed a downward trend in establishment response rates
over the past years, with rates at 75% for 2012–2014, 68% for 2015–2017, and more recently
55% for 2018–2020.
The above results for establishment surveys, summarized in Appendix Exhibit E-2,
suggest that both the O*NET cumulative establishment response rate of 72% and the more recent
experience of 55% for 2018–2020 are comparable to those reported for similar surveys and
reflective of trends in data collection.
Appendix Exhibit E-2.

Studies of Establishment-Level Response Rates
Study

Cycyota & Harrison (2006)
Tarnai & Paxton (2004)
Baruch & Holtom (2008)
Petroni et al. (2004)
W. Davie (2011)
National Institute of Occupational Safety and Health (2003)
ECI (2020)
JOLTS (2020)
O*NET (cumulative)
O*NET (2018–2020)

Response Rate
32%
49%
36%
66%–87%
66%–80%
76%
54%
51%
72%
55%

The O*NET employee response rates can be compared with those of surveys that directly
sample employees within an establishment. Because federally sponsored surveys of employees
within organizations are rare, the literature on their response rates is sparse. Most surveys of this
type are employee satisfaction surveys. For example, one well-documented, governmentsponsored survey of employees is the Public Service Employee Survey, administered to about
258,000 employees of the Public Service of Canada and conducted by Statistics Canada in 2008
(Treasury Board of Canada Secretariat, 2009). A questionnaire was delivered to each employee
by a government agent who personally requested that the employee complete the questionnaire
and return it by mail. Multiple follow-ups of nonrespondents were made by e-mail and
interoffice mail to maximize the response rate. No incentive was used, however; because all

Appendix E-3

Appendix E:
Summary of Response Rate Experience to Date

O*NET Data Collection Program
OMB Control No. 1205-0421
February 2021

sample members were also employees of the organization conducting the survey and could fill
out the survey on government time, the use of an incentive was thought to be unnecessary. The
final overall response rate for the survey was 66%.
The Office of Personnel Management (OPM) developed the Organizational Assessment
Survey (OAS) and has encouraged all federal agencies to survey their employees in order to
evaluate organizational performance, benchmark best practices, and align performance with
important and measurable outcomes. The experience OPM has had in implementing these
surveys in numerous federal agencies provides some evidence of response rates for employee
surveys conducted by the U.S. government. The OAS design closely resembles that of the
Canadian Public Service Employee Survey. The surveys are self-administered and are conducted
by each agency for its own employees. Furthermore, like the Public Service Employee Survey,
the OAS request to participate is personalized and made directly to the employee by his or her
employer. The features of the design offer a significant advantage over the O*NET survey
design, as previously noted.
Although the results of the OAS surveys are not publicly available, an official at OPM
was able to provide some general information regarding OAS response rates (C. Simons,
personal communication, March 21, 2002). According to OPM, response rates for OAS surveys
vary considerably by agency, from 30% to 80%. However, the average response across all
agencies is approximately 57%.
The Employee Viewpoint Survey, an annual census administered to hundreds of
thousands of full-time federal employees across U.S. government agencies, has yielded response
rates ranging from 46% to 50% during the period 2011–2016 (U.S. Office of Personnel
Management, 2016).
Additionally, Anseel, Lievens, Schollaert, and Choragwicka (2010), who conducted a
meta-analysis of 2,037 surveys in the field of industrial and organizational psychology,
management, and marketing from 1995 through 2008, found that surveys implementing many of
the response-rate-enhancing features of the O*NET survey had an average response rate of 52%.
In a study examining implications of employees’ mode preference in completing a survey, Cole,
Bedeian, and Feild (2006) sampled 8,598 employees across 50 countries who worked in the
manufacturing industry and who varied in age, job tenure, and job functions. The study offered
targeted respondents the same survey mode options as are given in O*NET: a paper-and-pencil
version or a Web-based option. The overall response rate was 57%.

Appendix E-4

O*NET Data Collection Program
OMB Control No. 1205-0421
February 2021

Appendix E:
Summary of Response Rate Experience to Date

The above results for employee surveys are summarized in Appendix Exhibit E-3. Again,
the data indicate that the O*NET cumulative employee response rate of 63% and the more recent
experience of 54% for 2018–2020 are comparable to those of similar surveys.
Appendix Exhibit E-3.

Studies of Employee-Level Response Rates

Study

Response Rate

Treasury Board of Canada Secretariat (2009)

66%

OPM (C. Simons, personal communication, 2002)

57%

OPM (2016)

46%–50%

Anseel et al. (2010)

52%

Cole et al. (2006)

57%

O*NET (cumulative)

63%

O*NET (2018–2020)

54%

Occupation Expert Method
The OE Method is a much smaller but still important component of the O*NET Data
Collection Program protocol. Appendix Exhibit E-4 shows our cumulative response rate
experience with this method as of December 31, 2020. As indicated, 13,809 of 19,463 eligible
OEs have participated, for a response rate of 71%. This response rate is higher than the employee
response rate for the Establishment Method, likely because of factors such as personalized direct
contact, greater total incentive, and generally higher education levels of the eligible population.
The OE Method has also seen declines in response rates over the past decade; rates of 75% for
2012–2014 stabilized at around 60-61% for both 2015–2017 and 2018–2020.
Appendix Exhibit E-4.

Occupation Expert Method Data Collection Results

Sampled occupation experts

25,901

Eligible occupation experts

19,463

Participating occupation experts

13,809

Occupation expert response rate (participating occupation experts/eligible occupation experts)

71%

Appendix E-5