Attachment 12 - ORS Sample Design Evaluation

Occupational Requirements Survey Sample Design October 2015
1

Gwyn R. Ferguson1 and Erin McNulty1
U.S. Bureau of Labor Statistics, 2 Massachusetts Ave. NE, Washington, DC 20212

Abstract
The Bureau of Labor Statistics (BLS) is working with the Social Security Administration
(SSA) to establish the Occupational Requirements Survey (ORS), an establishment survey
that collects information on the vocational preparation and the cognitive and physical
requirements of occupations in the U.S. economy, as well as the environmental conditions
in which those occupations are performed. In preparation for the first ORS production
sample, we studied several potential sample designs and assessed each for practicality and
coverage of the population. In addition, we considered how and if the ORS sample might
be coordinated with the sample of BLS’s National Compensation Survey (NCS). In this
paper, we evaluate each sample design option, describe the sample design that was selected
for the first ORS production sample, and outline plans for monitoring the effectiveness of
the selected sample design.
Key Words: survey design, sample allocation, sample selection
1. Introduction
In the summer of 2012, the Social Security Administration (SSA) and the Bureau of Labor
Statistics (BLS) signed an interagency agreement, which has been updated annually, to
begin the process of testing the collection of data on occupations. As a result, the
Occupational Requirements Survey (ORS) was established as a test survey in late 2012.
The goal of ORS is to collect and publish occupational information that will replace the
outdated data currently used by SSA. More information on the background of ORS can be
found in the next section. All ORS products will be made public for use by non-profits,
employment agencies, state or federal agencies, the disability community, and other
stakeholders.
An ORS interviewer attempts to collect close to 70 data elements related to the
occupational requirements of a job. The following four groups of information will be
collected:





Physical demand characteristics/factors of occupations (e.g., strength, hearing, or
stooping)
Specific vocational preparation requirements, which include educational requirements,
experience, licensing and certification and post-employment training
Mental and cognitive demands of work
Environmental conditions in which the work is completed

Section 2 of this paper provides additional background information on the ORS. Prior to
selecting the first ORS sample, which will go into collection in fall 2015, several sample
design options were studied and compared. Section 3 discusses the major sample design
options that were considered, including the results of simulations of each of the options.
Section 4 presents the sample design that will be used for the first ORS production sample
and explains why it was chosen. Section 5 describes several approaches taken to estimate
the number of occupations for which ORS data will be published under the selected sample

design. Section 6 describes a modification to the design for the first production sample.
Finally, section 7 outlines plans for using data collected from the first few ORS samples to
evaluate the selected sample design.
2. Background Information on ORS
In addition to providing Social Security benefits to retirees and survivors, the Social
Security Administration (SSA) administers two large disability programs which provide
benefit payments to millions of beneficiaries each year. Determinations for adult disability
applicants are based on a five-step process that evaluates the capabilities of the worker, the
requirements of their past work, and their ability to perform other work in the U.S.
economy. In some cases, if an applicant is denied disability benefits, SSA policy requires
adjudicators to document the decision by citing examples of jobs the claimant can still
perform despite restrictions (such as limited ability to balance, stand, or carry objects) [1].
For over 50 years, the Social Security Administration has turned to the Department of
Labor's Dictionary of Occupational Titles (DOT) [2] as its primary source of occupational
information to process the disability claims. SSA has incorporated many DOT conventions
into their disability regulations. However, the DOT was last updated in its entirety in the
late 1970’s, and a partial update was completed in 1991. Consequently, the SSA
adjudicators who make the disability decisions must continue to refer to an increasingly
outdated resource because it remains the most compatible with their statutory mandate and
is the best source of data at this time.
When an applicant is denied SSA benefits, SSA must sometimes document the decision by
citing examples of jobs that the claimant can still perform, despite their functional
limitations. However, since the DOT has not been updated for so long, there are some jobs
in the American economy that are not even represented in the DOT, and other jobs, in fact
many often-cited jobs, no longer exist in large numbers in the American economy.
SSA has investigated numerous alternative data sources for the DOT such as adapting the
Employment and Training Administration’s Occupational Information Network (O*NET)
[3], using the BLS Occupational Employment Statistics program (OES) [4], and
developing their own survey. But they were not successful with any of those potential data
sources and turned to the National Compensation Survey program at the Bureau of Labor
Statistics [5].
3. Potential Sample Designs for ORS
The NCS has a fully-developed survey platform available for potential coordination with
the ORS. This platform includes its management structure, collection staff, and survey
procedures, as well as its sample of establishments and occupations. Coordination with the
NCS sample, in particular, might provide efficient data collection opportunities for the
ORS. At the same time, the NCS is an established survey that includes the Employer Cost
Index, a principal federal economic indicator, so the ORS sample design cannot be allowed
to harm the NCS sample or its outputs. Therefore, we have studied two main categories of
sample designs: designs in which the ORS and NCS samples are coordinated for sampling,
and designs in which the ORS sample is selected independently from the NCS sample.

3.1 Coordinated Sample Design
In a coordinated sample design, the NCS sample would be selected as a subsample of the
larger ORS sample. All establishments selected for the NCS sample would also be included
in the ORS. The evaluation of coordinated ORS and NCS sample designs is described in
detail by Ferguson et al. in “Occupational Requirements Survey Sample Design
Evaluation” [6].
The research described in Ferguson et al. showed that coordinated sampling was possible
but might be difficult to implement. Multiple sample simulations were carried out with the
NCS sample selected as a subsample of the ORS sample. It was a challenge to maintain
NCS sample design with this approach because the preferred sample distribution for each
survey differs. The NCS sample is allocated among 120 cells that are defined by 5
aggregate industries and 24 geographic areas. The measure of size for each frame unit is
adjusted so that sample units are distributed among 24 major private sector industries in
certain proportions. In other words, there are effectively 576 cells for which NCS needs
sufficient frame units. NCS’s industry proportions, however, differ from the industry
proportions chosen for the ORS sample. Since the ORS sample is projected to be almost
three times larger than the NCS sample, it was assumed that the NCS sample should be a
subsample of the ORS sample in a coordinated design. But it was found to be difficult to
apportion the ORS sample in a way that maintains its own targeted industry proportions
while providing an adequate frame for the NCS subsample.
The most promising coordinated sample design required a change to the NCS sample cells.
With 24 detailed industry sample cells, implicitly stratified by geographic area, the ORS
sample and NCS subsample were both able to achieve overall target employment and
sample size goals. Slight shifts to NCS sample by geographic area were noted but not
considered a serious problem. The sample design also has the advantage of being
straightforward and easy to explain. However, changing the NCS sample design to this
degree would require resources that are not currently available, making this sample design
difficult to implement before the first production sample is needed.
3.2 Independent Sample Designs
Having studied a range of coordinated sample designs, we investigate independent sample
designs in more detail in this paper. With an independent sample design, the effect of the
ORS sample design on the NCS sample design is not a factor, allowing the ORS more
freedom to tailor its sample design to its needs. To test the effectiveness and utility of each
design, we followed procedures similar to those used for coordinated designs in Ferguson
et al. [6].
To test each sample design in private industry and to allow comparability, we took a recent
sample frame – the same frame used to test the coordinated designs – and selected 150
ORS samples from that frame. We then used those samples to calculate average sample
sizes and weighted employment totals in various categories of interest. For private industry,
we also selected 150 NCS samples from the same frame, according to the NCS sample
design. We calculated the average amount of overlap between each ORS sample and its
corresponding NCS sample.
For the state and local government sectors of the economy, the NCS sample is selected less
frequently, so we already have access to the NCS government sample that will be in

collection during the first few years of ORS production, as well as the frame from which it
was selected. Therefore, we selected the ORS simulation samples from this frame and
compared the ORS government simulation samples to the actual NCS sample.
Since we were not limited by the NCS sample cells, we used 23 detailed industry strata as
the ORS sample cells in private industry. We used a proportional to employment allocation
method to distribute the total sample size among the 23 private industry ORS sample cells.
Note that the NCS adjusts its measures of size to achieve targeted sample sizes in these 23
industries, but the sample sizes are not guaranteed and can vary by a few units more or less.
See Appendix A for a list of the industry sampling cells for private industry samples.
In government we used 10 detailed industry strata as the ORS sample cells. NCS sample
cells use 5 aggregate industries and 24 geographic areas for a total of 120 sampling cells.
For NCS, the 5 aggregate industries are implicitly stratified by the 10 detailed industries
used for ORS. NCS and ORS both allocate the government sample proportional to
employment. The ORS industry sampling cells for state and local industry samples are also
shown in Appendix A.
3.2.1 Maximum Overlap Independent Design
We tested two types of independent sample designs. The first type was a maximum overlap
independent design, in which the ORS sample is selected separately from the NCS sample,
but ORS data is collected from all NCS units. The total ORS sample comprises the
independently selected ORS-only sample and the independently selected NCS sample.
To allocate the ORS-only sample using the maximum overlap design, the composition of
the NCS sample must be taken into account even though it is independent from the ORS
sample. This is not a concern in the government sample because both ORS and NCS are
allocated proportional to employment in similar cells. In private industry, however, NCS
sample sizes for the ORS sample cells are not guaranteed. The NCS sample must be
selected, not just allocated, prior to allocating the ORS sample, so that the exact number of
NCS units in each ORS sample cell is known. Then the total ORS sample size, including
the NCS units, is allocated proportional to employment across the 24 sample cells. The
sample cell allocations are reconciled with the NCS sample counts; if the NCS sample size
is greater than the allocation for a sample cell, the ORS allocation is increased to meet the
NCS sample size and the remaining sample is reallocated among the other cells. The
reconciliation process continues until each total ORS cell allocation is at least as large as
the NCS sample size for the cell. The NCS sample sizes are then subtracted from the total
ORS allocations to create the ORS-only sample allocations.
Two ORS sample cells, Finance (Rest of) and Insurance, required increases to their original
allocations due to the size of the NCS sample in the cells, as shown in Table 1. Because
the NCS sample sizes in these cells were so large, all of their ORS sample units came from
the NCS sample, and there were no ORS-only units. On the other end of the range, only
9% of the Health Care, Social Assistance (Rest of) cell came from NCS. Overall, NCS
units accounted for 38% of the total private industry ORS sample.

Table 1. Independent Design, Maximum Overlap: Allocation Adjustments for
Selected ORS Sample Cells (Average over 150 Private Industry Samples)
ORS Sample Cell (Detailed Industry)
Health Care, Social Assistance (Rest of)
Accommodation and Food Services
Finance (Rest of)
Insurance
…
Total ORS Sample

Percent Change from
Original to Final
Allocation
-1%
-1%
13%
14%
…
0%

Percent of Cell from
NCS
9%
15%
100%
100%
…
38%

After the ORS-only sample allocations were determined, the ORS-only sample was
selected using systematic probability proportional to employment selection. The total ORS
sample clearly overlapped the NCS sample for the 38% that comes from the NCS private
industry sample (the proportion was nearly the same in government). However, because
the ORS and NCS samples were independently selected from the same frame, there was
the possibility of additional, unintentional overlap between the samples. These units would
be collected in any case, since ORS data is collected from all NCS units under the
maximum overlap design. However, the unintentional overlap reduces the yield of unique
ORS data that can be collected.
Table 2 shows that approximately 1.4% of a single year’s ORS-only sample overlapped
the NCS private industry sample. ORS estimates will ideally be based upon data collected
from several years of sample, and Table 2 shows that when a three-year span of samples is
considered, the percentage of overlapping units rose to 3.3%. The three-year overlap
percentage was not three times the single year’s overlap percentage because there was
overlap from year to year within a survey. For the ORS-only portion of the private industry
sample, about 6.9% of units overlapped with another ORS-only unit in one (or both) of the
other years in the three-year span of samples.
The government sample size for the ORS is about 15% of the total planned ORS sample
size, which is proportional to the share of government employment out of all employment
(government and private industry). However, government establishments have an average
of 80 employees per establishment, while private industry establishments have an average
of 14 employees per establishment. It follows that there were proportionally fewer
establishments from which to select the government sample, compared to the private
industry sample. Some establishments had particularly large employment, making it likely
that they would be selected for any sample, ORS or NCS. As a result, the amount of overlap
was higher for the government sample. Close to a quarter of ORS-only units overlapped
with an NCS government unit in any single sample group, and about a third overlapped
with NCS after three years of sample were combined. The amount of overlap between ORS
samples within a three year span was also relatively high at nearly 30%.

Table 2. Independent Design, Maximum Overlap: Percentage of ORS-Only/NCS
Overlap (Average over 150 Samples)
Type of
Overlap

Private
Industry

State & Local
Government

One year (one sample group)

ORS to NCS

1.4%

23.1%

Three years (one estimation group)

ORS to NCS

3.3%

33.1%

Three years (one estimation group)

ORS to ORS

6.9%

29.2%

Span

3.2.2 Minimum Overlap Independent Design
The second type of independent sample design that we studied was a minimum overlap
independent design. In such a design, the ORS sample is selected separately from the NCS
sample, and ORS data is collected only from units selected in the ORS sample. For this
design, the ORS simulation sample size was allocated across the 24 private industry sample
cells and 10 government sample cells strictly proportional to employment. There were no
adjustments or other complications, so the resulting simulation sample sizes and weighted
employment totals met all expectations.
Under the minimum overlap design, the entire ORS sample is an ORS-only sample. Since
the ORS would ideally obtain data from the same number of observations regardless of
design, the ORS-only sample under the minimum overlap design was about 62% larger
than the ORS-only sample under the maximum overlap design. As a result, though there
was no intentional overlap between the ORS and the NCS, there was more unintentional
overlap between ORS-only and NCS units under the minimum overlap design. Table 3
shows that 1.8% of a single year’s private industry ORS sample overlapped the NCS
sample, and 3.7% overlapped over a three-year span. These overlaps do not decrease the
amount of unique ORS data that is collected since the NCS sample is not part of the total
ORS sample. However, these overlaps do increase respondent burden for the overlapping
NCS units.
The amount of unintentional overlap between ORS-only units across years was slightly
lower under the minimum overlap design than under the maximum overlap design. In
private industry, it was 0.2 percentage points lower, and in government it was 5.3
percentage points lower. This means that there would be a somewhat larger amount of
unique ORS data collected.
Table 3. Independent Design, Minimum Overlap:
Overlap (Average over 150 Samples)

Percentage ORS-Only/NCS

Type of
Overlap

Private
Industry

State & Local
Government

One year (one sample group)

ORS to NCS

1.8%

17.2%

Three years (one estimation group)

ORS to NCS

3.7%

29.0%

Three years (one estimation group)

ORS to ORS

6.2%

24.2%

Span

4. Selecting the Sample Design
There was one coordinated sample design that met the sample-related needs of both the
ORS and the NCS. Also, there were maximum and minimum overlap independent sample

designs that resulted in satisfactory samples. Any of the three approaches could be used to
meet the sample-related needs of both the ORS and the NCS. Therefore, the choice of a
final sample design hinged upon other considerations: flexibility, staffing, cost, respondent
burden, and response.
Despite two years of survey field testing, it remains difficult to fully predict the experience
of a full production collection cycle. Many features of the ORS sample – such as total size,
industry distribution, and collection timing – require modifications during the first few
years of ORS production. Flexibility, therefore, is essential. On the other hand, changing
the sample selection process for the NCS, a stable and well-established survey, must be
undertaken with care and deliberation. To balance these competing requirements, it was
first decided that the ORS would implement an independent, rather than coordinated,
sample design.
The next decision was to determine the desired amount of overlap between the collection
of the ORS and NCS samples. As discussed in section 3, there would be some overlap
between the two samples regardless of whether a maximum or minimum overlap design
was chosen, but the amount of intentional overlap can be controlled. Past field testing
showed that collecting ORS data from NCS units produced some efficiencies. Established
relationships with NCS respondents could be used to encourage ORS response. Overall
staffing needs and associated costs were lowered because the collection of administrative
data and certain data elements common to both surveys could be combined. For the same
reasons, the aggregate collection time of a given amount of ORS and NCS data was lower
than if no units had overlapped, potentially allowing a shorter collection cycle. Aggregate
respondent burden was also lower, as fewer respondents were contacted to collect the same
amount of data across the two surveys.
On the other hand, it was found that the data collection process for a combined ORS and
NCS unit was lengthy, especially for units with a large number of occupational
observations. Large establishments can have eight or more occupations. The collection of
ORS items alone for four occupations has typically taken about an hour [7]. A combined
ORS/NCS sample unit would also involve the collection of all NCS elements: wages,
hours, and the availability, cost, and provisions of various employee benefits. As a result,
individual respondent burden is increased, which could discourage existing NCS
respondents from responding.
The potential danger posed by additional individual respondent burden is the largest
concern because it could affect the amount of response and quality of collected data for
both the ORS and the NCS. It was determined that the best way to mitigate this concern is
to limit overlap between the two surveys and implement the minimal overlap independent
sample design. This design will be implemented initially for both private industry and
government, but it was noted that the independent sample design would allow one or the
other ownership group to modify the amount of NCS overlap in the future. In particular,
response rates for government units are historically high for the NCS. Given the smaller
number of available units, unintentional overlap will be higher for the government sector
than for private industry, so it might make sense to increase intentional government overlap
at some point.

5. Occupation Count Projections
A key feature of ORS estimates is the focus on detailed occupational information at the 8digit Standard Occupational Classification (SOC) level. Because of this, it is important to
collect a large number of unique observations for as many detailed occupations as possible.
The sample design affects how many occupations will have enough data for publishable
estimates.
The sample frame available to ORS contains only establishment-level information and does
not have occupational data. Therefore, in our selected sample design, industry distribution
is being considered as a proxy for occupational distribution, and we are assuming that
occupational variety is roughly proportional to the number of employees in an industry. It
is difficult to gauge the accuracy of this assumption before collecting occupational data
under the design, but we have done three studies to get an idea of how many occupations
ORS will be able to publish. We have studied the former NCS National Wage publication,
the current NCS estimation sample, and the Occupational Employment Statistics (OES)
sample.
5.1 NCS National Wage Publication
Until 2011, the NCS program produced a publication of national wage estimates covering
private industry and state and local government. The estimates were based on a probability
proportional to employment sample of more than 35,000 establishments, making it similar
in industry composition and size to the planned ORS sample. Occupations were selected
using the same probability selection method (PSO) that will be used for the ORS and were
classified using the Standard Occupational Classification (SOC) codes [9]. For this study,
NCS wage establishment response rates are assumed to be comparable to response rates in
the ORS; while wages are more sensitive than most ORS data, a larger quantity of data will
be requested from an ORS respondent than from an NCS wage respondent. Unlike the ORS
sample, the NCS wage sample was not a national sample but an area-based sample; data
were collected from 227 localities, and the unit weights were adjusted with an area factor
so that the sample represented the nation. Also, six-digit SOC was the most detailed
occupational level for which data were collected and published. Nevertheless, its
similarities to the ORS sample make the NCS national wage publication a good source for
gauging how much occupational data might be available from the ORS sample.
Wage estimates were published at various levels of detail, from the broad occupation (twodigit SOC) down to the detailed occupation (six-digit SOC). To be publishable, an estimate
for an occupation had to be based on data from at least three unique responding companies
(by Employer Identification Number, or EIN) and six unique occupational observations;
its relative standard error had to be less than 50%; and it had to pass a P-percent dominance
test. These criteria are less stringent than the ORS criteria are likely to be since we have
less experience with ORS data and will be more cautious with the initial publication criteria
until BLS has enough experience to determine if the less stringent NCS wage criteria are
sufficient for protecting ORS respondent confidentiality. Even with these cautions, we
believe the wage data are a good starting point for assessing how much data might be
available.
We counted the number of occupation-specific estimates that were published in the All
Civilian wage table (covering private industry and government combined) [8]. Table 4
shows that over two-thirds of all possible occupations were published, and over 90% of all

possible occupations were published at the 2-digit and 5-digit SOC level. (Note that threedigit SOC was used only to group more detailed published occupations, so only a small
number of the eligible occupations at that level were published even though many more
could have been published.) At the most detailed level, six-digit SOC, almost threequarters of possible occupations were published.
Table 4. 2010 NCS National Wage Publication (Published May 2011), All Civilian:
Percent of Occupations Published
Occupational Level

Percent Published

2 Digit SOC

95.65%

3 Digit SOC

11.25%

4 Digit SOC*

---

5 Digit SOC

92.57%

6 Digit SOC

72.35%

All Levels

71.61%

* Four-digit SOC is the same as three-digit SOC.
5.2 2014 NCS Estimation Sample
The NCS sample used in estimation in 2014 included about 10,000 establishments. It
covered both private industry and state and local government units, in roughly the same
proportions as are planned for the ORS. The NCS sample is being changed from an areabased sample, similar to the national wage sample in section 5.1, to a national sample akin
to the ORS. The 2014 NCS estimation sample was a combination of several annual sample
groups and included both area-based sample groups and national sample groups.
As discussed earlier, the current NCS sample is selected probability proportional to
employment with certain target industry proportions, which results in a different
distribution of establishments by industry from the ORS. (The NCS national wage sample
discussed in section 5.1 did not have these target industry proportions.) Occupations were
selected using the same probability selection method (PSO) that will be used for the ORS.
Response rates by industry are known and, for the purpose of this study, assumed to be
similar to those that will be experienced in the ORS, because collection procedures and
staff will be largely the same.
We counted the number of unique occupations found in the NCS sample by various levels
of SOC detail. Also, we counted the number of unique responding establishments by EIN
within each SOC level. We applied an ownership-based factor to each count to adjust for
the smaller size of NCS sample compared to the planned ORS sample size of 30,000. We
determined the proportion of all possible occupations that met several EIN and occupation
criteria levels.
Tables 5 and 6 show that at the most detailed SOC levels, the amount of available data
quickly falls off as more stringent criteria are added. Requiring 3 establishments and 6
occupations yields about 80% of possible six-digit SOCs for both private industry and
government, which is slightly higher than the results from the NCS wage publication.
Requiring 15 establishments and 30 occupations drops the private industry yield to 53%.
As might be expected from the smaller government sample, its yield at this level is much

lower, at 37%; in fact, government yields for all but the most lenient criteria and broad
occupations are quite a bit lower than the corresponding private industry results.
Table 5. 2014 NCS Estimation Sample: Percent of Private Industry Occupations
That Are Publishable

Publishability Criteria

Percent of Private Industry
Occupations That Are Publishable
(by SOC detail level)
6-digit

5-digit

3-digit

2-digit

3 establishments, 6 occupations

80%

85%

90%

100%

15 establishments, 30 occupations

53%

67%

82%

100%

15 establishments, 60 occupations

38%

56%

80%

100%

15 establishments, 90 occupations

30%

46%

75%

95%

15 establishments, 120 occupations

25%

38%

70%

95%

Table 6. 2014 NCS Estimation Sample: Percent of State & Local Government
Occupations That Are Publishable

Publishability Criteria

Percent of State & Local
Government Occupations That Are
Publishable (by SOC detail level)
6-digit

5-digit

3-digit

2-digit

3 establishments, 6 occupations

81%

83%

92%

100%

15 establishments, 30 occupations

37%

47%

74%

95%

15 establishments, 60 occupations

23%

31%

66%

95%

15 establishments, 90 occupations

16%

24%

60%

95%

15 establishments, 120 occupations

11%

18%

49%

95%

5.3 2011 OES Sample
The Occupational Employment Statistics (OES) program is a BLS survey program that
produces wage estimates for over 800 occupations [10]. With OES cooperation, we were
able to access a version of the 2011 private industry OES sample that the NCS program
had used for previous research. The OES sample had three years of data covering over 1
million establishments and included data for all occupations in each of the establishments,
including the 6-digit SOC code and number of employees for each occupation. The earlier
NCS researchers had amplified the sample to produce a quasi-frame of establishments with
occupation information.
Since we began with a full private industry (quasi) frame, we were able to apply the
selected ORS sample design and select a private industry sample of 25,500 establishments
(the private industry portion of the planned ORS sample size of 30,000). We stratified by
23 detailed industries, combining the aircraft manufacturing and all other manufacturing

industries that had been separate in the studies described in section 3 because the first is
small subset of the other and is not a particular focus for the ORS. The sample was allocated
proportional to industry employment. Establishments were selected by systematic
probability proportional to size selection with employment used as the measure of size.
Occupation selection followed the planned ORS protocol as closely as possible. The
number of quotes selected was based on establishment size, and occupations were selected
using systematic probability proportional to size selection with the occupational
employment used as the measure of size.
Each frame establishment was considered to be unique, though there were no EINs
available to verify this assumption. The number of unique responding quotes was counted
at several SOC code levels. The actual counts were multiplied by 0.75 to simulate nonresponse.
Table 7 shows ORS simulation results that are similar to the results from the private
industry 2014 NCS estimation sample (similar establishment thresholds could not be used
because of differences in how establishments were identified). The OES-based percentages
are a bit lower for the most stringent criteria at the most detailed SOC levels; for example,
requiring 120 occupations for a six-digit SOC yields 16% of possible occupations from the
OES sample compared to 25% from the NCS sample. The general patterns hold, though,
so the occupation projection studies provided a rough but consistent idea of the amount of
data to expect from the ORS.
Table 7. 2011 OES Sample, Private Industry: Percent of Occupations That Are
Publishable
Publishability Criteria

Percent of Private Industry Occupations That Are Publishable
6-digit SOC

5-digit SOC

3-digit SOC

2-digit SOC

6 occupations

78%

89%

94%

96%

30 occupations

40%

60%

89%

96%

60 occupations

26%

42%

83%

93%

90 occupations

20%

34%

77%

91%

120 occupations

16%

29%

68%

91%

6. Year 1 Modification to Design
In addition to the ORS-only establishments selected in year one using the minimum overlap
independent design described above, a supplemental set of establishments from the current
NCS private sample will be included for collection of the ORS data elements. The
modification is being made to increase the amount of data that can be published from the
first production sample given the amount of trained and available field collection staff. The
additional units are all active for the NCS but will be rotating out of the NCS sample after
October of 2015. The NCS design at the time that these units were selected was a threestage stratified design with probability proportionate to size selection used at each level.
The first stage was the selection of 152 areas with the second stage being the selection of
establishments within each selected area. The final stage was the selection of occupations
within sampled establishments. For more details on this design see “Evaluating Sample
Design Issues in the National Compensation Survey” by Ferguson et al. [11].

Since the two parts of the year one sample are selected independent of each other and each
has weights that represent the entire frame, weight adjustment factors will be applied to all
sample units in order to properly represent the frame. The data from this full sample will
be used in all estimates generated from the first ORS production sample. Following the
year one sample, all subsequent ORS samples will be selected independent of the NCS
sample and will not be supplemented by the NCS.
7. Future Plans
The first production sample for the ORS has been selected according to the selected sample
design described in sections 4 and 6. Data collection will begin in the fall of 2015. The
following year’s sample will need to be selected before all data have been collected from
the first sample, so we will assess the sample design at intervals throughout collection as
well as at the end of collection.
We have assumed for our work so far that establishment (unit) response rates will be similar
to those in the NCS, but the different subject matter could lead to different response rates.
The first production sample will allow us to determine whether the sample is less efficient
in certain industry strata or establishment size classes, which could lead to refining strata
definitions and/or shifting sample. We will also monitor response rates of individual ORS
elements (item response) to see if the data yield per establishment differs by industry or
size class in a way that could be mitigated by the sample design. As part of our detailed
response analysis, we will evaluate the potential for non-response bias in the ORS survey,
focusing on the ORS data elements with the lowest item response rates. Similar response
analysis will be evaluated for the 2015 pre-production test sample that was collected earlier
this year. However, the pre-production test sample was selected using a different design in
order to meet the test needs of that sample [6]. In addition, staff were just beginning to
learn how to collect ORS data, so response rates from the test might not accurately reflect
the response that will be obtained during production.
The number and distribution of responding units will contribute to the amount of data the
ORS collects for detailed occupations. We will monitor the amount of usable data that we
collect by occupational SOC code as well as the industries in which SOC codes are
concentrated. Our studies so far have relied upon the six-digit SOC occupation as the most
detailed occupation, but the ORS data will allow us to study occupational distribution at
the more detailed eight-digit SOC level. We will observe how many estimates can be
published at various levels of detail and also assess standard errors to determine whether
more sample is needed from certain strata to ensure estimate reliability.
Researchers and members of the public have commented that it may be necessary to update
ORS data for certain occupations more frequently than for others occupations. The sample
designs studied so far do not allow for this flexibility, but we have identified and will
evaluate additional sample designs that could allow such flexibility.
Minor changes to the design could be implemented quickly, perhaps as soon as the second
production sample. Other observations might lead to more extensive research that could
result in sample design changes further in the future. We will continually gather and assess
data throughout the collection of all ORS samples.

8. References
[1] Social Security Administration, Occupational Information System Project,
http://www.ssa.gov/disabilityresearch/occupational_info_systems.html.
[2] U.S. Department of Labor, Employment and Training Administration (1991),
“Dictionary of Occupational Titles, Fourth Edition, Revised 1991”
[3] U.S. Department of Labor, O*Net Online, http://www.onetonline.org/.
[4] U.S. Bureau of Labor Statistics (2008) BLS Handbook of Methods, Occupational
Employment Statistics, Chapter 3. http://www.bls.gov/opub/hom/pdf/homch3.pdf.
[5] U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, National
Compensation Measures, Chapter 8. http://www.bls.gov/opub/hom/pdf/homch8.pdf.
[6] Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan. 2014. Occupational
Requirements Survey Sample Design Evaluation. In JSM Proceedings, Government
Statistics Section. Alexandria, VA: American Statistical Association.2924-2938.
[7] U.S. Bureau of Labor Statistics (2014), “Occupational Requirements Survey
Consolidated Feasibility Tests Summary Report:
Fiscal Year 2014.”
http://www.bls.gov/ncs/ors/fy14_feasibility_test.pdf.
[8] National Compensation Survey: Occupational Earnings in the United States, 2010,
U.S. Department of Labor, U.S. Bureau of Labor Statistics, May 2011, Bulletin 2753,
http://www.bls.gov/ncs/ncswage2010.htm.
[9] U.S. Department of Labor, Bureau of Labor Statistics, Standard Occupational
Classification System, http://www.bls.gov/soc/.
[10] U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, Occupational
Employment Statistics, Chapter 3. http://www.bls.gov/opub/hom/pdf/homch3.pdf.
[11] Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan. 2010. Evaluating
Sample Design Issues in the National Compensation Survey. In JSM Proceedings,
Survey Research Methods Section. Alexandria, VA: American Statistical
Association.2838-2850.

Any opinions expressed in this paper are those of the authors and do not constitute policy
of the Bureau of Labor Statistics or the Social Security Administration.

Appendix A – ORS Industry Stratification
ORS Production Stratification
Detailed Industry
Educational
Services (Rest of)
Elementary and
Secondary Schools
Junior Colleges,
Colleges and
Universities
Mining
Construction
Manufacturing
Healthcare, Social
Assistance (Rest of)
Hospitals
Nursing and
Residential Care
Facilities
Utilities
Wholesale Trade
Retail Trade
Transportation and
Warehousing
Information
Finance (Rest of)
Insurance
Real Estate,
Renting, Leasing
Professional,
Scientific, Technical
Management of
Companies and
Enterprises
Admin., Support,
Waste Management
Arts, Entertainment,
Recreation
Accommodation and
Food Services
Other Services
(excl. Public
Administration)
Public
Administration

Included
NAICS
Codes
61 (excl.
61116113)

Private Industry

State & Local Govt.

Establishments
in Universe

Expected
Sample
Size

Establishments
in Universe

Expected
Sample
Size

8,671

36

1,251

2

6111

80,655

22

61,828

247

6112,
6113

17,249

23

7,550

78

21

35,471

27

23

746,906

197

6,339

7

31-33
62 (excl.
622, 623)
622

336,416

385

1,277,003

103

8,470

11

8,829

311

2,644

34

623

74,363

149

2,081

7

22

17,382

18

42

618,652

184

44-45

1,036,005

483

12,863

20

48-49

228,101

138

51
52 (excl.
524)
524

149,018

86

282,821

65

187,344

109

53

357,544

69

54

1,104,197

262

55

59,734

68

20,678

29

56

494,313

273

71

131,361

74

72

656,131

405

81 (excl.
814)

572,205

127
109,075

202

92 excl.
928