Attachment 12 - ORS Sample Design Evaluation

Occupational Requirements Survey
Sample Design Evaluation
Gwyn R. Ferguson1, Erin McNulty1, and Chester Ponikowski1

Bureau of Labor Statistics, 2 Massachusetts Ave. NE, Washington, DC 20212

1

Abstract

The Bureau of Labor Statistics (BLS) is working with the Social Security Administration
(SSA) to carry out a series of tests to determine the feasibility of using the National
Compensation Survey (NCS) platform to accurately and reliably capture data that are
relevant to the SSA's disability program. The proposed new Occupational Requirements
Survey (ORS) is envisioned to be an establishment survey that collects information on the
vocational preparation and the cognitive and physical requirements of occupations in the
U.S. economy as well as the environmental conditions in which those occupations are
performed. This paper builds on the evaluation that was presented at the 2013 Federal
Committee on Statistical Methodology Research Conference that described the advantages
and disadvantages of integrating the sample design of ORS with the sample design of NCS.
This paper presents an analysis of the sample design options that were considered for this
proposed new survey and describes the sample design being used in the pre-production test
for ORS. It also describes the design issues that need to be resolved before full survey
implementation.

Key Words: survey design, occupational data, integrated surveys, survey cost
1. Introduction
In the summer of 2012, the Social Security Administration (SSA) and the Bureau of Labor
Statistics (BLS) signed an interagency agreement, which has been updated annually, to
begin the process of testing the collection of data on occupations. As a result, the
Occupational Requirements Survey (ORS) was established as a test survey in late 2012.
The goal of ORS is to collect and publish occupational information that will replace the
outdated data currently used by SSA. More information on the background of ORS can be
found in the next section. All ORS products will be made public for use by non-profits,
employment agencies, state or federal agencies, the disability community, and other
stakeholders.
An ORS interviewer attempts to collect close to 70 data elements related to the
occupational requirements of a job. The following four groups of information will be
collected:





Physical demand characteristics/factors of occupations (e.g. strength, hearing, or
stooping)
Educational requirements
Cognitive elements required to perform work
Environmental conditions in which the work is completed

Field testing to date has focused on developing procedures, protocols, and collection aids
using the NCS platform. These testing phases were analyzed primarily using qualitative
techniques but have shown that this survey is operationally feasible. Now it is time to turn
our attention to the survey design to ensure that we have the best possible sample design
to meet the needs of the ORS. This paper presents an analysis of the sample design options
that were considered for this proposed new survey and describes the sample design being
used in the pre-production test for the ORS. It also describes the design issues that need to
be resolved before full survey implementation.

2. Background Information on ORS
In addition to providing Social Security benefits to retirees and survivors, the Social
Security Administration (SSA) administers two large disability programs, which provide
benefit payments to millions of beneficiaries each year. Determinations for adult disability
applicants are based on a five-step process that evaluates the capabilities of the worker, the
requirements of their past work, and their ability to perform other work in the U.S.
economy. In some cases, if an applicant is denied disability benefits, SSA policy requires
adjudicators to document the decision by citing examples of jobs the claimant can still
perform despite restrictions (such as limited ability to balance, stand, or carry objects) [1].
For over 50 years, the Social Security Administration has turned to the Department of
Labor's Dictionary of Occupational Titles (DOT) [2] as its primary source of occupational
information to process the disability claims. SSA has incorporated many DOT conventions
into their disability regulations. However, the DOT was last updated in its entirety in the
late 1970’s, and a partial update was completed in 1991. Consequently, the SSA
adjudicators who make the disability decisions must continue to refer to an increasingly
outdated resource because it remains the most compatible with their statutory mandate and
is the best source of data at this time.
When an applicant is denied SSA benefits, SSA must sometimes document the decision by
citing examples of jobs that the claimant can still perform, despite their functional
limitations. However, since the DOT has not been updated for so long, there are some jobs
in the American economy that are not even represented in the DOT, and other jobs, in fact
many often-cited jobs, no longer exist in large numbers in the American economy. For
example, a job that is often cited is “envelope addressor,” because it is an example of a
low-skilled job from the DOT with very low physical demands. There are serious doubts
about whether or not this job still exists in the economy.
SSA has investigated numerous alternative data sources for the DOT such as adapting the
Employment and Training Administration’s Occupational Information Network (O*NET)
[3], using the BLS Occupational Employment Statistics program (OES) [4], and
developing their own survey. But they were not successful with any of those potential data
sources and turned to the National Compensation Survey program at the Bureau of Labor
Statistics.

3. Overview of Potential Survey Design Options
To develop sample design options for the ORS, BLS reviewed the sample designs for the
NCS [5] and other BLS surveys, including the OES and Survey of Occupational Injuries

and Illnesses [6]. Since the ORS will be collected by trained Field Economists who also
collect the NCS data, potential coordination with the NCS sample design was a key factor
of consideration. So we identified four basic categories of ORS survey designs allowing
for different potential levels of coordination with NCS:
1. Fully Integrated Survey Design – where the NCS establishment sample would be
a subsample of the ORS establishment
2. Independent Survey Design – where the ORS establishment samples would be
selected using a design appropriate for SSA’s needs, the NCS establishment
samples would be selected using the current NCS sample design, and there would
be no control on the amount of establishment sample overlap between the samples
selected for the two surveys
3. Separated Survey Design – where the NCS establishment sample would be
selected from the frame, the selected NCS establishments would be removed from
the frame, and an independent ORS establishment sample would be selected from
the rest of the frame
4. OES-ORS Integrated Design – where the ORS establishment sample would be
selected as a subsample of OES establishment sample
The next four sections of this paper will describe each of these options in more detail. For
each option, we will describe several pros (reasons to implement the design) as well as
some potential drawbacks from implementation. For the first two design options, we will
also describe the results of some empirical evaluations conducted using sample
simulations. Although the sample size for ORS has not been finalized yet, we assumed that
the ORS sample would need to collect data from 10,000 establishments annually for all
sample simulations and evaluation work. Section 8 of the paper describes the sample design
being used for the pre-production test of ORS that will start in the fall of 2014. Section 9
of the paper wraps up with some conclusions and recommendations for future work.

4. Fully Integrated Survey Design Options
Under this design, we would select an ORS establishment sample each year and then select
the NCS establishment sample as a subset of the bigger ORS sample. This approach was
used by the NCS program when a locality wage sample was a part of the integrated NCS
sample design. We would collect the ORS data from all establishments in the NCS
initiation sample. If a current NCS update establishment is selected for the ORS portion of
the sample, we would need to collect the ORS data elements for the existing NCS jobs.
For simulation purposes, we assumed an annual ORS sample size of 10,000 establishments.
Since the current NCS sample size includes approximately 3,200 establishments to be
initiated each year, we assume that we would collect the ORS data elements (and not the
NCS data elements) from the 6,800 ORS establishments not included in the NCS
subsample.
This design option poses some challenges that will need to be overcome in order to
implement it. The primary challenges are listed below.
• How should we select the samples to meet the needs of both sets of desired outputs
– NCS and ORS?
• NCS samples State & Local government establishments once each decade. Will
this approach work for ORS? If not, how will we handle this segment of the
economy for ORS?

•

Will the same sample rotation strategy work for both?

Some of the reasons we would want to use a fully integrated survey design option include
the following:
• Ability to generate linked outputs such as wage percentiles by portion of the day
lifting/carrying specified weights;
• Eliminates individual establishment burden associated with having to provide data
separately for more than one survey program;
• Lower data collection costs;
• May be able to use the existing NCS sampling, nonresponse adjustment, and
benchmarking systems; and
• NCS has experience doing this.
Some of the reasons we may not want to use this approach include:
• Increased interview length for establishments in both surveys is likely to have
negative impact on establishment response rates for NCS and ORS;
• Respondent fatigue may result in lesser quality data; and
• Potential increase in item nonresponse rates.
For these purposes, testing a design meant that we took a recent sample frame and selected
at least 150 samples from that frame. We then computed some basic information from each
of those samples and averaged the data across all the samples. In general, we were looking
for a sample design that allowed us to select an ORS sample to meet the ORS needs – that
is a sample that has enough establishments in each industry but not too many and resulted
in the total frame employment when we calculated the sum of the weighted employment
across all establishments. For NCS samples, we wanted to be able to select the NCS desired
sample sizes for each industry and geographic area AND have a total weighted employment
that matches the frame employment. For each option considered, we used the NCS sample
design and establishment selection methods with only the changes described below.

4.1 Fully Integrated – All NCS Design

Under this approach, we used the NCS design and allocation to pick both the ORS sample
and the NCS subsample. This meant that we selected the ORS sample the same way that
we pick the NCS sample – just with a larger sample size. So when we oversampled an
industry in NCS due to wide dispersion of wages and/or lower response, we used the same
target percentage of sample for the ORS allocation.
As expected, simulations showed that this design worked relatively well for NCS in that
the industry distribution was very close to the targeted distribution, the full sample size was
achieved, and the weighted employment was close to the frame total. However, the ORS
sample was too big in industries that may not need a particularly large ORS sample, such
as the Finance, Insurance, and Colleges & Universities industries. Given the unweighted
results from the feasibility tests conducted in 2013, described in the Phase 1 [7], Phase 2
[8], and Phase 3 [9] Summary reports, we feel that the NCS industry distribution may not
be ideal for ORS as it provides too much data in some industries where the occupational
requirements data do not show a wide dispersion.

The following table highlights some industries where the ORS sample size was at least
two times larger under NCS Design than under a proportional to employment design:
Average ORS Sample Size
Using…
Percent
Industry
Proportional to
Difference
NCS
Employment
Design
Design
Insurance
1,505
506
198%
Finance (Rest of)
2,489
845
194%
Junior Colleges, Colleges &
Universities
704
259
172%
Utilities
307
136
126%

4.2 Fully Integrated Design with Different ORS and NCS Allocations

With the current NCS sample design, we explicitly ensure that we select the desired number
of establishments for each aggregate industry. We distribute the total number to be selected
in each industry to the geographic cells in proportion to the total employment in each area
of the country for that aggregate industry. Then we implicitly stratify and adjust the
measures of size to select the establishments in a manner that ensures that we select the
target number of establishments in each of the detailed industries across the country as a
whole.
Under this approach, we allocated the ORS sample size to each industry and geographic
area stratum in proportion to the stratum employment. This allowed us to set allocations
for ORS directly in proportion to total employment, reducing the number of establishments
we sample in some industries compared to the all NCS design. We were satisfied with the
ORS sample sizes and employment totals that resulted from this approach. The average
weighted employment totals were very close to the frame employment.
The following table provides simulation data for the industries with the largest percentage
difference between the frame employment and the average ORS sample weighted
employment:
Average Employment
Percent
Detailed Industry
Difference
Frame
ORS Sample
Elementary and Secondary Schools
Junior Colleges, Colleges &
Universities
Mining
Insurance
Other Services (except Public
Admin.)

661,077

657,960

-0.5%

1,164,474
617,765
2,076,306

1,167,206
616,707
2,072,942

0.2%
-0.2%
-0.2%

3,741,681

3,737,766

-0.1%

However, we were unable to achieve acceptable NCS sample sizes and employment counts
using this approach. We tried this five different times with slight variations on the process
each time in an effort to achieve better results. But the results were pretty much the same.
For all simulations in which ORS was selected with an allocation strictly proportional to
employment, there are ‘shortages’ of frame units in certain cells targeted by our measure

of size adjustments. These cells are smaller than the sample cells, so the ORS sample design
does not ensure enough units in each geographic area. Therefore, the measure of size
(MOS) adjustments do not help us attain our target sample sizes in all detailed industries.
The following table highlights the industries where over 10% of NCS sample size was
lost using MOS adjustment factor approach to subsampling NCS:
Average NCS Sample
Size
Independent Subsample
Percent
Industry
Design
of ORS
Difference
Real Estate, Renting, Leasing
215
162
-24.7%
Mining
86
66
-23.8%
Hospitals
251
210
-16.3%
Total

9,804

9,792

-0.1%

Note that the sampling method allowed much of the “loss” in these industries to be
absorbed by other related industries (for example, Construction and Manufacturing
absorbed some of the losses from the Mining industry). However, not all losses could be
completely accounted for, so the full sample was short of the target size.

4.3 Fully Integrated Design – Inside Out Option

For the Inside Out Design, we turned the current NCS sampling cells inside out while
ensuring that we still selected the target number of establishments in each aggregate
industry and in each detailed industry. Under this design, we created 24 sampling cells, one
for each detailed industry including aircraft manufacturing and implicitly stratified within
each sampling cell for 24 geographic areas. The ORS sample was allocated to each industry
in proportion to industry employment, and the NCS sample allocations were set to the
current NCS sample sizes in each industry.
We were again satisfied with the sample sizes and employment counts of the ORS sample.
The sample distribution was not substantially different from that under the previous
approach (see section 4.2).
The NCS sample was also acceptable under this approach. For private industry, the overall
NCS detailed industry counts met the NCS targets. NCS counts by the 24 geographic areas
were close to the area counts using the current NCS sample design. The largest difference
between the two designs was an increase of 3.5% in Rest of West North Central Census
Region. Fourteen of the 24 NCS geographic areas had sample sizes that differed by less
than 1% compared to the current design. We believe that these differences were acceptable.
But we will need to conduct much further analysis to evaluate regional workloads,
anticipated response rates, impact on publications, impact on the number of quotes
available for estimation, etc. before making a choice to implement this design.
The table below shows the industries with largest percentage difference between the
sample sizes for the current NCS design and the sample sizes for the design where NCS is
a subsample of ORS using the Inside Out design:

Detailed Industry
Educational Services (Rest of)
Management of Companies and
Enterprises
Elementary and Secondary
Schools
Arts, Entertainment, Recreation
Mining

Average NCS Sample Size
Current NCS Subsample of
Design
ORS
58
57
73
74
91
92
104
103
86
87

Percent
Difference
-1.7%
1.3%
0.7%
-0.7%
0.6%

This table identifies the geographic areas with the largest percentage difference between
the sample sizes for the current NCS design and the sample sizes for the design where NCS
is a subsample of ORS using the Inside Out design:

Area
West North Central Census Division
San Jose-San Francisco-Oakland, CA
CSA
Detroit-Warren-Flint, MI CSA
New England Census Division

Average NCS Sample Size
Current
Subsample
NCS Design
of ORS
597
617
253
154
224

246
150
229

Percent
Difference
3.5%
-3.0%
-2.7%
2.3%

5. Independent Survey Design Options
In this design, we would select a sample of ORS establishments from the sample frame
using a sample design that works well for ORS. Each year, we would also select an
independent sample of NCS establishments from the same frame, using the current NCS
sample design. No special procedures would be implemented to control, reduce, or
maximize overlap between the two independent establishment samples. Under this survey
design, BLS will have two separate collection options, one in which ORS data elements
are collected for all NCS and ORS sampled establishments (i.e. Joint Collection) and one
in which ORS data elements are collected only for establishments selected in the ORS
sample (i.e. Independent Collection). With Independent Collection, some NCS
establishments will be asked to provide ORS data elements if they are also sampled in the
ORS sample but the rest of the NCS establishments would not be asked to provide ORS
data elements.
Under the independent design, the survey scope will match the NCS survey scope which
encompasses all businesses in the 50 States and the District of Columbia that are Private,
State Governments, or Local Governments. It will exclude private households and
agriculture industry. The ORS sample will be selected from the same frame as the NCS –
primarily the BLS Quarterly Census of Employment and Wages Database which is
compiled based on Unemployment Insurance filings by businesses across the country.
Unlike NCS which creates sampling strata based on 5 aggregate industries and 24
geographic areas, we will stratify by 24 major industry groups. ORS Sample allocation will
be proportional to employment in each of the major industries with no adjustments for

response rates or level of accuracy (since these are unknown at this time). If we are using
the Joint Collection model, the ORS-only allocations will be reduced by the number of
establishments selected in the NCS sample for each of the 24 industries before we select
the ORS independent sample. The decision about whether or not to collect ORS data from
NCS respondents has not yet been made and will be evaluated using the results from a
feasibility test on joint collection conducted in FY 2014 and from the pre-production test
results (see Section 8).
For ORS, we will select 24 independent establishment samples each year, one in each of
the 24 industry groups. We will implicitly stratify the sample by geographic area to ensure
a fair distribution of the sample across the country. Each sample will be selected using a
systematic PPS technique without replacement. For every establishment in the sample, a
sample of jobs will be selected with input from the respondent during initiation. This
process is also a PPS selection. When a job is selected, we collect data for all workers in
the job. The measure of size for each of the two stages of sampling is the employment in
the establishment or job.
We know that there will be some overlap of establishments from sample to sample and
from the NCS to ORS samples under this approach and that the amount of overlap is
important. So we selected 150 NCS simulated samples, 150 ORS simulated samples
assuming that ORS data will be collected from the NCS sample (joint collection with an
ORS sample size of 6,800 establishments a year), and 150 ORS simulated samples
assuming that ORS data will NOT be collected for every establishment in the NCS sample
(independent collection with an ORS sample size of 10,000 establishments a year). Both
sets of samples were split into three groups (Year 1, Year 2, and Year 3) assuming a three
year rotation.
In our research of the option under which ORS data are collected from all NCS
establishments, the amount of overlap between the ORS-only portion of the sample and the
NCS portion of the sample was minimal. Less than 5% of each NCS sample overlapped
with ORS at some point during the three-year sample design.
Amount of overlap in NCS sample between NCS and ORS-only establishments
Year of NCS Sample Design
Type of NCS-to-ORS
Overlap
Year 1
Year 2
Year 3
NCS unit is ORS-only unit in
ANY year of three-year design
4.24%
4.20%
4.20%
NCS unit is ORS-only unit in
SAME year of three-year
design
2.24%
2.11%
2.13%
Since the ORS-only portion of the sample is larger than the ORS/NCS portion, there was
less overlap when looked at from the ORS perspective.
The amount of overlap between ORS-only establishments and NCS establishments was a
bit higher under the option where ORS data are not intended to be collected from any NCS
establishments. The higher amount of overlap for this option was expected, because the
size of the ORS-only sample was larger at 10,000 establishments a year. The amount of
overlap is about twice as large for the NCS sample, and about a tenth of each NCS sample
overlaps with ORS at some point during the three-year sample design.

Amount of overlap in NCS sample between NCS and ORS-only establishments
Year of NCS Sample Design
Type of NCS-to-ORS
Overlap
Year 1
Year 2 Year 3
NCS unit is ORS-only unit in
ANY year of three-year
design
10.27% 10.11% 9.95%
NCS unit is ORS-only unit in
SAME year of three-year
design
6.51%
6.43%
6.35%
Some of the reasons for using an independent survey design option include:
 More flexibility to make changes in the future than any variation of the fully
integrated design option
 This option gives us the ability change either (NCS or ORS) design without
changing the other design.
 This option allows us to make changes to either design at the best point in time
for that design without forcing us to change both designs at the same time.
 This option allows us to move from Joint Collection to Independent Collection
for NCS establishments without changing the sample design, if necessary.
 In the unlikely event that BLS funding for ORS should go away, there would
be no impact on the NCS sample design.
 This option allows us to proceed with an optimal design for ORS to best meet the
needs of the SSA.
 This option does not have any impact on the NCS sample design since we can have
a different sample design for ORS than for NCS.
 This option allows us to have different rotation patterns for NCS and ORS such as
3 years for NCS and 3, 5, or 10 years for ORS, if desired.
 The joint collection option provides us with the ability to generate linked outputs
such as wages by amount of time sitting each day, if desired.
 The joint collection option results in lower data collection costs than an
independent data collection option.
Some of the challenges to using an independent survey design option include:
 If we implement the independent collection option, we would not have the ability
to generate linked outputs.
 Increase in resource demand during sample selection and systems development
and maintenance.
 Increased data collection costs for independent collection option when compared
to the joint collection option due to the larger ORS sample size.
 Increase in resource demand during data review and analysis if we are not able to
collect ORS data from all NCS respondents.
 Some added complexity to the computation of final ORS weights for estimate
calculations due to the different sample designs for joint collection option
 Some added complexity to the ORS variance computation methods due to the
different sample designs for joint collection option
Based on the evaluation of this design to date, it appears that the independent design for
sampling establishments will be the best choice for implementation. However, we would

still like to evaluate potential response rates for this design, develop expected levels for
published outputs, and further analyze the most appropriate way to handle large enterprise
firms with many establishments in the sample frame. This includes evaluating response
rates for all ORS samples and for establishments in the portion of the sample for which
both ORS and NCS data elements would be collected. In addition, we need to evaluate the
potential for nonresponse bias for these samples, especially if establishment response rates
fall below 80% during the pre-production test described in Section 8. Research is
proceeding in these areas and will be completed before final long-term production
decisions are made.

6. Separated Survey Design Options
Under this design, we would pull the NCS sample from the frame first. Then we would
remove those units from the ORS frame and draw a sample of ORS units from what is left
over. We would collect both NCS and ORS data from the establishments in the NCS
sample. We would collect only the ORS data from the establishments in the Rest of ORS
sample. So this only works with joint collection for NCS establishments.
Potential issues with this design:
 Establishments are sampled independently so one establishment within a large
enterprise in which all collection occurs centrally (usually at corporate
headquarters) could appear in the NCS part of the sample and other establishments
in the same enterprise could show up in the rest of ORS sample. Is there a way to
sample these large enterprises as a group (for NCS and ORS) to reduce this issue?
 This approach also requires a more complex weighting scheme for ORS estimation
than either the Fully Integrated or Independent Sample Design options, but we
believe that it is doable.
Due to a lack of resources, this option has not been fully evaluated for use in a production
environment using sample simulations. While it would eliminate the overlap between NCS
and ORS selected samples each year while providing flexibility in the design for the Rest
of ORS portion of the sample, there are several challenges with this option. First, it is quite
likely that establishments in a Rest of ORS sample could already be reporting data for a
prior NCS sample so the option does not eliminate all sample overlap. Also, the Rest of
ORS sample would not reflect a full sampling frame so the weighting process for
computing estimates would be more complex than under the previous design options.
Additionally, it is very likely that large enterprises would have establishments in both
portions (NCS and Rest of ORS) of the sample making is more difficult to collect data for
these large firms. At this time, the benefits of controlling initiation overlap are outweighed
by the other issues with the design so it will not be considered further for selection of
production samples.

7. OES-ORS Integrated Design
The Bureau of Labor Statistics conducts another survey which collects occupational
information from establishments in the United States, the Occupational Employment
Statistics survey. This is a mail survey which is used to generate mean annual wages and
employment for detailed occupations in the U.S. economy. Two samples are selected and
collected for this survey each year with data from the most recent six samples (three years)
used to compute annual estimates. The samples are drawn in a manner that ensures that

establishments appear in one and only one of the six samples used in each set of estimates.
Data on the occupations employed by each business establishment, the number of
employees, and the wages for those employees is collected via a mail survey. Since this
survey collects data about the occupational mix in each establishment, it may be
advantageous to use the collected data from the OES as the sample frame for ORS. This
could allow ORS to target specific occupations needed for disability determination
decisions as part of the sample design. However, there is some concern about the age of
the data and the potential survey error and lack of efficiency that this could introduce. So,
we have not yet begun an evaluation of this potential design but plan to do so at some point
in the future.

8. ORS Pre-Production Test Sample Design
Beginning in the fall of 2014, BLS plans to conduct a nation-wide pre-production test to
evaluate ORS survey processes and operations in a possible production environment at the
request of the Social Security Administration (SSA). Data collection and capture will run
for approximately six months and will conclude in the spring of 2015. A full evaluation of
the data elements captured for this pre-production test will be followed by an evaluation of
the processes, survey design, and other test program elements. In order to fully evaluate
the potential for implementing the ORS in a production environment, BLS will use the
Separated Sample Design approach to select the establishments included in the test as
described below. All of ORS Pre-production’s projected 2,550 sample establishments will
be collected once for all of the ORS data elements.
The ORS Pre-production sample will include a combination of both ORS-only
establishments as well as those that currently exist within the NCS (National Compensation
Survey). All units will be selected using a 2-stage stratified design with probability
proportional to employment sampling at each stage. The first stage of sample selection will
be a probability sample of establishments, and the second stage of sample selection will be
a probability sample of jobs within sampled establishments. For more information on the
current NCS sample design as well as factors explored for an ORS sample design, see the
American Statistical Association (ASA) papers by Ferguson et al titled, “Evaluating
Sample Design Issues In the National Compensation Survey” [10], “Update on the
Evaluation of Sample Design Issues in the National Compensation Survey” [11], and “State
and Local Government Sample Design for the National Compensation Survey" [12] as
well as the Federal Committee on Statistical Methodology (FCSM) paper by Rhein et al
titled, “Sample Design Considerations for the Occupational Requirements Survey” [13].
Each sample of establishments will be drawn by first stratifying the establishment sampling
frame by defined industry and ownership. The industry strata for private industry as well
as state and local government are shown on the next page and are based on the North
American Industry Classification System (NAICS).
After the sample of establishments is drawn, jobs will be selected in each sampled
establishment. The number of jobs selected in an establishment will range from 4 to 8
depending on the total number of employees in the establishment, except for government
and aircraft manufacturing units and units with less than 4 workers. In government, the
number of jobs selected will range from 4 to 20. In aircraft manufacturing, the number of
jobs selected will range from 4 for establishments with less than 50 workers to 32 for
establishments with 10,000 or more workers. In establishments with less than 4 workers,

the number of jobs selected will equal the number of workers. The probability of a job
being selected will be proportionate to its employment within the establishment.
The total ORS pre-production sample will consist of approximately 2,550 establishments.
The private portion of this sample will be approximately 85% (2,168) with one-third (716)
of that coming from current NCS sample units. The remaining two-thirds (1,452) of the
private sample will be selected from a national frame not to include any other existing NCS
sample units. This frame will be stratified by NAICS based on the 24 detailed industry cells
as defined above. The state and local government sample will be approximately 15% of
the total sample (382) with one-third (126) of the units selected from the existing NCS
sample and the remaining two-thirds (256) selected from a national frame not to include
existing NCS sampled establishments.
The sample allocation process starts with a total budgeted sample size. Since some of the
sample for the ORS Pre-production test will be selected from the NCS design, the same
industry definitions (based on ownership and NAICS as defined in charts below) will be
used to select both the NCS overlap sample as well as the ORS-only sample. Due to the
differences in the selection of the original NCS samples, different sampling strata will be
used for the samples selected from existing NCS samples than from the new ORS-only
samples. The sample will be allocated proportionally by ownership and industry using total
employment within each sample cell for ORS-only samples and total weighted
employment for samples drawn from NCS samples.
The ORS Pre-production test will select a sample consisting of both NCS sample units as
well as ORS-only units. The portion selected from the existing NCS sample units will use
systematic sampling with probability proportionate to measure of size. The measure of size
(MOS) will be the sample unit employment times its NCS sample weight.
For the ORS only sample, units will be selected from a frame that excludes all existing
NCS sample units. This frame will be stratified by ownership and industry as defined
above, with each sample cell being sorted by area (using the 24 area definitions in the NCS
design – see below), establishment employment, and establishment identification number.
These units will be selected using a probability proportional to size approach based on the
unit’s employment as it was reported to on the state unemployment file.
Sample weights will be assigned to each of the selected establishments in the sample to
represent the entire frame. Units selected as certainty will be self-representing and will
carry a sample weight of one. The sample weight for the non-certainty units will be the
inverse of the probability of selection.

ORS Pre-production Stratification for Private Industry
Aggregate
Industry
Education
Education
Education
Finance, Insurance
and Real Estate
Finance, Insurance
and Real Estate
Finance, Insurance
and Real Estate
Goods Producing
Goods Producing
Goods Producing
Health Care, incl.
Hospitals and
Nursing Care
Health Care, incl.
Hospitals and
Nursing Care
Health Care, incl.
Hospitals and
Nursing Care
Service Providing
Service Providing
Service Providing
Service Providing
Service Providing
Service Providing
Service Providing
Service Providing
Service Providing
Service Providing
Service Providing

Detailed Industry
Educational Services
(Rest of)
Elementary and
Secondary Schools
Junior Colleges,
Colleges and
Universities
Finance (Rest of)
Insurance
Real Estate,
Renting, Leasing
Mining
Construction
Manufacturing
Healthcare, Social
Assistance (Rest of)
Hospitals
Nursing and
Residential Care
Facilities
Utilities
Wholesale Trade
Retail Trade
Transportation and
Warehousing
Information
Professional,
Scientific, Technical
Management of
Companies and
Enterprises
Admin., Support,
Waste Management
Arts, Entertainment,
Recreation
Accommodation and
Food Services
Other Services (excl
Public
Administration)

Included
NAICS
Codes
61 (excl 61116113)
6111

#
Companies
in Universe

Sample
Size

78,008

11

16,899

14

8,023

15

52 (excl 524)

279,462

64

524

185,133

39

53

349,578

43

21
23
31-33

34,579
744,370
334,610

14
122
224

62 (excl 622,
623)

1,230,175

194

622

8,419

68

623

72,659

66

22
42
44-45

17,130
619,782
1,031,277

10
121
308

48-49

225,026

75

51

143,541

47

54

1,075,999

177

55

58,245

40

56

485,943

161

71

127,658

38

72

647,059

240

81 (excl 814)

563,765

76

6112, 6113

ORS Pre-production Stratification for State and Local Government Industry
Aggregate
Industry
Education
Education

Included
NAICS
Codes

Establishments
in Universe

Sample
Size

6111

62,349

150

6112, 6113

7,416

39

61 excl
6111-6113

1,281

1

51, 52-53

1,961

2

21, 23, 3133

6,350

4

Hospitals

622

2,377

21

Nursing Homes

623

1,679

5

Rest of Health

62, excl
622-623

8,546

9

42, 44-45,
48-49, 22

12,764

14

92 excl 928

107,694

122

54-56, 7172, 81 excl
814

18,462

15

Detailed Industry
Elementary and
Secondary
Education
Colleges and
Universities

Education

Rest of Education

Financial
Activities

Other Serviceproducing - Part A

Goods Producing

Goods-Producing

Health Care, incl.
Hospitals and
Nursing Care
Health Care, incl.
Hospitals and
Nursing Care
Health Care, incl.
Hospitals and
Nursing Care
Service Providing
Service Providing
Service Providing

Trade,
Transportation, and
Utilities
Public
Administration
Other Serviceproducing - Part B

9. Conclusions and Future Work
BLS has conducted much testing to evaluate various alternative sample designs and has
determined that there are statistically viable designs under both the Fully Integrated and
Independent Design options. Given the additional flexibility of the Independent Design
and the ability of implement that design with no change in the NCS design, the Independent
Design is currently being strongly considered for implementation with the first production
sample selection process for ORS. However, no decision has been made at this time about
the amount of acceptable overlap between the ORS and NCS samples. Selection of the
final design for use in an on-going production survey is dependent on many factors such
as response rates, anticipated survey accuracy levels, the impact of ORS data collection
from NCS respondents, and survey budget for which we do not yet have data. Response
rates, anticipated survey variances, and the impact of joint NCS and ORS data collection

will need to be evaluated using the results of the pre-production test to help guide the final
design decision. Survey budget numbers also need to be fully evaluated and vetted with
the Social Security Administration. These activities are scheduled to occur in Fiscal Year
2015 and will be shared when available.

References/Footnotes
[1] Social Security Administration, Occupational Information System Project,
http://www.ssa.gov/disabilityresearch/occupational_info_systems.html.
[2] U.S. Department of Labor, Employment and Training Administration (1991),
“Dictionary of Occupational Titles, Fourth Edition, Revised 1991”
[3] U.S. Department of Labor, O*Net Online, http://www.onetonline.org/
[4] U.S. Bureau of Labor Statistics (2008) BLS Handbook of Methods, Occupational
Employment Statistics, Chapter 3. http://www.bls.gov/opub/hom/pdf/homch3.pdf
[5] U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, National
Compensation Measures, Chapter 8. http://www.bls.gov/opub/hom/pdf/homch8.pdf
[6] U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, Occupational
Safety and Health Statistics, Chapter 9.
http://www.bls.gov/opub/hom/pdf/homch9.pdf
[7] U.S. Bureau of Labor Statistics (2013), “Occupational Requirements Survey, Phase 1
Summary Report, Fiscal Year 2013”, http://www.bls.gov/ncs/ors/phase1_report.pdf
[8] U.S. Bureau of Labor Statistics (2013), “Occupational Requirements Survey, Phase 2
Summary Report, Fiscal Year 2013”, http://www.bls.gov/ncs/ors/phase2_report.pdf
[9] U.S. Bureau of Labor Statistics (2013), “Occupational Requirements Survey, Phase 3
Summary Report, Fiscal Year 2013”, http://www.bls.gov/ncs/ors/phase3_report.pdf
[10] Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan (2010), “Evaluating
Sample Design Issues in the National Compensation Survey”, 2010 Proceedings of
the Section on Survey Research Methods, Alexandria, VA: American Statistical
Association.
[11] Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan (2011), "Update on the
Evaluation of Sample Design Issues in the National Compensation Survey", 2011
Proceedings of the Section on Survey Research Methods, Alexandria, VA; American
Statistical Association.
[12] Ferguson, Gwyn R., Ponikowski, Chester H., and McNulty, Erin (2012), “State and
Local Government Sample Design for the National Compensation Survey", 2012
Proceedings of the Section on Survey Research Methods, Alexandria, VA: American
Statistical Association.
[13] Bradley D. Rhein, Chester H. Ponikowski, Erin McNulty, (November 2013), “Sample
Design Considerations for the Occupational Requirements Survey,” FCSM Papers
and Proceedings, Federal Committee on Statistical Methodology Research
Conference
Any opinions expressed in this paper are those of the authors and do not constitute policy
of the Bureau of Labor Statistics or the Social Security Administration.