Attachment 7 - Testing the Collection of ORS ASA Papers and Proceedings

JSM 2013 - Government Statistics Section

Testing the Collection of Occupational Requirements Data October 2013
Gwyn R. Ferguson
Bureau of Labor Statistics, 2 Massachusetts Ave. NE, Washington, DC 20212

Abstract
In FY 2013, the Bureau of Labor Statistics (BLS) is working in conjunction with the
Social Security Administration (SSA) to design, develop, and carry out a series of tests to
assess the feasibility of using the National Compensation Survey (NCS) platform as a
means to accurately and reliably capture data that are relevant to the SSA's disability
programs. These tests will include but are not limited to collecting data on an indicator of
"time to proficiency," physical demand (PD) characteristics/factors of occupations, and
environmental conditions. This year, the BLS is evaluating survey design options,
developing protocols, aids, and a collection approach to meet SSA data needs, collecting
data to test and refine the protocols and aids, and documenting the work performed,
conclusions drawn and recommendations for future data collection. This work is being
done with three general phases of testing: an initial proof of concept phase, collection
protocol testing, and broad scale testing. This paper will describe the approaches being
used to test the feasibility of this potential data collection effort and describe the current
status of the testing efforts.
Key Words: Collection Testing; Field Testing; Sample Design; Collection Tools;
Survey Design

1. Introduction/Background
In addition to providing Social Security benefits to retirees and survivors, the Social
Security Administration (SSA) administers two large disability programs which provide
benefit payments to millions of beneficiaries each year. Final determinations about which
citizens, or claimants, are eligible to receive benefits are based on a five step process that
evaluates the capabilities of the worker, the requirements of their past work (prior job),
and their ability to perform work for any job in the U.S. economy. If an applicant is
denied disability benefits, SSA policy requires adjudicators to document the decision by
citing examples of jobs the claimant can still perform despite their restrictions (such as
limited ability to balance, stand, or carry objects)1.
For over 50 years, the Social Security Administration has turned to the Department of
Labor's Dictionary of Occupational Titles (DOT)2 as its primary source of occupational
information to process the disability claims3. SSA has incorporated many DOT
conventions into their disability regulations. However, the DOT was last updated in its
entirety in the late 1970’s, although a partial update was completed in 1991.
Consequently, the SSA adjudicators who make the disability decisions must continue to
refer to an increasingly outdated resource because it remains the most compatible with
their statutory mandate and is the best source of available data at this time.
1

See and Social Security Administration, Occupational Information System Project
See U.S. Department of Labor, Dictionary of Occupational Titles
3
See Occupational Information Development Advisory Panel, 2010
2

2291

JSM 2013 - Government Statistics Section

So, when an applicant is denied SSA benefits, SSA documents the decision by citing
examples of jobs that the claimant can still perform. But some of the jobs in the
American economy are not even represented in the DOT and other jobs, in fact many
often cited jobs, don’t exist in large numbers in the American economy any longer. For
example, a job that is often on the list for applicants is “envelope addressor.” If this job
still exists in our economy, there aren’t too many of them and the positions are hard to
find.
SSA has investigated numerous alternative data sources for the DOT such as adapting the
Employment and Training Administration’s O*NET4 (occupation information network),
using the BLS Occupational Employment Survey5 (OES), and developing their own
survey. But they were not successful with any of those potential data sources and turned
to the National Compensation Survey (NCS) at the Bureau of Labor Statistics.
NCS is a national survey of business establishments conducted by the Bureau of Labor
Statistics6 (BLS). Initial data from each sampled establishment are collected during a one
year sample initiation period. Many collected data elements are then updated each quarter
while other data elements are updated annually for at least three years. The data from the
NCS is used to produce the Employer Cost Index (ECI), Employer Costs for Employee
Compensation (ECEC), and various estimates about employer provided benefits.
Additionally, data from the NCS is combined with data from the OES to produce
statistics that are used to help the President’s Pay Agent and the Federal Salary Council
recommend changes in GS pay under the Federal Employee’s Pay Comparability Act.
In order to produce these measures, the NCS collects information about the sampled
business or governmental operation and about the occupations that are selected for
detailed study. Each sample unit is classified using the North American Industry
Classification System (NAICS). Each job selected for study is classified using the
Standard Occupational Classification system (SOC). In addition, each job is classified by
work level – from entry level to expert, nonsupervisory employee to manager, etc. These
distinctions are made by collecting information on the knowledge required to do the job,
the job controls provided, the complexity of the tasks, the contacts made by the workers,
and the physical environment where the work is performed. Many of these data elements
are very similar to the types of data needed by SSA for the disability determination
process.
All NCS data collection is performed by professional economists or statisticians,
generically called field economists. Each field economists must have a college diploma
and is required to complete a rigorous training and certification program before collecting
data independently. As part of this training program, each field economist must complete
several calibration exercises to ensure that collected data are coded uniformly no matter
which field economist collects or codes the data.
So SSA asked the NCS to partner with them under an annual interagency reimbursable
agreement to test the NCS ability to use the NCS platform to collect data on three groups
of information related to the demands of work for each occupation:

4

See O*NET OnLine at www.onetonline.org
See U.S. Bureau of Labor Statistics, Handbook of Methods, Chapter 3
6
See U.S. Bureau of Labor Statistics, Handbook of Methods, Chapter 8
5

2292

JSM 2013 - Government Statistics Section

x

physical demand (PD) characteristics/factors of occupations (e.g. strength, hearing, or
stooping)
x an indicator of "time to proficiency", and
x environmental conditions in which the work is completed.
If BLS is able to collect these data about work demands, SSA would have new and better
data to use in its disability programs. SSA cited three key advantages of using NCS to
provide this updated data:
x Reputation - SSA was impressed with the BLS reputation for producing high quality,
statistically accurate data that are trusted by our data users and follow statistically
accepted methods and principles.
x Trained Workforce – SSA was also impressed that NCS field economists have
experience collecting information about occupations in America’s work force and
collecting data similar to that needed by SSA.
x Survey Infrastructure - After attempting to develop their own survey (without
success), SSA was also impressed with the fact that NCS has a complete
management infrastructure across the country to manage and implement a new
survey to meet their data needs as well as systems and processes to support all the
steps of the survey.
In summer 2012, NCS began an effort to test our ability to collect these new data
elements using the NCS survey platform. Initial testing plans focused on developing
procedures, protocols, and collection aids using the NCS platform. The initial testing
phases were analyzed primarily using qualitative techniques. Once testing shows that it
is feasible to collect this data, more quantitative analysis and testing will be conducted.
The rest of this paper will describe the FY 2013 feasibility tests, provide some initial
results from those tests, discuss the work being done to develop a sample design for this
survey, and provide an overview of the steps of work for this effort.

2. FY2013 Test Plan and Initial Results
In FY 2013, the BLS worked to design, develop, and carry out a series of tests to assess
the feasibility of using the National Compensation Survey (NCS) platform as a means to
accurately and reliably capture data that are relevant to the SSA's disability program
purposes. These tests include but are not limited to collection of the following target data
elements:
1) An indicator of "time to proficiency," defined as the amount of time required by the
typical worker to learn the techniques, acquire the information, and develop the
facility needed for average job performance, comparable to the Specific Vocational
Preparation (SVP) used in the Dictionary of Occupational Titles (DOT).
2) Physical Demand (PD) characteristics/factors of occupations measured in such a way
to support SSA disability determination needs, comparable to measures currently
defined in the Selected Characteristics of Occupations7 (SCO).
3) Environmental conditions (e.g., high risk due to extreme temperatures) that replicate
as closely as possible those that the SSA currently uses and as listed in the SCO, or
specific revisions or additions to these factors.

7

See U.S. Department of Labor, Selected Characteristics of Occupations Defined in the
Revised Dictionary of Occupational Titles

2293

JSM 2013 - Government Statistics Section

This year, BLS is collecting data from a set of establishments and State and local
government units that are within the scope of the NCS but are not fully representative of
the universe of all businesses in the United States. The data collected during the tests
include SSA-specific data elements and selected NCS data elements but will not be used
to generate any weighted estimates representative of the population of interest.
BLS conducted three phases of field testing in FY 2013: an initial proof of concept
phase, a collection protocol testing phase, and a broad scale testing phase. The next three
sections of this paper will describe the goal, approach, and initial results of each of these
phases of testing.

2.1 Initial Proof of Concept Testing – Phase 1
The primary goal of this phase of testing was to ensure that the BLS field economists
knew how to describe the survey and ask for the new data elements. In addition, the BLS
created and tested an initial set of data collection protocols and a preliminary set of data
collection aids.
Field collection for Phase 1 testing was conducted in the greater Washington, D.C. area
from November 28, 2012 through December 6, 2012. The target sample size for Phase 1
testing was 25 – 30 establishments representing as broad a mix of industries as possible
given the size and time constraints of the test. Experienced NCS field economists (FE)
from the BLS regional offices collected the data and each interview was observed by a
BLS national office staff member or an SSA representative.
Interviewer training for Phase 1 collection followed regular BLS practices and consisted
of three components:
x Self-study – BLS field economists read several SSA background materials,
researched the SSA website, and reviewed the Dictionary of Occupational Titles
(DOT), among other documents.
x In-person training – In an ORS Orientation meeting in October, 2012, project
managers introduced BLS field economists to the ORS program and SSA
representatives presented an overview of the disability determination process and the
required data elements. At the Phase 1 Test Kick-off Meeting in November, 2012,
SSA provided additional background about the disability adjudication process
(including physical residual functioning capacity assessment), and the BLS field
economists received training and a document with detailed instructions for this phase
of testing.
x On-the-job training (OJT) – Leading up to and throughout the Phase 1 Test fielding
period, BLS field economists engaged in a number of activities designed to reinforce
formal ORS training concepts. Prior to data collection, each field economist
conducted at least two practice interviews with BLS staff unaffiliated with the ORS
and NCS (e.g., Human Resources personnel, program managers), and participated in
a calibration exercise in which all ORS field economists observed the same practice
interview, individually recorded ORS information, and then compared and discussed
their answers with one another. During data collection, OJT was provided through
formal daily debriefing sessions and informal conversations between BLS field
economists and other staff in which ‘lessons learned,’ ‘best practices,’ and
challenging issues were identified.

2294

JSM 2013 - Government Statistics Section

BLS field economists tested three alternative collection approaches. Each approach was
designed to collect the required data elements (i.e., Specific Vocational Preparation
(SVP), Physical Demands (PD), and Environmental Conditions (EC)), but they varied in
their format, question flow, and collection method.
x Collection Approach A was designed to obtain the occupational requirements one
occupation at a time.
x Collection Approach B also collected occupational requirements one occupation at a
time. The main difference was that the response options in Approach B were grouped
together under a single question stem where appropriate. The purpose of this
approach was to obviate the need for field economists to repeat the same question for
related items.
x Collection Approach C collected the occupational requirements of all selected
occupations simultaneously.
In order to test the effectiveness of each collection approach, field economists were asked
to use a different collection approach during each of their three personal visit interviews.
A set of aids for respondents and interviewers was also developed. The field economist
visual aid listed the SSA-provided definition and examples for each data element, as well
as definitions of the measurements of frequency (i.e., never, occasionally, frequently,
constantly). BLS field economists were encouraged to consult this aid during the
interview, and to share it with the respondent as necessary. An additional, two-sided
respondent visual aid provided a place for BLS field economists to list each of the jobs
being surveyed, the frequency definitions, and examples for the noise intensity level
response categories. BLS field economists were told to use the respondent aid during
every interview. Finally, BLS field economists were provided with a document that
contained answers to questions the respondent may ask; they familiarized themselves
with this document prior to the interview, and could refer to it in the event that questions
arose during the appointment.
Upon completion of the ORS data collection, respondents were asked to complete a short
questionnaire to gauge their reactions to the survey. Daily debriefing sessions were held
with field economists, observers, and other staff for the purposes of discussing
interviewing experiences, identifying potential issues with the materials and procedures,
and sharing lessons learned. A final end-of-phase debriefing session summarized the
major findings identified during the test field period and allowed for expanded discussion
of these and other issues between ORS and SSA staffs.
The results of the Phase 1 test were very promising overall.8 Test objectives were
successfully met and ORS had a strong foundation on which to build for future
development and testing.
x BLS field economists completed interviews with 27 establishments, collecting
detailed information for 104 occupations. It took between 8 and 10 minutes to collect
information for each occupation.
x BLS field economists had minimal difficulty gaining cooperation of sampled
establishments. Many BLS field economists noted that one of the most effective
pieces of information to gain cooperation was the high name recognition of the SSA.
8

See U.S. Bureau of Labor Statistics, ”Occupational Requirements Survey, Phase 1
Summary Report, Fiscal Year 2013”

2295

JSM 2013 - Government Statistics Section

x
x
x
x
x
x
x

Most data were collected from traditional NCS respondents such as human resource
directors, small business owners, and location managers.
Cooperating establishments were able to answer the vast majority of questions asked
in the survey resulting in a very high item response rate.
The training approach used in Phase 1 worked very well and will be repeated in
subsequent tests.
The use of daily debriefings where BLS field economists, observers, and other staff
could exchange information and suggestions about collection issues was particularly
successful.
The materials prepared to aid ORS cooperation were very well received by
respondents.
It is not enough to simply record the respondent’s answer; it is important to verify
consistency across questions. For example, is it possible to sit 6 hours a day and also
climb stairs frequently?
Certain words and concepts proved to be unclear or confusing to respondents. These
included “required,” “accommodations” and prior work experience.

2.2 Collection Protocol Testing – Phase 2
The primary goals of Phase 2 testing were to expand the number of BLS field economists
that understand how to describe and collect the ORS data elements from respondents and
to obtain information on new additions to the Phase 1 collection procedures. Some key
additions to the tests were:
1. Probability Selection of Occupations (PSO) – used a disaggregating technique to select
occupations randomly from a list of employees working for an establishment;
2. Rarely /“Almost Never” – offered an additional frequency choice in order to better
capture the existence of elements that occur very infrequently (approaching never);
3. Phone Interviews – provided guidance for the use of this collection method;
4. Dictionary of Occupational Titles (DOT) Coding – matched establishment jobs to the
occupational list within the DOT; and
5. Work Setting – determined the work environment in which the job occurs.
Another goal of the Phase 2 test was to assess the effectiveness of revised ORS collection
tools. BLS staff created streamlined collection tools based on the results of the Phase 1
test and the Phase 2 test objectives and evaluated a modified set of data collection
protocols and data collection aids.
Field collection for Phase 2 testing was conducted in the Indianapolis - Anderson Columbus, IN metropolitan area (January 28, 2013 – February 20, 2013) and the Portland
- Vancouver - Hillsboro, OR-WA metropolitan area (February 28, 2013 - March 21,
2013). Probability Selection of Occupations (PSO) was used to determine the occupations
selected for collection of detailed job requirement information.
Experienced NCS field economists from each BLS regional office collected the data and
more than 40 percent of the interviews were observed by BLS staff. The senior field
economists who had participated in Phase 1 testing also collected in both Phase 2 test
cities and served as mentors to the field economists new to ORS. Interviewer training for
Phase 2 collection followed regular BLS practices and consisted of five components:
x Self-study – The self-study program from Phase 1 was repeated.
x Webinars – In the week prior to the start of test data collection, field economists
participated in web-based training sessions in which they were in given an overview

2296

JSM 2013 - Government Statistics Section

x

x

x

of the SSA disability process and information about the ORS test objectives,
procedures, and materials. There were two webinar training sessions prior to
Indianapolis collection and one prior to Portland collection. The imbalance in
webinar sessions between test cities was offset by an additional day of in-person
training in Portland.
In-person training – In-person training consisted of: review of the technical
memorandum; practice with the data capture spreadsheet; an overview of instrument
edits and data analysis objectives; a calibration exercise; mock interviews; and a
mock interview debrief session.
Mentoring - In addition to formal training, each field economist new to ORS was
assigned a mentor who collected in prior ORS testing. The mentor served as the field
economists’ primary resource for technical advice. Mentors made all first-week
appointments for the mentee. Mentees observed mentors conducting two interviews
prior to conducting their own interviews, and then were themselves observed by their
mentor on their first two interviews. Mentors also reviewed the mentees’ write-up of
their initial schedules.
On-the-job training (OJT) – The OJT program from Phase 1 was repeated.

Three paper data collection tools were tested in Phase 2. Each version was designed to
collect the required data elements: Job Details (e.g., title, occupational employment size,
SOC/DOT code, worker characteristics, and work environment); Specific Vocational
Preparation (SVP); Physical Demands (PD); and Environmental Conditions (EC).
x A Single Quote Tool was designed to obtain the occupational requirements one
occupation at a time. Definitions for select terms were provided at the bottom of the
page.
x A Multiple Quote Tool was designed to collect the occupational requirements of all
selected jobs simultaneously. Field economists were instructed to ask each item about
all occupations before proceeding to the next item.
x An Advanced Multiple Quote Tool was developed for small-scale feasibility testing
in Portland. This tool changed the order in which the elements were asked (e.g.,
asking Environmental Conditions questions prior to Physical Demand items), and
was designed to allow field economists to first determine whether each job required
certain characteristics or not (e.g., does the job require the worker to be exposed to
extreme heat or not?), and then to go back and administer follow-up questions only
for eligible occupations (e.g., how often does the job require the worker to be
exposed to extreme heat?).
Based on feedback received from field economists and observers during the Indianapolis
data collection, all three versions of the collection tools were modified prior to the start of
Portland collections in an effort to improve the understanding of questions related to
literacy, reaching, the frequency of lifting/carrying items, and crawling/kneeling/stooping
questions.
Use of the Advanced Multiple Quote tool was restricted to limited testing by senior BLS
field economists in Portland. For all remaining Phase 2 interviews, the decision of
whether to use the Single Quote tool or standard Multiple Quote tool (or combination
thereof) was left up to the individual field economists, though the Multiple Quote tool
was recommended for schedules in which it appeared likely that many of the selected
occupations would have similar requirements. Field economists were instructed to read

2297

JSM 2013 - Government Statistics Section

the questions as worded on the tool for testing consistency, but were permitted to provide
additional explanations and ask follow-up questions as necessary.
In addition to these collection forms, a number of aids were developed to help clarify the
survey task and definitions. Respondent visual aids provided frequency definitions and
examples for the noise intensity level response categories and of the type of frequency
and weight information that respondents should consider when answering the
lifting/carrying items. Field economist aids provided definitions and clarifying examples
of ORS elements and answers to commonly asked questions. Field economists were
encouraged to consult these aids during the interview and to share them with respondents
as necessary.
Both in-person and remote data collection were tested in Phase 2. The preferred method
of collection was personal visit; if the respondent indicated a willingness to provide
information by personal visit and another method (e.g., phone, email), it was mandatory
to select personal visit. In the event that a respondent refused or was unable to schedule
an in-person appointment, field economists were permitted to collect information over the
phone or by email. In addition, a small group of schedules was assigned to phone
collection conducted by senior BLS field economists in each Phase 2 test city. Remote
collection protocols were very similar to those for in-person collection. Field economists
were required to use the collection tools, collect all of the ORS Phase 2 elements, ask
questions as worded, and probe unclear answers or in situations in which respondents’
answers did not match expected job patterns. They also were encouraged to frequently
refer to the field economist aids to provide more detailed definitions, explanations, and
examples to the respondent as needed.
The results of the Phase 2 test overall were very promising9. Test objectives were
successfully met and ORS had a strong foundation on which to build for future
development and testing.
x BLS staff completed interviews with 227 establishments from a wide variety of
industry groups, collecting detailed job information for 1,094 occupations.
x The materials prepared to explain the background, purpose, and methods of the Phase
2 test and aid ORS cooperation were well received by respondents.
x Probability Selection of Occupations was implemented successfully in more than
90% of Phase 2 interviews.
x Most data were collected from traditional NCS respondents such as human resource
directors, small business owners, and location managers.
x Cooperating establishments were able to answer the vast majority of questions asked
in the survey, resulting in a very high item response rate. This was true of both
personal visit and phone interviews.
x The training approach used in Phase 2 worked well and will be used in subsequent
tests.
x The use of daily debriefings where BLS staff could exchange information and
suggestions about issues was particularly successful.
x Certain words and concepts continued to be unclear or confusing to respondents.
These include “required,” “accommodations,” and time to “average performance.”

9

See U.S. Bureau of Labor Statistics, ”Occupational Requirements Survey, Phase 2
Summary Report, Fiscal Year 2013”

2298

JSM 2013 - Government Statistics Section

x

Some respondents struggled to provide frequency estimates for tasks that were short
but repetitive, or which varied from day to day or between employees in a given
occupation, and for nonstandard work shifts (e.g., 12 hour shifts).

2.3 Broad Scale Testing – Phase 3
The main objective of Phase 3 was to test whether ORS field economists from across the
country could collect all of the ORS data elements and occupational wages and leveling
information in a uniform and efficient manner. In addition, BLS ran supplemental tests to
assess the feasibility of Central Office Collection (COC), assess the feasibility of joint
collection of ORS and Employment Cost Index (ECI) elements, and identify more
efficient ways to conduct the ORS interviews. Throughout these activities, BLS
continued to refine its methods and tools to advance the broad goal of meeting SSA’s
data needs in an efficient and uniform manner. To this end, Phase 3 activities included
testing of a new data capture/write-up system and new schedule review procedures.
Feasibility of COC collection: Some large firms provide data for all sampled
establishments across the country to the NCS from a central office in a single location.
For the traditional ECI data elements, central office reporting can be a straight-forward
activity for large firms when wage and employee benefit data are stored in computer
systems that are managed centrally. But NCS needs to know if information about the
environmental conditions and physical demands of work will be available from a central
location. The goals of this supplemental test were to assess the extent to which remote
respondents could provide ORS data; determine whether current NCS Central Office
Authorization (COA) and COC collection procedures will need to be changed to
accommodate ORS; develop an initial list of best practices for handling COC/COA
establishments; and document situations where ORS data are not available through
central office respondents.
Feasibility of Joint ORS/ECI Collection: Although a final sample design has yet to be
developed, there will almost certainly be establishments that will be asked to provide data
to both the NCS (in support of the ECI, ECEC, and benefits products) and ORS. It will be
important to ensure that NCS field economists can obtain high quality data for both
surveys when needed. The goals of this supplemental test were to develop an initial list of
best practices and evaluate the feasibility of incorporating fully integrated benefits
collection in future testing.
Efficient Collection Interviewing: Throughout Phases 1 and 2 of the ORS testing efforts,
field economists were required to ask every question of every respondent using the exact
same wording in a structured interview process. This approach is very different than the
typical NCS data collection experience of conversational interviewing and seemed to take
longer than many field economists felt was necessary to collect all the needed data for
ORS. So a small Efficiency Burden supplemental test was added to Phase 3 testing to
identify some collection best practices that produce high quality data but were more
efficient than the current structured interview; and to determine the impact of modified
collection protocols on data quality.
Test Collection Activities: Field collection for Phase 3 testing was conducted in areas
surrounding six cities: Nashville, TN; Providence, RI; Cincinnati, OH; Kansas City, MO;
Pittsburgh, PA; and Orange, CA. Tests ran concurrently in all of these cities between
April 22, 2013 and July 19, 2013. In addition, some data collection was done in
Baltimore, MD in order to provide SSA staff the opportunity to observe interviews.

2299

JSM 2013 - Government Statistics Section

Phase 3 data were collected by more than twice the number of experienced NCS field
economists from BLS national and regional offices than participated in Phase 2 testing.
The field economists who had participated in previous ORS phases served as field data
collectors in Phase 3 and as mentors to the field economists new to ORS. Interviewer
training for Phase 3 collection followed regular BLS practices and consisted of five
components:
x Self-study – The self-study program from Phase 2 was repeated.
x Webinar – The webinar program from Phase 2 was repeated.
x In-person training – As in the second part of Phase 2 testing, this occurred over a two
day period in each test city and consisted of the same material as in Phase 2.
x Mentoring – The Phase 2 mentoring program was repeated.
x On-the-job training – Throughout the Phase 3 Test fielding period, field economists
engaged in a number of activities designed to reinforce formal ORS training
concepts. During data collection, on-the-job training was provided through daily
debriefing sessions, formal technical guidance. In addition, informal conversations
between field economists and project staff helped to identify challenging issues,
lessons learned, and best practices.
For the Phase 3 broad scale test, field economists used separate tools to collect SVP and
leveling-related data elements and information on Physical Demands and Environmental
Conditions. A two-sided, paper Quote Info Leveling and SVP tool collected information
about education, experience, training, and core function requirements for a specific job,
and captured job characteristics such as union/non-union, full time/part time, supervisory
duties, number of incumbents and work setting. On the back of the tool were fields to
capture the amount of time the incumbent spent driving, the type of vehicle driven, and
information regarding Generic Leveling as captured in NCS.
To capture the Physical Demands and Environmental Conditions elements, field
economists had three paper data collection tool options: a Single Quote tool, a Multiple
Quote tool, and an Advanced Multiple Quote tool. Each tool was designed to collect all
of the required data elements, but they differed in their visual format and collection
method. The Single Quote tool was configured to collect the occupational requirements
one occupation at a time. The Multiple Quote tool was designed to collect this
information by element for all selected jobs simultaneously. The Advanced Multiple
Quote tool was designed to allow field economists to first determine whether each job
required certain elements or not (e.g., does the job require the worker to be exposed to
extreme heat or not?), and then to go back and administer follow-up questions only for
eligible occupations (e.g., how often does the job require the worker to be exposed to
extreme heat?).
Use of the Advanced Multiple Quote tool was restricted to field economists who
collected in prior ORS test phases. For all remaining Phase 3 test interviews, the decision
of whether to use the Single Quote tool or Multiple Quote tool (or combination thereof)
was left up to the individual field economists. Regardless of the tool selected, field
economists were instructed to read the questions as worded on the tool for testing
consistency, but were permitted to reorder the elements and provide additional
explanations and ask follow-up questions as necessary.

2300

JSM 2013 - Government Statistics Section

In addition to Phase 3 collection tools, a number of aids were developed to help clarify
the survey task and definitions. A Respondent Visual Aid provided frequency definitions.
The Field Economist Reference Guide provided definitions and clarifying examples of
ORS elements. Field economists were encouraged to consult these aids during the
interview and to share them with respondents as necessary.
Both in-person and remote data collection were tested in Phase 3. The goal was to collect
data through personal visit for 85% of the establishments and through telephone for the
remaining 15% of schedules. In the event that a respondent refused or was unable to
schedule an in-person appointment but was willing to participate, additional phone
interviews were permissible. Phone collection protocols were very similar to those for inperson collection. Field economists were required to use the collection tools, collect all of
the ORS Phase 3 elements, ask questions as worded, and probe unclear answers or
situations in which respondents’ answers did not match expected job patterns.
Field economists entered data from the collection tools into a new web-based data
capture system on a flow basis. The data capture tool was designed to permit easy data
entry by field economists, the ability to review the captured data, and capacity to tabulate
results. Review parameters were developed for Phase 3 and were used to evaluate data
elements for internal consistency. Specifically, three internal consistency review types
(i.e., data capture system edits; secondary review edits; cross-schedule edits) were used in
the data review and analysis. The parameters identified expected relationships within
individual ORS data elements and compared these against DOT coding.
Initial Phase 3 Results10: In Phase 3, BLS field economists completed interviews with
638 companies representing a total of 667 establishments. Personal visit interviews were
conducted for 92.3% of the schedules, 6.3% were conducted by telephone, and 1.2%
involved some combination of phone and personal visit collection. During these
interviews, field economists collected ORS and NCS data elements for a total of 3,259
sampled occupations, representing 496 eight-digit classification levels within the SOC
structure.
The field economists were very positive about the set of materials and resources that were
developed or refined for Phase 3 to explain the background, purpose, and methods of the
test and to secure respondent participation. In particular, a two-sided ORS flyer with a
supporting letter from SSA on back was widely used, as were area economic summary
sheets and an introductory letter tailored to industry sectors and BLS region.
Phase 3 training activities were effective in conveying the key ORS concepts and
procedures. Field economists particularly valued the pre-collection ‘mock’ interviews,
the debriefing following the mock interview, and the personal-visit mentoring
appointments. The daily and weekly debriefing sessions continued to be an excellent
training tool as well, offering field economists the opportunity to exchange information,
clarify their understanding of materials, and share suggestions about collection issues.
The Phase 3 mentoring process also was well received, though field economists
recommended that in future rounds of testing mentees should observe schedules before
training and have additional time to write up their schedules and have them reviewed
before independent collection. Field economists recommended that ORS expand training
10

See U.S. Bureau of Labor Statistics, ”Occupational Requirements Survey, Phase 3
Summary Report, Fiscal Year 2013”

2301

JSM 2013 - Government Statistics Section

content and explore different delivery options to maximize engagement and decrease the
time needed to independently collect ORS.
The Multiple Quote tool – which collected information by element for all jobs
simultaneously – was the most popular choice among the field economists collecting data
in Phase 3 and seemed to provide a good balance between collecting quality data and
reducing respondent burden. Field economists would have liked additional space on the
tool to record explanatory notes. As in previous rounds of tests, certain words and
concepts continued to be unclear or confusing to respondents (e.g., required; time to
average performance). Field economists relied on their professional understanding and
professional judgment when interpreting respondent-provided information and
administering follow-up probes. Applications of professional judgment were most
common when there were apparent inconsistencies among elements, with combination
jobs and combined tasks, and when respondents reported higher-than-expected frequency
estimates (e.g., especially for speaking and reaching). Field economists emphasized that
the use of professional judgment is essential when collecting ORS information and
recommended additional training be developed on this topic to ensure that it is applied
consistently and documented with sufficient detail. They noted the need for additional
refinements to question wording, probing guidelines, and development of better examples
for each physical demand and environmental factor (possibly tailored to specific
industries or establishment types). Despite these areas for improvement, respondents
reported quite positive responses about the ORS initiative and experience, and there was
consensus among field economists that the ORS materials and procedures significantly
improved between test phases and throughout Phase 3.
The results of the small-scale supplemental tests conducted in Phase 3 also were very
promising. For example, although only 60 schedules were completed in the efficiency
burden test, the results point to the potential benefits of asking questions about
combination elements (i.e., driving, keyboarding, and writing) at the start of the interview
to provide better context for the subsequent elements and improve the application of
professional judgment. Together, the findings from these supplemental tests underscore
the importance of continuing to develop strategies for increasing the efficiency of ORS
element collection, collecting high-quality NCS and ORS data simultaneously, and
securing ORS elements from large employers with multiple locations.
The introduction of a new internet-based system for data capture in Phase 3 also yielded
valuable information. Overall, field economists and reviewers liked the web-based
system, but cited the inability to use the system without internet access as a shortcoming.
Some field economists expressed the need for a wage-import feature for bigger
establishments and the ability to enter compound wages. Field economists and reviewers
also recommended that additional development and testing be carried out in fiscal year
2014 to better integrate the data capture system with other components (e.g., the
appointment calendar, review questions, time-reporting spreadsheets, etc.).
The main objective of the work that was completed as part of the ORS project in fiscal
year 2013 was the assessment of whether it was feasible for BLS to collect data relevant
to the SSA’s disability determination program. The results of this broad scale test suggest
that the collection of the ORS data elements using a probability selection of occupations
in conjunction with selected NCS data elements is viable.

2302

JSM 2013 - Government Statistics Section

3. FY 2013 Survey Design Research and Administrative Activities
In addition to the three phases of feasibility testing, BLS began research and evaluation
for several other activities associated with creating new federal surveys. This work
broadly fits into three categories, sample design research, estimation and data
dissemination vision, and administrative activities.
Research into potential survey designs has begun with the focus on two general survey
designs. Under the first design being evaluated, a sample of establishments to support the
ORS would be selected each year. The sample of NCS establishments would be selected
from the ORS sample yielding a fully integrated survey where all NCS establishments
would be asked to provide ORS data in addition to the NCS data elements. Under the
second design, two independent samples would be selected each year – one for ORS and
a second one for NCS. When the same establishment appears in both samples, the
respondent would be asked to provide data for both surveys. Initial alternatives for both
designs have been identified and are being tested. Research has also looked into the types
of occupations for which SSA needs data and the industries in which those occupations
appear and the various potential survey rotation schemes. While this research continues,
we have learned that it is possible to build a single survey platform to support both the
NCS and ORS samples and collection processes. However, it has been difficult to
identify a single sample design to meet the different estimation goals of the two surveys.
For NCS, it is critical to be able to publish estimates by industry and Census Regions and
Divisions. ORS estimates are needed for detailed occupations across the entire country.
So, research continues and alternatives are being explored which could permit separate
sample designs with joint collection for establishments selected by both surveys.
BLS has developed an initial vision for the types of estimates to be produced for the ORS
survey which is still subject to refinement based on input from SSA and other potential
data users. In general, this would include estimates of the proportion of workers in each
occupation that are required to perform a given physical demand or are exposed to a
given environmental condition with different frequencies. For example, one estimate
could be the proportion of workers in a given occupation who are exposed to toxic
chemicals less than 1/3 of the time or the proportion of workers in that occupation who
need to reach above their shoulders more than 2/3 of the time. All estimates will be
generated with a measure of standard error or relative standard error as appropriate and
will be reviewed to ensure that they meet standard confidentiality restrictions. Initial
ideas for disseminating the data are being developed based on stakeholder needs. Plans at
this time call for making all data that meets publication criteria available to all users in a
variety of forms, possibly including tables, query tools, charts, and graphs.
BLS has also completed many administrative activities related to the ORS this year
including procuring clearance from the Office of Management and Budget for the tests,
reporting on the project to the SSA and Congress, learning about the disability
community and potential data users by attending several professional conferences, and
planning for future testing activities.

4. Future Work
While the FY 2013 testing was successful, there are several areas where further testing is
needed before BLS will be ready to move into production with the Occupational
Requirements Survey. In addition, SSA has notified BLS that some new data elements

2303

JSM 2013 - Government Statistics Section

are desired for the disability determination process, specifically the mental-cognitive
demands of work. NCS would also like to validate the tasks performed for each
occupation to help field economists understand the jobs being collected better and help
explain any data estimates that do not match expectations obtained from other data
sources including the DOT and O*NET. In FY 2014, NCS plans to conduct several
additional feasibility tests, each focused on a specific issue as listed below.
x Design, develop, and conduct a test to determine how best to collect occupational
requirements data elements and NCS data elements from the same establishment.
x Design, develop, and conduct a test to determine how best to collect occupational
requirements data elements from America’s largest firms.
x Design, develop, and conduct tests for the new mental and cognitive demands of
work data elements for use in SSA’s disability evaluation process, and evaluate the
use of occupational task lists as developed by the Employment and Training
Administration’s O*NET program during data collection.
x Design, develop, and conduct a test to refine the methods to develop more efficient
approaches for data collection as identified during FY 2013 testing.
x Design, develop, and conduct a test to determine how best to collect occupational
requirements data elements when a personal visit is not optimal due to respondent
resistance, collection costs, or other factors.
x Capture and evaluate changes in data coding as a result of observing the work
environment, the sampled occupation, or both.
At the conclusion of Feasibility Testing, the BLS will conduct a nationwide test to
evaluate all aspects of the survey in a possible production environment. Data collection
and capture for this Pre-Production Test are expected to run for two or more quarters and
will conclude in FY 2015. The data capture portion of this test will be followed by a full
evaluation of data, processes, survey design, and other program elements. This
evaluation will include qualitative reviews of the processes and procedures as well as
quantitative analysis of the collected data, response rates, respondent burden, test
estimates, and other survey processes.
The BLS will use data collected in testing, input from SSA, analysis from internal BLS
experts, and/or its institutional survey knowledge to research various issues identified in
FY 2013. These research projects include the following:
x Sample selection, data review, estimation, validation, and other issues related to
survey design;
x Development of measures of reliability of estimates;
x Measurement of survey error; and
x Resolution of how best to handle jobs whose duties and responsibilities cross
multiple occupational classifications.
If testing is successful, BLS plans to move into a production mode in FY 2015 with the
goal of releasing some initial estimates in FY 2016.

5. References
Ferguson, Gwyn R., Coleman, Joan, and Ponikowski, Chester (2011), “Update on the
Evaluation of Sample Design Issues in the National Compensation Survey”, 2011
Proceedings of the Section on Survey Research Methods, Alexandria, VA: American
Statistical Association.

2304

JSM 2013 - Government Statistics Section

Ferguson, Gwyn R., Ponikowski, Chester, and Coleman, Joan (2010), “Evaluating
Sample Design Issues in the National Compensation Survey”, 2010 Proceedings of the
Section on Survey Research Methods, Alexandria, VA: American Statistical
Association.
Ferguson, Gwyn R., Ponikowski, Chester, and McNulty, Erin (2012), “State and Local
Government Sample Design for the National Compensation Survey", 2012 Proceedings
of the Section on Survey Research Methods, Alexandria, VA: American Statistical
Association.
Occupational Information Development Advisory Panel, “Findings Report: A Review of
the National Academy of Sciences Report - A Database for a Changing Economy:
Review of the Occupational Information Network (O*NET)”, June 28, 2010, Report to
the Commissioner of Social Security,
http://www.ssa.gov/disabilityresearch/documents/COMPLETE%20FINAL-Findings%20Report%20OIDAP%20062810.pdf.
Office of Management and Budget, "Guidance on Agency Survey and Statistical
Information Collections" Memorandum, January 20, 2006,
http://www.whitehouse.gov/omb/assets/omb/inforeg/pmc_survey_guidance_2006.pdf.
Office of Management and Budget, "Standards and Guidelines for Statistical Surveys",
September 2006,
http://www.whitehouse.gov/omb/assets/omb/inforeg/statpolicy/standards_stat_surveys.
pdf.
Social Security Administration, Occupational Information System Project,
http://www.ssa.gov/disabilityresearch/occupational_info_systems.html.
U.S. Bureau of Labor Statistics (2013) BLS Handbook of Methods, National
Compensation Measures, Chapter 8. http://www.bls.gov/opub/hom/pdf/homch8.pdf.
U.S. Bureau of Labor Statistics (2008) BLS Handbook of Methods, Occupational
Employment Statistics, Chapter 3. http://www.bls.gov/opub/hom/pdf/homch3.pdf.
U.S. Bureau of Labor Statistics (2013), “Occupational Requirements Survey, Phase 1
Summary Report, Fiscal Year 2013”, http://www.bls.gov/ncs/ors/phase1_report.pdf.
U.S. Bureau of Labor Statistics (2013), “Occupational Requirements Survey, Phase 2
Summary Report, Fiscal Year 2013”, http://www.bls.gov/ncs/ors/phase2_report.pdf.
U.S. Bureau of Labor Statistics (2013), “Occupational Requirements Survey, Phase 3
Summary Report, Fiscal Year 2013”, http://www.bls.gov/ncs/ors/phase3_report.pdf.
U.S. Department of Labor, Employment and Training Administration (1993), “Selected
Characteristics of Occupations Defined in the Revised Dictionary of Occupational
Titles”.
U.S. Department of Labor, Employment and Training Administration (1991), “Dictionary
of Occupational Titles, Fourth Edition, Revised 1991”.

Any opinions expressed in this paper are those of the author and do not constitute policy
of the Bureau of Labor Statistics.

2305