Attachment 9 - ORS, Phase 2 Summary Report

U.S. DEPARTMENT OF LABOR
BUREAU OF LABOR STATISTICS

Occupational
Requirements
Survey
Phase 2 Summary Report

1|Page

Executive Summary
The National Compensation Survey (NCS) recently completed Phase 2 of the Occupational Requirements Survey (ORS)
tests conducted in cooperation with the Social Security Administration (SSA). The main objectives of this test were to
evaluate ORS collection protocols and aids that had been updated since Phase 1 testing (e.g., streamlined collection
tools; implementation of a probability-based establishment selection method; refined frequency questions; limited
phone collection), and to assess ORS collection outside the DC metropolitan area using an expanded number of BLS field
economists. The results of the Phase 2 test demonstrate the effectiveness of the revised materials and procedures and
the continued viability of BLS collection of data relevant to the SSA’s disability program. Respondents agreed to
participate in the test, BLS field economists were able to capture the required data from typical NCS respondents, and
individual data element response rates were very high.
Field collection for Phase 2 testing was conducted in the Indianapolis - Anderson - Columbus, IN and Portland Vancouver - Hillsboro, OR-WA metropolitan areas from January 28, 2013 through March 21, 2013. Establishments were
selected from the current NCS sampling frame, excluding establishments currently active in any NCS sample. Probability
Selection of Occupations (PSO), a method of randomly selecting individual items from a larger population (in this case
occupations), was used to determine the occupations selected for collection of detailed job requirement information.
Twenty-four experienced field economists from BLS regional offices collected the data. More than 40 percent of the
interviews were observed by BLS staff. The majority of interviews were done by personal visit, but a small number were
collected by phone.
Upon completion of the ORS collection, respondents were asked questions to gauge their reactions to the survey, and
BLS field economists completed a post-interview debriefing. Daily debriefing sessions were held with BLS field
economists, observers, and other staff for the purposes of discussing interviewing experiences, identifying potential
issues with the materials and procedures, and sharing lessons learned. End-of-city debriefing sessions held in
Indianapolis and Portland summarized the major findings identified during the test period and allowed for expanded
discussion of these and other issues between BLS and SSA staffs.

Background and Test Overview
In April 2012, the Bureau of Labor Statistics (BLS) signed an interagency agreement with the Social Security
Administration (SSA) for the purpose of designing, developing, and carrying out a series of tests to assess the feasibility
of using the National Compensation Survey (NCS) platform as a means to accurately and reliably capture data relevant to
the SSA’s disability program purposes. The resulting initiative – the Occupational Requirements Survey (ORS) Project –
recently completed the second of three phases of testing planned for fiscal year 2013 as outlined in the Interagency
Agreement Deliverable sent to SSA on September 28, 2012. That document outlines the work the BLS will perform, key
objectives, and a detailed test plan:
In fiscal year 2013, the BLS will perform work to meet the following objectives:
1. Evaluate survey design options and begin developing the protocols, aids, and final design to meet SSA data
needs;
2. Collect data to test and refine the protocols and aids; and
3. Provide documentation to the SSA summarizing the work performed by the BLS, conclusions drawn, and
recommendations for future data collection.
In order to accomplish these objectives, the BLS will conduct a series of field tests with general test goals as described
below:
2|Page

Phase 1 – Initial Proof of Concept Testing: The primary goal of this phase of testing will be to ensure that the BLS
field economists know how to describe the survey and ask for the new data elements. In addition, the BLS will
create and test and initial set of data collection protocols and a preliminary set of data collection aids.
Phase 2 – Collection Protocol Testing: The primary goal of this phase of testing will be to test collection of the new
data elements while performing a selection of occupations from each respondent. In addition, the BLS will refine
the collection protocols and aids based on an analysis of the test results.
Phase 3 – Broad Scale Testing: The primary goal of this phase of testing will be to test the BLS’s ability to select a
sample of occupations within each establishment, collect the new data elements needed by SSA, and collect other
NCS data elements that are of research interest to SSA such as wages and job leveling information. In addition, the
BLS will test the feasibility of collecting the data needed by SSA along with all the NCS data elements needed to
produce the Employment Cost Index, Employer Costs for Employee Compensation, and various benefits products.
Phase 1 testing was successfully conducted in the Washington, D.C. consolidated metropolitan statistical area (CMSA)
from November 28, 2012 through December 6, 2012. BLS staff completed interviews with 27 establishments, collecting
detailed information on SSA data elements for 104 occupations. This report details the methods and findings of the
Phase 2 test. Phase 3 is scheduled to start in early May.

Phase 2 Test Objectives
The primary goals of Phase 2 testing were to expand the number of BLS field economists that understand how to
describe the ORS and ask respondents for information regarding the data elements, and obtain information on new
additions to the Phase 1 collection procedures. Some key additions to the tests were:
1. Probability Selection of Occupations (PSO) – used a disaggregating technique to select occupations
randomly from a list of employees working for an establishment.
2. Rarely / “Almost Never” – offered an additional frequency choice in order to better capture the
existence of elements that occur very infrequently (approaching never).
3. Phone Interviews – provided guidance for the use of this collection method.
4. Dictionary of Occupational Titles (DOT) Coding – matched establishment jobs to the occupational list
within the DOT.
5. Work Setting – determined the work environment in which the job occurs.
Another primary goal of the Phase 2 test was to assess the effectiveness of revised ORS collection tools. BLS staff
created streamlined collection tools based on the results of the Phase 1 test and the Phase 2 test objectives, and
evaluated a modified set of data collection protocols and data collection aids. The experiences of Phase 2 testing will be
used to further refine data collection protocols and aids, analyze the collected data, revise the methods for computing
estimates, and validate future estimates.

Test Logistics and Methods
Field collection for Phase 2 testing was conducted in the Indianapolis - Anderson - Columbus, IN metropolitan area
(January 28, 2013 – February 20, 2013) and the Portland - Vancouver - Hillsboro, OR-WA metropolitan area (February 28,
2013 - March 21, 2013). Establishments were selected from the current NCS sampling frame, excluding establishments
currently active in any NCS sample. Probability Selection of Occupations (PSO) was used to determine the occupations
selected for collection of detailed job requirement information. Twenty-four experienced field economists from BLS
regional offices collected the data, and more than 40 percent of the interviews were observed by BLS staff. The majority
of interviews were done by personal visit, but a small number were collected by phone.
3|Page

BLS field economists tested three alternative collection approaches. Upon completion of the ORS collection,
respondents were asked questions to gauge their reactions to the survey, and BLS field economists completed a postinterview debriefing. Daily debriefing sessions were held with BLS field economists, observers, and other staff for the
purposes of discussing interviewing experiences, identifying potential issues with the materials and procedures, and
sharing lessons learned. End-of-city debriefing sessions held in Indianapolis and Portland summarized the major findings
identified during the test period and allowed for expanded discussion of these and other issues between BLS and SSA
staffs.

Key Findings
The results of the Phase 2 test were very promising overall.
BLS field economists completed 227 interviews representing a total of 240 establishments, collecting detailed job
information for 1,094 occupations. Probability selection of occupations was successfully implemented in over 90
percent of interviews. As was true in Phase 1 testing, a number of issues arose in Phase 2 that will be addressed and
evaluated in subsequent phases of testing. These include refined guidance and protocols for: collecting job leveling
information, physical demand strength and other frequency estimates, phone collection, and the use of clarifying
examples. However, the consensus opinion of BLS staff is that Phase 2 objectives were successfully met, and these
activities lay a strong foundation for future tests.
Phase 2 training consisted of self-study (background readings on the SSA disability program and relevant ORS concepts),
web-based and in-person instructions provided prior to test collection in each city, and on-the-job training (precollection practice interviews, debrief sessions, and informal field economist exchanges). The training approach used in
Phase 2 worked well and will be repeated in subsequent tests. The use of practice interviews prior to data collection
and the use of debriefing session where BLS field economists, observers, and other staff could exchange information and
suggestions about collection issues were particularly successful.
The materials prepared to aid ORS cooperation were very well received, particularly the two-sided ORS flyer with the
SSA letter on the back, the Zooming in on Compensation Statistics booklet, the city-specific economic summary sheets,
and the EA&I contact information.
BLS field economists expressed a strong preference for the collection approach that asked respondents for information
about each ORS element for all selected occupations before moving to the next element. Respondents and field
economists had very positive reactions to the collection aids, but recommended that a corpus of standard examples be
developed to help clarify ORS concepts for respondents and ensure testing consistency.
Most data were collected from typical NCS respondents such as human resource directors, small business owners, and
location managers.
Cooperating establishments were able to answer the vast majority of questions asked in the survey, resulting in a very
high item response rate. This was true of both personal visit and phone interviews. However, as in Phase 1, there were
certain words and concepts that proved challenging to administer effectively. Respondents continued to vary in their
interpretations of the term required, and to have difficulty with the concept of average performance. Some respondents
struggled to provide frequency estimates for tasks that were short but repetitive, or which varied from day to day or
between employees in a given occupation, and for nonstandard works shifts (e.g., 12 hour shifts). The issue of
accommodations and how to explain why they are excluded in ORS was consistently discussed in the daily debriefs. One
successful approach used by a field economist was to explain that the focus of the survey is the job and not the
individual doing the job, and that individual accommodations at an establishment should not be collected. Field
economists said that it would be helpful to be able to clearly explain why they are being excluded in a manner that is
acceptable to all parties.
4|Page

Phase 2 Test Methods and Materials
Field collection for Phase 2 testing was conducted in the Indianapolis - Anderson - Columbus, IN metropolitan area
(January 28, 2013 – February 20, 2013) and the Portland - Vancouver - Hillsboro, OR-WA metropolitan area (February 28,
2013 - March 21, 2013).

Selected Establishments
Establishments were selected from the current NCS sampling frame for businesses in the two metropolitan areas,
excluding all establishments currently active in any NCS sample but including some units recently rotated out of NCS
production. The target number of completed interviews for Phase 2 testing was 250 establishments (125 per test city)
representing as broad a mix of industries as possible given the size and time constraints of the test. At the conclusion of
the testing period, BLS completed 120 interviews in each test city (see the Results section for more details about
participating establishments).

Field Economists
Phase 2 data were collected by 24 experienced NCS field economists from BLS regional offices. Eight senior field
economists who had participated in Phase 1 testing also collected in both Phase 2 test cities, and served as mentors to
the field economists new to ORS. Interviewer training for Phase 2 collection followed regular BLS practices, and
consisted of five components:


Self study – field economists were provided the SSA background materials, ORS high-level test plans, and
element definitions.



Webinars – In the week prior to the start of test data collection, field economists participated in web-based
training sessions in which they were in given an overview of the SSA disability process, and information about
the ORS test objectives, procedures, and materials. There were two webinar training sessions prior to
Indianapolis collection and one prior to Portland collection. The imbalance in webinar sessions between test
cities was offset by an additional day of in-person training in Portland).



In-person training – This occurred during the field economists’ first days in the test city. In-person training
consisted of: review of the technical memorandum; practice with the data capture spreadsheet; an overview of
instrument edits and data analysis objectives; a calibration exercise; mock interviews; and a mock interview
debrief session.



Mentoring - In addition to formal training, each field economist new to ORS was assigned a mentor who
collected in prior ORS testing. The mentor served as the field economists’ primary resource for technical
advice. Mentors made all first-week appointments for the mentee. Mentees observed mentors conducting two
interviews prior to conducting their own interviews, and then were themselves observed by their mentor on
their first two interviews. Mentors also reviewed the mentees’ write-up of their initial schedules.



On-the-job training (OJT) – Throughout the Phase 2 Test fielding period, field economists engaged in a number
of activities designed to reinforce formal ORS training concepts. During data collection, OJT was provided
through formal daily debriefing sessions and informal conversations between field economists and project staff
in which lessons learned, best practices, and challenging issues were identified.

Refinement Procedures
Field economists were provided with their collection assignments prior to their arrival in the test city, and immediately
began the refinement process (e.g., using Internet searches, phone calls) to validate the assigned establishments. Only
establishments that could provide data locally (i.e., in Indianapolis or Portland areas) were considered in scope. Field
economists were instructed to proceed to the next establishment if data had to be procured from outside the test area
5|Page

(e.g., from a corporate office in another city), or when refinement proved problematic after reasonable effort had been
made.
To secure appointments, field economists contacted potential respondents and attempted to speak with company
personnel with the same job titles that NCS traditionally works with (e.g., HR Directors, Personnel Managers, and Hiring
Officials). If those individuals were not available, the field economist worked with the establishment contact to identify
the person most familiar with the target data elements to interview. When the appropriate company contact was
located, the field economist used a standardized script or “spiel” to explain the purpose and importance of the survey,
and attempted to schedule an appointment. Some of the field economists sent e-mails to establishment contacts to
provide additional background information about the purpose of and procedures for the interview. All potential
respondents were informed that the data collection effort was part of a test being done at the request of the Social
Security Administration and that participation was voluntary.
When contact could not be made with listed establishments, or the establishment points of contact expressed
reluctance or indicated that they were unavailable during the test field period, the field economists were instructed to
forego the normal NCS conversion attempts and simply advance to the next establishment on their assignment list. This
process continued until each field economist had secured their allotted appointments.

Occupational Selection for Responding Units
The Phase 2 tests evaluated the use of NCS Probability Selection of Occupations (PSO) procedures as an occupational
selection tool. Field economists were instructed to use standard NCS coverage and procedures for PSO as much as
possible. They attempted to obtain an employee list, refine the list based on NCS inclusion/exclusion criteria, and then
select the appropriate number of occupations for collection. Field economists selected 8 occupations for the largest
establishments (i.e., those with more than 250 employees), and fewer occupations for smaller establishments (e.g., 4
occupations for companies under 50 people). If the respondent was unwilling or unable to do formal PSO, a fall-back
selection procedure was used. Field economists attempted to identify the most highly populated occupations in the
establishment and then select between four and eight of those that also spanned different major SOC classifications.
PSO could be done at the start of the interview or in advance.
Standard NCS worker characteristics (e.g., full-time/part-time, union/non-union, time/incentive, supervisor/lead worker
information, etc.) were collected for all selected occupations. These characteristics were used to help identify unique
occupations.

Data Collection Tools and Protocols
Three paper data collection tools were tested in Phase 2. Each version was designed to collect the required data
elements: Job Details (e.g., title, occupational employment size, SOC/DOT code, worker characteristics, and work
environment); Specific Vocational Preparation (SVP); Physical Demands (PD); and Environmental Conditions (EC). The
tools differed in their visual format, question flow, and collection method.




A Single Quote Tool was designed to obtain the occupational requirements one occupation at a time. Definitions
for select terms were provided at the bottom of the page.
A Multiple Quote Tool was designed to collect the occupational requirements of all selected jobs
simultaneously. Field economists were instructed to ask each item about all occupations before proceeding to
the next item.
An Advanced Multiple Quote Tool was developed for small-scale feasibility testing in Portland. This tool
changed the order in which the elements were asked (e.g., asking Environmental Conditions questions prior to
Physical Demand items), and was designed to allow field economists to first determine whether each job
required certain characteristics or not (e.g., does the job require the worker to be exposed to extreme heat or
not?), and then to go back and administer follow-up questions only for eligible occupations (e.g., how often does
the job require the worker to be exposed to extreme heat?).

6|Page

Based on feedback received from field economists and observers during Indianapolis data collection, the collection tools
were modified prior to the start of Portland collections.
o
o
o
o

An item was added to the SVP element that asked if there was a literacy requirement for the job.
The ‘reaching’ items in the PD element were reordered such that the “any reaching” question was asked
prior to the directional reaching questions.
The frequency questions for ‘lifting/carrying’ items in the PD element were reversed (i.e., “constantly”
was asked first, followed by “frequently” and “occasionally”).
The order of the crawling, crouching, kneeling, and stooping items were reversed to ask first about
stooping, then crouching, kneeling, and crawling.

Use of the Advanced Multiple Quote tool was restricted to limited testing by senior BLS field economists in Portland. For
all remaining Phase 2 interviews, the decision of whether to use the Single Quote tool or standard Multiple Quote tool
(or combination thereof) was left up to the individual field economists, though the Multiple Quote tool was
recommended for schedules in which it appeared likely that many of the selected occupations would have similar
requirements. Field economists were instructed to read the questions as worded on the tool for testing consistency, but
were permitted to provide additional explanations and ask follow-up questions as necessary.
In addition to these collection forms, a number of aids were developed to help clarify the survey task and definitions.
Respondent visual aids provided frequency definitions and examples for the noise intensity level response categories
and of the type of frequency and weight information that respondents should consider when answering the
lifting/carrying items. Field economist aids provided definitions and clarifying examples of ORS elements and answers to
commonly asked questions. Field economists were encouraged to consult these aids during the interview and to share
them with respondents as necessary.
Both in-person and remote data collection were tested in Phase 2. The preferred method of collection was personal
visit; if the respondent indicated a willingness to provide information by personal visit and another method (e.g., phone,
email), it was mandatory to select personal visit. In the event that a respondent refused or was unable to schedule an
in-person appointment, however, field economists were permitted to collect information over the phone or by email. In
addition, a small group of schedules was assigned to phone collection conducted by senior BLS field economists in each
Phase 2 test city. Remote collection protocols were very similar to those for in-person collection. Field economists
were required to use the collection tools, collect all of the ORS Phase 2 elements, ask questions as worded, and probe
unclear answers or in situations in which respondents’ answers did not match expected job patterns. They also were
encouraged to frequently refer to the field economist aids to provide more detailed definitions, explanations, and
examples to the respondent as needed.

Data Capture and Review
Data from the Collection Tools and Establishment Information sheet were entered into a data capture spreadsheet on a
flow basis. The tool was designed to permit easy data entry by field economists, the ability to review the captured data,
and capacity to tabulate results. Table 1 lists the data elements collected in the capture instrument.
Review parameters were developed for Phase 2 and were used to evaluate data elements for internal
consistency. Specifically, three internal consistency spreadsheets (one for each SSA data element area – SVP, PD, EC)
were used in the data review and analysis. The parameters identified expected relationships between the individual
ORS data elements, as well as consistencies between ORS and NCS data elements.

7|Page

Table 1. Information Recorded in the Phase 2 Test Data Capture System
Element
Items Captured

General Information

Schedule number; establishment employment size;
NAICS code; selected occupations (title, SOC,
employment, FT/PT, union/nonunion, incentive/time,
supervisory status, work environment, job description)

Specific Vocational Preparation (SVP)

Title; SVP; SVP level; education level; training and
certifications; time to average performance; literacy
requirement

Physical Demands (PD)

Standing/walking/sitting; reaching; lifting/carrying;
pushing/pulling; gross manipulation/fine manipulation;
keyboarding/words per minute; hand arm controls/foot
leg controls;
climbing/stooping/kneeling/crawling/crouching;
visual acuity; speaking; noise intensity level; loud
noise/moderate sound discernment

Environmental Conditions (EC)

Extreme cold/extreme heat; wetness/humidity;
vibrations; fumes/odors/gases/poor ventilation;
toxic caustic chemicals; high exposed places; moving
mechanical parts

Observers
To help garner feedback about the interview and data collection processes, an observer accompanied the field
economists on approximately forty percent of the data collection appointments. The observers represented a mix of
roles and stakeholders within the ORS Project (e.g., NCS/ORS management; BLS Assistant Regional Commissioners and
Branch Chiefs; ORS mentors; BLS survey methods and operations staff), with priority given to those individuals serving
on the ORS Project. An attempt was made to have each observer observe three collection interviews.
The observer did not participate directly in ORS collection, but was responsible for taking notes about the collection
process. They noted the duration of the interview (and the time needed to administer sections within the interview),
and their observations for each item and the interview as a whole.

Other Debriefing Activities
In addition to the Observer tool(s), the following debriefing activities were conducted to assess the effectiveness of
Phase 2 ORS materials and procedures:


Respondent and Field Economist Debriefing – at least 10 minutes were set aside at the end of each ORS
interview to ask respondents about their interview preparation and experiences. This debrief was
administered either by the field economist or the observer, and consisted of a brief set of questions targeting
potential comprehension issues, perceptions of burden, reactions to the visual aids, etc. In addition, field
economists were asked to complete a post-interview questionnaire about their impressions of respondent
comprehension issues and the effectiveness of the interview process and materials.

8|Page



Daily Collection Debriefs – each day during the Phase 2 test in which there was a collection, an in-person
debriefing session was held to assess the day’s interviews, discuss problems or questions that arose, and share
ideas for best practice solutions. All available field economists and observers who had participated in that day’s
collection interviews attended these daily debrief meetings. Others involved in the ORS project attended, as
available.



City Debriefs – city debriefs were held in Indianapolis on February 8, 2013, and in Portland on March 7, 2013.
The purpose of these sessions was to summarize key findings from the Phase 2 tests in the respective cities
based on all the information collected to date through the various assessment activities, to solicit additional
feedback about components of the test, and begin to identify issues and recommendations relevant to future
ORS test collections. These meetings were attended by all available ORS Project staff and by SSA sponsoring
officials.



End-of-Phase Questionnaire – field economists were asked to complete a brief questionnaire at the end of
Phase 2 testing. The purpose of the questionnaire was to collect feedback on aspects of the Phase 2 test that
were not covered in depth by the other debrief activities (e.g., the effectiveness of Phase 2 training, calibration
exercises, use of the data capture tool, and debriefing sessions).

Phase 2 Test Results
This section reports on results compiled throughout the Phase 2 test. They reflect information obtained from each
debriefing component and feedback collected from observers and field economists during the collection and city
debriefing sessions. Although the information garnered from each of these components is summarized here and not
reported individually by source as was true in the Phase 1 summary, each source contributed significantly to the overall
Phase 2 findings and conclusions. This section begins with a broad assessment of the feasibility of collecting the data
needed by SSA through the ORS, and then provides an overview of the collection effort along with more detailed
descriptions of issues that arose during the test.

Feasibility
The main objective of the development and evaluative work that will be done for the ORS project in fiscal year 2013 is to
assess whether it is feasible for BLS to collect data relevant to the SSA’s disability determination program. The results of
this collection protocol test suggest that the collection of the ORS data elements using a probability selection of
occupations is viable. In Phase 2, BLS field economists completed interviews with 227 respondents representing a total
of 240 locations. New in Phase 2 was the use of probability selection of occupations to determine on which jobs to
collect detailed job requirement information. Probability selection of occupations was completed in approximately 92%
of the interviews while the remainder used the same process as Phase 1 testing. These processes combined resulted in
1,094 occupations for Phase 2 testing. As was true in Phase 1 testing, a number of issues arose in Phase 2 (e.g., with
certain data elements, aspects of the collection and assessment, etc.) that will be addressed and evaluated in
subsequent phases of testing; many of those are reflected in the sections that follow. However, the consensus opinion
of BLS staff is that Phase 2 objectives were successfully met, and these activities lay a strong foundation for future tests.

Participating Establishments
Table 2 provides additional details about the establishments that participated in Phase 2 and the occupations for which
ORS data were collected. As can be seen, BLS staff continued to be successful in securing interviews from a variety of
industry groups and collecting data for a range of occupations in an extremely tight fielding period.

Data Collection Assignments and Appointments
Field economists’ pre-collection activities continued to be very similar to NCS and Phase 1 testing. As was true in Phase
1, efforts to turn around reluctant respondents were not required. Field economists were instructed to move on to the
next establishment on their list in the event that they encountered reluctance.
9|Page

Indianapolis field economists were given their assignments one week prior to arrival in the test city, and during
debriefings indicated that this did not allow them enough time to make appointments. Instead of coming into the test
city with most of the appointments secured, Indianapolis field economists reported that they had to juggle making
appointments with other week-one test activities such as training and mentoring appointments. In response to this
concern, Portland field economists were given their assignments two weeks before they arrived in the city. This
improved their ability to make appointments before formal training and mentoring appointments began, and an effort
will be made in Phase 3 to provide field economists their assignments as early as possible.
Field economists thought that having their assignments organized geographically was helpful. Not only was it efficient in
terms of scheduling appointments in the same general proximity within the metropolitan area, but it also allowed those
who were unfamiliar with the area to gain familiarity with at least a portion of the metropolitan area.
When making assignments where some ownership/industry sectors were limited in the number of establishment units,
field economists often found that sectors closed for collection before they were able to fully work through their assigned
units. This caused the need for additional switching of units to provide enough opportunities for field economists to
reach their individual collection goals, and in turn, caused a delay in their ability to schedule appointments.

10 | P a g e

Table 2. Description of Establishments and Selected Occupations for Phase 2 Testing
Ownership

Industry
Group

# of Participating Units
Overall

By City

Employment
Size
(range)

# of Jobs
Collected

Indianapolis = 19

Education

43

4 – 522

165

Portland = 24

Jobs Collected
(selected)







Indianapolis = 16
Financial
Activities

33

1 – 1200

143

Portland = 17

Indianapolis = 21











Goods
Producing

39

1 – 717

187

Portland = 18





Private
Industry

Indianapolis = 23
Healthcare

40

4 – 1400

189

3 – 675

156

Portland = 17

Indianapolis = 19

Service
Producing

















39



Portland = 20




Total

11 | P a g e

194

Indianapolis = 98
Portland = 96

1 – 1400

840

Teachers (All grade levels; Special Ed)
Tutors
Self-Enrichment Education Teachers
Teacher Assistants
Secretaries & Administrative
Assistants
Vocational Education Teachers,
Postsecondary
Insurance Sales Agents
Insurance Policy Processing Clerks
Loan Officers
Computer User Support Specialists
Tellers
Advertising Sales Agents
Customer Service Representatives
Molding, Coremaking and Casting
Machine Setters, Operators and
Tenders, Metal & Plastic
First-line Supervisors of Production
and Operating Workers
Industrial Production Managers
Inspectors, Testers, Sorters, Samplers
& Weighers
Mechanical Engineers
Customer Service Representatives
Painters, Construction &
Maintenance
Assemblers & Fabricators, All Other
Registered Nurses
Nursing Assistants
Personal Care Aides
Medical Secretaries
Medical Assistants
Child, Family & School Social Workers
Dental Assistants
Waiters & Waitresses
Heavy & Tractor-Trailer Truck Drivers
Sales Representatives, Wholesale &
Manufacturing except Technical &
Scientific Products
Laborers and Freight, Stock and
Material Movers, Hand
Refuse and Recyclable Material
Collectors
First-Line Supervisors of Retail Sales
Workers
Retail Salespersons
Office Clerks, General

Table 2, continued
Ownership

Industry
Group

# of Participating Units
Overall

By City

Employment
Size
(range)

# of Jobs
Collected

Indianapolis = 5
Education

9

92 - 1809

52

4 - 570

64

Portland = 4

Indianapolis = 7
Financial
Activities

13
Portland = 6

Goods
Producing

4














Indianapolis = N/A
State and
Local
Government

Jobs Collected
(selected)

16-91

17

Portland = 4






Indianapolis = 5
Healthcare

10

6 - 4063
Portland = 5

64

Indianapolis = 5
Service
Producing

10

Total

46

38 - 4530

57

4 - 4530

254

Portland = 5
Indianapolis = 22
Portland = 24












Teacher (All grade levels; Special Ed)
Teacher Assistant
Janitor & Cleaner
Secretary & Administrative Assistant
Bus Driver
Dining Room & Cafeteria Attendant
Librarian
Library Technician
Library Assistants, Clerical
Usher, Lobby Attendant & Ticket
Taker
Janitor & Cleaner
Landscaping & Groundskeeping
Worker
General & Operations Manager
Electrician
Production, Planning & Expediting
Clerk
Heating & Air Conditioning
Mechanics & Installers
Regulatory Affairs Specialist
Office Clerk, General
Social & Human Service Assistant
Registered Nurse
Probation Officers & Correctional
Treatment Specialist
Police Patrol Officer
Office Clerk, General
Construction & Building Inspector
Municipal Firefighter
Childcare Worker

Marketing
For Phase 2, BLS assembled the following set of marketing items: ORS one-page flyer with a copy of the letter that SSA
sent to BLS; a one-page ORS flyer with reference to the SSA website; the BLS Customer Service Guide; a Zooming in on
Compensation Data booklet; Indianapolis and Portland Area Economic Summary sheets; the most recent ECI/ECEC/EBS
releases; and business cards listing the contact information of the Economic Analysis and Information (EA&I) units for
the Chicago and San Francisco Regions. BLS also created two sets of letters aimed at respondents in Phase 2 testing one targeting government units and the other targeting private sector units. In addition, field economists used the SSA
Occupational Information System Project (OIS) and the BLS Occupational Requirements Survey websites to inform
respondents about the ORS collection efforts.
The Phase 2 debriefing results demonstrate that field economists especially liked the two-sided ORS flyer with the SSA
letter on the back. The other materials that were widely used by field economists were the Zooming in on
Compensation Statistics booklet, the Indianapolis/Portland Economic Summary sheets, and the EA&I contact
information. There was not widespread use of the ORS regional letters or respondent-specific marketing packets. This
12 | P a g e

likely was due to the nature of the ORS assignment and the Phase 2 test study; field economists had a short window to
make appointments and were instructed to move to another unit if the first few attempts were unsuccessful. For
collection in Phase 3 and beyond, use of the respondent letters and tailored marketing packets may increase as fuller
recruitment efforts are implemented.

Interview Duration
Based on the approved testing plan submitted to the Office of Management and Budget (OMB), field economists were
allotted one hour to collect the ORS data and conduct the respondent debriefing in each establishment. On average,
data collection took 53 minutes per schedule, and data was collected on 4 occupations at each establishment. The
average occupation took 13.4 minutes to complete. PSO took an average of 5 minutes to complete when not done in
advance. Job details and education and training took an average of 16 minutes to complete, or 4 minutes per
occupation. Physical demands took the longest to complete of the sections, at an average of 26.8 minutes per schedule
or approximately 6 minutes per occupation. Environmental conditions took 4.7 minutes to complete on average. The
post-interview debrief averaged 2.5 minutes.

Collection Protocols
The Collection Approaches
The most popular choice among the three collection approaches tested in Phase 2 was the Multiple Quote method in
which field economists collected data about each element for all the selected jobs at one time before moving on the
next element. BLS field economists reported that this approach seemed to provide the best balance between collecting
quality data and reducing respondent burden, and they liked that the definitions of the various elements were
incorporated directly into the collection tool. Feedback from the senior BLS field economists who used the Advanced
Multiple Quote method were similarly positive, with some staff indicating that they would like to use this tool
exclusively in future collection efforts. The approach of asking all of the ORS data elements in their entirety for a single
selected occupation before moving on to the next occupation received positive feedback in limited situations, such as
small companies with one or two selected occupations. However, most field economists felt that the single-quote
approach did not result in an efficient interview, and produced a higher level of respondent burden when more than one
or two occupations were selected. When given a choice of which tool to use (as opposed to being required to use each
tool equally, as in Phase 1), staff almost universally opted not to use the single-quote tool.
The Spiel
The spiel consisted of written introductory language designed to aid ORS cooperation, as well as a series of bullet points
covering topics that field economists were required to address when initially contacting selected establishments. The
Phase 2 spiel incorporated a number of changes based on recommendations made during Phase 1, and field economists
said that these changes resulted in significantly improved interactions with respondents and greater chance of securing
cooperation.
Respondent Visual Aids
Results from the field economist and the respondent debriefings revealed nearly universal agreement that the
respondent visual aid (with concept definitions and frequency scale) was well liked and effective. All of the respondents
who saw the visual aid reported that it was helpful. The multi-colored weight chart, on the other hand, proved difficult
for respondents to understand and for field economists to explain. Comments such as “the multi-colored weight chart
did not work visually” were fairly common in the debriefings.
Field Economist Reference Guide
This document essentially served as technical manual that field economists could consult during the interview. It
contained definitions of the factors, as well as some examples pulled from the Revised Handbook for Analyzing Jobs
(RHAJ). Staff working in Phase 2 found it to be very helpful, and referred to it quite frequently during interviews. The
main shortcoming that field economists found with this document was that it had insufficient examples of the
conditions and factors that one encounters in a modern work environment. Most of the examples either were
13 | P a g e

production related or so specific that they had minimal relevance to the respondent’s actual work situation. Field
economists recommended that more effort needs to be devoted to finding more common, contemporary, and
understandable examples for each factor. Other project participants noted that while examples have the power to
quickly crystallize a vague concept for respondents and to help with recall, they may have a biasing effect if not
implemented with care and consistency.

Phase 2 BLS Field Economist and Observer Training
Phase 2 test collections allowed for further development and refinement of the training procedures for staff new to
ORS. As the training staff gained greater insight into the ORS data collection process, they were able to create training
materials that helped get new staff up to speed more efficiently, as reflected in field economists’ responses to items on
the end-of-phase questionnaire (Tables 5 and 6).
Table 5. The Field Economist Training Adequately Prepared Me for Phase 2 Collection – Percent Agreement
Strongly Agree
Agree
Neither Agree nor Disagree
Disagree
Strongly Disagree

6.7%
80.0%
6.7%
6.7%
0.0%

Table 6. Field Economists’ Overall Rating of Phase 2 Field Economist Training
Excellent
Good
Fair
Poor

40.0%
46.7%
13.3%
0.0%

Beyond these general reactions, field economists and observers were largely positive about their experiences in a
number of the individual components of Phase 2 training. They reported that the pre-collection practice (or ‘mock’)
interviews gave them the opportunity to rehearse using the collection and observer tools, and gave them confidence
that BLS staff could ask the questions as intended and collect high-quality data pertinent to the SSA elements. The daily
debriefing sessions continued to be an excellent training tool, as well. The semi-structured setting once again provided
an exchange of information which allowed for training and clarifications of materials already provided. Finally, field
economists and observers emphasized the value of other informal, on-the-job learning opportunities. These on-going
conversations helped to solidify lessons learned, best practices, and to identify areas in need of further development.

Data Collection
Data collection for Phase 2 once again proved to be a valuable learning experience for field economists, as it did in Phase
1. Over the course of collecting data in the Phase 2 cites a number of recurring issues began to emerge. As has been the
case since the beginning, reaching and weight were the two data elements that posed the most problems, but some
additional areas of confusion began to surface as well.
Respondents
Field economists’ experience administering the Phase 2 ORS items and respondent debriefs provided valuable insight
into respondents’ understanding of the ORS concepts and their reactions to the survey experience, and it allowed field
economists to develop better and more consistent probes when answers appeared inconsistent. As in Phase 1, the term
require (e.g., “…does this job require the worker to…”) continued to be confusing for some respondents in Phase 2.
Phase 2 introduced language intended to clarify this concept (e.g., how the job is generally performed; core duties of the
occupation), but respondents varied in their interpretations of these descriptions, as well. For example, one respondent
said that a core duty of a job was not required if another person in the occupation was able to perform the task.
14 | P a g e

The concept of average performance also continued to challenge respondents. Field economists continued to provide
the given definition and description of the term. Some respondents said that for jobs requiring prior experience,
incumbents were required to be satisfactory on day one, while other respondents wanted to use the probationary time
as the indicator.
The issue of accommodations and how to explain why they are excluded in ORS was a frequent topic in the daily
debriefs. One successful approach used by a field economist was to explain that the focus of the survey is the job and
not the individual doing the job, and that individual accommodations at an establishment should not be collected.
Although it proved easy enough to explain that accommodations should be excluded, field economists said that it would
be helpful to be able to clearly explain why they are being excluded in a manner that is acceptable to all parties.
Respondent Reaction to ORS
Respondents generally were very receptive to ORS, and most said that they felt it is a worthy endeavor. The following
bullets list some examples of typical respondent comments made during the respondent debrief:






The survey is appropriate; there is an obvious need for this study.
This is a great undertaking.
The SSA test is a good idea and I like the idea that SSA is assessing their disability program.
It made me think about and review my own jobs and what I should add or change to my job descriptions.
I like the idea that SSA is considering a disability assessment.

In some instances, however, respondents had more pointed observations about the survey:





The shorter the survey the better for me, even though I understand you need to get your data.
It is hard to average out the work in a job as it varies day by day.
It was difficult to know all aspects of the job (e.g., some of the fine details about physical demands aren't so
well known).
I found it hard to distinguish between requirements and accommodations.

In anticipation of Phase 3 testing, Phase 2 respondents were asked questions during their debriefing about providing
ORS data through alternate methods of collection, and their ability to provide wage information for the selected
occupations. As the results presented in Table 8 (below) make clear, most respondents felt that some form of remote
collection of ORS data would be feasible, and that they would have been able to provide wage data about employees in
the selected occupations.
Table 8. Respondents’ Responses to Questions about Mode Preference and Wage Data Availability
Do you think we could have collected this information through email or a phone interview?
Yes

72%

No

20%

Maybe
9%
Would we have been able to obtain current wage rates for all of the employees in each of the discussed job
titles from you?
Yes

87%

No

6%

Other

7%

Field Economists
Phase 2 field economists in most cases felt that the collection steps for the ORS were very similar to the NCS program,
and the refinement of the sample units typically followed the same process as in NCS. This was reflected in daily debrief
discussions as well as field economists’ responses to items asked in the end-of-phase questionnaire (see Table 7).
15 | P a g e

Table 7. Field Economists’ Responses to Debriefing Questions about Phase 2 Pre-collection Activities
Were the refinement, contact and collection any different for this establishment than it would have been
for an NCS schedule?
Same as NCS

96%

Different than NCS

1%

Other

3%

How long did it take you to research the establishment and make the appointment?
Less than 15 minutes

18%

15-30 minutes

61%

30-60 minutes

15%

More than 60 minutes

6%

Debriefing Activities
Phase 2 debriefing activities remained unchanged from Phase 1 (e.g., interview observations; completion of observer
and respondent debrief, and post-interview debrief tools; attendance at the daily and end-of-city debriefing sessions).
Of the 120 interviews conducted in Indianapolis, 60 had observers; in Portland 42 of 120 interviews were observed.
The daily debrief sessions were attended by ORS project staff – including all field economists (unless they were on an
interview), observers, project staff, and managers – and the discussions again were active and productive. The end-ofcity debriefing sessions were attended by all available ORS Project staff and managers, with representatives from the
SSA attending via phone.
Each of the Phase 2 debriefing activities remained useful, albeit with some appearance of diminishing returns towards
the end of collection as trends solidified and less new information emerged. The respondent debriefings and daily
debriefing sessions were the most crucial assessment elements, offering insights into issues that will need to be
addressed and further evaluated in Phase 3 or fiscal year 2014 (e.g., the optimal order of ORS elements; the use
clarifying examples; and the length and repetitiveness of the survey/respondent burden).
The in-house debriefing sessions were larger than they had been in Phase 1, and often there many more interviews that
needed to be discussed. In Indianapolis, the sessions were structured around collection appointments, with field
economists and observers sharing their interview experiences, identifying potentially problematic items or procedures,
and exchanging approaches that they found helpful in clarifying meaning and securing collection goals. For Portland, the
decision was made to alter the way in which the meetings were structured, moving from a specific interview-focused
approach to an industry-focused approach that allowed BLS staff to more efficiently identify survey trends across
different industries. The end-of-city debriefings provided the opportunity for BLS staff to summarize their Phase 2
findings, and for SSA to ask questions about the Phase 2 test changes and to get feedback from field economists about
what was and was not working. As in Phase 1, significant problems were identified as well as smaller issues that need
adjusting, but there was consensus that the ORS materials and procedures significantly improved between the two test
phases and throughout Phase 2.
Phase 2 observers reported that the observer tools remained useful for recording timing information and noting
problems that occurred during the interview, but the utility and efficiency of those tools could be improved.
Suggestions included providing additional space for writing notes, requiring less item-by-item notation, and formatting
the tool as a booklet with major elements on one page (for reference) and a notes page facing it. Most of the field
economists favored the Multiple Quote collection tool, so the multi-quote observer tool was used most often. For
Phase 3, BLS plans to explore a redesign of the observer tool to make it easier to keep pace with the interview.

16 | P a g e

Data Capture System
The data capture excel spreadsheet was once again used in Phase 2. It was an updated version of the spreadsheet used
in Phase 1. Some changes from Phase 1 collection included:





Edits were added to the system, prompting field economists to review their coding and provide additional
documentation when coding is unusual for a specific job.
Each element had don’t know, no response and never added as an option.
The 9-digit DOT code and the 8-digit SOC code were required using the proper numerical formatting. The
system provided a prompt if there is an error and would not allow the incorrect format to be entered. The
formatting error is true for all other elements as well.
The updated spreadsheet included an area to indicate the supervisory level of the job.

The use of the spreadsheet proved to be a functional option in Phase 2 while work on the data capture system
continued, but field economists had somewhat mixed views about the usability of this tool and the perceived value of
the new Phase 2 edits.
Table 9. Field Economists’ Responses to Questions about Phase 2 Data Capture Tool
The Phase 2 data capture tool (spreadsheet) was easy to use.
Strongly Agree
Agree
Neither Agree nor Disagree
Disagree
Strongly Disagree

6.7%
33.3%
33.3%
20.0%
6.7%

The edits helped identify situations where the data may have been inconsistent or incorrect.
Strongly Agree
Agree
Neither Agree nor Disagree
Disagree
Strongly Disagree

0.0%
40.0%
46.7%
13.3%
0.0%

Work continued towards a Phase 3 Data Capture system during Phase 2 testing. Acceptance testing was conducted by
BLS. At the end of Phase 2, the first version of the production data capture application was deployed and used in a
parallel test along with the data capture spreadsheet. Staff were to report back any issues or suggestions that they have
with the system in order to improve it for Phase 3. A tracker was implemented to identify system bugs and issues. The
new system was shown in several demonstrations throughout Phase 2 to demonstrate progress and request feedback
from staff.

Data Analysis
BLS staff were tasked with preparing edits to determine the validity and accuracy of the data gathered in Phase 2 as well
as creating calibration activities to help ensure inter-coder reliability in data collection.
Calibration Activities
One activity was conducted in each of the two test cities prior to the beginning of collection to help ensure that different
field economists code the same specific vocational preparations, physical demands and environmental conditions for a
job the same way during a collection interview. This also served to augment the training provided to the field
economists by emphasizing best practices developed throughout the previous test phase.
Each field economist participating in collection for the city was given a dental hygienist job description and a single
quote data collection tool. Field economists were instructed to code all elements in which they felt there was sufficient
17 | P a g e

supporting documentation in the job description. They also were asked to make notes in places in which they thought
there was enough information to code the data element but would be more comfortable if they could ask additional
questions of the respondent. In such instances, the field economists also were asked to note the specific questions they
would ask. Additionally, field economists were asked to note instances in which they could determine the existence of a
required element but would be unable to determine the frequency.
Once all participants completed their coding based upon the job description, facilitators led the group through a
discussion that looked at each question on the collection tool as well as the answers arrived at by the field economists
and why they gave each particular answer. Additional discussion was generated by the facilitators once the targeted
answers were revealed so as to emphasize core collection concepts such as what is required for the job as opposed to
just preferred by the employer, the threshold which needs to be reached for certain elements to be coded as present,
and activities that are considered incidental to the job and should not be coded. An answer key with the target answers,
the job description point of reference supporting the targeted answer as well as the DOT coding was provided to the
field economists once discussion had concluded.
Data Analysis – Phase 2 Report Summary
BLS staff devised tools, procedures, and systems for completing analysis and validation of the micro data collected
during Phase 2. These included one calibration activity in each test city, the refinement of review parameters from
Phase 1 for data analysis, and the data review itself. The calibration activities were intended to ensure consistency in
coding among field economists with regards to SVP, physical demands, and environmental conditions. For the review
parameters, staff analyzed data collected during Phase 1 to develop expected relationships between each data element
which were then integrated into the data capture spreadsheets completed by the senior BLS field economists.
Excel formulas were developed to identify coded data elements that did not meet these expectations. For data review,
two approaches were taken. Schedules first were reviewed by BLS staff for consistency using the edits developed as part
of the review parameters. Review questions concerning any data entries which were flagged as potentially inconsistent
were sent to the field economist who collected the schedule in question. Their answers indicated whether data had
been changed, documentation added, or both. Further analysis of schedules then was performed using SAS software. A
comparison was made by SOC code between the coding of each element done in ORS Phase 2 and the value given by the
DOT.
ORS values were higher on the frequency scale than DOT values more often than DOT values were higher than ORS
values. Gross manipulation was the element with the largest proportion of occupations with higher ORS values than
DOT values: 40% of occupations had higher DOT values than ORS values. Speaking had the largest proportion of
differences, with 72% coded higher in ORS than the DOT value. The two elements dealing with the use of the hands,
Fine Manipulation (fingers) and Gross Manipulation (whole hand) had the lowest proportion of matches with large
proportions of ORS coding both higher and lower than the DOT value.

18 | P a g e