Attachment 9 - ORS Phase 1 Summary Report

U.S. DEPARTMENT OF LABOR
BUREAU OF LABOR STATISTICS

Occupational
Requirements
Survey
Phase 1 Summary Report
Fiscal Year 2013

1|Page

Executive Summary
The National Compensation Survey (NCS) recently completed Phase 1 of the Occupational Requirements
Survey (ORS) tests conducted in cooperation with the Social Security Administration (SSA). The main
objective of the three ORS tests in fiscal year 2013 is to assess whether it is feasible for BLS to collect
data relevant to the SSA’s disability program using the NCS platform. The results of the Phase 1 proofof-concept test suggest that this approach is viable. Respondents agreed to participate in the test, BLS
field economists were able to capture the required data from traditional NCS respondents, and
individual data element response rates were very high.
Field collection for Phase 1 testing was conducted in the greater Washington, D.C. area from November
28, 2012 through December 6, 2012. Establishments were selected from the current NCS sampling
frame, excluding establishments currently active in any NCS sample. Nine experienced field economists
from BLS regional offices collected the data, and each interview was observed by a BLS national office
staff member or an SSA representative.
Upon completion of the ORS collection, respondents were asked questions to gauge their reactions to
the survey, and BLS field economists and observers completed a post-interview debriefing. Daily
debriefing sessions were held with BLS field economists, observers, and other staff for the purposes of
discussing interviewing experiences, identifying potential issues with the materials and procedures, and
sharing lessons learned. A final end-of-phase debriefing session summarized the major findings
identified during the test period and allowed for expanded discussion of these and other issues between
BLS and SSA staffs.

Key Findings
The results of the Phase 1 test were very promising overall. Test objectives were successfully met and
ORS has a strong foundation on which to build for future development and testing.










BLS field economists completed interviews with 27 establishments, collecting detailed information
for 104 occupations. It took between 8 and 10 minutes to collect information for each occupation.
BLS field economists had minimal difficulty gaining cooperation of sampled establishments. Many
BLS field economists noted that one of the most effective pieces of information to gain cooperation
was the high name recognition of the SSA.
Most data were collected from traditional NCS respondents such as human resource directors, small
business owners, and location managers.
Cooperating establishments were able to answer the vast majority of questions asked in the survey
resulting in a very high item response rate.
The training approach used in Phase 1 worked very well and will be repeated in subsequent tests.
The use of daily debriefings where BLS field economists, observers, and other staff could exchange
information and suggestions about collection issues was particularly successful.
The materials prepared to aid ORS cooperation were very well received by respondents.
It is not enough to simply record the respondent’s answer; it is important to verify consistency
across questions. For example, is it possible to sit 6 hours a day and also climb stairs frequently?
Certain words and concepts proved to be unclear or confusing to respondents. These included
“required,” “accommodations” and prior work experience.

As a result of our collection efforts during Phase 1, BLS identified several refinements that will be tested
and evaluated as part of Phase 2. These include refined guidance and protocol for collecting the
Physical Demand Strength and additional choices for the frequency scale.
2|Page

Background and Test Overview
In 2012, the Bureau of Labor Statistics (BLS) signed an interagency agreement with the Social Security
Administration (SSA) for the purpose of designing, developing, and carrying out a series of tests to
assess the feasibility of using the National Compensation Survey (NCS) platform as a means to accurately
and reliably capture data relevant to the SSA's disability program. The resulting initiative–the
Occupational Requirements Survey (ORS)–recently completed the first of three phases of testing
planned for fiscal year 2013 as outlined in the Interagency Agreement Deliverable sent to SSA on
September 28, 2012. That document outlines the work the BLS will perform, key objectives, and a
detailed test plan:
In FY 2013, the BLS will perform work to meet the following objectives:
1) Evaluate survey design options and begin developing the protocols, aids, and final design to
meet SSA data needs;
2) Collect data to test and refine the protocols and aids; and
3) Provide documentation to the SSA summarizing the work performed by the BLS, conclusions
drawn, and recommendations for future data collection.
In order to accomplish these objectives, the BLS will conduct a series of field tests with general test goals
as described below:
Phase 1- Initial Proof of Concept Testing: The primary goal of this phase of testing will be to
ensure that the BLS field economists know how to describe the survey and ask for the new data
elements. In addition, the BLS will create and test an initial set of data collection protocols and a
preliminary set of data collection aids.
Phase 2- Collection Protocol Testing: The primary goal of this phase of testing will be to test
collection of the new data elements while performing a selection of occupations from each
respondent. In addition, the BLS will refine the collection protocols and aids based on an
analysis of the test results.
Phase 3- Broad Scale Testing: The primary goal of this phase of testing will be to test the BLS’s
ability to select a sample of occupations within each establishment, collect the new data
elements needed by SSA, and collect other NCS data elements that are of research interest to
SSA such as wages and job leveling information. In addition, the BLS will test the feasibility of
collecting the data needed by SSA along with all the NCS data elements needed to produce the
Employment Cost Index, Employer Costs for Employee Compensation, and various benefits
products.
This report details the methods and findings of the Phase 1 test. Phase 2 begins in late January and
Phase 3 is scheduled to start in April.

3|Page

Phase 1 Test Objectives
The primary goal of Phase 1 testing was to ensure that the BLS field economists understood how to
describe the ORS and ask respondents for information regarding the new data elements. The key data
elements of interest were:
1. Specific Vocational Preparation (SVP) - an indicator of “time to proficiency,” defined as the
amount of time required by the typical worker to learn the techniques, acquire the information
and develop the facility needed for average job performance.
2. Physical Demand (PD) - characteristics or activities that an occupation requires of a worker,
measured in such a way to support SSA disability determination needs
3. Environmental Conditions (EC) of occupations that replicate as closely as possible those that the
SSA currently uses.
In addition, BLS staff created, implemented, and evaluated an initial set of data collection protocols and
data collection aids, and used the experience of Phase 1 testing to begin learning about the methods for
computing estimates, analyzing the collected data, and validating future estimates.

Test Logistics and Methods
Field collection for Phase 1 testing was conducted in the greater Washington, D.C. area from November
28, 2012 through December 6, 2012. Establishments were selected from the current NCS sampling
frame, excluding establishments currently active in any NCS sample. The target sample size for Phase 1
testing was 25 – 30 establishments representing as broad a mix of industries as possible given the size
and time constraints of the test. Nine experienced NCS field economists (FE) from the BLS regional
offices served as collected the data, and each interview was observed by a BLS national office staff
member or an SSA representative.
BLS field economists tested three alternative collection approaches. Upon completion of the ORS
collection, respondents were asked to complete a short questionnaire to gauge their reactions to the
survey. Daily debriefing session were held with field economists, observers, and other staff for the
purposes of discussing interviewing experiences, identifying potential issues with the materials and
procedures, and sharing lessons learned. A final end-of-phase debriefing session summarized the major
findings identified during the test field period and allowed for expanded discussion of these and other
issues between ORS and SSA staffs.

Key Findings
The results of the Phase 1 test were very promising overall.
BLS field economists completed interviews with 27 establishments, collecting detailed information for
104 occupations. As with any new collection effort, there were a number of issues that arose in Phase 1
that will be addressed and evaluated in subsequent phases of testing. However, the consensus opinion
of BLS staff is that the Phase 1 test objectives were successfully met and that ORS has a strong
foundation on which to build for future development and testing.

4|Page

The Phase 1 test collected data on an average of 4 occupations at each establishment, and it took
between 8 and 10 minutes to collect the ORS information for each occupation, depending on the
instrument used.
Phase 1 training consisted of self-study (background readings on the SSA disability program and relevant
ORS concepts), in-person instruction provided during ORS orientation meetings, and on-the-job training
(pre-collection practice interviews, debrief sessions, and informal field economist exchanges). The
training approach used in Phase 1 worked very well and will be repeated in subsequent tests. The use of
practice interviews prior to data collection and the use of daily debriefings where BLS field economists,
observers, and other staff could exchange information and suggestions about collection issues were
particularly successful.
The materials prepared to aid ORS cooperation, in particular a 1-page flyer explaining the background,
purpose, and specifics of the ORS test and the prepared script were very well received. In addition, many
BLS field economists noted that one of the most effective pieces of information at their disposal was the
name recognition of the SSA.
BLS field economists expressed a preference for the collection approach which collected information
one occupation at a time. Respondents had very positive reactions to the respondent visual aid, and
field economists had some suggestions for improving the aid, such as removing the ‘extreme heat’ and
‘extreme cold’ scales and adding examples to improve collection of the lifting and carrying elements.
During the collection of data for Phase 1, it became very clear that it was not enough to simply record
the respondent’s answer. Answers to questions asked earlier in the interview may very well prove to
raise flags about answers to the current question. Is it possible to sit 6 hours a day and also climb stairs
frequently? Can a worker reach forward and to the side constantly?
The definitions in the Dictionary of Occupational Titles (DOT) do not provide enough information to code
all required data elements consistently. To supplement the definitions, BLS staff used examples of work
such as those provided in The Revised Handbook for Analyzing Jobs (RHAJ). These examples, too, had
limitations. For example, most are drawn from the manufacturing sector which has declined in
employment.
There were certain words and concepts that proved to be unclear or confusing to some respondents.
“Average performance” also was an issue in some interviews. The issue of accommodations and how to
explain why ORS excludes them came up consistently in the daily debriefs. One successful approach
used by field economists was to explain that the focus of the survey is the job and not the individual
doing the job, and that individual accommodations at an establishment should not be collected. The
matter of prior work experience also proved to require greater explanation for some respondents. There
are amounts of previous work experience that are preferred and there are amounts that are required.
Another topic of discussion among the BLS field economists was the measurement of the SVP scale. The
intervals are very wide between some steps and the credit allowed for undergraduate education
resulted in some coding abnormalities that yielded some unexpected results, with certain occupations
having an SVP much higher or lower than expected.

5|Page

Phase 1 Test Methods and Materials
Field collection for Phase 1 testing was conducted from November 28, 2012 through December 6, 2012.
The testing location was the greater Washington, D.C. area.

Selected Establishments
Establishments were selected from the current NCS sampling frame for businesses in the greater
Washington, D.C. area, excluding all establishments currently active in any NCS sample but including
some units recently rotated out of NCS production. The target number of completed interviews for
Phase 1 testing was 25 – 30 establishments representing as broad a mix of industries as possible given
the size and time constraints of the test. At the conclusion of the testing period, 27 interviews were
completed (see Results section for more details about participating establishments).

Field Economists
The data collectors for the Phase 1 test were 9 experienced NCS BLS field economists from the BLS
regional offices. Interviewer training for Phase 1 collection followed regular BLS practices and consisted
of three components:


Self study – BLS field economists were provided the SSA background materials, researched SSA
website, and reviewed the Dictionary of Occupational Titles (DOT), among other documents.



In-person training – This occurred during two meetings held at the BLS National Office. In the
ORS Orientation meeting (October 23-24, 2012), project managers introduced BLS field
economists to the ORS program and SSA representatives presented an overview of the disability
determination process and the required data elements. At the Phase 1 Test Kick-off Meeting
(November 27, 2012, also at BLS), SSA provided additional background about the disability
adjudication process (including physical residual functioning capacity assessment), and the BLS
field economists were provided a document containing detailed instructions for the Phase 1
test.



On-the-job training (OJT) – Leading up to and throughout the Phase 1 Test fielding period, BLS
field economists engaged in a number of activities designed to reinforce formal ORS training
concepts. Prior to data collection, each field economist conducted at least two practice
interviews with BLS staff unaffiliated with the ORS and NCS (e.g., Human Resources personnel,
program managers), and participated in a calibration exercise in which all BLS field economists
observed the same practice interview, individually recorded ORS information, and then
compared and discussed their answers with one another. During data collection, OJT was
provided through formal daily debriefing sessions and informal conversations between BLS field
economists and other staff in which ‘lessons learned,’ ‘best practices,’ and challenging issues
were identified.

Refinement Procedures
Field economists were provided with the establishment listings and case assignments upon arrival to the
National office. Each field economist was given a list of private sector establishments to recruit, and 5
field economists were additionally asked to collect for one government sector unit. The BLS field
economists were notified that the numerical order of their assignments represented the collection
preference, with the lower number being preferred.
Once the field economists had their assignments, they began to research the establishments using the
Internet. The collectors made phone calls to contact potential respondents and attempted to speak
6|Page

with company personnel with the same job titles that NCS traditionally works (e.g., HR Directors,
Personnel Managers, and Hiring Officials). If those individuals were not available, the field economist
worked with the establishment contact to identify the person most familiar with the target data
elements to interview. When the appropriate company contact was located, the field economist used a
standardized script to explain the purpose and importance of the survey, and attempted to schedule an
in-person appointment (phone interviews were not permitted as part of Phase 1 testing). Some of the
collectors sent e-mails to establishment contacts to provide additional background information about
the purpose of and procedures for the interview. All potential respondents were informed that the data
collection effort was part of a test being done at the request of the Social Security Administration and
that participation was voluntary.
When contact could not be made with listed establishments, or the establishment points of contact
expressed reluctance or indicated that they were unavailable during the test field period, the field
economists were instructed to forego the normal NCS cooperation attempts and simply advance to the
next establishment on their assignment list. This process continued until each field economist had
secured three viable appointments.

Occupational Selection for Responding Units
At the start of the interview, BLS field economists collected or verified information about the
participating establishment (e.g., total employment, NAICS code), and then selected between 4 and 6
occupations (as time and cooperation allowed) for the ORS collection. There was no formal sampling
process for occupational selection, but field economists attempted to select occupations that were (a)
highly populated in the establishment, (b) representative of a range of major Standard Occupational
Classification (SOC) codes, and (c) convenient for the respondent to report. Field economists were
instructed to select the most narrowly defined level of an occupation (i.e., the most detailed, distinct
establishment description for an individual job), and to avoid selecting combined jobs. Neither
occupational work schedules nor NCS occupational characteristics (e.g., full-time/part-time, union/nonunion, time/incentive) were collected, except for situations in which it was apparent that the
occupational requirements varied by these characteristics.

Data Collection Approaches and Protocols
Three collection approaches were tested in Phase 1. Each version was designed to collect the required
data elements (i.e., Specific Vocational Preparation (SVP), Physical Demands (PD), and Environmental
Conditions (EC)), but they varied in their format, question flow, and collection method.





Collection Approach A was designed to obtain the occupational requirements one occupation
job at a time.
Collection Approach B also collected occupational requirements one occupation at a time. The
main difference was that the response options in Approach B were grouped together under a
single question stem where appropriate. The purpose of this approach was to obviate the need
for field economists to repeat the same question for related items
Collection Approach C collected the occupational requirements of all selected jobs
simultaneously.

In order to test the effectiveness of each collection approach, field economists were asked to use a
different collection approach during each of their three personal visit interviews.
A set of aids for respondents and interviewers was also developed. The field economist visual aid listed
the SSA-provided definition and examples for each data element, as well as definitions of the
7|Page

measurements of frequency (i.e., never, occasionally, frequently, constantly). BLS field economists were
encouraged to consult this aid during the interview, and to share it with the respondent as necessary.
An additional, two-sided respondent visual aid provided a place for BLS field economists to list each of
the jobs being surveyed, the frequency definitions, and examples for the noise intensity level response
categories. BLS field economists were told to use the respondent aid during every interview. Finally,
BLS field economists were provided with a document that contained answers to questions the
respondent may ask; they familiarized themselves with this document prior to the interview, and could
refer to it in the event that questions arose during the appointment.

Data Capture and Review
Data, including establishment information, were entered into a data capture spreadsheet on a flow
basis. The spreadsheet was designed to permit easy data entry by BLS field economists, to provide the
ability to review the captured data, and to ensure the capacity to tabulate results. Table 1 lists the data
elements collected in the spreadsheet.
Initial review parameters were developed for Phase 1 and were used to review and analyze data
elements for internal consistency. Specifically, three internal consistency spreadsheets (one for each
SSA data element area – SVP, PD, and EC) were used in the data review and analysis. The parameters
identified expected relationships between the individual ORS data elements, as well as consistencies
between ORS and NCS data elements.

Table 1. Information Recorded in the Phase 1 Test Data Capture Spreadsheet
Element
Items Captured

General Information

Schedule number; establishment employment size;
NAICS code; selected occupations (title, SOC,
employment, FT/PT, work environment, job
description)

Specific Vocational Preparation (SVP)

Title; SVP; SVP level; education level

Physical Demands (PD)

Standing/Walking/Sitting; Reaching; Lifting/Carrying;
Pushing/Pulling; Gross Manipulation/Fine Manipulation;
Keyboarding; Hand Arm Controls/Foot Leg Controls;
Climbing/Stooping/Kneeling/Crawling/Crouching;
Visual Acuity; Speaking

Environmental Conditions (EC)

Extreme Cold/Extreme Heat; Wetness/Humidity;
Vibrations; Fume/Odors/Gases/Poor Ventilation;
Toxic Caustic Chemicals; High Exposed Places; Moving
Mechanical Parts; Noise Intensity Level; Loud
Noise/Moderate Sound Discernment

Observers
To help garner feedback about the interview and data collection processes, an observer accompanied
the field economist on each data collection appointment. The observers represented a mixture of roles
8|Page

and stakeholders within the ORS Project (e.g., BLS survey methods and operations staff; NCS/ORS
management; SSA officials); with priority given to those individuals serving on ORS related teams but not
serving as a collector. Each observer participated in only one interview.
The observer did not participate in ORS data collection, but was responsible for recording information
about the interview. They noted the duration of the interview (and the time needed to administer
sections within the interview), and their observations for each item and the interview as a whole.

Phase 1 Other Debriefing Activities
In addition to the observer form, BLS assessed the effectiveness of Phase 1 ORS materials and
procedures by conducting the following debriefing activities:


Respondent Debriefing – At least 10 minutes were set aside at the end of each ORS interview
to ask the respondent about their interview preparation and experience. This debrief was
administered either by the field economist or the observer, using a respondent debriefing
outline with 11 questions targeting potential comprehension issues, perceptions of burden,
reactions to the visual aids, etc.



Post-Interview Debrief – Immediately after leaving each ORS appointment, the field economist
and observer jointly answered 10 questions that asked whether they observed any data
element comprehension issues, if the interview process and materials seemed effective, and if
there were any suggested improvements for ORS collection (either from the respondent or the
field economist or the observer).



Daily Collection Debrief – Each day during the Phase 1 test in which there was a collection, an
in-person debriefing session was held to assess how the daily interviews went from the
perspective of both the field economist and the observer. These sessions were moderated by a
facilitator and had a dedicated note taker to ensure that all aspects of the discussion were
captured. All 9 BLS field economists (whether they conducted an interview that day or not) and
the observers who went out on collection interviews that day participated in these daily debrief
meetings. Others involved in the ORS project attended, as available.



End of Phase Debrief – On December 6, 2012, the Debrief Team conducted a final debriefing
session. The purpose of this session was to summarize key findings from the Phase 1 test
based on all the information collected in the various assessment activities, to solicit additional
feedback about components of the test, and begin to identify issues and recommendations
relevant to Phase 2 preparation. The meeting was attended by BLS staff and by SSA sponsoring
officials.

Phase 1 Test Results
This section reports on results compiled throughout the Phase 1 test. They reflect information obtained
from each debriefing component and feedback collected from observers and interviewers during the
collection and end-of-phase debriefing sessions. Although the information garnered from each of these
components is summarized here and not reported individually by source, each source contributed
significantly to the overall Phase 1 findings and conclusions. This section begins with a broad
assessment of the feasibility of collecting the data needed by SSA through the ORS, then provides an
overview of the collection effort and presents more detailed descriptions of issues that arose during the
test.
9|Page

Feasibility
The main objective of the development and evaluative work that will be done for the ORS Project in FY
2013 is to assess whether it is feasible for BLS to collect data relevant to the SSA’s disability program.
The results of this initial proof-of-concept test suggest that this approach is viable. Respondents agreed
to participate in the test and BLS field economists were able to capture the required data from
traditional NCS respondents. In Phase 1, BLS field economists completed interviews with 27
establishments, collecting detailed job requirement information for 104 occupations. As with any new
collection effort, there were a number of issues that arose in Phase 1 (e.g., with certain data elements,
or aspects of the collection and assessment) that will be addressed and evaluated in subsequent phases
of testing; many of those are reflected in the sections that follow. However, the consensus opinion of
BLS staff is that the Phase 1 objectives were successfully met and these activities lay a strong foundation
for future tests.

Participating Establishments
Table 2 provides additional detail about the establishments that participated in Phase 1, and the
occupations for which ORS data were collected. BLS field economists were successful in securing
interviews from a variety of industry groups and collecting data for a range of occupations in an
extremely tight fielding period.
Table 2. Description of Establishments and Selected Occupations for Phase 1 Test
Industry Group

Private Industry,
Education

Private Industry,
Financial Services

Number of
Participating Units

4

4

Employment
Size
(range)

15 – 200

4 - 80

Number of Jobs
Collected

17

14

Jobs Collected
(selected)










Private Industry,
Goods Producing

Private Industry,
Health Care

4

4

12 – 114

3 – 85

15

16

Private Industry,
Services

5

3 – 187

14

State and Local
Government

6

65 – 775

28

10 | P a g e















Teachers
Management
Office and Admin. staff
Installation, Maintenance
and repair
Property manager
Laundry and dry cleaning
Financial specialists
Audio/Video equipment
technicians
Mechanics, pipefitters,
and electricians
Management
Installation,
maintenance, and repair
Healthcare practitioners
Office and administrative
support
Home health aide
Sales
Hair stylist
Store manager
Housekeeping
Faculty
Registered nurses
Police officer
Clerical

.

Data Collection Appointments
ORS field economists reported that their pre-collection activities included searching for the company’s
phone number, conducting Internet searches to find establishment websites, gaining cooperation and
similar activities. For the Phase 1 test, BLS field economists were instructed to move on to the next
establishment on their list in the event that they encountered reluctance. This allowed BLS field
economists to focus their time on collecting the needed test data as opposed to spending hours
attempting to get cooperation from respondents who did not want to cooperate.
Notwithstanding these procedural differences, field economists reported that it took 30 to 60 minutes
to secure an appointment. Once BLS field economists were able to speak with the respondents, they had
a range of experiences securing the appointment: Some set up appointments in as little as 5 minutes,
while others had to go through several layers of contacts and information sharing (e.g., with an
establishments’ national corporate office or legal department) before being able to make an
appointment. Another limiting factor encountered by ORS field economists was the ever-presence of
voicemail which sometimes made initial contacts difficult.
The BLS field economists reported that the materials prepared to aid in data collection were very useful
in addressing any concerns raised by their establishment contacts. Several field economists referred
respondents to the SSA and BLS websites as well, and found that to be a useful tactic. One of the most
effective pieces of information at BLS field economists’ disposal was the name recognition
establishment contacts had for SSA – most field economists reported that this was instrumental in
securing test participation.

Interview Duration
Based on the approved testing plan submitted to the Office of Management and Budget (OMB), BLS
field economists were allotted one hour to collect the ORS data and conduct the respondent debriefing
in each establishment. On average, data was collected on 4 occupations at each establishment.
Collection of the average occupation took between 8 and 10 minutes.

Collection Protocols
The Collection Approaches
Each of the three collection approaches had things that worked and things that did not, but overall there
were some universal comments and suggestions that applied to all three approaches:
1. Add questions that address prior work experience and other types of training besides a formal
college degree.
2. Group similar questions together. Switching from Yes/No to Occasionally/Frequently/Constantly
often caught the respondent off guard.
3. The order of the reaching questions could be tweaked to ask overall reaching last.
4. Ask about moderate sounds before asking about loud sounds.

11 | P a g e

Other Protocol Outputs
Introductory Script
During the Phase 1 collection the field economists reported that they were all using the prepared
introductory script, although they were making minor changes to it somewhat to reflect their own
personal style. There seemed to be no specific issues with the introductory script as it was presented in
Phase 1. Some field economists concluded there should be a greater emphasis on what specific
elements ORS would be asking about, while others thought confidentiality needed to be addressed at
greater length. However, neither of these views was universal. Some felt that accommodations needed
to be addressed up front at first contact, but this was also a minority opinion.
Visual Aids
One clear theme can be gleaned from reviewing all of the notes from the various debrief meetings – the
respondents loved their visual aids. Whether it is actually useful or just helps them feel involved in the
process, there was universal praise. There were, of course, some suggestions for improving the aid,
such as removing the extreme heat and cold scales and adding some weight examples.

Phase 1 BLS Field Economist and Observer Training
Phase 1 training consisted of self-study (background readings on the SSA disability program and relevant
ORS concepts), in-person instruction provided during ORS orientation meetings, and on-the-job training
(pre-collection practice interviews, debrief sessions, and informal field economist exchanges). BLS field
economists and observers reported that all of these activities were important and helped to ensure that
they were prepared for the Phase 1 test. The orientation meetings and SSA presentations provided
crucial information about the ORS Project objectives and elements, and the supporting SSA
documentation was mined heavily by ORS team when developing Phase 1 materials and procedures.
The observer information session was attended by all Phase 1 observers, and participants found the
session extremely useful, and even offered a number of suggestions that resulted in improved materials
and procedures.
Everyone was particularly positive about the benefits of doing practice interviews prior to data
collection. These interviews gave field economists and observers the opportunity to practice using the
collection approaches and observer methods, and affirmed that BLS field economists could ask the
questions and collect the data on all the SSA data elements. The daily debriefings also were excellent
training tools. This semi-structured forum provided an avenue for valuable information exchange about
collection issues. Because data collection scenarios all are unique, BLS field economists found these
debriefing sessions allowed them to discuss different types of situations and possible solutions to these
scenarios. Finally, the end-of-phase debriefing gave the BLS staff the opportunity to synthesize and
report on information gleaned from the spectrum of assessment activities, to raise questions relevant to
Phase 2 testing, and share Phase 1 experiences with our SSA sponsors.
BLS field economists and observers also underscored the importance of other informal, on-the-job
learning opportunities. These on-going conversations helped to solidify lessons learned, best practices,
and to identify areas in need of further development. For Phase 1 testing there was no formal
mentor/mentee process, but it did not stop the field economists from discussing the program objectives
and collection strategies amongst themselves.

12 | P a g e

Data Collection
The process of going out and collecting live data from actual respondents proved to be a fruitful and
valuable endeavor. Once the field visits began, the feedback at the debrief meetings began to flow. Field
economists became more familiar with the various concepts, procedures and collection approaches in a
more substantial and concrete way. It also was at this point that a lot of very specific questions
pertaining to definitions, concepts and fallbacks began to surface.

Respondents
Meeting with respondents and actually asking the questions allowed BLS field economists to gather
some valuable information on how these questions might be perceived from the other side of the table.
There were certain words or concepts that proved to be unclear or confusing to some respondents. A
good example of this was our use of the word “required.” Some respondents shied away from this
word, not wanting to be seen as “requiring” anything specific of their employees that could potentially
cause a legal issue. Some field economists theorized that the trepidation may also have stemmed from
fear of appearing to discriminate against those with disabilities.
Another concept that proved to be rather perplexing to respondents was the issue of “average
performance.” There was much discussion with certain respondents trying to clarify the concept. Many
respondents said something along the lines of “I wouldn’t hire them if they weren’t ready to work on
day one.” Some respondents also said it was difficult trying to answer this item when they were asked
to exclude any orientation period. Consistent interpretation of this concept across the respondent
population seems unlikely until BLS and SSA can provide additional guidance to field economists and
respondents.
The issue of accommodations and how to explain why ORS excludes them came up consistently in the
daily debriefs. One successful approach used by field economists was to explain that the focus of the
survey is the job and not the individual doing the job, and that individual accommodations at an
establishment should not be collected. Although it proved easy enough to explain that accommodations
should be excluded, it would be helpful to be able to clearly explain why they are being excluded in a
manner that is acceptable to all parties.
The matter of prior work experience also proved to require greater explanation for some respondents.
There are amounts of previous work experience that are preferred and there are amounts that are
required. Along those same lines there were ranges of prior work experience (5-7 years) collected.

BLS Field Economists
The BLS field economists tasked with the collection of the data in Phase 1 proved that they are skilled in
all aspects of their jobs - locating respondents, explaining complex survey concepts in plain language,
selling the importance of cooperating in the survey, guiding the respondent in their answers, and further
acting as a resource during the interview.
During the collection of data for Phase 1, it became very clear that it was not enough to simply check the
box corresponding to the respondent’s answer in the forms. Answers to questions asked earlier in the
interview may very well prove to raise flags about answers to the current question. Is it possible to sit 6
hours a day and also climb stairs frequently? Can a worker reach forward and to the side constantly?
The BLS field economists also found themselves having to act as a “human visual aid,” physically
demonstrating the various movements during the interview.

13 | P a g e

Another topic of discussion among the BLS field economists was the measurement of the SVP scale. The
intervals are very wide between some steps and the credit for undergraduate education resulted in
some coding abnormalities that yielded some unexpected results, with certain occupations having an
SVP much higher or lower than expected.

Phase 1 Debriefing Activities
Participation in the various Phase 1 debriefing activities (e.g., interview observations; completion of
observer, respondent debrief, and post-interview debrief forms; attendance at the daily and end-ofphase debriefing sessions) was very high. All interviews had observers, and there was a 100 percent
return rate on the observer forms and post-interview debrief forms. Respondent debriefing questions
were asked in all but two interviews (where time constraints prevented their administration), and the
team received completed forms for all available interviews. On average, the five daily debrief sessions
were attended by 20 BLS staff – including field economists, observers, team representatives, and
managers – and the discussions were active and productive. The end-of-phase debriefing session was
attended by all available BLS staff and managers, as well as representatives from the SSA.
Reactions to the five debriefing components were very positive. In particular, staff singled out the
respondent debriefings and daily debriefing sessions as crucial to Phase 1 assessment. Most collectors
and observers indicated that the respondent debriefings gave them insights that they would not have
had otherwise into respondents’ perceptions of the survey, potential confusions, and suggestions for
improving the questions and survey process. In addition, respondents seemed to really appreciate being
asked about their reactions and suggestions. Several observers felt that additional time could be spent
on this activity in future testing, and a number of people suggested that the questionnaire items should
be slightly changed for the next round of tests. It was thought by some that a greater emphasis should
be placed on gathering data pertaining to specific aspects of the collection, the questions asked, and the
order of questions as opposed to open-ended or general questions.
The in-house debriefing sessions were lively discussions in which both BLS field economists and
observers shared their interview experiences, identified potentially problematic items or procedures,
and exchanged approaches that they found helpful in clarifying meaning and securing collection goals.
These meetings also provided an opportunity for BLS field economists and observers to share stories
they heard from respondents about their personal connections to disability issues, and how that helped
to put a human face to the ORS. The end of phase debriefing provided the opportunity for BLS field
economists to summarize their Phase 1 experiences, and for SSA to ask questions about the Phase 1 test
design and to get answers from the people doing the actual collection.
Phase 1 also provided valuable information about how best to collect information during and
immediately following the ORS interview. The post-interview debriefing process entailed Field
economists and observers discussing their impressions of the interview and completing a structured
questionnaire. Many field economists and observers said that this process served as a good way to
organize their thoughts, particularly if they had a long commute back from their collection location.
Others, however, felt that the information coming out of the exercise was redundant and unnecessary,
and that the same thing would be covered in the daily debriefings.

14 | P a g e

Data Analysis
The Data Analysis team was tasked with preparing tools, procedures, and systems to determine the
validity and accuracy of the data gathered as well as creating calibration activities to help ensure intercoder reliability in data collection. The team conducted two calibration activities, developed initial
review parameters for data analysis, and completed the data review itself.

Calibration Activities
Two activities were conducted to help ensure that different field economists code the same specific
vocational preparations, physical demands, and environmental conditions for a job the same way during
a collection interview. The first activity was a "zero-point" calibration which took place during the Phase
1 collection training session prior to any data collection. The purpose was to get a baseline reading on
how similar the collectors' coding was for the same information and to help train the collectors on
where differing interpretations of the same information might arise. As a result of this activity, it was
apparent that there were some discrepancies between the ways different field economists coded
certain elements, especially when the information given by the respondent seems incomplete. The
second activity was conducted at the end of the Phase 1 debriefing session and after the Phase 1 data
had already been collected and coded. The purpose of this activity was to gather another reading from
the field economists and to facilitate reflection on individual collector's interview techniques and
understanding of the questions. Of particular salience during this activity was the extent to which a
collector was willing to challenge a respondent's answer to an element or ask more probing questions to
verify if the respondent understood the question correctly and was providing accurate information. This
activity showed much greater agreement in coding among field economists than the first activity.

Data Analysis Tool Development and Results of Data Review
For the initial review parameters, the team reviewed all individual data elements to develop expected
relationships between each data element which were then integrated into the data capture
spreadsheets completed by the field economist. A total of 515 unexpected conditions were created and
Excel formulas were developed to identify data elements coded that did not meet these expectations.
For data review, two approaches were taken. Schedules were first reviewed by members of the Data
Analysis and Review staff for consistency using the edits developed as part of the initial review
parameters. Review questions concerning any data entries which were flagged as potentially
inconsistent were sent to the field economist and the answers indicated whether data had been
changed, documentation added, or both. Further analysis of schedules was then performed on the
data. SVP variability was compared by SOC group, with the result being that the SVP range for several
groups was lower than expected. Frequency tables of responses were also compiled, indicating that the
answer “Occasionally” was chosen far more often by respondents in Phase 1 testing than would be
expected based on comparison to the DOT. In SOC codes which had jobs from multiple establishments
there was more variation in all the elements – SVP, physical demands, and environmental conditions –
than was seen for shared SOC codes within a single establishment, which might be a sign of differing
interpretations or confusion with certain concepts. The analysis and review staff also tabulated data on
each edit to evaluate their effectiveness. Eighty-three total questions were asked of the field
economists and resulted in 31 changes to data. In 14 of the cases where the field economist did not
change the data, however, the field economist indicated they would have called back the respondent for
clarification had policies been different.

15 | P a g e