ORS Response to 30 day comments

Joint BLS and SSA replies to ORS 30day FRN Sept. 30, 2014

Basic Methodology comments and replies

Comment [SkillTRAN, IARP, NASPPR, Barros-Bailey, NIOSH, Veith]:

These comments relate to the lack of a clear provision for observing the work performed.

Reply 1:

The collection of information in a Federal survey must balance the needs of maximum utility for data users against the burden placed on the respondent participating in a voluntary survey. The tasks performed by many occupations vary over a workday or longer period, increasing the complexity of coordinating representative observations for each company job. Including formal observation as part of the data collection would increase respondent burden, which typically decreases initial response rates. Access to worksites for performing job observations is an additional concern due to safety, security, and cost issues.

Comment [SkillTRAN]:

33,000 responses will be obtained at the O*NET level of occupational classification during the full survey. That works out to about 33 responses per occupation. A huge issue that has not been addressed in any of the NCS or SSA documents is whether that is a sufficient sample size given the wide range of Strength and Specific Vocational Preparation (SVP) expected within many of the O*NET groups.

Reply 2:

Examination of the amount of data available from the ORS at the occupation level is a topic under continued evaluation and will receive further attention in the OMB clearance package for collecting data in a production environment. The final number of establishments to be included in ORS estimates, the number of occupations for which data will be provided by each, and the expected survey response rates will all factor into this evaluation. BLS will also be conducting evaluations of the dispersion of data estimates within individual SOC codes as more data is collected for ORS.

Comment: [SkillTRAN]

...there will be enormous variability in reported results and that the reported ranges will be so large as to render the data essentially useless in developing an understanding of what an occupation requires of the worker.

Reply 3:

ORS will classify occupations initially using the SOC as federal agencies gathering occupational data are mandated to do. We will actually use the O*NET-SOC which provides slightly more detail. We will analyze the data spread once we have a sufficient amount of data collected to populate a usable number of SOC codes. We will determine at that time whether we need to further break down the O*NET-SOC coding, possibly by industry.

Comment: [Karman]

The BLS-sponsored ICR for the Occupational Requirements Survey is premature for these reasons at a minimum:

1. No report of quantitative analyses of the 2013 occupational data results has been published

2. Sample design concerns that Rhein et al raise regarding possible compromise of the NCS and/or the ORS and three integrated sample simulations indicate some but inadequate overlap of industries needed for NCS; and,

3. The 2014 ORS feasibility test results have not been published.

Reply 4:

1. The 2013 testing was designed to determine the feasibility of collecting the data and was not collected from samples representative of the universe. No estimates meeting BLS publication criteria were produced from the data. Information from the 2013 tests is available on the BLS internet page at www.bls.gov/ors as part of the Phase 1, Phase 2, and Phase 3 Summary Reports.

2. The sample design continues to be evaluated and revised using the information obtained through the FY 2013 and FY 2014 feasibility tests as well as additional statistical analysis and simulations. Results from the continued design work are documented in a paper titled “Occupational Requirements Survey Sample Design Evaluation” by Gwyn Ferguson, Erin McNulty, and Chester Ponikowski (See Attachment 14). This paper will be published on the BLS website in October 2014.

3. The 2014 ORS feasibility tests did not conclude until July 2014. Information from these tests are now available in Attachment 13 and will be published on the BLS website in October 2014.

Comment: [SkillTRAN]

How many people are employed at a specific level of Strength within an occupation also
remains as a critical missing dimension in actually using this data to adjudicate claims at
the ODAR hearing level. While NCS has indicated that reporting of values may be
broken out at the two digit level of NAICS industry classification, this will NOT be a
sufficient level of detail to reduce this variability to a tolerable level with practical utility.

Reply 5:

The strength level is a calculated estimate from several discrete collected elements. The level of detail published on an occupation or industry basis will be determined by confidentiality and other publication criteria. We anticipate that many occupations, aggregated at the O*NET-SOC, will be represented in significant numbers at varying “exertional” levels. If this is the case, BLS might be reporting ranges of exertion for occupations. We will not know the utility of using the NAICS coding in combination with SOC coding until we have data that we can analyze.

Comment: [SkillTRAN, Karman, IARP, NASPPR, Barros-Bailey, A-T Voc Services, NIOSH, Veith]

These comments all question of the validity of collecting the cognitive and physical demands without direct observation of the work performed, direct measurement, or direct collection from workers. NIOSH comment regarding similar O*Net content and coverage is relevant since that data set is a potential ORS data validation source.

Reply 6:

Collection of occupational data may take multiple forms, including those direct methods recommended. Surveys, tasks lists, observation, recording jobs being performed, and examining archival materials (training manuals or videos) are some common methods.  Sources of data may include job incumbents, supervisors, human resources officials, or subject matter experts.

Validation of data may involve obtaining data through a different data collection approach, through different data sources, or both.  Many studies exist that seek to validate occupational data and the approaches to validating the data vary widely depending on the context of the study. Validation is a broad term that can refer to verifying both collection methods and outputs. BLS processes typically validate final estimates against expectations developed using internal and external sources.

In the case of the Occupational Requirements Study, there are multiple possible approaches to validating data.  The “best” approach is likely dependent on both the characteristics of the occupation and the data elements to be validated.  Research to identify appropriate approaches for validating ORS elements began in FY 2014 and is on-going.  A contractor is being retained to assist us with that effort and validation tests will be structured and performed on subsets of the pre-production data (both subsets of the data elements and occupations) during FY 2015.

In June the BLS issued a Request for Information (RFI), "Response to Occupational Requirements Literature Review RFI" and received interest from contractors.  As a result, in July BLS issued an “Occupational Requirements Literature Review” Request for Proposal (RFP).  The statement of work specifies that the contractor will evaluate BLS internal research on existing literature regarding the methods of collection of occupational requirements data and approaches for testing validity and reliability of such data.  The contractor will complete the literature review and provide recommendations regarding further research efforts centered on measuring data validity and reliability.

This RFP requests the following technical experience from the contractor: One subject matter expert will perform the work under this task.  This expert should have at minimum a Masters degree and eight years of experience in the research of occupational requirements.  Research experience in occupational requirements should include the following topics: trends in occupational requirements, methods for collecting national data on occupational requirements, and analysis of occupational requirements data.

Comment: [IARP]

The interview period for each occupation remains significantly low when considering the data being gathered for multiple occupations. ... Based on our experience conducting field job analyses, this is a very limited amount of time to gain a clear understanding of the demands of an occupation given the number of data elements under Physical Demands and Environmental Conditions.

Reply 7:

The collection of information in a Federal survey must consider the burden placed on the respondent participating in a voluntary survey. Based on BLS experience collecting ORS data in FY2013 and FY2014, the time allocated for the interview is an appropriate average to use for pre-production collection.

Comment: [IARP, NASPPR]

As we have commented on in the past, it is important to collect not only duration of an activity, but also frequency.

Reply 8:

During data collection the field economist discusses the duration and frequency of activities with a respondent. The average total duration for an activity is reflected in the coding. The level of data captured reflects the intended data output as well as the costs of data entry and developing the required systems for data capture, review, and estimation.

Comment: [NASPPR, Workability, Veith]

Although HR personnel are the appropriate point of contact when gathering information on wages and benefits for the NCS, they do not always understand or know the real physical demands of many jobs the way the worker or the direct supervisor does. We understand that the field economists are collecting data directly from businesses through their human resources and management staff as they do when collecting data for the NCS. However, there is neither clear provision for direct observation of the work being performed, nor is there interaction with direct supervisors or workers whom have more intimate knowledge of the requirements of the work, especially the physical and cognitive demands of the occupations.

Reply 9:

The actual respondent title and expertise varies by firm and firm size. Respondents reflect individual corporate policies on who is authorized to provide information. Field economists often initiate contact with companies through personnel or human resource staff, but may have multiple respondents within a company providing different information based on expertise. The goal is always to find the best data source while balancing response and burden concerns. Human Resource and management staff are familiar with the requirements of an occupation from the employer hiring and performance perspective. This helps ensure the answers reflect the establishment needs and not how a job has been adapted to a particular worker’s skills or personal style. During data collection, many respondents will either consult with supervisors or employees with the field economist present, or will obtain information from more knowledgeable staff and provide it to resolve particular questions.

Workability

Coding: The NAICS Industry Classification for the establishment should be captured along with the O*NET-SOC Code

Reply 10:

This is already done.

Comment: [Barros-Bailey]

Scaling: BLS made some slight scaling adjustments to the instrument that were appreciated. Not withstanding, the duration range of “seldom” set at 2% is too narrow, while the remaining ranges that break down a workday into thirds are overly broad when considering the frequency and duration of human function data elements considered.

Reply 11:

The conceptual division of the workday into thirds is driven by the construct of the DOT output. BLS added the 2% range during early testing to provide an easier slotting of those activities that occur only on a rare basis. Never is not an accurate categorization and respondents were reluctant to include rare tasks in a category that spanned up to a third of the day. For most elements, the duration will be captured on a finer basis at the microdata level. This provides BLS flexibility to adjust the ranges in published outputs to reflect user needs.

Comments: [Workability, A-T Voc Services, Veith]

Both of these comments mention the training of Field Economists.

Reply 12:

BLS field economists are not trained in industrial hygiene or physiology. They are trained in interview methodology to obtain information from establishments on a voluntary survey basis, how to perceive the attentiveness of the response to the question, and how to evaluate the data collected to determine when additional information is needed or conflicting information is reported. In respect to the ORS collection, staff are provided classroom conceptual training and on-the-job mentoring on the very specific concepts collected. Calibration activities are conducted to improve consistency and reliability across field economists.

Comment: [SkillTRAN]

SkillTRAN advocates for all levels of appropriate technology, and particularly measuring instruments such as weight scales, tape measures, and decibel meters be used to provide objective and reliable assessment of these key values in the vocational planning/testimony process.

Reply 13:

We must balance the need for occupational data with data quality, respondent burden, and the needs of the SSA disability program. Disability adjudication does not require the level of detail described. Because SSA processes large numbers of claims in a short period of time, the program makes more generalized assessments of claimants' ability to perform work. For example, SSA primarily uses information about the workplace noise intensity level to restrict claimants with non-listing level hearing loss from noisy environments.

Comment: [Karman]

Regarding the ORS’s physical, mental-cognitive, and other data elements, the exact origins and rationale for inclusion of the selected constructs is unclear, particularly those intended to capture mental-cognitive requirements. The BLS may find SSA’s prior work, Disability Evaluation Constructs (2011), to be helpful.

Reply 14:

When SSA developed the mental-cognitive data elements, the Disability Evaluation Constructs (DEC) document was the starting point. The DEC contains a list of all of the possible elements that may be useful to disability adjudication that were gathered from external experts and public comment from the Occupational Information Development Advisory Panel. SSA also reviewed the results of an internal study that gathered information from 5,000 disability claims decided at the initial and hearings level at steps 4 and 5 of sequential evaluation. SSA used the information on mental limitations from claims folders to examine the DEC and develop a manageable set of data elements that describe the mental demands of work, with a focus on the demands of low skilled work. SSA also leveraged the expertise of the BLS regarding the NCS leveling procedures when it was necessary to make refinements and adjustments to mental-cognitive questions based on the results of testing.

Comment: [Barros-Bailey]

Cognitive Elements: In my 5/23/14 public comment, I questioned the origin of mental-cognitive data elements and importance of understanding the body of literature from where these emanated. There was no response to this query. The majority of cases undergoing disability adjudication at Steps 4 and 5 in SSA’s disability program involve some aspect of mental-cognitive factors. An attempt to find these elements within the ORS instrument in the academic and popular literature did not result in support for these elements, particularly vis-à-vis the existing elements upon which SSA now makes disability decisions as reflected in the Mental Residual Functional Capacity form (MRFC).

Reply 15:

When SSA developed the mental-cognitive data elements, the Disability Evaluation Constructs (DEC) was our starting point. The DEC contains a list of all of the possible elements that may be useful to disability adjudication that were gathered from external experts and public comment from the Occupational Information Development Advisory Panel. SSA also reviewed the results of an internal study that gathered information from 5,000 disability claims decided at the initial and hearings level at steps 4 and 5 of sequential evaluation. SSA used the information on mental limitations from disability claims folders to examine the DEC and develop a manageable set of data elements that describe the mental demands of work, with a focus on the demands of low skilled work. SSA also leveraged the expertise of the BLS regarding the NCS leveling procedures when it was necessary to make refinements and adjustments to mental-cognitive questions based on the results of testing.

The goal with the development of these data elements is to support current SSA policy. The mental-cognitive questions will not map to the MRFC checkboxes like the physical elements do because the checkboxes do not represent the MRFC. The MRFC is the medical consultant’s text summary, and it is unstructured.

SSA policy for evaluating whether a claimant has the ability to perform the basic mental demands of unskilled work includes, but is not limited to, evaluating whether the claimant can do the following:

• Understand, carry out, and remember simple instructions. We believe our Complexity domain in the ORS addresses this factor.

• Make judgments that are commensurate with the functions of unskilled work, i.e., simple work-related decisions. We believe our Work Controls domain in the ORS addresses this factor.

• Respond appropriately to supervision, coworkers and work situations. We believe our Personal Interaction domain in the ORS addresses this factor.

• Deal with changes in a routine work setting. We believe our Work Routine domain in the ORS addresses this factor.

SSA relied heavily on their policy requirements when drafting the mental-cognitive elements.

Comment: [Karman]

[T]he Federal government continues to face the same problem it always has with developing occupational data for disability evaluation: Psychometric data and labor market & compensation statistics differ in ways that preclude the immediate, unaltered use of a ready-made labor-market data collection/survey process (Barros & Karman in Foundations of Forensic Vocational Rehabilitation (Springer 2013)).

Reply 16:

The ORS survey collects occupational information about physical job requirements for SSA evolved over the full year of 2013 feasibility testing. We began testing the questions about the mental/cognitive job requirements in FY 2014.  The mental/cognitive survey questions are similar to what BLS uses to level occupations for the NCS, but they were tailored to obtain the type of mental/cognitive occupational requirements that relate to the functional limitations assessed by SSA in claimants with severe mental impairments. After SSA completed a thorough process to develop data elements that describe the mental and cognitive demands of work, many similarities were observed between SSA’s data elements and BLS’s leveling questions. In FY 2015 pre-production testing, we will continue to test the mental/cognitive survey questions to ensure that they are collecting the occupational requirements that SSA needs to adjudicate claims.

Comment: [IARP, NASPPR, Veith]

We note that no Industrial Organizational Psychologists are in the Social Security nor the NCS work groups. We view this as a deficiency, as IO Psychologists can bring great insight and clarity on the issue of occupational requirements. We understand from the 2014 Supporting Statement for a nationwide Pre-Production Test of the proposed Occupational Requirements Survey that cognitive psychologists are now used in the ORS program, but reiterate the need for the expertise of also including Industrial Organizational Psychologists who can address the convergence of the world of work and cognitive elements.

Reply 17:

The first three year of the OIS project were spent on research. At that time, SSA worked with Industrial and Organizational Psychologists through special contracts, and had I/O psychologists on the OIDAP. Professional organizations whose members are I/O psychologists have provided input. Once we started developing the survey questions, we considered the input from all the experts and input from SSA medical consultants and disability adjudicators. SSA only considered input that promoted the adjudication of disability claims under SSA's current policy.

Comment: [Workability]

I recommend that more opportunity be provided for professional discussion in a forum that engages all stakeholders. The development of ORS is a vital initiative that should not be dictated solely by the specific interests and funding by Social Security Administration. It deserves additional funding and there should be more opportunity for discussion with occupational health professionals who evaluate disability. There should be more collaboration on design of the ORS data collection with researchers from other agencies such as the National Institute of Occupational Safety and Health (NIOSH).

Reply 18:

The first four years of the OIS project were devoted to professional discussion and a forum that provided the opportunity for input from all stakeholders. First and foremost, SSA must develop an occupational information system that meets the program needs of SSA and facilitates disability adjudication. The data collected are established by SSA and reflect earlier input on the OIS project from multiple stakeholders.

The pre-production test is a final step in testing the ability of ORS to deliver these specific data to support SSA disability adjudication needs. As we transition beyond this test to a new phase of survey development, BLS will work with SSA to determine the best approaches to involving researchers from other agencies.

Comment: [Barros-Bailey]

Usability Analyses: At this juncture, BLS has had various rounds of data collection at different levels in the development and refinement of the ORS prototype instrument to OMB. There has been no mention throughout the materials as to the review of these initial data collection efforts with Disability Determination Services personnel (at the minimum) to get preliminary feedback from the front-line users of the usefulness of such data. Thus, there exists insufficient information even after a $41,000,000 cost to the taxpayer for the ORS that the front-line users of these data have had regular input into this process beyond research staff and management at SSA.

Reply 19:

SSA has received a significant amount of input into the development of the survey questions from experienced DDS and hearings-level adjudicators. SSA also plans to conduct usability analysis and seek front-line user input once BLS has shared data from pre-production testing that SSA can use to populate and test the OIS IT Platform, which is under development.

Comment: [SkillTRAN]

After the formal data collection is done, DO ask about accommodations.

Reply 20:

ORS collection is based on an establishment’s requirements for an occupation as the work is performed without accommodation. Not all employers can offer the same accommodations and work performed with an accommodation is not reflective of how a job is generally performed. Some employers may make what is generally thought of as an accommodation available to all employees regardless of disability. Adding a question about accommodations would increase respondent burden not directly related to the purpose of the survey.

Comment: [Workability]

Serious concerns about validity (usefulness and accuracy) of data gathered in this limited survey collection, given the design plan to:

• Exclude several very useful DOT factors that have related tests of worker abilities (manual dexterity, finger dexterity, general education development).

• Rely heavily on survey reporting from company officials that may not be the best source for information on occupational demands (There is no plan to statistically validate the data accuracy of this brief survey with measurement, observation and interviews with the actual job supervisor or lead workers).

• Have the survey administered by economists that don’t have formal training or expertise in job analyses (Functional job analysis was the foundation of DOT).

Reply 21:

SSA believes the data gathered in the pre-production test will be useful for the following reasons:

• SSA is tailoring the new OIS for use in their disability programs. Under SSA policy, adjudicators only consider mental and physical limitations that result from medically determinable impairments when determining whether claimants can perform work. Aptitudes in the DOT are defined as specific abilities required of an individual to perform a given work activity. SSA does not use aptitudes and temperaments in their program because these represent personal interests, natural abilities, and personality characteristics of job incumbents rather than limitations or restrictions resulting from medically determinable impairments.

• SSA entered into an interagency agreement with BLS because BLS was already collecting occupational information for the National Compensation Survey that is similar to the type of information needed for disability adjudication. BLS field economists are experienced with collecting not only wage data from employers but also information about the strength and skill requirements of occupations. In FY 2013, BLS performed testing to see if it would be feasible to use the NCS platform to collect occupational information for SSA disability adjudication purposes. Feasibility testing showed that in most cases, traditional NCS respondents, such as human resources representatives, were capable of answering ORS survey questions. In FY 2015, BLS will perform validation studies to determine whether this data is reliable.

• BLS field economists are highly-skilled and trained in collecting occupational data from employers. Data collected by BLS for the NCS is used to determine federal pay scales. Also, they are expert at gaining the cooperation of employers to voluntarily participate in a government survey.

Forms

Comment: [SkillTRAN]

“Getting Low” is a terrible heading – use “Posture” or something else.

Reply 22:

We will revise the heading.

Comment: [IARP, NASPPR]

Recommend use of single quote tool to capture information.

Reply 23:

The only collection forms submitted for use are single quote tools. Field staff found note-taking during an interview can be more efficient in certain situations when captured in a “multi-quote” layout and created this layout for use in testing. The layout did not prevent the interview from being conducted on a quote by quote basis as appropriate. Field staff may

Comment: [NASPPR]

Under the Lifting/Carrying section, the form uses the word 'Seldom' with two different definitions

Reply 24:

BLS is unable to locate this issue on the two forms published with the 30day FRN. This was changed after the 60 day notice.

Elements/concepts

Comment: [SkillTRAN, IARP, NASPPR]

These comments relate to definitions of elements.

Reply 25:

This information is directly collected via interview. There are definitions and additional procedural instructions provided to the field economists to ensure consistency of data collection. These materials are not intended for respondent use and are not part of the clearance package.

The definitions are designed to meet SSA’s disability program policy needs. Many, such as the definitions for gross and fine manipulation, originated from the Revised Handbook for Analyzing Jobs.

Comment: [SkillTRAN]

Driving omits collection of Distance, Frequency, Duration, and Terrain

Reply 26:

Driving is collected as a screening question in the Pre-Production test but not captured as a unique physical demand. The presence of driving indicates related physical requirements and environmental conditions. Any driving duration is included in the relevant elements such as foot/leg controls. Details may be captured in pre-production schedule documentation if those details relate to overall environmental conditions or other demands.

The required level of detail collected for any element must weigh the benefit against respondent burden and the level of detail captured must reflect the costs of developing systems and data entry. For disability adjudication purposes, SSA only needs to know whether driving is required. SSA does not need the duration of driving separately. If a claimant is unable to drive due to a seizure disorder or vision impairment, they would not be able to drive for any length of time.

Comment: [SkillTRAN]

“Other Hazards:” should be captured so that they could be categorized later if encountered often enough

Reply 27:

If a respondent mentions other hazardous conditions, these would be noted by a field economist, captured in schedule remarks, and raised through collection debriefings. We cannot gather all of the physical/mental requirements and environmental exposures for occupations. Asking an open-ended “other” question as a formal part of the interview would increase respondent burden with no clear research goal.

Development of additional categories would be an ongoing survey development effort and is outside the scope of the pre-production test. We must balance the need for occupational data with data quality, respondent burden, and the needs of SSA. Some adjudicative judgment will be necessary in certain situations as it is now in the SSA program.

Comment: [SkillTRAN, IARP, NASPPR]

Standing and walking. Standing and walking are distinct functions, and although relational... recommend asking questions specific to each factor.

Reply 28:

Collecting these as separate durations was not requested by SSA and has not been tested by BLS.

SSA policy does not separate standing and walking into two distinct functions. SSA uses the combined standing and walking requirements to determine the “exertional” requirements of work. For example, work is sedentary when standing and walking are only required occasionally. Additionally SSA’s definition of medium work indicates that standing and walking are required for a total of approximately 6 hour in an 8 hour day. Therefore, since standing and walking together are used to determine the exertional level of work, it is not necessary to separate standing and walking into two distinct functions. BLS believes additionally that the time-split between the two activities in certain occupations is determined by the incumbent as well as the specific tasks.

Comment: [Workability]

Standing/Walking: Capturing only the total time per shift is not an adequate measure to evaluation the impact of disabilities that affect mobility. It is therefore recommended that one new factor “Ambulation Agility” be added under this section that can be related more directly to standardized tests for balance and agility that rate the ability more like an aptitude.

Reply 29:

Capturing the total time per shift spent standing/walking is appropriate for SSA's disability adjudication purposes. This data element is an improvement over what SSA has been using for decades based on the DOT's aggregated exertional information. As previously mentioned, SSA does not consider a person's aptitude to perform certain types of occupations. SSA only considers physical and mental limitations that result from medially determinable impairments.

Comment: [Workability]

Sitting vs. Standing/Walking at will: Yes/No does not seem to be an adequate scale to capture this demand factor. It would make more sense to capture total duration in hours per shift that the worker has flexibility to alternate between sit and standing (same scaling as for sitting or standing).

Reply 30:

It makes sense for disability adjudication to collect the sit/stand at will element in this way. If any part of the day requires someone to assume a certain posture, then it does not allow the worker to sit stand at will. We do capture time spent sitting and standing/walking for every occupation. We only record a "yes" for alternate sit/stand for an occupation that allows a worker an “at will” choice throughout the entire day to sit or stand.

Comment: [Workability]

Manipulation: The level of skill required for manipulation has been excluded from this survey. This was previously captured with an aptitude rating scale for finger dexterity and manual dexterity and represents a critical dimension for job placement.

Reply 31:

Our definitions for gross and fine manipulation originate from the Revised Handbook for Analyzing Jobs. The DOT used these definitions, and SSA adopted them into its disability program policy. Under SSA policy, adjudicators only consider mental and physical limitations that result from medically determinable impairments when determining whether claimants can perform work. Aptitudes in the DOT are defined as specific abilities required of an individual to perform a given work activity. SSA does not use aptitudes and temperaments in the disability program because they represent personal interests, natural abilities, and personality characteristics of job incumbents rather than limitations or restrictions resulting from medically determinable impairments.

Comment: [IARP, NASPPR]

It would be helpful to have clarification on the “proximity to moving mechanical parts” factor—i.e.: what type of machinery; guarded assembly machinery, fork trucks, large construction equipment etc. so that the Vocational Expert can comment on the risk based on the disability.

Reply 32:

Details such as type of machinery may be captured in pre-production schedule documentation when necessary to explain overall environmental conditions or other demands. Documentation fields are not designed to be tabulated. SSA has indicated that the tools and technology information from the O*Net is expected to be incorporated into their OIS system and data collection would duplicate this information.

Comment: [Workability]

Hours/Shift: I believe that it would be really important for the survey to ask about the usual (median), minimum and maximum scheduled shift lengths in total hours for workers in the job.

Hours/Week: To better understand how much time worker spend in a given occupation, it would also be helpful to know the usual (median), minimum and maximum number of hours worked per week by job incumbents in the occupation.

Reply 33:

The standard work schedule – hours per day, hours per week, weeks per year – is already being captured for each occupation. ORS estimates are designed to support current SSA policy, which defines a workday as 8 hours. There is a separate question related to how often there are deviations from an occupation’s standard work schedule. Based on NCS collection of hours paid associated with wages, and overtime, BLS believes the schedule minimum and maximum detail would be difficult to collect on a meaningful basis given the various reasons for fluctuations.

Comment: [Workability]

Scaling Method for two elements: The scale used for frequency is not adequate for “4. How often does the occupation verbally interact (work related) with regular contacts?” and “6. How often does the
occupation verbally interact (work-related) with people other than regular contacts?” This frequency scale is not useful to capture the level of exposure of these dimensions during the work shift. It would make more sense to capture the duration in hours per shift, similar to the approach used for physical demands and environmental conditions. In some respects, these questions seem redundant with a later function listed under physical demands “Communicating verbally”.

Reply 34:

As noted, the physical demand “Communicating verbally” captures the typical duration during a shift. The frequency of contact questions provide insights into who workers have to interact with and broadly how much. This information is particularly important for SSA when considering whether claimants with mental impairments can perform a job. Furthermore, ORS is capturing whether communication on a job is face to face under speech. These elements are useful when evaluating claimants with hearing impairments. The redundancy of the questions is considered carefully for respondent burden reasons. Answers to these questions are expected to meet different survey needs defined by SSA.

Comment: [Workability]

Physical Demands: Operational Definitions: It is recommended that BLS work with the professional associations to establish and communicate operational definitions for all the factors represented.

Reply 35:

The definitions used are established by SSA to support their disability adjudication needs, and reflect earlier input on the OIS project from multiple stakeholders. Since undertaking the ORS work, BLS has been in contact with multiple professional associations representing a broad range of stakeholders for the separate goal of developing BLS professional knowledge of this specific subject area. The BLS focus has been determining the feasibility of collecting data to meet the needs as defined by SSA and the limits of voluntary establishment response.

The pre-production test is a final step in testing. As we transition beyond this test to a new phase of survey development, BLS will work with SSA to determine the best approaches to communicate with additional stakeholders about the primary goal of the data collection effort and expected outputs. Presentations to professional associations by BLS and SSA are planned in FY2015.

Comment: [Workability]

Lifting/Carrying: I like the general approach to asking about categories of exposure based on definitions of frequency; however, putting a time range such as 1/3 to 2/3 of the time creates confusion when shift duration is so widely variable. A better approach would be to operationally define the categories based on repetition and time duration, similar to what is done for the ACGIH Threshold Limit Value for Lifting.

Reply 36:

BLS has looked at the American Conference of Industrial Hygienists (ACGIH) information mentioned. The current Lift/Carry definitions meet the goals of the pre-production test and the program needs of SSA. Information is collected about shifts that are more or less than 8 hours, but the days are then scaled to an 8 hour day because that is how it is defined in SSA’s disability policy.

BLS will discuss the repetition suggestion further with SSA and consider improvements to the definitions for future collection efforts. Any future changes must meet SSA’s disability adjudication needs and not increase respondent burden.

Comments: [A-T Voc Services]

Employment issues include the number of hours scheduled and also the work pattern (part-time on-call, part-time scheduled, split shift, temp to hire, contract, and permanent), and hiring methods such as direct hire or use of an employment service.

Reply 37:

The number of scheduled hours and the full/part time status of the occupation will be collected. Information about the scheduling pattern and hiring methods are outside the current scope of the survey. Prior research on these concepts indicates they often vary by economic functions unrelated to the occupation itself. SSA disability adjudication considers whether claimants have the functional ability to perform work. Consideration is not given to employment trends or whether claimants will be hired in occupations.

Comment: [SkillTRAN, IARP, NASPPR]

Bending and Twisting are critical aspects of Posture that are difficult for persons with back, hip, knee, and foot injuries. Balancing is missing from this list and is relevant for some occupations. Missing from the Climbing area is the Terrain on which the climbing occurs.

Reply 38:

The ORS survey is capturing very specific information about bending. ORS is capturing information about “stooping” which SSA defines as “bending the body downward and forward by bending the spine at the waist", as well as “Crouching” which means, “bending the body downward and forward by bending the legs and spine.” “Stooping” and “crouching” are more precise definitions of bending. Twisting is not a common limitation that SSA has observed in disability claims. When there is a specific twisting limitation, it will be noted on the claimant’s assessment of functioning. In these cases, the adjudicator must use his or her judgment to determine if the twisting limitation prevents the claimant from doing past work and other work. Balancing is defined in the DOT as “maintaining body equilibrium to prevent falling when walking, standing, crouching…or performing gymnastic feats.” SSA evaluates a claimant’s ability to balance when they evaluate a claimant’s ability to stand, walk and crouch. In addition, the SSA program experience indicates that balance alone is almost never a factor in determining disability and adjudicators will need to use judgment in cases where balancing is an issue. Therefore, SSA does not need to capture specific information about the ability to balance. ORS will capture specific information about climbing on two distinct types of terrains: 1. Ramps and stairs, and 2. ladders, ropes, or scaffolds.

Comment: [SkillTRAN]

Hearing – First example is doubled up. There is no example or instance in which hearing must happen when there is background noise or sounds present that could easily impede full comprehension of spoken words.

Reply 39:

We are capturing information about the noise levels in the worker's environment as a separate data element. We are also capturing under speech whether a communication on a job is face to face or not. These elements will be useful when evaluating people with hearing impairments.

Comment: [SkillTRAN, IARP, NASPPR, Workability]

Vision omits capture of any requirement for depth perception or color discrimination … important in many occupations. Yes/No scaling for Hearing and Vision should be modified to reflect an aptitude style of skill.

Reply 40:

ORS cannot capture all the physical requirements for an occupation. The survey must balance the SSA disability program need for the occupational data with the cost of collection and respondent burden. We have to measure the elements that are most critical to the disability adjudication, impact the largest number of cases, and where other data sources are insufficient for the program needs. For physical requirements that are not included in our OIS, adjudicators will need to use judgment and rely on claimants' descriptions of job requirements.

Comment: [Workability]

ENVIRONMENTAL CONDITIONS: It makes no sense that the ORS survey contains a number of non-specific environmental factors that cannot be related directly to a person’s abilities and limitations. For example, I don’t believe that information about general exposure to toxic/caustic chemicals is helpful for determining whether claimants with respiratory impairments or skin conditions can perform a job. This entire section should be reviewed with input from occupational health professionals and agencies to eliminate and add factors that are more relevant to worker-job match decisions. For example, it would be better to ask of respiratory protection or skin protection is recommended for workers who perform the occupation.

Reply 41:

The environmental elements that we are collecting are the ones that have proven most necessary to experienced adjudicators. In our efforts to record the requirements of occupations, we are not mapping them to the natural abilities of people. Our goal is to measure the physical and mental requirements of occupations that map to limitations commonly seen in disability adjudication that are caused by physical and mental impairments. In cases of people with skin disorders, respiratory disorders, and certain types of allergies, understanding the risk of exposure to fumes and toxic, caustic chemicals and whether safety equipment mitigates this exposure is critical.

Comment: [A-T Voc Ser]

Production expectations are the eminent issue in job performance. Tenure in a work setting is based on production not physical capacity. For example, the capacity for Frequent Fingering is a relevant physical characteristic, but the ability to keyboard at the rate of 5,000 keystrokes per hour with zero errors, is the relevant vocational characteristic. Use of the previously developed, critical aptitude profiles, may be helpful in assessing the interaction between these variables. Relevant questions about a job is, “what is the hardest part of this job?” and “what injury patterns occur from this work?”

Reply 42:

SSA has tailored information collected by ORS, and is tailoring their OIS, for use in the SSA disability programs. Under SSA policy, adjudicators only consider mental and physical limitations that result from medically determinable impairments when determining whether claimants can perform work. Aptitudes in the DOT are defined as specific abilities required of an individual to perform a given work activity. SSA does not use aptitudes and temperaments in the disability program because they represent personal interests, natural abilities, and personality characteristics of job incumbents rather than limitations or restrictions resulting from medically determinable impairments.

Comment: [A-T Voc Ser]

In addition, knowing how employers respond to and tolerate production deficits is critical to understanding the capacity to sustain work. Related issues are off task behavior, extra breaks, and the expectations and tolerances for the use of sick time.

Reply 43:

SSA policy indicates that an individual must be able to maintain concentration and attention for extended periods (with customary breaks), and perform activities within a schedule, maintain regular attendance, and be punctual within customary tolerances in order to perform the duties of all occupations. Therefore, it is not necessary to collect information about small variances that may exist within industries concerning production tolerances.

Comment: [A-T Voc Ser]

Work station design, production style (assembly line, batch processing, proximity to coworkers, etc.) and “at will” behaviors, such as stand up, sit down, walk around, are issues also relevant to the intended use of this data.

Reply 44:

ORS is capturing information about the incumbent’s ability to sit or stand at will, and information about work controls which SSA defines as, “the level of supervision and the requirements to adhere to established guidelines,” and personal interaction which measures, “with whom the employee interacts with as well as the frequency and types of interaction.”

Cognitive and related factors

Comment: [SkillTRAN]

There is no collection of Temperament factors, which are work situations to which the worker must adapt in order to perform the occupation satisfactorily.

There is no estimation of Aptitude values being attempted here.

Reply 45:

Under SSA policy, SSA staff can only consider mental and physical limitations that result from medically determinable impairments when determining whether claimants can perform work. Temperaments, like aptitudes, represent personal interests, natural abilities, and personality characteristics of job incumbents rather than limitations or restrictions resulting from medically determinable impairments.

Comment: [SkillTRAN]

There is no collection of any of the traditional Reasoning, Math, and Language factors.

Reply 46:

SSA policy directs that adjudicators consider whether claimants can perform work that is unskilled, semi-skilled, or skilled. The definitions of these skill levels are based on the SVPs assigned to occupations in the DOT. SSA will use a similar system in the new OIS. We do not have the resources to collect detailed information about claimants' proficiency in math, English, or reasoning so we will not collect this information about occupations.

Comment: [IARP, NASPPR]

(1) “Time to proficiency,” … It would be important to understand not only must the worker read and write but at what level—i.e.: recognize numbers/letters, write short sentences, read narrative instructions, understand safety signs etc. The issue of education required vs. preferred would also be important to be clear about.

Reply 47:

SSA policy directs that SSA consider whether claimants can perform work that is unskilled, semi-skilled, or skilled. SSA based our definitions of these skill levels on the SVPs assigned to occupations in the DOT. SSA will use a similar system in the new OIS and ORS is designed create a similar skill level. SSA does not have the resources to collect detailed information about claimants' proficiency in math, English, or reasoning. ORS will collect information from employers about the level of education that is necessary to perform the occupation.

Comment: [Workability]

Task Complexity is not specific enough to adequately address the occupational impact of various learning disabilities. It is recommended that this dimension be supplemented by inclusion of at least three of the key cognitive elements describing complexity that have well-established precedent with aptitude levels described in the DOT. Minimal adaptation of the description levels would be needed to include the DOT factors for Reasoning, Math and Language

Reply 48:

BLS started testing the mental-cognitive data elements at the beginning of 2014. In FY 2015, BLS will continue to test the collection of these elements. Following pre-production testing, SSA and BLS staffs will work together to determine whether the mental-cognitive elements need to be changed. However, SSA does not consider an individual's aptitude to perform certain occupations as part of disability adjudication. The new OIS will be tailored for use in SSA’s disability programs. Under SSA policy, adjudicators only consider mental and physical limitations that result from medically determinable impairments when determining whether claimants can perform work. Aptitudes in the DOT are defined as specific abilities required of an individual to perform a given work activity. SSA does not use aptitudes and temperaments in the disability program because they represent personal interests, natural abilities, and personality characteristics of job incumbents rather than limitations or restrictions resulting from medically determinable impairments.

Comment: [IARP, NASPPR]

Reaching. Data should be collected for above shoulder, waist to chest height and below waist. For example, an assembler may be able to reach at bench height but not below the waist. These are very important distinctions in particular with the high number of claims involving back injuries.

Reply 49:

ORS captures data concerning overhead reaching and reaching at or below the shoulder. We initially tested collecting reaching at a more detailed level, but found that level of detail increased the burden on respondents. SSA believes this data will be sufficient to adjudicate most claims. When a claimant has reaching limitations that do not fall specifically within these domains, the actual limitations will be recorded on the claimant’s residual functional capacity assessment (RFC) and the adjudicator will exercise adjudicative judgment when evaluating the claim at steps 4 and 5 of sequential evaluation.

Comment: [SkillTRAN]

Cognitive elements identified in the survey reflect terminology used internally by NCS far
more than they reflect questions that are posed by the ALJ to a vocational expert in the
ODAR hearing.

Reply 50:

When developing the mental-cognitive questions, SSA reviewed the results of a study they performed that gathered information from 5,000 disability claims decided at the initial and hearings level at steps 4 and 5 of sequential evaluation. SSA recorded the mental limitations most commonly cited by SSA psychiatric and psychological consultants in these cases and tried to address these with our mental-cognitive questions. SSA also reviewed a list of mental-cognitive data elements compiled from recommendations made by a panel of experts, stakeholders, O*NET, and disability policy. Disability policy experts condensed the list of over 300 elements, and then reworded them to focus on the requirements of lower-skilled jobs.

There were many similarities between the list of what SSA needs and the information BLS collects with leveling questions. SSA leveraged the expertise of the BLS regarding the NCS leveling procedures when it was necessary to make refinements and adjustments to mental-cognitive questions based on the results of testing. Although the mental-cognitive questions may use leveling terminology and concepts, SSA made sure that ORS is capturing the concepts most important to the disability program.

Comment: [SkillTRAN]

Missing from the cognitive mix is the ability of the person to reason.

Reply 51:

The mental-cognitive questions are designed to specifically capture information about low-skilled occupations. We focused on the information that is most important to disability adjudication. We will not be able to address all of the mental-cognitive requirements of occupations so some adjudicative judgment will be required. SSA policy does not allow consideration of a person's natural abilities when assessing his or her ability to work. SSA staff must only consider limitations resulting from medically determinable impairments (MDI).

Comment: [IARP, NASPPR]

[T]he Cognitive Element lexicon comes from the NCS and is being superimposed onto the ORS. This lexicon has no relation to the Social Security Residual Mental Functional Capacity Questionnaire which is the foundation for much of the questioning in Social Security Hearings. The language and factors measured should be consistent with factors typically explored in SS disability hearings. Is there literature/research that supports this group of cognitive elements and can these elements be cross walked or mapped to standardized psychometric testing instruments?

Reply 52:

When developing the mental-cognitive questions, SSA reviewed the results of a study they performed that gathered information from 5,000 disability claims decided at the initial and hearings level at steps 4 and 5 of sequential evaluation. SSA recorded the mental limitations most commonly cited by SSA psychiatric and psychological consultants in these cases and tried to address these with our mental-cognitive questions. SSA also leveraged the expertise of the BLS regarding the NCS leveling procedures when it was necessary to make refinements and adjustments to mental-cognitive questions based on the results of testing. Although the mental-cognitive questions may use leveling terminology and concepts, SSA made sure that ORS is capturing the concepts most important to the disability program.

SSA policy for evaluating whether a claimant has the ability to perform the basic mental demands of unskilled work includes, but is not limited to, evaluating whether the claimant can do the following:

• Understand, carry out, and remember simple instructions, (SSA believes the Complexity domain in the ORS addresses this factor);

• Make judgments that are commensurate with the functions of unskilled work, i.e., simple work-related decisions; (SSA believes the Work Controls domain in the ORS addresses this factor);

• Respond appropriately to supervision, coworkers and work situations; (SSA believes our Personal Interaction domain in the ORS addresses this factor);

• And deal with changes in a routine work setting. (SSA believes the Work Routine domain in the ORS addresses this factor.);

These four factors were very important drivers of ORS mental/cognitive data elements. SSA policy indicates that the check blocks on the Mental Residual Functional Capacity (MRFC) form are not the actual residual functional capacity assessment. The MRFC is the narrative at bottom of the form that describes the most the claimant can do despite his or her impairment(s). This allows the adjudicative team to document limitations that a claimant may possess, which may not be included in the check blocks. Therefore, it is not necessary that the ORS lexicon match the check blocks on the MRFC.

Comment: [SkillTRAN]

The definition of complexity (Item 1) has improved from the initial draft, but again, multiple dimensions remain for this factor. The number of tasks is one dimension and the amount of judgment required is another. The ability to “analyze” is also another dimension. It is better to break this into 3 questions than one.

Reply 53:

ORS is measuring the complexity of work for an occupation.  The tasks, the judgment required, and the analysis needed for completing an occupation’s work, are seen as interrelated. The mental/cognitive questions are still being tested during pre-production. BLS and SSA will discuss testing results and determine whether we are collecting the type of information most needed to adjudicate claims involving mental impairments or whether further refinement of questions is needed.

Comment: [SkillTRAN]

Item 2 --“Closely controlled” – remains another multiple dimension item. Number or level of instructions is intermixed with autonomy, supervision, ingenuity, and goal orientation. Break these out separately IF these are relevant to the questions that SSA asks about during the ODAR hearings

Reply 54:

We started testing ORS mental-cognitive questions in FY 2014 and will continue to test them in FY 2015. Once FY 2015 pre-production testing is complete, SSA will consider comments made about the questions and issues that arose during interviews with survey respondents to determine if revisions to the questions are needed.

Comment: [SkillTRAN]

Item 3 is vastly improved. Missing from this area is capture of “Time off task”.

Reply 55:

SSA worded this question to capture the information needed to adjudicate claims for individuals with mental impairments. The element "time off task" seems to be more of a "person-side" element rather than something that describes the requirement of an occupation.

Comment: [SkillTRAN]

Perhaps the better way to describe these dimensions is “communication” rather than “verbal interaction”.

Reply 56:

The other types of communication outlined are not in scope of the element. Verbal interaction correctly defines the scope.

Page 19