BLS Responses for OMB Reviewer to the 30 day Federal Register public comments for the Occupational Requirements Survey
The BLS received three public comments on the Federal Register 30 day notice published in the Federal Register, Vol. 80, No. 119, published on June 22, 2015 (OMB ICR Reference Number: 201502–1220–006.)
Mr. Tager and Mr. Truthan both commented that the SOC and O*Net do not have the number of unique occupations that the DOT had. Quoting Mr. Truthan:
“The venerable but aged DOT has 12,761 unique occupations. These are being aggregated down to 974 O*NET occupations only. Some of these groups contain hundreds of DOT occupations. The variability of values obtained for critical factors such render them functionally useless.
A glimmer of hope exists if enough data can be gathered particularly for the most troublesome SOC groups (those with the highest variability of SVP/Strength) by sufficiently sampling enough of the NAICS groups in which the occupations occur. A SOC occupation performed within an industry is likely to be more consistent in terms of physical demand and SVP. NAICS coding should go to at least a 3-digit level of coding and probably to a 4- or 5- digit NAICS coding level for greater precision. This will permit direct linkage to critical labor market data for numbers of people employed nationally in these targeted industries. This may well require an increase in the sample size to achieve results at a reliable level … but so be it.”
BLS believes that the proposed survey design is appropriate for the initial years of the survey, and is committed to evaluating the results of collection to ensure the design can support needed outputs while meeting confidentiality and reliability constraints. In Part A, Section 8, BLS stated:
ORS will classify occupations using the SOC as federal agencies gathering occupational data are mandated to do. Data for the ORS will be coded at an eight digit SOC level defined by the O*NET (O*NET-SOC). As such, we will be coding data to approximately 1,100 individual occupations, not just the 820 SOC codes. Although the ORS samples will be selected using sampling strata defined by North American Industry Classification System (NAICS) economic sector codes (first two digits), all establishments in the sample will be coded into industries using the full NAICS national industry codes (six digits). BLS plans to analyze variability within a given O*NET-SOC code using as many digits of the NAICS codes as possible with the given sample size. Should this analysis show that variability within an O*NET-SOC code can be better explained by generating estimates for an occupation at the more detailed occupation by industry level, BLS will release the data by both occupation and industry whenever sufficient data are present to meet our publication criteria. Decisions about changes to the sample will depend on detailed analysis of the responses from the first several samples as well as the need to balance the cost of the survey with respondent burden.
Also in Section 8, BLS stated:
SSA performed a folder study of 5,000 claim folders decided at steps 4 and 5 at the initial and hearings level to inform its Occupational Information System (OIS) development. SSA associated the 15 year work history of each claimant with DOT codes, and found that they were associated with a small percentage – about 10 percent – of the DOT’s titles. SSA and BLS have worked together closely to ensure that the occupational data collected meets the needs of SSA’s disability program. SSA will ensure that the final OIS contains descriptions of a sufficient number of occupations that are common to the disability applicant population. If necessary, SSA and BLS will develop a strategy to disaggregate O*NET-SOC codes for which” the data are too spread, by industry or sub-SOC groupings, and to describe occupations which are not captured through sampling. …Decisions about changes to the sample will depend on detailed analysis of the responses from the first several samples as well as the need to balance the cost of the survey with respondent burden.
Mr. Tager and Mr. Truthan both commented that the ORS survey should not be establishment based, but occupational based as ORS will not get enough occupational data. This is similar to an earlier public comment that proposed using an occupation based approach similar to the Occupational Employment Statistics survey.
In Part A, Section 8, BLS stated:
NCS uses a probability sampling method of selecting occupations within each company in proportion to the number of workers within each occupation. This method, called a Probability Selection of Occupations, has been shown to yield data for a broad range of occupations within each industry and to ensure that all occupations within an industry, even those with low employment, have a chance of being included in the data being collected. While it may be possible to meet these goals using an approach proposed by the commenter, further research needs to be done to explore and test this approach. That research is included in future testing plans but has not been started to date.
Mr. Truthan and Mr. Wickstrom both feel that direct job observations are the best way to collect ORS data. Mr. Truthan stated, “I support this proposed data collection effort, particularly with development and validation against the proposed “gold standard” of direct job observation, which has been clearly shown as the more valid approach for collection of both physical demand and environmental factors.”
BLS in Section 8 stated:
As part of ORS validity tests, field economists will directly observe previously selected jobs and use meters to measure temperature, humidity and noise levels. The BLS Office of Survey Methods Research (OSMR) will provide guidance for observation training. BLS will continue to evaluate our procedures and survey outputs for validity and reliability as producing accurate and reliable statistics is central to our mission.
BLS is currently conducting initial validity research using the job observation method. BLS will use the results of the test to inform further research and procedure improvements. BLS believes that the proposed interview based survey design will provide necessary data to produce quality estimates without the formal inclusion of direct job observation.
Mr. Wickstrom commented on several areas where he thought additional details should be captured, such as the specific types of licenses or languages required.
While such information could be captured via this survey, these are not required data elements and would require additional resources to collect as well as potentially add burden. In Section 8, BLS states:
The survey must balance the SSA disability program need for the occupational data with the cost of collection and respondent burden. The definitions used are established by SSA to support their disability adjudication needs, and reflect earlier input on SSA’s Occupational Information System project from multiple stakeholders.
Mr. Truthan felt the last cognitive element of “work-related interactions continues to mix too many concepts into a single rating scale. It is a confusing mix of factors such as: information sharing, coordination, problem solving, persuading/softselling, interacting, influencing, asserting control, calm under fire (resolving controversy), planning, supervision, leadership, and presentation skills.”
By contrast, Mr. Wickstrom’s only comment on the cognitive elements was: “There are many changes to the survey questions in this section. In general, I really like many of the improvements.”
In Section 8, BLS stated:
Continuing into FY 2015, BLS has tested these elements and worked with SSA to improve them. The ORS currently captures information related to the frequency and type of communications required. For ORS production collection, questions on work pace were added and the question on task complexity was refined to focus on decision-making. Furthermore, the work controls question was revised to focus on the instruction and review of work provided to the occupation. … If SSA and BLS are not confident that the mental-cognitive questions are accurately and reliably measuring the type of occupational information most important to disability adjudication, they will continue to refine these elements.
ORS is currently testing cognitive elements revisions that were based on pre-production test experiences and comments made on the 60 and 30 day notices. The element that Mr. Truthan mentions is not an element that was modified. The purpose of contacts that are covered is varied, yet the scale is closely aligned with the job leveling factor distinctions in the NCS and has not presented issues in testing.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Carney, Paul - BLS |
| File Modified | 0000-00-00 |
| File Created | 2021-01-22 |