Supporting Statement Part B ORS Production_2018

Occupational Requirements Survey (ORS)

1220-0189

April 2018

2018 Supporting Statement for the Occupational Requirements Survey

Justification, Part B.

Collection of Information Employing Statistical Methods

For detailed technical materials on the sample allocation, selection, and estimation methods as well as other related statistical procedures see the BLS technical reports and American Statistical Association (ASA) and Federal Committee on Statistical Methodology (FCSM) papers listed in the references section. The following is a brief summary of the primary statistical features of the Occupational Requirements Survey.

The Occupational Requirements Survey (ORS) is an establishment survey that the Bureau of Labor Statistics (BLS) is conducting to collect information about the requirements of occupations in the U.S. economy, including the vocational preparation, the cognitive and physical requirements, and the environmental conditions in which the work is performed. The Social Security Administration (SSA), one of several users of this occupational information, is funding the survey through an Interagency Agreement (IAA). Prior planning for ORS involved several feasibility tests throughout Fiscal Years 2013 and 2014 and the first phase of a three-year production wave beginning in Fiscal Year 2015. The BLS is currently collecting ORS data for the third year of a three-year production wave using a two-stage stratified design with probability proportional to employment sampling at each stage. Under the current design, occupations with low employment in the current economy have a smaller probability of selection resulting in an insufficient number of observations to publish ORS estimates for these low employment occupations. Sections 1-3 of this document describe the new selection process of the ORS production samples, the collection process for the ORS data elements, and planned estimates to be produced. Data from the samples will be used to produce outputs, such as the "time to proficiency" of occupations, the mental-cognitive and physical demands of work, and the environmental conditions in which work is performed. Section 4 of this document describes the efforts conducted by the BLS to prepare for this new wave of production of the ORS.

In late FY 2018, ORS production will begin this new wave by selecting samples using the methodology described in this document. Current plans call for collection of the first ORS production sample under this new design to begin in September 2018 and continue for approximately twelve consecutive months.

1. Universe and Sample Size

1a. Universe

The ORS will measure constructs such as time to proficiency, mental-cognitive and physical demands, and environmental conditions and produce national-level estimates by occupation of percentages, means, percentiles, and modes of variables derived from measurements capturing information about ORS constructs.

The frame for the ORS sample under this new design will be developed from several sources:

• The Occupational Employment Statistics (OES) sample of establishments and occupations. The OES sample contains over 1 million establishments from private industry and state and local government.

• A modeled occupation frame created by the OES program. The OES uses the private industry portion of their sample to predict occupational distributions for not sampled for or non-responding to the OES private industry establishments.

• The Quarterly Contribution Reports (QCR) filed by employers subject to State Unemployment Insurance (UI) laws. The BLS receives the QCR for the Quarterly Census of Employment and Wages (QCEW) Program from the 50 States, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands. The QCEW data, which are compiled each calendar quarter, provide a comprehensive business name and address file with employment, wage, detailed geography (i.e., county), and industry information at the six-digit North American Industry Classification System (NAICS) level.

• In many states railroad establishments are not required to report to the State UI. BLS obtains railroad establishment information from State partners that work directly with staff in the office of Occupational Employment Statistics (OES).

The ORS universe will include all establishments in the 50 States and the District of Columbia with ownerships of State and Local governments and private sector industries, excluding agriculture, forestry, and fishing (NAICS Sector 11) and private households (NAICS Subsector 814). The estimate of the current universe size, based on the most recent QCEW data, is about 9,000,000 establishments.

Since the OES modeled frame only includes establishments in the private industry, separate sampling frames will be created for private industry versus government (state and local combined).

• To create the private industry frame, data from the OES modeled frame will be combined with the establishment-level data from the private-industry QCEW and railroad files to create lists of occupations at the establishment level. The modeled information will then be supplemented with the collected OES sample of establishments and occupations to create the full private industry frame of occupations at an establishment level.

• The frame for state and local government will be the government establishments from the QCEW.

All ORS production sampled establishments (approximately 10,000 per year) will be interviewed once in an attempt to capture all of the needed ORS data elements.

1b. Sample Size

Scope - The ORS production frame is as defined above. The sampling design for the five-year sample will be a two-stage stratified random sample of establishments and occupations within selected establishments.

Sample Stratification – Both the private industry and government sector samples will be stratified, however, the sample cells (i.e., strata) will be defined differently for each sector and the input frames separate.

For private industry, based on data found in the modeled OES frame file, all private industry establishments will first be identified as either having a “Rare Occupation” or not. For the purposes of sample selection, in most cases a “rare occupation” is defined as one of the 200 6-digit Standard Occupational Classifications (SOCs) with the lowest May 2017 OES employment, across all ownerships.

Strata are then formed by the cross-classification of the predicted presence/absence of a “rare occupation” in the establishment, Census Region (Northeast, Southeast, Midwest, West), and aggregate industry (Education, Goods Producing, Health Care, Financial Activities, Service Providing), leading to forty strata.

Since the OES modeled frame does not include State and local government entities, Rare Occupation Status does not apply to the government sector for the purposes of stratification. Thus, for the state and local government frame, strata are formed by the cross-classification of Census Region (Northeast, Southeast, Midwest, West), and detailed industry, also leading to forty strata.

Sample Allocation – The ORS must determine the number of units intended for each sample cell before it selects its establishment sample. The allocation process is run separately for each ownership sector (Private industry or Government).

The total ORS production sample will consist of approximately 50,000 establishments for each five-year production wave. The private portion of this sample will be approximately 85% (42,500) and State and Local government portion will be approximately 15% of the total sample (7,500). In order to accommodate the goal of ORS, to produce estimates of occupational requirements for as many Standard Occupational Classification (SOC) codes as possible, a higher proportion of the total private industry sample size will be allocated to the twenty “rare-occupation” strata than to the twenty “non-rare occupation” strata. Establishment allocation to the cells within the “rare/non-rare” strata is proportional to total employment within the cell. Establishment allocation to the sample cells for the state and local government sector will be proportional to the total employment within the cell.

Collection of the first ORS production sample under this new five-year design will span a consecutive twelve month period, beginning in the September of 2018.

Sample Selection – Sample selection will involve two stages: establishment selection and occupation selection. For private industry, both stages will be completed before the sample is fielded, with the exception of establishments that are also in the National Compensation Survey (NCS). However, for private industry sample units that are also sampled, by chance, for the NCS and for all sampled state and local government establishments, occupational selection will be done after establishment contact.

At the first stage of sample selection, all establishments are selected with probability proportional to employment size of the establishment.

For each private industry establishment that is not in the NCS, an occupational quote allocation is assigned based on establishment size, noting that there is one quote per SOC and the quote allocation can only be as large as the total number of distinct SOCs.

BLS acknowledges that some of the allocated quotes will not exist in the sampled establishment. This is because the occupational distribution information for each establishment is a prediction, or a best guess of the occupations that exist in the establishment. Due to imperfections in SOC quote information, BLS will sample twice as many occupations as needed for each of the establishment size classes in the following manner:

• Up to 4 employees: Total number of distinct SOCs

• 5 – 49 employees: Up to 8 SOCs/quotes

• 50 – 249 employees: Up to 12 SOCs/quotes

• 250+ employees: Up to 16 SOCs/quotes

Within each selected establishment, the allocated occupations will be selected based on the predicted occupational distribution in the following manner.

• In the twenty “non-rare” strata, BLS will select all SOCs/quotes using a systematic sampling strategy.

• In the twenty “rare” strata, if the selected establishment has only “rare” SOCs, BLS will select SOCs/quotes with certainty or using a systematic sampling strategy in accordance with the quote allocation.

• In the twenty “rare” strata, if the selected establishment has a mix of “rare” and “non-rare” SOCs, BLS will select no more than one less the quote allocation from the “rare” SOCs either with certainty or using a systematic sampling strategy, depending on the total number of “rare” SOCs. BLS will select the remaining quotes from the “non-rare” SOCs using a systematic sampling strategy.

The selected occupations will be ordered for each establishment. The field economist follows the order until the total number of quotes needed for the establishment size is identified and collected.

• Up to 4 employees: Total number of distinct SOCs

• 5 – 49 employees: Up to 4 SOCs/quotes

• 50 – 249 employees: Up to 6 SOCs/quotes

• 250+ employees: Up to 8 SOCs/quotes

For the private portion of the sample that overlaps with the NCS and the government portion of the sample, jobs will be selected in each sampled establishment during the collection phase. The probability of a job being selected within this segment of the ORS sample will be proportionate to its employment within the establishment. The number of jobs selected in an establishment will range as follows:

• Up to 4 employees: Total number of distinct SOCs

• 5 – 49 employees: 4 SOCs/quotes

• 50 – 249 employees: 6 SOCs/quotes

• 250+ employees: 8 SOCs/quotes

A team within the BLS is currently identifying, recommending solutions to, and providing procedural guidance on the field collection issues that arise from having a pre-selected set of SOCs/quotes to collect ORS data elements from sampled establishments. This team is conducting a field test to refine the collection procedures and tools needed for the new sample design. This team is scheduled to deliver all outputs prior to the start of production.

Sample weights will be assigned to each of the selected establishments and jobs in the sample to represent the entire frame. Units selected as certainty will be self-representing and will carry a sample weight of one. The sample weight for the non-certainty units will be the inverse of the probability of selection.

2. Sample Design

2a. Sample Rotation

The new design plan for the ORS will use a five-year rotation with complete estimates published after a full five-year sample has been fielded and collected. Limited interim results will be produced on an annual basis for estimates that meet all BLS confidentiality and SSA interagency agreement guidelines. The full five-year sample will be split evenly and collected over a five-year period with approximately one-fifth collected each year. The data for establishments in each sample year will be collected once and will not be collected again for the ORS until a new production wave is fielded.

2b. Estimation Procedure

The ORS production plan is to produce estimates as described in the formulas below. Computation of these estimates will include weighting the data at both the unit (establishment and occupation/job) and item (individual ORS data element) level. The final weights will include the initial sample weights, adjustments to the initial sample weights, two types of adjustments for non-response, and benchmarking. The initial sample weight for a job in a particular establishment will be a product of the inverse of the probability of selecting a particular establishment within its stratum and the inverse of the probability of selecting a particular job within the selected establishment. Adjustments to the initial weights will be done when data are collected for more or less than the sampled establishment. This may be due to establishment mergers, splits, the inability of respondents to provide the requested data for the sampled establishment, or inaccuracies in the predicted occupational distribution information for the sampled establishment which results in Probability Sampling of Occupations (PSO) being used. The two types of adjustments for non-response will include an adjustment for establishment refusal to participate in the survey and an adjustment for respondent refusal to provide data for a particular job.

Benchmarking, or post-stratification, is the process of adjusting the weight of each establishment in the survey to match the distribution of employment by detailed industry at the reference period. Because the sample of establishments is selected from a frame that is approximately two years old by the time the data is used in estimation and sample weights reflect employment when selected, the benchmark process will update that weight based on current employment.

ORS will calculate percentages, means, percentiles, and modes for ORS data elements for the nation as a whole by occupation, defined by SOC. ORS will use an 8-digit SOC code designed to coordinate with O*NET, resulting in the potential of data for 1,110 SOC codes. Before estimates of characteristics are released, they will first be screened to ensure that they do not violate the BLS confidentiality pledge. A promise is made to each private industry respondent and those government sector respondents who request confidentiality that BLS will not release its reported data to the public in a manner that would allow others to identify the establishment, firm, or enterprise.

Calculate Estimates

ORS estimates will be defined in two dimensions. A set of conditions describes the domains and a separate set of conditions describes the characteristics. Domain conditions may include specific occupations, occupational groups, worker characteristics, and geographic region. Characteristic conditions depend on the ORS data elements, such as previous experience or the required number of hours an employee must stand in a typical day. Each characteristic is calculated for each domain. If a quote meets the domain condition for a particular estimate, the X_ig value in the formulas below is 1; otherwise, it is 0. Likewise, if a quote meets the characteristic condition for a particular estimate, the Z_ig value in the formulas below is 1; otherwise, it is 0. The final quote weight ensures that each quote used in estimation represents the appropriate number of employees from the sampling frame.

Estimates that use the mean or percentile formulas require an additional quantity for estimation, Q_ig, the value of the variable corresponding to this quantity. For more information, see “Estimation Considerations for the Occupational Requirements Survey” by Rhein (see Attachment 1).

Estimation Formulas (All estimates use quote-level records, where quote represents the selected workers within a sampled establishment job.)

1. Percent of employees with characteristic: Percent of employees with a given characteristic out of all employees in the domain. These percentages would be calculated for categorical elements (e.g., type of degree required) and for element durations within SSA categories (e.g., Seldom, Frequently).

Estimation Formula Notation

i = Establishment

g = Occupation within establishment i

I = Total number of establishments

G_i = Total number of quotes selected in establishment i

X_ig = 1 if quote ig meets the condition set in the domain (denominator) condition

= 0 otherwise

Z_ig = 1 if quote ig meets the condition set in the characteristic condition

= 0 otherwise

OccFW_ig = Final quote weight for occupation g in establishment i

To calculate the percent of employees with a given characteristic out of all employees in the domain, add the final quote weights across only those quotes that meet the domain (denominator) condition and characteristic condition. Then divide that number by the sum of the final quote weights across quotes that meet the domain (denominator) condition. Multiply the final quotient by 100 to yield a percentage.

2. Mean: Average value of a quantity for a characteristic. These estimates would be calculated for element durations and other numeric elements.

Estimation Formula Notation

i = Establishment

g = Occupation within establishment i

I = Total number of establishments in the survey

G_i = Total number of quotes in establishment i

X_ig = 1 if quote ig meets the condition set in the domain condition

= 0 otherwise

Z_ig = 1 if quote ig meets the condition set in the characteristic condition

= 0 otherwise

OccFW_ig = Final quote weight for occupation g in establishment i

Q_ig = Value of a quantity for a quote g in establishment i

To calculate the average value of a quantity for a characteristic, multiply the final quote weight and the value of the quantity for those quotes that meet the domain (denominator) condition and characteristic condition; add these values across all contributing quotes to create the numerator. Divide this number by the sum of the final quote weights across only those quotes that meet the domain (denominator) condition and characteristic condition.

3. Percentiles: Value of a quantity at given percentile. These estimates would be calculated for element durations and other numeric elements.

The p-th percentile is the value Q_ig such that

• the sum of final quote weights (OccFW_ig) across quotes with a value less than Q_ig is less than p percent of all final quote weights, and

• the sum of final quote weights (OccFW_ig) across quotes with a value more than Q_ig is less than (100-p) percent of all final quote weights.

It is possible that there are no specific quotes ig for which both of these properties hold. This occurs when there exists a quote for which the OccFW_ig of records whose value is less than Q_ig equals p percent of all final quote weights. In this situation, the p-th percentile is the average of Q_ig and the value on the record with the next lowest value. The Q_ig values must be sorted in ascending order.

Include only quotes that meet the domain condition and the characteristic condition – i.e., where:

.

Estimation Formula Notation

i = Establishment

g = Occupation within establishment i

X_ig = 1 if quote ig meets the condition set in the domain condition

= 0 otherwise

Z_ig = 1 if quote ig meets the condition set in the characteristic condition

= 0 otherwise

OccFW_ig = Final quote weight for occupation g in establishment i

Q_ig = Value of a quantity for a specific characteristic for occupation g in establishment i

4. Modes: The category with the largest weighted employment from among all possible categories of a characteristic. These estimates will be calculated for all categorical elements (e.g., type of degree required) among the appropriate categories (e.g., bachelor’s degree, master’s degree).

2c. Reliability

Measuring the Quality of the Estimates

The two basic sources of error in the survey estimates are bias and variance. Bias is the amount by which estimates systematically do not reflect the characteristics of the entire population. Many of the components of bias can be categorized as either response or non-response bias.

Response bias occurs when respondents’ answers systematically differ in the same direction from the correct values. For example, this occurs when respondents incorrectly indicate “no” to a certain ORS element’s presence when that ORS element actually existed. Another example may occur when, in providing the requested ORS data elements, the respondent focuses only how the selected employee performs the duties in his position, rather than what is required by the position. Response bias can be measured by using a re-interview survey. Properly designed and implemented, this can also indicate where improvements are needed and how to make these improvements. For production, the ORS data will be reviewed for adherence to ORS collection procedures using a multi-stage review strategy. Approximately five percent of the sampled establishments will be re-contacted to confirm the accuracy of coding for selected data elements. The remaining ORS units will either be reviewed in total or for selected data elements by an independent reviewer in the Regional or National Offices. All schedules in the sample will be eligible for one and only one type of non-statistical review, in other words a responding establishment may or may not be re-contacted at most once for an additional review. Additionally, all schedules will be reviewed for statistical validity to ensure the accuracy of the sample weight with the data that was collected.

Non-response bias is the amount by which estimates obtained do not properly reflect the characteristics of non-respondents. This bias occurs when non-responding establishments have ORS element data that are different from those of responding establishments. Non-response bias is being addressed by efforts to reduce the amount of non-response. Another BLS establishment based program, the National Compensation Survey (NCS), has analyzed the extent of non-response bias using administrative data from the survey frame. The results from this analysis are documented in the 2006 ASA Proceedings of Survey Research Methods Section (See Attachment 2). A follow-up study from 2008 is also listed in the references (See Attachment 3). Details regarding adjustment for non-response are provided in Section 3 below. These studies provide knowledge that can be incorporated into ORS. See Section 3c for more information about non-response studies.

Another source of error in the estimates is sampling variance. Sampling variance is a measure of the variation among estimates from different samples using the same sample design. Sampling variance for the ORS data will be calculated using a technique called balanced half-sample replication. For national estimates, this is done by forming different re-groupings of half of the sample units. For each half-sample, a "replicate" estimate is computed with the same formula as the regular or "full-sample" estimate, except that the final weights are adjusted. If a unit is in the half-sample, its weight is multiplied by (2-k); if not, its weight is multiplied by k. For all ORS estimates, k = 0.5, so the multipliers will be 1.5 and 0.5. Sampling variance computed using this approach is the sum of the squared difference between each replicate estimate and the full sample estimate averaged over the number of replicates and adjusted by the factor of 1/(1-k)² to account for the adjustment to the final weights. This approach is similar to that used in the NCS. For more details, see the NCS Chapter of the BLS Handbook of Methods (See Attachment 4).

For ORS production, the goal is to generate estimates for as many 8-digit SOCs as possible, given the sample size and BLS requirement to protect respondent confidentiality and produce accurate estimates. Additional estimates for aggregate SOC codes will be generated if they are supported by the data. Estimates of means should be accurate with a relative standard error less than 33% on average and the percent estimates are expected to be within 5 percent of the true (population) percent at the 90 percent confidence level.

2d. Data Collection Cycles

ORS production data collection under this new design will begin collection in September 2018 after the conclusion of the first three-year wave of production and upon receipt of OMB approval. Collection will span 60 months (12 months for each one-fifth portion of total sample assigned each year) with complete estimates produced at the conclusion of the total five-year design. Limited interim results will be produced on an annual basis for estimates that meet all BLS confidentiality and SSA interagency agreement guidelines. The BLS will conduct ORS as a national survey composed of no more than 50,000 establishments. Approximately 15 percent of these establishments will be selected from State and Local government and the remainder of the sample will be selected from private industry.

3. Non-Response

There are two types of non-response for ORS: total establishment non-response and partial non-response with the latter occurring at the occupation or data element level. The assumption for all non-response adjustments is that non-respondents are similar to respondents.

To adjust for establishment or occupation non-response, weights of responding units or occupations that are deemed similar will be adjusted appropriately. Establishments will be considered similar if they are in the same “rare/non-rare” strata, ownership, and 2-digit NAICS. If there are not sufficient data at this level, then a broader level of aggregation will be considered. For partial non-response at the ORS element level, ORS will compute estimates that include a replacement value imputed based on information provided by establishments with similar characteristics.

For ORS, the un-weighted and weighted establishment response rates for the second production sample that ended collection in July 2017 were the same at 77%. At the occupation level, the un-weighted response rate for the same sample group was 89% and weighted response rate was 90%.

3a. Maximize Response Rates

To maximize the response rate for this survey, field economists will initially refine addresses ensuring contact with the appropriate employer. Then, employers will be mailed or emailed a letter explaining the importance of the survey and the need for voluntary cooperation. The letter will also include the Bureau’s pledge of confidentiality. A field economist will call the establishment after the package is sent and attempt to enroll them into the survey. Non-respondents and establishments that are reluctant to participate will be re-contacted by a field economist specially trained in refusal aversion and conversion. Additionally, respondents will be offered a variety of methods, including personal visit, telephone, fax, and email, through which they can provide data.

3b. Non-Response Adjustment

As with other surveys, ORS experiences a certain level of non-response. To adjust for the non-respondents, ORS will divide the non-response into two groups, 1) unit non-respondents and 2) item non-respondents. Unit non-respondents are the establishments (or occupations) for which no ORS data was collected, whereas item non-respondents are the establishments that report only a portion of the requested ORS data elements for the selected occupations.

The unit (establishment or occupation) non-response will be treated using a Non-Response Adjustment Factor (NRAF). Within each sampling cell, NRAFs will be calculated based on the weighted ratio of the number of viable, i.e., in-scope and sampled, establishments to the number of usable, i.e., provided any data, respondents in the sample cell. Item non-response will be adjusted using item imputation.

3c. Non-Response Bias Research

Extensive research was done to assess whether non-respondents to the NCS survey differ systematically in some important respect from respondents and would thus bias NCS estimates. Details of this study are described in the two papers by Ponikowski, McNulty, and Crockett referenced in Section 2c (See Attachments 2 and 3). These studies provided knowledge that can be incorporated into future ORS non-response bias research.

BLS also analyzed survey response rates from the Pre-production test of the ORS sample at the establishment, occupational quote, and item (i.e., individual data element) levels. The data was analyzed using un-weighted response rates and response rates weighted by the sample weight at each level of detail. Results from the Pre-production test are detailed in the paper by Yu, Ponikowski, and McNulty (see Attachment 5). In a continued effort to monitor response rates at the establishment, occupation, and item levels, the BLS will run the same non-response analysis at the conclusion of each production sample. BLS plans to present response rate results from the first production sample at the annual Federal Committee on Statistical Methodology Meetings in March 2018.

We plan to review the response rates in aggregate and by available auxiliary variables such as industry, occupation, geography, e.g., Census and BLS data collection regions, and establishment size. BLS will use the results from the analysis to identify the auxiliary variables that are most likely to contribute significantly to bias reduction. Once these variables are identified they will be used in the data processing system to reduce potential nonresponse bias.

4. Testing Procedures

Various tests have been completed both prior to the start and during the first wave of the ORS production samples. Field testing focused on developing procedures, protocols, and collection aids. These testing phases were analyzed primarily using qualitative techniques but showed that this survey was operationally feasible. Survey design testing was also conducted to ensure that we have the best possible sample design to meet the needs of the ORS. Data review processes and validation techniques were also analyzed to ensure quality data can be produced.

4a. Tests of Collection Procedures

The timeline below is an overview of past and current testing of collection procedures.

Past

• Fiscal Year 2012

The BLS signed an interagency agreement with SSA to design, develop, and conduct a series of tests using the NCS platform. The purpose was to assess the feasibility of using the NCS to accurately and reliably capture data relevant to SSA’s disability program. The resulting initiative–the ORS–was launched to capture data elements new to NCS using the NCS survey platform.

• Fiscal Year 2013

BLS completed three initial phases of ORS testing: a Proof-Of-Concept Test, a Collection Protocol Development Test, and a Broad Scale Test of various protocols. Results from this testing indicated that it is feasible for BLS to collect data relevant to the SSA’s disability program using the NCS platform. Details on these collection tests are documented in the “Testing the Collection of Occupational Requirements Data” report found in the 2013 ASA Papers and Proceedings (see Attachment 6).

The results of Phase 1’s Proof-Of-Concept Test suggested that BLS’ approach is viable. Respondents agreed to participate in the test. BLS field economists were able to capture the required data from traditional NCS respondents, and individual data element response rates were very high. Additional information on this test and the lessons learned are available in the “Occupational Requirement Survey, Phase 1 Summary Report, Fiscal Year 2013” (see Attachment 7).

Phase 2’s Collection Protocol Development Test evaluated ORS collection protocols and aids that had been updated following Phase 1 testing (e.g., streamlined collection tools; implementation of a probability-based establishment selection method; refined frequency questions; limited phone collection). This test was also developed to assess ORS collection outside the DC metropolitan area using an expanded number of BLS field economists. The results of Phase 2 testing, which can be found in the “Occupational Requirement Survey, Phase 2 Summary Report, Fiscal Year 2013” (see Attachment 8), demonstrated the effectiveness of the revised materials and procedures and the continued viability of BLS collection of data relevant to the SSA’s disability program. Respondents agreed to participate in the test. BLS field economists were able to capture the required data from typical NCS respondents, and individual data element response rates were very high.

Phase 3’s Broad Scale Testing was designed to show whether ORS field economists from across the country could collect all of the ORS data elements in addition to wages and leveling information in a uniform and efficient manner. Phase 3 testing also included supplemental tests to assess the feasibility of Central Office Collection (COC), joint collection of ORS and Employment Cost Index (ECI) elements, and conducting “efficiency” interviews. The Phase 3 testing demonstrated the effectiveness of the revised materials and procedures and the continued viability of BLS collection of data relevant to the SSA’s disability program. The details of this test and the results are further documented in the “Occupational Requirement Survey, Phase 3 Summary Report, Fiscal Year 2013” (see Attachment 9). As in the prior two tests, respondents agreed to participate in the test. BLS field economists were able to capture the required data from traditional NCS respondents, and individual data element response rates were very high.

• Fiscal Year 2014

The BLS completed six feasibility tests to refine the ORS methodology. Five tests were conducted nationally, across all six BLS regions; and one test, the New Data Elements Test, was conducted only in two Metropolitan Areas: Washington, D.C. and San Diego, CA.

The six feasibility tests were designed to:

• refine the methods to develop more efficient approaches for data collection as identified during fiscal year 2013 testing (ORS Only Efficiency Innovations Test);

• determine how best to collect occupational requirements data elements and NCS data elements from the same establishment (NCS/ORS Joint Collection Test);

• determine how best to collect the new mental and cognitive demands for work data elements, and evaluate the use of occupational task lists as developed by ETA’s O*NET program during data collection (New Data Elements Test);

• determine how best to collect occupational requirements data elements from America’s largest firms and State governments (Central Office Collection Test); and,

• determine how best to collect occupational requirements data elements when a personal visit is not optimal due to respondent resistance, collection costs, or other factors (Alternative Modes Test).

In general, the results from these tests confirmed BLS’ viability at collecting data relevant to ORS and demonstrated the effectiveness of the revised materials and procedures tested. All test objectives were successfully met and these activities established a strong foundation for the Pre-Production Test. More detailed information on these feasibility tests as well as key findings can be found in the “Occupational Requirement Survey, Consolidated Feasibility Tests Summary Report, Fiscal Year 2014” (see Attachment 10).

• Fiscal Year 2015

The Pre-Production test was designed to test all survey activities by mirroring production procedures, processes and protocols as closely as possible. Pre-Production data collection spanned approximately six consecutive months. ORS collection included both NCS schedules as well as a set of ORS-only schedules. The field economists followed the standard non-response follow-up protocols to attempt to collect as many assigned schedules as possible. All ORS data elements planned for Production were to be collected from all schedules. For NCS establishments that were already initiated, NCS data elements, such as employment and list of occupations were not collected again.

Every normal production activity associated with each of BLS’ product lines was conducted during Pre-Production testing. Production activities included selecting ORS samples, training staff, conducting calibration exercises, collecting the data, conducting all review activities, calculating estimates and standard errors, validating the estimates, and applying publication criteria to the computed estimates. All staff collecting data during the Pre-Production test were trained and participated in calibration testing. Staff used an ORS data capture system available for regional and national office use and ORS data review processes were conducted. More detailed information on the Pre-Production test can be found in the “The Occupational Requirements Survey: estimates from preproduction testing” (see Attachment 11).

In Fiscal Year 2015, ORS production sample units were selected using a 2-stage stratified design with probability proportional to employment sampling at each stage. The first stage of sample selection will be a probability sample of establishments and the second stage will be a probability sample of jobs within sampled establishments. The total ORS production sample in the first year was 4,250 establishments and 10,000 establishments for the next two years. The private portion of the sample was approximately 85% while the State and Local government sample was approximately 15% of the total sample each year. The establishment sample allocation to the industry strata was proportional to stratum employment.

Collection of the ORS production sample for Year 1 took approximately 9 months, beginning in the fall of 2015. Collection of the samples after Year 1 spanned a period of 12 months after an initial three month refinement period that overlapped collection of the prior sample. For more details on this design see paper by Ferguson and McNulty on “Occupational Requirements Survey Sample Design” (Attachment 12).

BLS also conducted a job observation test during the summer of 2015 to provide validation for the ORS physical elements by comparing the data collected during pre-production to those collected through direct job observation, which is more typical among small scale studies of job tasks. As part of this test, Field Economists (FE) re-contacted establishments who had responded to the ORS pre-production survey and observed workers actually performing their jobs to obtain data on the physical requirements of the job. The purpose of the job observation test was to provide validation for the ORS physical elements by comparing the data collected during pre-production to those collected from a different source – observation. Two field economists were assigned to observe the same job for 60 minutes and record the duration of each of the physical elements of the job.

Initial results showed high levels of inter-rater reliability among the two observing FEs, suggesting that any future observations could be done without pairs of FEs. Comparing the observed data to that collected during pre-production proved somewhat more complicated due to the limited length of the observation time resulting in some elements classified as “not present” that were more likely present with very low frequency. The measures of agreement for duration were relatively strong, however, suggesting that the collected data and observed data have high levels of agreement across most elements. More details and results from this test can be found in the paper titled “Occupational Requirements Survey Job Observation Report” (see Attachment 13).

• Fiscal Year 2016 to Present

The ORS program began collection of the second sample from the first production wave in FY 2016 and the third and final sample to be used in the first wave of estimates will conclude in the summer of 2018. During FY2017 and FY2018, three tests of data collection methods are being conducted. The first test continues the BLS work to validate the ORS data and methodology. It is a larger scale version the FY2015 Job Observation test, and is focused on ORS elements and occupations that are amenable to testing by observation. This test will produce information on the rates of agreement between data collection methods, and will give insight into the availability of data via observation that was not able to be collected via interview. A research paper will be produced and published in the Monthly Labor Review. This research paper will be made available on the www.bls.gov website.

Data for the second test was collected between September and November 2017 and focused on a comprehensive set of questions on the mental/cognitive demands for a job. Earlier cognitive data collection questions did not yield data that would meet SSA’s needs for adjudication and were discontinued in August 2017 after OMB approval was received on 4/28/2017. New questions were designed and tested on a limited basis through the BLS Office of Survey Methods Research (OSMR) generic Clearance 1220-0141 in the first half of 2017. The latest testing was a larger field test to refine question wording and response clarity using a more diverse group of occupations and industries. Overall, field economists that collected the data indicated the revised cognitive questions were easier for the respondent to comprehend and were an improvement over those administered in previous testing and production samples. The questions were more readily understandable as well as more efficient to collect, resulting in limited issues during the collection and coding of the cognitive data elements. The outcome of the test is a revised set of mental/cognitive questions and response answers incorporated into the ORS survey. More details and results from this test can be found in the paper titled “Occupational Requirements Survey (ORS) Cognitive Test Summary Report Fiscal Years 2017-2018” (see Attachment 14).

The third test is the FY2018 Occupational Selection Test. The primary goal of this test is to evaluate the new occupational quote selection described in section 1 above and the impact this change will have on training and collection procedures. The test includes a range of establishments in order to accurately refine data collection procedures and provide insights for field economist training. This test will begin in April 2018 and continue for approximately six weeks.

4b. Tests of Survey Design Processes

Sample Design Options

To further ensure the BLS met the needs of the ORS by producing statistically valid and high quality data, testing on possible sample design options was also conducted. In FY 2013, the BLS began work to evaluate sample design options for ORS by reviewing the sample designs used for the NCS. More details on this initial sample design testing is available in the November 2013 FCSM Proceedings, “Sample Design Considerations for the Occupational Requirements Survey” (see Attachment 15). This research continued into FY 2014 and expanded to look at other BLS surveys, including the Occupation Employment Statistics (OES) and Survey of Occupational Injuries and Illnesses (SOII). Since the ORS will be collected by trained field economists who also collect the NCS data, potential coordination with the NCS sample design was a key factor of consideration. As a result, four basic categories of ORS survey designs were identified to allow for different potential levels of coordination with NCS. These design options, which are documented in the ASA 2014 Papers and Proceedings titled “Occupational Requirement Survey, Sample Design Evaluation” by Ferguson et al (see Attachment 16) are:

1. Fully Integrated Survey Design – where the NCS establishment sample would be a subsample of the ORS establishment sample

2. Independent Survey Design – where the ORS establishment samples would be selected using a design appropriate for SSA’s needs, the NCS establishment samples would be selected using the current NCS sample design, and there would be no control on the amount of establishment sample overlap between the samples selected for the two surveys

3. Separated Survey Design – where the NCS establishment sample would be selected from the frame, the selected NCS establishments would be removed from the frame, and an independent ORS establishment sample would be selected from the rest of the frame

4. OES-ORS Integrated Design – where the ORS establishment sample would be selected as a subsample of the OES establishment sample

While desirable for the ORS sample design to be integrated with NCS, it was unclear whether the NCS sample design would meet the goals of ORS. There are many things to consider when choosing a sample design for the ORS. Cost, individual respondent burden, overall respondent burden, response rates, data quality, the effect on the ECI, and whether the surveys could be integrated were all factors. After various testing on the four basic categories of ORS survey designs, the BLS determined that the most viable design options, among those considered, were the Fully Integrated and the Independent Survey Designs. However, among the integrated sample designs considered, no design afforded the ability to meet fully the goals of both the ORS and the NCS; therefore, it was determined that the Independent Survey Design was the optimal design option for implementation of the first production wave. This design, as demonstrated through the most recent two years of first wave production, met the requirements of being able to produce reliable estimates for ORS data elements; however, it did not meet the needs of the SSA in terms of its ability to produce reliable estimates for ORS data elements for the vast majority of 8-digit SOC codes.

In order to improve the balance of the number of observations (quotes) sampled across all occupations and increase the publication rate across a greater number of occupations while maintaining current resource levels, in FY2017 the BLS began additional research into alternative sample design options for the ORS. For each of the options extensive research, including simulating hypothetical samples, analyzing sample allocations, and estimating the predicted number of observations per occupation per hypothetical sample, was completed prior to coming to a final design. The options studied included:

1. Modify current ORS industry sample allocations but maintain the remaining design features

• Description: Use OES collected data to determine industries where certain occupations exists. Allocate sample based on this output. Continue to use current Probability Selection of Occupations (PSO) procedures to identify the occupations to collect.

• Observations: Although the alternative allocations slightly increased the number of potentially publishable occupations, the SOC coverage did not improve greatly. The same amount of occupations continued to have very high sample quote counts, so sample balance was not improved.

2. Modify ORS industry sample allocations and PSO procedures but maintain the remaining design features

   • Description: Use OES collected data to determine industries where SSA jobs are located. Allocate sample based on this output. Develop and implement a new method for selection of occupations that will yield more occupations for SSA needs.

   • Observations: Resources and time were not available to modify changes to PSO so this option has not been tested.

3. Construct sample from subsamples that each target a specific group of occupations

   • Description: Using knowledge (or assumption) of the occupations that are present in each establishment, sample groups of six occupations using a PPS sample design where the measure of size is based on whether or not an establishment employs people in any of those six occupations. Group the occupations by how likely they are to appear in the same establishment.

   • Observations: Among the occupations tested, the number of potentially publishable occupations decreased. Rare occupations were still hard to sample because they were in so few establishments within even their major industry. On average, only around two of the six occupations were found in each establishment, so collection would not be especially efficient.

4. Target ORS sample to pairs of low employment occupations

   • Description: Create 200 pairs of low-employment occupations where each occupation in the pair is likely to exist in the same establishment. Stratify the QCEW sample frame by industry and allocate the sample size to each industry proportional to the employment in the paired occupations. For each establishment in a selected industry, collect data for the sampled pairs of occupations. Implement standard PSO for the rest of the employees in the establishment and collect data for the sampled occupations.

   • Observations: Sample quote counts improved for occupations that are specific to and common in just one or two industries. Some rare occupations were still hard to sample because they were in so few establishments within even their major industry. Some occupations were spread across numerous industries, diluting the advantage of focusing on just two SOCs at a time.

5. Two-phase stratified sampling to target specific occupations of interest

   • Description: Select a probability proportionate establishment employment sample of n units. Perform a match of this sample with OES sample. For matched (overlap) units extract occupations that are present. It is expected that about 50 percent of ORS sample units will overlap with OES units. Units that do not overlap with OES will need to be surveyed to obtain what occupations are present within those non-overlapping sampled establishments. In the second phase identify units that contain specific occupations or a group of occupations of interest and take a subsample of those phase 1 units that would yield a desired number of observations for a given occupation. Grouping of occupations is desirable when occupations included in a group exist in the same establishment.

   • Observations: Due to resource constraints and respondent burden issues, this option has not been tested.

6. Multiple frames stratified sampling to target specific occupations of interest

   • Description: Stratify OES modeled frame data by establishments with 100 or more employees and fewer than 100 employees. Identify 200 occupations to be surveyed in sample year one. The occupations that are eligible to be included in this list are occupations that do not have sufficient number of observations (less than 30) collected in 702-704 samples. For each occupation identify the list of establishments (sampling frame) that contains this occupation. If the sampling frame contains 50 or fewer establishments, then include all establishments in the sample. If there are more than 50 establishments, then allocate 38 units (75% of the sample) to stratum with 100+ employees and 12 units (25% of the sample) to stratum with less than 100 employees. Sort establishments by 23 detailed industries and establishment employment size and select systematic sample of establishments. After selection of the 200 samples of 50 units each check (using match on LDB number) if there are units that appear in more than one sample. For matched units data collection would be combined. If there are more than 50 units that appear in more than one sample, then select additional sample(s) of 50 units each to cover additional occupation(s). The above process should be repeated for a new set of 200 occupations to be included in each of the samples in years two through five.

   • Observations: Assuming a response rate of 75 percent and 50 percent of units in stratum with less than 100 employees will have desired occupation this design should yield about 32 observations for each of the 1,000 surveyed occupations. However, this design is less efficient from data collection standpoint since, on average, data for only one occupation is collected from each sampled establishment.

7. Two-stage stratified sampling to target rare occupations

   • Description: The sampling design for the five-year private industry sample is a two-stage stratified sample of private industry establishments and occupations within selected establishments. Strata are formed by the cross-classification of the predicted presence/absence of a “rare occupation” in the establishment, Census Region (Northeast, Southeast, Midwest, West), and aggregate industry (Education, Goods Producing, Health Care, Financial Activities, Service Providing), leading to forty strata. For the purposes of sample selection, a “rare occupation” is defined as one of the 200 6-digit SOCs with the lowest May 2017 OES employment, across all ownerships.

   • Observations: This design limits oversampling of higher employment occupations by allocating more sample in occupations that would have a lower probability of selection under a probability proportional to occupational employment. Based on simulation results after five years almost 800 6-digit SOCs would meet publication criteria.

After reviewing the results of each of the above design approaches, option 7 yielded the most promising results to reach the goal of publishing estimates for a broader number of the nearly 1,100 8-digit SOC codes. This new design also has the potential to save time for both the Field Economist as well as the respondent by reducing the number of establishments for which the selection of occupations is completed during collection.

Data Review and Validation Processes

BLS has developed a variety of review methods to ensure data of quality are collected and coded. These methods include data review and validation processes and are available in more detail in the 2014 ASA Papers and Proceedings under the title “ Validation in the Occupational Requirements Survey: Analysis of Approaches” by Smyth (see Attachment 17).

The ORS Data Review Process is designed to create the processes, procedures, tools, and systems to check the micro-data as they come in from the field. This encompasses ensuring data integrity, furthering staff development, and ensuring high quality data for use in producing survey tabulations or estimates for validation. The review process is designed to increase the efficiency of review tools, build knowledge of patterns and relationships in the data, develop expectations for reviewing the micro-data, help refine procedures, aid in analysis of the data, and set expectations for validation of tabulations or future estimates.

To further ensure the accuracy of the data, the ORS Validation Process focuses on aggregated tabulations of weighted data as opposed to individual data. This entails a separate but related set of activities from data review. The goal of the validation process is to review the estimates and declare them Fit-For-Use (FFU), or ready for use in publication and dissemination, as well as confirming that our methodological processes (estimation, imputation, publication and confidentiality criteria, and weighting) are working as intended. Validation processes include investigating any anomalous estimates, handling them via suppressions or correction, explaining them, documenting the outcomes, and communicating the changes to inform any up/down-stream processes. All results of validation are documented and archived for future reference if necessary.

Overall, the ORS poses review and validation challenges for the BLS because of the unique nature of the data being collected. In order to better understand the occupational requirements data, the BLS engaged in a contract with Dr. Michael Handel, a subject matter expert. From the fall of 2014 through January 2015, Dr. Handel reviewed and analyzed literature related to the reliability and validity of occupational requirements data. At the conclusion of his work, Dr. Handel provided the BLS with the recommendations below with the understanding that the ORS is complex in nature and there is no “one size fits all” approach for testing reliability and validity of the data items:

• The development of a strategic documentation to guide methodological research. The guide should include:

• Background on the data needs, intended uses, and feasible collections.

• A list of variables, levels of measurement, response options, and methods for calculating composite measures.

• Variables and response options that are highest priority for testing based on the needs of SSA.

• A list of occupations and data elements of highest priority for SSA.

• A clear statement of different data collection methods under considerations and rankings of cost feasibility.

• A description of the format and content of data products.

• An evaluation on the existence of “gold standard” benchmarks for methods of data collection and for data elements. The evaluation should include:

• Using field economists to observe occupations as the standard for physical demands and environmental conditions.

• Using physical measuring devices for environmental conditions (such as noise)

• Comparing alternative methods of data collection to determine their accuracy relative to the gold standard

• For data elements without any gold standards, multiple approaches may be used.

• When ORS data elements have overlap with variables in existing microdata sets (e.g. education and training requirements), these databases should be used to measure agreement between ORS data and other data sources.

• When there is little or no overlap between ORS data elements and existing databases (e.g. cognitive requirements), subject matter experts should be consulted to structure tests to determine validity. BLS should contract with an IO Psychologist to assist with this effort.

• Measures of agreement for ORS data should consist of assessing data agreement within method, as opposed to across methods. Because there are many characteristics of the interview that may cause variability (e.g. characteristics of the respondent, length of interview, characteristics of the job and establishment, identity of the field economist/field office), it would be significant to use debriefs with the field economists to identify the key characteristics of the interview to focus on for measures of reliability.

• Consideration should be given to variation caused by errors in coding occupations.

BLS management agreed with the recommendations provided by Dr. Handel. As a result, the BLS began a review initiative in FY 2015 including the development of a methodological guide, evaluation of “gold standard” benchmarks for data collection, and testing of inter-rater reliability (see “Occupational Requirements Survey Job Observation Report,” Attachment 13). More detailed information on Dr. Handel’s proposals are explained in an Executive Summary paper titled “Methodological Issues Related to ORS Data Collection” by Dr. Handel (see Attachment 18). These recommendations, as well as refinements of the ORS manual, the data review process, and the validation techniques developed to date ensured ORS products are quality occupational data in the areas of vocational preparation, mental-cognitive and physical requirements, and environmental conditions.

Throughout the testing stages as well as the first wave of production for the ORS, BLS has conducted various calibration activities. As stated in a paper by Mockovak, Yu & Earp (see attachment 19), calibration training is a type of refresher training that compares interviewer performance against predetermined standards to assess rating accuracy, inter-rater reliability, and other measures of performance. In addition to those described in this paper, the BLS conducts calibration exercises to test staff understanding and adherence to problematic concepts. Most recently, in FY2018, calibration activities focused on SOC coding to better ensure the accuracy and consistency across all National and Regional Office staff that are involved in collection, coding and/or review of ORS microdata. Information obtained during the various calibration activities is used to enhance procedural guidance as well as training materials.

5. Statistical and Analytical Responsibility

Dr. Jeffrey Gonzalez, Chief, Statistical Methods Group of the Office of Compensation and Working Conditions, is responsible for the statistical aspects of the ORS production. Dr. Gonzalez can be reached on 202-691-7517. BLS seeks consultation with other outside experts on an as needed basis.

6. References

Bradley D. Rhein, Chester H. Ponikowski, and Erin McNulty, (October 2014), “Estimation Considerations for the Occupational Requirements Survey,” ASA Papers and Proceedings, http://www.bls.gov/ncs/ors/estimation_considerations.pdf (Attachment 1)

Chester H. Ponikowski and Erin E. McNulty, (December 2006), "Use of Administrative Data to Explore Effect of Establishment Nonresponse Adjustment on the National Compensation Survey", ASA Papers and Proceedings, http://www.bls.gov/ore/abstract/st/st060050.htm, (Attachment 2)

Chester H. Ponikowski, Erin McNulty and Jackson Crockett (October 2008) "Update on Use of Administrative Data To Explore Effect of Establishment Nonresponse Adjustment on the National Compensation Survey Estimates", ASA Papers and Proceedings, http://www.bls.gov/osmr/abstract/st/st080190.htm, (Attachment 3)

Bureau of Labor Statistics’ Handbook of Methods, Chapter 8, Bureau of Labor Statistics, 2010 http://www.bls.gov/opub/hom/homch8.htm, (Attachment 4)

Alice Yu, Chester H. Ponikowski, and Erin McNulty (October 2016), “Response Rates for the Pre-Production Test of the Occupational Requirements Survey,” ASA Papers and Proceedings,

https://www.bls.gov/ors/response_rates_for_the_preproduction_test.pdf (Attachment 5)

Gwyn R. Ferguson, (January 2014), "Testing the Collection of Occupational Requirements Data," ASA Papers and Proceedings, http://www.bls.gov/osmr/abstract/st/st130220.htm, (Attachment 6)

The ORS Debrief Team, (January 2013) “Occupational Requirements Survey, Phase 1 Summary Report, Fiscal Year 2013," Bureau of Labor Statistics, http://www.bls.gov/ncs/ors/phase1_report.pdf, (Attachment 7)

The ORS Debrief Team, (April 2013) “Occupational Requirements Survey, Phase 2 Summary Report, Fiscal Year 2013," Bureau of Labor Statistics, http://www.bls.gov/ncs/ors/phase2_report.pdf, (Attachment 8)

The ORS Debrief Team, (August 2013) “Occupational Requirements Survey, Phase 3 Summary Report, Fiscal Year 2013," Bureau of Labor Statistics, http://www.bls.gov/ncs/ors/phase3_report.pdf, (Attachment 9)

The ORS Debrief Team, (November 2014) "Occupational Requirements Survey, Consolidated Feasibility Tests Summary Report, Fiscal Year 2014," Bureau of Labor Statistics, http://ocwc.sp.bls.gov/ncsmanagers/392coord/fy14final/ORS_FY14_Feasibility_Test_Summary_Report.pdf, (Attachment 10)

Nicole Dangermond, (November 2015) “The Occupational Requirements Survey: estimates from preproduction testing,” Monthly Labor Review, https://www.bls.gov/opub/mlr/2015/article/the-occupational-requirements-survey.htm, (Attachment 11)

Gwyn R. Ferguson and Erin McNulty, (October 2015), “Occupational Requirements Survey Sample Design,” http://www.bls.gov/osmr/pdf/st150060.pdf, (Attachment 12)

The ORS Job Observation Test Team, (November 2015), “Occupational Requirements Survey Job Observation Report,” http://www.bls.gov/ncs/ors/preprod_job_ob.pdf, (Attachment 13)

The ORS Cognitive Test Team, (April 2018), “Occupational Requirements Survey (ORS) Cognitive Test Summary Report Fiscal Years 2017-2018,” http://www.bls.gov/ncs/ors, (Attachment 14).

Bradley D. Rhein, Chester H. Ponikowski, and Erin McNulty, (November 2013), “Sample Design Considerations for the Occupational Requirements Survey,” FCSM Papers and Proceedings, https://fcsm.sites.usa.gov/files/2014/05/H4_Rhein_2013FCSM.pdf, (Attachment 15)

Gwyn R. Ferguson, Erin McNulty, and Chester H. Ponikowski (October 2014), “Occupational Requirements Survey, Sample Design Evaluation,” ASA Papers and Proceedings, http://www.bls.gov/ncs/ors/sample_design.pdf, (Attachment 16)

Kristin N. Smyth, (October 2014), “Validation in the Occupational Requirements Survey: Analysis of Approaches,” ASA Papers and Proceedings http://www.bls.gov/ncs/ors/validation.pdf, (Attachment 17)

Michael Handel, (February 2015), “Methodological Issues Related to ORS Data Collection,” www.bls.gov/ncs/ors/handel_exec_summ_feb15.pdf, (Attachment 18)

William Mockovak, Alice Yu & Morgan Earp, (September 2015), “Using Calibration Training to Assess the Quality of Interviewer Performance,” ASA Papers and Proceedings, https://www.bls.gov/osmr/pdf/st150010.pdf, (Attachment 19)

24