Supporting Statement Part B 9-30--14

Supporting Statement – Revised 09/30/2014

Occupational Requirements Survey (ORS) Pre-Production Test

B. Collection of Information Employing Statistical Methods

For detailed technical materials on the sample allocation, selection, and estimation methods as well as other related statistical procedures see the BLS technical reports, and ASA papers listed in the references section. The following is a brief summary of the primary statistical features of the ORS Pre-production test.

Prior planning for the ORS Pre-production test involved several feasibility tests throughout Fiscal Years 2013 and 2014.  These tests included considerations about the basic information desired and the availability of data, the efficiency of alternative collection procedures, and the probable degree of cooperation from respondents. Additionally, a final approved FY 2014 Interagency Agreement between the Bureau of Labor Statistics (BLS) and the Social Security Agency (SSA) called for the start of a Pre-production test that would attempt to closely mirror production procedures and protocols. As described in Sections 1 – 3 of this document, the ORS Pre-production sample will be selected using a 2-stage stratified design with probability proportional to employment sampling at each stage.  The first stage of sample selection is a probability sample of establishments and the second stage of sample selection is a probability sample of jobs within sampled establishments.  Data from all sampled establishments will be used to produce products, such as the indicator of "time to proficiency" of occupations, the mental-cognitive demands of work, the physical demand (PD) characteristics/factors of occupations, and the environmental conditions of occupations.

In 2014, ORS Pre-production will begin by selecting samples using the methodology described in this document. Current plans call for collection of the first schedules for the Pre-production test in the fall of calendar year 2014. Section 4.b of this document describes the efforts conducted to develop and test the proposed sample design. Prior to FY 2014, the BLS conducted three phases of feasibility testing for the ORS. The selection of establishments for these three phases were based on various criteria depending on the test, but as a result of the small sample size were not designed to produce statistically sound estimates. The ORS Pre-production test will have a greater sample size with a plan to produce top level estimates from the data obtained.

1a. Universe

The ORS plans to measure information such as "time to proficiency," mental-cognitive demands, physical demand (PD) characteristics/factors, and environmental conditions by median, mean, and distribution for national level estimates by occupation. The universe for this survey will consist of the Quarterly Contribution Reports (QCR) filed by employers subject to State Unemployment Insurance (UI) laws. The BLS receives these QCR for the Quarterly Census of Employment and Wages (QCEW) Program from the 50 States, the District of Columbia, Puerto Rico, and the U.S. Virgin Islands. The QCEW data, which are compiled for each calendar quarter, provide a comprehensive business name and address file with employment, wage, detailed geography (i.e., county), and industry information at the six-digit North American Industry Classification System (NAICS) level. This information is provided for over nine million business establishments, most of which are in the scope of this survey.

The potential respondent universe that will be used in the selection of the ORS Pre-production sample of establishments is derived from the QCEW and a supplementary file of railroads for each stratum in the sample. The ORS universe will include all establishments in the 50 States and the District of Columbia with ownership of State and local governments and private sector industries, excluding agriculture, forestry, and fishing (NAICS Sector 11) and private households (NAICS Subsector 814). Estimates of the current universe and sample size are about 9,000,000 and 2,550 establishments, respectively. Data for the duties and responsibilities of a sample of jobs will be collected in all sample establishments.

All of ORS Pre-production’s projected 2,550 sample establishments will be collected once for all of the ORS data elements.

1b. Sample

Stratification, Sample Allocation, and Sample Selection

The ORS Pre-production sample will include a combination of both ORS-only establishments as well as those that currently exist within the NCS (National Compensation Survey). All units will be selected using a 2-stage stratified design with probability proportional to employment sampling at each stage. The first stage of sample selection will be a probability sample of establishments, and the second stage of sample selection will be a probability sample of jobs within sampled establishments. For more information on the current NCS sample design as well as factors explored for an ORS sample design, see the American Statistical Association (ASA) papers by Ferguson et al titled, “Evaluating Sample Design Issues In the National Compensation Survey” (See Attachment 1), “Update on the Evaluation of Sample Design Issues in the National Compensation Survey” (See Attachment 2) and “State and Local Government Sample Design for the National Compensation Survey" (See Attachment 3) as well as the Federal Committee on Statistical Methodology (FCSM) paper by Rhein et al titled, “Sample Design Considerations for the Occupational Requirements Survey” (See Attachment 4). Each sample of establishments will be drawn by first stratifying the establishment sampling frame by defined industry and ownership. The industry strata for private industry as well as state and local government are shown below and are based on the North American Industry Classification System (NAICS).

ORS Pre-production Stratification for Private Industry

Aggregate Industry	Detailed Industry	Included NAICS Codes	Establishments in Universe	Sample Size
Education	Educational Services (Rest of)	61 (excl 6111-6113)	78,008	11
Education	Elementary and Secondary Schools	6111	16,899	14
Education	Junior Colleges, Colleges and Universities	6112, 6113	8,023	15
Finance, Insurance and Real Estate	Finance (Rest of)	52 (excl 524)	279,462	64
Finance, Insurance and Real Estate	Insurance	524	185,133	39
Finance, Insurance and Real Estate	Real Estate, Renting, Leasing	53	349,578	43
Goods Producing	Mining	21	34,579	14
Goods Producing	Construction	23	744,370	122
Goods Producing	Manufacturing	31-33	334,610	224
Health Care, including Hospitals and Nursing Care	Healthcare, Social Assistance (Rest of)	62 (excl 622, 623)	1,230,175	194
Health Care, including Hospitals and Nursing Care	Hospitals	622	8,419	68
Health Care, including Hospitals and Nursing Care	Nursing and Residential Care Facilities	623	72,659	66
Service Providing	Utilities	22	17,130	10
Service Providing	Wholesale Trade	42	619,782	121
Service Providing	Retail Trade	44-45	1,031,277	308
Service Providing	Transportation and Warehousing	48-49	225,026	75
Service Providing	Information	51	143,541	47
Service Providing	Professional, Scientific, Technical	54	1,075,999	177
Service Providing	Management of Companies and Enterprises	55	58,245	40
Service Providing	Admin., Support, Waste Management	56	485,943	161
Service Providing	Arts, Entertainment, Recreation	71	127,658	38
Service Providing	Accommodation and Food Services	72	647,059	240
Service Providing	Other Services (excl Public Administration)	81 (excl 814)	563,765	76

ORS Pre-production Stratification for State and Local Government Industry

Aggregate Industry	Detailed Industry	Included NAICS Codes	Establishments in Universe	Sample Size
Education	Elementary and Secondary Education	6111	62,349	150
Education	Colleges and Universities	6112, 6113	7,416	39
Education	Rest of Education	61 excl 6111-6113	1,281	1
Financial Activities	Other Service-producing - Part A	51, 52-53	1,961	2
Goods Producing	Goods-Producing	21, 23, 31-33	6,350	4
Health Care, including Hospitals and Nursing Care	Hospitals	622	2,377	21
Health Care, including Hospitals and Nursing Care	Nursing Homes	623	1,679	5
Health Care, including Hospitals and Nursing Care	Rest of Health	62, excl 622-623	8,546	9
Service Providing	Trade, Transportation, and Utilities	42, 44-45, 48-49, 22	12,764	14
Service Providing	Public Administration	92 excl 928	107,694	122
Service Providing	Other Service-producing - Part B	54-56, 71-72, 81 excl 814	18,462	15

After the sample of establishments is drawn, jobs will be selected in each sampled establishment. The number of jobs selected in an establishment will range from 4 to 8 depending on the total number of employees in the establishment, except for government and aircraft manufacturing units and units with less than 4 workers. In government, the number of jobs selected will range from 4 to 20. In aircraft manufacturing, the number of jobs selected will range from 4 for establishments with less than 50 workers to 32 for establishments with 10,000 or more workers. In establishments with less than 4 workers, all jobs will be selected. The probability of a job being selected will be proportionate to its employment within the establishment.

Scope - The ORS Pre-production sample will be selected from the populations as defined above.

Sample Allocation— The total ORS Pre-production sample will consist of approximately 2,550 establishments. The private portion of this sample will be approximately 85% (2,168) with one-third (716) of that coming from current NCS sample units. The remaining two-thirds (approximately 1,452) of the private sample will be selected from a national frame not to include any other existing NCS sample units. This frame will be stratified by NAICS based on the 23 detailed industry cells as defined above. The state and local government sample will be approximately 15% of the total sample (382) with one-third (126) of the units selected from the existing NCS sample and the remaining two-thirds (256) selected from a national frame not to include existing NCS sampled establishments.

The sample allocation process starts with a total budgeted sample size. Since some of the sample for the ORS Pre-production test will be selected from the NCS design, the same industry definitions (based on ownership and NAICS as defined in charts above) will be used to select both the NCS overlap sample as well as the ORS-only sample. The sample will be allocated proportionally by ownership and industry using total employment within each sample cell.

Sample Selection - The ORS Pre-production test will select a sample consisting of both NCS sample units as well as ORS-only units. The portion selected from the existing NCS sample units will use systematic sampling with probability proportionate to measure of size.  The measure of size (MOS) will be the sample unit employment times its NCS sample weight.  

 

For the ORS only sample, units will be selected from a frame that excludes all existing NCS sample units. This frame will be stratified by ownership and industry as defined above, with each sample cell being sorted by area (using the 24 area definitions in the NCS design – see below), establishment employment, and establishment identification number. These units will be selected using a probability proportional to size approach based on the unit’s employment as it was reported to on the state unemployment file.

Sample weights will be assigned to each of the selected establishments in the sample to represent the entire frame. Units selected as certainty will be self-representing and will carry a sample weight of one. The sample weight for the non-certainty units will be the inverse of the probability of selection.

24 Geographic Areas from the NCS Design

Atlanta-Sandy Springs-Gainesville, GA-AL CSA

Boston-Worcester-Manchester, MA-NH CSA

Chicago-Naperville-Michigan City, IL-IN-WI CSA

Dallas-Fort Worth, TX CSA

Detroit-Warren-Flint, MI CSA

Houston-Baytown-Huntsville, TX CSA

Los Angeles-Long Beach-Riverside, CA CSA

Minneapolis-St. Paul-St. Cloud, MN-WI CSA

New York-Newark-Bridgeport, NY-NJ-CT-PA CSA

Philadelphia-Camden-Vineland, PA-NJ-DE-MD CSA

San Jose-San Francisco-Oakland, CA CSA

Seattle-Tacoma-Olympia, WA CSA

Washington-Baltimore-No. Virginia, DC-MD-VA-WV CSA

Miami-Fort Lauderdale-Miami Beach, FL MSA

Phoenix-Mesa-Scottsdale, AZ MSA

Rest of New England Census Division (excl. areas above)

Rest of Middle Atlantic Census Division (excl. areas above)

Rest of East South Central Census Division (excl. areas above)

Rest of South Atlantic Census Division (excl. areas above)

Rest of North Central Census Division (excl. areas above)

Rest of West North Central Census Division (excl. areas above)

Rest of West South Central Census Division (excl. areas above)

Rest of Mountain Census Division (excl. areas above)

Rest of Pacific Census Division (excl. areas above)

2. Sample Design

2a. Sample Rotation

Since the ORS Pre-production is a one-time test, there is no sample rotation. The units selected for this test will be contacted this one time to capture ORS data and will not be contacted again for the ORS unless selected in a future sample. Establishments that are also in the NCS will be asked to update the NCS data elements as approved under the OMB Clearance for the NCS (220-0164) which expires on April 30, 2015.

2b. Estimation Procedure

The ORS Pre-production plan is to produce estimates as described below. Computation of these estimates will include weighting the data at both the unit (establishment) and item (occupation) level. The final weights will include the initial sample weights, adjustments to the initial sample weights, two types of adjustments for non-response, and benchmarking.  The initial sample weight for a job in a particular establishment will reflect the probability of selecting a particular establishment within one of the pre-defined geographic areas and industry along with the probability of selecting a particular job within the selected establishment.  Adjustments to the initial weights will be done when data are collected for more or less than the sampled establishment.  This may be due to establishment mergers, splits, or the inability of respondents to provide the requested data for the sampled establishment.  The two types of adjustments for non-response will include adjustment for establishment refusal to participate in the survey and adjustment for respondent refusal to provide data for a particular job.

Benchmarking, or post-stratification, is the process of adjusting the weight of each establishment in the survey to match the distribution of employment by geography and industry at the reference period.  Because the sample of establishments used to collect ORS data was chosen over the past several years, establishment weights will reflect their employment when selected.  The benchmark process will update that weight based on current employment. 

BLS has begun to develop procedures and implement processes for the estimation, estimate review, and validation for the ORS tests. The table below summarizes the recommended approach for estimation and validation in FY 2014 and FY 2015:

Recommended Estimation and Validation Approach for FY 2014 and FY 2015

Estimation

• Produce means, medians, and percentage estimates for ORS characteristics, by domain

Validation

• Use graphics to investigate distributions of micro data

• Use feasibility test data to validate future estimates

Non-response

• Investigate the feasibility of imputing for item non-response within specific cells.

Outputs

• Develop basic tables that are geared towards the more experienced data user and allow them to pull a number of estimates at one time for their own data analysis needs.

• Create charts that will be developed within a dashboard framework to analyze detailed occupational data as proposed in the ORS Outputs Vision Paper

• Produce an interactive Occupation Finder that will assist users in determining what occupations meet multiple job requirements

• Produce stand-alone charts for occupational groups and other aggregate estimates

Means, medians, and percentages for ORS will be calculated the same way as in the NCS. ORS domains follow many of the same patterns as NCS, though one difference is that ORS focuses on reporting data for individual occupations more heavily than NCS. ORS will also use an 8-digit SOC code defined by O*Net, resulting in the potential of data for 1,110 SOC codes. These 8-digit SOC codes will be linked directly to the NCS 6-digit SOC. All NCS occupational groups and sub-cells will be used for ORS. New sub-cells for supervisory, non-supervisory, and lead workers will be added. Before estimates of characteristics are released, they will first be screened to ensure that they do not violate the BLS confidentiality pledge. A promise is made to each private industry respondent, and to those government sector respondents who request confidentiality, that BLS will not release its reported data to the public in a manner which would allow others to identify the establishment, firm, or enterprise.

Estimation Formulas

All estimates use quote-level records.

1. Percent of employees with characteristic: Percent of employees with a given characteristic out of all employees in the domain.

To calculate the percent of employees with a given characteristic out of all employees in the domain, add the final quote weights across only those quotes that meet the domain (denominator) condition and characteristic condition. Then divide that number by the sum of the final quote weights across quotes that meet the domain (denominator) condition. Multiply the final quotient by 100 to yield a percentage.

Estimation Formula Notation

i = Establishment

g = Occupation within establishment i

I = Total number of establishments

G_i = Total number of quotes selected in establishment i

X_ig = 1 if quote ig meets the condition set in the domain (denominator) condition

= 0 otherwise

Z_ig = 1 if quote ig meets the condition set in the characteristic condition

= 0 otherwise

OccFW_ig = Final quote weight for occupation g in establishment i

2. Mean: Average value of a quantity for a characteristic.

To calculate the average value of a quantity for a characteristic, multiply the final quote weight and the value of the quantity for those quotes that meet the domain (denominator) condition and characteristic condition; add these values across all contributing quotes to create the numerator. Divide this number by the sum of the final quote weights across only those quotes that meet the domain (denominator) condition and characteristic condition.

Estimation Formula Notation

i = Establishment

g = Occupation within establishment i

I = Total number of establishments in the survey

G_i = Total number of quotes in establishment i

X_ig = 1 if quote ig meets the condition set in the domain condition

= 0 otherwise

Z_ig = 1 if quote ig meets the condition set in the characteristic condition

= 0 otherwise

OccFW_ig = Final quote weight for occupation g in establishment i

Q_ig = Value of a quantity for a quote

3. Median: Value of a quantity at 50^th percentile (median).

The median is the value Q_ig such that

• the sum of final quote weights (OccFW_ig) across quotes with a value less than Q_ig is less than 50 percent of the sum of all final quote weights, and

• the sum of final quote weights (OccFW_ig) across quotes with a value more than Q_ig is less than 50 percent of the sum of all final quote weights.

It is possible that there are no specific quotes ig for which both of these properties hold. This occurs when there exists a quote for which the sum of OccFW_ig of quotes whose value is less than Q_ig equals 50 percent of the sum of all final quote weights. In this situation, the median is the average of Q_ig and the value on the record with the next lowest value. The Q_ig values must be sorted in ascending order.

Include only quotes that meet the domain condition and the characteristic condition – i.e., where:

.

Estimation Formula Notation

i = Establishment

g = Occupation within establishment i

X_ig = 1 if quote ig meets the condition set in the domain condition

= 0 otherwise

Z_ig = 1 if quote ig meets the condition set in the characteristic condition

= 0 otherwise

OccFW_ig = Final quote weight for occupation g in establishment i

Q_ig = Value of a quantity for a specific characteristic for occupation g in establishment i

Calculate Estimates

ORS estimates will be defined in two dimensions. A set of conditions describes the domains, and a separate set of conditions describes the characteristics. Each characteristic must be calculated for each domain (alternatively, each domain must be calculated for each characteristic). If a quote meets the domain condition for a particular estimate, the X_ig value in the formula is 1; otherwise, it is 0. Likewise, if a record meets the characteristic condition for a particular estimate, the Z_ig value in the formula is 1; otherwise, it is 0.

Estimates that use the mean or median formulas require an additional quantity for estimation, Q_ig. The value of the variable corresponding to this quantity is placed directly into the formula.

2c. Reliability

Measuring the Quality of the Estimates

The two basic sources of error in the estimates are bias and variance. Bias is the amount by which estimates systematically do not reflect the characteristics of the entire population. Many of the components of bias can be categorized as either response or non-response bias.

Response bias occurs when respondents’ answers systematically differ in the same direction from the correct values. For example, this occurs when respondents incorrectly indicate no to a certain ORS element’s presence when that ORS element actually existed. Another possibility of having response bias is when data are collected for a unit other than the sampled unit. For example, the respondent’s focus on the question may be altered from what is required of the employee in that position to what the current employee is doing in that position. This misunderstanding would alter the sampled unit to a particular person rather than the sampled occupation. Response bias can be measured by using a re-interview survey. Properly designed and implemented, this can also indicate where improvements are needed and how to make these improvements. For the Pre-production test, the ORS data will be reviewed for adherence to ORS collection procedures using a multi-stage review strategy. Approximately five percent of the sampled establishments will be re-contacted to confirm the accuracy of coding for selected data elements. The remaining ORS units will either be reviewed in total or for selected data elements by independent reviewer in the Regional of National Offices. All schedules in the sample will be eligible for one and only one type of non-statistical review. Additionally all schedules will be reviewed for Statistical validity to ensure the accuracy of the sample weight with the data that was collected.

.

Non-response bias is the amount by which estimates obtained do not properly reflect the characteristics of non-respondents. This bias occurs when non-responding establishments have ORS elements records that are different from those of responding establishments. Non-response bias is being addressed by efforts to reduce the amount of non-response. NCS is analyzing the extent of non-response bias using administrative data from the survey frame. The results from initial analysis are documented in the 2006 ASA Proceedings of Survey Research Methods Section (See Attachment 5). A follow-up study from 2008 is also listed in the references (See Attachment 6). Details regarding adjustment for non-response are provided in Section 3 below. These studies that the NCS has completed provide knowledge that can be incorporated into ORS as well.

Another source of error in the estimates is sampling variance. Sampling variance is a measure of the fluctuation between estimates from different samples using the same sample design. Sampling variance for the ORS data will be calculated using a technique called balanced half-sample replication. For national estimates, this is done by forming 128 different re-groupings of half of the sample units. For each half-sample, a "replicate" estimate is computed with the same formula as the regular or "full-sample" estimate, except that the final weights are adjusted. If a unit is in the half-sample, its weight is multiplied by (2-k); if not, its weight is multiplied by k. For all ORS publications, k = 0.5, so the multipliers will be 1.5 and 0.5. Sampling variance computed using this approach is the sum of the squared difference between each replicate estimate and the full sample estimate averaged over the number of replicates and adjusted by the factor of 1/(1-k)² to account for the adjustment to the final weights. This approach is similar that that in the NCS. For more details, see the NCS Chapter of the BLS Handbook of Methods (See Attachment 7). Standard error, which is the square root of variance, for primary aggregate estimates of the index of quarterly change is typically less than 0.5 percent. Relative standard error, which is the square root of variance divided by the estimate, for aggregate estimates of compensation, wage, or benefit levels are typically less than 5 percent.

Variance estimation also serves another purpose. It identifies industries and occupations that contribute substantial portions of the sampling variance. Allocating more sample units to these domains often improves the efficiency of the sample. These variances will be considered in allocation and selection of the future replacement samples.

For the ORS Pre-production test, the goal is to generate estimates for several high-employment, 8-digit Standard Occupational Codes as maintained by O*Net.  Additional estimates for aggregate SOC codes will be generated as are supported by the data.  Estimates of levels should be accurate with a relative standard error less than 33% on average and the percent estimates are expected to be within 5 percent of the true (population) percent at the 90 percent confidence level.

2d. Data Collection Cycles

The ORS Pre-production test is a one-time test to begin at the conclusion of the Feasibility Testing described above and upon receipt of OMB approval. The BLS will conduct a nationwide test to evaluate ORS processes and operations in a possible production environment. The goal of the Pre‐Production Test is to identify any issues or problems that BLS would then resolve prior to the full production start. This test will be composed of approximately 2,550 establishments selected from a nationwide sample design. Approximately fifteen percent of these establishments will be selected from State and local government establishments and the remainder of the sample will be private industry businesses.

3. Non-Response

There are two types of non-response for ORS: total establishment non-response and partial non-response. The non-responses can occur at the establishment level, occupation level, or ORS element level. The assumption for all non-response adjustments is that non-respondents are similar to respondents.

To adjust for establishment or occupation non-response, weights of responding units or occupations that are deemed to be similar will be adjusted appropriately. Establishments will be considered similar if they are in the same ownership and 2-digit NAICS. If there are no sufficient data at this level, then a broader level of aggregation will be considered.

For partial non-response at the ORS element level, a replacement value will be imputed based on information provided by establishments with similar characteristics. Imputation will be done separately for each ORS element using processes currently being evaluated but not yet determined. In some cases, BLS may compute estimates based solely on data provided and provide a percentage of non-provided data.

There is a continuous effort to maximize response rates. We are continually exploring alternative methods for respondents to collect their data. Testing alternative modes of data collection for ORS was conducted as part of the FY 2014 Feasibility Tests approved by OMB under control number 1220-0164. Results from this test are included in the summary report: “Occupational Requirements Survey: Consolidated Feasibility Tests Summary Report Fiscal Year 2014” (See Attachment 13).

The response rate, based on weighted employment, is expected to be about 80 percent based on historical NCS response rates as the data is being collected from typical NCS respondents.

3a. Maximize Response Rates

To maximize the response rate for this survey, field economists will initially refine addresses ensuring appropriate contact with the employer. Then, employers will be mailed a letter explaining the importance of the survey and the need for voluntary cooperation. The letter will also include the Bureau’s pledge of confidentiality. A field economist will call the establishment after the package is sent and attempt to enroll them into the survey. Non-respondents and establishments that are reluctant to participate will be re-contacted by a field economist specially trained in refusal aversion and conversion. Additionally, respondents will be offered a variety of methods, including personal visit, telephone, fax, and email, through which they can provide data.

3b. Non-Response Adjustment

As with other surveys, ORS experiences a certain level of non-response. To adjust for the non-responses, ORS plans to divide the non-response into two groups, 1) unit non-respondents and 2) item non-respondents. Unit non-respondents are the establishments that do not report any ORS data elements, whereas item non-respondents are the establishments that report only a portion of the requested ORS data elements, such as years of prior work experience for a sub-set of sampled jobs.

The unit non-response will be treated using a Non-Response Adjustment Factor (NRAF). Item non-response will be adjusted using item imputation. Within each sampling cell, NRAFs will be calculated based on the ratio of the number of viable establishments to the number of usable respondents in the sample cell.

3c. Non-Response Bias Research

Extensive research was done to assess whether the non-respondents to the NCS survey differ systematically in some important respect from the respondents of the survey and would thus bias NCS estimates. Details of this study are described in the two papers by Ponikowski, McNulty, and Crockett referenced in Section 2c (See Attachments 5 and 6). These studies that NCS have completed provided vast knowledge that can be incorporated into ORS as well because ORS data will be collected from the same respondents and there are many similarities between the data elements collected for both NCS and ORS.

BLS will analyze survey response rates from the Pre-Production test at the establishment, occupational quote, and item (i.e. individual data element) levels. The data will be analyzed using unweighted response rates and response rates weighted by the sample weight at each level of detail. We plan to review the response rates in aggregate and by available auxiliary variables such as industry, occupation, geography (i.e. Census regions and BLS data collection regions), and establishment size. BLS will use the results from the analysis to identify the auxiliary variables that are most likely to contribute significantly to bias reduction.  Once these variables are identified they will be used in the data processing system to reduce potential nonresponse bias.  Other methods for assessing bias, such as re-contact of a subsample of refusals may be considered in the future but will not be conducted during FY 2015 as part of the work planned for the Pre-Production Test.

4. Testing Procedures

4a. Tests of Collection Procedures

The Occupational Requirements Survey (ORS) is under development by the BLS’ National Compensation Survey program in association with the SSA. Several tests have already been completed prior to the start of the Pre-production test. See the ASA paper by Ferguson et al titled “Testing the Collection of Occupational Requirements Data” (See Attachment 8) for more information on the tests conducted by the BLS. The timeline below provides a brief overview of past, present, and future testing.

Past

In summer 2012, NCS began an effort to test our ability to collect ORS data elements using the NCS survey platform. Initial testing plans focused on developing procedures, protocols, and collection aids using the NCS platform. The initial testing phases were analyzed primarily using qualitative techniques. Once testing showed that it is feasible to collect this data, more quantitative analysis and testing was conducted.

The initial three phases of testing indicated that it is feasible for BLS to collect data relevant to the SSA’s disability program using the NCS platform. The results of the Phase 1 proof-of-concept test suggest that this approach is viable. Respondents agreed to participate in the test, BLS field economists were able to capture the required data from traditional NCS respondents, and individual data element response rates were very high. Phase 2 evaluated ORS collection protocols and aids that had been updated since Phase 1 testing (e.g., streamlined collection tools; implementation of a probability-based establishment selection method; refined frequency questions; limited phone collection), and to assess ORS collection outside the DC metropolitan area using an expanded number of BLS field economists. The results of the Phase 2 test demonstrate the effectiveness of the revised materials and procedures and the continued viability of BLS collection of data relevant to the SSA’s disability program. Respondents agreed to participate in the test, BLS field economists were able to capture the required data from typical NCS respondents, and individual data element response rates were very high. Phase 3 tested whether ORS field economists from across the country could collect all of the ORS data elements and occupational wages and leveling information in a uniform and efficient manner. Phase 3 also included supplemental tests to assess the feasibility of Central Office Collection (COC), joint collection of ORS and Employment Cost Index (ECI) elements, and conducting “efficiency” interviews. The results of the Phase 3 test demonstrate the effectiveness of the revised materials and procedures and the continued viability of BLS collection of data relevant to the SSA’s disability program. Respondents agreed to participate in the test, BLS field economists were able to capture the required data from traditional NCS respondents, and individual data element response rates were very high. For more information in regards to Phase 1, 2, or 3 testing, please read their summary reports: "Occupational Requirements Survey, Phase 1 Summary Report, Fiscal Year 2013" (See Attachment 9), "Occupational Requirements Survey, Phase 2 Summary Report, Fiscal Year 2013" (See Attachment 10); "Occupational Requirements Survey, Phase3 Summary Report, Fiscal Year 2013" (See Attachment 11).

Current

In FY 2014, the BLS performed work to meet the following objectives:

• Collect data to continue evaluating whether the National Compensation Survey (NCS) can produce estimates of sufficient quantity and quality to meets the needs of the SSA;

• Continue to evaluate survey design options and develop the processes, protocols, aids, and collection procedures to meet SSA data needs;

• Provide documentation to the SSA summarizing the work performed by the BLS, conclusions drawn, and recommendations for future data collection, testing, and research.

In order to accomplish these objectives, the BLS conducted various Feasibility Tests and completed related research.

The series of Feasibility Tests completed in July 2014. For these tests, BLS identified six independent Feasibility Tests and one test was conducted during all Feasibility Testing:

1. NCS/ORS Joint Collection Test – to determine if and how BLS can collect NCS and ORS data from the same establishments without diminishing data quality in any product line.

2. Central Office Collection Test – to determine how BLS will collect ORS data from America's largest firms and State governments while balancing data quality—as measured by overall and item level response and cost of collection—without negatively impacting NCS.

3. New Data Element Tests – to determine if and how to collect the new mental and cognitive demands of work data elements and evaluate the use of occupational task lists as developed by the Employment and Training Administration’s O*NET program during data collection.

4. ORS-Only Efficiency Innovations Test – to identify the most efficient way to collect the ORS data elements tested in FY 2013 when not also collecting NCS data elements.

5. Alternative Modes Test – to determine how to collect high quality ORS data via phone, e‐mail, or both so that BLS can balance cost and data quality.

6. Educational Requirements Test – to ensure that BLS is accurately capturing the minimal education requirements needed for each occupation in support of the computation of SSA's SVP data element while providing estimates desired by the BLS Occupational Outlook Handbook and the Department of Education.

7. Observation Test – to determine if and when data coding is changed as a result of observing the work environment, the sampled occupation, or both. These data will be captured during each of the five independent tests and the Pre‐Production Test.

As BLS conducted each of these Feasibility Tests, BLS captured the results of the data collection effort in a database for review and analysis purposes. All data were reviewed for accuracy and completeness. BLS will use this information to develop data edits and edit ranges. BLS will not compute any weighted estimates from the data obtained during these Feasibility Tests, but some un-weighted tabulations may be calculated for internal BLS use only. For more information in regards to FY 2014 ORS Feasibility Tests, please read “FY 2014 ORS Test Plans” (See Attachment 12) and “Occupational Requirements Survey: Consolidated Feasibility Tests Summary Report Fiscal Year 2014” (See Attachment 13).

Future

The current plan is to start collection for the Pre-production test in October 2014. The goal of the Pre-production test is to identify any problems that would occur from mirroring production procedures, processes and protocols as close as possible. These problems would then be resolved prior to the start of production data collection.

Collection will run for approximately six consecutive months. ORS collection will occur on live NCS schedules as well as a set of ORS-only schedules. The field economists should follow the standard non-response follow-up protocols and make as much of an attempt to collect every assigned schedule as possible. Collection, data capture, and data review milestones are set. For the ORS only establishments, all data elements planned for full ORS production will be collected. For NCS establishments that have already been initiated, only the additional ORS data elements will be collected. All NCS data elements will be extracted from the NCS database and will not be collected again.

Aside from data elements, every normal production activity associated with each of our product lines will be conducted. Production activities include selecting ORS samples, identifying NCS establishments for inclusion in the test, training staff, conducting calibration exercises, collecting the data, conducting all review activities, calculating estimates and standard errors, validating the estimates, and applying publication criteria to the computed estimates. All staff collecting data during the Pre-production test will be trained and will participate in calibration testing. Prior to test commencement, training will be conducted for all collection staff. Staffs will have an ORS data capture system and ORS data review processes available for regional use. This system will minimize, or avoid, the need to enter the same data in both ORS data capture system and NCS data capture system. A decision on what, if any, of these data will be released has yet to be finalized. However, any such release will need to identify the estimates as the result of a research test and clearly describe why they are or are not suitable for SSA disability determinations. Due to the sample size, the BLS only expects to be able to compute and release data for a very limited number of occupations or occupational groups.

4b. Tests of Survey Design Procedures

In addition to the specific Feasibility Tests described above in Section 4a, BLS plans to conduct cognitive testing on a subset of ORS questions. This will allow BLS to test and determine the best way to phrase questions for specific ORS data elements. While these tests will not provide data that could be used for computing SSA requested tabulations or estimates, they will allow BLS to conduct a qualitative evaluation of alternative ways to ask questions that were problematic during FY 2013-2014 ORS testing. Specific test questions will be based on input provided from Field Economists and their experiences in collecting ORS data. No establishments will be contacted for these tests, rather, questions will be asked of individuals identified by the Office of Survey Methods Research at BLS. Each set of test questions may be asked from a maximum of 300 people, with each person asked up to four ORS questions and four additional open-ended questions about the wording used in the those questions.

Respondents might be contacted more than once. These tests will be coordinated and evaluated by the BLS’s Office of Survey Methods Research. A separate Information Collection Request will be submitted to OMB for these tests.

The BLS will use data collected in testing, input from SSA, and analysis from internal BLS experts, to research various issues identified in FY 2013. These research projects in FY 2014 include the following:

• sample selection, data review, estimation, validation, and other issues related to survey design;

• development of measures of reliability of estimates;

• measurement of survey error; and

• resolution of how best to handle jobs whose duties and responsibilities cross multiple occupational classifications.

BLS prepared a summary report to document the results of all FY 2014 Feasibility Testing and Research with recommendations for the future (See Attachment 13). These recommendations address the possible need for future testing in FY 2015 beyond this Pre‐production Test.

BLS understands that some potential data users question the validity of collecting the cognitive and physical demands without direct observation of the work performed, direct measurement, or direct collection from workers. BLS recognizes that collection of occupational data may take multiple forms, including those direct methods recommended. Surveys, tasks lists, observation, recording jobs being performed, and examining archival materials (training manuals or videos) are some common methods.  Sources of data may include job incumbents, supervisors, human resources officials, or subject matter experts.

Validation of data may involve obtaining data through a different data collection approach, through different data sources, or both.  Many studies exist that seek to validate occupational data and the approaches to validating the data vary widely depending on the context of the study. Validation is a broad term that can refer to verifying both collection methods and outputs. BLS processes typically validate final estimates against expectations developed using internal and external sources.

In the case of the Occupational Requirements Study, there are multiple possible approaches to validating data.  The “best” approach is likely dependent on both the characteristics of the occupation and the data elements to be validated.  Research to identify appropriate approaches for validating ORS elements began in FY 2014 and is on-going.  A contractor is being retained to assist us with that effort and validation tests will be structured and performed on subsets of the pre-production data (both subsets of the data elements and occupations) during FY 2015.

In June the BLS issued a Request for Information (RFI), "Response to Occupational Requirements Literature Review RFI" and received interest from contractors.  As a result, in July BLS issued an “Occupational Requirements Literature Review” Request for Proposal (RFP).  The statement of work specifies that the contractor will evaluate BLS internal research on existing literature regarding the methods of collection of occupational requirements data and approaches for testing validity and reliability of such data.  The contractor will complete the literature review and provide recommendations regarding further research efforts centered on measuring data validity and reliability.

This RFP requests the following technical experience from the contractor: One subject matter expert will perform the work under this task.  This expert should have at minimum a Masters degree and eight years of experience in the research of occupational requirements.  Research experience in occupational requirements should include the following topics: trends in occupational requirements, methods for collecting national data on occupational requirements, and analysis of occupational requirements data.

5. Statistical and Analytical Responsibility

Ms. Gwyn Ferguson, Chief, Statistical Methods Group of the Office of Compensation and Working Conditions, is responsible for the statistical aspects of the ORS Pre-production test. Ms. Ferguson can be reached on 202-691-6941. As mentioned in the above paragraph, BLS seeks consultation with other outside experts on an as needed basis.

6. References

Gwyn Ferguson, Chester Ponikowski, Joan Coleman, (August 2010), “Evaluating Sample Design Issues in the National Compensation Survey,” ASA Papers and Proceedings, www.bls.gov/osmr/abstract/st/st100220.htm, (Attachment 1)

Gwyn Ferguson, Joan L. Coleman, Chester Ponikowski, (August 2011), “Update on the Evaluation of Sample Design Issues in the National Compensation Survey,” ASA Papers and Proceedings, (http://www.bls.gov/osmr/pdf/st110230.pdf), (Attachment 2)

Gwyn Ferguson, Chester Ponikowski, Erin McNulty, (August 2012), “State and Local Government Sample Design for the National Compensation Survey" ASA Papers and Proceedings, www.bls.gov/osmr/abstract/st/st120280.htm, (Attachment 3)

Bradley D. Rhein, Chester H. Ponikowski, Erin McNulty, (November 2013), “Sample Design Considerations for the Occupational Requirements Survey,” FCSM Papers and Proceedings, (Attachment 4)

Chester H. Ponikowski and Erin E. McNulty, (December 2006), "Use of Administrative Data to Explore Effect of Establishment Nonresponse Adjustment on the National Compensation Survey", ASA Papers and Proceedings, http://www.bls.gov/ore/abstract/st/st060050.htm, (Attachment 5)

Chester H. Ponikowski, Erin McNulty and Jackson Crockett (October 2008) "Update on Use of Administrative Data To Explore Effect of Establishment Nonresponse Adjustment on the National Compensation Survey Estimates", ASA Papers and Proceedings, http://www.bls.gov/osmr/abstract/st/st080190.htm, (Attachment 6)

Bureau of Labor Statistics’ Handbook of Methods, Chapter 8, Bureau of Labor Statistics, 2010 http://www.bls.gov/opub/hom/homch8.htm, (Attachment 7)

Gwyn R. Ferguson, (January 2014), "Testing the Collection of Occupational Requirements Data," ASA Papers and Proceedings, http://www.bls.gov/osmr/abstract/st/st130220.htm, (Attachment 8)

The ORS Debrief Team, (January 2013) “Occupational Requirements Survey, Phase 1 Summary Report, Fiscal Year 2013," Bureau of Labor Statistics, http://www.bls.gov/ncs/ors/phase1_report.pdf, (Attachment 9)

The ORS Debrief Team, (April 2013) “Occupational Requirements Survey, Phase 2 Summary Report, Fiscal Year 2013," Bureau of Labor Statistics, http://www.bls.gov/ncs/ors/phase2_report.pdf, (Attachment 10)

The ORS Debrief Team, (August 2013) “Occupational Requirements Survey, Phase 3 Summary Report, Fiscal Year 2013," Bureau of Labor Statistics, http://www.bls.gov/ncs/ors/phase3_report.pdf, (Attachment 11)

The ORS Debrief Team, (September 2013) "Fiscal Year 2014 Occupational Requirement Survey (ORS Test Plans)," Bureau of Labor Statistics, http://www.bls.gov/ncs/ors/fy2014_test_plans.pdf, (Attachment 12)

The ORS Debrief Team, (September 2014) “Occupational Requirements Survey: Consolidated Feasibility Tests Summary Report Fiscal Year 2014,” Bureau of Labor Statistics (Attachment 13)

Gwyn R. Ferguson, Erin McNulty, Chester Ponikowski (September 2014), “Occupational Requirements Survey Sample Design Evaluation”, ASA Papers and Proceedings (Attachment 14)

Bradley D. Rhein, Chester H. Ponikowski, and Erin McNulty (September 2014), “Estimation Considerations for the Occupational Requirements Survey,” ASA Papers and Proceedings (Attachment 15)

Kristin N. Smyth (September 2014), “Validation in the Occupational Requirements Survey: Analysis of Approaches,” ASA Papers and Proceedings (Attachment 16)

19