Note to Reviewer

October 18, 2012

MEMORANDUM FOR: REVIEWER of 1220-0164

FROM: Hilery Simpson

Chief, Division of Compensation Data Analysis and Planning

Office of Compensation and Working Conditions

Bureau of Labor Statistics

SUBJECT: Non-substantive Change Request for the
National Compensation Survey (NCS)

Clearance is being sought for the Bureau of Labor Statistics’ (BLS) National Compensation Survey (NCS) program for NCS feasibility testing for additional occupational characteristics in cooperation with the Social Security Administration (SSA). Information on this upcoming test was included in Part A, Section 12 of the most recent OMB Clearance which was approved on May 1, 2012. This request covers the first phase of testing and details the plans for future testing. Updated procedures for subsequent tests will be sent to OMB for review and approval prior to testing.

The NCS program, working with Social Security Administration, plans to do feasibility testing on three types of data elements – Specific Vocational Preparation, Physical Demands, and Environmental Conditions in FY 2013. NCS has estimated that 988 private industry and 151 government establishments (1139 establishments’ total) are available for this test in our current OMB Clearance. This will cover our FY 2013 testing requirements. No additional burden hours are being requested. The first phase of testing is planned for November to December 2012. After learning from phase 1 testing, the second phase of testing is planned for January to March of 2013, followed by a third phase from April to July of 2013.

Updated procedures for subsequent tests will be sent to OMB for review and approval as developed prior to testing. Plans call to use the same data elements and measurements for all phases of testing. Changes to collection materials, aides, and methods use in data collection are expected between testing phases, and other procedural changes may occur to add better methods for collection.

Testing Plan Overview

Section I – Initial work on feasibility testing

The BLS will work in conjunction with the Social Security Administration to design, develop, and carry out a series of tests to assess the feasibility of using the National Compensation Survey (NCS) platform as a means to accurately and reliably capture data that are relevant to the SSA's disability program purposes. To ensure that the data the BLS collects will serve the SSA's disability program purposes, these tests will include:

1. An indicator of "time to proficiency," defined as the amount of time required by the typical worker to learn the techniques, acquire the information, and develop the facility needed for average job performance, comparable to the Specific Vocational Preparation (SVP) used in the Dictionary of Occupational Titles (DOT).

2. Physical Demand (PD) characteristics/factors of occupations measured in such a way to support SSA disability determination needs, comparable to measures in Physical Demands of the Selected Characteristics of Occupations (SCO).

3. Environmental conditions (e.g., high risk due to extreme temperatures) that replicate as closely as possible those that the SSA currently uses Environmental Conditions of the SCO, or specific revisions or additions to these factors.

In FY 2013, the BLS will collect data from a set of establishments and State and local government units that are within the scope of the NCS to meet SSA needs in terms of the type of data elements collected, the measures associated with these elements, and the range and level of detail of occupations identified and classified. The data collected during the tests will include SSA-specific data elements and selected NCS data elements. Depending on the results of the FY 2013 tests, the BLS and the SSA will determine whether additional testing is needed in the subsequent fiscal year.

Section II – Phase 1 Testing: Initial Proof of Concept Testing

The primary goal of this phase of testing will be to ensure that the BLS field economists know how to describe the survey and ask respondents for information regarding the new data elements. In addition, the BLS will create and test an initial set of data collection protocols and a preliminary set of data collection aides for the data elements needed by SSA. During this phase of testing, BLS plans to begin learning about the methods for computing estimates, analyzing the collected data, and validating future estimates.

Establishment Identification: The BLS Quarterly Census of Employment and Wages (QCEW) serves as the sampling frame for the current NCS survey and will serve as the frame of establishments for this pilot test. The specific establishments to be contacted will be identified from a list of frame establishments in the Washington, D.C. geographic region as defined by OMB and will exclude all establishments currently active in any NCS sample.

Test Size/Geographic Areas/Industry Coverage: The size for this test will be 25 – 30 establishments in the Washington, D.C. geographic region. The BLS plans to include as broad mix of industries in the test establishments as possible, but will not be able to include all industries in this phase of testing due to the small test size.

Data Elements: The field economists will collect all data elements in Appendix A., plus basic occupational information for each job identified by the respondent.

Procedures Development: The BLS will develop the Phase 1 collection procedures using input from SSA, the Dictionary of Occupational Titles, data gathered through environmental scanning, and standard BLS collection approaches. The procedures will be documented in written materials that will be provided to the testers, SSA, and others as appropriate. Before use in testing, the procedures will be reviewed carefully by the BLS Cognitive Research Lab to ensure that the processes and terminology are appropriate for establishment data collection efforts based on generally accepted behavioral science practices.

Collection Tools: For Phase 1, the field economists will collect the data by taking unstructured notes using pen and paper. The collected data will be captured in an electronic format for preliminary review and analysis purposes after the interviews are conducted.

Data Review: During this phase, the BLS will identify possible data review approaches and needs, begin developing expectations for these new data through internal and external environmental scanning, start framing initial edits around the collection elements, begin developing initial review parameters, and review the data collected for the test.

Training: Field economists and others assigned to the SSA testing will receive initial training using distance education methods (webinar, videoconference and teleconference along with self study activities) and classroom training. At the beginning of this test phase, participants will receive in person classroom training covering test content and procedures.

In summary, the test activities for Phase 1 will be evaluated by debriefing the employees involved in conducting the test collection efforts periodically during the test period and at the end of the test period and by reviewing comments and notes about the test collection activities recorded during the interviews.

In all debriefing sessions in each phase of testing the data collectors and observers will be asked:

• Length of interview AND/OR time spent by the respondent in preparation for and during the interview

• Observed respondent reactions to specific data elements requested

• Number of respondents needed to provide answers to all the questions

• Issues using the collection aides & suggestions for improvements to the collection aides

• Issues that arose when asking for the data elements & suggestions for improving the process of asking for the data elements

To complete the test evaluations, we will also review the collected data and evaluate response rates at the establishment, occupation, and data element levels.  Although the establishments included in these tests are not a fully representative probability sample, information about the response rates for them will help guide our future planning efforts.

Section III – Phase 2 Testing: Collection Protocol Testing

The primary goal of this phase of testing will be to test collection of the new data elements from a selection of occupations from each respondent. In addition, the BLS will refine the collection protocols and aides based on an analysis of the test results. During this phase of testing, BLS plans to continue learning about the methods for computing estimates, analyzing the collected data, and validating future estimates.

Establishment Identification: The specific establishments to be contacted during Phase 2 testing will be identified from the same frame of establishments as used in Phase 1 testing although these establishments will be selected from different geographic areas. This list of establishments will exclude all establishments currently in any active NCS sample,

Test Size/Geographic Areas/Industry Coverage: This phase of testing will be conducted in two OMB defined separate geographic areas at two separate times. The specific geographic test areas will be chosen to maximize the distribution of establishments across the industries while controlling federal travel costs. By maximizing the industry mix of the establishments in this phase of testing, the BLS goal is to collect data for as broad a range of occupations as possible given the limited test size. This test will consist of 225 – 250 establishments with half of the units selected from each of the two geographic areas. BLS plans to include as broad mix of industries in the test establishments as possible.

Data Elements: The field economists will collect all data elements in Appendix A., plus NCS data needed to obtain a probability selection of occupations (PSO). The extra NCS data elements include SOC codes, Time/Incentive, Union/Non-union, and Work Level. The NCS data will be collected using standard NCS procedures and processes.

Procedures Development: The BLS will develop the Phase 2 collection procedures by making the necessary changes identified as a result of Phase 1 testing and input from SSA while adding procedures for any new data elements to be collected. The procedures will be documented in written materials that will be provided to the testers, SSA, and others as appropriate. Before use in testing, the procedures will be reviewed carefully by the BLS Cognitive Research Lab to ensure that the processes and terminology are appropriate for establishment data collection efforts based on generally accepted behavioral science practices.

Collection Tools: For this test, the economists will collect the data by taking structured notes using pen and paper. The collected data will be captured in an electronic format for preliminary review and analysis purposes after the interviews are conducted and may be captured using a prototype data capture system. The structured collection notes will be used to continue preparing requirements for data capture systems to be used in future phases of testing.

Data Review: During this phase, the BLS will test data review approaches, make recommendations on the review approaches for Phase 3 testing, and continue developing and implementing collection and review tools.

Training: Field Economists assigned to this phase of testing who did not participate in Phase 1 testing will receive initial training using distance education methods (webinar, videoconference and teleconference along with self study activities). At the beginning of the test phase, all test participants will receive in person classroom training covering test content and procedures. Throughout the test period, test participants will also participate in calibration, training and exercises to help ensure consistent coding of similar situations regardless of who is doing the coding.

Estimation Methods: For this Phase 2 of testing, BLS will calculate some preliminary unweighted aggregates using the collected test data. These aggregate tabulations will be used for preliminary analysis and initial estimate validation purposes only. During this phase, the BLS will test analysis and validation approaches, processes and tools, make recommendations on the validation approaches and tools for Phase 3 testing, and continue developing and implementing processes to support validation activities.

In summary, the test activities for Phase 2 will be evaluated by debriefing the employees involved in conducting the test collection efforts periodically during the test period and at the end of the test period and by reviewing comments and notes about the test collection activities recorded during the interviews

Section IV – Phase 3 Testing: Broad Scale Testing

The primary goal of this phase of testing will be to test the BLS ability to select a sample of occupations within each establishment, collect the new data elements needed by SSA, and collect other NCS data elements that are of research interest to SSA such as wages and job leveling information. In addition, the BLS will test the feasibility of collecting the data needed by SSA at the same time it is collecting all the NCS data elements needed to produce the Employment Cost Index, Employer Costs for Employee Compensation, and various benefits products. During this phase of testing, BLS plans to continue learning about the methods for computing estimates, analyzing the collected data, and validating future estimates and may generate some estimates for broad occupational groups and a few specific occupations which appear frequently in the collected establishments.

Test Size/Geographic Areas/Industry Coverage: This phase of testing will be conducted concurrently in six OMB defined separate geographic areas, one in each BLS region. The specific geographic test areas will be chosen to maximize the distribution of establishments across the industries, maximize the mix of city sizes, and control federal travel costs. By maximizing the industry mix of the establishments in this phase of testing, the BLS goal is to collect data for as broad a range of occupations as possible given the limited test size. By selecting a mix of metropolitan and non-metropolitan areas, the BLS goal is to ensure that the tests better reflect the full geography of the country. The test size for this test will be 700 - 800 establishments with approximately one-sixth of the units selected in each of the test areas for this phase of testing.

Establishment Identification: The specific establishments to be contacted during this phase of testing will be identified from the same frame of establishments as used in Phase 1 and 2 testing although these establishments will be selected from different geographic areas. The specific establishments will be identified in a manner to ensure that we test collection across a broad range of industries and occupations.

Data Elements: The field economists will collect all data elements in Appendix A., plus NCS data needed to obtain a probability selection of occupations (PSO), and other NCS data elements needed by SSA for research purposes such as wages and supervisory/non-supervisory status. For a few establishments, the field economists may attempt to collect all data elements in Appendix A as well as all current NCS data elements.

Procedures Development: The BLS will develop the Phase 3 collection procedures by making the necessary changes identified as a result of Phase 2 testing and input from SSA while adding procedures for any new data elements to be collected. The procedures will be documented in written materials that will be provided to the testers, SSA, and others as appropriate. Before use in testing, the procedures will be reviewed carefully by the BLS Cognitive Research Lab to ensure that the processes and terminology are appropriate for establishment data collection efforts based on generally accepted behavioral science practices. Feedback from field economists and test observers will be used to update the collection procedures before any future use.

Collection Tools: During this phase of testing, the field economists will capture the collected data on structured interview forms during the data collection contacts. They will enter the collected data into a prototype data capture system for preliminary review and analysis purposes during or after the interviews are conducted. The structured collection notes will be used to continue preparing requirements for a more refined data capture system to be used in future efforts.

Data Review: During this phase, the BLS will implement review strategies for the collected data based on Phase 1 and 2 testing and will review all collected data using these strategies to get information on the consistency and accuracy of the collected microdata.

Training: At the beginning of this test phase, participants will receive training covering test content and procedures, either in person or using distance education methods (webinar, videoconference and/or teleconference). During the testing period, test participants will participate in additional training sessions using a combination of classroom and distance education methods as needed to clarify collection and coding issues. Throughout the test period, test participants will also participate in calibration, training and exercises to help ensure consistent coding of similar situations regardless of who is doing the coding.

Estimation Methods: For this Phase 3 of testing, BLS will calculate some preliminary unweighted aggregates using the collected test data. These aggregate tabulations will be used for preliminary analysis and initial estimate validation purposes and some may be released as examples of potential future BLS estimates. During this phase, the BLS will implement validation approaches, processes, and tools determined to work most effectively during Phase 2 testing. The BLS will analyze and validate the preliminary unweighted tabulations using these validation processes and tools and assesses the reasonableness of the results against expectations and test objectives. Additionally, the BLS will review the test results to see if the analysis-validation results have improved because of further refinements of our validation processes during Phase 2 testing.

At the conclusion of Phase 3 testing, BLS will prepare a summary report that documents the results of all FY 2013 testing with recommendations for the future. These recommendations will address the possible need for additional testing in FY 2014, and to the extent the test data allow, revisions to the survey data elements, including possible disaggregation of occupation classifications.

If you have any questions about this nonsubstantive change request, please contact Paul Carney at 202-691-5180 or e-mail at [email protected] or Hilery Simpson at 202-691-5184 or e-mail at [email protected].

Appendix A. Data Elements Table

These data elements are grouped as Physical Demands, and Environmental Conditions, and Specific Vocational Preparation. Definitions for most of the terms used in this Appendix can be found in the Dictionary of Occupational Titles: www.occupationalinfo.org.

Definitions of coding:

• Not present (none or never)

• Occasionally present (up to 1/3 of the time)

• Frequently present (1/3 up to 2/3 of the time)

• Constantly present (2/3 or more of the time)

Data Elements, Descriptions, and Measurements

Element	What We Need	Measurement
Physical Demands
Standing/Walking	How many hours does the worker stand/walk (combined) in a typical 8 hour day?	In an 8 hour day:  2 hours or less  Greater than 2 hours to 6 hours  Greater than 6 hrs to 8 hrs
Sitting	How many hours does the typical worker sit in a typical 8 hour day (should add up to 8 hours when combined to stand/walk)?	In an 8 hour day:  6 hours or less  Greater than 6 hrs to 8 hrs
Alternate Sit/Stand	Does this job offer the typical worker the opportunity to alternate sitting and standing at will throughout the day?	 Yes  No
Lifting/Carrying - Strength	We need to know the most weight the worker typically has to lift at each frequency.	Occasionally:  Less than or equal to 10 lbs  greater than 10 lbs to 20 lbs  Greater than 20 lbs to 50 lbs  Greater than 50 lbs to 100 lbs  Greater than 100 lbs Frequently:  Less than or equal to 10 lbs  Greater than 10 lbs to 25 lbs  Greater than 25 lbs to 50 lbs  Greater than 50 lbs Constantly: • No constant lifting required • Negligible amount to 10 lbs • Greater than 10 lbs to 20 lbs • Greater than 20 lbs
Pushing/pulling (Unilateral or bilateral) Arm-hand controls Foot-leg controls	We need to know: 1. The frequency of pushing and pulling required for the job, 2. Whether it is with the upper or lower extremities, 3. Whether it is unilateral or bilateral. 4. If upper extremity push/pull is required, we need to know whether the worker has to operate hand-arm controls, 5. If lower extremity push/pull is required does it require the operation of foot-leg controls. Note: The operation of hand/arm controls should also be reflected in reaching/handling/fingering as appropriate.	• Constantly  Frequently • Occasionally • Never
Climbing ramps/stairs	With what frequency does the job require the worker to climb ramps/stairs (combined)?	 Constantly  Frequently  Occasionally  Never
Climbing ladders/ropes/scaffolds	With what frequency does the job require the worker to climb ladders/ropes/scaffolds (combined)?	 Constantly  Frequently  Occasionally  Never
Stooping	With what frequency does the job require the worker to stoop?	• Constantly • Frequently • Occasionally • Never
Kneeling	With what frequency does the job require the worker to kneel?	• Constantly • Frequently • Occasionally • Never
Crouching	With what frequency does the job require the worker to crouch?	• Constantly • Frequently • Occasionally • Never
Crawling	With what frequency does the job require the worker to crawl?	• Constantly • Frequently • Occasionally • Never
Reaching	We need to know: 1. The frequency of reaching the job requires, 2. Whether it is unilateral or bilateral, 3. Whether the reaching is above the shoulders or shoulder level and below 4. Whether the reaching is in front or laterally.	• Constantly • Frequently • Occasionally • Never
Handling (gross manipulation)	We need to know: 1. The frequency of handling the job requires, and 2. Whether it is unilateral or bilateral.	• Constantly • Frequently • Occasionally • Never
Fingering (fine manipulation)	We need to know: 1. The frequency of fingering the job requires, and 2. Whether it is unilateral or bilateral.	• Constantly • Frequently • Occasionally • Never
Keyboarding	We need to know: 1. The frequency of keyboarding the job requires, and 2. Whether there is a speed requirement (wpm)?	• Constantly • Frequently • Occasionally • Never  Yes  No
Near Acuity	Does the job require good near acuity?	 Yes  No
Far Acuity	Does the job require good far acuity?	 Yes  No
Field of Vision	Does the job require good field of vision?	 Yes  No
Hearing	We want to discern the level of hearing that is required for the job: Does the job require the ability to discern loud sounds such as sirens or alarms? Does the job require the ability to discern moderate sounds such as conversational speech?	 Yes  No  Yes  No
Speaking	We need to know the frequency that speaking is required	• Constantly • Frequently • Occasionally • Never
Environmental Conditions
Extreme Cold	With what frequency is the worker exposed to extreme cold?	• Constantly • Frequently • Occasionally • Never
Extreme Heat	With what frequency is the worker exposed to extreme heat?	• Constantly • Frequently • Occasionally • Never
Wetness	With what frequency is the worker exposed to wetness?	• Constantly • Frequently • Occasionally • Never
Humidity	With what frequency is the worker exposed to humidity?	Constantly • Frequently • Occasionally • Never
Noise	What is the noise intensity level in the environment where this worker typically performs the job?	1 = Very Quiet (deep sea diving) 2 = Quiet (private office) 3 = Moderate (department store) 4 = Loud (heavy traffic) 5 = Very loud (Rock Concert front row)
Vibration	With what frequency is the worker exposed to vibrations?	• Constantly • Frequently • Occasionally • Never
Fumes, odors, dusts, gases, poor ventilation	With what (combined) frequency is the worker exposed to fumes, odors, gases, or poor ventilation?	• Constantly • Frequently • Occasionally • Never
Proximity to moving mechanical parts	With what frequency is the worker exposed to moving mechanical parts?	• Constantly • Frequently • Occasionally • Never
Working in high, exposed places	With what frequency is the worker required to be in high, exposed places?	• Constantly • Frequently • Occasionally • Never
Exposure to Toxic, Caustic Chemicals	With what frequency is the worker exposed to toxic, caustic chemicals?	• Constantly • Frequently • Occasionally • Never

Specific Vocational Preparation
Specific Vocational Preparation (SVP)	Period of time required by a typical worker to learn the techniques, acquire the information, and develop the facility needed for average performance in the specific job worker situation for each occupation?	Time: Short demonstration only Anything beyond short demonstration up to and including 1 month Over 1 month up to and including 3 months Over 3 months up to and including 6 months Over 6 months up to and including 1 year Over 1 year up to and including 2 years Over 2 years up to and including 4 years Over 4 years up to and including 10 years Over 10 years

Appendix B.  Tentative Timeline for FY 2013 Test Survey Tasks

Note that all dates are subject to change and a more complete timeline will be developed for each phase of testing prior to the start of that phase of the project.

Task Name	Tentative Dates
Manage Pilot Project	Full Fiscal Year
Learn about SSA Data Requirements	Full Fiscal Year
Determine goals of survey collection and outputs including data elements needed	Early Fall 2012
Prepare for Phase 1 Testing	Early Fall 2012
Conduct Phase 1 Testing	November – December 2012
Prepare for Phase 2 Testing	January 2013
Conduct Phase 2 Testing - Location A	January-February 2013
Conduct Phase 2 Testing - Location B	February – March 2013
Identify Needed Changes to Methods/Procedures/Tools/Capture Process/etc.	Late March 2013
Prepare for Phase 3 Testing	April 2013
Conduct Phase 3 Testing	Late April – July 2013
Evaluate Results of FY 2013 Testing	July – August 2013
Document Test Results and Recommendations for FY 2014	August – September 2013
Disseminate Conclusions & Results to SSA and Public	September 2013
Conduct Research	Full Fiscal Year

12