Note to Reviewer

April 14, 2014

MEMORANDUM FOR: REVIEWER of 1220-0188

FROM: Bryandt Rose Dickerson, Chief

Division of Compensation Data Analysis and Planning

Office of Compensation and Working Conditions

Bureau of Labor Statistics

SUBJECT: Non-substantive Change Request for the
Pre-Production Occupational Requirements Survey (ORS) –

Cognitive Element Feasibility Test and Job Observation Test

Clearance is being sought for the Bureau of Labor Statistics’ (BLS) Occupational Requirements Survey (ORS) program for the testing of revised cognitive questions and testing of occupational requirements collection by job observations in cooperation with the Social Security Administration (SSA).

Based on pre-production collection information, the questions and wording for the cognitive elements have been revised to clarify the underlying concepts being measured. The testing of these revised questions in FY 2015 will provide further information on respondents’ perception of the wording.

The job observation pre-test is in response to Federal Register Notice public comments and an external researcher’s recommendations on achieving high levels of reliability, validity, and accuracy given the ORS’s parameters. The FY 2015 test is designed to provide information to ORS on whether systematic differences for physical and environmental elements may exist between the interview collection method in the ORS design and directly observed job collection method.

The ORS Cognitive Element Feasibility Test

ORS will collect data from 66 establishments. This cognitive test will consist of establishments from prior ORS feasibility tests in the following cities: Nashville, Tennessee; Providence, Rhode Island; Cincinnati, Ohio; Kansas City, Missouri; Pittsburgh, Pennsylvania; and Portland, Oregon. These establishments will be selected based on pre-determined industry groups. For each city, ORS plans to collect cognitive data elements from eleven establishments (nine from private industry, two from state and local governments).  Sample weights will not be assigned to the selected establishments or occupations in the sample. The cognitive element test respondent burden is expected to average 30 minutes per establishment. The revised questions will be verbally asked to respondents.

This test will add an additional respondent burden hours of 33 hours.

ORS Cognitive Element Test Respondent Burden Hours
Private burden hours all establishments	State and local burden hours all establishments	Total burden hours all establishments
27	6	33

ORS Job Observation Pre-test

This pre-test is in response to both Federal Register Notice public comments and an external researcher’s recommended testing and validation of ORS survey design. The FY 2015 job observation pre-test will provide information to ORS for analysis of whether systematic differences for physical and environmental elements may exist between ORS interview collection methods and a direct observation method.

The ORS job observation pre-test sample will consist of 560 sampled establishments from existing ORS Pre-production test establishments (476 from private industry, and 84 from State and local governments) where the pre-production collected jobs included targeted occupations. The targeted occupations were determined by ORS using a list of occupations SSA identified as more commonly encountered in disability cases. An independent pre-test sample will be selected using systematic simple random sampling with an equal probability of selection within each of the targeted occupations. The sample frame units will be ordered by a combination of geography, industry and size class to ensure a good distribution of available establishments within each of the targeted occupations. Sample weights will be assigned to each of the selected quotes in the sample to represent the entire frame.

Re-contacting sample establishments to explain the observation test and obtain appointments is expected to take on average 5 minutes of respondent’s time. The ORS job observation pre-test respondent burden is expected to average 85 minutes per establishment. Two field economists will go to each establishment for the pre-test, observe the same occupation, and independently write up their observations.

This pre-test is oversampled as ORS expects a high non-response rate (approximately 50 percent), due to a quick test turnaround and the possibility of firms being reluctant to allow observations.

This pre-test will add an additional respondent burden hours of 444 hours (based on the high non-response expected of 50 percent).

The ORS Job Observation Test of physical and environmental elements is to measure the coding agreement/differences between an observed job and an unobserved job for the tested elements. ORS will test if job elements are coded differently for observed and unobserved occupations.

Basic Collection Methodology:

For setting up appointments BLS Field Economist will introduce themselves and explain the reason for this follow up contact following the attached script (Appendix A.)

• ORS field economists will get an appointment for a day and time when an incumbent in the selected job is doing their normal and primary duties on a typical work day/schedule (usual and customary) for that occupation. Explaining they will need at least an hour to observe the job.

• Two field economist will observe each job being performed at same time and independently code and record their observations on the physical and environmental elements measured.

• The data from these observations and the supporting documentation will be analyzed and evaluated by national office staff against the previously collected ORS data for these jobs ORS obtained from earlier visits when interviewing company officials and using written position descriptions.

• Data discrepancies from this test survey of selected jobs will be measure and there will be a written study of these findings.

ORS Job Observation Pre-test Appointment Contact Respondent Burden Hours
Private burden hours all establishments	State and local burden hours all establishments	Total burden hours all establishments
40	7	47
ORS Job Observation Pre-test Collection Respondent Burden Hours
Private burden hours all establishments	State and local burden hours all establishments	Total burden hours all establishments
337	60	397

ORS Job Observation Pre-test Total Burden Hours
Private burden hours all establishments	State and local burden hours all establishments	Total burden hours all establishments
377	67	444

For both tests, the total number of additional respondent burden hours is 477.

Further ORS Pre-Production Testing Respondent Burden Hours
Private burden hours all establishments	State and local burden hours all establishments	Total burden hours all establishments
404	73	477

If you have any questions about this non-substantive change request, please contact Paul Carney at 202-691-5180 or e-mail at [email protected], or Bryandt Dickerson at 202-691-7744 or e-mail at [email protected].

Attachment A. Appointment script for explaining job observation pre-test to respondents

Attachment B. Revised cognitive questions

Attachment C. BLS staff note taking aid for job observation pre-test

4