Note to Reviewer

February 13, 2020

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Erica Yu Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of materials for the Pre-testing of ORS Incumbent Contact Information Collection Study

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study is 36 hours.

If there are any questions regarding this project, please contact Erica Yu at 202-691-7924.

1. Introduction and Purpose

To address ongoing interest in the validity and reliability of data collected by the Occupational Requirements Survey (ORS) program, the Bureau of Labor Statistics is exploring the potential for collecting ORS data elements from incumbents. Collection of data from incumbents is a complex undertaking that requires two stages of cooperation (establishment and incumbent) and identification of the appropriate mode of data collection from incumbents. The purpose of this study is to collect preliminary evidence about incumbent contact procedures and rates to inform a decision as to whether to conduct a subsequent large-scale incumbent test, and if so how it should be designed.

2. Research Design

This study is designed to be implemented in two stages: contact a previous ORS respondent to collect incumbent contact information for a research survey, and then contact the incumbent to conduct a brief survey of occupational requirements.

For the purposes of this feasibility study, OCWC management determined that collecting incumbent contact information from 20 establishments would provide sufficient evidence for determining whether to proceed with a large-scale test. Because a study of this size is not representative of the ORS population, we are not expecting the contact and response rates to be generalizable to the ORS population and will not rely on them for evaluation. Instead, the primary measures of interest from this study will be qualitative assessments of whether contact and response procedures (including mode of incumbent contact and interview) are feasible on a larger scale.

Sample

A sample of 150 establishments was drawn from the 721 ORS sample group for this study. This group, which will likely not be contacted for ORS for the remainder of the 5 year collection wave, was chosen to avoid over-burdening ongoing ORS respondents¹. To determine this sample size, we considered expected response rate and concerns about minimizing respondent burden and preserving on-going collection relationships. For reference, we considered the response rates of other BLS surveys that have re-contacted employers². Given the differences in survey context, mode of contact, and differences in resources available for non-response follow-up, we used a hypothetical response rate of 15%, resulting in a minimum sample size of 134 establishments to secure cooperation from 20 establishments. We increased the sample size to 150 to enable the Office of Field Operations to review the final sample and remove any additional establishments as needed to minimize re-contact of sensitive reporters.

All establishments in the frame are in scope for sampling, including all ownership and industry types. To account for the likely variation in incumbent contact rates and procedures by establishment size, the sample is stratified by size class with three equal-sized strata: small (1-49 employees), medium (50-249), and large (250 or more).

Incumbent Selection

Given that any large-scale incumbent test will likely target certain occupations (SOCs) and require contact information for incumbents specifically in those occupations, this study has been designed with a second stage of selection to approximate the complexity of such a survey request. The selection of SOCs at an establishment is driven by a random process rather than allowing the interviewer or the respondent to choose a SOC, which mimics the likely design of any future large-scale test. After SOCs are identified, the ORS respondent will be asked to provide contact information for any two incumbents in those SOCs (four total incumbents or fewer if a SOC does not have two current incumbents).

Mode

Data will be collected from both employers and incumbents by interviewers over the telephone. If possible logistically, incumbents will also be offered an option for the interviewer to conduct a personal visit to the workplace.

Contact Procedures

Researchers will manage contact of respondents across interviewers so as not to contact more respondents than needed to complete 20 employer interviews to collect incumbent contact information. At the start of each week, interviewers will receive a new batch of establishments and will be required to contact each establishment at least one time. Interviewers will attempt to call those establishments until they successfully make contact and either gain cooperation or receive a refusal. There will be no refusal conversion. Interviewers will begin the next batch of establishments the next week.

When requesting incumbent contact information from the employer, interviewers will emphasize the follow-up nature of the study and that the request is for 10 minutes of the incumbent’s time. Interviewers will attempt to collect both telephone number and email addresses for each incumbent, to increase the likelihood of reaching the incumbent. Any collected contact information will be saved to a secure network location accessed only by the interviewers and researchers. Scripts of these contacts are attached as Appendix A.

When contacting the incumbent for the survey, interviewers will first contact incumbents by telephone and emphasize that the employer has already completed its portion of the survey and that the request is for 10 minutes of the incumbent’s time. Subsequent contact attempts may be by telephone or email. Scripts of these procedures are attached as Appendix B.

Survey

Items for the survey were selected to represent a range of content likely to be on a large-scale incumbent test survey. Questions were written to match the questions asked by interviewers in production as closely as possible. The full questionnaire is attached in Appendix C. Interviewers will administer either an electronic or paper version of the survey and all data will be stored in secure network locations accessed only by the interviewers and researchers.

3. Participants

Up to 150 individuals who previously responded to ORS will be contacted to provide contact information for incumbents. Up to four incumbents working in randomly selected occupations from each establishment will be interviewed.

4. Burden Hours

The total burden hours for this study will be 36.

Up to 150 participants will be contacted. The introduction to this study is expected to take no more than 5 minutes per establishment, for a total of 12.5 burden hours (150 participants x 0.0833 hours). Of those participants, we expect to collect incumbent contact information from up to 20 establishments, which takes no more than 10 minutes including time to gather payroll lists or other records, for a total of 3.3 burden hours (20 participants x .1667 hours). Up to 80 incumbent participants will be contacted (up to four from each establishment), with recruitment taking no more than 5 minutes and interviews no more than 10 minutes, for a total of 6.7 burden hours (80 participants x 0.0833 hours) and 13.3 burden hours (80 participants x .1667 hours). These calculations are described in the table below. In total, this study may require up to 36 burden hours should survey cooperation be secured from up to four incumbents from each establishment.

Type of participant	Activity	Number of participants	Time spent per participant	Total time
Employers	Recruitment	150	5 min each	12.5 hours
	Collection of incumbent contact information	20	10 min each	3.3 hours
Incumbents	Recruitment	80	5 min each	6.7 hours
	Interview	80	10 min each	13.3 hours
				35.83 hours

5. Payment to Participants

Participants in this study will not receive any payment.

6. Data Confidentiality

During the survey request, participants will be informed as to the voluntary nature of the study and assured that the data will be held confidentially. Participants will also be informed that the study will be used for internal purposes to improve the quality of data collected by the Occupational Requirements Survey. Interviewers will read aloud the following statement at the start of the survey:

The purpose of the survey is for internal research only to improve our data collection efforts. The BLS will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act (44 U.S.C. 3572) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. This collection is authorized under O.M.B. Number1220-0141, which expires March 31^st 2021. We estimate that it will take an average of 10 minutes total time to participate in this voluntary survey.

Appendices

Appendix A: Scripts for requesting incumbent contact information

Appendix B: Scripts for requesting incumbent participation in the survey

Appendix C: Survey

Appendix A: Scripts for requesting incumbent contact information

Interviewers will be given the following outlines to use when contacting respondents:

Feel free to use the wording or bullet points when talking to your respondents. The language highlighted in bold is what you are required to say.

• Introduce self

• Describe survey and past participation

• Tell them why you’re contacting them

• Answer any questions they have about the follow-up study

  • In-person visit or telephone call

  • Participation is voluntary

  • Data held confidentially and used only for statistical purposes

  • Purpose is to help us understand how to improve data quality for the survey

  • Questions are about education, physical and cognitive demands of the job, and work environment

  • Survey should only take 10 minutes of the incumbent’s time; no additional contact with the incumbent after participating

  • Once they give the incumbent’s information, there are no further questions for them to answer

• Identify incumbents in the selected SOCs and collect incumbent contact information

  • Two job titles will be selected

  • Any two incumbents with each job title are eligible – there are no restrictions other than job title

Sample introduction with previous production respondent:

My name is __{name}__ and I am [an economist/a statistician] with the U.S. Department of Labor. We are conducting a follow-up study of occupational requirements for jobs in today’s economy. You participated in our survey in __{month}__ of __{year}__, and now I’m calling to speak with a few of your employees to ask follow-up questions about their jobs. I will need about 10 minutes of their time. Can you provide contact information for them?

Sample introduction with new respondent, in cases where the previous respondent changed jobs or is no longer at the establishment:

My name is __{name}__ and I am [an economist/a statistician] with the U.S. Department of Labor. We are conducting a follow-up study of occupational requirements for jobs in today’s economy. [Previous contact name] participated in our survey in __{month}__ of __{year}__, and now I’m calling to follow-up on that survey.

Let me give you a bit of background first. I’m calling for the Occupational Requirements Survey, which is a national survey about jobs and their education, cognitive, physical, and environmental requirements. Participation is voluntary. All data we collect are held confidentially and used only for statistical purposes.

What we need is to speak with a few of your employees to ask follow-up questions about their jobs. I will need about 10 minutes of their time. Would you be willing to help us out and let us ask them to participate?

If respondent says no and gives a reason, thank them and record the reason.

If respondent says no and doesn’t give a reason:

Ok, I understand and we will not contact anyone else at your business. For our records, may I ask the reason why you are not willing to let us ask them to participate?

If says yes and schedule has two SOCs:

Great! I have here that you have a job title at your company called something like “__{job title}__”, is that right? [What are the names of two of your employees/What is the name of the employee] with that job title? And what is their direct phone number? And direct email address?

And now, last one, I have here that you also have a job title at your company called something like “___{job title}__”, is that right? [What are the names of two of your employees/What is the name of the employee] with that job title? And what is their direct phone number? And direct email address?

Great, thanks very much for that.

If says yes and schedule has only one SOC:

Great! I have here that you have a job title at your company called something like “__{job title}__”, is that right? [What are the names of two of your employees/What is the name of the employee] with that job title? And what is their direct phone number? And direct email address?

Great, thanks very much for that.

If a sampled SOC is no longer present at the establishment, ask the respondent for the job title most similar to the originally sampled SOC.

If incumbent does not have a direct phone number or email address:

What is the best way to reach this employee directly?

Appendix B: Scripts for requesting incumbent participation in the survey

Interviewers will be given the following outlines for contacting incumbents:

Feel free to use the wording or bullet points when talking to your respondents. The language highlighted in bold is what we are required to say.

• Before calling, check to see if a personal visit is possible for this case (these cases will be clearly marked).

• Introduce self

• Describe survey and past participation

• Tell them why you’re contacting them

• Answer any questions they have about the follow-up study

  • Name of the ORS respondent who provided the incumbent’s contact information

  • Participation is voluntary

  • Data held confidentially and used only for statistical purposes

  • Purpose is to help us understand how to improve data quality for the survey

  • Questions are about education, physical and cognitive demands of the job, and work environment; no identifying details about you

  • Employer’s portion was about requirements for jobs at the business; no details about particular employees

  • Survey should take 10 minutes and can be done on work time; no additional contact after participating

• The wording about survey participation that is required to be read, including OMB control number, is in the survey introduction.

Telephone Introduction:

My name is __{name}__ and I am [an economist/a statistician] with the U.S. Department of Labor. We are conducting a follow-up study of occupational requirements for jobs in today’s economy. Your employer already completed their portion of the survey and now I need about 10 minutes of your time to collect information from your perspective as the person actually doing your job. Would you be willing to help us out and answer questions?

If says no and gives a reason, thank them, ask about web, and record the reason.

Ok, I understand and we will not contact you again. For our records, would you have been willing to complete the survey online if we had offered that option?

If says no and doesn’t give a reason, accept refusal and ask about web and the reason:

Ok, I understand and we will not contact you again. For our records, would you have been willing to complete the survey online if we had offered that option? And may I ask the reason why you are not willing to participate?

If says yes:

Great! Can we go ahead and get started now?

If says yes, then complete interview by telephone

If says no:

Can we set-up an appointment for me to call you later this week?

If says no or says yes but sounds like s/he may not follow through AND personal visit is possible:

If it would work better for you, I could come out to do the interview in-person.

If sounds like a soft refusal (“too busy right now”), then accept refusal, ask about web:

Ok, I understand and we will not contact you again. For our records, would you have been willing to complete the survey online if we had offered that option?

Email Introduction:

Dear __{Incumbent name}__,

I am an economist with the U.S. Department of Labor. We are conducting a follow-up study of occupational requirements for jobs in today’s economy.

Your employer already completed their portion of the survey and now I need about 10 minutes of your time to collect information from your perspective as the person actually doing your job. Would you be willing to help us out and answer questions?

The next step is to set up an interview appointment. Please let me know your availability and we’ll find a time that works.

I am happy to answer any questions. I look forward to speaking with you.

__{FE name}__

__{FE phone number}__

BLS Logo

Appendix C: Survey

Interviewers should read aloud the full question and response options as scripted; blue text indicates interviewer instructions or help material that may be read aloud at the interviewer’s discretion. Interviewers will make collection notes about any substantial deviations from script.

Mandatory confidentiality language to be read aloud before collecting information from the incumbent:

The purpose of the survey is for internal research only to improve our data collection efforts. The BLS will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act (44 U.S.C. 3572) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. This collection is authorized under O.M.B. Number 1220-0141, which expires March 31st 2021. We estimate that it will take an average of 10 minutes total time to participate in this voluntary survey.

I will start by asking you a few questions about what it’s like to do your job.

Please consider only critical job duties, which are the main tasks you are required to perform.

Include all machinery and equipment you typically use.

Include all worksites where you typically work.

But do NOT include any work that you do that happens only once in a while or is not a required part of performing your job.

How many hours per day or shift do you usually work?

[open end, validate between 0 - 48]

Are you permitted to work from home or telework on a regular basis?

Yes

No

Not sure

What is the minimum level of education required for your job? The level required by your employer for your job may be different from your own level of education.

No minimum education required

High school diploma

Vocational high school diploma

Associate’s degree

Vocational associate’s degree

Bachelor’s degree

Master's degree

Professional degree (for example, M.D., D.M.D., J.D., Pharm.D.)

Doctorate (for example, Ph.D., Ed. D.)

Not Sure

On a typical workday, are supervisors or lead workers physically present in your work area and able to monitor your work?

Yes

No

Not sure

As part of your critical job duties, are you required to work with the general public, that is, anyone other than co-workers?

Yes

No

Not sure

As part of your critical job duties, must you be able to hear and understand others speak in person?

Yes

No

Not sure

Do you typically have the ability to choose whether you sit or stand while performing your critical job duties?

Yes

No

How many hours of a typical day or shift do you spend sitting or lying down?

[open end, validate between 0 – 48]

As part of your critical job duties, do you typically drive or operate motorized vehicles?

Yes

No

Not sure

As part of your critical job duties, do you use one or both of your feet or legs to move pedals, levers, or other controls on equipment?

Yes, one leg and/or foot only (go to follow up)

Yes, both legs and/or feet (go to follow up)

No

Not sure (go to follow up)

(If yes/Not sure) On a typical workday, how much time do you spend using at least one of your feet or legs to move pedals, levers, or other controls on equipment?

Very rarely (less than 10 minutes of a typical 8-hour workday)

Up to 33% of the time (up to 2.5 hours of a typical 8-hour workday)

Between 33% to 66% of the time (between 2.5 – 5.5 hours of a typical 8-hour workday)

More than 66% of the time (5.5 hours or more of a typical 8-hour workday)

Not sure

As part of your critical job duties, do you typically stoop, kneel, crouch, or crawl?

Here are a few examples of what I mean:

-Stooping is bending down like a mechanic to reach under the hood of a car

-Kneeling is getting down on one or two knees like a floor installer laying tile

-Crouching is getting low like a baseball umpire during a pitch

-Crawling is moving on all fours like a concrete worker smoothing the surface of a sidewalk

Yes

No

Not sure

Which of the following best describes the noise level of your typical work environment?

Quiet (For example - art museum)

Moderate (For example - grocery store)

Loud (For example - heavy traffic)

Very loud (For example - jackhammer)

Not sure

As part of your critical job duties, do you typically spend time outdoors?

Yes (go to follow-up)

No

Not sure (go to follow-up)

(If yes/Not sure) On a typical workday, how much time do you spend outdoors?

Very rarely (less than 10 minutes of a typical 8-hour workday)

Up to 33% of the time (up to 2.5 hours of a typical 8-hour workday)

Between 33% to 66% of the time (between 2.5 – 5.5 hours of a typical 8-hour workday)

More than 66% of the time (5.5 hours or more of a typical 8-hour workday)

Not sure

As part of your critical job duties, do you typically come into contact with water or other liquids excluding weather, such as when washing hands?

Yes (go to follow-up)

No

Not sure (go to follow-up)

(If yes/Not sure) On a typical workday, how much time do you spend in contact with water or other liquids?

Very rarely (less than 10 minutes of a typical 8-hour workday)

Up to 33% of the time (up to 2.5 hours of a typical 8-hour workday)

Between 33% to 66% of the time (between 2.5 – 5.5 hours of a typical 8-hour workday)

More than 66% of the time (5.5 hours or more of a typical 8-hour workday)

Not sure

Thank you. Now we have a few final questions about what it was like to participate in this survey. You will NOT be contacted again but your responses now will help us improve this survey in the future.

Would you have preferred to be contacted by:

Phone

Email

Letter in the mail

Would you have preferred to answer these questions:

Over the phone, talking to an interviewer

In person, talking to an interviewer

Online

Paper form, submitted by mail

Would you have been willing to provide more detailed information about the requirements of your job for this survey?

Yes

No

Would you have preferred us to ask your supervisor for permission before contacting you?

Yes

No

Thank you! Do you have any questions for me about this survey?

I can also provide contact information so you can get in touch with us later if any questions come up - [email protected].

1 We also applied rules at the time of sampling to exclude establishments known to be sensitive or that otherwise should not be contacted for participation in a research study in order to preserve ongoing collection relationships.

2 Previous ORS jobs observation tests achieved 60% (2016), 41% (2017), and 38% (2018) establishment response rates. The Survey of Employer Provided Training achieved a 74% (1995) response rate.

Page 17 of 17