Note to Reviewer - ORS Cognitive Questions

July 6, 2017

NOTE TO THE REVIEWER OF:	OMB CLEARANCE #1220-0141 “Cognitive and Psychological Research”
FROM:	Bill Mockovak Survey Methodologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Testing of ORS Cognitive Questions

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141, “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the research, and the materials to be used in the research.

The maximum number of burden hours is estimated to be 15.

If there are any questions regarding this project, please contact Bill Mockovak at (202) 691-7414.

Attachments

I. Introduction and Purpose

The Occupational Requirements Survey (ORS) is a survey conducted by the Bureau of Labor Statistics' (BLS) National Compensation Survey (NCS) program. The ORS gathers job-related information regarding physical demands, environmental conditions, mental and cognitive demands, and vocational preparation requirements. The focus of the cognitive testing described in this clearance is on questions that measure the mental and cognitive demands of jobs.

ORS data are collected to meet the Social Security Administration’s (SSA) needs for disability eligibility determination. The cognitive domains of interest include decision making, work review, work pace, control of work pace, adaptability, task variety, and interaction with people.

A BLS/SSA working group, the Mental/Cognitive Job Requirements Working Group, (MCWG) was formally chartered in January 2017, to review and revise existing ORS cognitive elements to better meet the Social Security Administration’s (SSA) needs for disability eligibility determination. Specifically, the objectives were to (a) evaluate existing ORS data elements to determine the extent to which they covered the key dimensions of mental and cognitive domains, (b) refine ORS questions and response options pertaining to these domains to move in the direction of greater concreteness and objective measurement, and (c) identify and develop questions for any additional cognitive domains relevant to disability claims commonly present in SSA adjudications.

The team completed its work in June 2017, and proposed a revised set of cognitive questions for ORS, which are shown in Attachment A. These questions will be included in an ORS field test expected to start in September 2017. However, prior to that test we plan to conduct some in-house and online cognitive testing to identify potential issues with the questions and better inform the development of training materials, which will be developed this summer for the fall field test.

II. Respondents

A total of thirty people will participate. Twenty online respondents will be recruited from TryMyUI, and 10 in-house participants will be recruited through the Office of Survey Method’s Research database of volunteers. Only U.S. citizens, age 30 or older, will be allowed to participate.

III. Research Design

Participants who come into the OSMR lab will be given an introduction to the study and asked to sign a consent form. They will then be asked the cognitive questions without interruption. After the questions have been asked, follow-up probes will be asked about the questions to better gauge the participant’s level of understanding.

Since the online testing sessions are limited to 20 minutes, participants will complete all the questions using a self-administered approach, and then proceed to answer some follow-up questions (see Attachment C).

IV. Burden Hours

Recruitment. There are no burden hours for recruitment of the TryMyUI participants. We estimate that it will take, on average, 12 minutes to recruit the 10 in-house participants, and 2 minutes to screen out other ineligible or disinterested individuals. With an expected recruitment rate of 40 percent, there will be 30 minutes of burden associated with screening (15 individuals x two minutes = 30 minutes), and 2 burden hours for recruitment (10 participants x 12 minutes = 120 min divided by 60 = 2 hours), for a total of 2.5 burden hours.

Data Collection. We estimate that it will take an average of 18 minutes for the online TryMyUI participants to complete their assignment, for a total of 6 burden hours (20 participants x 18 min = 360 min divided by 60 = 6 hours).

In-house sessions are expected to last an average of 40 minutes, resulting in 6.7 burden hours (10 participants x 40 min = 400 min divided by 60 = 6.7 hours).

Total burden hours are therefore 15.2, rounded to 15 hours (2.5 + 6 + 6.7 = 15.2).

V. Payment to Respondents

In-house participants will receive an incentive payment of $40. TryMyUI participants receive an incentive payment of $10 each.

V. Data Confidentiality

BLS will not provide a guarantee of confidentiality for the TryMyUI participants. In-house participants will be asked to read and sign a consent form (see Attachment C).

VI. Attachments

A - Revised ORS Cognitive Elements Items

B - Protocol for In-house Cognitive Testing

C - Consent form

D – PDF file Showing Online Protocol

Attachment A

Revised ORS Cognitive Elements Items

Introduction

Unlike some tasks you’ve completed on TryMyUI, we’re not looking for you to evaluate a website, rather we’re looking for you to answer a series of questions and give us your written and verbal feedback. Please talk aloud, giving your reactions and thoughts as you move through the pages.

This is a survey for the US Bureau of Labor Statistics. This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141 (Expiration Date: April 30, 2018). This survey will take approximately 20 minutes to complete. You are not required to respond to this survey unless it displays a currently valid OMB control number.

We will use the information you provide for statistical purposes only. Your participation is voluntary and you have the right to stop at any time. This survey is being administered by SurveyMonkey and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

We are looking for information about how respondents answer our surveys. Please take your time as you answer these questions. The information you provide will contribute to valuable research at the BLS, one of the leading federal statistical agencies.

Control of Work Pace

Which of the following most determines the pace of work? (select one)

□ Machinery, equipment, or software controls pace

□ Numerical performance targets or quotas enforced by company determine pace

□ Other persons determine pace (e.g., co-workers, supervisors, customers, members of the public)

□ Other external source (specify) ___________________

□ Self-paced by worker within general performance guidelines

Work Flow

How would you describe the work flow for this job? Would you say that it ….

□ Has few (if any) slack periods

□ Has few (if any) rush periods (or large build-ups of work)

□ Varies, with fluctuations of rush periods (or large build-ups of work) and slow periods on a daily or weekly basis

Pause Control

Can workers step away from their work area easily outside of scheduled breaks (e.g., short bathroom breaks)

□ Yes

□ No

Work Review

How frequently is work checked in the occupation?

□ More than once per day

□ Once per day

□ At least once per week, but not every day

□ Less than weekly

Are supervisors generally present in the same work area as workers?

□ Yes

□ No

Problem Solving

The next question is about “problem solving” tasks that the worker does in his/her job. Think of “problem solving” as what happens when workers are faced with a new or difficult situation which requires them to think for a while about what to do next. How often is the worker responsible for solving problems that take more than 5 minutes to find a good solution?

□ More than once per day

□ Once per day

□ At least once per week, but not every day

□ Less than weekly, but at least once per month

□ Less than monthly, including never

Personal Contacts

How often does this occupation require verbal, work-related interactions?

□ Constantly, every few minutes

□ More than once per hour, but not constantly

□ More than once per day, but not more than once per hour

□ No more than once per day, including never

What level of “people skills” does this job require?

□ Minimal/Basic

□ More than basic

Checklist Items

Are workers in this occupation:
Required to work with the general public	Yes No
Required to work around crowds	Yes No
Required to supervise others as part of job	Yes No
Permitted to work from home or telework	Yes No
Required to work outside	Yes No

Attachment B – Protocol for In-house Cognitive Testing

• Hi! Thank you for coming in today.
  
  

• [Respondent reads and signs consent form.]

• [If team member is observing] I have a colleague in the next room that will be observing and taking notes.

• Today we’re going to be evaluating questions for a new survey that collects information about the demands of occupations.

• When this survey is conducted, we collect information about jobs by speaking to supervisors, managers, or the human relations staff in a company. Today I’m going to be asking you questions about the mental or cognitive demands of a job that you supervise. For example, the questions will ask about things like what is the pace of work and what determines the pace, how closely supervised is the position, who does the worker interact with, and so on.

• Since you’ve been a supervisor, we want to ask you the questions about a job that you supervise.

• So, to begin, can you tell me the title or name of the job that you supervise?

• Title/Name of Job supervised: ___________________

• How long have you supervised this job? __________

• Were you responsible for assigning some type of an annual job performance rating?

  • Yes

  • No (if no – make sure he/she had other supervisory responsibilities such as assigning work, monitoring progress, giving feedback, etc.)

• I want to assure you that all the information we discuss today will be kept completely confidential, and you won’t be identified in any reports that we prepare. So please read and sign this consent form, and let me know if you have any questions. Notice that we want to record the session. I do that just to help me with note taking.

• Before we begin, I want to emphasize that we’re not here to evaluate you or how good a supervisor you are. We’re only interested in how well the questions work. For example, are the questions clear and easy to understand?

• If a question isn’t clear or doesn’t make sense, please tell us. You’re not going to hurt our feelings. These questions are going to be asked all around the country with thousands of people so they need to be clear and easy to understand.

• Do you have any questions before we begin? I’m going to start the tape recorder now.

• Again, these questions are being asked about the job [title/name] that you supervise. Here’s the first question.

1. Which of the following most determines the pace of work on this job? (Select one. Show card with response options)

□ Machinery, equipment, or software controls pace

□ Numerical performance targets or quotas enforced by company determine pace

□ Other persons determine pace (e.g., co-workers, supervisors, customers, members of the public)

□ Other external source (specify) ___________________

□ Self-paced by worker within general performance guidelines

2. How would you describe the work flow for this job? Would you say that it ….

□ Has few (if any) slack periods

□ Has few (if any) rush periods (or large build-ups of work)

□ Varies, with fluctuations of rush periods (or large build-ups of work) and slow periods on a daily or weekly basis

3. Can workers step away from their work area easily outside of scheduled breaks (for example, for short bathroom breaks)

□ Yes

□ No

4. How frequently is work checked in the occupation?

□ More than once per day

□ Once per day

□ At least once per week, but not every day

□ Less than weekly

5. Are supervisors generally present in the same work area as workers?

□ Yes

□ No

6. The next question is about “problem solving” tasks that the worker does in his/her job. Think of “problem solving” as what happens when workers are faced with a new or difficult situation which requires them to think for a while about what to do next. How often is the worker responsible for solving problems that take more than 5 minutes to find a good solution?

□ More than once per day

□ Once per day

□ At least once per week, but not every day

□ Less than weekly, but at least once per month

□ Less than monthly, including never

7. How often does this occupation require verbal, work-related interactions?

□ Constantly, every few minutes

□ More than once per hour, but not constantly

□ More than once per day, but not more than once per hour

□ No more than once per day, including never

8. What level of “people skills” does this job require?

□ Minimal/Basic

□ More than basic

Are workers in this occupation:
Required to work with the general public?	Yes No
Required to work around crowds?	Yes No
Required to supervise others as part of job?	Yes No
Permitted to work from home or telework?	Yes No
Required to work outside?	Yes No

• Those are all the questions. Now that you've had a chance to answer each one, I am going to go through them one more time to ask you a few additional questions to help me better understand how you interpreted the question. Here is the first question once again (show card with responses).

1. Which of the following most determines the pace of work on this job? (Select one. Show card with response options)

□ Machinery, equipment, or software controls pace

□ Numerical performance targets or quotas enforced by company determine pace

□ Other persons determine pace (e.g., co-workers, supervisors, customers, members of the public)

□ Other external source (specify) ___________________

□ Self-paced by worker within general performance guidelines

• Did you have any difficulty picking one factor that most controlled the pace of work?

  • Yes – if yes, please explain

  • No

• What does the phrase “Self-paced by worker within general performance guidelines” mean to you?

• Here’s the next question.

2. How would you describe the work flow for this job? Would you say that it ...

□ Has few, if any, slack periods

□ Has few, if any, rush periods (or large build-ups of work)

□ Varies, with fluctuations of rush periods (or large build-ups of work) and slow periods on a daily or weekly basis

• Can you explain what the question means by “work flow?”

• Here’s the next question.

3. Can workers step away from their work area easily outside of scheduled breaks (for example, for short bathroom breaks)

□ Yes

□ No

• Does the [job supervised] have scheduled work breaks? (If yes) How often are these? So, in addition to these regular breaks, it’s not a problem if someone has to take a short bathroom break?

• Here’s the next question.

4. How frequently is work checked in the occupation?

□ More than once per day

□ Once per day

□ At least once per week, but not every day

□ Less than weekly

• What is involved in how you check work in [job supervised)?

• Here’s the next question.

5. Are supervisors generally present in the same work area as workers?

□ Yes

□ No

• In your own words, what does it mean for a supervisor to be “in the same work area as workers?”

• Does a supervisor have to be visible to the worker to be considered to be in the same area?

  • Yes

  • No

  • Depends

6. The next question is about “problem solving” tasks that the worker does in his/her job. Think of “problem solving” as what happens when workers are faced with a new or difficult situation which requires them to think for a while about what to do next. How often is the worker responsible for solving problems that take more than 5 minutes to find a good solution?

□ More than once per day

□ Once per day

□ At least once per week, but not every day

□ Less than weekly, but at least once per month

□ Less than monthly, including never

• This question asked about problem solving in [job supervised]. Can you give me some examples of problems this worker would have to solve?

• Was the 5 minute guideline helpful?

7. How often does this occupation require verbal, work-related interactions?

□ Constantly, every few minutes

□ More than once per hour, but not constantly

□ More than once per day, but not more than once per hour

□ No more than once per day, including never

• Can you think of a different way to ask this question?

8. What level of “people skills” does this job require?

□ Minimal/Basic

□ More than basic

• How would you define minimal or basic people skills?

• Finally, I asked you a series of yes/no questions about the job.

Are workers in this occupation:
Required to work with the general public?	Yes No
Required to work around crowds?	Yes No
Required to supervise others as part of job?	Yes No
Permitted to work from home or telework?	Yes No
Required to work outside?	Yes No

• Who do you consider the “general public” to be?

• (If yes to “work around crowds”) Can you describe the types of crowds this person works around?

• (If yes to “supervise others”) Please describe the types of supervisory responsibilities this work has.

• (If yes to “work outside”) What does it mean to work outside in this job

Attachment C – Consent form

OMB Control Number: 1220-0141

Expiration Date: April 30, 2018

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 40 minutes to participate in this research.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. OMB control number is 1220-0141 and expires April 30, 2018.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974 (DOL/BLS – 14 BLS Behavioral Science Research Laboratory Project Files (81 FR 47418)), as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. Per the Federal Cybersecurity Enhancement Act of 2015, Federal information systems are protected from malicious activities through cybersecurity screening of transmitted data.

12