Form Appendices A-F Appendices A-F Interview Protocol

Appendix A: Screening Questions for In-Person Interviews and MTurk

1. Are you currently employed either part-time or full-time?

   • Yes

   • No

2. (If Yes to Q1) What is your current job title?

3. (If eligible for study) How long have you worked as a [fill with occupation from Q3] for your current employer?

• Less than a year

• 1 – 2 years

• 3 – 5 years

• 6 – 10 years

• More than 10 years

4. (If eligible for study) What is the highest level of education you have achieved, or the highest grade you have completed?

• Less than HS

• HS diploma or equivalent

• Some college but no degree

• BA or BS

• Post Graduate (MA/MS, PhD, MD)

• Other, specify

5. (If Yes to Q1) How old are you?

Appendix B: Introduction

• Hi! Thank you for coming in today.

• [If team member is observing] I have a colleague in the next room that will be observing and taking notes.

• Explanation of the study purpose:

  • Today we’re going to be evaluating a new survey that collects information about the mental and cognitive demands of occupations. BLS currently collects this information by speaking to managers or HR staff in a company, but we’re exploring the feasibility of collecting this information directly from the employees themselves. We will be asking you to complete this survey, and then we will spend some time at the end discussing your reactions (e.g., what you liked or disliked, how easy or difficult you found it, suggestions for how we can improve things).

  • It’s important to note that we’re not here to evaluate you or your abilities. We’re speaking to a number of people with different backgrounds and in different jobs, and really just trying to learn what works and what doesn’t. And we’re pretty early in this process, so we know our survey isn’t perfect by any stretch.

• Consent Form [for interviewer-administered conditions] or confidentiality acknowledgement [for self-administered conditions]

• Any questions before we begin?

Appendix C: Consent Form

OMB Control Number: 1220-0141

Expiration Date: April 30, 2018

CONSENT FORM

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you will be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 30 minutes to participate in this research.

Your participation in this research project is voluntary and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires April 30, 2018.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Appendix D: ORS Cognitive Demand Questions

Decision-making

What is the highest level of independent judgment a worker is expected to use to perform the tasks of this occupation?

• Employee uses independent judgment to select from a limited number of predetermined actions.

• Employee uses independent judgment to determine the most appropriate course of action in situations that do not have set responses.

• Employee uses independent judgment to make decisions by choosing from a large number of possibilities in situations where a high degree of uncertainty or complexity may exist.

Work Review

How frequently is work checked in the occupation?

• More than once per day.

• Once per day.

• At least once per week, but less than daily.

• Less than weekly.

Pace

Are there faster and slower periods of work?

• Yes

• No

What is the fastest pace performed?

• Rapid with no periods of waiting.

• Steady with rare periods of waiting.

• Unhurried with much time spent observing or waiting, rushed periods rarely or never occur.

Control of Work Flow

Can a worker intervene and control the flow of work?

• Yes. The worker can change the priority of work tasks or the amount of time allotted to complete them.

• No. The work is primarily driven by business processes, production line speed, or customer demands.

Adaptability

Work tasks are the regular duties of an occupation. How often do work tasks change in this occupation?

• At least once per day.

• At least once per week, but less than daily.

• At least once per month, but less than weekly.

• Less than monthly, including never.

Work location is the physical site where work is performed. How often does the work location change in this occupation?

• Does not change unless it is permanent.

• Changes up to four times a year.

• Changes more than four times a year.

Work schedule is the work hours and days for the occupation set by the employer. Does the work schedule change in this occupation?

• Yes

• No

Appendix E: Debriefing Questions

Thanks for completing that survey. We are going to switch gears now and talk a bit about what it was like for you to answer those questions. I have some follow-up questions that will help us better understand how people are responding to this survey. I want to emphasize again that there are no ‘right’ or ‘wrong’ answers to these questions – we are just interested in your interpretations and reactions.

What were your reactions to the survey?

Cognitive Elements

1. The first question asked about decision making. What are your reactions to this question?

1. In your own words, please explain what it means to use “independent judgment” on the job?

2. How would you describe what a “predetermined action” is?

3. How would you describe a job task that does not have a set response?

4. The question presents three levels of independent judgment. Please give one or more examples of job tasks at each of the following three levels (the job tasks can be from any job, not just your own):

   1. Selecting from a limited number of predetermined actions.

   2. Determining the most appropriate course of action in situations that do not have set responses.

   3. Making decisions by choosing from a large number of possibilities in situations where a high degree of uncertainty or complexity may exist.

5. How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). What made it [fill]?

2. How about the question about work review. What are your reactions to this question?

1. What does it mean to have “worked checked” in your job? (What is being checked? Who is doing the checking?)

2. Is your work checked on a regular, predictable basis, or is checked only occasionally, or even randomly?

3. Are there any terms or concepts in this question that you think might be confusing for others? (If so, which ones and why?)

4. Can you give me some examples of the types of situations or activities that you thought about when answering this question?

5. How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult)

3. The next sections asked several questions about the pace of work in your occupation. What are your reactions to these questions?

1. For the fastest pace and control of work flow questions, what types of situations or activities did you think about when answering these questions? (e.g., What are examples of fast period of work? Slow periods?)

2. In your own words, what does it mean to perform work at a rapid pace, at a steady pace, at an unhurried pace?

3. Are there any terms or concepts in this set of questions or the instructions that were confusing to you, or that you think might be confusing for others? (If so, which ones and why?)

4. How easy or difficult was the Pace of Work question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult)

5. For the Control of Work pace question, please explain what “intervene and control the flow of work” means to you.

6. What does it mean to have work driven by business processes?

7. How easy or difficult was the Control of Work Flow question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult)

4. You were asked several questions about changes in work routines (tasks, schedule, and location).

1. Work Tasks:

   1. How would you define a “work task” in your current job? Please give a couple of examples, and examples of when or how your work tasks change.

2. How easy or difficult was it to figure out how often your work tasks change? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Please tell me more about that - what made you give that answer? How did you arrive at this answer?

3. How much variability is there in this job in terms of how frequently work tasks change (e.g., is the pattern basically the same throughout the year, or does it vary? Besides time of year, are there other factors that affect this?)

4. Do you think that other individuals working in this occupation in this company would report the same level of changes in work tasks?

2. [For both the work schedule and work location questions, ask:] How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Why is that?

3. What do you consider changes in work schedule to be? If your employer allows you to occasionally change the times you begin and end work, would you consider that a change in work schedule?

4. [For participants that reported some change in work location, ask:] How did you arrive at your answer? Can you describe the types of location changes you thought about when answering this question? (Probe to gauge extent to which change is unique to this employee, etc.)

5. Sometimes two people looking at the same job might view it differently – for example, one person might view something as a requirement of the job while others view it as simply a choice, or they could have different estimates about the amount of time a person is required to spend doing something on a typical day. With that in mind, imagine that we had your supervisor fill out the same survey for your occupation.

1. To what extent do you think you and your supervisor would agree on the level of cognitive demands in your occupation? (100% agreement, 75%, 50%, 25%, 0% - don’t know). Why is that?

Appendix F: Coding Scheme for Participant Responses

Question Comprehension

a	A comment was made that wording of the question stem was ambiguous, unclear, too abstract, or confusing
b	A comment was made that wording of a question response option was ambiguous, unclear, too abstract, or confusing
c	Respondent interpreted question incorrectly
d	(In-person) Respondent asked that the question be reread / (For TryMyUI respondents) Respondent had to read the question more than once
e	A comment was made that question was long or wordy
f	(For in person) Respondent interrupted reading of the question or response options / (For TryMyUI respondents) Respondent did not fully read the question and all of the response options out loud / (For Embedded-probe condition) Respondent indicated that he/she did not read all of the question or response options

Response Selection

g	None of the response option(s) applied; correct response option was missing
h	Struggled with selection of best response option
i	Answered question but expressed uncertainty or lack of confidence about accuracy

Retrieval

j	Gave (in-person) or Entered (online) a “don’t know” response Reason: could not recall information (e.g., number of hours of leave taken in past 7 days)
k	Reason: did not have the knowledge to answer question (e.g., was not sure how often employer reviews work)
L	A comment was made that a question was not relevant to respondent
m	The respondent has difficulty recalling, formulating, or reporting the answer (e.g., remembering/calculating the frequency of work changes)

Other

n	A positive comment was made, for example, question was easy, straightforward, etc.
o	Answer to a current question is not in agreement with, or is inconsistent with, an answer to a previous question

No problems

p	No problems noted

2