Note to Reviewer

July 27, 2016

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Scott Fricker Senior Research Psychologist Office of Survey Methods Research
SUBJECT:	Submission of Materials for ORS Incumbent Survey Development Testing

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, I am submitting a brief description of the study.

The total estimated respondent burden hours for this study are 37 hours.

If there are any questions regarding this project, please direct them to Scott Fricker (202-691-7390).

1. Introduction and Purpose

The current design of the Occupational Requirements Survey (ORS) utilizes establishment respondents (typically human resource managers or occupational safety personnel) to report the occupational requirements of selected jobs in the organization. One of the challenges of this approach is that establishment-level respondents may not always have the knowledge necessary to provide accurate reports of incidence and duration for all of the ORS data elements.

A potential solution to address knowledge gaps introduced by reliance on establishment-respondent reports is to collect data directly from employees (job incumbents) in the organization about the requirements of their occupation. Job incumbents have intimate, firsthand knowledge of their job tasks and activities, and in principle should be able to supply ORS data more reliably and accurately than less knowledgeable establishment respondents. In addition, incumbents may be able to provide data for occupations and/or elements that ORS currently has difficulty collecting (e.g., duration of physical elements). Despite these advantages, the methodological concern with utilizing incumbent self-reports is that employees may over- or underestimate true values due to social desirability effects, or issues with comprehension, recall, and estimation. Incumbents may also focus on their own experience rather than the general requirements of their position.

The intent of this study is to understand and evaluate the measurement properties and performance of an incumbent survey of occupational requirements. How do employees interpret the questions they are asked? How does the survey format (e.g., interviewer administered vs. self-administered; diary vs. stylized questionnaire; in-person vs. phone) affect the work tasks/ORS data elements that are captured from incumbent reports? What memory and estimation processes do incumbents use to make those reports? Are there any changes that can be made to survey protocols and materials that would improve the respondent experience and data quality? To begin to answer these questions, participants in this study will complete an incumbent survey and then be asked about their experiences using cognitive interviewing techniques.

2. Research Design and Procedures

This is an early-stage, qualitative study designed to evaluate potential approaches for collecting occupational-requirements information from job incumbents. The focus of data collection and analysis in this study will be on the ORS cognitive and physical-demand data elements. In addition, participants will be asked to provide information about general occupational characteristics (e.g., job title, work schedule), and to generate a list of their primary work tasks. No Personally Identifiable Information (PII) will be collected as part of the ORS incumbent interview. All participants will be currently employed in one of the following occupations: nursing assistant; childcare worker; truck driver; security guard; or bank teller.

Study participants will come to the Office of Survey Methods Research (OSMR) research lab at BLS. Observers from the test team and ORS program office will monitor the session from an adjacent room. The session facilitator will begin by introducing the purpose of the study, obtaining informed consent, and answering respondent questions (Appendix B). Study participants will receive one of four incumbent-survey collection methods: (a) interviewer-administered, in-person; (b) interviewer-administered, phone; (c) self-administered, stylized questionnaire; or (d) self-administered, diary.

The interviewer-administered incumbent surveys will be conducted by experienced ORS Field Economists (FEs) following existing ORS collection procedures. As in production, FEs in this study will rely on their knowledge of the ORS elements and conversational interviewing techniques to help streamline data collection. They will begin by asking incumbents about their job characteristics and primary work tasks, and use this information to code as many of the ORS data elements as possible. FEs will then ask targeted follow-up probes as needed to capture the remaining cognitive and physical demand elements. The format and ordering of these probes is at the FEs’ discretion (see Attachment D for ORS Interviewer Instrument). FEs in the in-person condition will interview the participants face-to-face in the OSMR testing room; FEs in the phone condition will call in from another location to conduct the interview. After the incumbent survey has been completed, the study facilitator (an OSMR researcher) will conduct a debriefing to assess the participant’s experience and response processes (Appendix D). The test team plans to conduct 10 interviewer-administered surveys (five in each mode).

In the self-administered, stylized questionnaire condition study participants will complete the incumbent survey (Attachment A) in the OSMR lab. The survey will be administered through an online tool (SurveyMonkey) that the participant will access from the testing room; the survey is intended to take respondents not more than 20 minutes. The test facilitator and observers will be able to view participants as they complete the survey from an adjacent room, and will note behavioral indicators of confusion or frustration. Upon completion of the survey, participants will be asked a series of cognitive interview questions. The test team plans to run 10 participants in this condition.

Due to the nature of the collection method, participants in the self-administered, diary condition will complete their ORS diary prior to coming in for the test session. Each study participant in this condition will be assigned a diary day on which to complete their diary. OSMR study recruitment staff will send participants the diary instructions and tool (Attachments B and C), notify them of their assigned diary day, and schedule a follow-up appointment for the cognitive interviewing session. The test team plans to run 10 participants in this condition.

Cognitive interviews provide an in-depth understanding of a respondent’s thought processes and reactions to survey questions and procedures. The cognitive interviews for this study will be conducted in the Office of Survey Methods Research (OSMR) laboratory by staff experienced in qualitative research.

3. Participants

Thirty participants will be recruited from a database maintained by OSMR, and through advertisements targeting selected occupations. During the recruiting process, recruiters will identify individuals who are incumbents in the target occupations, and individuals will be directed to come to the OSMR lab at BLS. Participants will be balanced across these occupational groups, as well as by education and age (see Appendix A for screener questions).

4. Burden Hours

The target sample size for this study is 30 participants. The test team anticipates that each in-lab session will last no longer than one hour. The five participants in the diary condition will have additional burden due to needing to complete their work diary prior to their cognitive interview session; we expect that the incumbent diary will take a total of 20 minutes to complete. Screening potential participants is estimated to take five minutes per participant. Since individuals will be responding to targeted recruitment ads and/or will have been participants in prior OSMR studies, we expect a fairly high success rate (approximately 70%). This means we will have to screen 43 people in order to find 30 eligible participants. Total burden hours for screening and participation are expected to be 35.3.

Interview Task	Number of Responses	Minutes per respondent	Total Number of minutes	Burden Hours
Recruiting/Screening questions	43	5	215	4
Interview administered – in person Interview administered – by phone	5 5	60 60	300 300	5 5
Self-administered	10	60	600	10
Diary	10	80	800	13
Totals				37

5. Payment

Participants will receive $40 for their time.

6. Data Confidentiality

Participants will be informed as to the voluntary nature of the survey, and that the study will be used for internal research purposes only. Participants in the interviewer-administered conditions will be given a consent form to read and sign (Appendix C). Participants in the self-administered conditions will be not be given a pledge of confidentiality because data collected as part of this study will be stored on SurveyMonkey servers. The following notice will be verbally read to participants at the start of the study session, and placed on the first page of the survey:

This survey is being administered by surveymonkey.com and resides on a server outside of the BLS domain. BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response.

Appendix A – Screening Questions

Appendix B – Introduction

Appendix C – Consent Form – Interview Administered (In person and by phone)

Appendix D – Debriefing Questions

Attachment A – Self-Administered Stylized Incumbent Survey

Attachment B – Diary Instructions

Attachment C – Diary Instrument

Attachment D – ORS Production Interviewer Instrument

Appendix A: Screening Questions

1. Are you currently employed either part-time or full-time?

   • Yes

   • No

2. (If Yes to Q1) What is your current job title?

3. (If Yes to Q1, and not obvious from Q2) Do you work in one of the following occupations?

• Nursing Assistant

• Childcare Worker

• Truck driver (Long-haul)

• Security Guard

• Bank Teller

• DOES NOT WORK IN THE TARGET JOBS (Stop)

4. (If eligible for study) How long have you worked as a [fill with occupation from Q3] for your current employer?

• Less than a year

• 1 – 2 years

• 3 – 5 years

• 6 – 10 years

• More than 10 years

5. (If eligible for study) What is the highest level of education you have achieved, or the highest grade you have completed?

• Less than HS

• HS diploma or equivalent

• Some college but no degree

• BA or BS

• Post Graduate (MA/MS, PhD, MD)

• Other, specify

6. (If Yes to Q1) How old are you?

Appendix B: Introduction

• Hi! Thank you for coming in today.

• I have a couple colleagues in the next room that will be observing and taking notes.

• Explanation of the study purpose:

  • Today we’re going to be evaluating a new survey that collects information about the physical and mental demands of occupations. BLS currently collects this information by speaking to managers or HR staff in a company, but we’re exploring the feasibility of collecting this information directly from the employees themselves. We will be asking you to complete this survey, and then we will spend some time at the end discussing your reactions (e.g., what you liked or disliked, how easy or difficult you found it, suggestions for how we can improve things).

  • [For interviewer-administered survey participants, introduce FE]. This is [Name]. S/He is a BLS interviewer for the Occupational Requirements Survey, and will be administering the survey to you today. I’ll duck out of the room while you two go through the survey questions. When you’ve finished, I will come back with a few questions about your experience.

  • It’s important to note that we’re not here to evaluate you or your abilities. We’re speaking to a number of people with different backgrounds and in different jobs, and really just trying to learn what works and what doesn’t. And we’re pretty early in this process, so we know our survey isn’t perfect by any stretch.

• Consent Form [for interviewer-administered conditions] or confidentiality acknowledgement [for self-administered conditions]

• Any questions before we begin?

Appendix C: Consent Form

OMB Control Number: 1220-0141

Expiration Date: April 30, 2018

CONSENT FORM

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you will be observed.

We estimate it will take you an average of 60 minutes to participate in this research.

Your participation in this research project is voluntary and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires April 30, 2018.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Appendix D: Debriefing Questions

Thanks for completing that survey. We are going to switch gears now and talk a bit about what it was like for you to answer those questions. I have some follow-up questions that will help us better understand how people are responding to this survey. I want to emphasize again that there are no ‘right’ or ‘wrong’ answers to these questions – we are just interested in your interpretations and reactions.

1. What were your reactions to the survey?

2. Were there any instructions that you found confusing or questions that were difficult for you to answer? If so, what were they? Tell me a bit more about that – what made them difficult or confusing?

3. You were asked to provide a list of your primary work tasks. What does that phrase “work task” mean to you?

1. How did you decide what kinds of activities to report for your primary work tasks?

2. How easy or difficult was it for you to come up with your list of work tasks? (Very Easy, Somewhat Easy, Neither Easy Nor Difficult, Somewhat Difficult, Very Difficult) Why was it [fill with R’s answer]?

3. The purpose of the task list is to give us a sense of your main job duties and responsibilities, and in particular to better understand the physical and cognitive demands of your current occupation. Are there any other work activities that you didn’t include in your task list that you think might affect our understanding of the physical or cognitive demands of your occupation?

1. If so, what are they?

2. Why did you decide not to include them in your original task list?

3. How would these activities change the picture we have of the physical or cognitive demands of your job?

Diary Debriefing

Now I’d like to ask you some specific questions about filling out the diary. Let’s start at the beginning:

1. When you agreed to participate in this study, we sent you an email that indicated your assigned diary day, and provided instructions for completing this survey. [Show respondent a copy of the Diary Instructions].

   1. What are your reactions to these materials? Were the instructions clear? Did you run into any “surprises” once you started filling out the diary?

2. What could we change to improve the instructions or the diary itself?

3. Did you review the definitions and visual examples of the ORS Physical Demands that were included in the instructions?

1. If not, why not?

2. If so, to what extent did you find that they helped you when completing the survey (and how – did you refer back to them as you filled out the diary? Were any of the definitions of the ORS concepts different from ones you already had on your own?)?

2. When did you fill out the diary? Did you complete it on the assigned day? (If not, why not? What day did you fill it out?)

1. Did you complete the diary all in one sitting (e.g., at the end of the day), or did you work on it throughout the day?

1. What led you to this approach (e.g., were there work/time constraints)?

2. [If the respondent completed the diary at the end of the day] How easy or difficult was it for you to remember the work tasks you did during the day? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Why was that?

1. Do you think that it would have been any different (easier or more difficult) if you had been assigned a different diary day (e.g., are there variations across work days, seasons, project cycles, etc.)?

2. Looking back at your diary entries now, how accurate do you feel you were with your task list(s) for that day? (Very, Somewhat, Not Very, Not at all) Why is that? Is there anything we could do to improve the accuracy?

3. How long did it take you in total to complete the diary (not including the other occupational/cognitive questions)? _________

4. Would you say that the amount of time and effort that it took you to complete the diary was (Very reasonable, Somewhat reasonable, Not very reasonable, Not at all reasonable)? Why is that?

5. Do you think that your employer would give you time to complete this diary during your work day?

2. How easy or difficult was it for you to estimate the durations of your tasks? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). How did you go about estimating durations?

3. How easy or difficult was it for you to decide which ORS Physical Demand elements applied to each of your task entries? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult).

1. Which elements gave you the most difficulty (and why)?

2. Which elements were the easiest for you (and why)?

4. What changes could we make to the layout of the diary that would make it easier to complete or give us a better sense of the physical demands of your occupation?

Stylized Questions Debriefing

1. How long did it take you in total to complete this survey? _________

1. Did you complete the survey all in one sitting, or did you fill out parts of it over time? What led you to fill out the survey this way?

2. Would you say that the amount of time and effort that it took you to complete this survey was (Very reasonable, Somewhat reasonable, Not very reasonable, Not at all reasonable)? Why is that?

2. You were kind enough to volunteer for this study, and are being paid to help us evaluate and improve this survey. If this was not the case – if your employer simply asked you to complete this survey for the BLS, and you were NOT getting paid:

   1. How likely would you be to complete the survey (Very likely, Somewhat likely, Not very likely, Not at all likely). Why is that?

2. Do you think the information you’d provide would be more thorough and accurate than what you provided for this study, less thorough and accurate, or about the same? Why is that?

Cognitive Elements

Now, let’s take a look at the questions we asked about the cognitive demands of your current occupation.

1. The first question asked about decision making. What are your reactions to this question?

1. What does the phrase “independent judgment” mean to you?

2. Can you give me some examples of the types of situations or activities that you thought about when answering this question?

3. How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). What made it [fill]?

4. Did you view the example how decision-making is coded? If so, did it make sense – was it helpful? If not, why not? What could we do to make this example or this question clearer?

2. How about the question about work review. What are your reactions to this question?

1. Are there any terms or concepts in this question that were confusing to you, or that you think might be confusing for others? (If so, which ones and why?)

2. Can you give me some examples of the types of situations or activities that you thought about when answering this question?

3. How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult)

3. The next sections asked several questions about the pace of work in your occupation. What are your reactions to these questions?

1. Are there any terms or concepts in this set of questions or the instructions that were confusing to you, or that you think might be confusing for others? (If so, which ones and why?)

2. For the fastest pace and control of work flow questions, what types of situations or activities did you think about when answering these questions?

3. [For each Pace item ask:] How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult)

4. You were asked several questions about changes in work routines (tasks, schedule, and location).

1. Work Tasks:

   1. How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Tell me more about that - what made it [fill]?

2. You reported that your occupation experiences changes in work tasks [fill: at least once per day, etc.]. How did you arrive at this answer? Could you give me examples of how your work tasks change?

3. How much variability is there in this job in terms of how frequently work tasks change (e.g., is the pattern basically the same throughout the year, or does it vary? Besides time of year, are there other factors that affect this?)

4. Do you think that other individuals working in this occupation in this company would report the same level of changes in work tasks?

2. [For both the work schedule and work location questions, ask:] How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Why is that?

3. [For participants that reported some change in work location, ask:] How did you arrive at your answer? Can you describe the types of location changes you thought about when answering this question? (Probe to gauge extent to which change is unique to this employee, etc.)

5. The last set of questions about the cognitive demands of your job asked about work-related personal-interactions – with “regular contacts” and “other contacts”).

1. First, in your own words, how would you describe what a “regular” contact is and what an “other” contact is?

2. How easy or difficult is it in this job to identify those who you’d consider “regular” contacts from those you’d consider “other” contacts? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Tell me more about that - what made it [fill]

3. We asked you what the highest level of work-related verbal interactions this occupation has with contacts [show question and response options] – what are your reactions to this question?

1. Does it make sense, are the question and response options clear? If not, why not?

2. How did you come up with your answer to this question – what sorts of things did you think about/consider?

3. Was this question more or less difficult for “regular” contacts than “other” contacts, and why?

4. The last question asked “what is the highest level of work-related interactions that you have with contacts. [Show question and response options] – What are your reactions to this question?

1. Does it make sense, are the question and response options clear? (If not, why not?) Are there any terms you found confusing or difficult?

2. You reported that this occupation has interactions that involve [fill]. How did you decide on this category? What sorts of things did you think about/consider?

3. Was this question more or less difficult for “regular” contacts than “other” contacts, and why?

Physical Demand Elements

Now, let’s take a look at the questions we asked about the physical demands of your current occupation.

1. First, I’d like to get a general sense of what it was like to answer these questions. If you had to give three words to describe what it was like, what words would you choose? (Follow up probes as needed)

2. [show 1^st page of definitions] Were these instructions and definitional guidelines clear and understandable to you? If not, why not?

1. We asked you to report the amount of time spent performing the physical demand on a typical work day/shift. For your occupation, does it make sense to talk about a typical work day? If not, why not? How did you decide what a typical day looks like in your job? Do you think this affected the accuracy of your responses?

3. [Show participant the list of elements] Here is a list of the physical demands that you were asked about. In general, how easy or difficult was it for you to answer questions about these elements? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). What made it [fill]?

1. Which elements gave you the most difficulty (and why)?

2. Which elements were the easiest for you (and why)?

3. What could we do to make it easier for respondents to report accurately about the physical demands of their job (e.g., more/better formatted instructions, additional aids/resources, different collection procedures, etc.?)

4. [If not evident from Q3, ask:] How easy or difficult was it for you to estimate the amount of time this occupation spends in each of the physical demand behaviors? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult).

1. Which elements were hardest in terms of estimating durations (and why)?

2. Which elements were the easiest in terms of estimating durations (and why)?

3. How did you go about estimating durations? What could we do to improve the accuracy of respondents’ duration estimates?

4. How precise did you think you needed to be in your duration estimates (e.g., ballpark, to the nearest hour or within a certain percent range, to the minute)?

Interviewer-Administered Debriefing

1. Would you say that the amount of time and effort that it took you to complete this survey was (Very reasonable, Somewhat reasonable, Not very reasonable, Not at all reasonable)? Why is that?

2. You were kind enough to volunteer for this study, and are being paid to help us evaluate and improve this survey. If this was not the case – if your employer simply asked you to complete this survey for the BLS, and you were NOT getting paid:

   1. How likely would you be to complete the survey (Very likely, Somewhat likely, Not very likely, Not at all likely). Why is that?

2. Do you think the information you’d provide (as an unpaid ORS respondent) would be more thorough and accurate than what you provided for this study, less thorough and accurate, or about the same? Why is that?

3. One of the things we’re hoping to learn in this study is how we can improve our procedures for collecting the data. In a minute we’ll talk more about the specific questions you were asked. But first I’m curious about your impressions of how we collected the data.

1. In your view, did the interview process go smoothly? If not, why not?

2. Did you feel that the person who interviewed you today listened and understood the information that you shared about your job? If not, why not?

3. This survey is designed to be more conversational than most government surveys. Did it seem conversational to you?

1. If so, how so – could you give me an example?

2. If not, why not?

3. Did you like this approach or would you prefer a more traditional, structured interview?

4. If your company was selected to participate in the Occupational Requirements Survey, and you were asked to provide information about the requirements of your occupation, how would you prefer to report your data:

2. Have an interviewer interview me, in person (at work)

3. Have an interviewer interview me, by phone (at work)

4. Complete a self-administered survey (paper, web)

5. Fill out a daily diary of your work tasks and job requirements (paper or web)

6. Some other method (e.g., have supervisor provide): ___________________

EXPLAIN

Ask Stylized Questions and Social Desirability debriefings questions

Debriefing - Social Desirability

1. Were there any questions (in the diary or survey) that you thought were too personal or intrusive? Which ones and why?

2. [For diary participants] Do you think there is an appropriate number of tasks that an employee in your occupation should list in this diary?

1. What is the minimum number of tasks that you feel should be reported in order to accurately capture a typical day in this occupation?

2. How much variation is there in this job in the number or type of tasks performed each day?

3. Sometimes two people looking at the same job might view it differently – for example, one person might view something as a requirement of the job while others view it as simply a choice, or they could have different estimates about the amount of time a person is required to spend doing something on a typical day. With that in mind, imagine that we had your supervisor fill out the same survey for your occupation.

   1. To what extent do you think you and your supervisor would agree on the level of cognitive demands in your occupation? (100% agreement, 75%, 50%, 25%, 0% - don’t know). Why is that?

2. To what extent do you think you and your supervisor would agree on the types of physical demands in your occupation? (100% agreement, 75%, 50%, 25%, 0% - don’t know) Why is that?

3. To what extent do you think you and your supervisor would agree on the duration of those physical demands in your occupation? (100% agreement, 75%, 50%, 25%, 0% - don’t know) Why is that?

4. In some ways, asking employees about their job requirements is a little like reading through a resume – it can be extremely informative but we also know that some people may tend to exaggerate or put a positive spin on things. This is completely normal, but it can be a challenge for a survey organization that’s trying to collect the most accurate data possible.

   1. Thinking about other employees in this occupation, how likely do you think they would be to overstate the requirements of their job? (Very likely, Somewhat likely, Somewhat unlikely, Not at all likely) Why?

2. [If Q4a is Very or Somewhat likely, ask:] What could we do to encourage respondents not to overstate their job requirements?

3. Are there situations in which you think that respondents might underestimate the requirements of their job? If so, why? Which requirements?

6