Appendix D Debriefing Questions

Appendix D: Debriefing Questions

Thanks for completing that survey. We are going to switch gears now and talk a bit about what it was like for you to answer those questions. I have some follow-up questions that will help us better understand how people are responding to this survey. I want to emphasize again that there are no ‘right’ or ‘wrong’ answers to these questions – we are just interested in your interpretations and reactions.

1. What were your reactions to the survey?

2. Were there any instructions that you found confusing or questions that were difficult for you to answer? If so, what were they? Tell me a bit more about that – what made them difficult or confusing?

3. You were asked to provide a list of your primary work tasks. What does that phrase “work task” mean to you?

1. How did you decide what kinds of activities to report for your primary work tasks?

2. How easy or difficult was it for you to come up with your list of work tasks? (Very Easy, Somewhat Easy, Neither Easy Nor Difficult, Somewhat Difficult, Very Difficult) Why was it [fill with R’s answer]?

3. The purpose of the task list is to give us a sense of your main job duties and responsibilities, and in particular to better understand the physical and cognitive demands of your current occupation. Are there any other work activities that you didn’t include in your task list that you think might affect our understanding of the physical or cognitive demands of your occupation?

1. If so, what are they?

2. Why did you decide not to include them in your original task list?

3. How would these activities change the picture we have of the physical or cognitive demands of your job?

Diary Debriefing

Now I’d like to ask you some specific questions about filling out the diary. Let’s start at the beginning:

1. When you agreed to participate in this study, we sent you an email that indicated your assigned diary day, and provided instructions for completing this survey. [Show respondent a copy of the Diary Instructions].

   1. What are your reactions to these materials? Were the instructions clear? Did you run into any “surprises” once you started filling out the diary?

2. What could we change to improve the instructions or the diary itself?

3. Did you review the definitions and visual examples of the ORS Physical Demands that were included in the instructions?

1. If not, why not?

2. If so, to what extent did you find that they helped you when completing the survey (and how – did you refer back to them as you filled out the diary? Were any of the definitions of the ORS concepts different from ones you already had on your own?)?

2. When did you fill out the diary? Did you complete it on the assigned day? (If not, why not? What day did you fill it out?)

1. Did you complete the diary all in one sitting (e.g., at the end of the day), or did you work on it throughout the day?

1. What led you to this approach (e.g., were there work/time constraints)?

2. [If the respondent completed the diary at the end of the day] How easy or difficult was it for you to remember the work tasks you did during the day? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Why was that?

1. Do you think that it would have been any different (easier or more difficult) if you had been assigned a different diary day (e.g., are there variations across work days, seasons, project cycles, etc.)?

2. Looking back at your diary entries now, how accurate do you feel you were with your task list(s) for that day? (Very, Somewhat, Not Very, Not at all) Why is that? Is there anything we could do to improve the accuracy?

3. How long did it take you in total to complete the diary (not including the other occupational/cognitive questions)? _________

4. Would you say that the amount of time and effort that it took you to complete the diary was (Very reasonable, Somewhat reasonable, Not very reasonable, Not at all reasonable)? Why is that?

5. Do you think that your employer would give you time to complete this diary during your work day?

2. How easy or difficult was it for you to estimate the durations of your tasks? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). How did you go about estimating durations?

3. How easy or difficult was it for you to decide which ORS Physical Demand elements applied to each of your task entries? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult).

1. Which elements gave you the most difficulty (and why)?

2. Which elements were the easiest for you (and why)?

4. What changes could we make to the layout of the diary that would make it easier to complete or give us a better sense of the physical demands of your occupation?

Stylized Questions Debriefing

1. How long did it take you in total to complete this survey? _________

1. Did you complete the survey all in one sitting, or did you fill out parts of it over time? What led you to fill out the survey this way?

2. Would you say that the amount of time and effort that it took you to complete this survey was (Very reasonable, Somewhat reasonable, Not very reasonable, Not at all reasonable)? Why is that?

2. You were kind enough to volunteer for this study, and are being paid to help us evaluate and improve this survey. If this was not the case – if your employer simply asked you to complete this survey for the BLS, and you were NOT getting paid:

   1. How likely would you be to complete the survey (Very likely, Somewhat likely, Not very likely, Not at all likely). Why is that?

2. Do you think the information you’d provide would be more thorough and accurate than what you provided for this study, less thorough and accurate, or about the same? Why is that?

Cognitive Elements

Now, let’s take a look at the questions we asked about the cognitive demands of your current occupation.

1. The first question asked about decision making. What are your reactions to this question?

1. What does the phrase “independent judgment” mean to you?

2. Can you give me some examples of the types of situations or activities that you thought about when answering this question?

3. How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). What made it [fill]?

4. Did you view the example how decision-making is coded? If so, did it make sense – was it helpful? If not, why not? What could we do to make this example or this question clearer?

2. How about the question about work review. What are your reactions to this question?

1. Are there any terms or concepts in this question that were confusing to you, or that you think might be confusing for others? (If so, which ones and why?)

2. Can you give me some examples of the types of situations or activities that you thought about when answering this question?

3. How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult)

3. The next sections asked several questions about the pace of work in your occupation. What are your reactions to these questions?

1. Are there any terms or concepts in this set of questions or the instructions that were confusing to you, or that you think might be confusing for others? (If so, which ones and why?)

2. For the fastest pace and control of work flow questions, what types of situations or activities did you think about when answering these questions?

3. [For each Pace item ask:] How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult)

4. You were asked several questions about changes in work routines (tasks, schedule, and location).

1. Work Tasks:

   1. How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Tell me more about that - what made it [fill]?

2. You reported that your occupation experiences changes in work tasks [fill: at least once per day, etc.]. How did you arrive at this answer? Could you give me examples of how your work tasks change?

3. How much variability is there in this job in terms of how frequently work tasks change (e.g., is the pattern basically the same throughout the year, or does it vary? Besides time of year, are there other factors that affect this?)

4. Do you think that other individuals working in this occupation in this company would report the same level of changes in work tasks?

2. [For both the work schedule and work location questions, ask:] How easy or difficult was this question for you to answer? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Why is that?

3. [For participants that reported some change in work location, ask:] How did you arrive at your answer? Can you describe the types of location changes you thought about when answering this question? (Probe to gauge extent to which change is unique to this employee, etc.)

5. The last set of questions about the cognitive demands of your job asked about work-related personal-interactions – with “regular contacts” and “other contacts”).

1. First, in your own words, how would you describe what a “regular” contact is and what an “other” contact is?

2. How easy or difficult is it in this job to identify those who you’d consider “regular” contacts from those you’d consider “other” contacts? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). Tell me more about that - what made it [fill]

3. We asked you what the highest level of work-related verbal interactions this occupation has with contacts [show question and response options] – what are your reactions to this question?

1. Does it make sense, are the question and response options clear? If not, why not?

2. How did you come up with your answer to this question – what sorts of things did you think about/consider?

3. Was this question more or less difficult for “regular” contacts than “other” contacts, and why?

4. The last question asked “what is the highest level of work-related interactions that you have with contacts. [Show question and response options] – What are your reactions to this question?

1. Does it make sense, are the question and response options clear? (If not, why not?) Are there any terms you found confusing or difficult?

2. You reported that this occupation has interactions that involve [fill]. How did you decide on this category? What sorts of things did you think about/consider?

3. Was this question more or less difficult for “regular” contacts than “other” contacts, and why?

Physical Demand Elements

Now, let’s take a look at the questions we asked about the physical demands of your current occupation.

1. First, I’d like to get a general sense of what it was like to answer these questions. If you had to give three words to describe what it was like, what words would you choose? (Follow up probes as needed)

2. [show 1^st page of definitions] Were these instructions and definitional guidelines clear and understandable to you? If not, why not?

1. We asked you to report the amount of time spent performing the physical demand on a typical work day/shift. For your occupation, does it make sense to talk about a typical work day? If not, why not? How did you decide what a typical day looks like in your job? Do you think this affected the accuracy of your responses?

3. [Show participant the list of elements] Here is a list of the physical demands that you were asked about. In general, how easy or difficult was it for you to answer questions about these elements? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult). What made it [fill]?

1. Which elements gave you the most difficulty (and why)?

2. Which elements were the easiest for you (and why)?

3. What could we do to make it easier for respondents to report accurately about the physical demands of their job (e.g., more/better formatted instructions, additional aids/resources, different collection procedures, etc.?)

4. [If not evident from Q3, ask:] How easy or difficult was it for you to estimate the amount of time this occupation spends in each of the physical demand behaviors? (Very easy, Somewhat easy, Neither easy nor difficult, Somewhat difficult, Very difficult).

1. Which elements were hardest in terms of estimating durations (and why)?

2. Which elements were the easiest in terms of estimating durations (and why)?

3. How did you go about estimating durations? What could we do to improve the accuracy of respondents’ duration estimates?

4. How precise did you think you needed to be in your duration estimates (e.g., ballpark, to the nearest hour or within a certain percent range, to the minute)?

Interviewer-Administered Debriefing

1. Would you say that the amount of time and effort that it took you to complete this survey was (Very reasonable, Somewhat reasonable, Not very reasonable, Not at all reasonable)? Why is that?

2. You were kind enough to volunteer for this study, and are being paid to help us evaluate and improve this survey. If this was not the case – if your employer simply asked you to complete this survey for the BLS, and you were NOT getting paid:

   1. How likely would you be to complete the survey (Very likely, Somewhat likely, Not very likely, Not at all likely). Why is that?

2. Do you think the information you’d provide (as an unpaid ORS respondent) would be more thorough and accurate than what you provided for this study, less thorough and accurate, or about the same? Why is that?

3. One of the things we’re hoping to learn in this study is how we can improve our procedures for collecting the data. In a minute we’ll talk more about the specific questions you were asked. But first I’m curious about your impressions of how we collected the data.

1. In your view, did the interview process go smoothly? If not, why not?

2. Did you feel that the person who interviewed you today listened and understood the information that you shared about your job? If not, why not?

3. This survey is designed to be more conversational than most government surveys. Did it seem conversational to you?

1. If so, how so – could you give me an example?

2. If not, why not?

3. Did you like this approach or would you prefer a more traditional, structured interview?

4. If your company was selected to participate in the Occupational Requirements Survey, and you were asked to provide information about the requirements of your occupation, how would you prefer to report your data:

2. Have an interviewer interview me, in person (at work)

3. Have an interviewer interview me, by phone (at work)

4. Complete a self-administered survey (paper, web)

5. Fill out a daily diary of your work tasks and job requirements (paper or web)

6. Some other method (e.g., have supervisor provide): ___________________

EXPLAIN

Ask Stylized Questions and Social Desirability debriefings questions

Debriefing - Social Desirability

1. Were there any questions (in the diary or survey) that you thought were too personal or intrusive? Which ones and why?

2. [For diary participants] Do you think there is an appropriate number of tasks that an employee in your occupation should list in this diary?

1. What is the minimum number of tasks that you feel should be reported in order to accurately capture a typical day in this occupation?

2. How much variation is there in this job in the number or type of tasks performed each day?

3. Sometimes two people looking at the same job might view it differently – for example, one person might view something as a requirement of the job while others view it as simply a choice, or they could have different estimates about the amount of time a person is required to spend doing something on a typical day. With that in mind, imagine that we had your supervisor fill out the same survey for your occupation.

   1. To what extent do you think you and your supervisor would agree on the level of cognitive demands in your occupation? (100% agreement, 75%, 50%, 25%, 0% - don’t know). Why is that?

2. To what extent do you think you and your supervisor would agree on the types of physical demands in your occupation? (100% agreement, 75%, 50%, 25%, 0% - don’t know) Why is that?

3. To what extent do you think you and your supervisor would agree on the duration of those physical demands in your occupation? (100% agreement, 75%, 50%, 25%, 0% - don’t know) Why is that?

4. In some ways, asking employees about their job requirements is a little like reading through a resume – it can be extremely informative but we also know that some people may tend to exaggerate or put a positive spin on things. This is completely normal, but it can be a challenge for a survey organization that’s trying to collect the most accurate data possible.

   1. Thinking about other employees in this occupation, how likely do you think they would be to overstate the requirements of their job? (Very likely, Somewhat likely, Somewhat unlikely, Not at all likely) Why?

2. [If Q4a is Very or Somewhat likely, ask:] What could we do to encourage respondents not to overstate their job requirements?

3. Are there situations in which you think that respondents might underestimate the requirements of their job? If so, why? Which requirements?

14