Attachment E - Cognitive Testing Results for the 2024 Leave Module

TO: Rose Woods

Office of Employment and Unemployment Statistics

FROM: Robin Kaplan and Victoria Narine

Office of Survey Methods Research (OSMR)

DATE: July 17, 2023

SUBJECT: Final Report on Cognitive Testing for the 2024 ATUS Leave Module

Introduction

The ATUS is a continuous survey that started in January 2003. The Leave and Job Flexibilities Module includes questions about workers' access to and use of paid and unpaid leave, job flexibility, and their work schedules. Sponsored by the U.S. Department of Labor Women’s Bureau, in 2011 and 2017, BLS conducted the ATUS Leave module. A 2024 Leave Module to the ATUS will offer researchers the chance to examine how alternative work schedules affect time spent in nonmarket activities such as housework, childcare, and volunteer activities and how the COVID-19 pandemic may have affected these activities.

As in the 2017-2018 module, the 2024 questionnaire will include five sections:

1. Access to Paid Leave

2. Access to Unpaid Leave

3. Leave taken in the last week

4. Job flexibility and work schedules

5. Non-use of leave

The 2024 ATUS Leave module will accomplish similar objectives as the 2017 module. Although many questions remain the same, some have been dropped, and some have been added to obtain better information about the availability and use of flexible and alternative work schedules. As in 2017, data will be collected on employees’ access to paid and unpaid leave (e.g., what types of leave are available and for what purposes, work schedules, and flexibility in work schedules).

The purpose of this memo is to summarize the Office of Survey Methods Research (OSMR) results from two rounds of cognitive testing of the newly proposed questions for the 2024 leave module. Because the new questions appear in Section 4 on Job flexibility and work schedules, cognitive testing focused on that section of the questionnaire only.

OSMR conducted cognitive interviews in Spring 2023 with 20 participants (10 participants per round). All interviews were conducted remotely via Microsoft Teams. The purpose of the cognitive interviews was to determine whether (1) the questions measure what they were intended to measure, (2) respondents can complete the survey and accurately answer these questions in the context of an ATUS interview, and (3) to make recommendations for question wording. An online survey was also conducted to complement the cognitive interviews. These participants were administered the leave module questions and similar follow-up probes (open and closed-ended) as the cognitive interview participants.

Methods

OSMR recruited 10 participants for the first round of cognitive testing and 10 participants for the second round of cognitive testing. Using an iterative design, an interim memo summarizing the findings from the first round of testing was provided. Based on the results and recommendations from the first round, some questions were dropped or revised for the second round of testing. See Appendix A, B, and C for the questionnaires used in the first and second round of testing and in the online study.

All participants indicated that they worked full or part time for the government, a private company, or a non-profit. Participants were recruited via advertisements on Craigslist and were offered a $50 incentive. OSMR interviewers administered a brief version of the ATUS diary, asking how they spent their time on the previous day. Afterward, participants answered the leave module questionnaire about job flexibility and work schedules. After completing the questionnaire, participants answered debriefing questions about their experience answering the leave module questions.

Online testing from a nonprobability panel with 404 participants was also conducted. Participants were asked the same questions using similar probes to the cognitive testing study, but in a self-administered format. Results from the online testing are also included to help support the recommendations.

As with any qualitative pretesting research, there are limitations to the conclusions that can be made from this study. The current study uses a convenience sample that is small and not representative of the survey population, and so generalizations should be made with caution.

Participants

Round 1

A total of 10 participants were included in the first round of testing. The average age of the participants was 38 years old, with a range of 24 to 59 years old. A total of 9 participants reported being employed full-time and 1 was employed part-time. Participants represented a mix of different occupations, salaried workers, hourly workers, and shift workers.

Round 2

A total of 10 participants were included in the second round of cognitive testing. The average age of the participants was 39 years old, with a range of 26 to 63 years old. A total of 7 participants reported being employed full-time, 3 were employed part-time, and represented a mix of different occupations. In round 2, hourly and shift workers were prioritized in recruitment to complement the overrepresentation of salaried and non-shift workers recruited in round 1. Table 1 compares the demographics between both rounds of testing.

Table 1. Participant demographics in both rounds of cognitive testing

	Gender		Employment Type		Worker Type		Shift Work		Telework
	Female	Male	Part-time	Full-time	Salary	Hourly	Yes	No		Yes	No
Round 1 (n=10)	8	2	1	9	6	4	3	7		6	4
Round 2 (n=10)	5	5	3	7	3	7	7	3		1	9
Total	13	7	4	16	9	11	10	10		7	13

Online Study

A total of 404 participants completed the online survey. The average age was 40 years old, with a range from 20 to 75 years old. All participants (49% female) reported working full-time (83%) or part-time (17%) and were not self-employed. Participants represented a mix of different occupations, industries, and whether they were paid by the hour (60%) or salaried employees (40%).

General Findings – Cognitive interviews

1. Length and timing of the module

ATUS supplements are required to take an average of five minutes total to administer. The first round of testing took five minutes to administer, with a median of 7 minutes, and a range of 5 to 10 minutes. In the second round, the module took an average of 6.5 minutes to administer, with a range of 4 to 11 minutes. Timing data should be interpreted with caution, as participants in the cognitive testing frequently elaborated on their answers during question administration, and this is less likely to occur during production.

2. Question difficulty

Most participants did not find the leave module questions difficult to answer during question administration. However, participants showed some comprehension issues with concepts included on the leave module. For example:

Confusion about the concept of “work schedule”

Several questions referred to “work schedule.” Work schedule refers to the days and hours worked, with most questions asking about the times respondents begin and end work. Despite this, several participants answered this question in terms of their work calendars or itemized schedule for the day, that is, their individual meetings, tasks, site visits, etc. that they perform in a typical workday. This confusion was only uncovered during debriefing when participants described how they arrived at their answers.

Confusion about paid versus unpaid work and work schedules

For several questions, salaried employees, versus those who were paid hourly, tended to have different interpretations of some questions, including those about employers changing work hours, being available to work extra hours, and performing additional work outside of scheduled work hours. These questions also took the longest to administer, with lengthy question stems and several response options. The confusion stemmed from most salaried employees conceptualizing the nature of their work as “working until the job is done,” and considered finishing or catching up on work outside of normal business hours to be part of their job and their salary. The hourly and shift workers indicated that any extra time they worked would be paid. For these reasons, the questions did not resonate with salaried workers, especially those working remotely or who telework often. They noted these questions seemed tailored more to those who work shifts.

3. Question sensitivity

Participants generally did not find the questions sensitive or personal. One participant mentioned the question about whether they are required to do any work at home was personal for them.

4. Other issues

Reference periods:

Several questions had different reference periods asking about the last week or month. Participants often did not pay attention to these reference periods, instead answering about the entire period they worked at the job, in general, or on a typical week or month at their job. Some participants indicated the answer would not differ if asked in general versus the last week or month. However, depending on the analytic needs of each question, this could potentially introduce measurement error.

Expanded flexibilities:

Several participants mentioned having expanded flexibilities at their workplace since the COVID-19 pandemic, including the ability to work at home (which they were not offered prior to the pandemic), hybrid work schedules, increased telework, hoteling, and downsizing of the office space such that employees are required to work from home some of the time. Sometimes participants conflated the telework policy at their job with their work schedule, that is, the days and hours they work. Another issue to consider with these expanded flexibilities is the ability to work remotely – at least two participants explained there is no physical office to report to (i.e., they are remote workers).

Most participants thought of work at home, but some mentioned they might work from another location, such as a family member’s house, a hotel while they are on vacation, or a café. Using broader language to capture telework that is done outside the home might be beneficial, for example, the new CPS questions about telework ask, “Did you telework or work at home for pay?” These expanded flexibilities seemed to color participants’ interpretation or response to several questions about work at home.

Specific Findings

This section details the findings for questions tested in Rounds 1 and 2, including the question wording used and the frequencies for each response option. Results from the online study are also included where applicable.

JF_1

Do you have flexible work hours that allow you to vary or make changes in the times

you begin and end work?

1. Yes [Go to to JF_1A]

2. No [Go to to D1]

Don’t know, Refusal [Go to D1]

Findings:

Response frequencies by round:

	Yes	No
Round 1 (n = 10)	5 (50%)	5 (50%)
Round 2 (n = 10)	6 (60%)	4 (40%)
Total	11 (55%)	9 (45%)

This question was generally well understood in both rounds of cognitive testing. Several participants did not answer “yes” or “no” to the initial question administration, requiring the interviewer to repeat the question or probe for a yes or no response. Several participants wanted to elaborate on their answer, saying “it depends,” or “within reason,” or “I can make slight changes.” At least one participant initially said yes, but then explained, “It’s not really up to me to change the hours,” and changed their answer to no. One participant explained he creates his own work flexibility by sometimes working through breaks and leaving a little early at the end of the day. Some participants also had different conceptions of what they considered flexible work hours. Some said they could ask their manager to make changes, whereas others indicated they could mostly make their own hours.

A couple participants mentioned that they have inherent flexibility in their work schedule due to time zone differences between members of their team, or due to meetings they must attend. Sometimes they might start work earlier or later to accommodate these differences but may take a break during the day or leave earlier another day to compensate for this.

We probed about whether the answer to this question had changed after the COVID-19 pandemic began. A few participants mentioned increased ability to telework (or going fully remote during the pandemic), with hybrid workplaces, where they telework some days and report to the office on other days. Some participants mentioned telework was unavailable to them prior to the pandemic. A couple participants mentioned the square footage of their company’s office space was significantly reduced after the pandemic, or the office took on a hoteling model, meaning that employees are required to telework some part of the week since the new office space could not accommodate all employees at the same time.

In the online study, of the 398 participants who responded to this question, 66% answered “yes” and 33% answered “no.” Of those, about 30% of participants indicated this had changed since the pandemic. When those participants were administered a follow-up question asking them to select all of the reasons for this change, 36% indicated they had switched jobs during that time, and 79% indicated that their employer policy on work flexibility had changed. This provides more evidence that workplace flexibility has likely changed since the start of the pandemic.

Recommendations: Only minor problems were observed with this question. No changes to question wording were recommended.

JF_1A

Can you change the times you begin and end work on a frequent basis,

occasionally, or only rarely?

1. Frequent basis

2. Occasionally

3. Rarely

Don’t know, Refusal

Findings:

Response frequencies by round:

	Frequent Basis	Occasionally	Rarely	I am entirely free to decide when I start and end work
Round 1 (n = 5)	2 (40%)	3 (60%)	0 (0%)	0 (0%)
Round 2 (n = 6)	3 (50%)	1 (17%)	2 (33%)	N/A1
Total	5 (45.5%)	4 (36.4%)	2 (18.2%)	0 (0%)

This question worked well in both rounds of testing, and participants did not display any major issues with comprehension. All participants were able to arrive at an answer, and indicated understanding the terms frequent basis, occasionally, and rarely. In the online testing, of the 259 participants who received this question, 35% answered “frequent basis,” 56% answered “occasionally” and 9% answered “rarely.”

Recommendations: No major problems were observed with this question. No changes were recommend to question wording.

JF_2

Is your flexible work schedule part of a formal, written program or policy offered by your

employer, or is it just an informal arrangement?

1. Formal program or policy

2. Informal arrangement

Don’t know, Refusal

Findings:

Response frequencies in Round 1:

	Formal Program or Policy	Informal Arrangement
Round 1 (n = 5)	1 (20%)	4 (80%)
Total	1 (20%)	4 (80%)

In the first round of testing, participants had some problems answering this question related to comprehension of formal versus informal, conflating their written telework or remote work arrangements with policies about their scheduled days and hours they work, and judgment of whether their work schedule would fall into a formal versus informal arrangement.

Comprehension issues:

As mentioned, the concept of work schedule was confusing for many participants. We found that several participants included their remote or telework agreements in their responses to this question. However, the question does not ask whether they have a formal or informal remote/telework arrangement. Some participants seemed to conflate their work schedule in terms of the days and hours they work with their telework or remote work arrangements, or the hours worked versus the location in which they worked. During the debriefing, these participants often wavered on their response upon seeing the question again or wanted to change their original response. This seemed to occur because the remote/telework arrangement could impact their schedule:

• “I’d say informal because remote workers have flexible options. At first, I was thinking more in terms of the questions about work hours and flexibility of hours, and that part of it is more informal. What’s throwing me is whether it’s asking if my work schedule is formal or informal, or whether the remote work itself is formal or informal.”

• “When I first answered, I thought of the work schedule in terms of when I have to start and stop work, but now I might change my answer because I am considered a part-time telecommuter I can decide when I go into the office and when I work at home. So the flexibility of me choosing where I work is codified in our policy.”

Other participants pointed out that the definition of formal and informal seemed subjective to them, and they weren’t sure what these concepts meant:

• “Everyone's definition of formal and informal is different. Would it make sense to say informal is just a chat or phone call with your employer, and formal means you made a change with HR? Formal for me is something that needs to be written, like email. As soon as my employer asks for something in writing, it becomes formal.”

• “I’m not 100% sure about that one. I guess maybe it could be an informal arrangement, I don’t know what the difference between them is. I was thinking about a policy or rule.”

• “I based my answer on the form of communication used to make changes to my schedule, if I use a Teams chat that is informal. Because it’s more a heads-up than asking for permission, that makes it informal.”

We probed about whether this changed after the COVID-19 pandemic. Some participants noted that the start and end times of their work did not change, but the telework policy did.

Because of the problems listed above, this question was dropped from the module and was not tested in round two or in the online study.

D1

Which of the following statements best describes how the times you start and END work are decided at (your main job)? <randomize to go 14 or 41>

1. Start and end times are decided by my employer and I cannot change them on my own.

2. Start and end times are decided by my employer but with my input.

3. I can decide the time I start and end work, within certain limits.

4. I am entirely free to decide when I start and end work.

5. Don't know/refuse

Findings:

Response frequencies in Round 1:

	Start and end times are decided by my employer and I cannot change them on my own	Start and end times are decided by my employer but with my input	I can decide the time I start and end work, within certain limits	I am entirely free to decide when I start and end work
Round 1 (n = 10)	4 (40%)	2 (20%)	4 (40%)	0 (0%)
Total	4 (40%)	2 (20%)	4 (40%)	0 (0%)

This question was tested in the first round of cognitive testing. Most participants seemed to understand what this question was asking and could arrive at an answer. However, problems with comprehension and response formation occurred. This question was also lengthy to administer due to the number and length of the response categories. Many participants also needed the question stem or response options repeated several times, indicating issues with comprehension.

Comprehension problems:

A couple participants expressed confusion about how to answer because they provided input on their work hours when they were hired, but now work a set schedule:

• “It’s not clear what is being asked, do you mean when I started the job or now?”

• “This one was a bit confusing. I kind of wanted to pick 2 because I did initially have input before accepting the position. If my availability worked with what the employer wanted, then I get the position. So I had a bit of input. But I went with 1 because now that I have the job, this is what I'm stuck with for now.”

• “A typical shift at this job would have been 8 hours. Because of the time my kids get on the bus to go to school, I could not work those hours. So we were able to work together. I start work half an hour later than the person who worked here before me and I leave half an hour earlier than she left.”

One participant noted she does not have a formal start time, and she sometimes works late to accommodate for meetings:

• “I don't have a technical start time. We have normal business hours but sometimes it's later, sometimes it's earlier. I can work those hours within certain limits is, I can't come in at midnight to do work because there isn't a business need. Normal business hours are defined as 8AM - 5PM, but we have reasons to work later, say, until 7PM or so.”

One participant noted that she was unclear on who controls the schedule. She explained changes are often made at a co-worker’s request to attend a meeting or cover for another meeting that might take place outside of normal business hours due to time zone differences:

• “I was confused about how much is dictated by the employer and how much control I have. There are sometimes last-minute changes, but co-workers ask to move meetings and I schedule around that. It’s not my employer making these changes, it’s the nature of job. And similarly, I work with a global team with colleagues on a 7- or 8-hour difference in time zone. Is that my work schedule or the nature of the work?”

Response formation problems:

Several participants had difficulty distinguishing between categories 2 and 3. For instance, participants who have some flexibility in their schedules tended to select response option 3, but often wavered between response options 2 and 3 in the debriefing. For example:

• “Maybe I would lean more toward #2 after seeing the question again, because I sometimes leave at 4:45pm (instead of 5pm).”

Participants mentioned the “limits” on deciding their work schedule tended to be making sure they are available for meetings or when their co-workers are working, working typical 9 to 5 “business hours,” plus or minus a couple hours on either end, or not asking to work a night shift, for example.

Participants who had rigid schedules or did shift work tended to choose response option 1 or 2 (and sometimes wavered a bit between them) if they had appointments or needed to start or leave slightly earlier or later than usual. For example:

• “They have a set time, I work 4am to 12pm, that's decided by the employer, if you need to change it, you can come in 2 hours late. Or if you needed to start at 5am every day, they would be okay with that.”

Participants tended to say they would imagine freelancers would select response option 4, and no participants chose this option.

Recommendations:

Because this question did not perform well, we recommended that the original question from the 2017 leave module (JF_3) replace this question. This question was dropped from the leave module and was not tested in the second round of cognitive interviews or in the online study.

JF_3

Universe: JF_1 = 2, DK, RF

Do you have any input into the hours you begin and end work or does your employer decide?

1. Worker has some input

2. Employer decides

3. Other

Don’t know, Refuse

Findings:

Response frequencies in Round 2:

	Worker has some input	Employer decides
Round 2 (n = 4)	0 (0%)	4 (100%)
Total	0 (0%)	4 (100%)

In the second round of cognitive testing, participants understood this question well and no major issues were observed. Participants usually knew the answer to this question because they reported working a set schedule assigned by their employer:

• “I work for an organization, and they have rules and principles and a time frame. When they hired me, they gave me a job description with the times I have to work. So, my employer decides everything that I do and don't do at work, so I’d say my employer.”

• “Because my employer decides totally. Except I'm taking some days off for a very good reason. But my employer has the say. It wasn't difficult [to decide between 1 and 2].”

• “Because I know the working hours that they have. They do business from literally 9-5 and that's when they need me. That's how they decide. I can't go away from that, unfortunately.”

A shift worker described being given hours to work by her employer:

• “They don't ask you what you want. They tell you what's available.”

In the online testing, of the 133 participants administered this question, 26% selected “I have some input” and 72% selected “Employer decides.”

Recommendations: No major problems were observed with this question. No changes to question wording were recommended.

JF_4

How far in advance do you know your work schedule (at your main job)?

*Interviewer instructions: if the respondent reports that he has a set work schedule, it is

permissible to record “4 or more weeks” without any additional probing.

*Interviewer probe if needed: By work schedule, we mean the days and hours you work at your [current/main] job.

1. 1 day or less

2. 2-3 days

3. 4-6 days

4. 7-10 days

5. 11 days to 4 weeks (not including 4 weeks)

6. 4 weeks or more

Don’t know, Refusal

This question remained the same as in the 2017 leave module, but new response categories were added to capture “just in time scheduling.”

Findings:

Response frequencies in Round 1:

	Less than 1 day	2-3 days	4-6 days	7-10 days	11 days to 4 weeks (not including 4 weeks)	4 weeks or more
Round 1 (n = 10)	1 (10%)	1 (10%)	0 (0%)	0 (0%)	3 (30%)	5 (50%)

In round one of cognitive testing, this question was relatively straightforward and well-understood for participants with a set work schedule and little variability. They tended to fall under option 5 or 6 because their schedule did not really change. The new response options were well-understood but added length to the survey. We observed several problems with comprehension of this question. Some participants needed the question repeated several times, adding length to the questionnaire.

This confusion stemmed from differential comprehension of the term “work schedule” in the question. Although an optional interviewer probe was provided defining the work schedule, several participants answered the question on the basis of the individual tasks, meetings, site visits, or other events on their daily calendar to be their “work schedule” when the question was designed to capture how far in advance they know the days and hours they work. The confusion was not discovered until the debriefing. For example:

• “I was thinking, 'Is she referring to my general working hours of the day, or is she referring to what my day to day schedule would be on any given day?’”

• “I just thought about reoccurring meetings, and webinars are decided weeks in advance.”

Participants who might have meetings pop-up in another time zone tended to think more about their scheduled meetings and how this may affect their work hours, for example:

• “Before the end of the day, I'll look at my calendar for next week and see which meetings are already on the calendar. And in reference to the west coast folks setting up meetings in their time zone. But typically, it's going to be the week before where I can see if the west coast people scheduled meetings.”

Recommendations:

Due to the confusion about the meaning of work schedule, we recommended observing in round 2 of cognitive testing whether reading the definition of “work schedule” as part of the question stem would improve comprehension. The revised version of the question is below.

JF_4

How far in advance do you know your work schedule (at your main job)? By work schedule, we mean the days and hours you work at your [current/main] job.

*Interviewer instructions: if the respondent reports that he has a set work schedule, it is

permissible to record “4 or more weeks” without any additional probing.

<DO NOT READ RESPONSE OPTIONS>

1. 1 day or less

2. 2-3 days

3. 4-6 days

4. 7-10 days

5. 11 days to 4 weeks (not including 4 weeks)

6. 4 weeks or more

Don’t know, Refusal

The above question was tested in the second round of cognitive testing and in the online study. The definition of work schedule was read aloud or presented to participants in the question stem.

Findings:

Response frequencies in Round 2:

Less than 1 day	2-3 days	4-6 days	7-10 days	11 days to 4 weeks (not including 4 weeks )	4 weeks or more
2 (20%)	2 (20%)	1 (10%)	1 (10%)	0 (0%)	4 (40%)

In round two of cognitive testing, this question was easy for participants who had set schedules or always knew their schedule well in advance. However, participants with more variable schedules had some difficulty arriving at an answer. For example, the interviewer would often have to probe when respondents gave an answer like “3-4 days” and ask them to choose if it’s closer to help select between response categories 2 and 3.

Some participants gave an answer during the interview, then changed their response after seeing the categories during the debriefing. For example, one participant initially answered, “at least 24 hours” and then after seeing the response categories, clarified that she has access to the schedule 4-6 days in advance, in between the weeks when the shifts rotate. Another participant indicated that he had a set schedule, and the response was coded as category 6, but upon seeing the categories, he changed his response to “7-10 days.”

As mentioned, in the first round of testing, there was a misinterpretation of the term “work schedule” so in round 2, the definition of work schedule was read by default to all participants. While most participants interpreted this as intended, several participants showed variability in their interpretation of the term “work schedule” despite the definition being read. One participant thought the question was asking about which days and hours he worked (likely due to the definition of work schedule being read). After repeating the question, and seeing it written during the debriefing, he seemed to understand what it was trying to collect. Another participant seemed to include the time preparing for work as part of the “work schedule,” then later noted this would be excluded from their total hours worked.

At least two participants had a broader interpretation of the term “work schedule” than was defined in the question. Similar to round 1, one participant conflated the work location or flexibility inherent in working from home as part of their work schedule. Another included activities during the workday in her answer. This tended to happen more for those with very flexible schedules. They tended to rely on information about the work tasks they do day-to-day, such as meetings or phone calls on their work calendar. For instance, one participant said the work schedule includes the location of work: “It means where I'll be working, whether I'll be working, if I'm going to be with a client. This week, I had a few Zoom meetings and Teams meetings so I blocked those times off so people wouldn't bother me.” This participant indicated that she marks her work schedule on a joint calendar, which indicates the days, hours, and location she will work, which may explain why the work location was included in her answer. Similarly, another participant indicated that the work schedule would include: “The things I have to do at work. The activities I have to do during the day. And the hours I have to spend at work. And it can also include the work that is done during work days. And the leave days. It's more than that, not just starting and end time. And everything you do after work because it affects what you do at work.”

Most participants in the online study correctly understood the work schedule as the days and times they worked, providing responses such as:

• “I define my work schedule as my normal hours worked Monday through Friday.”

• “The hours and days I work every week.”

• “I teach at a school, so the times for everything are very regimented. It's strictly 8:00AM-4:00PM, unless there's a rare after school activity.”

Online participants who answered that they have 3 days or less of advance notice were asked for the reasons for this. Answers included the nature of the job (e.g., healthcare) where shifts vary and can be unpredictable, working on a small team and needing to be flexible, the work being unpredictable or to accommodate busier times, or having control of one’s own schedule and making changes at the last minute based on workload.

Recommendations:

Because some misinterpretations of the term “work schedule” was observed despite reading the definition in the cognitive testing, but this was not observed in the online study, we did not recommend any additional changes to this question.

C5

In the past month, did your employer change the timing or the length of your SCHEDULED WORK HOURS at your [main/current] job by having you…

1. Start work earlier than scheduled? [yes/no/don’t know/refuse]

2. Start work later than scheduled? [yes/no/don’t know/refuse]

3. End work later than scheduled? [yes/no/don’t know/refuse]

4. End work earlier than scheduled? [yes/no/don’t know/refuse]

5. Add a shift? [yes/no/don’t know/refuse]

6. Cancel a shift? [yes/no/don’t know/refuse]

This was a new question tested in the first round of cognitive testing. The question was designed to collect information on how workers experience last minute changes, and whether these changes generally result in more work, less work, or the same amount of work with just schedule adjustments.

Findings:

Response frequencies in Round 1 (n=10):

Start work later than scheduled?		Start work earlier than scheduled?		End work later than scheduled?		End work earlier than scheduled?		Add a shift?		Cancel a shift?
Yes	No	Yes	No	Yes	No	Yes	No	Yes	No	Yes	No
2 (20%)	8 (80%)	3 (30%)	7 (70%)	3 (30%)	7 (70%)	0 (0%)	10 (100%)	1 (10%)	9 (90%)	0 (0%)	10 (10%)

Participants generally understood what this question was asking and could arrive at an answer. However, the question is a series of 6 yes/no items, which added significant length to the interview. Several participants interrupted during question administration or needed the full question stem, reference period, or certain response options repeated multiple times. Participants who were salaried employees remarked that response categories 5 and 6, which ask about adding or canceling a shift, did not apply to them because they don’t do any shift work. These participants found the question a bit out of place for them since last minute changes to their scheduled hours and shift work do not occur – the only exception might be traveling for work. For example:

• “For me my work schedule is always fixed, it's not based on different hours or shifts. There's sometimes an event or work travel that might affect work, but the typical work is always the same time and days of the week, there's not much that could be changed. A nurse or someone in retail may have more variance.”

We also observed some problems with response formation. For example, several participants seemed to miss the portion of the question stem specifying that the employer made the change to the work schedule and mentioned changes that they had requested:

• “I started late one day, but this was due to my schedule. I had to request this.”

A couple of participants described ambiguous situations where they were unsure whether the answer should be yes or no. For example, one participant caveated that her employer does not control these schedule changes, but rather it’s “a negotiation with her colleagues” as to who covers which meetings. Another participant remarked that she was not required to stay late one night during the prior week, but “there was a meeting last night at 5pm- I could have gotten out of it. I felt I needed to be there; it wasn’t mandatory but important.”

A couple participants described these variations in their schedule as within their control, not the employers. Participants described them as being part of their job or the nature of the work they do. For example:

• “I'm thinking of the board meeting. My employer required that the length of my work hours end later than the norm as a result of the board dinner that I was responsible for organization and that I needed to attend. I see that as a requirement of my employer. If the operative was 'did the change come about in the last month?' then no, because I set the date of the evening dinner. But the fact that I worked later than what is the normal schedule of what is required by my employer and the nature of the job -- I had a later than usual schedule once this past month because of my job.”

• “If I did have to start a day earlier, it wouldn't be a result of my employer directing me to start my day earlier, or to end my day later. If there was any variation in my workday, it was because of my autonomy to schedule a meeting earlier in the day.”

• “Last week when I worked a little bit later, that was my own accord because I wasn't able to finish something. But that was on me. Nobody asked me to do it.”

A couple of participants who had indicated yes to some of these items explained that they were asked to either come in early, stay late, or were offered additional shifts. No participants indicated they had an early dismissal or had a shift canceled. For instance, one participant was a social worker who sometimes had to start or end late to meet with families. Another participant worked in a factory and often picks up extra shifts voluntarily to earn overtime. He described this situation as more prevalent during certain seasons, like the holidays and summertime.

Recommendations:

We recommended that this question either be cut or shortened due to the length of time it took to administer. To make sure the question works well for both salaried and hourly workers, we also recommended that the question remove reference to adding or canceling a shift. As a result, a revised version of the question was tested in the second round of cognitive testing and in the online study; those findings are described below.

C5 In the last month, did your employer require you…

1. To change the time you started or ended work? [yes/no/don’t know/refuse]

2. To work more hours? [yes/no/don’t know/refuse]

3. To work fewer hours? [yes/no/don’t know/refuse]

The revised version of C5 above was tested in the second round of cognitive testing and in the online study. The revised question was shorter and consolidated the individual items about starting or ending work earlier or later and the items on shift work.

Cognitive Interview Findings:

Response frequencies in Round 2 (n=10):

To change the time you started or ended work?		To work more hours than scheduled or planned?		To work fewer hours than scheduled or planned?
Yes	No	Yes	No	Yes	No
4 (40%)	6 (60%)	5 (50%)	5 (50%)	0 (%)	10 (100%)

In round two of the cognitive testing, participants expressed confusion answering this question. Some participants thought that items 2 and 3 were mutually exclusive; that is, if they answered “yes” to item 2, they were confused about why item 3 was asked. They assumed if they worked more hours, that would imply they could not have also worked fewer hours. Some asked for the question to be repeated.

The question was straightforward for those with rigid schedules without much variability:

• “The moment the shift is over you must leave, in shift work there are rules and regulations.”

• “My hours are 9-5. I’m on a strict time frame. The only time my hours vary is when I take leave. But sometimes I work overtime but it’s rare – not in the last month.”

• “I just know I do my 40 hours, I badge in and out, and I’m done.”

Some participants seemed to pause and had to think about the past month and whether any of these situations happened. This seemed to occur during administration of the second item, “to work more hours.” This also occurred with participants who had more variability in their schedule, either due to being asked to cover a shift for a colleague in the last month, or due to a busier season at their job. These changes were sometimes requested, and not required:

• “Unexpectedly, someone called up. Couldn’t get in touch with that person so supervisor asked if I could cover. I was never asked to work fewer hours. This was voluntary. Even though they ask you if you can cover, you can say no.”

• “In the last month, there has been a larger workload. And the other month worked fewer hours. More work hours this month made up for fewer hours last month. These changes were voluntary, accepting request to work more hours. No punishment if I didn’t accept.”

• “Because I’ve had to change the time I started and ended work. And I’ve worked more hours recently.”

A few participants mentioned that they worked more hours but this was not a requirement of their job:

• “Because my employer did not ask me to work more than I was required. Although I did spend more hours at work, but not due to my employer, due to me trying to help out with something.”

• “Yes, but the way it’s worded sounds more severe than my actual life. They didn’t require me to do that. I had to take a meeting, but they didn’t require me to do that. I didn’t imagine my employer yelling at me saying do that. We have clients and vendors in Europe and have to hop on a call at 6 or 7 in the morning. It’s not like it’s a demand. It’s part of my job and it’s what I have to do.”

• “I was asked to do this because my colleague made some mistakes and wanted me to correct some of his errors. So I had my schedule changed to cover for him. It was voluntary.”

Most participants answered the question based on the hours they usually work or were actually scheduled to work. One participant with a very flexible schedule noted that she answered the question based on the need to meet with a client at 7am, which is not something she would normally do. She saw this as part of her job.

Online Study Findings:

In the online testing, participants were asked the same series of questions with a follow-up for the reasons for any change(s) and whether they were required or voluntary. The table below shows the distribution of responses.

In the last month, did your employer require you to… (n=383)	Yes	No
Change the time started/ended work	31.0%	69.0%
Work more hours	42.0%	58.0%
Work fewer hours	25.0%	75.0%

A total of 40% of participants reported their employer was their direct supervisor or manager, and 60% reported their employer was not an individual person but the company or organization in which they work.

Online participants who answered “yes” were eligible for the following follow-up question, asking for all the reasons for the change(s):

What were the reasons for this change? Select all that apply. (n=208)
Finish or catch up on work	44.7%
Filled in for another worker	30.0%
Employer added a shift	22.1%
Employer canceled a shift	9.6%
More workers were needed	31.7%
Fewer workers were needed	12.0%
Other (please specify)	11.1%

Examples of “Other” responses included:

“Semester ended.”
“Overtime was taken away due to financial reasons.”
“Changed times due to conflicts with my fiancé’s work schedule.”
“The earlier shift was required due to a lack of males.”
“Was waiting for occupancy license from the state.”
“Not enough work to do.”
“Our catered events went longer than planned.”
“There are often mistakes in the schedule leaving unfilled shifts.”
“Overtime on Saturday.”
“Had personal issues to take care of.”
“Special event happening at work required me in earlier.”
“Had to be flexible to others schedule.”
“Holiday and less work.”
“There was a promotional event for our company that needed my specific skills.”
“Low workload within my contracted project.”
“Specific promotional events were happening during this time that are under my job description.” Online participants then answered a follow-up question on whether this was a requirement or not:
Was this change a requirement of your job, or voluntary? (n=207)
Required	61.3%
Voluntary	38.7%

As seen above, around half of participants were asked to change their hours in some way. The top reason seemed to be to finish or catch up on work. Importantly, almost 40% of participants indicated the change was voluntary, not required. If the changes were required, we’d expect 100% of participants to select “required.” This was similar to the cognitive testing results, where participants indicated some of the changes were not technically a requirement of the job.

Recommendations: We recommended cutting this question due to measurement error, and it was dropped from the module. Some participants thought items 2 and 3 were mutually exclusive. Participants in both the cognitive testing and online study included voluntary changes in their answers. Although the question asked for required changes, some changes seemed to have been voluntary or “soft requirements,” where respondents see the change as part of their job. Participants may vary in their interpretation of what’s required; on one end of the spectrum, it may mean being fired from a job if they refused, on the other end, it may have been a request to work additional hours. This question as worded is likely to capture a mix of required and voluntary schedule changes.

B7ALT1

Some workers may keep their personal schedule open to work additional hours or shifts. How often do you keep your personal schedule open or cancel personal plans in order to be available for additional work at your main job?

Interviewer probe if needed: Personal schedule refers to your free time outside of working hours. Include plans you canceled or left early in your response.

<READ OPTIONS>

1. Every day

2. At least once a week (but less than every day)

3. At least once a month (but less than once a week)

4. Less than once a month

5. Never

Don’t know, Refusal

This was a new question added in the first round of cognitive testing to get at the concepts of the imposition of work even when not working, workers feeling they always have to be at the ready (perhaps to the exclusion of other non-work activities), and work spilling over into personal time.

Findings:

Response frequencies in Round 1 (n=10):

Every day	At least once a week (but less than every day)	At least once a month (but less than once a week)	Less than once a month	Never
1 (10%)	2 (20%)	0 (0%)	3 (30%)	4 (40%)

Most participants seemed to understand what this question was trying to get at and understood the concept of “personal schedule” well. However, there were problems with comprehension and measurement error associated with this question. These issues were mostly related to salaried versus non-salaried workers having different interpretations of the question, leading to problems with response formation. Due to the length of the question stem and the response options, it also took significant time to administer.

Problems with comprehension:

Salaried versus non-salaried employees had different conceptualizations of what this question was asking. For example, some salaried workers considered it a part of their job and compensation to occasionally leave their schedule open to catch up on work outside of their normal business hours during busy times:

• “If there's a project that has a deadline and I see that we're pushed for time, I'll work in the evening to move it along.”

• “I will do it, but it would be on my own accord to finish or catch up on work, especially if I took a break for childcare during the day.”

• “I keep my personal schedule on Thursday afternoons to go on a walk with my neighbor and friends during lunch. But sometimes that has to be postponed or cancelled outright because of a meeting. I'm not going to convey that my personal plans interfere with the call.”

In contrast, other salaried employees didn’t think this question related to them because they wouldn’t be paid for this additional work. They felt this should be asked of someone working wage, shift work, or night shift work because the option to pick up extra work wasn’t really available to them in the first place:

• “I am a full-time salaried employee. I don't get paid overtime. So I don't pick up additional shifts.”

• “My hours are set so even if I wanted to make extra money, I can't do that. My hours are my hours and I know ahead of time. My hours are from 8:30AM - 4PM and I never make plans during those times. It's not like at a restaurant when you may pick up a shift.”

• “So I think it's asking...I think it relates to someone working in a position that's a wage position, or you could work a different shift -- a day shift or a night shift. But, in other words, someone who's potentially looking to do overtime. And that doesn't relate to my job because we don't have that. The only thing like that that ever comes up for me would be if there's work travel, or a conference that would be an addition or an exception to my regular schedule. But it wouldn't be an ongoing option to pick up additional work for pay.”

Several participants mentioned this type of situation rarely occurs for them because their employer promotes work/life balance, they try to avoid their work spilling over into their personal life, or because they know their schedule in advance and do not make plans during working hours.

Problems with response formation:

A couple participants who work shifts mentioned policies where they could indicate open availability or “voluntary time on,” where their employer is aware they are generally available to take on additional shifts.

One participant had problems answering this question because he chose to have an “open availability policy,” that is, when he was hired, he informed his employer he would have open availability to take on additional shifts as needed. Thus, his answer to his question would change depending on whether it was asking how frequently he keeps his schedule OPEN versus how frequently he cancels plans, revealing a double-barreled component of the question:

• “I guess like maybe it sounds like they want to know how often you keep your availability open. I hadn't really thought about the second part, how often you keep your personal plans open for your job. For me, it seems like two different questions. It's asking how often you keep your schedule open and how often you cancel personal plans for it. Yeah, if they asked me if I kept my personal schedule open to work I would say yes. But if they asked me how often I cancelled personal plans, then I would say once a month.”

Similarly, one participant who did shift work as a Certified Nurse Assistant indicated that the answer to the question would be “no” for her main job. However, during debriefing she indicated that she uses an App to pick up additional “per diem” shifts from a different employer about once per week. She keeps her schedule open to pick up these shifts but would not cancel plans to do so.

A couple participants also indicated they wanted to answer “rarely,” but this wasn’t a response option. They didn’t want to say “never” but the next closest response option of less than once per month might inflate the frequency in which this occurs:

• “It’s rare that I let my personal life affect work life. I do cancel things if I find out that I have to work. If I'm invited to something, I may have to decline based on my work schedule.”

• “For me, I can't say never because I have in the past. But it's less than once a month because it's not very frequent at all.”

Finally, several participants pointed out that it was unclear whether keeping their schedule open was a requirement of the job, or asking about something they would do voluntarily. Depending on whether the change is supposed to be voluntary, involuntary (or both), responses were different for salaried employees and seemed to measure voluntary "catching up" on emails in the evenings. For everyone else, it captured paid overtime or extra shifts.

Recommendations:

We recommended dropping this question due to length, problems with comprehension, inconsistent interpretation by salaried workers, and difficulty with response formation.

ALTJF_56

On your [main/current] job, do you USUALLY work days, evenings, nights, a rotating shift, a split shift, an irregular schedule, or some other shift?

0. A day shift – most work is done between 6 a.m. and 6 p.m. <GO TO JF_8 / B7ALT2>

1. An evening shift – most work is done between 2 p.m. and midnight

2. A night shift – most work is done between 9 p.m. and 8 a.m.

3. A rotating shift – hours change periodically from days to evenings or night

4. A split shift – hours consist of two distinct periods each day

5. An irregular schedule

6. Some other shift

Don’t know, refusal

Cognitive Interview Findings:

Response frequencies by round:

	Day shift	Evening shift	Night shift	Rotating shift	Split shift	Irregular schedule	Some other shift
Round 1	9 (90%)	0 (0%)	0 (0%)	1 (10%)	0 (0%)	0 (%)	0 (%)
Round 2	4 (40%)	0 (0%)	1 (10%)	3 (30%)	2 (20%)	1 (10%)	0 (0%)

This question performed well and no problems were observed in the first round of cognitive testing. Participants generally understood this question and were able to find a response option that fit their schedule.

In the second round of cognitive testing, however, more issues were observed. One participant answered about his second (not his main) job, and then changed his answer during debriefing. Another participant who had a lot of flexibility in selecting her work hours choose “an irregular shift,” despite reporting doing most of her work during the day with some evening work:

• “I think it's pretty evident, every day is different for me. It's within the confines of a day shift, but it can bleed into the evening. So, I wouldn't say every week is the same, so that's why I picked irregular shift. I usually work in the daytime with some evening work, and that doesn’t happen every day or evening, which is why it’s irregular.”

In addition, the new question stem makes the question take longer to administer than in previous leave modules, where a forced-choice question about working a day shift or some other shift was administered, and only if the answer was “some other shift” would respondents select from the alternative shift types.

Online Study Findings:

In the online study, participants provided the following answers, with most selecting a day shift (72%), and almost 8% selecting an irregular shift:

On your main job, do you USUALLY work days, evenings, nights, a rotating shift, a split shift, an irregular schedule, or some other shift? (n=383)
A day shift – most work is done between 6 a.m. and 6 p.m.	72.3%
An evening shift – most work is done between 2 p.m. and midnight	5.5%
A night shift – most work is done between 9 p.m. and 8 a.m.	5.5%
A rotating shift – hours change periodically from days to evenings or night	5.5%
A split shift – hours consist of two distinct periods each day	1.3%
An irregular schedule	7.8%
Other (please specify)	2.1%

Recommendations: Because cognitive interview participants and online study participants may have under-selected “day shift”, when most of their work was done during the day, we recommended using the wording from the 2017 leave module with a forced-choice lead-in about day shifts. This was also expected to save time during the interview, since the lengthier question stem would be eliminated and only read to those who do not work most of their hours during the day.

Recommended wording:

On your [main/current] job, do you USUALLY work a daytime schedule or some other schedule? *Read if necessary: By daytime, I mean a schedule in which most work is done between the hours of 6 a.m. and 6 p.m.

1. Daytime– most work is done between 6 a.m. and 6 p.m.

2. Some other schedule [ask follow-up question about type of shift]

JF_7

(Read list and choose one)

What is the main reason why you work this type of shift?

*Read if necessary: Do you work this shift because of family or childcare arrangements, better pay, to allow time for school, because you could not get another shift, it’s the nature of the job, because it’s your personal preference, or some other reason?

1. Better arrangements for family or childcare

2. Better pay

3. Allows time for school

4. Could not get any other shift

5. Nature of the job

6. Personal preference

7. Some other reason – specify [Go to JF_7_SP]

<1-6, Don’t Know, Refusal>

Cognitive Interview Findings:

In the first round of cognitive testing, only one participant was administered this question, and she responded “Other,” specifying commuting time was the main reason she works double shifts at her main job. She understood the question but felt the response options did not fit her reason.

In the second round of cognitive testing, participants understood this question but often wanted to select multiple reasons. When probed, they were able to arrive at a single answer. At least one participant answered about his second job, which was in the evening. Like in round 1, those who had multiple reasons often ended up selecting personal preference, as it encompassed all of the other reasons.

Online Study Findings:

In the online testing, participants eligible for this question selected the following reasons shown in the table below.

What is the main reason why you work this type of shift? (n=106)
Better arrangements for family or childcare	16.0%
Better pay	10.4%
Allows time for school	4.7%
Could not get any other shift	5.7%
Nature of the job	28.3%
Personal preference	31.1%
Other (please specify)	3.8%

Recommendations: No changes to this question were recommended.

JF_8/B7ALT2 In the last week, for your [current/main job], how many hours did you perform additional paid or unpaid work, such as responding to texts or emails, or other tasks outside of your regular working hours?

0. NONE

1. An hour or less

2. 2-90. ACTUAL HOUR AMOUNT

3. DON’T KNOW/REFUSED

This question was asked in the first round of cognitive testing to get at spillover of work into personal life.

Findings:

Response frequencies in Round 1 (n=10):

None	An hour or less	An actual hour amount
4 (40%)	3 (30%)	3 (30%)2

Participants generally understood this question to be getting at how much work they do outside of their regular working hours. However, problems with comprehension and response formation occurred. Several participants asked for the question to be repeated, or needed reference period repeated, indicating issues with comprehension, and adding length to the survey.

Comprehension problems:

Participants who were salaried versus paid hourly or did shift work had different interpretations of this question. Most salaried workers tended to view their job as “working until the job is done,” regardless of whether that exceeded 40 hours per week; this was just part of their job (not additional work), and they’d consider it paid work. A minority of salaried workers considered anything over their 40 hours as “donating time” and that this would be unpaid work. This led some inconsistencies in comprehension of the question and what counts as “additional work.” For example:

• “This is what I would interpret as unpaid work like either I choose to open my email on my phone or my work laptop prior to 9 or after 5PM. Or keep working at other times either because someone has contacted me and I have chosen to respond or I didn't have time to finish something. I know it's minimal because I really try not to do it. So for me, it was only unpaid work for me to consider...because I'm paid the same either way.”

• “I spent about an hour. But again I'm salary so I see it as outside my work hours.”

• “I did not meet that deadline so I had to work extra so I did work outside my regular hours but it was not additional pay and it was not unpaid. It was just part of my hours. And that's because I had not worked those hours during the day. So they were part of my hours essentially.”

Other comprehension issues were related to the examples embedded in the question. Some participants had a narrow interpretation of the types of work tasks they should report on, due to the examples of responding to texts or emails. These participants thought the question was only interested in collecting when they responded to communications, and not other general work tasks.

Participants who were hourly or shift workers tended to respond “none” or “5 minutes” to this question. Even some of those workers did some related tasks from home, such as checking their work schedule on the company app or work calendar, or staying in the office a few extra minutes to finish a phone call. However, the question asks for the number of hours (rather than minutes) that they worked, and some participants felt they should not report smaller tasks.

Recommendations: We recommended revising this question in the second round of cognitive testing to improve comprehension. We recommended removing the phrase “paid or unpaid” from the question, as salaried workers often got stuck on wondering if they were paid for the additional work or not, and this distracted from the intention of the question. We also recommended removing the examples of responding to texts or emails so that participants will consider all types of work tasks instead of only responding to communications; we recommend replacing this text with “any type of work tasks” to broaden the type of work tasks respondents would consider in their answers. In addition, we recommended adding an optional interviewer probe to collect smaller tasks if respondents mention doing very short, work-related tasks, and specifying that the question applies to respondents’ main job. The revised version of the question (below) was tested in the second round of cognitive testing and the online study.

JF_8/ B7ALT2

In the last week, how much time did you spend performing any type of work tasks outside of your regular working hours for your main job?

*Interviewer probe if needed: Include any additional paid or unpaid work you might have done in the last week, even as little as a few minutes.

*Interviewer probe if needed: Work tasks include finishing or catching up on work, responding to calls, texts or emails, and so forth

0. None/no time

1. 1-59 actual minute amount: ____

2. 1-90 actual hour amount:_______

3. DK/RF

Cognitive Interview Findings:

Response frequencies in Round 2 (n=10):

None/No time	1 – 59 Actual Minute Amount	1 – 90 Actual Hour Amount
1 (12.5%)	1 (12.5)3	7 (87.5%)4

In the second round of cognitive testing, almost every participant asked for this question to be repeated, indicating an issue with question comprehension. Some reported expecting to hear response categories or a request for a unit of time (hours, minutes). Upon repeating, most participants could arrive at an answer. Some wanted to provide a range (e.g., 5-8 hours) and needed prompting to select a single answer. One participant reported being on vacation during the last week and was unable to provide an answer.

Most participants seemed to understand the question as asking about work they performed outside of their scheduled working hours or their shifts, such as overtime or catching up on work from home. At least two participants seemed to have different interpretations of the question. One thought the question was asking about how strenuous or labor intensive one’s job was and how one spends their time outside of work in general – after probing for more information, it was unclear where this interpretation came from:

• “I think it was asking if I've done anything extra during that last week, and if it was strenuous. Like physical labor.”

A couple participants seemed to think the question was interested in collecting whether someone has a part-time job on the side to supplement their income:

• “Something outside of work I'm doing. Anything else on the side.”

• “If you had a part-time job, to supplement your income.”

Most participants thought about the typical tasks they do at work, and considered whether any of them were completed outside of regular working hours. At least one participant had a much broader interpretation of this question, and considered things they do in their free time (non-work-related tasks) as part of their response:

• “Asking me what I am doing outside of work. Normally I’m just hanging out.”

Most participants understood their “regular working hours” to mean the days and hours they are required to work. Most participants also reported that they did the extra work by choice to catch up on work or cover for another worker.

Online Study Findings:

In the online testing, participants reported performing an average of 5.5 hours of additional work. However, responses ranged from 0 hours to 55 hours. A total of 31 participants reported working upwards of 20 additional hours, suggesting some participants may have misinterpreted the question. Probes about the type of tasks performed revealed participants did think about in-scope work tasks, including paperwork, lesson plans for students, responding to emails, phone calls, catching up on work from home, or completing training. However, several out-of-scope responses were also provided, including:

• “Side gigs.”

• “Working on a second job.”

• “Completing online surveys.”

• “I did the dishes or assisted with other chores.”

• “Working out or meal prepping.”

• “Cooking lunch.”

Recommendations: Participants did not consistently understand this question, or what was meant by work tasks performed outside of their regular working hours. Some considered any tasks they do outside of work in general, while others seemed to think the question was asking about additional side jobs or gigs. This was observed in both the cognitive interviews and online testing. Because of issues with comprehension and measurement error, we did not recommend adding this question and it was dropped from the leave module.

2a. In the last month, what is the greatest number of hours you’ve worked in a week at your [current/main] job? Include any extra hours, overtime, work you did at home, and so forth. [ACTUAL HOUR AMOUNT]

2b. In the last month, what is the fewest number of hours you’ve worked in a week at your [current/main] job? Do not include weeks in which you missed work because of illness or vacation. [ACTUAL HOUR AMOUNT]

Findings: In both rounds of cognitive testing ( _{2aRound 1} = 36.33, _2bRound1 = 32.5, _2aRound2 = 41.1, _2bRound2 = 29.2), participants understood this question well. However, because of the open-ended nature of the question, it seemed to create more respondent burden and pauses to think. Some participants interrupted the interviewer during question administration, likely due to the long question stem and include/exclude instructions.

In the online testing, a total of 382 participants answered this question, and 23 skipped it. The average number of hours reported for 2a was 41 hours, and the average number of hours reported for 2b was 31 hours. This suggests that participants understood the question, as ‘most hours worked’ exceeded the ‘fewest number of hours worked’ on average.

Recommendations: No wording changes were recommended.

NEW ITEM

How difficult would it be for you to take an hour or two off during work hours, to take care of personal or family matters?

<READ OPTIONS>

1. Not difficult

2. Somewhat difficult

3. Very difficult

This question was added in the second round of cognitive testing and the online study to assess work schedule flexibility.

Cognitive Interview Findings:

Response frequencies in Round 2 (n=10):

Not difficult	Somewhat difficult	Very difficult
5 (50%)	5 (50%)	0 (0%)

Most participants seemed to understand this question and did not have difficulty arriving at an answer. Some participants who indicated that this would be somewhat difficult tended to think about built-in breaks, like their lunch hour, and trying to take time off during those set breaks. Those answering ‘somewhat difficult’ also tended to say that they would need supervisor permission. One participant thought the question was asking about whether or not they could take leave in general.

Another participant initially responded “somewhat difficult” but during debriefing changed their response to “not difficult”:

• “My supervisor is very understanding, if it's family matters, knowing that family is so important, a reason like that would be allowed. I've not taken more than 2 hours in any leave. When I said somewhat difficult earlier, I started thinking about occasions when my family might need him, it then makes it complex on when to decide to go to work or go to my family, but my manager always allows me to go with my family.”

For those responding that this would be difficult, it was usually because they’d need to find someone to cover a shift:

• “If I don't do that, I have to look for someone to fill in for me. And it's not really easy to get a coworker or a colleague to want to fill in for me, except for the one who's really close to me. So it depends on if she's around.”

• “Once you are clocked in, it's not that easy to clock out. It’s a strict schedule, it's up to you to talk to a colleague to get them to cover a shift. It’s up to you to sort it out, make an arrangement with a colleague to assist you.”

• “You only get 15 minute break and 30 minute lunch break. The only way I could see doing that is if I got all work done and had an extra hour of down time. That's if able to get work done early and nothing else required. Other than that, 30 minute lunch break and 15 minute break. Would say somewhat because able to get work done fast.”

• “It's somewhat difficult. Being a social worker, works with children. Leaving them in the middle and tending to personal schedule is difficult. Personal things can be taken care of later. Care is for the children so have to be careful when dealing with children.”

Those with a flexible schedule or who were salaried employees tended to respond that this would not be difficult for them, and they give more of a “heads up” than asking for explicit permission to use leave:

• “Normally, I get an hour break. For example, if I have to go to a doctor's appointment, it's no problem. I can be away for at least 2-3 hours without any issues. Of course I want everyone to know exactly where I am. I don't want to leave them in the dark. It's pretty much a heads up. I don't need approval.”

• “If anything changes, I can change the hours I work.”

Participants tended to think of medical appointments, child care, running errands, providing transportation for older relatives, or emergency situations. Some participants thought of other reasons, including:

• “Needing to go see my daughter play piano at school. The super wants to clean the filter of the heater. I've even done it to take a yoga class during the day. As long as I get in my full 8 hours of work, I'm fine.”

• “Laundry, traveling, visiting someone.”

• “Trying to take care of a bill or plan a vacation.”

Most participants reported imagining a typical or average day at their job when answering this question, and not busy seasons or other atypical time periods.

Online Study Findings:

In the online testing, participants were asked a similar series of questions on this topic. The table below shows the response distributions in the online study.

How difficult would it be for you to take an hour or two off during work hours, to take care of personal or family matters? (n=379)
Not difficult	52.2%
Somewhat difficult	39.0%
Very difficult	8.7%

Participants were asked an open-ended probe about how they came up with their answer. These tended to fall into three categories: it depends, it would be easy, or it would be difficult.

Examples of “it depends” situations included:

• “With advance notice it would be fine. But on short notice, If the main part of my work finishes early I could take off a little early but if it runs later I would not be able to.”

• “it would depend on the time of day and our coverage.”

• “This would be difficult on the days when I am physically at work. I commute with another person to a city that is 1.5 hours away. When I am working from home, this would be possible.”

• “It really does vary. Some days it would be nearly impossible because there simply isn't anyone to cover my position that day then other days they could get by without me there.”

Examples of situations where it would be easy included:

• “I've worked for the company for over three years and have had some personal matters to take care of. In those instances, I was allowed to make up the time that I was away from work at a later time.”

• “I would be easily able to leave work for an hour or two because I can catch up when I return.”

• “I know what my workload is and how long I need to get it done. As long as I get it done by the deadline, my managers don't care what hour of the day I do it.”

Example situations where it would be difficult included:

• “I have a limit of hours I can be absent from work before being held accountable or fired.”

• “Based on my company's policies; I wouldn't be permitted to leave during working hours without going through the formal process & getting permission, and it might require documentation related to why I needed time off.”

• “It is heavily frowned upon to have an issue that requires you to leave work. It is seen as not being responsible enough to take care of things outside of working hours, or prioritizing your personal life. I have seen people get scheduled for less hours as a punishment for needing to leave to pick up their kid from school and things like that.”

Participants in the online study also thought mostly about family emergencies and doctors’ appointments as “personal or family matters.” Other responses included taking care of and walking dogs, automobile issues, childcare, or providing transportation for a family member. Of the 378 participants who answered this question, a total of 70% indicated their response would be consistent week-to-week, and 30% said it would vary week-to-week.

Recommendations:

This question worked well. Participants reported a wide variety of activities that would count as “personal or family matters,” ranging from emergencies to leisure activities. All participants seemed to understand the intent of the question and could arrive at an answer. Responses from the online testing seemed aligned with the findings from the cognitive interviews. We did not recommend any wording changes.

JF_10

As part of your [current/main] job, can you work at home?

1. Yes [Go to JF_11]

2. No [Go to NOLV_Intro]

Don’t know, Refusal [Go to NOLV_Intro]

Cognitive Interview Findings:

Response frequencies by round:

	Yes	No
Round 1 (n = 10)	6 (60%)	4 (40%)
Round 2 (n = 10)	4 (40%)	6 (60%)
Total	10 (50%)	10 (50%)

In both rounds of cognitive testing, participants seemed to understand this question well and did not display any major problems with comprehension or arriving at an answer. Participants understood work at home as any work they can do offsite. Most mentioned that would include work done at home, and some mentioned it could include the home or another remote location. For participants whose work can only be performed on-site, the question seems a little out of place. For example, one participant who worked in a factory remarked:

• “I guess depending on your job, some people can work on their computer and do their work-related jobs. What I do, I work in a factory, it's not even possible. I load trucks for my job. What am I going to do, load trucks from my house?”

We were interested in how this may have changed for participants after the COVID-19 pandemic. Many participants noted expanded telework policies following the pandemic, with several participants explaining that they have “hybrid” policies in which they report to an office some days and work from home other days.

In round two of testing, we also asked about the meaning of the term “telework.” Some participants were unfamiliar with the term. Others thought it was using a system like a laptop or phone connected to the Internet. Two participants thought it was related to making calls at home (perhaps telemarketing). Participants reported using terms like work at home, hybrid schedule, and remote work.

Online Study Findings:

In the online study, participants (n=328) seemed to understand the concept of “work at home” as intended. They described work at home as doing their normal duties from home instead of on-site or in an office. Like the cognitive interview participants, they also used terms like remote work, hybrid schedule, or telecommuting. Most only mentioned work done in the home, and not other remote locations.

In the online study, of the participants answering this question (n=330⁵), a total of 53% of reported they can work at home. Of the 301 participants who received a follow-up question about whether this changed since the COVID-19, 30% answered yes, it had changed since the pandemic, and 70% indicated no, it had not.

Of those answering yes, participants were asked for all of the reasons this had changed since the pandemic; the table below shows the distribution of responses.

For what reasons has this changed since the COVID-19 pandemic began? Select all that apply. (n=91)
I changed jobs	19.8%
Employer policy changed	62.6%
Job changed to hybrid schedule	38.5%
Job became fully remote	18.7%
On-site work limited or no longer possible (for example, office building closure)	8.8%
Other (please specify)	2.2%

We also asked participants if “work at home” and “telework” mean the same thing, or if they are different. Of the 329 participants who received the question, a total of 70% answered they are the same and 30% indicated they are different. For those saying they were different, reasons tended to be that telework implied digital, telephone, or electronic communications while work at home didn’t:

• “Telework implies the use of telecommunications to perform the task, while work at home is ambiguous.”

• “For me telework refers to work that can be done digitally, work from home is all inclusive, any task.”

• “Work at home is any task you can do from your house and get paid for telework is any work you can do on a computer or phone and get paid for.”

Some thought that telework could include other locations than one’s home:

• “Telework can be anywhere whereas work at home is at one’s specific home.”

• “Telework is being able to complete your work tasks anywhere outside of your work location and working at home means to work only at home.”

• “The difference between work at home and telework to me is work at home means working from home and telework means working at a location other than the main office location.”

• “Work at home generally refers to working from a personal residence while telework involves working from any remote location.”

• “I think telework means that you can bring your work to other places to accomplish them such as a public cafe with wifi, for instance, while work at home means to literally be at your home and work to do your job.”

Other participants thought that telework implied you were physically located near an office and work at home included remote work:

• “Work at home could be from anywhere in the world, whereas telework would be geographically closer to the main office.”

• “Working at home means working 100 percent at home. Telework means that it is a hybrid thing work home and on site.”

• “Work at home is working at home the entire day and telework is working both at home mostly and occasionally going into work.”

Some participants also seemed to think telework implied telemarketing:

• “Work at home is completing takes on your desktop at home while telework is trying to persuade people to buy things or take surveys. This is mostly done at the business location due to easy monitoring of the employees by the employer.”

• “Work at home could mean office type stuff. Telemarketer means bothering people trying to sell them crap they don’t want.”

Recommendations: Since participants seemed to understand this question well, we do not recommend any changes. Participants in both the cognitive interviews and online testing had different conceptions of telework – so we do not recommend introducing the phrase “telework” at this time. Although “work at home” implied working in a home location for some participants, some also thought it implied other remote locations as well. An optional probe or interviewer instructions to include other locations besides one’s home could be considered (e.g., “Include any other remote or off-site location where you CHOOSE to work.”).

JF_11

Universe: JF_10 = 1

Do you ever work at home?

1. Yes [Go to JF_12]

2. No [Go to JF11.5 ]

Don’t know, Refusal [Go to NOLV_Intro]

Findings: No problems were observed with this question in either round of cognitive testing. In the online study, of the 297 participants who received the question, 55% answered yes. No changes to question wording were recommended.

JF_12

Are you paid for the hours that you work at home, or do you just take work home

from the job?

1. Paid

2. Take work home

3. Both

Don’t know, Refusal

[Go to JF_13]

Cognitive Interview Findings:

In both rounds of cognitive testing, only minor problems were observed for this question. A couple participants asked for the question to be repeated, and changed their response when the question was repeated a second time. At least one participant thought that the question was asking for a yes or no answer.

Online Study Findings:

In the online study, participants eligible for this question mostly indicated their work at home was paid:

Are you paid for the hours that you work at home, or do you just take work home from the job? (n=161)
Paid	74.5%
Take work home	8.7%
Both	16.8%

Recommendations: We did not recommend any changes to this question. Only minor issues were observed.

JF_13

What is the main reason why you work at home?

*Read if necessary: Do you work at home to finish or catch up on work, because your job requires working at home, to coordinate your work schedule with personal or family needs, to reduce your commuting time or expense, because it’s your personal preference, or for some other reason?

<DO NOT READ RESPONSE OPTIONS>

1. Finish or catch up on work

2. Job requires working from home

3. Coordinate work schedule with personal or family needs

4. Reduce commuting time or expense

5. Personal preference

6. Other-specify <GO TO JF_13_SP>

Don’t know/refused

Cognitive Interview Findings:

Response frequencies by round:

	Job requires working from home	Coordinate work schedule with personal or family needs	Reduce commuting time or expense	Personal preference	Other
Round 1 (n = 6)	1 (1.7%)	1 (1.7%)	0 (0%)	4 (66.7%)	0 (0%)
Round 2 (n = 4)	1 (25%)	0 (0%)	0 (0%)	1 (25%)	0 (0%)

In both rounds of cognitive testing, participants generally understood this question but had problems mapping their answers onto the available response options. Some of these problems stemmed from the changing landscape of work from home following the COVID-19 pandemic. For example, several participants noted that they either work remotely and there is no physical office for them to report to, or they have a hybrid schedule where they telework and report to an office. These participants may be required to work remotely because there is no physical location to report to, or their employer may require them to come in certain days and not others. These participants felt the reasons for teleworking did not completely reflect their situation, for example:

• “It's a combo of preference and an additional option should be 'no local office' or 'it's a remote position, maybe'. After reading response options, I would choose “personal preference.”

• “This is one where I'm trying to decide what fits for those choices, when we came back from fully remote from COVID, the employer decided everyone would be in the office 2 days/week, and work at home 3 days per week. If I had the option, I guess then I would answer Personal Preference.”

• “Our organization now has hoteling arrangements for use of the office, it's expected that some people will telework, and some are in the office.”

• “I think I work from home because I'm allowed to work from home. But I do prefer it. If we were allowed to work from home 5 days a week, I would do it. It is a personal preference. I find it quieter. I find it easier to concentrate. It is easier to coordinate with my family and getting things done around the house during breaks here and there, like throwing the laundry in or putting dinner in the oven. So it makes things easier. I would answer that I prefer to do it over all the others [response options].”

A few participants mentioned it was difficult to select just one reason because their employer requires some amount of telework, for example:

• “This question threw me off because I could have chosen more than one. The main reason I work from home is because that is what the company policy is, a hybrid work schedule.”

• “It’s a remote position, but I could have chosen other options. Flexibility should be another option; I have friends who work from home because they do it for flexibility.”

Participants’ responses did not always map onto the response categories. Three participants provided the response “convenience.”

• “Why not? I don't have to leave my couch. Convenience. Ease of childcare. Ease of errands. Things that can't be done during evenings or weekends. And again, just one choice? Personal preference, and it encompasses everything you mentioned.”

• “It makes it very convenient.”

After probing, participants could arrive at a response, but this often involved using the “read if necessary” instructions so that participants could hear the options, adding time to the interview. Some participants also wanted to change their response during the debriefing, when they could see the answer choices listed. Others indicated that multiple response options apply to them, but they selected personal preference, since it seemed to encompass all of the other reasons. Another participant said she could select multiple reasons, but catching up on work was probably the mean reason if forced to pick one.

Response option 5, personal preference, seemed to be a default answer for participants who were having problems arriving at an answer. Sometimes this response category took precedence over when the job requires some amount of telework (e.g., those with hybrid schedules).

Online Study Findings:

In the online study, participants who reported working at home were asked an open-ended question about the reasons they work at home, followed by the closed-ended question tested in the leave module. Themes from the open-ended responses included work at home being required of the job, policy changes after the COVID-19 pandemic (e.g., being fully remote or hybrid), catching up on work, cutting time and expense on company office space or lack of office space after the pandemic, avoiding a commute, convenience and efficiency, work/life balance, avoiding office distractions, saving on gas money, childcare, increased flexibility, and sometimes merely because participants were “allowed to do it.”

Response distributions to the closed-ended question were as follows.

From the options below, what is the main reason why you work at home? (n=159):
Finish or catch up on work	18.2%
Job requires working from home	22.6%
Coordinate work schedule with personal or family needs	10.7%
Reduce commuting time or expense	17.6%
Personal preference	25.2%
Other (please specify)	5.7%

Like in the cognitive testing, personal preference seemed to be a common response. A similar number of participants also indicated that their job requires them to work at home.

Recommendations: We did not recommend any wording changes to this question. With changes in work at home since the pandemic, interviewers may have to probe more to help respondents arrive at a codable answer, or read the response categories, which may add time to the interview.

JF_13.5

Universe [JF_11 = 1, AND JF_12 = 1 or 3 or DK/RF, AND JF_13 does not equal 2]

At your main job, are you required to work at home on a regular basis?

1. Yes

2. No

3. DK/RF

Because it was observed that some participants indicated that their job required some amount of work at home, an additional question to collect whether work at home is required was tested in the second round of cognitive testing.

Cognitive Interview Findings:

Response frequencies in Round 2 (n=4):

Yes	No
1 (25%)	3 (75%)

Participants seemed to understand this question but some minor issues came up. One participant answered “no” because the requirement was not on a regular basis. For the participants who regularly work at home, they tended to answer “no” because work at home was not technically a requirement:

• “Again, it's my choice. If I want to go work in the office, I can. I have a desk there. I have an office there. Sometimes I have stuff to do here. I love the ease of just getting presentable and getting on the phone.”

• “It's not a requirement to work at home or in the office. No policy exists. How being in the office/working at home differs depending on the job: Obviously the maintenance crew has to be there to clean the building. The administrative assistant has to be there for the director because he goes in every day and they have to be there for him.”

Online Study Findings:

In the online study, of the 159 participants answering this question, 53% indicated that they are required to work at home on a regular basis. When asked for the reasons they’re required to work at home, this tended to catch the remote workers:

• “Virtual team, no value being in the office.”

• “That is my primary location assigned.”

• “The office is located out of state.”

• “We have gone fully remote and do not have access to any co-working spaces to work from.”

• “There is no in-person option.”

Others reported there was no on-site location:

• “Company has no official office.”

• “Our office changes location constantly. It’s often easier to work from home.”

Others mentioned office space was reduced following the pandemic:

• “We have gone fully remote and do not have access to any co-working spaces to work from.”

• “That was part of the job description. They also really don't have much space for me at the office.”

A few participants mentioned that this was their personal preference, and it was unclear if the job truly required work at home:

• “I almost always work from home unless they need me to come to a specific study in person.”

• “I am a hybrid employee. Part of my job is required to be in person, but there are tasks that I am allowed to work at home for pay.”

• “I can be more productive and my work does not require face-to-face interaction with the public.”

• “I could go into the office but I'd have to reserve a desk, it would take time and I already said I'd work from home. I could probably do it with effort, but for practical purposes I have to work at home now.”

• “It is easier right now because the office is moving to another location.”

• “It's easier than at any other location.”

• “It saves time.”

We also asked participants in the online study how their employer refers to this arrangement, response distributions are included in the table below. Work at home was most common, followed by remote work:

How does your employer refer to this arrangement? (n=83)
Work at home	50.6%
Telework	3.6%
Telecommuting	4.8%
Hybrid schedule	12.0%
Remote work	26.5%
Other (please specify)	2.4%

Recommendations: No wording changes were recommended for this question. Although it seemed like participants generally understood this question, the term “required” seemed to be ambiguous to some participants in the online study. When asked about reasons they are required to work at home, responses included reasons that seemed related to personal preference, when they may have had the option to report to an office. Thus, the question may have some measurement error in these types of situations.

JF_14

Universe: JF_11 = 1

Are there days when you work only at home?

1. Yes [Go to JF_15]

2. No [Go to NOLV_Intro]

Don’t know, Refusal [Go to NOLV_Intro]

Findings:

Response frequencies by round:

	Yes		No
Round 1 (n = 6)	6 (100%)		0 (0%)
Round 2 (n = 4)	3 (75%)		1 (25%)

In both rounds of cognitive testing, most participants seemed to understand this question and had no problems arriving at an answer. However, one participant thought this question referred to days of the week she teleworked consistently (i.e., having set days of the week in which she works from home), not whether there are any entire days in which she worked at home. Of the 158 participants who received this question in the online study, 82% answered yes.

Recommendations: No wording changes were recommended for this question.

JF_15

Universe: JF_14 = 1

How often do you work only at home?

<DO NOT READ RESPONSE OPTIONS>

1. 5 or more days a week

2. 3 to 4 days a week

3. 1 to 2 days a week

4. At least once a week

5. Once every 2 weeks

6. Once a month

7. Less than once a month

Don’t know, Refusal

Findings:

Response frequencies by round:

	5 or more days a week	3-4 days a week	1-2 days a week	At least once a week	Once every 2 weeks	Once a month	Less than once a month
Round 1 (n = 6)	2 (33.3%)	4 (66.7%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)
Round 2 (n = 3)	1 (33%)	1 (33%)	1 (33%)	0 (0%)	0 (0%)	0 (0%)	0 (0%)

In both rounds of cognitive testing, only minor issues occurred for this question in terms of mapping participants’ responses onto the available response categories. For example, participants provided answers like “2 or 3 days” which required probing to select between categories 2 and 3. At least one participant indicated confusion on the unit of time the question was asking for and inquired if wanted the answer reported, “per week?”

In the online study, for participants who received this question, over one-third reported working only at home 5 or more days per week:

How often do you work only at home? (n=129)
5 or more days a week	37.9%
3 to 4 days a week	24.0%
1 to 2 days a week	26.4%
At least once a week	3.9%
Once every 2 weeks	3.9%
Once a month	1.6%
Less than once a month	2.3%

Recommendations: No changes were recommended for this question. We note that interviewers may need to do additional probing to help respondents map their answers onto the available categories, which may add length to the interview.

Appendix A: Round 1 questionnaire and debriefing protocol

Appendix B: Round 2 questionnaire and debriefing protocol

Appendix C: Online survey questionnaire

1 The response option “I am entirely free to decide when I start and end work” was dropped in Round 2.

2 Three participants reported an actual hour amount. Two reported 2 hours and one reported 6 hours.

3 One participant reported an actual minute amount and an actual hour amount.

4 = 4.45 hours (Note: includes total time reported i.e., actual minute amount and actual hour amount).

5 Due to a server glitch, some participants were unable to access this set of questions, leading to inconsistent number of participants responding.

42