Attachment J - Cognitive Testing Results for the 2017 Leave Module

Summary of Cognitive Interviewing Testing for the 2017 ATUS Annual Leave Module

Prepared by

Bill Mockovak

Robin Kaplan

Office of Survey Methods Research

1. Overview

The American Time Use Survey (ATUS) is an ongoing, nationally-representative telephone survey of the civilian, non-institutional population age 15 and over. Respondents are asked to sequentially describe what they did “yesterday,” from 4 a.m. of the previous day to 4 a.m. of the interview day. The sample is chosen from households that completed the eighth (final) CPS interview, and in each of these households, one person is randomly selected to be interviewed for the ATUS. Each person is also assigned to be interviewed on a particular day (for example, Tuesday), with interviewers attempting to reach the designated person for up to eight weeks. About half of the respondents are interviewed on a weekday and about half on a weekend day.

In 2011, BLS conducted an ATUS Leave module that captured information about Americans’ access to and use of paid and unpaid work leave. The 2017 ATUS Leave module will accomplish similar objectives as the 2011 module. Although many questions will remain the same, some have been dropped, and some have been added to obtain better information about the availability and use of flexible and alternative work schedules, including how flexible, alternative work schedules affect time spent in nonmarket activities such as housework, childcare, and volunteer activities. As in 2011, data will be collected on employees’ access to paid and unpaid leave (for example, whether respondents have paid or unpaid leave and, if so, for what purposes workers can take leave from their jobs) and their leave activities (for example, instances of leave taking, leave denials, and non-use of leave).

This report describes results from cognitive tests of the proposed leave items in order to gain insight about respondents’ understanding of the terms, definitions, and concepts associated with leave. The tests were conducted by BLS staff in the Office of Survey Methods Research (OSMR) and observed by staff from the Office of Employment and Unemployment Statistics (OEUS).

This report focuses on results from in-house testing. Although some online testing was done as well, the majority was conducted after the final ATUS Leave Module was completed and submitted. Since those results were largely not used to inform the final questions included in the final questionnaire, they will be supplied in a follow-up report at a later time. However, small-scale, online, supplemental testing involving five participants was conducted after the in-house testing had been completed to test some last-minute wording changes, and those results will be presented in this report.

2. Methodology

There are five sections in the 2017 ATUS leave module questionnaire (see Appendix A for the detailed question specifications). A general outline of the module follows:

1. Section 1: Access to Paid Leave. Is paid leave available, for which reasons could a participant take paid leave, what types of leave plans are offered by the employer, and did workers have a PTO, or “paid time off” plan? A PTO plan generally provides a fixed number of hours which workers can use for different reasons, such as vacation or sick leave.

2. Section 2: Access to Unpaid Leave. Is unpaid leave available, for which reasons could a participant take unpaid leave, and has unpaid leave ever been used?

3. Section 3: Leave Taken in the Last Week. If paid or unpaid leave was taken in the prior week, how much was taken, what was the reason, and was it paid or unpaid?

4. Section 4: Job Flexibility and Work Schedules. These questions ask about worker flexibility in arranging work schedules, the flexibility of working at home, and reasons for working at home.

5. Section 5: Nonuse of Leave. This set of questions asks about times in the past month when workers needed to take time off from work, but could not, the reasons why they needed to take leave but did not, and the reasons why they did not take leave.

Many of the 48 questions in Sections 1, 2, 3, and 5 were tested when the module was last conducted in 2011. Section 4, which deals with a worker’s flexibility arranging start and stop times and work schedules, consists of many new questions. In-house participants answered all the leave-module questions.

Research psychologists from OSMR interviewed in-house participants individually in the OSMR behavioral science research laboratory. In addition, to expand the sample and include participants with desired characteristics (for example, fire fighters with unusual work schedules), several online and phone interviews were conducted.

Each in-house session generally included a monitor from the ATUS staff, who took notes and timed the interview, coded question responses and possible problems (see Appendix B for the list of response codes), and entered the results into a database after reaching consensus with the OSMR interviewer about the assignment of codes. Depending on the availability of ATUS and OSMR staff, almost all sessions had two observers (usually one from the ATUS staff and one from OSMR). In these sessions, the two observers reached a consensus on the assignment of codes, reviewed that consensus with the OSMR interviewer, and then entered the codes into the database. In only two interviews (two conducted by phone), when monitors were not available, did the OSMR interviewer conduct the interview and assign the codes individually. Each in-house interview was conducted as follows:

1. The participant was introduced to the objectives of the session, and asked to sign a consent form (see Appendix C for the detailed in-house protocol).

2. The interviewer and observer(s) then left the room, closed the door, and the interviewer called the participant on the phone. The interview was conducted entirely over the phone and was scripted using an online survey tool, SurveyMonkey, which allowed automatic control of skip patterns.

3. Once the interview was complete, the interviewer and observer(s) returned to the room and proceeded through the debriefing interview, which essentially asked if there were any questions that were difficult to answer or understand and asked specific probes about the questions (see Appendix C).

4. After the debriefing, participants were thanked for their help and given a $40 check.

5. The OSMR interviewer and observer(s) then discussed their observations of the interview and how the questions had worked.

3. Participants

Thirty-one in-house participants participated in this study. The in-house participants were recruited from OSMR’s database of general population study participants and from advertisements on Craigslist and a local listserv. During recruitment and screening, we collected the following information to ensure that we were interviewing a cross-section of the target population: employment status (full- vs. part-time), class of worker (government, private, not-for-profit), age, and education. A special effort was made to interview eight shift workers (for example, nurses, fire fighters, factory workers, fast-food workers, security guards, etc.). Online testing used participants who were recruited by TryMyUI.¹

4. Results

1. Timing

One purpose of the testing was to determine the approximate time the module would take. An observer (or in a couple of situations, the interviewer) timed each interview. The results are shown in Table 1 below.

Protocol 1 reflects the question wording and order as they appeared in the original questionnaire specifications, dated March 8, 2016. Midway through the testing, the decision was made to modify and move a question about the availability of types of leave plans to the beginning of Section 1 to see if this would shorten the interview, improve comprehension, and alleviate perceived redundancy between questions that asked about reasons for taking paid leave and types of paid leave plans. This modification is referred to as Protocol 2. Appendix D contains both Protocols 1 and 2. These protocols show how the questions were administered using the Survey Monkey software and thus they have a slightly different appearance than the specified questionnaires appearing in Appendix A.

As shown in Table 1, this strategy did not have the desired effect of shortening the length of the module. The average length of each interview took about the same amount of time as measured by the mean and median values. Note however that in the context of cognitive interviews, participants are instructed to think out loud as they answer survey questions and provide feedback about the questions in real time. They also receive compensation for participating. Because of these factors, many participants elaborated on their responses more than we would expect real respondents to do during a production survey. This likely inflated the total amount of time it took to complete the survey.

Table 1. Timing of the Module Using Two Different Protocols

	Mean	Median	N
Protocol 1	6:05 min	6:00 min	15
Protocol 2	6:11 min	5:57 min	16

The following sections summarize the results of Protocols 1 and 2, and we provide suggestions for the final questionnaire. If a particular survey question is not mentioned, that indicates it performed well.

Results

Unless indicated otherwise, the question names used in this section come from the March 8^th version of the questionnaire, which was used to program Protocol 1 (see Appendix A for the full version of the March 8^th questionnaire).

Section 1: Access to Paid Leave

As mentioned, Section 1 focused on questions about whether respondents have access to paid leave from their main job.

PAID:

“Do you receive paid leave on your [current/main] job? [Fill: By main job, we mean the one at which you usually work the most hours.]”

This question worked well and functioned as intended. Most participants (14/15) seemed to understand the intent of this question. Only one participant expressed minor confusion, at first saying he has no paid leave, then stating he could use paid leave for emergency situations only, and finally saying that he does have a paid leave plan. Most likely, his paid leave plan is situational and contingent on the level of need to take leave.

The next question focused on reasons why people might have to miss work and whether respondents are allowed to take paid leave for these reasons. Only participants (8/15) who earlier indicated they receive paid leave answered these questions.

RSNPD_A-G:

“I’m going to read you a list of reasons why you might have to miss work. For each reason, please tell me if you are able to take paid leave in your [fill: current/main job.]”

A: Own illness or medical care

B: Illness or medical care of another family member

C: Childcare, other than for illness

D: Eldercare

E: Vacation

F: Errands or personal reasons

G: Birth or adoption of a child

In general, these questions performed well. Most respondents seemed to understand the intent of the questions and were able to provide a yes or no response. Participants seemed to immediately know that they could use paid leave for their own illness or medical care, but sometimes struggled with providing responses for the other reasons. This occurred primarily when a reason did not personally apply to them (e.g., they do not have children or do not have eldercare obligations). Some participants weren’t familiar with their employer’s policy simply because the reason did not apply to them, so they answered “Don’t Know.” Other times, they were aware that they could likely take paid leave for that reason, but noted they never personally had to do so. They based their guesses on knowledge about their co-workers using paid leave for that reason.

Occasional problems that emerged for each item are summarized below.

B: Illness or medical care of another family member: Generally, few problems were observed for this question. One participant answered “yes” at first, then later requested to go back and change her answer to no. This implied she struggled with selection of the best response option.

C: Childcare, other than for illness: Three out eight participants who answered this question struggled with providing a response, usually because they did not have children and this situation did not apply to them. One participant asked for clarification on what we meant by childcare – another participant noted that, “it depends on the situation.”

D: Eldercare: Participants understood the meaning of eldercare, but like the question about childcare, it did not apply to some participants. Two out of eight participants noted they didn’t know their employer’s policy about taking paid leave for eldercare.

G: Birth or adoption of a child: Generally, no major problems were observed for this question. Two participants out of eight again noted it wasn’t personally relevant to them, so they weren’t sure what their employer’s policy on this was. One participant said it depends on the situation.

After asking questions about the reasons for taking paid leave, the next question focused on types of leave plans offered by employers:

PDTYPE_A1-A5

“Now I’d like to ask you about the types of paid leave plans offered by your employer. Some employers offer leave plans that require workers to use paid leave for specific purposes. For example, sick leave might be available but it cannot be used for vacation time. I’m going to read a list of different types of paid leave plans.  Please tell me if you have a separate leave plan available to you for...”

[Interviewer: If the respondent mentions that he/she has a consolidated leave plan or PTO, mark that column for vacation, sick leave, and personal days]

	Yes	No	PTO	DK
A1: Vacation
A2: Sick leave
A3: (females only) Separate plan for maternity leave
A4: (males only) Separate plan for paternity leave
A5: Personal days

After several in-house testing sessions, it became clear that the order of questions in Section 1 seemed redundant to respondents, took a long time to complete, and led to some seemingly inconsistent responses. The question, PDTYPE_A1-A5, sounded redundant because questions immediately preceding it had asked participants if they could take paid leave for a variety of reasons, three of which were repeated in the table in PDTYPE_A1-A5. So, for example, after answering that they could take paid leave for vacation and sick time, some participants wondered why another question asked if their work offered a separate leave plan that covered vacation and sick leave. The question also had a relatively long transition statement, which some participants asked to be repeated.

Similar to the section on reasons for taking paid leave, two participants also commented that they lacked specific knowledge about the existence of a separate plan for maternity leave because they had not encountered the need for it. Once again we found that several participants expressed uncertainty about certain features of their leave plans unless they had the occasion to use or ask about that particular feature. In addition, at least two participants weren’t familiar with the term “personal days.”

The next question asked respondents whether they had a Paid Time Off (PTO) plan.

PTO:

“Some employers also offer a paid leave plan, called a consolidated leave plan or PTO. In this type of plan you get a fixed number of hours, which you can then use for whatever reason you would like to take leave — whether you’re sick, going on a vacation, or need some personal time. Does your employer offer this type of plan?”

1. Yes

2. No

3. Don’t Know/Refusal

Based on results from the 2011 module, it was found that participants over-reported having a PTO plan. In an effort to avoid this issue, questions about leave types were asked before questions about having a PTO. For participants who have PTOs, the questions about leave types might be awkward because, at least for traditional PTO plans, employers would not offer separate types of paid leave plans for employees. Instead, all of these leave types would generally be consolidated into a single plan.²

Because of this, we anticipated that participants might mention having a PTO in response to the question PDTYPE_A1-A5 without being asked directly. But we found that this was rarely the case, and participants had different reactions to the questions about PTOs. Three participants answered as expected (“no”) to the question about having a paid sick leave plan, and other types of paid leave category in Question PDTYPE_A1-A5. When asked the PTO question later, they then revealed they had a PTO. One participant answered “yes” to having a separate paid sick leave plan, and also later revealed having a PTO. Another participant with a PTO was confused about how to answer, and remarked, “We only get one type of leave.” A final participant indicated that she was a bit confused about how to answer the questions about leave types since she has a single PTO that includes 15 days of leave for any reason. Once she got to the question about the PTO, it was clearer for her.

Revised Question about Types of Leave Plans Used in Protocol 2

In Protocol 2, the question about types of leave plans was asked as an open-ended question as shown in the box below, and was the first question asked in Section 1:

The next few questions ask if you get paid leave on your job. Can you briefly tell me how your current leave plan works? (for example, do you get paid vacation time or paid sick leave)

• Yes – has paid leave

• No

Interviewer Instruction -
If necessary, ask about paid vacation and sick leave using the wording below. Do not ask about a PTO plan, unless the respondent mentions one.

     Vacation:  Does your leave plan include a separate category for paid vacation time?
   Sick leave:  Does your leave plan include a separate category for paid sick leave?
            PTO:  Record answer, if mentioned.

	Yes	No
Vacation
Sick Leave
PTO

After asking this opening question, a brief conversation ensued in which respondents described the type of paid leave that was available (or not available) to them. This conversation provided enough information for the interviewer to answer the first yes/no question, and usually enough information to indicate yes or no in the table for vacation and sick leave (no participant mentioned a PTO at this time). If enough information was not obtained for the table entries, the interviewer would probe about vacation and sick leave plans, but not about a PTO because a later question asked directly about that type of plan. Thirteen participants answered this question using Protocol 2, with six indicating they had paid leave plans, and seven indicating that they did not.³

From an interviewer’s perspective, this approach seemed natural and conversational, and the following questions flowed well without a perception of redundancy. However, contrary to expectations, no respondent mentioned the existence of a PTO in the initial question, despite the fact that when asked directly about a PTO later in the interview, two respondents said they had that type of plan.

While testing the questions about types of leave plans, we learned PTO plans are more complex than we initially had realized. Employers offer different types of leave plans and different bundles of leave as PTO. For example, some PTOs bundle vacation, sick, and personal leave, whereas others combine sick and personal days while offering a separate category for vacation leave. Some of the seemingly inconsistent responses observed during the cognitive testing may have been due to respondents’ employers offering variations of a traditional PTO plan. Because of concerns the questions did not account for the complexity of leave plans, and additional concerns that respondents were not understanding the questions as intended, the question about PTOs was ultimately dropped from the questionnaire.

Sections 2 and 3

Section 2 dealt with “Access to unpaid leave,” and Section 3 dealt with “Leave taken in the last week.” No problems were observed in either section.

Section 4. Job flexibility and work schedules

JF_2

“Is your flexible work schedule part of a program or policy offered by your employer?”

1. Yes

2. No

DK/R

Although several people answered yes to this question, their responses and the follow-up debriefing indicated that in most cases the flexible schedule resulted from an informal agreement with their employer and not a program or policy generalized to everyone working in the business. As a result, the decision was made to change the wording to “Is your flexible work schedule part of a formal, written program or policy offered by your employer, or is it just an informal arrangement?”

JF_8:

“What is the main reason why you work this type of shift?”

1. Better arrangements for family or childcare

2. Better pay

3. Allows time for school

4. Could not get any other shift

5. Nature of the job

6. Personal preference

7. Some other reason – specify

Don’t know, Refusal

Ten participants answered the question about why they work a non-daytime shift. We found several minor problems, mostly with participants providing non-codeable responses that required additional interviewer probing. For instance, one graduate assistant with extremely flexible hours explained that she worked better during certain hours of the day. The interviewer asked whether this meant personal preference, which the participant confirmed. One participant said it was “flexibility,” which also seemed to fall under personal preference. Another participant elaborated, “This was the only shift available at the time,” and that her employer wanted her to work that shift because of her qualifications. In the final draft of the questionnaire, the instructions “If needed, read response Options 1-6” were added to address these types of issues.

Afterward, participants were asked a sequence of questions about what days of the week they work and about working at home. Most participants answered these questions without difficulty, and upon debriefing, interpreted them correctly.

JF_10:

“Which days of the week do you usually work?”

*Select all that apply

• Monday through Friday

• Sunday

• Monday

• Tuesday

• Wednesday

• Thursday

• Friday

• Saturday

• It varies

Don’t know, Refusal

Some minor problems arose with these questions. For instance, when asked what days of the week she usually worked (question JF_10), a participant who was a graduate assistant at first mentioned only three days per week, then requested to go back and clarify that she does some sort of work every single day of the week, but it depends on how we define “work.” Some days she only does small tasks, such as filing papers.

In this case, since work is respondent defined, interviewers should be instructed to accept the respondent’s definition.

JF_11:

“As part of your job, can you work at home?”

1. Yes

2. No

Don’t know, Refusal

During the question about working at home (JF_11), one participant asked for clarification: “Do I, or can I?” One participant answered, “Sometimes.” One participant, a bail bonds worker, noted that she is required by her job to work at different locations. When asked again, she said she does sometimes post bonds from home. Another participant mentioned during the debriefing that this question was redundant as she'd already indicated that she works from home earlier in the interview.

Interviewer instructions should clarify that any instance of an employer who allows working at home counts. If a respondent indicates that he or she works at various locations in the field or outside of the employer’s office, interviewers should probe as to whether one of these locations includes the home. If a respondent had previously indicated that he or she worked at home, this information can be verified.

JF_14:

“What is the main reason why you work at home?”

• Finish or catch up on work

• Coordinate work schedule with personal or family needs

• Reduce commuting time or expense

• Personal preference

• Other

Of the ten participants who said that they could work from home, very few problems occurred

when asked about whether they are paid for the hours that they work at home, or if they just take

work home from the job (question JF_13). However, the question about the main reason why they work at home (question JF_14) posed some minor issues, for example, seven participants did not provide answers that were immediately codeable. For instance, one participant said there were multiple reasons and probing was required to narrow it down to the main reason. Three participants seemed to give answers that mapped onto personal preferences, such as increased flexibility. Other reasons included being sick and one respondent mentioned slow days in the office (e.g., Fridays) where working at home or in the office would not matter because many people are out of the office. For this reason, an interviewer instruction should be added that says, “If needed, read response options 1-5.”

JF_15:

“Are there days when you work only at home?”

In addition, three participants had minor problems with the question about whether there are days when they work only at home. One participant answered, “Somewhat.” Another seemed to misinterpret the question as asking about people who have a set day on which they work from home, and one teacher said "yes" because she is allowed to work at home one weekday of the entire school year (to do planning, etc.). An interviewer instruction should be added that says “A person who works only at home does not go into the office on that work day. This does not have to be a set day or a certain number of days.”

Participants who worked at home were then asked the following question about how often they worked at home:

JF_16:

“How often do you work only at home?”

1. Everyday

2. At least once a week

3. Once every 2 weeks

4. Once a month

5. Less than once a month

Four out of 8 participants had problems with this question. One participant asked for clarification "on average?", and answered "2-3 times a month." The interviewer wondered about the best response option. Another participant noted that she mainly worked at home, but went into the office for occasional meetings. She worked only at home nearly every day, but not "everyday" (option 1) and quite a bit more than "at least once a week" (option 2). Another said that on average, she worked at home 2 days a week. But it “varies from week to week." This indicates that the participant was trying to compute an average number to answer this question. In each example, participants provided answers that didn’t map cleanly onto one of the response options. As such, the final questionnaire was revised to include more clear-cut categories.

Section 5: Nonuse of Leave

The final section was about times when workers needed to take leave in the past month, but could not. Of the 30 participants, most did not have trouble with this question (NOLV), but one participant asked for the question to be repeated, and another said it was difficult to recall these instances during the one month reference period. The following question was used to ask about the main reason they needed to take leave:

NEEDLV

“Why did you need to take off work?”

1. Own illness or medical care

2. Illness or medical care of another family member

3. Childcare, other than for illness

4. Eldercare

5. Vacation

6. Errands or personal reasons

7. Birth or adoption of a child

8. Other, specify

Don’t Know/Refusal

Of the 9 out of 30 participants who indicated that they needed to take leave in the past month but could not, only one participant gave an uncodeable response (“family obligations”), which could be coded as “personal reasons.”

Those nine participants were also asked the follow-up question about why they didn’t take leave:

RSNNO

“Why did you decide not to take leave?”

1. Too much work

2. Wanted to save leave

3. Leave was denied

4. Did not have enough leave

5. Fear of job loss or other negative employment-related consequence

6. Could not afford the loss in income

7. Other, specify

Don’t Know/Refusal

Some responses did not map onto the response categories. For instance, one participant indicated there was a school night at her daughter's school that was scheduled at the same time as an important work meeting. Another said she was unable to find work coverage. Again, these might require skilled probing from interviewers to arrive at a codeable response.

Finally, during debriefing, a hypothetical question was asked to learn more about how people think about leave taking. It stated, “Assume you worked in a job where you occasionally had a week or more off with no scheduled work, for example, because you were a landscaper who did not work when there was no work, or you were a teacher who had summers off. How would you answer the following question for the landscaper and the teacher if you had not worked the preceding week: ‘In the past seven days, how many hours of paid or unpaid leave did you take in total?”

Participants’ responses to this question varied, indicating that people think of leave taking in different ways, and people who are in professions that experience breaks in their work schedules may have difficulties answering questions about leave taking.

Detailed Coding Results

Appendix E shows codes that could have been assigned to individual questions to cover a variety of reactions to the questions. Codes that were assigned to individual questions during the testing are shown in Appendix F. These codes were categorized into the following four high-level categories:

1. Question comprehension

2. Response selection

3. Retrieval

4. Other, for example, answers that appear to be inconsistent with previous answers, negative comments about a question (it’s repetitive, difficult to read, awkward transition), or positive comments.

Although almost all of these codes reflect a possible problem with a question, one of the codes in the “Other” category included positive comments, but this code was dropped in the analysis described in this report.

Table 2, below, presents a summary of the codes assigned to the four high-level coding categories. (Note: Seven positive comments that were made in the Other category about Questions RSNUN_A to B have been deleted from Table 2).

Across all questions, the largest number of possible problems were noted in the categories of Question Comprehension and Response Selection, with 112 codes being assigned in those two categories. In addition, a count of the codes for each individual question has been summed across rows, and the order of the rows sorted based on the percentage of times a code was assigned. So, for example, Question RSNPD_C (“Please tell me if you are able to take paid leave for childcare, other than for illness”) generated a total of 10 codes across 15 test respondents (those participants who received paid leave), ⁴ with confusion about response selection being the most common cause. Similarly, Question RSNPD_D (“Please tell me if you are able to take paid leave for Eldercare”), generated nine codes across 15 participants who reported having paid leave.

It’s interesting to note that RSNPD_A (“Please tell me if you are able to take paid leave for your own illness or medical care”) and RSNPD_E (Vacation) generated no problems, whereas questions that asked about other reasons such as RSNPD_C (Childcare, other than for illness) and RSNPD_D (Eldercare) generated more confusion or reflected a simple lack of knowledge. As noted previously, participants seemed to immediately know that they could use paid leave for their own illness or medical care, but sometimes struggled with providing responses for the other reasons. This occurred primarily when a reason did not personally apply to them (for example, they did not have children or did not have eldercare obligations).

Table 2. Summary of Codes Assigned to Individual Items

(Question names refer to those used with the initial questionnaire in Protocol 1)

Question	Question Comprehension	Response Selection	Retrieval	Other	Sum of All Codes	No. of Times A Question Was Tested	Total % of Times A Code Was Assigned
RSNNO	1	1	0	3	5	9	55.6%
JF_8	0	4	0	1	5	10	50.0%
JF_7	1	1	1	2	5	10	50.0%
JF_16	1	3	0	0	4	8	50.0%
RSNPD_F	1	3	1	2	7	15	46.7%
RSNPD_D	1	2	3	1	7	15	46.7%
RSNPD_B	0	2	1	4	7	15	46.7%
PDTYPE_A2	1	2	1	3	7	15	46.7%
RSNPD_C	1	4	3	2	10	22	45.5%
JF_9A	3	3	0	0	6	14	42.9%
RSNUN_D	5	0	3	1	9	22	40.9%
RSNPD_G	0	1	3	2	6	15	40.0%
JF_2	4	1	1	0	6	16	37.5%
RSNUN_G	2	2	3	0	7	22	31.8%
JF_14	0	3	0	0	3	10	30.0%
JF_15	1	1	0	1	3	10	30.0%
PTO	2	1	1	1	5	17	29.4%
RSNUN_F	3	2	1	0	6	22	27.3%
RSNUN_A	5	1	0	0	6	22	27.3%
PDTYPE_SL	1	1	2	0	4	15	26.7%
PDTYPE_A3	0	0	4	0	4	15	26.7%
PDTYPE_A1	1	1	1	1	4	15	26.7%
JF_1	5	2	0	0	7	30	23.3%
PDTYPE_A5	0	1	1	1	3	15	20.0%
RSNUN_C	1	2	1	0	4	22	18.2%
RSNUN_B	1	0	3	0	4	22	18.2%
JF_5	3	0	2	0	5	30	16.7%
PDWK	0	0	0	1	1	6	16.7%
LVMAIN	1	0	0	0	1	6	16.7%
NEEDLV	1	2	0	1	4	30	13.3%
JF_6	0	3	1	0	4	30	13.3%
PDTYPE_A4	1	1	0	0	2	15	13.3%
NOLV	1	0	3	0	4	30	13.3%
JF_3	1	1	0	0	2	16	12.5%
PAID	1	1	1	0	3	30	10.0%
JF_13	1	0	0	0	1	10	10.0%
JF_12	1	0	0	0	1	10	10.0%
RSNUN_E	1	1	0	0	2	22	9.1%
UNPAID	0	1	1	0	2	30	6.7%
JF_11	1	1	0	0	2	30	6.7%
JF_4	1	0	0	0	1	16	6.3%
JF_1A	0	1	0	0	1	16	6.3%
JF_9	0	1	0	0	1	19	5.3%
JF_10	0	1	0	0	1	30	3.3%
RSNPD_E	0	0	0	0	0	15	0%
RSNPD_A	0	0	0	0	0	15	0%
LVHRS	0	0	0	0	0	8	0%
LEAVE	0	0	0	0	0	21	0%
*UNEVER	0	0	0	0	0	NA	NA
Total	54	58	42	33	180

*UNEVER was not asked in testing because it could not be programmed adequately in the online scripting language used.

Based on feedback from the observers, and considering the preceding results, the following changes were made to the questions as originally written in the specifications. Not all of these changes were based solely on results of the cognitive testing. Instead, results obtained from the prior administration of the question in the 2011 Annual Leave Module were also taken into consideration, as was the objective to limit the duration of the Module to five minutes.

1. PDTYPE_A. This question was dropped because participants were not making the distinction between the reasons for taking leave (asked in RSNPD_A–G)) versus the types of leave categories offered by their employers (asked in PDTYPE_A). Because of this, the question was not effective and sounded redundant to study respondents. Additionally, as noted previously, it took a relatively long time to ask and complete PDTYPE_A.

2. PDTYPE_SL. This question asked “Can you use paid sick leave for family members who are ill or need care?” It was a follow-up question for people who indicated they had a paid leave category specifically for sick leave. This question was dropped because the decision was made to drop questions about the types of leave plans employers offer. A preceding question (RSNPD_B) asked if the respondent could take paid leave (of any type) for “the illness or medical care of another family member,” and this question was retained.

3. PTO. A question that asked if an employer offered a “paid time off” or PTO plan was dropped.

4. RSNUN_A. The transition statement for this question that asked about approved reasons for taking unpaid leave was significantly shortened to “Would your employer approve of you taking unpaid leave for the following reasons …?” (and then individual reasons were read)

5. JF_2. We found in debriefing that the response options to this question didn’t always capture the worker’s arrangement with their employer. For instance, in many cases the flexible schedule resulted from an informal agreement with their employer and not a program or policy generalized to everyone working in the business. As a result, the decision was made to change the wording to “Is your flexible work schedule part of a formal, written program or policy offered by your employer, or is it just an informal arrangement?”

6. JF_3. A question that asked about core hours, “Are the times that you can begin and end work completely flexible, or are there certain hours of the day, such as 10 a.m. to 2 p.m., that you have to work each day?” was dropped to reduce the duration of the Leave Module.

7. JF_9. A question that asked respondents if an alternative work schedule was available was dropped to reduce the duration of the Leave Module.

8. HEALTH. A question about a respondent’s general health was dropped to shorten the length of the module.

5. Follow-up Online Testing

Results from the cognitive testing were discussed on a regular basis during the lab testing. During one discussion, a decision was made to change the wording for RSNUN_A and JF_2 as shown in Table 3 below. However, since these changes were suggested at the end of in-house testing when there were no additional participants, the decision was made to test these changes using an online, cognitive testing protocol.

Five online testers participated in this follow-up study. Because online sessions are limited to 20 minutes, which includes time spent explaining the testing process, conducting the test, and receiving feedback, testers were asked an abbreviated set of module questions that included the two revised questions. Participants were asked to talk aloud, to explain their answers, and how they arrived at their answers. These data were used to analyze how well the questions performed.

Table 3. Revisions Made to RSNUN_A and JF_2 after In-House Testing

	Original Version	Revised Version
RSNUN_A	I’m going to read a list of reasons why you might have to miss work. Assuming that you have received your employer’s approval, for each reason please tell me whether or not you are able to take time off from work without pay. Yes/No/DK/R	Would your employer approve of you taking unpaid leave for the following reasons …? Yes/No/It depends/DK/R
JF_2	Is your flexible work schedule part of a program or policy offered by your employer? Yes/No/DK/R	Is your flexible work schedule part of a formal, written program or policy offered by your employer, or is it just an informal arrangement? Yes, formal program or policy/No, informal arrangement/DK/R

The revised questions were tested and worked well. No problems, or potential problems, were noted in any of the five interviews. On a positive note, in RSNUN_A, three of the five participants made the distinction between the “it depends” and “don’t know” answers, which supported adding an “it depends” answer to this question.

Other changes that were made to the questionnaire based on testing results follow. Variable names refer to those in the final questionnaire (see Appendix A below).

1. In testing, it was determined that the response options for LVMAIN do not need to be read aloud.

2. The response options for JF_4 were clarified as distinct, non-overlapping categories.

3. In testing, it was determined that the response options for JF_6 (work shifts) do not need to be read aloud.

4. The response options for JF_15 (“How often do you work only at home?”) were modified to capture more detail.

5. The response options for NEEDLV are now read only if necessary.

6. The response options for RSNNO are now read only if necessary.

Appendix A: ATUS Leave Module Questionnaire with Specifications

Final version:

Protocol 1 (from March 8):

Appendix B: Coding Scheme for ATUS Leave Module Cognitive Testing

Question Comprehension

a	A comment was made that wording of the question stem was ambiguous, unclear, too abstract, or confusing
b	A comment was made that wording of a question response option was ambiguous, unclear, too abstract, or confusing
c	Respondent interpreted question incorrectly
d	(In-person) Respondent asked that the question be reread / (Online) Respondent had to read the question more than once
e	A comment was made that question was long or wordy
f	(For in person) Respondent interrupted reading of the question or response options / (For online) Respondent did not fully read the question and all of the response options out loud

Response Selection

g	None of the response option(s) applied; correct response option was missing
h	Struggled with selection of best response option
i	Answered question but expressed uncertainty or lack of confidence about accuracy

Retrieval

j	Gave (in-person) or Entered (online) a “don’t know” response Reason: could not recall information (e.g., number of hours of leave taken in past 7 days)
k	Reason: did not have the knowledge to answer question (e.g., does not know what type of leave their employer offers)
L	A comment was made that a question was not relevant to respondent
m	The respondent has difficulty recalling, formulating, or reporting the answer (e.g., remembering/calculating the number of hours/days of leave taken in past 7 days)

Other

n	A positive comment was made, for example, question was easy, straightforward, etc.
o	Answer to a current question is not in agreement with, or is inconsistent with, an answer to a previous question
p	Question was repetitive or asked about information provided earlier (e.g., Respondent indicated s/he has a PTO earlier, but must listen to the whole question/definition of PTO)
Q	(For in-person) Interviewer reading question stem or response options was awkward
r	(For in-person) Needs smoother transition from one section of the questionnaire to another

No problems

s	No problems noted

Question not asked

t	Check this box

Appendix C. In-House Consent Form and Testing Protocol

Consent Form

The Bureau of Labor Statistics (BLS) is conducting research to increase the quality of BLS surveys. This study is intended to suggest ways to improve the procedures the BLS uses to collect survey data.

The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent. The Privacy Act notice on the back of this form describes the conditions under which information related to this study will be used by BLS employees and agents.

During this research you may be audio and/or videotaped, or you may be observed. If you do not wish to be taped, you still may participate in this research.

We estimate it will take you an average of 60 minutes to participate in this research.

Your participation in this research project is voluntary, and you have the right to stop at any time. If you agree to participate, please sign below.

Persons are not required to respond to the collection of information unless it displays a currently valid OMB control number. The OMB control number is 1220-0141 and expires April, 30, 2018.

------------------------------------------------------------------------------------------------------------

I have read and understand the statements above. I consent to participate in this study.

___________________________________ ___________________________

Participant's signature Date

___________________________________

Participant's printed name

___________________________________

Researcher's signature

PRIVACY ACT STATEMENT

In accordance with the Privacy Act of 1974, as amended (5 U.S.C. 552a), you are hereby notified that this study is sponsored by the U.S. Department of Labor, Bureau of Labor Statistics (BLS), under authority of 29 U.S.C. 2. Your voluntary participation is important to the success of this study and will enable the BLS to better understand the behavioral and psychological processes of individuals, as they reflect on the accuracy of BLS information collections. The BLS, its employees, agents, and partner statistical agencies, will use the information you provide for statistical purposes only and will hold the information in confidence to the full extent permitted by law. In accordance with the Confidential Information Protection and Statistical Efficiency Act of 2002 (Title 5 of Public Law 107-347) and other applicable Federal laws, your responses will not be disclosed in identifiable form without your informed consent.

Introductory Script for In-House Testing

Hi! Thank you for coming in today. My name is ( ). I work for ( ) in the Bureau of Labor Statistics. [This is my colleague ( ).]

I [we] look forward to working with you today.

With your permission, I will be taping this session. This allows me to ensure the accuracy of your answers, and also allows me to focus on you and your responses instead of taking notes. Do I have your permission to audiotape?

Before we get started, please read over the consent form. If you agree to participate in the study, print and sign your name as indicated on the bottom of the form. If you do not agree, please let me know now.

Let me explain a bit about what we’ll be doing today. We are going to be working with some questions that might be asked as part of the American Time Use Survey, which is an ongoing national survey that provides information about how Americans spend their time. This survey first asks how you spent your time in the previous 24 hours. After these questions, some questions about the availability of paid and unpaid time off at your job will be asked. These questions are important because the ability to take time off is likely related to how people spend their time.

Today, we will be going through the survey questions together as if you were a real respondent, and then we will spend a few minutes talking about your experience – what you liked or disliked about the questions, what you found confusing, etc. Because this is a phone survey, I’ll actually be leaving the room in a minute to conduct the survey with you over the phone, but then I’ll come back in here afterward so we can discuss your reactions to the questions face-to-face.

It’s important to point out the purpose of today’s session is to help us find out more about how people respond to these questions. We want the questions to be clear, easy to understand, and easy to answer. So if a question is confusing or difficult for you to answer, or if it uses a word that you don’t understand, or the possible answers don’t apply to your situation, I want you to stop and tell me that immediately. That’s important information for us. Is that clear? All the information you give us will be kept completely confidential, and it will only be used to improve the survey questions. Do you have any questions so far?

Debriefing Questions

1. GENERAL REACTIONS

So, tell me. What were your general reactions to the questions about paid and unpaid leave? Did any of the questions stand out as particularly confusing?

2. COGNITIVE/LINGUISTIC ISSUES

Note: In addition to the scripted probes shown in the following protocol, emergent probes will be used during the interview.

III. Name/title of occupation: What is the name or title of your occupation?

Q1. (PAID)

Now I’d like to spend some time talking about some of the individual questions.

The first question asked was, “Do you receive paid leave on your job?”

What does the term paid leave mean to you?

Can you think of a simpler way to ask this question, or is it okay as is?

Q2. (PDTYPE_A)

This next question is a pretty long one. It asks:

“Now I’d like to ask you about the TYPES of paid leave plans offered by your employer. Some employers offer leave plans that require workers to use paid leave for specific purposes. For example, sick leave might be available, but it cannot be used for vacation time. I’m going to read a list of different types of paid leave plans. Please tell me if you have a separate leave plan available to you for ...

• Vacation

• Sick leave

• (females only) Separate plan for maternity leave

• (males only) Separate plan for paternity leave

• Personal days

What did you think of when you heard the phrase “paid leave for specific purposes?” Was the example about not using sick leave for vacation time clear?

Was it clear this question was asking about types of leave plans, and not the reasons for taking leave?

What did you think of when you heard the choice “Separate leave plan (for maternity/paternity leave)?” What does that mean to you?

What did you think of when you heard the choice “Personal days?” What does that mean to you?

Q3 (PTO)

Another question asked about a new type of leave plan that some employers are offering, called a consolidated leave plan or PTO. Had you ever heard of this type of plan prior to this interview?

What's the simplest explanation you can give of a consolidated leave plan or PTO?

Q4. (UNPAID)

Did you have any difficulties answering the question “[In addition to your paid leave/In your main job,] Are you allowed to take time off from work without pay?

Was this question difficult or easy to answer? Why is that?

Q5. (LVHRS)

One of the questions asked if you had taken any time off in the past seven days.

Was it difficult to remember if, and why, you had taken time off from work in the last seven days?

Was it clear that the question was asking about both paid and unpaid leave?

(if took time off) How confident are you in your estimate of the number of hours you had taken off? Would you say you are very confident, confident, slightly confident, or not at all confident?

What if we had asked about the previous two weeks? Would you say your estimate would be very accurate, accurate, inaccurate, or very inaccurate?

Q6. (JF_1, and JF_1A)

Some of the questions asked about how much flexibility you had in determining your start and end times for work, as well as your overall work schedule.

What did it mean to you when you heard the question “Do you have flexible work hours that allow you to vary or make changes in the time you begin and end work?”

(If can vary start & end times) We asked if you can change the times you begin and end work on a frequent basis, occasionally, or only rarely? How did you define frequent, occasionally, and rarely?

Q7 (JF_7)

(If some other schedule)

Did you find a good match for your work schedule?  If not, please explain why.

(If yes) How would you describe it?

Q8 (JF_9A)

One question asked “Do you work an alternative work schedule where you can

Work fewer days during a pay period by working more hours per day?”

 Is it clear what this question is asking?

What does an “alternative work schedule” mean to you in your own words?

Q9 (JF_13)

(If worked at home)

One question asked “Are you paid for the hours that you work at home, or do you just take work home from the job?”

Was this question easy or hard to answer?

Are you a salaried worker, a contract worker, or do you get paid by the hour?

(if salaried worker) Do you consider the hours that you work at home “paid hours?” Why or why not?

Q10 (JF_14)

(If worked at home)

Did you have any problem answering the question “What is the main reason why you work at home?” (Please explain)

Q11 (JF_15)

Another question asked:

Are there days when you work only at home?

In your own words what does it mean to work only at home?

Q12 (NOLV)

We asked you if there were situations in the past month when you needed to take off time from work, but did not.

Was this question easy or difficult to answer? (Why?)

[If R said ‘NO’ to NOLV, ask:] Can you think of any times when you might have needed to take off from work but did not – it doesn’t have to be in the last seven days?

Vignette

I want to ask you about a hypothetical situation.  Assume you worked in a job where you occasionally had a week or more off with no scheduled work, for example, because you were a landscaper who did not work when there was no work, or you were a teacher who had summers off.

How would you answer the following question for the landscaper and the teacher if you had not worked the preceding week? 

In the past seven days, how many hours of paid or unpaid leave did you take in total?”

Wrap-Up

Are there any questions that I didn’t think to ask you about how to collect this information well?

If I can relay only one message to the team that’s developing these questions, what’s the most important thing that you think I should tell them about collecting information about leave?

Do you have any other reactions or comments about the survey or this cognitive interview before we end this session?

Thanks for your time and assistance!!

Appendix D: Protocols 1 and 2

Appendix E: Cognitive Interviewing Codes

Comprehension

Code	Frequency	Description
A	7	A comment was made that wording of the question stem was ambiguous, unclear, too abstract, or confusing
B	0	A comment was made that wording of a question response option was ambiguous, unclear, too abstract, or confusing
C	21	Respondent interpreted question incorrectly
D	21	(In-person) Respondent asked that the question be reread / (Online) Respondent had to read the question more than once
E	0	A comment was made that question was long or wordy
F	5	(For in person) Respondent interrupted reading of the question or response options / (For online) Respondent did not fully read the question and all of the response options out loud

Response Selection

G	25	None of the response option(s) applied; correct response option was missing
H	30	Struggled with selection of best response option
I	3	Answered question but expressed uncertainty or lack of confidence about accuracy

Retrieval

J	0	“Don’t know” response Reason: could not recall information (e.g., number of hours of leave taken in past 7 days)
K	29	Reason: did not have the knowledge to answer question (e.g., does not know what type of leave their employer offers)
L	4	A comment was made that a question was not relevant to respondent
M	9	The respondent has difficulty recalling, formulating, or reporting the answer (e.g., remembering/calculating the number of hours/days of leave taken in past 7 days)

Other

N	7	A positive comment was made, for example, question was easy, straightforward, etc.
O	7	Answer to a current question is not in agreement with, or is inconsistent with, an answer to a previous question
P	9	Question was repetitive or asked about information provided earlier (e.g., Respondent indicated s/he has a PTO earlier, but must listen to the whole question/definition of PTO)
Q	9	(For in-person) Interviewer reading question stem or response options was awkward
R	1	(For in-person) Needs smoother transition from one section of the questionnaire to another

No problems

S

790

No problems noted

Question not asked

T

579

Check this box

Appendix F: Codes Assigned to Individual Questions

Question

a

b

c

d

e

f

g

h

i

j

k

l

m

n

o

p

q

r

s

t

PAID	0	0	1	0	0	0	0	1	0	0	0	0	1	0	0	0	0	0	29	0
RSNPD_A	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	24	7
RSNPD_B	0	0	0	0	0	0	0	2	0	0	0	0	1	0	1	1	1	1	14	12
RSNPD_C	0	0	0	0	0	1	1	3	0	0	2	1	0	0	0	1	1	0	11	12
RSNPD_D	1	0	0	0	0	0	0	2	0	0	2	1	0	0	0	1	1	0	13	12
RSNPD_E	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	24	7
RSNPD_F	0	0	0	1	0	0	2	1	0	0	1	0	0	0	0	1	1	0	13	12
RSNPD_G	0	0	0	0	0	0	0	1	0	0	2	1	0	0	0	1	1	0	16	12
PDTYPE_A1	0	0	1	0	0	0	1	0	0	0	0	0	1	0	1	0	0	0	16	11
PDTYPE_A2	0	0	1	0	0	0	1	1	0	0	0	0	1	0	3	0	0	0	12	12
PDTYPE_A3	0	0	0	0	0	0	0	0	0	0	4	0	0	0	0	0	0	0	11	18
PDTYPE_A4	0	0	1	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	6	23
PDTYPE_A5	0	0	0	0	0	0	0	1	0	0	1	0	0	0	0	1	0	0	15	13
PDTYPE_SL	0	0	0	1	0	0	0	1	0	0	2	0	0	0	0	0	0	0	15	12
PTO	0	0	1	1	0	0	0	0	1	0	1	0	0	0	0	1	0	0	14	12
UNPAID	0	0	0	0	0	0	0	1	0	0	1	0	0	0	0	0	0	0	29	0
RSNUN_A	1	0	2	2	0	0	0	1	0	0	0	0	0	1	0	0	0	0	19	8
RSNUN_B	0	0	1	0	0	0	0	0	0	0	3	0	0	1	0	0	0	0	20	7
RSNUN_C	0	0	1	0	0	0	2	0	0	0	1	0	0	1	0	0	0	0	19	8
RSNUN_D	0	0	2	3	0	0	0	0	0	0	2	1	0	1	0	0	0	0	18	7
RSNUN_E	0	0	1	0	0	0	0	1	0	0	0	0	0	1	0	0	0	0	20	9
RSNUN_F	1	0	2	0	0	0	2	0	0	0	1	0	0	1	0	0	0	0	18	8
RSNUN_G	0	0	1	1	0	0	2	0	0	0	3	0	0	1	0	0	0	0	17	7
UNEVER	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	17	14
LEAVE	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	20	11
LVHRS	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	0	9	21
PDWK	0	0	0	0	0	0	0	0	0	0	0	0	0	0	1	0	0	0	4	26
LVMAIN	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	7	23
JF_1	0	0	4	1	0	0	1	1	0	0	0	0	0	0	0	0	0	0	24	0
JF_1A	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	16	14
JF_2	1	0	0	3	0	0	0	1	0	0	1	0	0	0	0	0	0	0	11	15
JF_3	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	11	19
JF_4	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	20	10
JF_5	1	0	1	1	0	0	0	0	0	0	1	0	1	0	0	0	0	0	27	0
JF_6	0	0	0	0	0	0	2	1	0	0	0	0	1	0	0	0	0	0	27	1
JF_7	0	0	0	0	0	1	0	1	0	0	0	0	1	0	0	1	1	0	6	21
JF_8	0	0	0	0	0	0	2	2	0	0	0	0	0	0	0	0	1	0	7	21
JF_9	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	31	0
JF_9A	1	0	1	1	0	0	0	2	1	0	0	0	0	0	0	0	0	0	16	10
JF_10	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	30	0
JF_11	0	0	0	1	0	0	0	1	0	0	0	0	0	0	0	0	0	0	28	1
JF_12	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	13	17
JF_13	0	0	0	1	0	0	0	0	0	0	0	0	0	0	0	0	0	0	11	19
JF_14	0	0	0	0	0	0	2	1	0	0	0	0	0	0	0	0	0	0	9	19
JF_15	0	0	0	0	0	0	0	1	0	0	0	0	0	0	0	0	0	0	11	19
JF_16	1	0	0	0	0	0	3	0	0	0	0	0	0	0	0	0	0	0	7	21
NOLV	0	0	0	1	0	0	0	0	0	0	1	0	2	0	0	0	0	0	21	7
NEEDLV	0	0	0	0	0	1	2	0	0	0	0	0	0	0	0	0	1	0	8	20
RSNNO	0	0	0	0	0	1	1	0	0	0	0	0	0	0	1	1	1	0	6	21

1 https://www.trymyui.com/ TryMyUI is an online testing website where respondents can complete a set of self-administered tasks while thinking “out loud” and can respond to follow-up, scripted probes. TryMyUI provides a video recording of the output and each test can last up to 20 minutes.

2 We discovered during the testing that provisions of PTO plans were more complex than we realized.

3 There were actually 15 participants in Protocol 2, but the SurveyMonkey instrument used to collect the data was missing data for two participants for this question.

4 There were 31 total respondents in the in-house testing, but one of them was a phone interview that was conducted on paper and was not entered in the online system for inclusion in Table 2.