Note to Reviewer - ATUS Sleep Patterns Study

October 14, 2015

NOTE TO THE REVIEWER OF:	OMB CLEARANCE 1220-0141 “Cognitive and Psychological Research”
FROM:	Robin Kaplan Research Statistician Office of Survey Methods Research
SUBJECT:	Submission of Materials for the Online Testing of American Time Use Survey Sleep Measures and Sleep Estimate Questions

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 429 hours.

If there are any questions regarding this project, please contact Robin Kaplan at

202-691-7378 or Polly Phipps 202-691-7513.

1. Introduction and Purpose

The American Time Use Survey (ATUS) uses a time diary format to ask respondents about the activities they performed on the prior day. In 2012, the ATUS reported that members of the civilian, non-institutionalized population in the United States spend an average of 8.7 hours per day sleeping, and that number has remained similar over time. ATUS estimates of sleep are consistently higher than surveys that ask respondents directly about the amount of sleep they get (referred to in this protocol as stylized questions). The National Sleep Foundation (NSF), for example, asks respondents to estimate the amount of time they spend sleeping on an average weekday and weekend day. These numbers are then combined into an overall estimate. The NSF reports that American adults get an average of 6.9 hours of sleep per day. Other national surveys that use stylized questions about sleep, such as The National Health Interview Survey (NHIS), the Behavioral Risk Factor Surveillance System (BRFSS), and the National Health and Nutrition Examination Survey (NHANES) report sleep duration estimates ranging from 6.9 to 7.1 hours.

The ATUS differs from surveys that use stylized questions in methodology, sleep definition, and population. It is unclear how these differences influence the estimates. OSMR conducted a previous cognitive interview study to help understand the measurement properties of both the ATUS interview as it relates to sleep and of a sample of stylized questions about sleep. In that study, we also found that diary and stylized measures differ from one another, consistent with prior research (e.g., Kan & Pudney, 2007). In particular, we also found that the diary sleep estimates were greater than the stylized sleep estimates. In addition, we found a number of differences across participants in how they answered questions about sleep in general and also across the diary versus stylized questions at each stage of the response process (as defined by Tourangeau, 1984).

The main findings included differences in comprehension – participants with a broad definition of sleep also reported getting more sleep than those with a narrower conception of sleep. Participants relied more on general estimation cues than memory to answer the stylized questions about sleep versus the diary questions. We also found evidence that stylized questions may be more susceptible to social desirability concerns than the Diary measure, that is, social desirability concerns about getting too much or too little sleep seemed to affect how participants answered the stylized questions about sleep, but not the diary measure. Participants also indicated that if they thought the survey was about jobs and employment, it would be more embarrassing to report oversleeping. In contrast, if a survey was about health, it would be more embarrassing to report undersleeping. These findings indicate that the context of the survey may also impact estimates of sleep duration. Since surveys across various agencies are framed differently, this may affect the sleep duration estimates obtained.

OSMR’s previous cognitive interview study had a very small sample, with only 29 participants who were all recruited from the Washington, DC metro area. Because of this, it wasn’t possible to make meaningful comparisons to determine whether demographic factors such as employment status or the presence of children in the household affected sleep estimates. The goal of the present study is to assess the difference between diary and stylized measures of sleep duration using a larger sample size that more closely resembles the U.S. population. We also hope to test some key hypotheses generated by the cognitive interview study, including whether providing respondents with a more specific definition of sleep affects their answers to the diary and stylized measures, and whether the context of the survey affects people’s estimates of how much they slept. To do so, participants in this study will again be asked both ATUS time diary questions and stylized sleep questions via an online instrument. Participants will be recruited using Amazon.com’s Mechanical Turk.

2. Research Design

Participants will be recruited from the general population using Amazon.com’s Mechanical Turk platform. Participants who agree to participate will be routed to an online survey, hosted on Qualtrics.com, where they will be asked to complete a modified version of the ATUS daily recall diary interview and answer a set of stylized questions regarding how much they sleep on an average weekday and about other activities (e.g., time spent engaged in work-related activities, physical exercise, household chores, travel, and use of information-technology devices). Participants will complete the surveys only on Tuesday – Friday so their ATUS recall of the prior day’s activities will always cover a weekday. Questions about where activities took place and who else was present (typical ATUS recall questions) will not be asked. After completing the diary and stylized time measures, participants will be asked questions aimed at understanding how they answered the questions, including questions about scheduling, social desirability, and demographics. See Appendix A for the full study protocol.

The order of the ATUS interview and the stylized questions will be counterbalanced such that half of the participants will be asked the ATUS interview questions followed by the stylized questions and the other half will answer the stylized questions followed by the ATUS interview questions. A number of Stylized questions related to common activities in the ATUS will also be included to make the topic of sleep less salient and to fit with the framing of the survey. In addition to answering the diary and stylized time-use questions, we will also vary the following factors: Framing of the study (i.e., participants will be told the survey is about jobs and employment, about health and wellness, or just about how Americans spend their time), and whether or not they are provided definitions of a subset of activities (including sleep) that are asked in the ATUS.

The table below illustrates the study design, a 3x2x2 mixed-model design with 2 between-subjects factors (Framing and Definition) and one within-subjects factor (type of measure), as follows: 3 Study Framing types (Jobs, Health, or No framing) X 2 Definition types (definitions provided or not provided) X 2 Types of Measures (Diary vs. Stylized, with order counterbalanced). For the different framing conditions, a banner will appear across the top of each survey indicating whether the survey is about Jobs and Employment, Health and Wellness, or how people spend their time. See Appendix B for the survey banners.

Between-Subjects Manipulations		Within-Subjects Manipulation
Survey Framing	Activity definitions provided	Type of Measure (Order Counterbalanced)
Framing of “Survey about Jobs and Employment”	Definitions provided No definitions provided	Diary then Stylized Stylized then Diary
Framing of “Survey about Health and Wellness”	Definitions provided No definitions provided	Diary then Stylized Stylized then Diary
Control (no framing)	Definitions provided No definitions provided	Diary then Stylized Stylized then Diary

*N = 1283, n=106 per condition

The analyses will focus on possible reasons for a difference between the diary and stylized measures, if prior research showing a difference between these types of questions is replicated. We believe it is possible that framing the survey about jobs and employment will discourage people from reporting a lot of sleep, and framing the survey as about health and wellness will encourage people to report more sleep, or what is considered a “normative” amount of sleep (e.g., 8 hours). We further hypothesize that providing definitions of sleep and other activities will decrease variation and eliminate some measurement error in reported sleep duration (e.g., by discouraging people from including non-sleep activities, such as resting in bed without actually being asleep). We will examine all main effects and interactions across the variables, along with participant demographics.

In addition, we will also include two individual differences measures that might be related to how participants self-report about their sleep habits.

1. Balanced Inventory of Socially Desirable Responding (BIDR; Paulhus, 1984). As evidenced from the cognitive interview results from previous studies, social desirability concerns may affect how respondents report on their sleep. Participants who exhibit high versus low levels of social desirability concerns may be differentially affected by the framing and context of the survey in which they report on sleep. Thus, a potential covariate for sleep duration in this study is participants’ own tendency to respond in socially desirable ways. Participants will complete a short version of Paulhus’s (1984) Balanced Inventory of Socially Desirable Responding (BIDR) impression management subscale, which assesses deliberate self-presentation to an audience. It contains 10 items (e.g., I sometimes tell lies if I have to.) See the full protocol in Appendix for the entire scale.

2. Time Crunch Scale (TCS; Robinson & Michelson, 2010). Another factor related to sleep duration estimates is the desire to “appear busy.” Researchers have documented that people might be motivated to underreport how much they sleep as it is highly desirable in American culture to appear very busy (Michelson, 2014; Robinson & Michelson, 2010). As such, they developed the “Time crunch” scale to assess people’s feelings about different aspects of their daily life as they relate to how pressed for time they feel throughout the day, or their perceptions of how busy they are. The greater the number of the time crunch statements participants agree with (e.g., “When you need more time, do you tend to cut back on your sleep?”), the greater the degree of a “time crunch” they experience. See the full protocol in Appendix for the entire scale.

3. Participants

Participants will be recruited using a convenience sample from Amazon Mechanical Turk of adult, U.S. citizens (18 years and older); this study is focused on internal validity rather than representativeness of any population. This research design requires a large sample of 1283 participants in order to sufficiently explore the range of variables of interest and because we expect a very small effect size since, as the study manipulations are subtle for online surveys of this nature. This sample size also takes into account break-offs, incomplete data, and participants who do not follow the task instructions, and this is similar samples used for studies of this nature. These participants will be randomly assigned to the 12 groups described (a 3x2 design repeated over two survey orders with 106 participants per group).

An additional 3 participants will be recruited for an initial pilot test from TryMyUI.com. TryMyUI is an online testing website where respondents can complete a set of self-administered tasks while thinking “out loud” and respond to follow-up, scripted probes. TryMyUI provides a video recording of the output and each test can last up to 20 minutes. These pilot participants will be asked to think aloud while completing the Diary questions and answer questions about the experience afterwards, which will help to confirm whether the instructions on how to fill out the online time diary is effective, the task is clear, the questions are worded clearly, and the instrument works as intended. The pilot tests will be conducted iteratively so that any modifications can be tested with pilot participants before launching the full study.

4. Burden Hours

Our goal is to obtain responses from 1283 participants recruited from Amazon Mechanical Turk. Each session is expected to take no more than 20 minutes to complete, for a total of 428 burden hours. In addition, the 3 pilot participants are expected to take no more than 20 minutes each, for an additional 1 burden hour. Total burden hours are expected to total no more than 429 hours. The survey will be administered completely online at the time and location of the participant’s choosing.

5. Payment to Respondents

We will recruit 1283 participants from the Amazon Mechanical Turk database. Participants will be compensated $2.00 for participating in the study, a typical rate provided by Mechanical Turk for similar tasks. A total of $2,566.00 will be paid to respondents for their participation in the study. The pilot participants recruited from TryMyUI will receive the standard TryMyUI fee of $10 each for their participation. The payment for these additional 3 pilot participants will total $30.

6. Data Confidentiality

Recruiting of participants will be handled by Amazon Mechanical Turk. Participants will be informed that the study is about their perceptions of different types of questions. Once participants are recruited into the study, they will be sent a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers. Participants in this study will not be asked to provide any personally identifiable information. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Appendix A – ATUS Sleep Measures Online Study Protocol

Welcome!

Thanks for your interest in our research. We’re conducting this study to better understand how people spend their time as it relates to <JOBS AND EMPLOYMENT / HEALTH AND WELLNESS / CONTROL FRAMING>.

…Work and work-related activities. We will use this information to estimate the amount of time people spend working and doing other related activities. [JOB FRAMING]

…Health and wellness-related activities, such as exercise, sleep, and leisure activities. We will use this information to estimate the amount of time people spend doing healthy activities. [HEALTH FRAMING]

…How people spend their time. [CONTROL]

Unlike some surveys or online tasks, we ask that you complete this task all at one time. Please begin only when you are in a quiet place where you won't be disturbed for about 20 minutes.

Please do not use your browser's back button.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Select ‘Next’ to continue

---page break---

Diary Introduction

On the next few pages, you will be prompted to think back to the last 24 hours, starting at 4 am yesterday morning and continuing through 4 am TODAY <INSERT DATE>. Please think back to each activity you did in as much detail as possible and what time you started and ended each activity. By activity, we mean anything you did during that time frame.

[Definition condition only]: Here are some examples of what we mean by activities:

• Work and work-related activities: By work and work-related activities, we mean time spent working, doing activities as part of one’s job, engaging in income-generating activities, and looking for jobs and interviewing. It does not include regular breaks, lunch breaks, or commuting to and from work.

• Sleeping: By sleep, we mean the number of hours you actually spend sleeping. This may be different from the number of hours you spend in your bed, time you spend preparing to go to sleep, resting with your eyes closed but not actually asleep. Please include any times you were sleeping during the day (or napping).

• Sports, exercise, and recreation: By sports and recreation, we mean organized physical activities that may be competitive in nature. By exercise, we mean activities that require physical exertion for fitness purposes.

On the next screen, you will be asked to select the type of activity you did from a drop-down list and what time you started and stopped that activity. If you don’t see the activity you were doing, please select ‘Other Activity’.

Please be sure to account for your whole day, so the end time of one activity should be the start time of the next activity. There should not be any gaps.

Below is an example of a completed version of how the activity log should appear:

*Note that the drop-down menus for this study includes the digits 0-12 for hour, and the digits 0-60, in intervals of 5, for minutes. When we ask for clock time, we will use the singular (i.e., hour, minute) to signify a precise start and stop time. When we ask for duration, we will use the plural (i.e., hours, minutes) to signify to participants that they should indicate the number of hours and minutes they spent on an activity.

Select ‘Next’ to continue

---page break---

Think back to 4:00 am YESTERDAY morning.

Please select the activity that you were doing.

[Drop-down menu]:

• Sleeping

• Grooming

• Watching TV

• Work and Work-Related Activities

• Eating and Drinking

• Household work

• Shopping

• Traveling and going from place to place

• Leisure activity

• Sports, exercise, and recreation

• Studying or learning

• Other activity, specify: ___________

Think back to what time you stopped doing that activity.

Enter what time this activity ENDED:

hour: ___; minute ___ ; _____ AM/PM

---page break---

Thank you. Please fill out what you did next until 4 am this morning <INSERT DATE>. Please remember to account for your whole day, so the end time of one activity should be the start time of the next activity. There should not be any gaps.

<Participants will be presented with a 20-row matrix of drop-down menus (as seen above in the image of the Time Diary) where they can enter each activity they did, including start and stop times, for the remainder of the 24-hour period>

---page break---

Last Week Stylized Questions Instructions

Thank you for completing the activity log. Next, we’d like to know more about your activities during the previous week, that is, during the past 7 days from <DATE> to <DATE>.

Please think carefully before giving your answers and be as accurate and as specific as possible.

Select ‘Next’ to continue

---page break---

1.) [INSERT DEFINITION IF IN DEFINITION CONDITION]___Thinking back to the past week (that is, during the previous <DATE> to <DATE>), how many hours did you work, or engage in work-related activities? _____hours; _____ minutes

2.) How many hours per week do you usually work at your main job (or engage in work-related activities)? ____ hours; ____ minutes

3.) During the past week, on average, at what time did you start working each day?

____ hour; ____ minute; _____ AM/PM

4.) During the past week, on average, at what time did you stop working each day?

____ hour; ____ minute; _____ AM/PM

---page break---

1.) [INSERT DEFINITION IF IN DEFINITION CONDITION] Thinking back to the past week (that is, during the previous 7 days), how many hours did you engage in sports, exercise, or other recreational activities? ___ hours; ____ minutes

2.) How many hours per week do you usually engage in sports, exercise, or other recreational activities? ___ hours; ____ minutes

---page break---

1.) Thinking back to the past week (that is, during the previous 7 days), how many hours did you engage in household chores and other cleaning activities? ___ hours; ____ minutes

2.) Thinking back to the past week (that is, during the previous 7 days), how many hours did you use technological devices connected to the Internet, such as your personal computer, cellular phone, tablet, or e-Reader, for leisure (non-work related) purposes? ___ hours; ____ minutes

3.) Thinking back to the past week (that is, during the previous 7 days), how many hours did you spend traveling and going from place to place, such as commuting to and from work and other places? ____ hours; ____ minutes

---page break---

1.) [INSERT DEFINITION IF IN DEFINITION CONDITION]Thinking back to the past week (that is, during the previous 7 days), how many hours of sleep did you get on average each weeknight (excluding weekends)?

____ hours; ____ minutes

2.) During the past week, on average, at what time did you fall asleep each weeknight (excluding weekends)?

____ hour; ____ minute; ____AM/PM

3.) During the past week, on average, at what time did you wake up each weeknight (excluding weekends)?

____ hour; ____ minute; ; _____ AM/PM

4.) During the past week, did you take any naps during the weekdays? By naps, we periods of short sleep episodes during the day.

• Yes

• No

5.) (If yes), During the past week, how many naps did you take during the weekdays? ____

6.) During the past week, on average, how long did your nap(s) last?

____ hours; ____ minutes

---page break---

1.) The previous questions asked about your sleep over the past week. Now we’d like to know about your sleep in general. How many hours do you sleep on a typical weeknight?

___ hours; ___ minutes

Follow-up Questions

1.) How confident are you in the estimate you gave of the number of hours you slept on average each weeknight during the past week, from <DATE> to <DATE>?

• Extremely confident

• Very confident

• Moderately confident

• Slightly confident

• Not at all confident

2.) Do you usually wake up at the same time every weekday morning?

• Yes

• No

3.) Do you usually go to sleep at the same time every weeknight?

• Yes

• No

4.) How many hours do you think is appropriate for an average person, with a similar schedule to yours, to sleep in one night?

_____ hours; ____ minutes

5.) How many hours do you think most people would have to sleep before feeling embarrassed for sleeping too MUCH in one night? ____ hours; ____ minutes

6.) How embarrassing do you think it would be for most people to admit they slept more than <Insert Answer from #5> hours in one night?

• Extremely embarrassing

• Very embarrassing

• Moderately embarrassing

• Slightly embarrassing

• Not at all embarrassing

7.) How few hours do you think most people would have to sleep before feeling embarrassed for sleeping too LITTLE in one night? ____ hours; ____ minutes

8.) How embarrassing do you think it would be for most people to admit they slept fewer than <insert answer from #6> hours in one night?

• Extremely embarrassing

• Very embarrassing

• Moderately embarrassing

• Slightly embarrassing

• Not at all embarrassing

---page break---

Thank you for completing the previous section. We’re almost done – please answer the following questions about yourself.

Balanced Inventory of Socially Desirable Responding

1 2 3 4 5

(strongly disagree) (strongly agree)

1. I sometimes tell lies if I have to.

2. I never cover up my mistakes.

3. I always obey laws, even if I am unlikely to get caught.

4. I have said something bad about a friend behind his or her back.

5. When I hear people talking privately, I avoid listening.

6. I have received too much change from a salesperson without telling him or her.

7. When I was young I sometimes stole things.

8. I have done things that I don’t tell other people about.

9. I never take things that don’t belong to me.

10. I don’t gossip about other people’s business.

---page break---

Time Crunch Scale

Please answer the following questions about yourself.

How frequently do you feel rushed?

• Always

• Often

• Sometimes

• Rarely

• Never

How frequently do you have time on your hands that you didn’t know what to do with?

• Always

• Often

• Sometimes

• Rarely

• Never

(All questions below will appear on a ‘Yes’ or ‘No’ scale)

1.) Do you plan to slow down in the coming year?

2.) Do you consider yourself a workaholic?

3.) When you need more time, do you tend to cut back on your sleep?

4.) At the end of the day, do you often feel that you have not accomplished what you had set out to do?

5.) Do you worry that you don’t spend enough time with your family or friends?

6.) Do you feel that you’re constantly under stress trying to accomplish more than you can handle?

7.) Do you feel trapped in a daily routine?

8.) Do you feel that you just don’t have time for fun anymore?

9.) Do you often feel under stress when you don’t have enough time?

10.) Would you like to spend more time alone?

---page break---

Demographic information

1.) Which of the following best describes you?

• Employed full time

• Employed part time

• Unemployed

• Student

• Retired

2.) What is your gender?

• Male

• Female

• Other

3.) What is your age as of today?

____ years

4.) What is your household size? By household, we mean the number of people currently living in your residence, including yourself.

• 1 person (including yourself)

• 2 people

• 3 people

• 4 people

• 5 or more people

5.) How many of the <insert answer from #4> people in your household are under the age of 16?

6.) How many of the <insert answer from #4> people in your household are under the age of 5?

7.) How personal a topic do you think it is for most people to talk about how much they sleep?

• Not at all personal

• A little personal

• Moderately personal

• Very personal

• Extremely personal

8.) Is talking about how much you sleep a sensitive topic for you?

• No

• Somewhat

• Yes

9.) (If Yes or Somewhat), How sensitive a topic is it for you to talk about how much you sleep?

• A little sensitive

• Moderately sensitive

• Very sensitive

• Extremely sensitive

Please explain (optional): ________________________

10.) How well-rested do you feel right now?

• Very rested

• Somewhat rested

• A little rested

• Not at all rested

---page break---

Thank you for your participation! If you have any additional thoughts on this survey, please provide them in the space below. ______________

Appendix B – Survey Banners

Health and Wellness Time Use Survey
_________________________________________________________________________________________

Work-Related Activities Time Use Survey
_________________________________________________________________________________________

[Control; no picture] Time Use Survey

_________________________________________________________________________________________

References

Kan & Pudney (2007). Measurement Error in Stylised and Diary Data on Time Use. Sociological Methodology 38, 101-132.

Michelson, W. (2014). Unraveling the mystery of sleep duration dynamics; Sleep in the objective and subjective lives of employed men and women. electronic International Journal of Time Use Research,, 57.

Paulhus, D. L. (1984). Two-component models of socially desirable responding. Journal of personality and social psychology, 46(3), 598.

Robinson, J. P., & Michelson, W. (2010). Sleep as a victim of the “time crunch”–A multinational analysis. electronic International Journal of Time Use Research, 7(1), 61-72.

Tourangeau, R. (1984). Cognitive science and survey methods. Cognitive aspects of survey methodology: Building a bridge between disciplines, 73-100.

Page 18 of 18