Note to Reviewer - ATUS Sleep Study

Note to Reviewer - ATUS Sleep Online Panel Study.docx

Cognitive and Psychological Research

Note to Reviewer - ATUS Sleep Study

OMB: 1220-0141

⚠️ Notice: This form may be outdated. More recent filings and information on OMB 1220-0141 can be found here:

Document [docx]

Download: docx | pdf

April 6, 2017

NOTE TO THE

REVIEWER OF:

OMB CLEARANCE 1220-0141

“Cognitive and Psychological Research”

FROM:

Robin Kaplan

Research Statistician

Office of Survey Methods Research

SUBJECT:

Submission of Materials for the Online Panel Testing of American Time Use Survey Sleep Measures and Sleep Estimate Questions

Please accept the enclosed materials for approval under the OMB clearance package 1220-0141 “Cognitive and Psychological Research.” In accordance with our agreement with OMB, we are submitting a brief description of the study.

The total estimated respondent burden hours for this study are 84 hours.

If there are any questions regarding this project, please contact Robin Kaplan at

202-691-7378.

Introduction and Purpose

The American Time Use Survey (ATUS) uses a time diary format to ask respondents about the activities they performed on the prior day. In 2014, the ATUS reported that members of the civilian, non-institutionalized population in the United States spend an average of 8.7 hours per day sleeping, and that number has remained similar over time. ATUS estimates of sleep are consistently higher than surveys that ask respondents directly about the amount of sleep they get (referred to in this protocol as stylized questions). The National Sleep Foundation (NSF), for example, asks respondents to estimate the amount of time they spend sleeping on an average weekday and weekend day. These numbers are then combined into an overall estimate. The NSF reported that American adults get an average of 7.1 hours of sleep per day in 2015. Other national surveys that use stylized questions about sleep, such as The National Health Interview Survey (NHIS), the Behavioral Risk Factor Surveillance System (BRFSS), and the National Health and Nutrition Examination Survey (NHANES) report sleep duration estimates ranging from 6.9 to 7.1 hours in 2014.

The ATUS differs from surveys that use stylized questions in methodology, sleep definition, and population. It is unclear how these differences influence the estimates. OSMR conducted a previous cognitive interview study to help understand the measurement properties of both the ATUS interview as it relates to sleep and of a sample of stylized questions about sleep. In that study, we found that diary and stylized measures of sleep duration differ from one another, with diary estimates exceeding stylized ones, consistent with prior research (e.g., Kan & Pudney, 2007).

The cognitive interview study results revealed a number of differences across participants in how they answered questions about sleep in general, and also across the diary versus stylized questions at each stage of the response process (Tourangeau, 1984). The main findings included differences in comprehension – participants with a broad definition of sleep (including activities such as lying in bed trying to fall asleep, resting, or dozing off) also reported getting more sleep than those with a narrower conception of sleep. Participants used a number of estimation strategies to answer the stylized questions about sleep versus the diary questions, where participants used memory retrieval strategies. We also found evidence that stylized questions may be more susceptible to social desirability concerns than the diary measure, that is, social desirability concerns about getting too much or too little sleep seemed to affect how participants answered the stylized questions about sleep, but this was not the case for the diary measure. Participants also indicated that if they thought the survey was about jobs and employment, it would more embarrassing to report oversleeping. In contrast, if a survey was about health, it would be more embarrassing to report undersleeping. These findings indicate that the context of the survey may also impact estimates of sleep duration. Since surveys across various agencies are framed differently, this may affect the sleep duration estimates obtained.

OSMR’s previous cognitive interview study had a very small sample, with only 29 participants who were all recruited from the Washington, DC metro area. Because of this, we conducted an online study to assess differences in sleep duration across diary and stylized estimates. We found that participants’ diary estimates exceeded their stylized sleep estimates, as expected. In addition, providing a definition of sleep seemed to bring participants’ diary and stylized sleep estimates slightly closer together, suggesting that including a precise definition of “sleep” might help reduce measurement error. We also found a large order effect where when stylized questions preceded diary questions, the diary sleep duration estimates were significantly lower, and the two measures matched more closely. We suspect that participants may have altered their response processes when the stylized sleep question came first, anchoring their diary answers to the stylized ones, consistent with other research (Schulz & Grunow, 2012; Schwarz et al., 1991). In addition, we found no effect of the survey context (telling participants the survey was about health, jobs, or general time use) on sleep duration estimates.

While we learned a lot from these previous two studies, our sample was recruited from Amazon.com’s Mechanical Turk, and therefore we could not control the demographics of our sample. The Mechanical Turk sample was not demographically representative of the U.S. population – for instance, they tended to be female and more highly educated. The goal of the present study is to assess the difference between diary and stylized measures of sleep duration using a sample size that more closely resembles the U.S. population. The online survey software company, Qualtrics, will recruit participants from their panel. Qualtrics will ensure participants match U.S. demographics in key areas (gender, education, age). Participants in this study will again be asked both ATUS time diary questions and stylized sleep questions via an online instrument.

2. Research Design

Qualtrics panel participants will be routed to an online survey, hosted on Qualtrics.com, where they will be asked to complete a modified version of the ATUS daily recall diary interview. They will enter each activity they did, along with their start and stop times, from 4am the prior day to 4am of the current day. Next, they will answer a set of stylized questions regarding how much they sleep on an average weekday and about other activities (e.g., time spent engaged in work-related activities, physical exercise, household chores, travel, and use of information-technology devices). These questions were adapted from other national surveys that ask for sleep estimates (e.g., the NHIS). Next, they will answer a set of stylized sleep questions aimed to assess the amount of time participants spend in bed versus actually sleeping. One source of measurement error in sleep duration might be because participants include the time they lie in bed awake or trying to fall asleep in their estimates, and this set of questions will tease that apart across weekdays and weekends. Finally, participants will answer a set of stylized questions adapted from other national surveys about the amount they typically sleep.

Participants will complete the surveys only on Tuesday – Friday so their ATUS recall of the prior day’s activities will always cover a weekday. Questions about where activities took place and who else was present (typical ATUS recall questions) will not be asked. After completing the diary and stylized time measures, participants will be asked questions aimed at understanding how they answered the questions, including questions about scheduling, social desirability, and demographics. See Appendix A for the full study protocol.

A number of stylized questions related to common activities in the ATUS will also be included to make the topic of sleep less salient and to fit with the framing of the survey. In addition to answering the diary and stylized time-use questions, we will also vary the framing of the study to be about sleep or general time use. This will allow us to determine whether the framing of the survey affects self-reports of sleep. This makes for a 2x2 study design with one between-subjects factor (Framing: sleep versus general time use) and two within-subjects factors (Diary vs. Stylized sleep questions).

The analyses will focus on possible reasons for a difference between the diary and stylized measures, if prior research showing a difference between these types of questions is replicated. We believe it is possible that framing the survey about sleep will cause people to report on their sleep in a manner consistent with their personal beliefs about whether it’s positive or negative to get a lot of sleep, or cause people to focus more on what is considered a “normative” amount of sleep (e.g., 8 hours) as they complete the survey. We will examine all main effects and interactions across the variables, along with participant demographics.

In addition, we will also include the following individual differences measures that might be related to how participants self-report about their sleep habits and their survey responses in general.

Balanced Inventory of Socially Desirable Responding (BIDR; Paulhus, 1984). As evidenced from the cognitive interview results from previous studies, social desirability concerns may affect how respondents report on their sleep. Participants who exhibit high versus low levels of social desirability concerns may be differentially affected by the framing and context of the survey in which they report on sleep. Thus, a potential covariate for sleep duration in this study is participants’ own tendency to respond in socially desirable ways. Participants will complete a short version of Paulhus’s (1984) Balanced Inventory of Socially Desirable Responding (BIDR) impression management subscale, which assesses deliberate self-presentation to an audience. It contains 10 items (e.g., I sometimes tell lies if I have to.) See the full protocol in Appendix for the entire scale.
Maximization Scale (Nenkov et al., 2008). This is a short scale used to determine whether people exhibit high versus low levels of a trait called maximization. Lower levels of maximization would tend to point to participants who satisfice. Satisficing is of great interest to survey researchers, as satisficers may tend to be less careful in completing surveys (e.g., straightline more on grids, speed through the survey instrument, or provide less thoughtful answers; Krosnick et al., 1996), which could lead to lower data quality. By taking a measure of participants’ tendency to maximize or satisfice, we can correlate this to other data quality indicators and paradata in the survey. See the full protocol in Appendix for the entire scale.
Mindfulness Attention Awareness Scale (Brown, 2003). The state MAAS is a 5-item scale designed to assess the short-term or current expression of a core characteristic of mindfulness, namely, a receptive state of mind in which attention, informed by a sensitive awareness of what is occurring in the present, simply observes what is taking place. Mindfulness has been associated with differences in how participants complete surveys and data quality (e.g., McCalin et al., 2015). By measuring participants’ level of mindfulness while completing the survey, we can correlate this to other data quality indicators and paradata in the survey. See the full protocol in Appendix for the entire scale.

Near the end of the survey, we will ask participants to report how burdensome they found it was to complete the entire survey and time diary portion, as well as related measures such as how interesting, effortful, and how sensitive they found the survey. As previous research has shown (e.g., Fricker et al., 2012), these perceptions of burden may have an effect on data quality and important survey outcomes. Since in our previous work, the time diary took a substantial amount of time to complete (6 minutes), these measures will help us to better understand the impact that perceived burden may have had on this survey as well as implications for perceptions of burden in the ATUS.

3. Participants

Participants will be recruited using a panel from Qualtrics, consisting of adult, U.S. citizens (18 years and older, using a quota sample of age, education, and gender breakdowns designed to be representative of the U.S. population); however, the focus of this study is focused on internal validity rather than representativeness of any population. This research design requires a sample of 250 participants in order to sufficiently explore the range of variables of interest and because we expect a very small size effect since, as the study manipulations are subtle for online surveys of this nature. This sample size also takes into account break-offs, incomplete data, and participants who do not follow the task instructions, similar to other samples used for studies of this nature.

An additional 3 participants will be recruited for an initial pilot test from TryMyUI.com. TryMyUI is an online testing website where respondents can complete a set of self-administered tasks while thinking “out loud” and respond to follow-up, scripted probes. TryMyUI provides a video recording of the output and each test can last up to 20 minutes. These pilot participants will be asked to think aloud while completing the Diary questions and answer questions about the experience afterwards, which will help to confirm whether the instructions on how to fill out the online time diary are effective, the task is clear, the questions are worded clearly, and the instrument works as intended. The pilot tests will be conducted iteratively so that any modifications can be tested with pilot participants before launching the full study.

4. Burden Hours

We will obtain responses from 250 participants recruited from Qualtrics. Each session is expected to take no more than 20 minutes to complete, for a total of 83.3 burden hours. In addition, the 3 pilot participants recruited via TryMyUI.com are expected to take no more than 20 minutes each, for an additional 1 burden hour. Total burden hours are expected to total no more than 84.3 hours. The survey will be administered completely online at the time and location of the participant’s choosing. There will be no recruitment burden associated with this research.

5. Payment to Respondents

Qualtrics will recruit 250 participants from their panel. Qualtrics requested $3,700 for 250 participants selected on 3 Census demographics (age, gender, education level). Per Qualtrics, the exact amount and form that each respondent receives can vary from respondent to respondent depending on their panelist profile, how they were recruited, and the form of incentives they have elected to receive (i.e. e-gift cards, points, cash, etc.). Generally speaking, though, respondents receive ~$1.00-$1.50 or a relative equivalent value for completing a 15-minute consumer survey. The remaining funds are paid to Qualtrics for licensing fees that are necessary for use of the panel. The pilot participants recruited from TryMyUI will receive the standard TryMyUI fee of $10 each for their participation. The payment for these additional 3 pilot participants will total $30.

6. Data Confidentiality

Recruiting of participants will be handled by Qualtrics. Participants will be informed that the study is about their perceptions of different types of questions. Once participants are recruited into the study, they will be sent a link to the survey, which is hosted by Qualtrics. The data collected as part of this study will be stored on Qualtrics servers. Participants in this study will not be asked to provide any personally identifiable information. Using the language shown below, participants will be informed of the voluntary nature of the study and they will not be given a pledge of confidentiality.

This voluntary study is being collected by the Bureau of Labor Statistics under OMB No. 1220-0141. We will use the information you provide for statistical purposes only. Your participation is voluntary, and you have the right to stop at any time. This survey is being administered by Qualtrics and resides on a server outside of the BLS Domain. The BLS cannot guarantee the protection of survey responses and advises against the inclusion of sensitive personal information in any response. By proceeding with this study, you give your consent to participate in this study.

Appendix A – ATUS Sleep Measures Online Study Protocol

Welcome!

Thanks for your interest in our research. We’re conducting this study to better understand how people spend their time as it relates to…

“…How much sleep people in America get. This survey includes questions about your sleep behavior, including times you typically go to bed and wake up and how much actual sleep you think you get.”

“…How people in America spend their time. This survey includes questions about your time use, used to find out how people balance work, child care, leisure, and other activities in their lives.”

Unlike some surveys or online tasks, we ask that you complete this task all at one time. Please begin only when you are in a quiet place where you won't be disturbed for about 20 minutes.

Please do not use your browser's back button.

Select ‘Next’ to continue

---page break---

Diary Introduction

On the next few pages, you will be prompted to think back to the last 24 hours, starting at 4 am yesterday morning and continuing through 4 am TODAY <INSERT DATE>. Please think back to each activity you did in as much detail as possible and what time you started and ended each activity. By activity, we mean anything you did during that time frame.

On the next screen, you will be asked to select the type of activity you did from a drop-down list and what time you started and stopped that activity. If you don’t see the activity you were doing, please select ‘Other Activity’.

Please be sure to account for your whole day, so the end time of one activity should be the start time of the next activity. There should not be any gaps.

Below is an example of a completed version of how the activity log should appear:

Drop-down menu/randomize order of choices, except ‘Other’ is always last]:

Sleeping
Grooming
Watching TV
Working
Eating / Drinking
Household chores
Shopping
Traveling /commuting
Leisure activity
Sports / exercise / recreation
Studying / learning
Socializing
Other activity

---page break---

[question order on this page will be randomized]

Note: For this set of questions, participants will use a drop down menu. The ‘hours’ drop-down menu will have the numbers 0-24 and the minutes drop down will have the numbers 0-55 in increments of 5.

Thank you for completing the activity log. Next, we’d like to know more about how you spend your time in general. Please think of a typical day while answering these questions, not one where you traveled, vacationed, or had family crises.

On average, how many hours per day do you participate in the following activities…?

1.) Working at your main job? ____ hours ____minutes

2.) Exercising (for example, sports, or physical activities) in your free time? ____ hours ____minutes

3.) Household chores and other cleaning activities? ____ hours ____minutes

4.) Watching television? ____ hours ____minutes

5.) Using technological devices connected to the Internet (for example, a personal computer, cell phone, or tablet, excluding television), in your free time? ____ hours ____minutes

6.) Traveling from place to place, such as commuting to and from work? ____ hours ____minutes

7.) Leisure activities, such as reading or other hobbies? ____ hours ____minutes

---page break---

The following questions are about your typical sleep routine on a weekday or workday, in which you work the next day. Please think of a typical weekday while answering these questions, not one where you traveled, vacationed, or had family crises.

Note: For this set of questions, participants will use a drop down menu. The hours drop-down menu will have the numbers 1-12 and the minutes drop down will have the numbers 0-55 in increments of 5.

1.) On a typical weekday (or workday) at what time do you get into bed to fall asleep?