Memo
To: Rachel Krantz-Kent, OEUS
From: Jennifer Edgar, OSMR
DATE: August 1, 2011
Subject: Results from Cognitive Testing of 2012 Changes to ATUS Well-Being Module
Executive Summary
Overview
This memo summarizes findings from cognitive testing of changes proposed to the American Time Use Survey’s (ATUS) Well-Being Module, including the Cantril Ladder question and a ‘typical day’ question. The purpose of this testing was to address the transition statement’s ability to focus participants beyond yesterday to life in general and move them to a 0 to 10 point scale. The testing also looked at the ability of respondents to compare yesterday to a ‘typical day’ and the sensitivity of both proposed questions. Comprehension and response process were not evaluated for the Cantril ladder question, or the existing affect questions, as the wording of those questions was established and not subject to revision.
Method
OSMR recruited 9 participants selected to obtain representation across age, gender and education. The ATUS time diary and well-being module, including the proposed changes, were administered over the telephone, followed by a face-to-face debriefing.
Results and Recommendations
Overall the transition statement and two questions worked as intended. Participants did not report finding them overly sensitive, and were able to answer the questions with high validity. The questions should be implemented as tested.
Overview
The purpose of this study was to test three proposed changes to the ATUS Well-Being Module. The new questions were provided by external researchers (Deaton, Krueger and Schkade) who the module sponsor –the National Institute on Aging—had designated as consultants on the module. These questions were provided in response to BLS indicating that the 2010 Well-being module did not take the full time available. The questions proposed are intended to add information useful in research on well being. OSMR staff proposed the wording for the transition statement, which will be placed after the current affect questions and before the Cantril ladder question. Additionally, three OSMR staff members conducted an expert review of the three proposed changes, leading to the tested wording:
Transition from affect questions: Thank you. We’re going to switch focus now. The next two questions ask you to think about your life in general, not just in terms of the specific activities we just talked about. And, rather than using a 6-point scale to represent how you feel, we’ll be using a 10-point scale.
Cantril ladder question: To begin, please imagine a ladder with steps numbered from zero at the bottom to ten at the top. The top of the ladder represents the best possible life for you and the bottom of the ladder represents the worst possible life for you. If the top step is 10 and the bottom step is 0, on which step of the ladder do you feel you personally stand at the present time?
Typical day question: Thinking about yesterday as a whole, how would you say that your feelings, both good and bad, compared to a typical [fill day of the week]? Were they better than a typical [fill day of the week], the same as a typical [fill day of the week], or worse than a typical [fill day of the week]?
The objective of the study was to evaluate:
Cantril Ladder Question
Are respondents able to use the 0 to10 point response scale after using a 0 to 6 point scale for the preceding affect questions?
How sensitive do respondents find the question?
Typical Day question
Are respondents able to compare yesterday to a ‘typical’ day?
Do respondents think of the whole day yesterday when answering the typical day question, just the three activities asked about in the affect questions, or some other portion of the day?
How sensitive do respondents find the question?
It is important to note that comprehension and response process were not evaluated for the Cantril ladder question, or the existing affect questions, as the wording of those questions was established and not subject to revision.
Method
Participants
OSMR recruited 9 participants from a database of general population research participants. Time constraints limited the study to a sample size of 9. Participants were selected to obtain representation across age, gender and education. There were four females and five males ranging in age from 26 to 62, with an average age of 40. Participant education level ranged from a high school degree to a masters degree.
Cognitive Interviews
For each interview, the ATUS time diary and well-being module, including the proposed changes1, were administered over the telephone, and then a face-to-face debriefing was conducted with each participant. Cognitive interviewers were from ATUS and OSMR staff. For each interview an ATUS program office staff member took notes, which form the basis of these results.
Interviews were conducted between June 14th and 22nd. For each interview, the cognitive interviewer greeted the participants, explained the purpose and method of the study, and obtained their signature to a consent form. To simulate actual ATUS production interviewing, the ATUS diary and well-being questions were administered over the telephone, with the cognitive interviewer in a different room. After the interview was completed, the interviewer returned to the interview room with the participant and conducted the debriefing face-to-face.
Analysis
Notes from each cognitive interview were reviewed to reach conclusions about each of the study objectives. It is important to note that with such a small sample size, conclusions are somewhat limited. Even though participants were selected to represent a range of respondent types, we cannot be sure that any problems identified during this small-scale test are the only problems that respondents will experience in production interviews. Also, the cognitive interview did not include the full ATUS interview (e.g., earnings, employment and other questions were not included). Context and order are important influences on participants’ responses, and it’s possible that they could affect participants’ responses.
However, the combination of expert review and cognitive testing, in combination with the relative simplicity of the proposed changes, leads to confidence in the final recommendations.
Results
Transition from affect questions:
“Thank you. We’re going to switch focus now. The next two questions ask you to think about your life in general, not just in terms of the specific activities we just talked about. And, rather than using a 6-point scale to represent how you feel, we’ll be using a 10-point scale. To begin,”
Purpose
The purpose of this transition statement was to move the participants’ focus away from the three activities asked about in the affect question, away from yesterday, and move it to their life in general. It was also designed to make the participant aware that the response scale was changing from 0 to 6 to 0 to 10.
Results
Yesterday focus. When asked what they were thinking about when answering this question, most participants indicated a broad, and often very general, reference period. One participant even indicated the question was difficult to answer because it included so many elements, such as work, family, relationships, and activities. Only one participant indicated they focused on “mostly yesterday” when answering this question. With additional probing, however , that participant was inconsistent; they said the question was asking how they feel about their life at the present rather than just yesterday. This inconsistency during the debriefing may indicate the participant was not able to recreate their thought process during the original survey question.
Scale differentiation. One participant used the 0 to 6 scale for the ladder question, giving a 6 during the interview, but then indicating during the debriefing that he felt he was near the top of the ladder and changing his answer to an 8. Although this one indicated that he had confused the scales, the rest of the participants’ transitioned to the new scale without any apparent issues.
Conclusion
Overall, the transition appears to have met the objectives, focusing participants away from yesterday and the specific activities, and informing them of the 0 to 10 point scale. Although only one participant used the wrong scale, with such a small sample size, we cannot conclude that it will not happen in production.
Recommendations
Implement as tested. Monitor ATUS interviews and the data to ensure that interviewers are using the transition statement and that respondents are using scale points 7, 8, 9 and 10, since a lack of use of these higher scale values may indicate that they are using the 0 to 6 point scale.
Cantril Ladder Question
“Please imagine a ladder with steps numbered from zero at the bottom to ten at the top. The top of the ladder represents the best possible life for you and the bottom of the ladder represents the worst possible life for you. If the top step is 10 and the bottom step is 0, on which step of the ladder do you feel you personally stand at the present time?”
Purpose
The purpose of this question was to add data the researchers deemed valuable.
Results
Question comprehension. Although not a goal of this study, due to the extensive existing research on the question, participant comprehension was evaluated as a part of the debriefing. All but one participant seemed to understand and be able to provide a response to this question. Participants commonly used terms like goals, ambitions, satisfaction and life standards. These all seemed consistent with the intended meaning of the question.
Participants were successfully able to interpret the top and bottom of the ladder, describing both appropriately. When asked what a ‘5’ indicated, responses included references to the middle, working towards goals, and struggling to reach the top.
Scale differentiation. All but one participant successfully moved from the 0 to 6 point scale used in the well-being questions to the 0 to 10 point scale in the ladder question. See above results for additional information.
Yesterday focus. Only one participant referred to yesterday when answering this question, but inconsistencies in his debriefing responses suggest a difficulty with the debriefing rather than the question. See above results for additional information.
Sensitivity. Although no participants reported that they had concerns about answering this question because of its sensitivity, several said that other people may. However, when probed on this, very few had specific reasons why others might find it sensitive, instead providing vague responses like “some people don’t like to give personal info.” The majority of reasons that this question may be found sensitive by other people were not specific to the Cantril ladder question, instead, they applied to the ATUS survey as a whole. There was only a single comment, “People won’t want you knowing how they’re doing in their life” specific to the ladder question.
Conclusions
Participants appeared to understand this question, and used appropriate reference periods (e.g., not just yesterday) and the 0 to 10 scale. The question was not seen as sensitive for any participants, though some participants thought it to be potentially sensitive for other people.
Recommendation
Implement as tested. See above reference to monitoring the interviews and data.
Typical Day Question
“Thinking about yesterday as a whole, how would you say that your feelings, both good and bad, compared to a typical [fill day of the week]? Were they better than a typical [fill day of the week], the same as a typical [fill day of the week], or worse than a typical [fill day of the week]?”
Purpose
The purpose of this question was to add explanatory data to the affect questions. The researchers were interested in identifying how typical the participant’s feelings were for that specific day. During the question development process, there was some concern that participants would not be able to easily identify typical feelings for a specific day, that identifying typical weekday or weekend feelings may be an easier task. The researchers indicated a preference to specify a single day in the question, and so that is the version that was tested.
Results
Question comprehension. Overall, participants seemed to understand this question as intended, with most participants correctly explaining that the question was asking them to compare yesterday to the same day of the week. There were some participants who focused more on the activities of the day than the feelings. However, this did not appear to impact their ability to respond to the question. They still provided an answer using the response options, and when debriefed seemed to easily move between discussing feelings and activities, suggesting a strong (and perhaps necessary) relationship between the two.
For the participants who did refer to feelings, the feelings mentioned included stress, happy, dread (in reference to the upcoming work week), hopeful, upset and bored.
It is interesting to note that only one participant used the word “typical” in his explanation of the question. Others referenced ‘other’ days and past days. This could be because we asked them to state the question in their own words, so they were deliberately trying to avoid the word typical since it was included in the question, or possibly because the concept of a typical Tuesday is not meaningful. Research with additional participants could provide further insight.
Sensitivity. As with the Cantril ladder question, no participants indicated that this question was too sensitive for them to answer. A few again indicated that other people may find it sensitive, but provided similar explanations, such that some people don’t like to share information about themselves or their lives. When compared to the Cantril Ladder question, fewer respondents indicated that the question was sensitive, either for themselves or for other people.
Reference for comparison. This question asked participants to compare their feelings yesterday to a typical day (e.g., typical Monday, Tuesday). Participants were asked several questions to identify what they compared yesterday to, and overall participants correctly referenced the correct days. They were also able to identify what a typical [fill]day meant to them, though most participants discussed what they did on a typical [fill]day, rather than how they felt.
There were two interviews conducted on a Tuesday, and both participants referenced going back to work after a weekend and the feelings associated with that. There was more variation in the feelings (and activities) reported by participants for the other days of the week.
Conclusions
The intended meaning of the question seemed to come across to most participants. Participants did seem able to identify a typical Monday, Tuesday, etc., though testing participants across all seven days would provide stronger confirmation of this. The question was not seen as overly sensitive.
Recommendation
Implement as tested.
Appendix A: Cognitive Testing Protocol
ATUS Well-being Module
Cognitive Interview Protocol
June 14, 2011
Protocol
Introduction
Explanation of study
Ask questions about what you did yesterday and then some questions about how you felt
need to know how people like you answer the questions, what you think when you hear certain terms and phrases, and how easy or difficult the question is
go through questions and get your answers, then go back and ask follow-up up questions to gauge your reactions and thoughts about those questions
first set of questions asked over the telephone, you’ll answer this phone and we’ll do the interview using it. this is to mirror the real survey, which is asked over the phone
Here to evaluate the questions, not you; no right or wrong answers, please respond honestly
Explanation of lab
Explain responses will be confidential
Get permission to tape
Introduce note taker
Get signature on consent form
Get signature voucher & give check
Show participant how to answer the phone
hit the speaker phone button,
you’ll then be on speaker phone and we’ll begin the interview.
Once the interview is finished, interviewer will come back to talk about interview questions
Phone #: 15037
ATUS Diary
1. Introduction
1. We are conducting a survey for the Bureau of Labor Statistics on how people in America spend their time. This information will help policy makers understand how people divide their time between work, family and other demands in our fast-paced society.
2. Household Roster
What are the names (just the first) of all the persons living in your household? Please start with your name. I have you as [FILL NAME].
What is the name of the next person?
What is [FNAME]’s sex?
How is [FNAME]’s related to you?
20. Spouse (husband/wife)
21. Unmarried partner
22. Child/stepchild
23. Grandchild
24. Parent (mother or father)
25. Brother/sister
26. Other related person (aunt, cousin, nephew, mother-in-law, etc.)
27. Foster child
28. Housemate/roommate
29. Roomer/boarder
30. Other nonrelative
Was [FNAME] older than 18 years old on [your / his or her] last birthday?
Is anyone else living here now?
S3: Diary (Interviewer, complete grid)
Now I'd like to find out how you spent your time yesterday, [yesterday's day & date], from 4 am yesterday morning to 4 am this morning. I'll need to know where you were and who else was with you. If an activity is too personal, there's no need to mention it.
So let's begin. Yesterday, [previous DAY] at 4:00 AM, what were you doing?
If simultaneous activities are listed: “What was the main activity?”
What time did you start [ACTIVITY]?; what time did you end ____[ACTIVITY]?
"Who was with you?"/ "Who accompanied you?" do not ask for sleeping, grooming, ‘personal’ activities and DK/R.
Where were you while you were [ACTIVITY]?
Interviewer Notes
Activities lasting longer than 2 hours (excludes sleeping, working, or attending high school), PROBE: Did you stop [ACTIVITY] to do anything else during that time? If yes, record the activity.
b. Work activities with a duration time greater than 4 hours are probed about, to determine if the DP took any breaks of 15 minutes or longer. PROBE: "Did you take any breaks of 15 minutes or longer?" for work activities that exceed the duration time of 4 hours.
(1) If yes, ask the probe "What did you do?" and record the activity.
If no, proceed on with the interview.
c. Probe for simultaneous activity: What would you say is your MAIN Activity?
If DP can estimate the time for each activity: Record each activity on its own line in the diary
If DP cannot estimate the time for each activity but reports a MAIN
activity: Record the main activity first followed by a / and then the second activity
Now I’m going to ask you a set of questions about how you felt yesterday during some particular times.
Please use a scale from 0 to 6, where a 0 means you did not experience the feeling at all, and a 6 means the feeling was very strong. You may choose any number 0,1,2,3,4,5 or 6 to reflect how strongly you experienced this feeling during this time.
Activity 1
Between X and X yesterday, you said you were Z. The next set of questions asks how you felt during this particular time. Again, 0 means you did not experience this feeling at all and 6 means the feeling was very strong.
Happy First, from 0 – 6, where a 0 means you were not happy at all and a 6 means you were very happy, how happy did you feel during this time?
Tired From 0 – 6, where a 0 means you were not tired at all and a 6 means you were very tired, how tired did you feel during this time?
Stressed From 0 – 6, where a 0 means you were not stressed at all and a 6 means you were very stressed, how stressed did you feel during this time?
Sad From 0 – 6, where a 0 means you were not sad at all and a 6 means you were very sad, how sad did you feel during this time?
Pain From 0 – 6, where a 0 means you did not feel any pain at all and a 6 means you were in severe pain, how much pain did you feel during this time if any?
Meaningful From 0 to 6, how meaningful did you consider what you were doing? 0 means it was not meaningful at all to you and a 6 means it was very meaningful to you.
7. Were you interacting with anyone during this time, including over the phone?
Activity 2
Between X and X yesterday, you said you were Z. The next set of questions asks how you felt during this particular time.
Stressed First, from 0 – 6, how stressed did you feel during this time?
Tired From 0 – 6, how tired did you feel during this time?
Happy From 0 – 6, how happy did you feel during this time?
Pain From 0 – 6, how much pain did you feel during this time, if any?
Sad From 0 – 6, how sad did you feel during this time?
Meaningful From 0 – 6, how meaningful to you was this time?
Were you interacting with anyone during this time, including over the phone?
Activity 3
Between X and X yesterday, you said you were Z. The next set of questions asks how you felt during this particular time.
Pain First, from 0 – 6, how much pain did you feel during this time, if any?
Sad From 0 – 6, how sad did you feel during this time?
Happy From 0 – 6, how happy did you feel during this time?
Tired From 0 – 6, how tired did you feel during this time?
Stressed From 0 – 6, how stressed did you feel during this time?
Meaningful From 0 – 6, how meaningful to you was this time?
Were you interacting with anyone during this time, including over the phone?
New Transition
Thank you. We’re going to switch focus now. The next two questions ask you to think about your life in general, not just in terms of the specific activities we just talked about. And, rather than using a 6-point scale to represent how you feel, we’ll be using a 10-point scale. To begin,
New Questions
Please imagine a ladder with steps numbered from zero at the bottom to ten at the top. The top of the ladder represents the best possible life for you and the bottom of the ladder represents the worst possible life for you.
If the top step is 10 and the bottom step is 0, on which step of the ladder do you feel you personally stand at the present time?
Thinking about yesterday as a whole, how would you say that your feelings, both good and bad, compared to a typical [fill day of the week]? Were they better than a typical [fill day of the week], the same as a typical [fill day of the week], or worse than a typical [fill day of the week]?
Thank you, I’ll be right in to discuss
Debriefing
Now I’d like to talk about the questions you just answered. Please think back to what you were thinking when you answered the questions, and tell me as much information as you can. I’m looking to hear about your thought processes when you answered the questions, as well as your overall reactions. This information will help us improve the questions.
Ladder Question
Another question asked you to think about a ladder. It said: “Please imagine a ladder with steps numbered from zero at the bottom to ten at the top. The top of the ladder represents the best possible life for you and the bottom of the ladder represents the worst possible life for you.
If the top step is 10 and the bottom step is 0, on which step of the ladder do you feel you personally stand at the present time?
Can you tell me in your own words what that question is asking?
For this question, what does a 10 mean to you?
What does a 0 mean?
What about a 5?
Did you have any problems answering this question?
What were you thinking about when you answered the question?
If necessary probe: Were you thinking about just yesterday? Just the activities we just talked about? Your life in general?
What is your overall reaction to this question?
Did you have any concerns about answering it because it was personal?
Do you think other people might find it too personal?
Typical Day question
Another question asked, “Thinking about yesterday as a whole, how would you say that your feelings, both good and bad, compared to a typical [fill day of the week]? Were they better than a typical [fill day of week], the same as a typical [fill day of week], or worse than a typical [fill day of week]?"
In your own words, what does this question mean?
How did you decide on your answer?
If necessary probe: Were you thinking about just yesterday? Just the activities we just talked about? Your life in general?
If necessary probe: What were you comparing yesterday to?
What feelings did you think of when you answered this question?
The question asked about ‘a typical [fill day of week],’ what does that mean to you?
How do you feel on a typical [fill day of week]?
Did you have any concerns about answering it because it was personal?
Do you think other people might find it too personal?
What is your overall reaction to this question about your typical day?
Do you have any other comments or feedback about these questions?
1 After 3 interviews, the response options of the typical day question were changed from “typical day” to “typical [fill day of the week]?” This was done based on interviewer reaction to the question, and to increase focus on the specific reference day.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Title | Proposed Questionnaire |
| Author | allard_d |
| File Modified | 0000-00-00 |
| File Created | 2021-01-14 |