Attachment L - Cognitive Testing Results 2010 Core Module

January 9, 2009

MEMORANDUM FOR Dori Allard

Office of Employment and Unemployment Statistics (OEUS)

From: Kathy Downey

Research Psychologist

Office of Survey Methods Research (OSMR)

Subject: Findings from the SECOND round of cognitive interviews for the American Time Use Survey’s proposed Well-Being Module

executive summary

Overview

This report presents findings from the second round of cognitive testing of the American Time Use Survey’s (ATUS) proposed Well-Being Module. The Office of Employment and Unemployment Statistics (OEUS) asked the Office of Survey Methods Research (OSMR) to test the module items developed by Alan Krueger and his colleagues. The purpose of these items was to determine the extent to which respondents felt, happy, stressed, sad, and in pain during sampled episodes from the time-use diary. Also, respondents were asked how meaningful the activities were. Additional questions were also added and tested about health status.

Methods and Analysis

OSMR recruited 28 participants from a database of general population research participants and the interviews were conducted by a team of cognitive interviewers. The respondents represented a cross-section of the general population based on sex, race, age, education, and employment status.

This round of testing used two different modules and protocols. Midway through testing (after Interview #17), the team decided to make minor changes to the module based on the results of the interviews. The decisions were based on an examination of interviewer notes and verbal discussions.

The final interviews (Interviews 18 through 28) tested a final copy of the module. These changes were made based on the length of the module, which is contracted to take an average of five minutes. The changes included deleting one of the health status questions, only asking about three activities (instead of five), adding the word “pain” to “prescription medication” in one of the items, and adding an explicit probe to determine whom the respondent was with at work.

After the final interviews, the interviewers examined their interviewer notes and held another debriefing for the whole team. Consensus was reached as to the problems seen during testing.

Findings and Recommendations

The version of the module used for the final portion of testing, Interviews 18 through 28, worked extremely well. As with Round 1, there was evidence of a lack of differentiation in participants’ responses for some of the affective dimension items (for example, ratings of pain and sadness).

Another issue that was found in both Rounds 1 and 2 of testing was respondents’ slight confusion with the “interaction” item: “Were you interacting with anyone during this time, including over the phone?” Respondents had varying definitions of “interacting.” This was noted in Round 1 and it was decided then that this item would be left up to respondent interpretation.

Also, a few respondents flipped the scale used for the affective dimension items; of these, all but two corrected themselves during the interviews and prior to the debriefing questions. Based on the findings, there are only two minor changes suggested for the module after Round 2 of testing.

• Activities deemed too personal should not be included in the module.

• The question that asks how meaningful an activity was should always appear last in the list of affective dimensions; the order of the other affective dimension items can be randomized. However, it should be noted that randomizing the order of the affective items was not tested in either round, so this recommendation reflects the opinion of the interviewing team.

Overview

The purpose of this research study was to examine possible problems with a new module for the American Time Use Survey (ATUS). BLS plans to add a “well-being module” to the ATUS in 2009. Gallup previously performed cognitive testing of about half of the items in the module. Therefore, the items not previously tested were to be examined through cognitive interviewing and an expert review. The Office of Employment and Unemployment Statistics (OEUS) asked the Office of Survey Methods Research (OSMR) to test the module items.

The “well-being module” was designed to collect data on measures of affective experience during randomly selected episodes of the day. (See work by Kahneman and Krueger, 2006, for a complete discussion of the measurement of subjective well-being.) The affect measures tested relate to the extent to which respondents felt tired, in pain, happy, stressed, sad or meaningful during the sampled episodes. Individuals are also asked if they were interacting with someone during sampled episodes. For sampled time periods spent at work, respondents are also asked whom they were with. Lastly, some health status information is collected. All of these survey items comprise the well-being module. This module would come at the end of the ATUS interview, which now finishes with items on schooling and earnings.

There were two rounds of testing for the module. The first round, reported elsewhere¹, consisted of an expert review of the module along with cognitive interviewing of 10 respondents. The second round, reported here, consisted of cognitive interviewing on revised drafts of the module.

Methods

Participants in the Cognitive Interviews

OSMR recruited 28 participants from a database of general population research participants. This database is compiled and maintained by OSMR. The participants were paid $40 for their time and travel to BLS.

During recruitment, OSMR made sure that approximately three-fourths of the participants were employed at the time of the interview (in order to have “work” appear as an activity during the interviews). OSMR also collected several other demographic characteristics, listed in Table 1, to try to include a cross section of the general population.

Table 1: Demographic Characteristics of Interview Participants

	Round 2 of Cognitive Testing		2007 ATUS
Demographic Characteristic	Number of Respondents	%	Number of Respondents (unweighted)	%
Sex
Male	8	28.6%	5,298	43.3%
Female	20	71.4%	6,950	56.7%
Race
White	11	39.3%	9,989	81.6%
Black	14	50.0%	1,592	13.0%
Other	3	10.7%	667	5.4%
Education
High school degree or less	7	25.0%	5,380	43.9%
Some college or bachelor’s degree	21	75.0%	5,593	45.7%
Age
20-29	8	28.6%	1,424	11.6%
30-39	9	32.1%	2,422	19.8%
40-49	1	3.6%	2,628	21.5%
50 +	10	35.7%	4,909	40.1%
Employment
Employed	20	71.4%	7,898	64.5%
Unemployed	8	28.6%	4,350	35.5%

The participants in the cognitive interviews were more likely to be female, Black, young, and highly-educated than the respondents in the 2007 ATUS. On one hand, this may mean that participants in the cognitive interviews (who were more likely female and highly educated/verbal) were more likely to be comfortable with affect questions. This, in turn, could lead them to over-analyze or over-report problems where none existed. On the other hand, as Willis (2005, p. 171) notes, “If tested subjects are more highly educated than the population to be surveyed, even relatively modest levels of documented comprehension problems might motivate the designers to simplify the questionnaire.”

Qualitative research differs from quantitative research on some fundamental paradigms regarding the necessity to exactly match potential survey respondents. As stated by Willis (2005, p. 225), “Cognitive laboratory subjects are not survey respondents. Volunteers for cognitive interviews are, by definition, self-selected, and are therefore not likely to be representative of the survey population as a whole.” He goes on to say:

The purpose of cognitive interviewing is not statistical estimation. The point is not to obtain sample sizes large enough to supply precision in statistical estimates. Rather we strive to interview a variety of individuals who will be useful in informing our decisions. Statisticians strive to minimize (error) variance, whereas cognitive interviewers maximize (subject) variance. (p. 227)

Therefore, while cognitive interviewing strives to gather a healthy mix of different types of respondents, it is not critical that they exactly match the potential survey respondents on demographics.

Protocol Questions for the Cognitive Interviews

The cognitive testing protocol was developed by OEUS and OSMR jointly. Based on findings from Round 1 of testing (reported elsewhere), changes were made to the module. These changes were carried over into a new version of the testing protocol and follow-up questions.

This round of testing used two different modules and protocols. Midway through testing (after Interview #17), the team decided to make minor changes to the module based on the results of the interviews. Therefore, Interviews 18 through 28 (termed “final interviews” for this report) tested a version of the module that hopefully would not require major changes.² The changes introduced were based on a meeting of the entire research team where we went over the interviewer notes to that point and reached a consensus on the changes. Although the changes made to the interview midway through testing will be discussed in greater detail later, they were as follows:

• Addition of an explicit probe to identify whom the respondent was with while at work;

• Deletion of two activities (three activities were sampled rather than five);

• Addition of the word “pain” to “prescription medication,” and

• Deletion of the high blood pressure question.

The first part of the cognitive interview gathered the respondent’s 24-hour recall of activities, the second part asked the well-being module, and the third part was debriefing questions. The module used for testing, shown with changes made midway through testing, is listed in Appendix A. The protocol used for the final interviews is Appendix B.

Implementation of the Protocol for the Cognitive Interviews

The interviewers were two research psychologists from OSMR and three staff members from OEUS. A note taker was also present for most of the interviews. All but one of the interviews were recorded.

Only one interviewer at a time conducted the interview. For each interview, the interviewer greeted the participants (in person) and gave them introductory information. To partially simulate the conditions that would exist in an actual ATUS phone interview, the interviewer then left the interview room and conducted the diary portion and the well-being module over the telephone (on speaker phone) with the participant. For the debriefing questions, the interviewer went back into the interview room with the participant and conducted the rest of the interview face-to-face.

The interviews lasted an average of 45 minutes. The average time spent during the interview on the affective dimension items was seven minutes for the first interviews and five minutes for the later interviews, as shown in Table 2. The contract for administering the module specifies that the module should take a maximum of five minutes. During the first portion of testing, the module was taking an average of seven minutes to administer. Therefore, the team decided to decrease the number of activities from five to three and delete one of the health status items, the one on high blood pressure. In the second portion of testing, the module took an average of about five minutes to administer, which met the goal.

Table 2: Average Administration Times for the Module, Before and After Changes to the Module

	First portion of Round 2 (in seconds)	Final interviews of Round 2 (in seconds)
Activity 1	176.08	150.00
Activity 2	66.15	67.73
Activity 3	49.31	60.45
Activity 4	47.69	--
Activity 5	46.23	--
Pain and Health Status Items	46.23	39.18
Total (in seconds)	431.69	317.36
Total (in minutes)	7.19	5.29

Analysis and Disclaimers/Limitations

Analysis of qualitative data. The cognitive interview approach seeks to analyze respondents’ understanding of survey questions and to identify possible sources of measurement error that might occur in the following hypothesized stages of cognitive processing (Tourangeau & Rasinski, 1988):

1. Comprehension

2. Recall/retrieval

3. Decision making

4. Response selection

As noted by Willis,³ cognitive interviewing explicitly focuses on the cognitive processes that respondents use to answer survey questions. Therefore, the approach examines both covert and overt processes. For cognitive testing, researchers do not analyze the actual survey responses per se, but as Willis (2005, p. 169) states, “…what is most important is a measure of reactions to the questions…” The investigative process results in a qualitative summary of the interviews and identifies important measurement issues. Willis (2005, p. 227) also states that:

…we do not evaluate survey questions simply by counting the number of interviews in which a problem occurs. Of course, if every interviewer reports a problem with a particular question in every interview, that is significant. However, a finding can be based on one interview; an interviewer may say that ‘I had a person with a particular disease for which this question does not work,’ and we need not verify this by testing a large number of other individuals with the same disease. Logical and structural problems in particular are sample-size independent, in the sense that sources of question illogic persist whether we test one subject or a thousand.

Specifically, issues with survey items were indicated when respondents seemed to misinterpret the question based on their verbal reports. The “data” were participants’ responses to probes as well as observers or interviewers’ notes and interpretations of the participants’ responses. Themes were developed according to respondents’ reactions to survey items and across survey items, such as when respondents sometimes flipped the endpoints of a scale and later corrected themselves after hearing the scale anchors multiple times.

For this project, there were two debriefings. The first was after Interview #17, when all of the cognitive interviewers read over their interview notes and then the research team reached consensus on what changes should be made to the module. The second debriefing came after the final interviews, when the interviewers again looked over their interview notes and the interviewers (not the whole research team) reached consensus on what issues were encountered during the final interviews.

Sample size. Cognitive testing uses relatively small sample sizes and produces qualitative data. Therefore, the approach should not be used to compute statistical estimates of the prevalence of identified problems. Because cognitive interview studies usually have small, non-probability samples, care must be taken in generalizing from cognitive testing results to the target population. As stated previously, the goal of cognitive testing is not to produce statistical estimates, but to interview a variety of people and to identify potential cognitive processing problems.

Limitations of context. During the cognitive interviews, we did not administer the whole ATUS interview to the respondents. The actual ATUS interview⁴ contains the following sections:

   Introduction

   Household roster

   Employment

   Time-use diary

   Summary questions (work, secondary childcare, volunteering)

   Trips

   Labor force status

   Earnings and schooling

For the cognitive interviews, we used the following:

   Introduction

   Household roster

   Time-use diary (with a “Who was with you?” probe for work activities)

   Summary questions (only secondary childcare)

• Well-being module

During production interviewing, the well-being module will follow the earnings and schooling section; however, due to skip patterns in the questionnaire, not all survey respondents will receive these questions. Therefore, during the cognitive testing the choice was made to have the well-being module follow the summary questions for secondary childcare, since all survey respondents would be asked these questions. Additional sections from the ATUS questionnaire were not included so that the cognitive interviews could be administered in less than one hour.

It is important to note that the cognitive interview protocol does not reflect the full complexity, order, and length of the actual ATUS interview.  Context and order are important influences on participants’ responses, and it’s possible that they could affect participants’ responses. 

Question-by-Question Review

Addition of a “Who was with you?” Probe for Sampled Work Activities

Purpose of the new question. During the collection of activities in the current ATUS interview (i.e., the “diary portion”), a question asks who was with the respondent after certain activities are mentioned. However, this question is not asked for personal activities, sleep, and work activities. We wanted to test asking “Who was with you?” during the diary portion for work activities. The research purpose of this data is to examine the impact of the presence of bosses or co-workers on respondents’ affect. The “Who was with you?” question will only be added when the whole well-being module is added.

An alternative version of the “Who was with you?” question was originally proposed for testing, but not tested. This version consisted of a two-part question. The first question was “During [FILL] time yesterday, were you alone for at least half of the time?” If the answer was no, the following probe would be asked: “During this time yesterday, were you with your boss or manager, co-workers, customers, people whom you supervise, or others?” These questions were designed to be asked during the well-being module and not during the diary portion. However, these alternative questions were not tested because the “Who was with you?” probe during the diary portion for work activities worked well. Plus, there are two advantages to having the “Who was with you?” question in the diary and not in the well-being module. First, the question flows better since other activities are followed up with the “Who was with you?” question. Second, it would eliminate the need for an additional question in the well-being module.

Item as tested for first portion of interviews. If a work activity was mentioned by the respondent during the diary portion, the telephone interviewer would immediately ask:

“Who was with you?” or “Who accompanied you?” [mark all that apply; list is not read]

1. Boss or manager

2. Co-workers

3. Customers

4. People whom you supervise

Item as tested for final interviews. If a work activity was mentioned by the respondent during the diary portion, the telephone interviewer would immediately ask:

“Who was with you?” or “Who accompanied you?” [mark all that apply; list is not read]

1. Boss or manager

2. Co-workers

3. Customers

4. People whom you supervise

If the respondent listed “co-workers” as part of his/her recall, the following probe was asked immediately: “By co-workers, did you mean you were with your manager/supervisor, people whom you supervise, or other co-workers?"

Results.

1. The respondents were asked whether it was easy or difficult to remember whether they were with someone at work. All of the respondents said it was easy to remember.

2. Half of the respondents had worked the previous day. Of those, a few had worked at home the previous day and said no one was with them. Of those that had worked in an office, a few people said that they were around people all day long and two people said that there was no one else in the same room with them all day long.

   1. However, respondents were imprecise in answering this question, especially if they worked in an office cubicle. For example, one person said she had “worked with coworkers all day.” Another person said that she had considered “all of them, including my boss” as co-workers. It quickly became apparent that when respondents said “co-workers,” they needed to be probed what they meant exactly. A probe was added for the final interviews.

   2. Even after the addition of the probe (“By co-workers, did you mean …?”), respondents were a little unclear whom they were with. One respondent said “didn’t know what you meant…the same room? What does ‘with people’ mean?”

Recommendations. Implement the version used in the final interviews. If a work activity was mentioned by the respondent during the diary portion, the telephone interviewer would immediately ask:

“Who was with you?” or “Who accompanied you?” [mark all that apply; list is not read]

1. Boss or manager

2. Co-workers

3. Customers

4. People whom you supervise

If the respondent listed “co-workers” as part of his/her recall, the following probe was asked immediately: “By co-workers, did you mean you were with your manager/supervisor, people whom you supervise, or other co-workers?"

Questions 1 through 6: Affective Dimension Items

Purpose of the new questions. These were developed by the client in order to learn more about the participants’ affect during specific activities. The preface to the module and the stressed and pain items had not been previously tested. Other items in this section had been cognitively tested by Gallup.

As stated previously, the affective dimension items, interaction item, pain medication item, and health status items comprise the well-being module. The module is delivered at the very end of the ATUS interview.

The order of the emotion questions will be randomized in implementation, but this wasn’t done for the cognitive testing. It is also important to note that the sponsor’s research uses a 0 to 6 rating scale for the affective dimension items. The previous testing by Gallup also used a 0 to 6 for the items.

Item as tested for first portion of interviews. [Please note that the CATI programming specifications are retained in brackets, as in this sentence.] Including the CATI programming specifications allows the program office to double-check text more easily against their programming specifications for the module.

Now I want to go back and ask you some questions about how you felt yesterday. We’re asking these questions to better understand people’s health and well-being during their daily lives. As before, whatever you tell us will be kept confidential.

[state this sentence if activity list is > 5, otherwise skip] The computer has selected 5 time intervals that I will ask about.

Between [starttime] and [stoptime] yesterday, you said you were doing [activity]. The next set of questions asks how you felt during this particular time.

Please use a scale from 0 to 6, where a 0 means you did not experience this feeling at all and a 6 means the feeling was very strong. You may choose any number 0,1,2,3,4,5 or 6 to reflect how strongly you experienced this feeling during this time.

1. Happy First, from 0 – 6, where a 0 means you were not happy at all and a 6 means you were very happy, how happy did you feel during this time?

2. Tired From 0 – 6, where a 0 means you were not tired at all and a 6 means you were very tired, how tired did you feel during this time?

3. Stressed From 0 – 6, where a 0 means you were not stressed at all and a 6 means you were very stressed, how stressed did you feel during this time?

4. Sad From 0 – 6, where a 0 means you were not sad at all and a 6 means you were very sad, how sad did you feel during this time?

5. Pain From 0 – 6, where a 0 means you did not feel any pain at all and a 6 means you were in severe pain, how much pain did you feel during this time if any?

6. Meaningful From 0 to 6, how meaningful did you consider what you were doing? 0 means it was not meaningful at all to you and a 6 means it was very meaningful to you.

[Interviewer note: ask about activities in chronological order. If asked, pain means both physical and mental pain.]

[Randomize the presentation of the affective dimensions (items 1-6).]

[Repeat below for activities 2-5. We are not repeating the paragraph that explains the scale.]

Between [starttime] and [stoptime] yesterday, you said you were doing [activity].

1. Happy From 0 – 6, how happy did you feel during this time?

2. Tired From 0 – 6, how tired did you feel during this time?

3. Stressed From 0 – 6, how stressed did you feel during this time?

4. Sad From 0 – 6, how sad did you feel during this time?

5. Pain From 0 – 6, how much pain did you feel during this time, if any?

6. Meaningful From 0 – 6, how meaningful to you was this time?

Item as tested for final interviews. The preface and rating items remained the same, but the number of activities sampled decreased from five to three.

Results. The affective dimension items are discussed individually below. Global issues, such as the respondent’s use of the scale, are discussed separately.

Global issues

1. Nature of the tasks to be sampled

One respondent mentioned sex as an activity during the prior day. The cognitive interviewer excluded this activity for the affective dimensions. It is important during implementation that activities deemed too personal not be selected for the module.

2. Reaction to the task

Respondents were asked “How did you react to being asked these questions about how you felt during an activity?” The majority (23) said that that either had no reaction at all or felt neutral about the questions (e.g., “the questions were fine”). Of the five respondents that had a non-neutral reaction, one person questioned “if you wonder why my ratings jumped around.” One respondent mentioned that it was “interesting because I don’t normally think about how I felt on days where my emotions aren’t extreme.” Another person said it was “a little revealing to [my]self by saying things out loud.” Yet another said there were “too many choices – too many questions about emotions. [I] felt like [I] had to really think about each…[I] felt I was confused about my activities [between] yesterday and [the] day before.”

3. Respondents’ self-reported difficulty with answering the items

Table 3 below shows the number of respondents who reported an affective dimension as being difficult to answer. The table also shows the affective dimension they were most confident in reporting. As with Round 1 of testing, none of the affective dimensions really stuck out as being difficult with more than one respondent; a majority of respondents (16) stated that none of the affective dimension items were difficult. Overall, the respondents felt that reporting the different affective emotions was “easy” or “very easy.”

Table 3. Respondents’ Reports of Most Difficult and Most Confident Ratings

		Number reporting confidence
Affective Dimension	Number reporting as most difficult	“Most confident”	“Not confident”*
Happy	5	7	1
Tired	0	5	0
Stressed	6	2	2
Sad	2	1	1
Pain	3	3	2
Meaningful	2	7	0
“All”	1	10	0
“None”	16	0	0

Note: Counts do not total 100 percent of respondents because respondents could list more than one dimension.

*When asked what affective dimension they were most confident about reporting, rather than answer that question, two respondents mentioned items that they were not confident about reporting.

As with Round 1, no respondent had problems with the items being “too personal” when asked explicitly about whether they were personal to them. The respondents were asked whether the affective dimension with the highest rating was the most intense feeling they had during the activity. All of the respondents said yes.

Respondents were also asked if there were other emotions not covered in the interview that were “a strong part of the experience.” The respondents only mentioned a couple of emotions, with the most common one being anxiety.

4. Scale differentiation

The respondents’ use of the scale was much better in Round 2 of testing than in Round 1. Repeating the scale more often during the interview (which was done for all of Round 2) improved differentiation. During Round 1, there were almost equal numbers of respondents who showed little differentiation and those that showed high differentiation (Downey, 2008):

Four respondents had a relatively high amount of differentiation. Three respondents had a moderate amount of differentiation and three respondents had little differentiation (and used exclusively zero or six).

Table 4 shows the number of point differences respondents gave across activities. This is sub-categorized by affective dimension and number of activities (mean earlier interviews versus final interviews). For example, in the row for happy, there were 17 respondents who were asked about five activities and nine that were asked about three activities. Of the 17 respondents that were asked about five activities, 11 had more than a two-point spread among their happiness ratings for the five activities, four had a only a one-point spread among their happiness ratings, and two did not have any point difference among their happiness ratings. When respondents used the same rating across all activities, this shows no differentiation; differences of two or more points show higher differentiation. Using these criteria, the ratings for tired and stressed showed the highest levels of differentiation, with more than half of the respondents using a larger (2+) range for their ratings. The sadness rating showed the lowest differentiation, with 53 to 55 percent of respondents using the same ratings across activities. The pain rating also showed low differentiation, with 35 to 55 percent of respondents using the same ratings across activities.

Table 4. Respondent Differentiation by Affective Dimension and Number of Activities

Affective dimension	Number activities	Number of point difference across activities
		2+		1		0
Happy	5	11	65%	4	24%	2	12%
	3	5	45%	4	36%	2	18%
Tired	5	13	76%	1	6%	3	18%
	3	8	73%	2	18%	1	9%
Stressed	5	13	76%	1	6%	3	18%
	3	5	45%	3	27%	3	27%
Sad	5	6	35%	2	12%	9	53%
	3	1	9%	4	36%	6	55%
Pain	5	7	41%	4	24%	6	35%
	3	4	36%	1	9%	6	55%
Meaningful	5	11	65%	3	18%	3	18%
	3	4	36%	4	36%	3	27%

As also stated in the Round 1 report, in general there is no solution for this lack of differentiation. This might truly reflect the way the respondent perceived their emotions, or it could be that 0-6 was an unusual scale for the respondents so that it encourages non-differentiated responding.

5. Use of 0-6 scale anchors

Seven respondents (out of 28) had a couple of places during the interview where they flipped the scale. One person commented that she “got mixed up I think…[I] realized what you meant after you repeated [the] explanation several times.” With most of the cases where there was an error, the respondent corrected himself during the interview after the scale was repeated.

Two people did not correct themselves during the interview and instead realized their mistakes during the follow-up questions. For example, one person felt that a “0” on the scale meant a good thing and “6” meant a bad thing.

6. Preference of 0-6 scale

In Round 1, most of the participants didn’t have a preference for a 0-6 scale or a 0-10 scale when asked explicitly. Fourteen respondents, or half, didn’t have a preference in Round 2. Of those that had a preference, six had a preference for 0-10, six had a preference for 0-6, and three had a preference for a 0-5 scale. It is interesting to note that the follow-up question mentions the option of 0-6 or 0-10 scales; the respondents who mentioned a preference for 0-5 came up with that option on their own. In general, when people preferred a 0-6 scale over a 0-10 scale, they felt it was because a 0-6 scale had fewer choices. When people preferred a 0-10 scale it was generally because it was more common and/or had heard of it more often.

One respondent had an interesting comment about the mid-points of the scale: “[I] used 5 because 6 [was] too far out; middles – 2, 3, 4 – mean the same to me.” For his ratings, he used nine 0’s, three 1’s, four 3’s, one 4, and one 5, though.

Introductory paragraph read to respondents

All but four of the respondents said they had no reaction to the explanation of the task or felt it was fine. One person said “it’s government, ‘Big Brother’…but [I] wouldn’t change anything; it’s straightforward.” Another person said she interpreted it as relating to daily lives, which might be different than how she felt yesterday in particular. Also, one person said he wasn’t sure if people completely trust whether it’s “kept confidential,” but he wouldn’t change anything and it “comes across as best as it can.” One woman said she questioned whether “the client is interested in a specific medication, is a drug company.” However, this was after she heard the whole module.

Happy

This affective dimension wasn’t discussed during the follow-up items, because it was tested by Gallup in a previous study.

Tired

This affective dimension wasn’t discussed during the follow-up items, because it was tested by Gallup in a previous study.

Stressed

Respondents were asked how they arrived at their stressed ratings. Their answers were in line with the research goals of the module, meaning that respondents gave 6’s to activities that were stressful and 0’s to those activities that were not stressful. Examples of activities that were labeled as 6’s were getting kids ready for school in the morning and getting an ultrasound. Examples of 0’s were watching TV, being at the library, and taking a walk with a friend.

Sad

Sad showed a lack of differentiation. Fifteen respondents had zeros across all activities. However, this affective dimension wasn’t part of the debriefing, because it was tested by Gallup in a previous study.

Pain

A few respondents included mental pain in their definition of pain. However, according to the module designers, this item is supposed to be left to respondent interpretation.

As expected, pain showed the most lack of differentiation. Twelve respondents had zeros across all activities.

Meaningful

Respondents were asked what meaningful meant to them. Meaningful meant different things to different people. For example, respondents defined it as “necessary for me to do,” “of meaning to me,” “beneficial to me…look forward to it,” and “personal importance.” As with pain, the module designers want respondents to apply their own definition of meaningful.

The meaningful item, though, should come after and not be randomized with the other affective dimension items. This allows for a more thoughtful answer than if it was the first question about the activity. Respondents can answer items about different affects, which will serve as cues for how meaningful the item was. Different orders of the affective items, though, were not tested.

Recommendations. The questions used in the final interviews worked well and should be used as previously detailed. However, there are two recommendations:

• Activities deemed too personal should not be included in the module.

• The “meaningful” item should appear last in the list; the other affective dimension items can be randomized.

Question 7: Interaction Question.

Purpose of the new question. The module designers would like to know whether interacting with others has an impact on people’s affect. This item was reported to have worked well in the Gallup testing without the interviewer note. During Round 1 of testing, it was found that an interviewer note was needed in case respondents asked what was meant by interacting. The module designers believe that this item can be left to respondent interpretation, meaning a definition is not needed in the question text.

Item as tested for first portion of interviews.

Were you interacting with anyone during this time, including over the phone?

If the interviewer is asked what “interacting” means, the interviewer can read this: “communicating or responding to someone in some way. Examples include talking to, emailing, or texting someone; listening to a conference call; or playing a game like tennis or chess.”

Item as tested for final interviews. This item was not changed for the final interviews.

Results. As in Round 1 of testing, respondents gave different definitions for interacting. For example, they said “involved,” “associating,” or “talking to.” Six people did not say that they were interacting during specific activities although technically they were:

• One woman did not include reading to her daughter as interacting.

• One respondent did not count being with others during a 70-minute work meeting as interacting. She also did not include taking a walk with a friend as interacting since she defined “interacting” as “phone calls or interrupting you.” She emphasized the phone portion of the question (“Were you interacting with anyone during this time, including over the phone?”)

• Another person said she “didn’t know if [you] meant just the people in the facility or if I had someone with me. [I] said no originally but [I] changed it when [I] realized the question may want [any type of] interacting...”

• Another person said she was not interacting when she was having breakfast with a friend because the friend did most of the talking.

• One woman said she was talking to someone (while having an ultrasound), but was not on the phone. Therefore, she said no for interacting.

• Finally, one woman said that she was not interacting with her son over breakfast because “he may come in and talk, but I do everything.”

During Round 1, differences in interpretation of interacting were also found and documented. However, the sponsor decided after Round 1 that the term interacting could be left to respondent interpretations and still meet the analytical purposes of the item.

Recommendations. Since this item meets the module designers’ goal without clarifying interacting to respondents, this item does not need to be modified.

Pain Item # 1: Pain Medication

Purpose of the new question. To determine if the pain the participant experienced was self-medicated.

Item as tested for first portion of interviews.

Did you take any pain medication yesterday, such as Aspirin, Ibuprofen or prescription medication? [INTERVIEWER NOTE: IF MENTIONS A DRUG, CODE AS A YES. FOR EXAMPLE, TYLENOL AND ALEVE ARE BOTH PAIN MEDS.]

• Yes

• No

Item as tested for final interviews. The word “pain” was added after “prescription.”

Did you take any pain medication yesterday, such as Aspirin, Ibuprofen or prescription pain medication? [INTERVIEWER NOTE: IF MENTIONS A DRUG, CODE AS A YES. FOR EXAMPLE, TYLENOL AND ALEVE ARE BOTH PAIN MEDS.]

• Yes

• No

Results. Respondents were asked if there were terms or words that were confusing; there were none. Additionally, there was only one problem found with this item. One person forgot to mention taking aspirin for a headache when she originally answered the pain medication item.

Recommendation. Implement the same question used in the final interviews.

Health Status # 1: Self-reported Health

Purpose of the new question. To gauge overall health and whether it relates to affect.

Item as tested for first portion of interviews.

Finally, I have a couple of questions about your health. Would you say your health in general is excellent, very good, good, fair, or poor?

1. Excellent

2. very good

3. good

4. fair

5. poor

9. dk/refused

Item as tested for final interviews. There were no changes made during testing.

Results. Item was not tested because it has received extensive cognitive testing by other researchers.

Recommendations. Implement the same question used in the interviews.

Health Status # 2: Blood Pressure

Purpose of the new question. To determine if measures of affect are related to hypertension.

Item as tested for first portion of interviews.

In the last five years, were you ever told by a doctor or other health professional that you have hypertension, also called high blood pressure, or borderline hypertension?

• Yes

• No

Item as tested for final interviews. There were no changes made during testing.

Results.

No problems were found with this item. However, this item was deleted from the module for the final interviews due to module time constraints.

Recommendations. Delete from module.

Health Status # 3: Well-rested

Purpose of the new question. To determine whether sleep is related to stress.

Item as tested for first portion of interviews.

When you woke up yesterday, how well-rested did you feel? Did you feel very rested, somewhat rested, a little rested, or not at all rested?

• Very

• Somewhat

• A little

• Not at all

Item as tested for final interviews. There were no changes made during testing.

Results. There were no problems found with this item. Respondents reported that this question was easy to answer.

Recommendations. Implement item as is.

Appendix A

Well-Being Module Tested for Round 2

• Questions added or changed mid-round (after the group debriefing on 9-09-08) are in highlighted yellow or in strikeout.

• Changes from Round 1 of testing are in red.

• Programming notes and cognitive interviewer notes in [ ].

WHO ITEM

[Add the “who was with you” question to the diary portion for work activities.

• 4 CATEGORIES ADDED TO THE DIARY WILL BE “Boss OR MANAGER; Co-workers; customers; AND people whom you supervise”

• BE SURE TO ASK THIS PROBE FOR WORK ACTIVITIES, FOR EXAMPLE: “Who was with you? / Who accompanied you?”

• IF THE PERSON SAYS CO-WORKERS: “By co-workers, did you mean you were with your manager/supervisor, people whom you supervise, or other co-workers?" ]

QUESTIONS 1 through 7

Now I want to go back and ask you some questions about how you felt yesterday. We’re asking these questions to better understand people’s health and well-being during their daily lives. As before, whatever you tell us will be kept confidential.

[STATE IF ACTIVITY LIST > 5, OTHERWISE SKIP THIS STATEMENT. BE SURE TO PICK TV AND WORK ACTIVITIES.]

The computer has selected 3 time intervals that I will ask about.

Between [Starttime of Episode] and [Stoptime of Episode] yesterday, you said you were doing [activity]. The next set of questions asks how you felt during this particular time.

Please use a scale from 0 to 6, where a 0 means you did not experience this feeling at all and a 6 means the feeling was very strong. You may choose any number 0,1,2,3,4,5 or 6 to reflect how strongly you experienced this feeling during this time.

1. Happy First, from 0 – 6, where a 0 means you were not happy at all and a 6 means you were very happy, how happy did you feel during this time?

2. Tired From 0 – 6, where a 0 means you were not tired at all and a 6 means you were very tired, how tired did you feel during this time?

3. Stressed From 0 – 6, where a 0 means you were not stressed at all and a 6 means you were very stressed, how stressed did you feel during this time?

4. Sad From 0 – 6, where a 0 means you were not sad at all and a 6 means you were very sad, how sad did you feel during this time?

5. Pain From 0 – 6, where a 0 means you did not feel any pain at all and a 6 means you were in severe pain, how much pain did you feel during this time if any?

6. Meaningful From 0 to 6, how meaningful did you consider what you were doing? 0 means it was not meaningful at all to you and a 6 means it was very meaningful to you.

[interviewer note: ask about activities in chronological order. If asked, pain means both physical and mental pain.]

[FOR PROGRAMMING IN IMPLEMENTATION, NOT IN COGNITIVE TESITNG: RANDOMIZE THE PRESENTATION OF THE AFFECTIVE DIMENSIONS (ITEMS 1-6)].

[FOR ACTIVITIES 2 THROUGH 3.] [NOTE: WE ARE NOT REPEATING THE PARAGRAPH THAT EXPLAINS THE SCALE.]

Between [Starttime of Episode] and [Stoptime of Episode] yesterday, you said you were doing [activity].

1. Happy From 0 – 6, how happy did you feel during this time?

2. Tired From 0 – 6, how tired did you feel during this time?

3. Stressed From 0 – 6, how stressed did you feel during this time?

4. Sad From 0 – 6, how sad did you feel during this time?

5. Pain From 0 – 6, how much pain did you feel during this time, if any?

6. Meaningful From 0 – 6, how meaningful to you was this time?

[interviewer note: If asked, pain means both physical and mental pain.]

[FOR PROGRAMMING IN IMPLEMENTATION, NOT IN COGNITIVE TESITNG: RANDOMIZE THE PRESENTATION OF THE AFFECTIVE DIMENSIONS (ITEMS 1-6)].

7. Were you interacting with anyone during this time, including over the phone?

[interviewer note: If asked, ‘interacting’ means “communicating with or responding to someone in some way. Examples include talking to, emailing, or texting someone; listening to a conference call; or playing a game like tennis or chess.]

Pain item # 1

8. Did you take any pain medication yesterday, such as Aspirin, Ibuprofen or prescription pain medication? [INTERVIEWER NOTE: IF MENTIONS A DRUG, CODE AS A YES. FOR EXAMPLE, TYLENOL AND ALEVE ARE BOTH PAIN MEDS.]

• Yes

• No

Health status # 1

9. Finally, I have a couple of questions about your health. Would you say your health in general is excellent, very good, good, fair, or poor?

1. Excellent

2. very good

3. good

4. fair

5. poor

9. dk/refused

Health status # 2

10. In the last five years, were you ever told by a doctor or other health professional that you have hypertension, also called high blood pressure, or borderline hypertension?

• Yes

• No

Health status # 3

11. When you woke up yesterday, how well-rested did you feel? Did you feel very rested, somewhat rested, a little rested, or not at all rested?

• Very

• Somewhat

• A little

• Not at all

Appendix B

Cognitive Testing Protocol

ATUS Well-being Module

Cognitive Interview Protocol – ROUND 2 – VERSION 4

September 11, 2008

Typographical conventions:

The text of the survey question is in plain text and surrounded by a box; however, the changes from Round 1 are in red.

Instructions to CATI interviewers or programmer notes are in ALL CAPS (for future reference when the items are deployed). Do not read aloud things in all caps.

Cognitive interviewer questions and notes are in plain text.

Questions added OR CHANGED mid-round (after the group debriefing on 9-09-08) are in highlighted yellow.

Since the ATUS is a telephone interview, the diary portion of the ATUS and the well-being questions and health questions will be administered via telephone. For the debriefings, the respondent and the interviewer will be in the same room.

Participant Recruitment

• General population

• During recruitment call, collect the following information (to get diverse sample):

  • Age

  • Race

  • Income

  • Education

• Quotas:

  • ¼ interviews to take place on Monday (to get Sunday recall)

  • ¾ interviews to take place rest of week

Beginning of Cognitive Interview

Introduction to respondent:

1. Explanation of tasks

   1. topic: go through a diary of what you did yesterday and then ask questions about health and well-being

   2. evaluate questions, not you

   3. need to know how people like you answer the questions, what you think when you hear certain terms and phrases, and how easy or difficult the question is

   4. no wrong answer

   5. ask a question, then ask follow-up questions after to determine how you arrived at an answer

   6. for the follow-up questions, think back to your thoughts as you originally answered the question

   7. one thing – you will be in this room and I’ll be in another – I will call you and we will do this over the phone. In real life the diary is asked over the phone, so want this to be similar.

   8. For the follow-up questions, where we explore your answers to the questions – I will come back to this room.

2. General issues

   1. confidentiality

   2. voluntary

   3. permission to tape

   4. consent form

3. I’ll leave now and call you on the phone. I’ll have you on the speaker phone and there might be another person with me taking notes. Then, I’ll come back into this room

Phone in Room 5: 15037

ATUS Diary

S1: Introduction

We will go ahead and start the interview now.

1. We are conducting a survey for the Bureau of Labor Statistics on how people in America spend their time. This information will help policy makers understand how people divide their time between work, family and other demands in our fast-paced society.

S2: Household Roster

1. What are the names (just the first) of all the persons living in your household? Please start with your name. I have you as [FILL NAME].

2. What is the name of the next person?

3. What is [FNAME]’s sex?

4. How is [FNAME]’s related to you?

20. Spouse (husband/wife)

21. Unmarried partner

22. Child/stepchild

23. Grandchild

24. Parent (mother or father)

25. Brother/sister

26. Other related person (aunt, cousin, nephew, mother-in-law, etc.)

27. Foster child

28. Housemate/roommate

29. Roomer/boarder

30. Other nonrelative

5. Was [FNAME] older than 18 years old on [your / his or her] last birthday?

6. Is anyone else living here now?

S4: Time-use Diary

1. Now I'd like to find out how you spent your time yesterday, [yesterday's day & date], from 4 am yesterday morning to 4 am this morning. I'll need to know where you were and who else was with you. If an activity is too personal, there's no need to mention it.

2. So let's begin. Yesterday, [previous weekday] at 4:00 AM, what were you doing? What did you do next?

IF PERSON MENTIONS THAT HE/SHE WAS DOING TWO OR MORE THINGS AT ONCE, ASK WHAT WAS THE MAIN ACTIVITY.

COMPLETE THE ACTIVITY GRID IN THE “INTERVIEWER NOTES” DOCUMENT.

3. How long did you spend [ACTIVITY]?

4. ASK THE “WHO WAS WITH YOU?” PROBE FOR ALL ACTIVITIES, EXCEPT SLEEPING, GROOMING, “PERSONAL” ACTIVITIES, AND DK/REFUSAL. FOUR CATEGORIES ADDED TO THE DIARY WILL BE “Boss OR MANAGER; Co-workers; customers; AND people whom you supervise”

Who was with you? OR Who accompanied you?

COGNITIVE INTERVIEWERS:

• BE SURE TO ASK THIS PROBE FOR WORK ACTIVITIES, FOR EXAMPLE: “Who was with you? / Who accompanied you?”

• IF THE PERSON SAYS CO-WORKERS: “By co-workers, did you mean you were with your manager/supervisor, people whom you supervise, or other co-workers?"

5. Where were you while you were [ACTIVITY]? (Comment: Was this asked for work activities?)

6. asked only if the respondent did not report any eating or drinking as a main activity

You did not report any eating or drinking yesterday. Did you do any eating or drinking yesterday as your main activity?

INTERVIEWER NOTES ON PROBING

1. For activities lasting longer than 2 hours (excludes sleeping, working, or attending high school), PROBE: Did you stop [ACTIVITY] to do anything else during that time? If yes, record the activity.

b. For work activities with a duration time greater than 4 hours, determine if the DP took any breaks of 15 minutes or longer. PROBE: "Did you take any breaks of 15 minutes or longer?" .

(1) If yes, ask the probe "What did you do?" and record the activity.

2. If no, proceed on with the interview.

c. Probe for simultaneous activity: What would you say is your MAIN Activity?

1. If DP can estimate the time for each activity: Record each activity on its own line in the diary

2. If DP cannot estimate the time for each activity but reports a MAIN

activity: Record the main activity first followed by a / and then the second activity

S5. Secondary Childcare

No Children under aged 13:

1. Now I'd like to talk with you about childcare. People often spend time with friends', neighbors' or relatives' children. During [FILL TIME THAT USED WILL USE FOR DIARY RECALL] yesterday was a child who is 12 years old or younger in your care? Please do not include any paid childcare.

IF YES - Ask: When was that?

OR ________________________________________________________

Has children under 13 years of age:

1. Now I'd like to talk with you in a little more detail about childcare. During which activities during that time was/were [FNAME] in your care and was he or she awake?

(fills name(s) of all the DP’s own children under 13 in the household)

* Probe: Any other times or activities?

1. Activities where child was in your care. [Go to next row]

96. All day.

97. None/no more childcare activities.

Don’t Know, Refused

2. Other than [FNAME] during [FILL TIME THAT USED WILL USE FOR DIARY RECALL] yesterday was a child who is 12 years old or younger in your care? Please do not include any paid childcare.

(fills household and own nonhousehold children’s names)

QUESTIONS 1 through 7

Now I want to go back and ask you some questions about how you felt yesterday. We’re asking these questions to better understand people’s health and well-being during their daily lives. As before, whatever you tell us will be kept confidential.

[STATE IF ACTIVITY LIST > 5, OTHERWISE SKIP THIS STATEMENT. BE SURE TO PICK TV AND WORK ACTIVITIES.]

The computer has selected 3 time intervals that I will ask about.

Between [Starttime of Episode] and [Stoptime of Episode] yesterday, you said you were doing [activity]. The next set of questions asks how you felt during this particular time.

Please use a scale from 0 to 6, where a 0 means you did not experience this feeling at all and a 6 means the feeling was very strong. You may choose any number 0,1,2,3,4,5 or 6 to reflect how strongly you experienced this feeling during this time.

1. Happy First, from 0 – 6, where a 0 means you were not happy at all and a 6 means you were very happy, how happy did you feel during this time?

2. Tired From 0 – 6, where a 0 means you were not tired at all and a 6 means you were very tired, how tired did you feel during this time?

3. Stressed From 0 – 6, where a 0 means you were not stressed at all and a 6 means you were very stressed, how stressed did you feel during this time?

4. Sad From 0 – 6, where a 0 means you were not sad at all and a 6 means you were very sad, how sad did you feel during this time?

5. Pain From 0 – 6, where a 0 means you did not feel any pain at all and a 6 means you were in severe pain, how much pain did you feel during this time if any?

6. Meaningful From 0 to 6, how meaningful did you consider what you were doing? 0 means it was not meaningful at all to you and a 6 means it was very meaningful to you.

[interviewer note: ask about activities in chronological order. If asked, pain means both physical and mental pain.]

[FOR PROGRAMMING IN IMPLEMENTATION, NOT IN COGNITIVE TESTING: RANDOMIZE THE PRESENTATION OF THE AFFECTIVE DIMENSIONS (ITEMS 1-6)].

7. Were you interacting with anyone during this time, including over the phone?

[interviewer note: If asked, ‘interacting’ means “communicating or responding to someone in some way. Examples include talking to, emailing, or texting someone; listening to a conference call; or playing a game like tennis or chess.]

[FOR ACTIVITIES 2 THROUGH 3.] [NOTE: WE ARE NOT REPEATING THE PARAGRAPH THAT EXPLAINS THE SCALE.]

Between [Starttime of Episode] and [Stoptime of Episode] yesterday, you said you were doing [activity].

1. Happy From 0 – 6, how happy did you feel during this time?

2. Tired From 0 – 6, how tired did you feel during this time?

3. Stressed From 0 – 6, how stressed did you feel during this time?

4. Sad From 0 – 6, how sad did you feel during this time?

5. Pain From 0 – 6, how much pain did you feel during this time, if any?

6. Meaningful From 0 – 6, how meaningful to you was this time?

[interviewer note:

If asked, pain means both physical and mental pain.]

[FOR PROGRAMMING IN IMPLEMENTATION, NOT IN COGNITIVE TESITNG: RANDOMIZE THE PRESENTATION OF THE AFFECTIVE DIMENSIONS (ITEMS 1-6)].

7. Were you interacting with anyone during this time, including over the phone?

[interviewer note: If asked, ‘interacting’ means “communicating or responding to someone in some way. Examples include talking to, emailing, or texting someone; listening to a conference call; or playing a game like tennis or chess.]

Pain item # 1

8. Did you take any pain medication yesterday, such as Aspirin, Ibuprofen or prescription pain medication?

[INTERVIEWER NOTE: IF MENTIONS A DRUG, CODE AS A YES. FOR EXAMPLE, TYLENOL AND ALEVE ARE BOTH PAIN MEDS.]

• Yes

• No

Health status # 1

9. Finally, I have a couple of questions about your health. Would you say your health in general is excellent, very good, good, fair, or poor?

1. Excellent

2. very good

3. good

4. fair

5. poor

9. dk/refused

Health status # 3

11. When you woke up yesterday, how well-rested did you feel? Did you feel very rested, somewhat rested, a little rested, or not at all rested?

• Very

• Somewhat

• A little

• Not at all

General follow-up questions

Now I’d like to ask some follow-up questions.

________________________________________

1. [ASK # 1 IF ONE OF THE ACTIVITIES WAS WORK]

When we were asking you about your activities, we also asked who was with you during the activity. We specifically asked you if someone was with your while you were at work.

1. Was it easy or difficult to remember whether someone was with you? How so?

2. Was it easy or difficult to remember who was with you? How so?

3. Were you with multiple people during this time? If so, how did you pick who you said you were with?

4. How long were you with other people?

__________________________________________________

For the next questions, I’ll ask you about how you gave your ratings for your feelings during yesterday’s activities. Try to think about how you came up with your answers about how you were feeling.

2. We asked you how you felt using this list of feelings [USE SHOW CARD].

   1. How did you react to being asked these questions about how you felt during an activity?

   2. Which feelings were you most confident about reporting?

   3. Was one of these questions more difficult to answer than the others?

      1. If yes: How so?

   4. Do you think other people might think these questions are too personal?

      1. If yes: How so?

      2. If yes: What about you?

3. Let me read the explanation of the task to you again:

Now I want to go back and ask you some questions about how you felt yesterday. We’re asking these questions to better understand people’s health and well-being during their daily lives. As before, whatever you tell us will be kept confidential.

What is your reaction to this explanation?

4. Now I’d like to ask you more about using the 0 to 6 scale.

   1. In general, what does a “0” on the scale mean? In general, what does a “6” on the scale mean?

   2. For the scale, we used a 0 to 6 scale. We could have used a different scale, such as 0 to 10. You have a preference for either a 0 to 6 scale or a 0 to 10 scale? Can you tell me more?

5. We used the word “interacting” in one of the questions. We asked “were you interacting with anyone during this time, including over the phone?” Can you think of a better word than “interacting” for this question?

EMOTION Questions by Activity

Next, I’d like to find out more about how you reacted when you heard the questions about your feelings during a specific activity. I’m going to go back to the first activity you discussed. Let’s talk about [FILL activity].

Again, I want to find out how you reacted to the questions about feeling pain, tiredness, stress, sadness, and so on. [USE SHOW CARD]

6. I’d like to ask you about your answer regarding stressed. You gave it a [VALUE].

   1. Can you tell me how you arrived at that answer?

7. Now for pain, we asked you “From 0 – 6, where a 0 means you did not feel any pain at all and a 6 means you were in severe pain, how much pain did you feel during [FILL ACTIVITY], if any?”

   1. When you heard the word “pain,” what did you think of? PROBE: Did you include both mental and physical pain?

   2. You gave the activity a [VALUE]. Can you tell me how you arrived at that answer?

8. Can you tell me how you came up with your rating for whether the activity was meaningful to you?

   1. [IF MEANINGFUL > 0] In what way was it meaningful? PROBE: Do you personally define meaningful as something that is interesting to you, or makes a contribution to society, or that has some personal value, or something else?

9. Now I’d like to ask you about whether you said you were interacting with someone while doing the activity. You said you were [INTERACTING/NOT INTERACTING].

   1. Can you tell me more about your answer?

10. Among happy, tired, stressed, sad, and pain, you gave [FILL FEELING] the highest rating while doing [FILL ACTIVITY].

    1. [IF HAD SOME VARIATION IN VALUES]

Was that the most intense feeling you had during this [FILL ACTIVITY]?

2. [IF BASICALLY RATED ALL THE FEELINGS 0]

Could you tell me more about how you were feeling during [FILL ACTIVITY]? You seemed to have a few feelings labeled as zeros and I want to make sure we capture your feelings accurately.

NOW WE ARE GOING TO START OVER WITH THE NEXT ACTIVITY.

[REPEAT 6 -10]

[COMPARE ACROSS ACTIVITIES TO FIND HIGHEST STRESSED, PAIN, AND HAPPY]

11. Of the 3 activities, you gave [FILL] activity your highest rating on [STRESSED /PAIN /HAPPY].

Was this the most [STRESSED/PAIN/HAPPY] you felt during the 3 activities?

	Notes
Happy
Tired
Stressed
Sad
Pain

12. During the activities we asked about, [FILL LIST OF 3], were there any emotions we haven’t talked about that were a strong part of the experience?

Rest of module items

PAIN ITEM # 1

I’d like to ask you about this question: “Did you take any pain medication yesterday, such as Aspirin, Ibuprofen or prescription pain medication?”

12. Were there any terms or words in this question that were confusing or for which you didn’t know the definition?

12. Now, I’d like to ask you how you arrived at your answer. What kind of pain medication did you think about when I read the question?

12. [IF TOOK MEDS] For what reasons did you take pain medication yesterday?

HEALTH STATUS 3

I asked you “When you woke up yesterday, how well-rested did you feel? Did you feel very rested, somewhat rested, a little rested, or not at all rested?”

12. Can you tell me how you arrived at this answer?

12. For this question, how easy or difficult was it to remember whether you were well-rested yesterday?

That is the end of the interview. Thank you so much!

1 Downey, K. (2008). Findings from the first round of cognitive interviews and expert review for the American Time Use Survey’s proposed Well-Being Module – REVISED.

2 Making mid-testing changes to the survey items and protocol is suggested by many researchers, including Willis (2005), and by the Constant Comparative Method (Lincoln and Guba, 1985) of qualitative research.

3 See http://appliedresearch.cancer.gov/areas/cognitive/interview.pdf

4 The ATUS questionnaire can be found here: http://www.bls.gov/tus/tuquestionnaire.pdf

100101900 1/14/2021 Page 3 of 48