Attachment D - Cognitive Testing Report 2006-08 EH Module

Attachment D - Cognitive Testing Report 2006-08 EH Module.docx

American Time Use Survey-Eating and Health Supplement

Attachment D - Cognitive Testing Report 2006-08 EH Module

OMB: 1220-0187

Document [docx]

Download: docx | pdf

FINAL REPORT

OMB # 1220-0141 - Cog Lab 2412

Cognitive Testing of a Proposed Economic Research Service Module for the American Time Use Survey: Phases 1, 2, & 3

Margaret K. Vernon

The American Time Use Survey

Bureau of Labor Statistics

2 Massachusetts Avenue, NE # 4675

Washington, D. C. 20212

February 2004

OVERVIEW

This report describes three phases of sequential testing conducted to test and refine questions proposed by the Economic Research Service for addition to the American Time Use Survey (ATUS). The actual questions tested and the research protocol followed are shown in Appendices A and B.

SUMMARY: PHASE 1

The following summary describes the first phase of cognitive testing of questions proposed by the Economic Research Service as a module to the American Time Use Survey. The summary describes the results and recommendations from Phase 1 of a 3-phase research plan. The first phase of testing was designed to improve the comprehension and quality of questions proposed by ERS. Phase 2 focused on possible order and ancillary effects associated with the addition of the ERS questions. The third and final phase of testing was undertaken to work out the last kinks in comprehension and response types to the secondary eating/drinking questions and the income questions. A detailed summary of the findings from Phase 2 of testing immediately follows the summary of Phase 1 findings. A summary of results from phase 3 follows the detailed report of Phase 2.

Background and Purpose

The Office of Survey Methods Research was asked to evaluate the cognitive and linguistic issues inherent in several questions proposed to be added to the American Time Use Survey by the Economic Research Service in the form of a 5-minute module. In Phase 1, OSMR was asked to test the effectiveness of a secondary eating question and the effectiveness of other questions pertaining to food planning and eating. In addition, OSMR was asked to test the wording and perceived sensitivity of questions about weight and income.

Phase 2 was undertaken to assess more quantitative measures of question quality, to assess length of the module, and to compare order effects. The final phase of testing was undertaken to refine comprehension and response types to secondary eating and drinking, and to test comprehension of an alternative income question. The first phase of testing included 8 participants, whereas the th second phase included 16 participants. The third phase of testing included 11 participants, four of which were low income. Testing occurred in the OSMR lab for all phases.

The participants were given the time diary and ERS questions in all phases (see Appendix A for the full protocol used in Phase 2). However, in Phase 3, respondents were only given the secondary eating/drinking questions and the income questions following the time diary as the other ERS questions were seen as adequate and were therefore finalized after the second phase of testing. Results and recommendations from Phase 1 are reported first. A more comprehensive report about the second phase of testing follows. A summary of findings from phase three concludes this report.

The goals of Phase 1of the study were to:

Determine whether or not the wording of the secondary eating question was understandable to participants.
Determine whether participants were able to accurately and comprehensively recall all instances of secondary eating or drinking.
Test whether or not the term “usually” was easily understood or created difficulty in the food shopping and food preparation questions.
Determine whether or not the term “food stamps” was generally understood and a commonly used reference for food benefits.
Determine whether or not people were able to recall and respond accurately if their child had eaten a breakfast or lunch that was prepared at a school or center.
Determine whether or not health questions including weight were perceived as sensitive and/or as valid questions to be asked on a time use survey.
Determine whether or not income was sensitive, whether or not people were able to respond accurately, and whether or not bracketing techniques would be effective if someone refused or didn’t know.

Findings

The principal findings from this study were:

The secondary eating/drinking questions did substantially increase mentions of eating/drinking. Among the five people who understood the question, several instances of eating and/or drinking were reported over the instances reported in the time diary.
However, the question about secondary eating confused respondents. Three out of eight respondents clearly did not understand the secondary eating question. Specifically, some respondents had trouble keeping both eating and drinking in mind when answering the question. For example, after hearing the qualifiers for drinking, participants concentrated on reporting drinking only and forgot to think about eating food¹. Moreover, the question qualifiers confused some respondents. One individual mentioned drinking water when the question wording said not to include it. Finally, one individual thought the question meant, “Were you doing anything else during the times you mentioned eating or drinking?” Two people responded that they didn’t drink alcohol.
Respondents forgot to include secondary eating and drinking activities. In the debriefing, the interviewer went through the time diary and asked if the respondents had forgotten to mention any instances of eating or drinking when responding to the secondary eating question. Five out of eight people had forgotten to mention an instance of eating or drinking, or both, when answering the secondary eating question.
The grocery shopping and food preparation questions did not cause a problem for anyone. One person said that he didn’t do any of the shopping or food preparation. One person said he and his wife split it equally. Six people said that they were the person in the household that usually did both. The term usually wasn’t problematic for anyone. All individuals were asked to give the reasoning behind their responses. All gave logical explanations.
The phrase “food stamp benefits” was well understood. All individuals were able to tell the interviewer accurately what “food stamp benefits” are. When asked if there is another term that they use, seven out of eight said that food stamp benefits was the term they would use to refer to food benefits from the government. One individual mentioned that he had heard the term WIC. When asked if he thought food stamp benefits or WIC was the better term to use, he said he thought ‘food stamp benefits’ was the better term because it is more generally understandable. Two of the individuals interviewed did receive benefits. They used an electronic card, but still called the benefits “food stamps.”
The questions about children eating breakfast and lunch, and who had prepared it, were well understood. However, there was some potential confusion about what constituted day care. All mothers and fathers in this study knew exactly where their child/ren had eaten breakfast and lunch, what their child had eaten for breakfast and lunch, and whether it had been eaten at home, packed by a parent, or prepared at a school or center. Six out of eight participants thought that a family member or neighbor providing care constituted day care. However, two individuals did not, and would not have counted a meal at this type of location in the question.
Some participants were confused by the question about general health. Several interesting issues came up in the health portion of the interview. First, in the general health question, three out of eight participants thought that the focus was on “house” or “household health,” instead of their individual physical health. Clearly, this was an unexpected interpretation. One speculation for why this might have occurred is that people are in the mind frame of thinking about household activities and eating and then the interview drastically switches gears.
The question about weight was viewed as very sensitive. Five out of eight people felt that weight was very sensitive and suggested that people might feel that it is too personal. One person refused to answer, one person said that it is more sensitive than income, and almost everyone provided some sort of behavioral cue that suggested they felt uncomfortable. In responding, there were some who provided explanations (health issues, pregnancy), some that told the interviewer in an embarrassed way that their doctor said they needed to lose a little bit of weight, and others who laughed nervously and told the interviewer that they wished they had been interviewed on a different day because of their bad eating the day before. The pregnancy was unexpected and the interview now has a screen included for this because BMI is meaningless in the case of pregnancy.
Income was less problematic than expected, but still raised several issues. No one refused or said that they didn’t know. Seven out of eight people reported income before taxes. One individual reported income after taxes. Three out of eight people said it could be a little sensitive. Four out of eight people said that they provided their income for the last calendar year, while four said that they provided their income from a year ago up to today. No one declined to answer the question so the bracketing technique could not be tested.

Additional Issues:

There were several idiosyncratic issues that came up, which should be addressed:

During the actual time diary, one individual reported buying and drinking a coffee. A probe was needed (was it plain black coffee?) so that this time spent drinking could be left out of the final estimate of time spent eating and drinking if it was a plain black coffee or included if not².
One individual mentioned that she gets child support/alimony but didn’t include this in her income because it wasn’t specifically mentioned in the question.
One individual had a niece temporarily staying with his family. He did not include any of her government benefits in their yearly income because she is not permanently part of the household.

Recommendations and Conclusions

At the conclusion of the first phase of testing, the interviewer suggested that the most problematic question was the secondary eating question. The interviewer recommended breaking it into two questions: asking the secondary eating question first, and then asking the drinking question separately. In addition, the interviewer suggested revising the question wording about drinking so that the qualifiers would come first, which might provide individuals with a little more time to process the question and think about instances of drinking during the day. It was suggested that even though splitting the question might be less confusing, there is still the likelihood that individuals will not remember to mention all instances of eating and drinking. However, it was concluded that the secondary eating and drinking summary questions would increase the estimates of time spent in secondary eating and drinking. Including these questions would provide ERS with data about time spent in secondary eating and drinking that adds to the estimates of total time spent eating overall.

To address the recommendations about the secondary eating question, several changes were made to the questions for the second phase of testing. First, the eating and drinking questions were broken into two; one that asked about eating and one that asked about drinking. In addition, ERS decided to simplify the drinking question so that only plain water was excluded (see Appendix A for exact wording). Finally, the term “drinking any beverages” was included in the question for the second phase of testing to see if that would cue that the drinking question is not just asking about alcohol.

After the first phase of testing, it was concluded that the questions about shopping, school or center meals, meal preparation, food stamps, height, and weight generally worked well. However, it was clear that the weight question was sensitive for many people, and possibly even more so than income. The interviewer suggested that one alternative might be to put it at the very end of the survey. Also, there was some confusion about the health question. The interviewer suggested that a more specific introduction should be developed for this section of the module and that the introduction should possibly say “physical health” instead of just health. In addition, the questions about school or day-care center meals were understandable and parents were able to answer this question with ease. It was suggested that perhaps a definition should be provided for interviewers so that if the question of a family (or other informal) day care provider comes up, interviewers can appropriately instruct respondents how to answer and appropriately code responses.

In order to address the suggestions from Phase 1of testing, several changes were made to the flow of questions in Phase 2. First, in the second phase of testing, half of the participants received the income question before the health questions, and half received it after the health (weight) questions. As part of the debriefing, a paper and pencil survey was given to assess the perceived sensitivity of income and weight and to test for possible order effects (for example, is the first sensitive question asked always perceived as the most sensitive?). In addition, a more specific introduction was given for the health questions that let the participants know that the interview was going to “switch topics” and ask about “physical health” in order to clarify the health questions.

The question about income was somewhat problematic with people using different time frames and tax deductions. However, this question has been known to have these problems and ERS’s purpose in asking it here is to update this question (which was asked exactly the same way in the previous study). For these reasons the recommendation was to keep the same wording. It was the interviewer’s opinion that more problems would be caused by changing the wording than by leaving it alone.

SUMMARY: PHASE 2

Purpose

Phase 2 of testing had the following research objectives:

Compare more quantitative measures of data quality such as the average number of secondary eating activities reported compared with the average number of secondary eating activities forgotten.

Determine the average interview length of the module.

Determine possible order effects associated with questions about weight and income.

Assess comprehension of questions changed between Phase 1 and Phase 2.

Study Design

Phase 2 of testing consisted of 16 cognitive interviews. The 16 participants were randomly assigned to two different conditions. In the first condition, participants received the income question after the health questions (general health, height, and weight). In the second condition, participants received the health questions after the income question. This random assignment experiment was undertaken because weight and income had previously been identified as “sensitive” questions. ERS and the ATUS staff thought that income would be the most sensitive question in the module and therefore recommended that it go last. Therefore, in the first phase of testing, weight was asked before income. Interestingly, one person declined weight in the first phase but no one declined income. In addition, in the first phase of testing, one person said that weight was more sensitive than income. Therefore, the interviewer began to wonder if weight (with other health questions) should be asked last, even after income. The second phase of testing varied the order of income and weight in order to test possible order effects of sensitive questions. After the ERS questions were asked, but before the debriefing, a paper and pencil survey was given that asked about the perceived sensitivity of weight and income. Question order on the survey was consistent with order of questions asked during the interview. For example, if the participant received income before weight in the interview, income appeared before weight on the survey. Likewise, weight appeared before income on the survey if it was asked in that order during the interview. See Appendix B for the full survey.

Flow of Cognitive Interview

Introduction

When participants arrived in the OSMR suite, they were met at the elevator and escorted into a room with a round table and a telephone. The interviewer explained that the American Time Use Survey is a relatively new survey that assesses how Americans spend their time. The interviewer explained that the survey is designed to assess things like how people balance time in work, family, and other daily activities such as commuting. The interviewer then told participants that BLS is interested in adding some new questions to they survey, but before questions can be added they need to be tested to make sure they make sense and that people find them understandable. The interviewer told the participants that they would be given the ATUS interview (over the phone) and that the new questions would be added to the interview. The interviewer then said that there would be a discussion about the questions, and that the participants would have the opportunity to give their thoughts and reactions to the new questions. The interviewer then left the room and administered the ATUS interview over the telephone.

Order of Questions in the ATUS Interview
1. Introduction
2. Confirmation of household composition
3. Time Diary
4. Secondary eating question
5. Secondary drinking question
6. Grocery shopping and meal preparation question
7. Food stamp question
8. School breakfast and lunch (if household contains a child under 18)
9. Health questions (group 1) OR Income question (group 2)
10. Income question (group 1) OR Health questions (group 2)

Follow-up survey

After the ATUS interview, the interviewer re-convened in the same room with the participant. The interviewer asked the participant to take a paper and pencil survey that was designed to gauge reactions to the new questions that were asked during the interview. The interviewer then went over the first question as an example of what the participant was being asked to do (see Appendix B for survey).

Debriefing

After participants were done with the survey, the interviewer engaged them in a standard debriefing where perceptions, interpretations, and meaning were assessed for each module question (see Appendix A for debriefing protocol).

Interviewer and Participants

The interviewer for this study was a Research Psychologist from the Office of Survey Methods Research (OSMR). One interviewer conducted all interviews in all Phases of testing.

Participants were recruited from a variety of sources. Several participants were recruited through newspaper advertisements, several were friends of employees at OSMR, some were recruited through word of mouth among contract employees at BLS, and several of the participants were friends or partners of individuals who were recruited through the newspaper and contract sources.

Unlike Phase 1, which focused on parents with children under 18 living with them, the second phase of testing did not have any specific demographic requirements for the participants. Sixteen people participated in the second round of the study. As noted previously, Phase 2 had two test conditions. There were four males and four females in each of the conditions. Balancing gender was seen as especially important because it was possible that there would be gender differences in the perceived sensitivity of weight. In Group 1, seven participants were African American (4 males and 3 females). One person was white (1 female). In Group 2, five participants were white (3 males and 2 females). Three participants were African American (2 females and 1 male). The income of the 16 participants ranged from $6,468 to $210,000 per year. The age of participants ranged from 21 to 62 years of age.

Results

A large amount of information came out of Phase 2 testing. There is information about the timing of the module overall, there is quantitative information about perceptions of the questions, and there is both qualitative and quantitative information about the response types and quality of the questions. This section will start by talking about findings for the module overall, and the following sections will delve into issues specific to each question.

Time Length of Module
1. The module overall took an average of 3.76 minutes (range: 2.23 – 7.63).
2. The secondary eating and drinking questions together took an average of 1.81 minutes (range: 0.67 – 4.06).
3. Parents spent significantly more time on the module than non-parents (F = 8.69; p = .01), presumably because they did have 2 extra questions.

Perceived sensitivity of questions (see Appendix C, Table 1 for means)
1. There were no significant group differences in how sensitive weight or income were perceived to be, in other words, it didn’t matter if weight or income came first or second.
2. People in both groups felt significantly less comfortable when asked about income compared with weight.
3. People in both groups thought that income was a significantly more personal question than weight.

Secondary eating and drinking questions.

These two questions were designed to get an estimate of time spent in secondary eating and drinking throughout the day. It was hoped that these questions might pick up on eating as a main activity (such as a meal) that was forgotten in the time diary. In addition, these questions were designed to pick up on eating as a secondary activity --- eating that might have occurred while the respondent was doing something else.

The eating question was read as follows:

We’re interested in finding out more about how people fit meals and snacks into their schedules.

Yesterday, you reported eating or drinking between (fill from time diary; or, in the rare event there are no Code 7s, “You didn’t report any eating or drinking yesterday”):

Were there any other times you were eating yesterday, for example while you were doing something else?

Yes—(Mark column with appropriate code) go back to Diary

No—Continue interview

If Yes, and if eating was secondary activity (probe, if necessary, “during which activities?”), ask:

Were you eating the entire time you were [fill activity]?

<1> Yes

<2> No

<-3> Refused

<-2> Don’t Know

<-1> Blank

If no or don’t know:

About how long would you say you were (eating) while you

were [fill activity]?

Were there any other times you were eating yesterday? If yes, repeat above questions, if no, continue on to next question.

The secondary drinking question was read as follows:

Not including plain water, were there any other times yesterday when you were drinking any beverages?

Yes—(Mark column with appropriate code) go back to Diary

No—Continue interview

If Yes, and if drinking was secondary activity (probe, if necessary, “during which activities?”), ask:

Were you drinking the entire time you were [fill activity]?

<1> Yes

<2> No

<-3> Refused

<-2> Don’t Know

<-1> Blank

If no or don’t know:

About how long would you say you were (drinking) while you

were [fill activity]?

Were there any other times yesterday when you were drinking anything other than plain water? If yes, repeat above, if no, move on to next question.

In some respects, these questions worked well with the time diary format. Having just completed the time diary, people were eager to fit their eating and drinking reports into the context of their day. Typically, when someone was telling about an eating or drinking episode that they forgot to report in the time diary, they would spontaneously tell which activity it was during or what time the eating took place, thus making it easy to code the eating/drinking during a specific activity. For example, one woman said, “While I was watching television last night I had a glass of juice and a couple of pistachio nuts”. This type of response did not necessitate using the probe “during which activity” to find out when the eating episode might have taken place. Another typical response was, “I had a glass of juice at 4:00, right when I got home”. Again, this type of response fit well with the time diary format because the juice drinking could be coded during whatever activity was happening at 4:00. Most people said that they weren’t eating or drinking the entire time they were doing a given activity and were able to give a time estimate for the eating or drinking episode with ease.

The second example about juice drinking raises two interesting points. First, it is hard to know if the juice drinking should be entered and coded as a main activity in this case (first thing I did when I got home), or if it should be coded as a secondary activity of whatever activity was reported at 4:00 previously during the time diary. The probe, “Was that your main activity or was it while you were doing (fill what was going on at 4:00)?”, might be necessary. The second issue is that people often reported drinking to the eating question. For example, in response to the eating question, someone might say that they had a glass of juice. This, however, did not seem to be a problem since the codes for eating and drinking are the same. Following the previous example, when the response of “drinking juice” was given to the eating question, the drinking was coded, and then the interviewer asked again, were there any other times you were eating yesterday? When the interviewer got to the drinking question, the respondents didn’t seem phased by the fact that they had already reported an instance of drinking.

There were several response types that did not fit well with the time diary format and probes. One type of response to the drinking question that did not fit well was when people reported drinking during a time that they had already reported they were eating, for example, during a meal. In this case, the interviewer did not probe further because that time had already been coded as time spent eating. This doesn’t seem to be a major problem, but ERS should decide if they want secondary drinking coded separately from eating in the event that there was both eating and drinking going on at the same time. It should also be noted that eating and drinking in the actual time diary have the same code --- so differentiating between eating and drinking as a primary activity is not possible.

A second thing that happened twice in response to the drinking question was that the person responded that they drink (tea or coffee or something like that) all day at work. In this case, there is only one main activity (working), so it is not possible to code multiple different drinking activities as secondary activities that correspond with the one main activity. Basically, the instrument is not set up to handle coding multiple episodes of eating/drinking that take place during one main activity. When this happened, the interviewer asked, “Can you give me an estimate of how long you spent drinking (tea) all together yesterday at work”?

Finally, some responses didn’t clarify during which activity the eating/drinking took place. For example, to the drinking question one woman said, “Yes, I had a glass of juice when I took my medicine.” However, taking medicine had not been reported during the time diary. The probe, “during which activity” didn’t seem to work because the activity was clearly, “taking medicine.” Therefore, the interviewer asked, and when was that?

These examples illustrate the many different types of responses that might and surely will come up to the secondary eating and drinking questions. It seems that one option would be to just train the interviewers to be flexible in their probing to find out when an episode took place. If ERS is really just interested in the total time eating and drinking (and not so much the number of episodes, or distinguishing between eating and drinking), this should work fine. However, this is something that will have to be decided before moving ahead with the module.

In addition to having a variety of different types of responses to contend with, the eating question was interpreted in two different ways. Several people thought that the question only referred to snacking, while others thought it was trying to get at any other eating --- meals or snacks not reported in the diary. When the interviewer asked the eating question during the interview, one person said, “We’re talking about snacking now, right?” During the debriefing when the interviewer asked what the eating question meant “in your own words,” one person said, “It meant, was I snacking at work?”

The drinking question was interpreted as intended; people understood that beverages referred to things like juice and soda. There was no problem understanding that water should not be included.

The erroneous interpretation of the eating question is clearly a problem. One suggestion would be to change the wording on the eating question to, “Were there any other times you were eating any meals or snacks yesterday, for example, while you were doing something else?” This should cue that the question is trying to get at both meals and snacks.

Descriptive statistics on the eating and drinking questions:

An average of 2.13 episodes of primary eating and/or drinking were reported during the time diary (range: 0-6)
An average of 85.13 minutes of primary eating and/or drinking were reported during the time diary (range: 0-235)
An average of 0.81 episodes of secondary eating were reported in response to the eating module question (range: 0-4)
An average of 0.81 episodes of secondary drinking were reported in response to the drinking module question (range: 0-4)
An average of 25.31 minutes of secondary eating and/or drinking were reported in response to the two (eating and drinking combined) module questions (range: 0-135)³

Were Any Episodes of Eating & Drinking Missed?

During the debriefing, a follow-up activity was done where participants walked through their day to see if they had forgotten to mention any eating or drinking either during the time diary or when the two new module questions about secondary eating and drinking were asked.

On average, people forgot to mention 1.4 episodes of eating and/or drinking in the ATUS interview (forgot to mention them in the time diary and in the secondary eating and/or drinking questions). (range: 0-5)
On average, 17.27 minutes of eating and/or drinking were missed with the time diary and 2 module questions (range: 0-85)

The interviewer kept track of the types of eating and drinking that were forgotten about in the interview. There was no pattern to the missing episodes. They consisted of both eating and drinking, snacking and meals. The only difference between the forgotten episodes and the reported episodes was that the forgotten episodes were shorter in length than the episodes reported in the time diary and module questions.

Child meals

As before, there were no problems with responses or interpretation of the school or center breakfast and lunch questions (see Appendix A for questions). Parents knew where their child had eaten breakfast and lunch and knew if they had or had not prepared these meals. As was mentioned in the executive summary, there should be instructions for the interviewers about how to tell the respondents to answer if their child eats a meal at a paid but informal daycare (friend, family member, or neighbor). In addition, in this round of testing there was a situation where a niece was living with the family for an extended period of time. It should be decided whether or not a temporary household child should be included in the school meal question.

Usually grocery shop and prepare meals

Unlike Phase 1, the “usually” term did confuse a couple of people. During one interview, in particular, the conversation went like this:

Are you the person who usually does the grocery shopping?

“Yes, most of it”

…prepares the meals?

“Yes, I do the majority”

When the rephrasing of these questions was probed about in the debriefing, the participant said that “usually” implies some sort of routine, some sort of regularity. She said that she does the majority or more often does it, but that it isn’t a routine like “usually” implies. Another individual did not respond with a yes or no answer to the usually question but said, it’s about 60-40.” Again, they said that they do it more, or most often, but “usually” wasn’t quite the right term.

Despite the concerns expressed about the term “usually,” it wasn’t hard for the interviewer to know how to code the responses. Clearly, in the first example, the individual does “usually” do it, even if she did not prefer that term. Keeping the question wording the same would probably be acceptable. However, changing the language to say something like, “Are you the person who does the grocery shopping most often in your household?” Or, “Are you the person who does the majority of the grocery shopping in your household?” could also be considered.

Food Stamps

Similar findings were reported in Phase 2 of testing for the food stamps question compared with Phase 1 of testing. There were no difficulties answering the question, and all participants knew what food stamp benefits are. The interviewer also asked if there were any other terms people used “in their everyday language” to refer to food benefits from the government. No one had ever heard it called anything else. Three out of 16 people did receive food stamps. They understood that the benefits come on a card but they insisted that they still call it food stamps or food stamp benefits. One person said that the interview shouldn’t refer to food stamp benefits with any kind of reference to the Ebt card because other benefits come on that card, as well.

Health, Height, and Weight

The new and more specific introduction to the health questions seems to have solved the problem of health being interpreted as referring to “house” or “household health.” There were no problems with understanding or interpretation of the health, height and weight questions. As in the first round of testing, people felt weight was sensitive (although the follow-up survey suggested that regardless of whether weight or income came first, income was more sensitive). There are a variety of explanations for why weight might have been perceived as more sensitive in Phase 1 compared with Phase 2. Most plausible was that it was an artifact of asking qualitative questions of a small, non-representative sample. One woman declined to answer the weight question in the second phase of testing. Anecdotally, women seemed to find weight more sensitive than men. However, there were no significant gender differences in the perceived sensitivity of the question based on the follow-up survey (mean level of comfort when asked the question about weight: women = 3.38; men = 4.43). Men did feel more comfortable, but this difference was not statistically significant (probably due to the small sample size).

Income

Similar problems were found for the income question in Phase 2 of testing as were found in Phase 1 of testing. Different participants used different time periods for calculating their estimate (fiscal year vs. calendar year vs. last 12 months). Several people reported after taxes while most reported before. One person said that the income estimate she gave included before-tax sources and after-tax sources added together. One person forgot to include dividends in her answer. Two people with irregular income said it was difficult to estimate because of the irregularity. One person gave a “don’t know” response and said “I really don’t know” when the interviewer attempted to bridge into the bracketing technique. One person gave income per month instead of one estimate for the last 12 months. A couple of people gave wide ranges that did not fit with the existing coding ranges (“between 50 and 60 thousand”). Most people perceived the question to be fairly personal and sensitive.

As was concluded in Phase 1, there are many problems with the income question. It is the opinion of the interviewer and ATUS management that the income question should not be considered for the module. Because of the problems inherent in this question --- especially the fact that about a third of people give income for the last 12 months, a third give income for the calendar year, and about a third give income for the fiscal year --- the question will not meet the analytical objectives of ERS. Specifically, ERS is concerned about recent changes in household income since the last CPS interview. Because individuals are not consistently giving an income estimate for the correct time frame (or an estimate that necessarily includes the few months between the CPS and ATUS interview), it is doubtful that ERS will be able to provide a good update of the income reported in the final CPS interview (Month-in-sample 8) . In addition, the differential reporting of pretax and after-tax income present a data quality concern if income were to be included in the module. An additional problem that would need to be addressed is how to deal with a response that covers a wide range of incomes and could fall into more than one coding category. One potential solution would be to insert a probe, such as “Was it closer to $50,000 or to $60,000? The response could then be coded into the appropriate category.

One final concern that arose with the income question had to do with asking about both food stamps and income. Specifically, one man expressed concern that people would feel very uncomfortable being asked about both food stamps and income on a government survey. This was a concern because some people might be receiving food stamps “illegally;” that is, be above the income level at which they are supposed to be receiving them. This concern, along with the fact that income was perceived to be the most sensitive, leads to the conclusion that income should be asked last. If income were to come before weight, because of the skip patterns, those people who are not parents would receive the income question directly after the food stamp question. This could lead to more discomfort for the participant than the income question already does on its own. Therefore, the recommendation, were all questions to be included, would be that the health questions come after school meals and income be asked last.

Conclusions

Despite the changes made from Phase 1 to Phase 2 of testing, the secondary eating and drinking questions are still problematic. Several issues still need to be addressed. First, the eating question should be changed to mention both meals and snacks. Second, the questions elicit many different answer types, some of which fit well with the time diary format and some of which do not. ERS will need to clarify which variables are of most interest to them: number of eating and drinking episodes or time spent eating and drinking?

In addition, because of the redundancies in collecting eating and drinking separately (even though they are often done together), BLS recommends that, unless ERS specifically needs to differentiate time spent drinking as a secondary activity vs. time spent eating as a secondary activity, that a focus be placed on asking questions that will enable the construction of a combined measure without unduly burdening respondents. (In the primary activity coding, they are lumped together.)

It should also be noted that although the secondary questions do add to the completeness of the estimates of time spent eating and drinking during the day over and above what the diary captures initially, the Phase 2 results present clear evidence that eating and drinking activities are still omitted at a fairly high rate, even after use of the secondary probing questions.

As was found in the first phase of testing, the school/center meal questions are not a problem for parents. It is recommended that a definition be provided for the interviewer about whether or not to include informal sources of daycare in this question. In addition, it should be decided if the question is to be asked about any child living in the household, or just about sons and/or daughters of the adult being interviewed.

Unlike the first round of testing, two people protested the use of the term “usually” in the grocery shopping and meal preparation questions during the second round of testing. Participants suggested that the language be changed to “the majority” or “most often.” This change is seen as optional because it seems to be addressing a simple preference suggested by some participants. Whether or not the wording is changed, the participants’ comprehension of the question and the interviewer’s ability to code the response probably won’t change.

Similar to what was found in Phase 1 of testing, the food stamp question did not pose a problem for participants. Everyone knew what food stamp benefits were and suggested that this was the term they used for food benefits in their everyday language. No one could think of an alternative term or something else they would use to refer to food benefits from the government.

The health questions, although perceived as personal and sensitive, did not pose a problem for anyone other than slight personal discomfort. The new introduction that was developed since Phase 1 seems to be providing the appropriate transition and is alerting people to the change of topics.

Finally, the income question is quite problematic. However, as was suggested after Phase 1, if ERS intends to update this information based on the exact same question asked in a previous survey, the question needs to remain the same. ERS and BLS should consider the problems inherent in this question, especially the fact that people are giving income responses for a variety of different time frames, and decide whether or not the question can meet the analytical objectives. If the question is fielded, one recommendation is to develop a probe in the event that a “range” response is given, and the range falls across two different income ranges in the coding scheme.

Follow-up Testing

In order to address the issues and problems that came up with the secondary eating and drinking questions, it was suggested that one final round of testing be undertaken. Specifically, this final round of testing should include the time diary portion of the interview, and just the secondary eating and drinking questions. Further testing of the other questions in the module is not seen as necessary. Two changes to the eating and drinking questions are suggested. First, it is recommended that the language of the eating question be changed to ask about eating both snacks and meals. In addition, it is suggested that the drinking question ask about drinking episodes that did not occur during previously reported eating episodes. This change should make answering the drinking question easier because the respondent will not have to repeat drinking during meals or other previously reported eating, thus reducing respondent burden.

With Census Bureau concurrence, specification and implementation of the proposed questions for the ATUS Blaise software will be undertaken simultaneously with the final phase of testing.

SUMMARY: PHASE 3

A third and final phase of testing was undertaken in order to work out some of the last comprehension issues surrounding the secondary eating and drinking questions. In addition, BLS in conjunction with ERS decided that the problems inherent in the income question were too great to be able to justify using that question in the module. Therefore, ERS requested that an alternative question about income be tested instead of the original CPS version.

The third round of testing had the following specific goals:

Test a secondary eating question that specifically mentions eating meals and snacks
Test a secondary drinking question that lets respondents know that they do not need to include drinking episodes if they occurred during an already reported eating episode
Develop a probe to get at time spent “actively” drinking in the event that a response is given such as, “I drink coffee all day at work”
Test an alternative income question

Participants included 11 adults living in the greater D.C. area. ERS thought that having lower income people test the income question was of particular importance. Therefore, 4 of the 11 participants were eligible for food stamps. These low income participants were recruited from a training program at the D.C. Central Kitchen. Other participants were recruited through the OSMR participant database.

Like previous phases of testing, the time diary and ERS questions were administered over the phone, and a standard debriefing followed. However, after the second round of testing, ERS and BLS decided that the questions about school and center meals, grocery shopping, food stamps, and health were working well and the wording used in phase two would be final. Therefore, these questions were excluded from the third round of testing, as further testing was seen as unnecessary. Only the secondary eating and drinking, and the new income questions were asked after the time diary portion of the interview and tested in the third phase.

Results – Secondary eating and drinking

During Phase two of testing, it became clear that some people were interpreting the secondary eating question to be only asking about snacking. Therefore, it was recommended that the language of the question be changed to mention both snacking and meal eating. Specifically, the question was changed in the following way:

Phase two wording - Were there any other times you were eating yesterday, for example while you were doing something else?

Phase three wording – Were there any other times you were eating any meals or snacks yesterday, for example, while you were doing something else?

This change in wording did seem to make the difference that was hoped for. Participants during the debriefing were asked to say in their own words what that question meant to them. Unlike in phase two of testing, participants in phase three seemed to understand that the question was trying to get at any other eating – whether it be main meals or snacks. For example, when asked what that question meant, one participant said, “It meant, other than the times I already told you I was eating, did I eat any other times at all yesterday”. The interviewer probed: would you have included both meals and snacks in your answer? All participants said, “yes.” In addition, in response to that question during the actual interview, participants did report both meal eating and snacking.

Wording was also changed from Phase two to Phase three for the secondary drinking question. During Phase two, it happened that in response to the secondary drinking question people would report drinking during times that they already reported eating. Since codes for drinking and eating are the same, it isn’t necessary or meaningful to capture both drinking and eating that occur at the same time. It is only necessary to capture drinking occasions that occurred outside of times already reported eating. In addition, it was thought that if people didn’t have to report drinking that occurred with eating, this would decrease respondent burden. In order to address this issue, the secondary drinking question was changed from Phase two to Phase three in the following way:

Phase two wording: Not including plain water, were there any other times yesterday when you were drinking any beverages?

Phase three wording: Excluding the times you already reported eating yesterday, were there any other times you drank a beverage other than plain water?

Unfortunately, these changes seemed to create more confusion and more burden. Two people during the debriefing said that they found it difficult to answer the drinking question because they found it hard to attend to the things they were not supposed to include. In addition, they found it difficult to think about all of the drinking they did, filter out the drinking behavior that occurred during meals or included water, and give an appropriate response over the telephone. Other people demonstrated that the question was difficult by the response type they gave during the actual interview. For example, one person only mentioned drinking during the eating occasions he had already mentioned (when probed in the debrief, he had done drinking outside of the eating occasions, but had left this out of his answer). In addition, one person ignored the instructions and just reported all drinking; both during previously reported eating and not. Finally, two people reported occasions of water drinking during the day. These types of mistakes, and their relatively high frequency, suggest that the drinking question in Phase three is quite confusing, difficult, and burdensome for respondents. Compared with Phase three of testing, there were relatively few mistakes made in Phase two of testing. For example, no one in Phase two of testing reported water drinking. In Phase two it seemed to be generally well understood that water wasn’t to be included. During Phase two, there was no evidence of confusion, burden, or comprehension problems with the drinking question. It is recommended, therefore, for both data quality reasons and to decrease respondent burden, that the wording used in Phase two be used in the module. Interviewers can be trained that there is no need to re-code drinking mentioned during a previously reported eating occasion, and to only code it when it occurs at a different time than already reported eating.

Finally, a probe was developed in the event that a response type of, “I drink XX all day at work” was given to the drinking question. When this occurred, the probe, “About how long in total would you say you were actively drinking throughout the day yesterday at work”, was used. It should be noted that this approach was used with some uncertainty as to how useful this probe might be. For example, OSMR experts were skeptical as to whether or not people would be able to understand how to interpret and give a quantitative estimate to the term, “actively.” However, this was the terminology proposed by ERS, and despite the uncertainty about the potential usefulness of this question, few alternatives were proposed. As it turned out, several people in the third phase of testing did respond with “drinking all day” answers. When given the probe, they were able to give a time estimate. In addition, during the debriefing, the interviewer asked what the probe meant to them. All participants gave plausible definitions. For example, one participant said, “you meant how much time I spent in total actually sipping my coffee. I added up all the sips.” Based on the ease with which participants were able to answer this probe and the plausible definitions given, it is recommended that this probe be adopted in the future module.

Results – Income

Due to the many difficulties with the original income question, ERS and BLS decided that it was not feasible to use it in the module. Therefore, ERS proposed using two questions from the Survey of Food Stamp Non-participants, Access Study, and the National Food Stamp Program Survey. The two questions assess whether the household is above or below the 185% poverty threshold (eligible for WIC), and if below the 185% threshold, whether they are above or below the 130% poverty threshold, and therefore eligible for food stamp benefits and free school breakfasts and lunches. The two new questions were read as follows:

1. Last month, was your total household income before taxes more or less than $ X,XXX per month?

People

in Household Income (monthly) corresponds to yearly (Probe, if

necessary)

1 $ 1,436.00 17,232

2 1,927.00 23,124

3 2,416.00 28,992

4 2,907.00 34,884

5 3,397.00 40,764

6 3,888.00 46,656

7 4,377.00 52,524

8 4,868.00 58,416

Each additional person: + $ 491.00

2. If answer to 1 was LESS: Was it more or less than $ X,XXX per month?

People

in Household Income Corresponds to yearly (Probe)

1 $ 1,009.00 12,108

2 1,354.00 16,248

3 1,698.00 20,376

4 2,043.00 24,516

5 2,387.00 28,644

6 2,732.00 32,784

7 3,076.00 36,912

8 3,421.00 41,052

Each additional person: + $ 345.00

The results from testing these questions were mixed. On the positive side, ten out of 11 people answered the questions correctly (and one person said, “don’t know”). That is, they said either above or below and when the interviewer went over the question in the debriefing, participants confirmed that they had responded correctly.

Three out of four low income participants had no trouble with the questions because, as predicted, they were acutely aware of exactly what they make per month both because the low amount is a struggle for day to day living, and because they need to know for benefit forms. One of the low income people was a 23 year old student living at home, and she didn’t know exactly what her mother and step-father make, so she said she didn’t know to the income questions.

Of the remaining seven people, three had no problems with the question. One person, when asked the question during the interview, stumbled a little and the interviewer repeated the question, and he was then able to answer. When probed about this in the debriefing, he said that he just doesn’t think about his monthly income before taxes, he just knows what’s deposited into his bank account. In retrospect, the interviewer should have probed with a yearly amount and this probably would have clarified the question for this respondent. Another respondent said that she doesn’t know what her husband makes on a monthly basis, but she does know yearly. The yearly probe enabled this participant to answer the question. One person asked spontaneously, “about how much is that per year”? When given the probe, she was able to answer easily. Finally, in the debriefing, one person said they didn’t hear the part about before taxes, but that she had still answered the question correctly.

These results suggest that the income questions should be easily answerable by people who are of lower incomes. In addition, the probe that gives a corresponding yearly salary enabled many people to answer the questions accurately. Although there was some confusion, all respondents were able to answer the questions correctly. One suggestion would be to train interviewers to use the yearly probe liberally if they sense that anyone is hesitating over the questions.

Conclusions

The third phase of testing had two main goals. First, the third round of testing assessed comprehension of refined eating and drinking questions. Second, the third round of testing assessed comprehension of two new income questions. The change to the eating question, to include a phrase about meals and snacks, worked well. Participants interpreted the eating question correctly and gave appropriate response types. It is recommended that the wording used in the third round of testing be adopted for the module.

The refinements made to the drinking question worked out less well, however. The additional phrase about excluding times already reported eating was confusing for participants and made the question difficult to answer. Two participants verbalized that the question was difficult, and several respondents gave incongruent responses that suggested that the question was not interpretable. It is recommended that the secondary drinking wording used in Phase two be adopted in the module instead of the wording used in Phase three.

Although the refined wording to the secondary drinking question did not work well, the probe developed to get at the amount of time spent actively drinking, in the event that a drinking all day response was given, worked well. Participants were both able to put a time estimate on their drinking behavior as well as verbalize the meaning of the question. It is recommended that this probe be adopted for the module.

Finally, the income questions were an improvement over the original CPS income question. All participants were able to answer the questions correctly, except one who said that she didn’t know. There is still some confusion about the before taxes part of the question, one person did not hear this part of the question. In addition, many people were better able to answer the question when a probe was given that calculated what the monthly income would translate into for yearly income. It is recommended that the income questions be adopted for the module, and that interviewers be trained to use the yearly income probe liberally.

Appendix A

Cognitive Interview Protocol for the ERS Module

Setup and Flow of the Cognitive Interview

The participant will sit in one of the interview rooms with a phone, while the interviewer goes into another room and calls him/her. The interviewer will ask the following questions:

Basic demographic information (for example, some of the questions later in the survey will require information about the number of children the participant has, along with children’s names; so this information will be collected in the demographic section).
The time diary.
ERS questions.

Order of ERS Questions

The first question will be the secondary eating question. The next question will be the secondary drinking question. Next, the participant will be asked if he/she is the usual grocery shopper and food preparer. They will then be asked questions about food stamp usage. After the questions about food preparation, participants will be asked about school breakfasts and lunches for each of their children. The order of the final two sections will vary. Half of the participants will receive the health questions first and then the income question, and half will receive the income question and then the health questions.

Proposed Protocol for the Cognitive Interview

Introduction. Hi, my name is Margaret and I work here at BLS helping to develop new questions for some of our on-going surveys. We invited you here today to help us get a better understanding about some new questions we have developed for the American Time Use survey. Have you by any chance heard of this survey? It has been in the news recently.

The American Time Use research project has been going on since 2003 and gathers data about how Americans spend their time. When we interview people for the American Time Use survey, we usually ask them the survey questions over the phone. Since we want to get a better understanding of how the survey will work when we are actually gathering data, I’d like to have you take the survey over the phone just as someone participating in the study normally would. When you are answering questions to the survey over the phone, I’d like you to try and respond to them as you normally would if you were taking the survey at home. After you answer the survey questions over the phone, we’ll get back together so you can share your thoughts about some of the survey questions I asked. The purpose here is really to make sure that the survey questions we are asking make sense to you.

If it’s alright with you, I’d like to tape record the telephone conversation and our conversation afterwards so that during the interview I can really focus on what you are saying and pay attention to you. I won’t share this tape with anyone; it’s really just to help me remember later what you said so I can get it right.

I also want to let you know that your participation here today and anything you say during the survey or our conversation afterwards is completely confidential. Also, you are here today as a volunteer which means that you can stop our conversation at any time and decline to answer any question you do not wish to answer. In order for you to participate, I’d like to ask you to sign this consent form which lets me know that you are agreeing to help out with our survey questions today.

Phone Interview Protocol

Hi, this is Margaret calling from the US census Bureau. We are conducting a survey for the Bureau of Labor statistics on how people in America spend their time. This information will help policy makers understand how people divide their time between work, family, and other demands in our fast-paced society. The survey will take about 15 to 20 minutes.

Before we get started, I need to verify that (X, Y, and Z) live with you at your home address. Do all of these persons still live here?

Now I’d like to find out how you spent your time yesterday, Sunday November 7, 2004, from 4:00 in the morning until 4:00 this morning. I’ll need to know where you were and who else was with you. If an activity is too personal, there’s no need to mention it.

So let’s begin. Yesterday, Sunday, at 4:00 am, what were you doing?

Go through time diary.

Eating as a Secondary Activity

We’re interested in finding out more about how people fit meals and snacks into their schedules.

Yesterday, you reported eating or drinking between (or, in the rare event there are no Code 7s, “You didn’t report any eating or drinking yesterday”):

[fill times when hard code 7 was entered].

Were there any other times you were eating yesterday, for example while you were doing something else?

Yes—(Mark column with appropriate code) go back to Diary

No—Continue interview

If Yes, and if eating was secondary activity (probe, if necessary, “during which activities?”), ask:

Were you eating the entire time you were [fill activity]?

<1> Yes

<2> No

<-3> Refused

<-2> Don’t Know

<-1> Blank

If no or don’t know:

About how long would you say you were (eating) while you

were [fill activity]?

Were there any other times you were eating yesterday? If yes, repeat above questions, if no, continue on to next question.

Not including plain water, were there any other times yesterday when you were drinking any beverages?

Yes—(Mark column with appropriate code) go back to Diary

No—Continue interview

If Yes, and if drinking was secondary activity (probe, if necessary, “during which activities?”), ask:

Were you drinking the entire time you were [fill activity]?

<1> Yes

<2> No

<-3> Refused

<-2> Don’t Know

<-1> Blank

If no or don’t know:

About how long would you say you were (drinking) while you

were [fill activity]?

Were there any other times yesterday when you were drinking anything other than plain water? If yes, repeat above, if no, move on to next question.

Now I’d like to ask a few questions about food preparation.

Are you the person who usually does the grocery shopping in your household?

<1> Yes

<2> No

<3> I split it equally with other household member

<-2> Don’t know

<-1> Refused

Are you the person who usually prepares the meals in your household?

<1> Yes

<2> No

<3> I split it equally with other household member

<-2> Don’t know

<-1> Refused

In the past 30 days, did (you or anyone in your household) get food stamp benefits?

<1> Yes

<2> No

<-3> Refused

<-2> Don’t Know

<-1> Blank

Now I’d like to ask about how your children fit their meals in when they are away from home.

Please think back over the past week starting last Monday up to today Monday. In the past week, did [Fill names of children in the HH under age 18] eat a BREAKFAST that was prepared and served at a school, a paid day care, or Head Start Center or a summer day program*? This question refers ONLY to BREAKFASTS prepared at the school or center—not meals brought from home.

<1> Yes

<2> No

<3> Child did not attend school or day care

<-3> Refused

<-2> Don’t Know

<-1> Blank

If yes,

Which child/ren?

<fill names>

*Note: only include summer day program seasonally: in the summer when this would make sense.

Also, head start is only for children 5 and under so we could leave this off if N/A.

What about LUNCH? In the past week, did [Fill names of children in the HH under age 18] eat a LUNCH that was prepared and served at a school, a paid day care, or Head Start Center or a summer day program*? This question refers ONLY to LUNCHES prepared at the school or center—not meals brought from home.

<1> Yes

<2> No

<3> Child did not attend school or day care

<-3> Refused

<-2> Don’t Know

<-1> Blank

Which child/ren?

<fill names>

*see comments above.

The next question is about your household income.

I’d like to ask you about the total combined income of your family during the last 12 months. This includes money from jobs, net income from business, farm or rent, pensions, dividends, interest, social security payments and any other money income received (by members of this FAMILY who are 15 years of age or older.) What was your total household income in the last 12 months?

If don’t know or refused;

Was your total household income over the last 12 months more or less than $25,000?

Response: less: was it more or less than 15,000?

If less, ask, was it more or less than 5,000?

If less, code 1; if more ask, was it more or less than 10,000?

If responded more to 25,000; then ask: was it more or less than 35,000?

If less, ask was it more or less than 30,000?

If less, code 8; if more code 9.

If responded more to 35,000; ask was it more or less than 50,000?

And so forth.

<1> Less than $5,000

<2> 5,000 to 7,499

<3> 7,500 to 9,999

<4> 10,000 to 12,499

<5> 12,500 to 14,999

<6> 15,000 to 19,999

<7> 20,000 to 24,999

<8> 25,000 to 29,999

<9> 30,000 to 34,999

<10> 35,000 to 39,999

<11> 40,000 to 49,999

<12> 50,000 to 59,999

<13> 60,000 to 74,999

<14> $75,000 or more

Alright, we’re almost done here. I’m going to switch topics and ask you a few final questions about your physical health that might affect how you use your time.

In general, would you say that your physical health is very good, good, fair, poor, or very poor?

<1> very good

<2> good

<3> fair

<4> poor

<5> very poor

<-3> Refused

<-2> Don’t Know

<-1> Blank

How tall are you without shoes?

_____ feet _____ inches

<-3> Refused

<-2> Don’t Know

<-1> Blank

How much do you weigh without shoes (women ages 15-45: Since pregnancy effects weight, please let me know if you are currently pregnant)?

_____ pounds

<-3> Refused

<-2> Don’t Know

<-1> Blank

Alright, that’s all I have for today, thank you very much for your time.

Follow-up Interview Protocol: allow the participant to vent for a minute by asking how they found the survey, have a little conversation/small talk, then do follow up survey followed by debriefing.

After the telephone interview has been completed, the interviewer and participant will re-convene in the same room. The interviewer will give the participant the follow up paper survey. The interviewer will read the following introduction:

Since some of the questions that you heard today are new questions, we are interested in how people like yourself found the questions to be before we add them to the survey. The evaluation form that I am giving you now has the new questions listed followed by specific questions about them. For example, the first question that I would like you to think about is one of the questions that was asked after you completed the diary. If you remember, I asked if there “Were there any other times you were eating yesterday, for example while you were doing something else?” The follow-up questions then ask some specific things about this question. Is it clear what I would like you to do?

The participant will then take the paper survey.

The survey will ask for the participant’s reactions to each question in the module. The participant will have the opportunity to tell us both how sensitive and personal they perceived each question in the module to be. See Appendix A for survey. The interviewer will then ask the following questions:

Now I’m going to ask about some of the specific questions on the survey.

Eating & Drinking Activities

Do you remember when I asked the question ‘Were there any other times you were eating yesterday, for example while you were doing something else?’ If they remember, move onto questions. If don’t remember, use extra memory jog: remember, you said XXX?

In your own words, can you tell me what that question meant to you?
Was it difficult or easy to answer that question? Why? What made it difficult?
When you were trying to remember all of the times yesterday when you were eating, what types of things did you think about to help you remember? For example, do you usually eat a snack when you watch tv?

What about the question, 'not including plain water, were there any other times yesterday when you were drinking any beverages?’

Can you tell me in your own words what that question meant to you?
- Probe: What did it mean to you when I said ‘not including water?’
- Probe: Did that change how you answered the question?
- Probe: What beverages did you think of when I asked that question?
- Probe: Are there any beverages that you left out of your answer?
Was it easy or difficult to answer that question?
What types of things did you think of to help you remember when you might have been drinking beverages yesterday?

Review Diary Activities for Missed Eating & Drinking

Now I’d like to go back over the activities you listed in your time diary. When I go through each activity you mentioned, I’d like you to think about whether or not you might have been eating or drinking anything other than plain water, that you forgot to mention during the telephone interview. If you forgot to mention a time when you were eating or drinking, that’s ok, we just want to make sure that we can get it now.

Review Diary Activities

Probe: for each forgotten activity, find out how much time they spent on that instance of eating and/or drinking.
Ask about the specific circumstances of the episode: eating or drinking? While doing what?

Food Shopping/Food preparation Questions

During the interview, I asked if you are the person who usually does the grocery shopping.
What does this question mean to you?
When you said that you do (or do not) usually do the grocery shopping, what types of things did you think about that helped you to answer this question? What does usually mean to you?
I asked if you are the person who usually prepares the meals.
What does this question mean to you?
When you said that you do (or do not) usually prepare the meals, what were you thinking about that helped you to answer this question? What does usually mean to you?

Food Stamps

You said that your household (did or did not) get food stamps. Do you know what food stamp benefits are?
- Do you know how someone might use these benefits? A coupon, a card?
Are there any other names for this program that you have heard of?
Are there any other names for this program that you use in your everyday language?
How did you react to being asked that question?

School Breakfast/Lunch

You mentioned that [Fill names of children in the HH under age 18] ate (or did not eat) a BREAKFAST that was prepared and served at a school, a paid day care, or Head Start Center or a summer day program in the past week.

Where did your child(children) eat their breakfast? (repeat for lunch)
When you heard the phrase “a breakfast (lunch) that was prepared and served,” what type of situation did you think of?
Does you child’s school serve breakfast (lunch)?
Do you know what type of breakfast was served? (repeat for lunch)
When you hear the phrase “paid day care,” what comes to mind? What would you include?
- Probe: would you include an informal or family day care - one where a neighbor, friend, or family member takes care of a few children in the home?

Health

I asked you if, in general, you thought your physical health is very good, good, fair, poor, or very poor?

How did you arrive at your answer to this question?
How did you react to being asked this question?

Height

I also asked you what your height is without shoes.

Can you tell me how you arrived at your answer?
When was the last time your height was measured?
Do you know your height exactly without shoes?
- Would you say your answer was exact or within a couple of inches?
How did you react to being asked this question?

Weight

I also asked you a question about your weight.

Can you tell me how you arrived at your answer for your weight without shoes?
When was the last time your weight was measured?
Do you know your weight exactly without shoes?
- Would you say your answer was exact or within a couple of pounds?
How did you react to being asked this question?

Does asking about health, height and weight together make sense to you?
If you had to give a good reason for asking them, what would it be?

Income

I asked you about the total combined income of your family during the last 12 months. This includes money from jobs, net income from business, farm or rent, pensions, dividends, interest, social security payments and any other money income received (by members of this FAMILY who are 15 years of age or older.)

How did you arrive at your answer for this question?
- What time period did you think about?
- Was this amount before or after tax deductions?
Which people did you include when estimating your family’s income?
What sources of income did you count?
How accurate an estimate of your household income were you able to give me?
- Would you say it was very accurate, accurate, or not very accurate?
How do you think people will react to being asked about their income

Appendix B

Paper survey

Follow-up Survey

Question 1. Were there any other times you were eating yesterday, for example while you were doing something else?

a. How hard was it for you to remember the information requested by this question? (Circle One)

(Very hard) (Hard) (Neutral) (Easy) (Very Easy)

b. If you were asked to judge how accurate your answers to this question were, would you say they were …

(Very inaccurate) (Inaccurate) (Neither accurate nor inaccurate) (Accurate) (Very accurate)

c. How personal do you think the information is that this question is requesting?