Memo
Date: July 10, 2013
Background and Purpose
The Office of Survey Methods Research (OSMR) was asked to evaluate proposed changes to the 2014 ATUS Eating and Health (EH) Module sponsored by the Economic Research Service (Appendix A). The goal of the evaluation was to ensure that the intended meaning of the questions were understood, that participants were able to accurately and comprehensively answer all the tested questions. For each of the 15 questions that were tested, there were specific testing goals, as described below.
The tight time frame of the study limited the sample to nine participants. Ideally, a larger sample size would have been used to gain broader insights and determine if the findings were idiosyncratic to the individuals included in the study or likely representative of issues that will occur for production respondents. Additionally, the small sample prevented alternative question wordings from being tested; although some minor changes were made during the testing, collecting enough information to warrant a change, and then being able to evaluate the revised wording is not possible with only nine participants. The researchers have made recommendations based on the findings, but caution that the small sample size limits the strength of the overall findings.
Participants
Nine participants were recruited from an existing OSMR participant database, which has obtained participants from a variety of sources (e.g. newspaper and online advertising, word of mouth recruiting). Participants were screened based on income, gender and whether they had children under 5 living in their home. To avoid biasing their responses during the interview, they were not screened on SNAP or WIC status. Four men and five women participated, with an average age of 47.4 years. Five participants receive SNAP benefits, and two received WIC benefits. Information about the participants is included in Table 1. All participants live in either Washington DC or Northern Virginia.
Table 1. Participant Information
Age |
Gender |
Food Stamp Recipient |
WIC Recipient |
47 |
M |
No |
No |
54 |
M |
Yes |
Yes |
55 |
M |
Yes |
No |
62 |
M |
No |
No |
26 |
F |
No |
No |
32 |
F |
Yes |
Yes |
34 |
F |
Yes |
No |
58 |
F |
Yes |
No |
59 |
F |
No |
No |
Method
The study followed a scripted protocol, developed based on a review of the proposed questions. Since the EH module questions tested didn’t incorporate the diary, participants were asked only part of the EH module and were not given a time diary. Participants answered the EH module (starting at the SODA question and ending before the INCOME question) and then were asked a series of follow-up questions for each of the 15 questions of interest. When necessary, interviewers used spontaneous probes to follow-up on participant comments. The testing protocol is available upon request. Two interviewers conducted the interviewers, with a note-taker participating in most sessions to take notes.
To analyze the data, the notes were systematically reviewed. Areas where potential issues were indicated based on participant original answers or during the debriefing are identified below (and were presented in the Initial Results Memo on June 25th). As noted above, with such a small sample size it is difficult to determine if a potential issue is a result of an individual participant’s idiosyncrasy, or is indicative of issues that are likely to plague many production respondents. Researchers used their experience to make this distinction.
Recommendations were made for changes based on the interviewers' experience with the participants and survey methods expertise. Since, in most cases, recommended revisions could not be tested due to time constraints; there is no empirical evidence that the revised questions would not also have issues, either the same issues or new ones. Additionally, there were some potential issues identified, that were deemed to be not significant enough (in either impact on the resulting data, or in the number of respondents likely to be impacted) to warrant a change in the question. A larger sample size would make these recommendations more certain, providing more information about these potential issues and the effectiveness of the final wording.
Results
Question-by-question results are reported below. There was one recurring issue that was seen across several questions: lack of consideration of the reference period. Although there was a specific period (e.g. ‘last thirty days’ and ‘last seven days’) stated in the question wording, most participants did not consider the reference period when answering questions. Instead they focused on their general patterns and based their responses on habits rather than specific behaviors or purchases.
This is a common issue for surveys, especially ones which ask question about frequent or recurring behaviors such as eating and drinking. As noted in the detailed findings, this issue will only cause a problem in the data if the past seven or thirty days were significantly different from the respondent’s typical patterns or habits. No changes to the question wordings are recommended to address the issue, as changing the wording to emphasize the reference period might introduce new issues that could not be evaluated without additional testing.
SODA
Question Tested
Were any of the beverages soft drinks? Such as cola, root beer, or ginger ale?
Main Findings
Overall this question was understood as intended. No participants had any issues with the concept of soft drinks and were able to accurately explain what the question was asking. All nine participants equated soft drinks and soda.
When asked to classify drinks as soft drinks or not soft drinks, all participants correctly classified coke, diet coke and orange soda (Table 1). A few participants incorrectly categorized root beer and ginger ale, despite these being included in the question wording. Some participants classified club soda, seltzer water and Gatorade as soda.
Table 2. Number of participants endorsing different drinks as ‘soft drinks’
Drink Type |
Number of Participants (out of 9) |
Coke |
9 |
Diet Coke |
9 |
Orange Soda |
9 |
Root Beer* |
7 |
Ginger Ale |
7 |
Club Soda |
5 |
Seltzer Water |
4 |
Gatorade |
4 |
*One participant responded ‘Don’t know’
The first question mark led to several participants interrupting the interviewer before the “such as cola…” portion of the question. This may explain why two participants didn’t include ginger ale or root beer in the question; the initial question mark was a cue that the question had ended, and they began formulating their response and stopped listening to the examples in the rest of the question.
To follow the scripted interview, as production interviews will be trained to do, the researcher read the rest of the question after being interrupted, but the flow of the question / answer series was disrupted. Removing the first question mark should change the interviewer’s intonation and reduce the likelihood that interruptions occur. This change is not expected to change the effectiveness of the question.
Question Recommended
Were any of the beverages soft drinks such as cola, root beer, or ginger ale?
Other Recommendations
Train interviewers on the correct classification of other drinks, and general guidance that if a respondent expresses uncertainty about a specific type of drink, it should be included.
DIET SODA
Question Tested
Was the soft drink diet, regular or did you have both kinds?
Main Findings
Overall this question was understood as intended. No participants had any issues with the concept of diet and regular drinks and were able to accurately explain what the question was asking.
One interesting finding was that several participants noted they “always” drink regular soda, and so the question was easy to answer. This suggests that they were not thinking of the actual sodas they drank yesterday, instead thinking of their general soda-drinking habits. As long as their soda-drinking habits are consistent, this will not lead to issues with their responses, but to the extent they have an unusual soda drink the day before the interview, they may misreport.
Participants were asked to provide examples of diet soft drinks to show their comprehension of the category (Table 3). All examples were correct and support the conclusion that the question clearly conveys the intended meaning.
Table 3. Drinks named by participants as examples of ‘diet soft drinks’
Drink Type |
Number of Participants (out of 9) |
Diet Coke |
9 |
Diet Pepsi |
8 |
Diet Mountain Dew |
2 |
Diet Sprite |
2 |
Diet Root Beer |
2 |
Any drink with no sugar or fewer calories |
2 |
Additional responses named by one participant only included Coke Zero, diet ginger ale, Pepsi Zero, and anything with 'diet' listed on the can.
Question Recommended
Was the soft drink diet, regular or did you have both kinds?
Other Recommendations
Train interviewers on the correct classification of other drinks, and general guidance that if a respondent expresses confusion about a specific type of drink, it should be included.
STORES
Question Tested
Where do you get the majority of your groceries? Do you get them at the
READ ALOUD:
Grocery store [Go to STREASON]
Warehouse club, such as Sam’s or BJ’s [Go to STREASON]
Drugstore or convenience store [Go to STREASON]
Supercenter, such as Wal-Mart or Target or [Go to STREASON]
Some other place? [Go to STREASON]
Don't know/Refused [Go to FASTFD]
Main Findings
Overall the question worked as intended, with eight participants1 understanding the intention of the question and generally understanding the response options (Tables 4 – 7).
The response options also seemed to be comprehensive; when participants were asked for other places they get groceries, they generally named specific grocery stores. Farmer’s markets and butchers were named as other stores you can get groceries, but participants noted they get fruits and vegetables or meat from these stores; they don’t do their general grocery shopping there.
There were three main findings identified with this question. The first is that a few participants reported more than one response option. The interviewer followed up with a probe asking for the store where they get the majority of their groceries and participants were able to provide a single response.
The second finding is that “supercenters” confused some participants. Two equated it with a certain type of grocery store (e.g. Flagship Giant) while another included Target and Wal-mart in both as a Supercenter and as a Warehouse club.
The third finding is that participants tended to focus on drugstores in the ‘drugstore or convenience store’ option. Of the twenty-four examples given, only five were convenience stores (7-11 and corner market). Some participants who didn’t name a convenience store were asked explicitly if 7-11 would be included, and they agreed it would.
Table 4. Participant examples of a ‘grocery store’
Place |
Number of Participants (out of 8) |
Giant |
7 |
Safeway |
6 |
Whole Foods |
3 |
Shoppers’ |
3 |
Kroger’s |
2 |
Additional responses named by one participant only included Wal-Mart, Harris Teeter, Trader Joe’s, Best Way, Food Lion, Wegman’s, and Lotte.
Table 5. Participant examples of ‘warehouse clubs, such as Sam’s or BJ’s’
Place |
Number of Participants (out of 8) |
Costco |
8 |
Sam’s |
4 |
Additional responses named by one participant only included Wal-Mart and BJ’s.
Table 6. Participant examples of a ‘drugstore or convenience store’
Place |
Number of Participants (out of 8) |
CVS |
7 |
Walgreen’s |
4 |
Rite-Aid |
4 |
7-11 |
3 |
Wal-Mart |
2 |
Corner store or mini-marts |
2 |
Additional responses named by one participant only included K-Mart and Target, which she explained were “convenient” for her to shop at.
Table 7. Participant examples of a ‘supercenter, such as Wal-Mart or Target’
Place |
Number of Participants (out of 8) |
Wal-Mart |
3 |
Target |
2 |
K-Mart |
2 |
Additional responses named by one participant only included Sam’s, Costco, flagship Giant stores, flagship Target stores, Food Lion, and Publix. One participant was not asked the question because she is not the person who usually does the grocery shopping in her household.
Although the tested order of the response options provided no evidence that it caused confusion, ERS requested 6/28 to change the order. The change is not expected to improve the confusion between the store types, but it is not expected to have a negative impact either, and so is included in the recommendation.
Question Recommended
Where do you get the majority of your groceries? Do you get them at the
READ ALOUD:
Grocery store [Go to STREASON]
Supercenter, such as Wal-Mart or Target or [Go to STREASON]
Warehouse club, such as Sam’s or BJ’s [Go to STREASON]
Drugstore or convenience store [Go to STREASON]
Some other place? [Go to STREASON]
Don't know/Refused [Go to FASTFD]
Other Recommendations
Although no changes are recommended to the question wording, we recommend adding definitions for a Warehouse club and a Supercenter that interviewers can provide to respondents who are unsure. Also, interviewer instructions should be given to probe for a single answer if multiple responses are initially provided.
STREASON
Question Tested
What is the primary reason you shop there? Is it because of
READ ALOUD ITEMS IN BOLD:
Price
Location
Quality of products
Variety of products or
Customer service?
Other
Main Findings
Overall, this question was understood as intended and participants2 were able to select an answer without issue. At least one participant chose each of the response options (Table 8).
Table 8. Reported Reasons for Store Choice
Reason |
Number of Participants (out of 8) |
Price |
3 |
Location |
1 |
Quality of products |
1 |
Variety of products |
2 |
Customer Service |
1 |
One participant was not asked the question because she is not the person who usually does the grocery shopping in her household.
As with the STORE question, a few participants initially provided more than one response option, but were able to identify a primary reason after probing.
When asked if there were other reasons for choosing a grocery store, four participants gave reasons that could be classified back into the original options, with the exception of “accepting food stamps” and “incentives.” The reasons named were:
Best deals and coupons
Convenience, best deals on certain items
Incentives, like cash back or gas points
Newer products, stock moves fast
It is likely that incentives would translate to ‘price’ for many respondents, and so no change is recommended based on that single response. Looking at the percent of respondents who receive food stamps, offering an ‘accepts food stamp option’ is not recommended as it would lengthen the question with an option that isn’t relevant for a majority of respondents.
When asked to explain each response option, participants were able to provide meaningful
explanations that aligned with the intended meaning. Price was explained to be related to lowest cost, value, and budgets. For seven of nine participants, location is related to convenience, that the store is near their home, work or bus route. Quality of products had some variety of meanings, with participants talking about the ‘look and feel’ of the food, the expiration or sold by date, general freshness and taste. One participant equated quality with quantity, but this may have been due to the fact she was thinking about explaining her Costco shopping choice.
Variety of products had a consistent meaning to participants, who used words such as “assortment,” “selection” and “choices” to explain the option. One participant talked about the brands (e.g. store brand vs. name brands) the store had to offer. Finally, participants explained customer service in terms of the staff and policies at the store. One participant talked about being able to get a rain check, while two others talked about getting help when it’s needed. Two participants mentioned the staff’s attitude (e.g. friendly, sulking) when explaining this response option.
Question Recommended
What is the primary reason you shop there? Is it because of
READ ALOUD ITEMS IN BOLD:
Price
Location
Quality of products
Variety of products or
Customer service?
Other
Other Recommendations
Provide interviewer training on what each response option means so if a respondent gives an “other” reason, they are able to code it back to the list if relevant.
5. FASTFD
Question Tested
Thinking back over the last seven days, did you purchase any prepared food from a deli, carry-out, delivery food, or fast food?
Yes [Go to FASTFD_FREQ]
No [Go to PRPMEL]
Don’t Know/Refused [Go to PRPMEL]
Main Findings
Generally, this question worked as intended, with participants seeming to understand the intention. When asked to explain what the question was asking, participants tended to have two explanations: “did you eat out” and “did you eat at …” for which they selected one of the outlets (prepared food from a deli, carry-out, delivery food, or fast food) options. For example, one participant interpreted the question as being about fast food only, while another focused on deli food.
When asked about the outlets individually, all participants were able to name at least one correct example from each category (Tables 9 and 10). Some participants showed confusion about the distinction, especially between carry-out and fast food, noting that you can carry-out almost all fast food. This overlap also was found with delivery, noting that you can also carry-out at most delivery places. Since the question does not ask respondents to distinguish purchases made from the outlets however, this should not impact the accuracy of the answer.
When asked to name examples of prepared food from a deli, four out of the nine participants named examples of delis and five named types of food that could be purchased from a deli. Examples of delis included Safeway deli, Whole Foods deli, Potbelly, Subway, Chipotle, Blimpey’s, and sandwich shops located downtown. Examples of food included potato salad (four participants), salads (two participants), chicken wings, macaroni and cheese, vegetable plates, pot roast, raisin and carrot salad, banana pudding, turkey salad, bakery bread, cold cuts, coleslaw, pasta salad, shrimp and seafood salad, and rotisserie chicken.
Table 9. Participant examples of ‘fast food’
Place |
Number of Participants (out of 9) |
McDonald's |
7 |
Burger King |
6 |
Wendy's |
5 |
Popeye's |
2 |
Subway |
2 |
Chipotle |
2 |
Additional responses named by one participant only included KFC, Quiznos, Cosi, Five Guys, Checker's, Chik-fil-a, food trucks, and hot dog stands. One participant named examples of types of fast food, including French fries, big macs, and smoothies.
When asked to name examples of carry-out, four out of the nine participants named examples of places from which you could carry out food and five named types of food that could be purchased from a carry-out. Examples of carry-out places included Chinese restaurants (two participants), Pizza Hut (two participants), sandwich shops, Taco Bell, Yum’s, Johnny’s, and Domino’s. Examples of carry-out food included Chinese food (three participants), shrimp fried rice, egg foo yung, steak and cheese, egg drop soup, fortune cookies, pizza, fried chicken, greens, and Mexican food. Two participants said that carry-out food was the same as fast food (the category asked about immediately before carry-out).
When asked to name examples of delivery food, four out of the nine participants named at least one place from which one could order delivery food; Domino’s (four participants), Pizza Hut (two participants), and Papa John’s (two participants). Four participants named at least one type of delivery food; pizza (four participants), pasta, Chinese food, and Thai food. Three participants stated that delivery food was the same as carry-out food (the category asked about immediately before delivery food).
Several participants excluded prepared foods from grocery stores, since they are not a ‘deli’ so the focus was on deli and not prepared food in general. Additionally, two included frozen prepared meals from the grocery store, but this is thought to be a result of the cognitive interviewing probing, and it is very unlikely that a production respondent who only purchased frozen prepared meals would respond yes to this question.
When asked if various establishments should be included in the question (Table 10), participants were generally correct, particularly in the clear-cut examples such as McDonald’s, pizza delivery and those including the word “deli.” As noted above, there were a few participants who incorrectly included items like groceries and frozen meals, and some who excluded prepared food from grocery stores.
Table 10. Number of participants endorsing different food types as ‘any prepared food from a deli, carry-out, delivery food, or fast food’
Food Type |
Number of Participants (out of 9) |
McDonald’s |
9 |
Chinese food delivered to your door |
9 |
Sub from a deli |
9 |
Pizza delivery |
9 |
Chicken noodle soup from a deli |
9 |
Salad from a salad bar in the cafeteria |
8 |
Fried chicken and potato salad from grocery store hot bar |
8 |
Sandwich from a vending machine* |
6 |
Pastry from a deli |
6 |
Groceries delivered to your door by a service such as Peapod |
3 |
Frozen meatloaf dinner from grocery store |
3 |
Olive Garden |
2 |
*One participant responded ‘Don’t know’
Although prepared food from a grocery store is intended to be collected by this question, the testing revealed that it may not be thought of by some respondents. However, without additional testing, we do not recommend making a change to the question. Adding “prepared food from a deli or grocery store” has the risk of inclusion of prepared meals (e.g. frozen meatloaf dinner) or leading respondents to include general groceries. Without additional information about how respondents will interpret the question, we do not recommend addressing this finding with a revision to the question wording.
There was evidence that not all participants were thinking about the last seven days, instead thinking about their usual eating patterns. Two participants responded “every day” because they always get at least one meal from fast food, without providing any evidence they were focusing on food purchased in the last seven days. Another said “no” because she said she never buys fast food.
Finally, although there was no evidence of this in the testing, some program office reviewers felt that respondents may be confused by whether the fast food questions were referring to a list of places to purchase food (deli, fast food restaurants, etc) or to a list of purchases. Adding a colon after “any” may help make it clearer that the question is referring to a list of purchases.
Question Recommended
Thinking back over the last seven days, did you purchase any: prepared food from a deli, carry-out, delivery food, or fast food?
Yes [Go to FASTFD_FREQ]
No [Go to PRPMEL]
Don’t Know/Refused [Go to PRPMEL]
Other Recommendations
Provide interviewer training on the question intention and the intent to include prepared food from grocery store delis and other outlets.
6. FASTFD_FREQ
Question Tested
How many times in the last seven days did you purchase prepared food from a deli, carry-out, delivery food, or fast food?
Main Findings
The focus of testing of this question was not on the outlets, as they were evaluated in the FASTFD question, but on the frequency concept in this question. In general, participants were able to understand and answer the question as intended.
As noted in the FASTFD question however, some participants appeared to be thinking about their general food patterns rather than the last seven days, basing their answers on their patterns. For respondents that have a very regular eating habit, this strategy is likely to be more common, but also likely not to have much impact on the data since the past seven days is likely to be similar to their general pattern. For respondents without a regular pattern, using a general impression recall strategy, rather than thinking of specific purchases may cause underreporting. Even in the small sample in this study, one participant revealed during the debriefing that he forgot a purchase in his original answer.
Question Recommended
How many times in the last seven days did you purchase: prepared food from a deli, carry-out, delivery food, or fast food?
Other Recommendations
None
7. MEAT
Question Tested (Original)
In the last 7 days, did you prepare any meals with meat, poultry, seafood or eggs?
Question Tested (Revised)
In the last 7 days, did you prepare any meals with meat, poultry or seafood?
Main Findings
This question changed during the study. The first five interviews revealed that participants were including eggs used for baking in the question, rather than egg-dishes. ERS gave approval (6/20) to drop ‘eggs’ from the question, and the final four interviews used the revised wording.
The revised wording worked as intended, participants understood the concept the question was asking for and were able to answer it accurately, with all participants responding “yes” to this question.
When explaining what the question was looking for, several participants talked mainly about “meat” but probing revealed this was a broad category that included beef, pork and poultry. Their responses were not impacted by this interpretation of the question because their definition of meat overlapped with the concepts covered in the question.
When asked what meals with meat, poultry or seafood he or she had prepared in the last seven days, participants’ responses included: chicken (two participants), shrimp (two participants), tilapia (two participants), stews with beans and vegetables made in a crockpot, steak, burgers, spaghetti and meat sauce with shrimp, baked chicken with string beans and rice, seafood at dinner, meatloaf, chicken on the grill, fried chicken, salmon, turkey burgers, scallops, and ground turkey with pasta. When asked what meals including those with eggs he or she had prepared, responses included: eggs for breakfast (two participants), and cornbread mix with eggs.
When asked to name some meat dishes that people could prepare, participants’ responses included: steak (three participants), beef stroganoff (two participants), meatloaf (two participants), fried chicken (two participants), meatballs, spaghetti and meatballs, pot roast with mashed potatoes and gravy, chicken cacciatore, chicken noodle soup, spare ribs, chicken marsala, shrimp fettuccine, barbecue ribs, lasagna, casserole, baked chicken, barbecue chicken, pork chops, beef, hamburger, chicken curries, and chicken nuggets.
When asked to name some poultry dishes that people could prepare, participants’ responses included: fried chicken (five participants), roasted chicken, baked chicken (five participants), barbecue chicken (three participants), chicken and rice, chicken marsala, barbecue chicken, chicken parmesan, chicken alfredo, chicken casserole, oven turkey, roasted turkey, and chicken nuggets.
When asked to name some seafood dishes that people could prepare, participants’ responses included: crab salad (two participants), shrimp with veggies, tuna casserole, salmon and potatoes, snow crab and biscuits, shrimp with rice and broccoli, baked tilapia, fried tilapia, shrimp, broiled fish, scallops, jambalaya, steamed shrimp, lobster, crab cakes, tilapia, cod, trout, seafood salad, shrimp salad, seared seafood, grilled seafood, seafood with pasta, and fish tacos.
When asked to name some egg dishes that people could prepare, participants’ responses included: scrambled eggs (three participants), egg salad (two participants), deviled eggs (two participants), boiled eggs, eggs with cheese, omelet with meat, omelets, cakes, pies, bread with eggs, egg custard, salad with hardboiled eggs and rotisserie chicken and vinaigrette, bacon and eggs, egg salad sandwich, quiche, egg casserole, breakfast food, and egg curries.
Question Recommended
In the last 7 days, did you prepare any meals with meat, poultry or seafood?
Other Recommendations
None
8. THERM
Question Tested
Did you use a food or meat thermometer when preparing any of those meals?
Main Findings
Overall, participants understood what this question was asking for. All participants reported knowing what a food or meat thermometer was. Although few were able to identify a difference between the two types, the distinction is not required for understanding of the question. Even the three participants who reported having “never” used a food thermometer knew what it was and what it was used for.
Question Recommended
Did you use a food or meat thermometer when preparing any of those meals?
Other Recommendations
None
9. THRMRSN
Question Tested
Did you use the thermometer to check for doneness or food safety?
Main Findings
Only one participant, who was a cafeteria manager, was able to explain the difference between doneness and food safety. The rest of the participants, including a past culinary student and the husband of a cafeteria manager, had difficulty distinguishing the difference between doneness and food safety.
The lack of meaning between the two concepts, with several participants thinking that doneness and food safety are the same thing, lead to some participants being unable to answer the question.
The difference between doneness and food safety is not likely to be meaningful to most respondents; even if they do provide an answer (some respondents are more likely to give an answer they are not sure of rather than ask for clarification or admit they don’t understand the question), the data is expected to be unreliable.
Question Recommended
Not recommended for implementation. The distinction between doneness and food safety is not likely to be meaningful to a majority of respondents
Other Recommendations
None
10. MILK
Question Tested (original)
In the last 7 days, did you serve unpasteurized or raw milk?
Question Tested (revised)
In the last 7 days, did you drink or serve unpasteurized or raw milk?
Main Findings
The first five interviews revealed that participants did not include their own raw milk consumption in the question; they only considered giving milk to someone else. ERS approved the addition of “drink or serve” to the question on 6/20, and the final four participants included both drinking milk and serving someone else in the question.
Eight participants responded no to this question, and debriefing revealed these were all accurate responses. One participant incorrectly said yes, including Lactaid milk.
Only three participants correctly defined “unpasteurized or raw milk.” The rest of the participants said either they’d never heard the term before or had incorrect definitions, such as “goat’s milk,” “evaporated milk,” and “buttermilk.” Some participants talked about the vitamins added to the milk, or what had been ‘taken out of’ the milk before it went to the store.
Without additional testing it is not possible to evaluate a revision to the question. Adding a probe (e.g. “directly from a farmer”) may help convey the intended meaning, or it may introduce new confusion about the outlet from which milk is purchased, or result in new issues that we can’t explore without testing.
Question Recommended
Consider dropping this question from the module.
Other Recommendations
If this question is used, recognize there may be some over reporting if respondents inaccurately say yes due to unfamiliarity with the concept. Provide a definition for interviewer guidance if a respondent seems unsure.
10. FDSIT
Question Tested
The next question is about the food eaten in your household in the last 30 days, and whether you were able to afford the food you need. Which of these statements best describes the food eaten in your household-- enough of the kinds of food (I/ we) want to eat, enough but not always the kinds of food (I/ we) want to eat, sometimes not enough to eat, or often not enough to eat?
1. Enough of the kinds of food (I/we) want to eat
2. Enough but not always the kinds of food (I/we) want to eat
3. Sometimes not enough to eat
4. Often not enough to eat
Main Findings
Overall this question was confusing. Table 11 shows the participants explanations of the question, and the key concept included. The question is attempting to capture all three concepts, but most participants focused on just one. This table shows that the intended meaning of the question was not understood by participants.
Table 11. Participant Explanation of FDSIT
|
Affordability |
Food Want |
Enough Food |
Do you have enough food |
|
|
X |
Do you have enough food to supply your hunger, your appetite |
|
|
X |
Did you have enough to eat, and have enough of the kinds of foods you wanted to eat |
|
X |
X |
Did we have enough of the food we preferred to eat |
|
X |
X |
Do I always have the exact food I want |
|
X |
|
If you can afford all the food you want, or maybe you can't afford food sometimes |
X |
X |
|
Am I able to purchase enough food where I wouldn't go hungry |
X |
|
|
What can you afford? |
X |
|
|
Affordability |
X |
|
|
When asked to explain the response options individually, additional issues were revealed. By reading each response option individually and asking participants to explain it, the clarity of the option is improved over the intended question administration when the respondent must listen and comprehend the question and all four options at once. Even with this focus on individual response options though, some participants struggled, with two defining the first option incorrectly, and three not defining the second option correctly. Participants included ideas like dieting (wanting to eat chocolate cake, but not buying it) or buying food for children (not buying what they want to eat all the time) in the second option.
Additionally, the question is long and difficult to ask. Participants often interrupted the interviewer before the question was finished. The long question stem and response option likely leads to participant confusion, and may also lead to interviewers stumbling when they ask it.
Finally, not a single participant mentioned the thirty-day reference period in describing how they arrived at their answer. As noted elsewhere, this issue was common throughout the module.
Question Recommended
We recommend use of an alternate question that was cognitively tested and found to work reasonably well at measuring food sufficiency. This question was used in the 1995 and 1996 Food Security Supplement of the Current Population Survey. We modified the wording slightly, changing the time period reference for consistency across the module.
The next question is about the food eaten in your household. Which of the following statements best describes the amount of food eaten in your household in the last 30 days—enough food to eat, sometimes not enough to eat, or often not enough to eat?
Enough food to eat
Sometimes not enough to eat
Often not enough to eat
12. FDSTP
Question Tested
In the past 30 days, did you or any member of this household receive [fill State SNAPNAME], SNAP, or food stamp benefits?
NOTE: The fill <SNAPNAME> is the State program name. If the State name is SNAP or Food Stamps, then the question reads, “…get SNAP or food stamp benefits.”
Main Findings
Overall this question worked as intended. Five participants said yes to this question, and the debriefing revealed their responses were accurate. All nine participants understood the question and were familiar with the food stamp program. The Note was not needed for any participants.
Only four participants were familiar with the SNAP program. One participant who was unfamiliar with the SNAP program said she thought it was different than the food stamp program. One DC resident said that only food stamps are available in DC. These findings show the importance of including both program names in the question.
Unfortunately, we were unable to recruit participants from Maryland, which has a state name for the SNAP program (FSP), so we were unable to evaluate the effectiveness of including the state name to the wording.
No participants mentioned the thirty-day reference period in describing how they arrived at their answer. This will only create a problem when a respondent starts or stops receiving benefits, but to the extent they regularly receive benefits, should not impact the accuracy of their answer.
Question Recommended
In the past 30 days, did you or any member of this household receive [fill State SNAPNAME], SNAP, or food stamp benefits?
NOTE: The fill <SNAPNAME> is the State program name. If the State name is SNAP or Food Stamps, then the question reads, “…get SNAP or food stamp benefits.”
Other Recommendations
None
13. WIC
Question Tested
In the last 30 days, did {you/you or any member of your household} receive benefits from the WIC program, that is, the Women, Infants and Children program?
Main Findings
Overall participants were familiar with the WIC program and were able to understand the question as intended. Although not all participants were able to say what the acronym stood for, they described the program and its benefits correctly.
Only two participants reported receiving WIC benefits, and debriefing revealed that their answers were likely accurate.
As with the SNAP question, no participants mentioned the reference period when explaining the question. Again, this is only an issue when a respondent starts or stops receiving benefits.
Question Recommended
In the last 30 days, did {you/you or any member of your household} receive benefits from the WIC program, that is, the Women, Infants and Children program?
Other Recommendations
None
14. EXERCISE
Question Tested
During the past seven days, other than your regular job, did you participate in any physical activities or exercises for fitness and health such as running, bicycling, working out in a gym, walking for exercise, or playing sports?
Main Findings
Overall, this question worked as intended; participants generally understood the question to be asking about physical activity or exercise.
The two potential issues with this question relate to the types of activities included and participants thinking of the general habits rather than actual behavior in the last seven days (described in the findings for EXFREQ).
When asked about what types of activities should be included in this answer, some participants seemed to over-include activities. Some included walking for transportation as exercise (one participant noting that she otherwise would take the bus if she didn’t want the exercise), taking a baby out for a walk, cleaning (‘sweeping the carpet is hard work’) or gardening.
When asked to name examples of activities that should be included in the question about exercise, participants’ responses included a wide variety of activities (Table 12)
Table 12. Participant Examples of ‘Exercise’
Activity |
Number of Participants (out of 9) |
Cycling/Bike ride |
7 |
Walking |
4 |
Swimming |
4 |
Running/Jogging |
3 |
Basketball |
2 |
Tennis |
2 |
Sports |
2 |
Aerobics |
2 |
Jump Rope |
2 |
Additional responses named by one participant include gardening, outdoor sports, motorcycling, yoga, stretching, football, jumping jacks, skating, hiking, home fitness DVDs, Zumba, water aerobics, weight lifting, and fitness classes.
When asked to name examples of activities that should not be included in the question about exercise, six participants’ responses included: pool (billiards), darts, ping pong, massage, gardening, construction work as an example of working that is not exercise, and skateboarding. Two participants stated that they were thinking of unsafe or dangerous activities that should not be included (such as skateboarding).
When asked what types of activities he or she was thinking of when answering the question, eight participants’ responses included: walking, walking up and down steps in the house, weightlifting, sweeping the rug, Zumba, running around at the park with children, modern jazz dance class, and bike riding.
Question Recommended
During the past seven days, other than your regular job, did you participate in any physical activities or exercises for fitness and health such as running, bicycling, working out in a gym, walking for exercise, or playing sports?
Other Recommendations
None
15. EXFREQ
Question Tested
How many times over the past seven days did you take part in these activities?
Main Findings
Overall, the question worked as intended and participants understood what the question was asking. Participants understood the connection to the EXERCISE question and were able to explain what the question was looking for.
As noted with several other questions, participants did not tend to think about specific events over the last seven days; rather, they answered based on general impressions of ‘how often I exercise.’ Again, this will have an impact on the data to the extent that participant exercise patterns are consistent or inconsistent. No evidence was seen in the debriefing of this. For example, one participant answered “every day” (which she was able to translate into a response of “seven”), and when probed, said she was thinking about her general patterns rather than the last week.
Also, as noted in EXERCISE question, some participants seemed to over-include activities, which led to over-counts of exercise.
Question Recommended
How many times over the past seven days did you take part in these activities?
Other Recommendations
None
Appendix A: EHM Module
American Time Use Survey
Eating & Health Module Cognitive Testing Questionnaire
June 2013
Section 1: Introduction
The EH Module begins with an introductory screen explaining the purpose of the module questions, and then proceeds to the screen asking the secondary eating questions.
Section 2: Secondary Eating
Secondary eating questions are designed to provide information on the amount of time that respondents spend eating while engaged in other activities during their diary day.
Section 3: Drinking
Section 4: Food Preparation
FDPRP Universe:
ALL Now
I'd like to ask a couple of questions about food preparation.
Enter
1 to Continue [Go to GROSHP]
GROSHP Universe:
ALL Are
you the person who usually does the grocery shopping in your
household?
1. Yes
[Go to STORES]
2. No
[Go to FASTFD]
3. I
split it equally with other household member(s) [Go to STORES]
Don’t
Know/Refused [Go to STORES]
Section 6: Income
1 One participant was not asked the question because she is not the person who usually does the grocery shopping in her household.
2 One participant was not asked the question because she is not the person who usually does the grocery shopping in her household.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | edgar_J |
| File Modified | 0000-00-00 |
| File Created | 2021-01-28 |