NASS Comments

November 3, 2009

OMBFSIS.DOCX

By Dave Dillard

FSIS plans to sponsor ads in the Oklahoma City area educating residents about safe food handling practices. These ads are part of a “Be Food Safe” campaign FSIS wants to eventually expand to the entire country. As part of the campaign, FSIS wants to conduct a survey of Oklahoma City women between the ages of 25 and 49 who prepare meals for children or older adults. Pre- and post-ad survey interviews will examine respondent attitudes, beliefs, and behavior relating to safe food handling practices. ICF Macro of Rockville, Maryland, will collect the data. This document is a response to a request FSIS made to NASS to review its proposal for conducting the survey.

The supporting documents FSIS supplied NASS provide evidence of a well-planned survey effort. For example, FSIS is employing some bilingual interviewers to increase response. It is using CATI to make the interviewing process more efficient. It has tested the survey instrument using a handful of respondents, and it made a few changes based on the results. The flow of the interview makes sense, and I consider the transition statements imbedded in the instrument to be quite good.

At one point FSIS stated that the Oklahoma City findings would be “generalizable” to the national level. Statistically, all inferences from the survey results apply to the population from which the sample was drawn. Even though certain demographics of Oklahoma City mirror the nation as a whole, other characteristics may be different. For example, a survey in rural areas, other regions of the country, or larger or smaller cities could produce different results. However, I don’t consider this a drawback for the purpose of the campaign.

There are a couple corrections FSIS should make to its supporting documents. Question 4 in the screener asks respondents to report age ranges. The stated ranges do not include a break at age 49, which is necessary to distinguish between members and non-members of the target population. Also, the range “55 64” is missing a hyphen. If respondents report an age range above 49, should the interviewer terminate the interview? At another point the text says “144.8 consumers will respond ….” The “144.8” refers to hours, not consumers.

I have the following suggestions for FSIS to consider regarding its survey. It may want to review these suggestions with ICF Macro.

In several places the text says 400 respondents will complete the survey and 172 will not agree to participate. It would be more accurate to say an estimated 400 respondents will complete the survey.

It says respondents are not asked to furnish information of a sensitive nature. Some respondents would consider the income question to be sensitive, even though the respondents are anonymous.

I don’t understand how FSIS obtained the cost to respondents to be $1,737, or why this is a cost to respondents. Respondents are not being paid to participate, and I don’t believe the time they spend answering the survey questions would result in any lost income to them The stated figure of $12 per hour is clearly $1,737 divided by 144.8 total hours. FSIS may want to explain these costs in more detail.

FSIS should probably be aware of potential biased reporting of safe food handling practices. For example, respondents may be unwilling to admit they do not wash their hands before preparing meals. I don’t know if such a bias exists, let alone its magnitude. I just want FSIS to be aware that it may exist.

Question A5 of the survey instrument asks whether respondents have seen advertising in the past three months about safe food practices. I realize three months is the length of the proposed ad campaign, but it is possible respondents may have seen advertising prior to the campaign regarding food handling practices.

If respondents answer “no” or “not sure” to question A7, should they branch to question A10?

I found the response choices in question A11 overly subjective. For example, I could not distinguish in my mind between “slight positive impact” and “a little impact.” Narrowing the choices to “strong positive impact,” “slight positive impact,” and “no impact” may improve the instrument.

Question Q6 asks if respondents have followed certain food safety practices in the past three months. Does this mean respondents followed these practices every time they prepared food, or does it mean they followed them at least once? FSIS may want to consider offering a menu of choices for each practice, such as “all the time,” “most of the time,” “some of the time,” or “not at all.”

Finally, Question Q12 contains a typo “Black of African America.”

Overall, the proposed survey associated with the “Be Food Safe” campaign seems well designed and should provide FSIS useful information about food handling practices.