The observational study will employ an experimental design in which participants are randomly assigned to a treatment group (exposure to food safety messaging) or to a control group (no exposure to food safety messaging). The observational study will be conducted in test kitchen facilities located at North Carolina State University (NCSU); thus, it will be necessary to recruit study participants who live within driving distance of the facility in the Raleigh-Durham area of North Carolina. As described below, convenience sampling will be used with quotas to ensure that study participants reflect the demographic characteristics of the U.S. population. When randomly assigning participants to the treatment and control groups, we will take necessary and reasonable steps to ensure that the demographic characteristics of each group are similar. Because probability-based sampling will not be used, inferences cannot be made to the U.S. population; however, by using an experimental design that is powered to detect change in the desired outcome (the power analysis is described in B.2), we can assess whether the food safety messaging has an impact on the desired outcome among study participants.
Assuming COVID-19 infection rates do not increase above the current level in Raleigh, North Carolina (the location for the in-person data collection) and as long as North Carolina State University’s (NCSU), the subcontractor for the in-person data collection, is permitted to do so by state, local, and university policy, which is based on continuous review of COVID-19 community transmission and adherence to social distancing guidelines, data collection for fiscal year 2020 will take place (starting in October 2020 and continuing through spring 2021).
Procedures will be in place to protect data collectors and study staff from being infected with COVID-19 and to prevent the transmission of COVID-19. Recruiting materials will indicate that COVID-19 screening will be part of the screening and eligibility process. Participants who have tested positive for COVID-19, have symptoms of COVID-19 (including temperature above 100°F), interacted with someone who has been diagnosed with COVID-19, or are age 65 or older will not be eligible to take part in the study. On the day of the observation, a COVID-19 screening tool will be administered, and participants must pass this screening to take part in the study (Appendix N). Data collectors and participants will be required to follow the COVID-19 procedures specifically established for this study (see Appendix M) and approved by NCSU’s Institutional Review Board. These procedures include wearing a mask, practicing social distancing, limiting the number of individuals allowed at the kitchen facility, disinfecting in between observations, and following the other procedures as detailed in Appendix M.
Respondent Selection Methods
Observational study participants will be recruited from the Raleigh-Durham area of North Carolina using convenience sampling via social media (see Appendix B) outlets and by sending emails to Expanded Food and Nutrition Education Program (EFNEP) participants to reach low-income consumers (see Appendix C). If needed, notices about the study will also be posted in grocery stores; food retailers; Women, Infant, and Children (WIC) clinics; and other organizations throughout the Raleigh-Durham area of North Carolina (see Appendix D). Recruitment materials will direct prospective participants to call or email the study team to be screened for eligibility or to a web link that will host the screening questionnaire (see Appendix E). For participants screened by phone, eligible participants will be invited to participate in the study and an appointment scheduled during the screening call. For participants who complete the web-based screener, eligible participants will be contacted by phone and invited to participate in the study and an appointment scheduled (see Appendix F). Appointments will be scheduled during work hours, evenings, and weekends to allow for a broader participant pool.
The respondent universe is English-speaking adults living in the Raleigh-Durham area of North Carolina. Participants must meet specific inclusion and exclusion criteria. The inclusion criteria are as follows:
Are aged 18 to 64
Speak English
Have primary responsibility for preparing food in the household
Have cooked meat or poultry at home in the past 3 months
The exclusion criteria are as follows:
Have taken ServSafe training in the past 5 years
Have cooked or worked professionally in a food preparation setting in the past 5 years
Do not pass COVID-19 screening criteria
As part of the screening process, we will also collect information on participant and household demographic characteristics to ensure that the demographics of recruited participants are similar to those of the U.S. population based on Census data. Table B-1 identifies the demographic characteristics and the distribution of these characteristics for the U.S. population. Participants will not be asked specifically about income, but including recruitment from the EFNEP list will help ensure that low-income individuals participate, because there is a maximum income requirement for program eligibility.
Table B-1. Demographic Characteristics for the U.S. Population (2018)
Characteristic |
Response Categories |
Percentage |
Race |
White |
76% |
|
Non-Whitea |
24% |
Ethnicity |
Not Hispanic or Latino |
82% |
|
Hispanic or Latino |
18% |
Age |
20–34b |
28% |
|
35–54 |
35% |
|
55+ |
37% |
Education |
Less than high school or high school diploma/GED |
40% |
|
Some college |
29% |
|
Bachelor’s degree |
19% |
|
Graduate or professional degree |
12% |
Children in Household |
Yes |
31% |
|
No |
69% |
Source: U.S. Census Bureau. (n.d.). 2014-2018 American Community Survey 5-year data profiles. Retrieved from https://www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/2018/
aNon-White includes Black or African American, American Indian or Alaska Native, Asian, Native Hawaiian and Other Pacific Islander, other races, or 2 or more races.
bAge is reported as ≤ 19; thus, the first age category starts at age 20; however, we will enroll individuals who are 18 or older.
Sample Size and Allocation
The power analysis (described in B.2) indicates a required sample size of 400 participants for each iteration of the observational study, divided equally between a treatment group and a control group, so 200 participants per group. We will use convenience sampling with set quotas. The expected allocation of the sample is shown in Table B-2. When randomly assigning participants to the treatment and control groups, we will take necessary and reasonable steps to ensure that the demographic characteristics of each group are similar.
Table B-2. Sample Allocation for Each Iteration of the Observational Study
Characteristic |
Response Categories |
Number |
Race |
White |
304 |
|
Non-Whitea |
96 |
Ethnicity |
Not Hispanic or Latino |
328 |
|
Hispanic or Latino |
72 |
Age |
20–34b |
112 |
|
35–54 |
140 |
|
55 to 64 |
148 |
Education |
Less than high school or high school diploma/GED |
160 |
|
Some college |
116 |
|
Bachelor’s degree |
76 |
|
Graduate or professional degree |
48 |
Children in Household |
Yes |
124 |
|
No |
276 |
Source: U.S. Census Bureau. (n.d.). 2014-2018 American Community Survey 5-year data profiles. Retrieved from https://www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/2018/
aNon-White includes Black or African American, American Indian or Alaska Native, Asian, Native Hawaiian and Other Pacific Islander, other races, or 2 or more races.
bAge is reported as ≤ 19; thus, the first age category starts at age 20; however, we will only enroll individuals who are 18 or older.
Note: When randomly assigning participants to the treatment and control groups, we will take steps to ensure that the demographic characteristics of each group are similar.
Response Rate
The expected response rate (show rate) for the observation study is 80% based on the contractor’s experience with similar studies. Thus, the starting sample size for each iteration of the observational study is 500, yielding 400 completed observations (500 x 0.80).
This section provides an overview of the study procedures, provides information on the degree of accuracy required for the study, and discusses the estimation procedures. The observational study is not employing statistical methodology for stratification and sample selection. There are no unusual problems requiring specialized sampling procedures. Participants are only being contacted one time, so periodic data collection cycles are not applicable.
Study Procedures
Before the observation and food preparation, the meat/poultry products will be inoculated with a harmless, realistic, and known amount of tracer bacteria. Under video observation, participants will be asked to prepare two recipes: one made with a raw meat or poultry product and one made with a ready-to-eat product (recipes and instructions for the first iteration of the study are provided as Appendix L). After receiving the appropriately assigned messaging (the treatment group will receive messaging on food safety specific to the behavior of interest), participants will receive instruction to cook the recipes as they would usually do so at home.
The study will be conducted in NCSU test kitchens (located in Raleigh, North Carolina) specifically designed for observation studies. Video recording equipment will be set up to record meal preparation. Trained research staff will conduct the video recording. Following the meal preparation and before cleanup, trained sample collectors will conduct surface swab sampling, and samples will be transported to an NCSU testing laboratory.
Recording of food handling and meal preparation will begin as soon as the participant enters the test kitchen and will end after the participant leaves. Participants’ cleaning and sanitizing of equipment and environment before and after preparation will also be recorded to evaluate intrameal and intermeal contamination risks. Following the observation portion of the study, trained sample collectors will take surface swab samples from surfaces such as kitchen surfaces, utensils, food containers, appliance handles, kitchen towels, cutting boards, and the ready-to-eat dish (up to 15 sites in total). The swabs will be delivered to the NCSU testing laboratory and plated to determine presence and concentration of the tracer. The presence of the tracer will indicate that cross-contamination occurred during food preparation. The level of cross-contamination will be compared across the sampling sites to determine the highest risk areas. Kitchen surfaces, appliances, and other potentially contaminated sites will be cleaned and sanitized after each participant.
Supplementing the observations, post-observation interviews (see Appendix I) will be conducted to provide insight into participants’ views, opinions, and experiences of their preparation practices of these products and their usual practices for preparing the products at home. Participants in the treatment group will also be asked about their exposure to the intervention. Collecting qualitative data will allow the project team to connect the knowledge, attitudes, and perceived behaviors with actual observed practices, allowing for more targeted intervention refinement or development.
Trained coders will use a coding rubric to evaluate the video observations based on the four food safety handling behaviors of clean, separate, cook, and chill. This rubric will be used to consistently define when the recommended behavior occurred or when one did not occur when it should have. Additionally, the coders will use notational analysis to assess recorded actions and their frequencies.
Degree of Accuracy Required for the Study
Sample size calculations were conducted to determine the minimum number of participants needed to provide a level of confidence that the experimental component of the observational study is sufficiently powered, meaning that a change of the anticipated size or greater would be interpreted as occurring beyond chance (i.e., statistically significant). By convention, we aim for 80% statistical power and a 95% level of confidence.
The purpose of the observational study is to evaluate the impact of FSIS educational materials on consumers’ demonstrated use of recommended safe food handling practices (clean, separate, cook, and chill). As noted in the initial approval for information collection, the primary outcome of interest for the first observational study was use of a food thermometer to check the doneness of meat and poultry. This is an important but not commonly practiced behavior in American kitchens. Based on recent estimates, it is anticipated that food thermometer use will be observed 5% of the time among the control group participants (Anderson, Shuster, Hansen, Levy, & Volk, 2004; Phang & Bruhn, 2011; Bruhn, 2014; Mazengia, Fisk, Liao, Huang, & Meschke, 2015; Scott & Herbold, 2010). Additionally, it is anticipated that the food safety messaging materials will provide medium effects among the treatment group participants. Table B-3 provides potential observed differences between control and treatment groups ranging from 4 to 12 percentage points. It is anticipated that the food safety messaging materials will be sufficient to generate differences in the middle of this range (i.e., the observed difference between the control and treatment groups is 8 percentage points).
Table B-3. Sample Size Requirements for Different Observed Differences between the Control and Treatment Groups
Proper Thermometer Use: Control Group |
Proper Thermometer Use: Treatment Group |
Observed Difference Between Control and Treatment Groups |
Total Sample Size (N) |
5% |
9% |
4% |
1,270 |
5% |
11% |
6% |
636 |
5% |
13% |
8% |
394 |
5% |
15% |
10% |
276 |
5% |
17% |
12% |
206 |
Accordingly, the proposed sample size of 400 participants (200 per group) takes into consideration several important features of the project: the anticipated base rate for thermometer use and the anticipated distributional characteristics of a dichotomous outcome, what research design is feasible given the logistical constraints of conducting an observational study at one location, and FSIS’ future plans for researching food safety education.
Estimation Procedures
We will conduct statistical analyses comparing the differences in handling behavior scores between the control and treatment groups for the four food handling behaviors. A comparative analysis will also be conducted on the samples collected from the designated kitchen sites and food samples to determine whether levels of cross-contamination differed between the two groups and to identify the kitchen sites with the highest levels of contamination.
To maximize response rate and address nonresponse, each participant will receive an honorarium of a $75 gift card and a small gift (food thermometer valued at $5.38) for their participation. In addition, we will send confirmation emails or letters (see Appendix G) with directions and make reminder calls (see Appendix H) to recruited individuals before their scheduled appointment.
On May 22, 2020, the contractor conducted instrument testing with three individuals. The purpose of the instrument testing was to test all study materials and the time allotted for each observation. The instrument testing confirmed the burden estimate of 2 hours for participation in the meal preparation and post-observation interview.
Sheryl Cates is the RTI Project Director and will manage the study. Dr. Jonathan Blitstein of RTI conducted the power analysis and is providing guidance on statistical aspects of the study. Dr. Benjamin Chapman of NCSU will manage the data collection for the observational study and oversee the analysis. Christopher Bernstein, an FSIS employee, will review the results of the observational study.
References
Anderson, J. B., Shuster, T. A., Hansen, K. E., Levy, A. S., & Volk, A. (2004). A camera’s view of consumer food-handling behaviors. Journal of the American Dietetic Association, 104, 186–191.
Bruhn, C. M. (2014). Chicken preparation in the home: An observational study. Food Protection Trends, 34(5), 318–330.
Mazengia, E., Fisk, C., Liao, G., Huang, H., & Meschke, J. (2015). Direct observational study of the risk of cross-contamination during raw poultry handling: Practices in private homes. Food Protection Trends, 35(1), 8–23.
Phang, H. S., & Bruhn, C. M. (2011). Burger preparation: What consumers say and do in the home. Journal of Food Protection, 74(10), 1708–1716.
Scott, E., & Herbold, N. (2010). An in-home video study and questionnaire survey of food preparation, kitchen sanitation, and hand washing practices. Journal of Environmental Health, 72(10), 8–13.
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | joconnell |
| File Modified | 0000-00-00 |
| File Created | 2022-03-07 |