Supporting Statement Part B- revised

Supporting Statement Part B

Supporting Statement (Part B: Statistical Methods) for

EHS-Net Kitchen Manager Certification Study

Change Request #2-09

Submitted under Generic Clearance #0920-0792

Environmental Health Specialists Network (EHS-Net) Program

June, 2011

Laura Green Brown

Centers for Disease Control and Prevention

National Center for Environmental Health

Emergency and Environmental Health Services

Environmental Health Services Branch

4770 Buford Highway, NE F – 60

Atlanta, GA 30341-3724

[email protected]

1. 770-488-4332

Table of Contents

B. Statistical Methods………………………………………………………………………..	3
1. Respondent Universe and Sampling Method…………………………………………	5
2. Procedures for the Collection of Information…………………………………………	3
3. Methods to Maximize Response Rates and Deal with Nonresponse…………………	7
4. Tests of Procedures or Methods to be Undertaken……………………………………	8
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data………………………………………………………………………	8

2. B. Statistical Methods

The results of this EHS-Net data collection will be used to generalize to the population of the EHS-Net catchment area, which includes Rhode Island, New York City, and selected counties in California, New York, Minnesota, and Tennessee. Financially and logistically it is not feasible to collect data from all states. While the number of states included is small, the states are demographically diverse and provide good geographical coverage of the U.S. (northeast, mid-west, south, and west). And within each state, the restaurants are randomly selected. These factors make the restaurants selected in this study representative of other restaurants in the U.S.

B.1. Respondent Universe and Sampling Methods

Respondent Universe

The respondent universe is all restaurants in the EHS-Net catchment area. Restaurant lists will be obtained from the restaurant databases maintained by the EHS-Net sites. CDC will use these restaurant lists to generate the sampling frame used to draw the sample for this study.

Sampling Methods

Sample Size and Power Calculation. We conducted a sample size and power calculation to estimate the total number of restaurants needed to provide the study with sufficient power to detect differences in outcomes between restaurants with and those without certified kitchen managers. Input values for the power calculation were drawn from previous EHS-Net studies and FDA reports that assessed the effects of kitchen manager certification on levels of compliance among a host of foodborne illness risk factors and/or environmental antecedents. The study’s hypothesis is that restaurants with certified kitchen manager(s) will have fewer foodborne illness risk factors than restaurants without a certified kitchen manager. We wished the study to be able to statistically detect at least a 10% reduction in the odds of the presence of a foodborne illness risk factor. That is, the study needs to have enough power to indicate whether an odds ratio of 0.9, or less, is statistically different from an odds ratio of 1, or no difference between groups. For the proposed study, design parameters are set at 5% type I error, 20% type II error, and 80% study power. The total required number of restaurants to sustain a study power of 80% and to detect an odds ratio of at least 0.9 is 383. With six EHS-Net sites participating, each site will need to enroll 64 restaurants. Since this is the minimum number of restaurants required, CDC will ask that each site enroll 80 restaurants in the study.

Sampling Design. This study will utilize a stratified random sampling design where each EHS-Net site serves as its own mutually exclusive stratum. There are two primary reasons for stratifying by EHS-Net site. The first is that food safety regulations vary by jurisdiction. For example, Tennessee state food safety regulations differ from New York state food safety regulations. These regulations can and do greatly influence restaurant food safety practices and policies. Thus, we felt that EHS-Net site/jurisdiction was a critically important factor for stratification. The second reason for stratifying by EHS-Net site is that EHS-Net sites participate in EHS-Net through a cooperative agreement. The nature of this agreement is such that one site cannot be expected to do a disproportionate amount of work in comparison to other sites (because each site receives relatively equal funding amounts). If we did not stratify by EHS-Net site, it is likely that some sites would have to collect data for this study in far more restaurants than other sites would, which is not practically feasible for EHS-Net at this time.

However, stratification on any other variables of interest, such as the number of violations the restaurant received on their last inspection, would require a larger sample size and place an additional burden on the EHS-Net sites. Thus, practical concerns limit our ability to stratify on other variables of interest. However, we will have data on the number of inspection violations and restaurant size for every restaurant in the EHS-Net population; we can use these data to create sampling weights for these variables, which will support some generalizations of findings (see the Sampling Weights section below for more details).

Restaurants will be randomly selected, with equal probability, within their respective EHS-Net site, independent of other sites. This process will give each restaurant on the list the same probability of being selected for study participation. We will sample without replacement, meaning that restaurants will not have the chance to be selected for the study sample more than once. Sampling with replacement is needed for the use of weighted estimates, which we plan to use. However, the use of sampling without replacement is a common and justifiable practice when the target population is considerably larger than the number of samples needed to be drawn from that population, as is the case with this study. In other words, we will sample without replacement but will use analytic methods that support weighted estimates.

The total target population of restaurants from all EHS-Net sites combined constitutes a highly heterogeneous group. To control for such heterogeneity in the total sample, restaurants will be stratified by EHS-Net site so they can be grouped into more homogeneous strata and then sampled within stratum independently. This reduction in heterogeneity of the total sample will lead to reduction in sampling error, which can improve representativeness of the selected sample, and provide weighted estimates (e.g., means) that tend to have less variability than estimates produced from a sample that were drawn using the un-stratified, simple random sampling method.

Sampling weights. This study was designed to support the generalization of findings from the sampled restaurants to all restaurants in the EHS-Net catchment area. To achieve generalizability, sampling weights will be used to indicate the degree of representativeness that each sampled restaurant has in relation to the total target population. Representation in this study is based on two distinguishing restaurant characteristics: restaurant size as indicated by seating capacity and most recent inspection status (number of critical violations found on last inspection). These data will be obtained from restaurant databases maintained by the EHS-Net sites and provided to CDC in their population restaurant lists. CDC will use these data to calculate the appropriate sampling weights for the sampled restaurants within each EHS-Net site.

Restaurants that refuse to participate will be considered the same as participating restaurants, unless data indicate otherwise. When possible, demographic information will be collected on restaurants that refused to participate. These data will be used to ascertain whether there are systematic differences between participating restaurants and those that refused to participate so that sampling weights can be adjusted accordingly. In instances where there are systematic differences between refused and participating restaurants, refused restaurants will be excluded from the total target population. All ineligibles will be excluded from the total target population. Careful considerations will be taken in calculating sampling weights as they can impact the overall population parameter estimates.

Response Rate. The two most recent EHS-Net studies that used methods similar to the proposed study yielded response rates around 70%. We expect a similar response rate for the proposed study.

B.2. Procedures for the Collection of Information

Sampling and Recruitment

As indicated earlier, each EHS-Net site will provide CDC with a list of all restaurants in the catchment area. This list will serve as the sampling frame for the site. CDC will use a random number generator in SAS 9.2 to produce a random sample of restaurants from this restaurant list for each site. As we expect some restaurants will refuse to participate and some will be ineligible to participate, we will select more than the needed number of restaurants--100 restaurants for each site. Once they receive their sample list from CDC, personnel in each site will contact establishments by telephone to recruit their participation in the study (see Appendix 9 for recruitment script). If the manager is willing to participate, the EHS-Net specialist will arrange a mutually convenient time to conduct the data collection.

In instances where an EHS-Net site is unable to recruit 80 restaurants from the first list of 100 restaurants, CDC will randomly select another group of 50 restaurants for the site to use to recruit additional respondents. Recruitment will be considered complete once 80 restaurants are selected. Recruitment will be done via the telephone and a log of each incident of contacts with the restaurants will be kept in order to document rates and reasons for refusal and/or ineligibility.

Data Collection

Data will be collected in the restaurants by the EHS-Net environmental health specialists. For the manager interview portion of the study, the EHS-Net specialist will obtain verbal informed consent and then conduct a face-to-face semi-structured interview with a manager with authority over the kitchen. This interview will include questions on establishment characteristics (number of employees, food safety training provided) and manager food safety attitudes. The manager will also complete a short written survey on their food safety knowledge. These activities will take about twenty minutes to complete. To increase cooperation, we will allow restaurant management to select the manager to be interviewed. Criteria for selection will be that the manager has authority over the kitchen and can speak English well enough to complete the interview in English.

For the worker interview portion of the study, the EHS-Net specialist will obtain verbal informed consent from and conduct a face-to-face semi-structured interview with a food worker. The interview will include questions on food safety knowledge and practices and will take about 10 minutes to complete. Data collectors will ask the manager to help identify a food worker who can spend about ten minutes being interviewed for the study. Criteria for selection will be that the worker handles food and can speak English well enough to complete the interview in English.

For the observation portion of the survey, the EHS-Net specialist will observe activity in the kitchen and observe food handling practices (e.g., cooking, hot and cold holding of food, cooling, prevention of cross contamination). This observation will take about an hour to complete. Thus, data collection will take about an hour and a half per restaurant. Although this may seem like a relatively long time for data collection, managers and workers are only engaged with the data collector for a relatively short time (20 and 10 minutes, respectively) because the kitchen observation does not actively involve them. We have conducted several studies using methods and data collection durations similar to this one, and have had response rates in the 70% range (Delea et al, 2010; Kirkland et al., 2009; Marcus et al, 2010).

Quality Control Procedures

The data collectors are experienced and knowledgeable in environmental health and food safety and will have received training from CDC on data collection for this study. Data entry will be double-checked by the EHS-Net administrator in each EHS-Net site.

Potential Biases

Managers’ concerns about the food safety of their restaurants may result in a lower rate of study participation among restaurants with worse food safety practices compared to restaurants with better food safety practices. We have conducted studies using methods similar to those used in this study in the past, and these studies have found a wide range of food safety practices, including poor ones (Delea et al, 2010; Green et al., 2006; Kirkland et al., 2009; Lee et al., 2004; Marcus et al, 2010). Nevertheless, in an attempt to assess the impact of this non-response bias on our data, we will compare data on inspection status for restaurants that participate in our study and restaurants that do not participate in our study. Although this inspection measure is not a perfect measure of food safety, this analysis may give us a better understanding of how food safety practices may differ between participating and non-participating restaurants.

The observation data collected for this study may be influenced by reactivity on the part of those observed. In other words, those observed may not respond naturally when they know they are being observed. However, observation data on behavior is considered to be more accurate than self-reported data, particularly when measures are taken to limit the observers’ influence on the observed (Leary, 2004). In this study, those measures include the following: 1) observers will attempt to remain relatively unobtrusive during the observation, and 2) when possible, the precise details on which aspects of behavior are being recorded will not be provided to those being observed. Additionally, the observation in this study is about 50 minutes, and research suggests that longer observations allow time for the observed to revert to more natural behavior over the course of the observation (Gall, Borg, & Gall, 1996).

The interview data collected for this study may be influenced by the social desirability bias- the tendency for people to report greater levels of socially desirable behavior (such as safe food preparation practices) than they actually engage in, or to report their best behavior rather than their typical or worst behavior. Although it is difficult to eliminate this bias altogether, it can be limited by ensuring respondents that the information they report will be anonymous, which we will do (Leary, 2004).

The fact that restaurant management will help select the manager and food worker to be interviewed may also introduce bias, as management may select managers and workers they believe are knowledgeable about food safety. However, we feel this selection technique is necessary to increase management and food worker participation.

We will only interview managers and workers that speak English well enough to be interviewed in English. The use of this criterion may introduce bias, as non-English speakers may have different food safety knowledge and practices than English speakers, but the resources are not available to include non-English speaking workers in the study. Currently, one of our EHS-Net sites is conducting a study in which food safety practice data will be collected from both Spanish-only speaking restaurant managers and workers and English-speaking restaurant managers and workers; the results from this study may give us a better understanding of how data from EHS-Net restaurant studies may be impacted by the restriction of participants to English speakers.

Any presentation of data from this study will acknowledge these potential biases and include a discussion of how they may impact data interpretation.

B.3. Methods to Maximize Response Rates and Deal with Nonresponse

We will engage in several activities designed to maximize response rates. First, all recruiters will receive training on the recruiting process. Second, multiple attempts will be made to contact potential respondents. Specifically, recruiters will make 10 attempts over 5 days to get a participation response from establishments they have not been able to contact, and 5 attempts over 5 days to get a participation response from establishments that have not provided a response (e.g., ‘call back later’). Third, recruiting scripts will emphasize two issues that have been shown to increase response rates—the anonymous nature of the data collection and the importance of the respondents’ participation in the study. The most recent EHS-Net data collections, conducted in food service establishments, used these techniques and yielded response rates around 70% (Kirkland et al., 2009; Sumner et al, in press).

We will also attempt to determine if restaurants participating in this study differ systematically from non-participating restaurants. To do this, we will compare participating and non-participating restaurants on two important restaurant characteristics- restaurant size and inspection status. If significant differences are found in these characteristics, any presentation of the data from this study will include a discussion of these differences and how they may impact data interpretation.

B.4. Test of Procedures or Methods to be Undertaken

All data collection materials were reviewed and evaluated by EHS-Net specialists familiar with collecting data in restaurants. Additionally, all data collection materials were evaluated in pretests with 9 restaurants. The pretests were used to improve the data collection materials.

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following people were primarily responsible for the design, including the statistical aspects, of the data collection and will be primarily responsible for data analysis.

Laura Green Brown, Ph.D.

Behavioral Scientist

Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

770-488-4332

Brenda Le, M.S.

Statistician

Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

770-488-3756

Personnel in the 6 EHS-Net sites will be responsible for data collection. These sites are listed below.

California Department of Health

Minnesota Department of Health

New York Department of Health

New York City Department of Health and Mental Hygiene

Rhode Island Department of Health

Tennessee Department of Health

References

Gall, M., W. Borg, and J. Gall. 1996. Educational Research. Longman. White Plains, NY.

Green, L., C. Selman, V. Radke, D. Ripley, J. Mack, D. Reimann, T. Stigger, M. Motsinger and L. Bushnell. 2006. Food worker hand washing practices: An observation study. J. Food Protect. 69:2417-2423.

Delea, K., K., Everstine and E. Coleman. 2010. Restaurant leafy green handling practices. Manuscript in preparation.

Marcus, R., and C., Monteilh. 2010. Restaurant chicken handling practices. Manuscript in preparation.

Kirkland, E., L. Green, C. Stone, D. Reimann, D. Nicholas, R. Mason, R. Frick, S. Coleman, L. Bushnell, H. Blade, V. Radke, C. Selman, and the EHS-Net Working Group. 2009. Tomato handling practices in restaurants. J. Food Protect. 72:1692–1698.

Leary, M. 2004. Introduction to behavioral science research methods. Allyn and Bacon. Boston, MA.

Lee, R., M. Beatty, A. Bogard, M. Esko, R. Anglulo, C. Selman, and the EHS-Net Working Group. 2004. Prevalence of high-risk egg-preparation practices in restaurants that prepare breakfast egg entre´es: An EHS-Net study. J Food Protect. 67:1444–1450.

Sumner, S., L. Brown, R. Frick, C. Stone, L. Carpenter, L. Bushnell, D. Nicholas, J. Mack, H. Blade, M. Tobin-D’Angelo, K. Everstine, and EHS-Net. In press. Factors associated with food workers working while experiencing vomiting or diarrhea. J. Food Protect.

9