REVISED Supporting Statement B

Supporting Statement Part B

EHS-Net Listeria Retail Deli Study

EHS-NET Generic Information Collection Request

OMB No. 0920-0792

Supporting Statement - B

Submitted: August, 2012

Program Official:

Laura Green Brown, Ph.D.

Behavioral Scientist

Centers for Disease Control and Prevention

National Center for Environmental Health

Emergency and Environmental Health Services

Environmental Health Services Branch

4770 Buford Highway, NE F – 60

Atlanta, GA 30341-3724

770-488-4332 (Phone)

770-488-7310 (Fax)

E-mail: [email protected]

9. Table of Contents

B. Study Methods
1. Respondent Universe and Sampling Method
2. Procedures for the Collection of Information
3. Methods to Maximize Response Rates and Deal with Nonresponse
4. Tests of Procedures or Methods to be Undertaken
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

10. Study Methods

1. Respondent Universe and Sampling Methods

Respondent Universe

EHS-Net is comprised of retail food establishments in selected geographical areas in California, Minnesota, New York City, New York State, Rhode Island, and Tennessee.  While the number of areas included in EHS-Net is small, they are demographically diverse and provide good geographical coverage of the U.S. (northeast, mid-west, south, and west). 

The respondent universe is all retail delis in the EHS-Net catchment area. A list of all retail establishments in the U.S. will be obtained from the 2008 and 2009 Homeland Security Infrastructure Program (HSIP) databases (A description of HSIP can be found here: https://www.hifldwg.org/; however, the database we will use is restricted access). CDC will use these lists of retail establishments to generate the sampling frame that will be used to randomly draw the samples of retail delis for this study.

Sampling Methods

The goal of the sampling methods outlined here is to maximize the extent to which we are able to characterize the extent to which work practices in deli establishments might lead to cross contamination. By enrolling a random sample of 50 retail delis from each of the six EHS_NET sites, we expect variation in the types of delis selected to characterize the potential influence of such factors as management models (e.g., corporate, mom and pop), volume of meats processes, sizes of work spaces, variation in products prepared behind the counter, variation in responsibilities of workers, age of deli equipment, types of refrigeration practices, etc. Without knowing the influence of these factors, it is not possible to stratify the sample in each location by the key variables in advance, thereby limiting our ability to conclude that we have a representative sample of the frequency with which delis in EHS-Net sites exhibit behaviors that might allow cross contamination. Furthermore, as discussed in Section 2 (page 6-8) there are a number of reasons that we might expect some response bias, which would suggest that we might miss some of the worst behaviors. However, the geographic and demographic variability across these sites, as well as the wide variety of types of delis present in each EHS-Net sites, suggests that CDC should be able to use data collected from these studies to draw conclusions about relationships that are likely relevant to establishments in many parts of the U.S. Thus, the information from this collection will be helpful in identifying a wide range of behaviors/characteristics as well as behaviors/characteristics that are common.

The primary limitations of the data collection in the context of USDA’s and CDC’s stated uses include the extent to which: a) the percentages of specific behaviors/characteristics observed in one geographic area (e.g., see table templates 16.2.B and C in Part A) might be generalized to another area without adjusting the factors that influence their distribution; and b) the distributions of such factors as age of equipment across the six EHS-Net areas are necessarily reflective of a the country as a whole. As such, USDA will caveat its model findings and CDC will caveat its national recommendations consistent with these limitation.

Sample Size. Each EHS-Net site will be required to enroll at least 50 retail delis (Table B.1). Since there are no previously published (population) studies that have examined workers’ behaviors and food safety policies and practices in retail delis, at this time, we are unable to determine whether this sample size will be able to support at least an 80% study power to detect statistical differences between study groups. Thus, data on expected prevalence of certain behaviors and food safety policies and practices between chain and independent retail delis are not available as inputs for proper calculation of study sample size and power. Enrollment of 50 retail delis per EHS-Net site, totaling 300 delis for the entire study, is a reasonable sample size and follows the precedent of previous EHS-Net studies (Brown et al., 2012; Delea et al., 2010; Green et al., 2006; Kirkland et al., 2009; Marcus et al., 2010). Data collected from this study will provide the necessary information for sample size and power calculation of future studies.

Table B.1

Strata (EHS-Net Sites)	Entity	Number of Entities
California	Retail Deli	50
Minnesota	Retail Deli	50
New York	Retail Deli	50
New York City	Retail Deli	50
Rhode Island	Retail Deli	50
Tennessee	Retail Deli	50

Sampling Strategy. A cross-sectional design using a stratified random sampling plan in which each EHS-Net site serves as its own mutually exclusive stratum. There are two primary reasons for stratifying by EHS-Net site. The first is that food safety regulations vary by jurisdiction. For example, Tennessee state food safety regulations differ from New York state food safety regulations. These regulations can and do greatly influence retail deli’s food safety practices and policies. EHS-Net site/jurisdiction, therefore, poses as the largest source of variability from a study design perspective. Thus, it is a critically important factor for stratification. The second reason for stratifying by EHS-Net site only is due to practical concerns that limit our ability to stratify on other variables of interest. EHS-Net sites participate in EHS-Net through a cooperative agreement. See Table B.2 for EHS-Net sites’ cooperative agreement numbers. The nature of this agreement is such that one site cannot be expected to do a disproportionate amount of work in comparison to other sites (because each site receives relatively equal funding amounts). If we did not stratify by EHS-Net site but by some other factor such as ownership (independently owned or belonging to a corporate chain), it is likely that some sites would have to carry a greater burden than other sites in term of recruiting and collecting data in a larger number of retail delis. However, we will be collecting data on these factors of interest and will account for their heterogeneity through statistical modeling. Finally, the need for each site to share an equal burden in data collection is the reason why a fixed-sample allocation method was used for each site (50 establishments per site), instead of a proportionate-sample allocation.

Table B.2

EHS-Net Sites	CDC-NCEH Cooperative Agreement Numbers
California	EH000704
Minnesota	EH000698
New York	EH000701
New York City	EH000692
Rhode Island	EH000700
Tennessee	EH000706

Retail delis will be randomly selected, with equal probability, within their respective EHS-Net site, independent of other sites. This process will give each retail deli in a particular sampling frame the same probability of being selected for study participation. There are three reasons for employing this sampling strategy: reducing sampling error, maintaining equal representation by site, and ensuring generalizability. First, as stated previously, the total target population of retail delis from all EHS-Net sites combined constitutes a highly heterogeneous group. To control for such heterogeneity in the total sample, retail delis will be stratified by EHS-Net site so they can be grouped into more homogeneous strata and then sampled within stratum independently. This reduction in heterogeneity of the total sample will lead to reduction in sampling error, which can improve representativeness of the selected sample and provide estimates (e.g., means) that tend to have less variability than estimates produced from samples that were drawn using the un-stratified, simple random sampling method. Second, with equal allocation of samples (50 retail delis per site), each EHS-Net site will have equal representation in the parameter estimates of the combined sample. An additional benefit is that even sites with small sampling frames will have sufficient data points to support their site-specific analyses. Third, by ensuring that the sampling of study units is done by an entity (CDC) separate from the data collectors (EHS-Net sites) and employing a random selection method, we are able to minimize the potential for selection bias. Parameter estimates or study findings obtained from an unbiased study sample could be generalized to the entire EHS-Net target population.

Study Design. A cross-sectional study design will be utilized.

Response Rate. The most recent EHS-Net study that used methods similar to the proposed study yielded a response rate approaching 80% (Brown et al., 2012). We expect a similar response rate for the proposed study.

2. Procedures for the Collection of Information

Sampling and Recruitment

As previously indicated, a list of all of the retail establishments in the U.S. will be obtained from the 2008 and 2009 HSIP databases. That list will then be refined to include only those retail establishments in the catchment area of each EHS-Net site. These lists of retail establishments will serve as the sampling frames for each EHS-Net site. CDC will use a random sampling algorithm in SAS 9.3 to produce a random sample of retail establishments from the sample frame for each EHS-Net site. As we expect some retail delis will refuse to participate and some will be ineligible to participate, we will select more than the needed number of retail establishments--75 retail establishments for each EHS-Net site. Once they receive their sample list from CDC, personnel in each EHS-Net site will contact establishments by telephone to recruit their participation in the study (see Attachment 10 for recruitment script). If the manager is willing to participate, the EHS-Net specialist will arrange a mutually convenient time to conduct the data collection.

In instances where an EHS-Net site is unable to recruit 50 retail establishments from the first list of 75 establishments, CDC will randomly select another group of 50 establishments for the EHS-Net site to use to recruit additional respondents. Recruitment will be considered complete once 50 establishments are selected. Recruitment will be done via the telephone and a log of each incident of contacts with the retail delis will be kept in order to document rates and reasons for refusal and/or ineligibility.

Data Collection

Data will be collected in the retail delis by the EHS-Net environmental health specialists. For the manager interview, the EHS-Net specialist will obtain verbal informed consent and then conduct a face-to-face semi-structured interview with a manager that has authority over the deli. This interview will include questions on establishment demographics, food safety policies, and manager food safety training and certification. The manager will also complete a short written survey on their food safety knowledge. The interview and survey will take about thirty-two minutes to complete. Criteria for selection will be that the manager has authority over the retail deli and can speak English well enough to complete the interview in English.

The data collector will ask the deli manager to help recruit a deli worker to participate in this study. For the worker interview, the EHS-Net specialist will obtain verbal informed consent before conducting a face-to-face semi-structured interview with a deli worker. The interview will include questions on policies and usage of the deli slicers and the deli worker’s food safety knowledge and beliefs. The deli worker interview will take about ten minutes to complete. Criteria for selection will be that the worker handles food, operates and cleans the deli slicers, and can speak English well enough to complete the interview in English.

For this study, the EHS-Net specialist will also conduct two types of observations, structured observations of the facility and equipment and notational observations of the sequences of work-related actions of one to three deli workers (depending on deli workers’ availabilities). Specifically, during the structured observation, the data collector will observe the retail deli to answer specific questions related to equipment, food storage and handling practices, etc. For the notational observation, the data collector will observe and notate every action performed by a deli worker once he or she begins a work-related task (e.g., slicing deli meat for a customer). The structured observations will take approximately 30 minutes; the notational observations will take from 30 to 60 minutes for each deli worker. Both of these observation methods will not require direct interactions with the deli workers, and therefore, will not interfere with their job duties nor create extra burden on the workers.

Data collection for this study will take about an hour and forty-five minutes per retail deli. Although this may seem like a relatively long time for data collection, managers and workers are only engaged with the data collector for a relatively short time, 32 and 10 minutes, respectively. Neither the structured or notational observation requires active involvement of the deli manager or workers. We have conducted several studies using methods and data collection durations similar to this one, and have had response rates approaching 80% (Delea et al., 2010; Green et al., 2006; Kirkland et al., 2009; Marcus et al., 2010).

Attachment 12 contains the study’s Data Collection Instruction Manual. This manual was designed solely for use by the study data collectors and contains detailed instructions for them on how to collect the data for this study. This manual is a working document, and may be revised several times throughout the data collection process to address data collection issues.

Quality Control Procedures

The data collectors are experienced and knowledgeable in environmental health and food safety and will have received training from CDC on data collection for this study. The EHS-Net administrator in each EHS-Net site and CDC staff will perform quality assurance procedures to check for data entry errors.

Potential Biases

Retail managers’ concerns about the food safety practices and policies of their establishments may result in selection bias- a lower rate of study participation among retail delis with worse food safety practices compared to retail delis with better food safety practices. We have conducted studies using methods similar to those used in this study in the past, and these studies have found a wide range of food safety practices, including poor ones (Delea et al., 2010; Green et al., 2006; Kirkland et al., 2009; Lee et al., 2004; Marcus et al., 2010). While the potential for selection bias is there, these studies indicate that biases, if they exist, may be minimal. Plus, the study protocol incorporates procedures to minimize the potential for and to detect any indication of selection bias. For example, EHS-Net staff will be trained in the recruitment process in order to keep non-response rate as low as possible, which will help minimize selection bias. Additionally, demographic information on non-respondents will be compared with information on respondents to assess for any evidence of selection bias.

The act of observing the deli worker may influence their performance and behavior on work-related tasks. In other words, the participants may not respond naturally when they know they are being observed. However, observation data on behavior is considered to be more accurate than self-reported data, particularly when measures are taken to limit the observers’ influence on those observed (Leary, 2004). In this study, those measures include the following: 1) observers will attempt to remain relatively unobtrusive during the observation, 2) when possible, the precise details on which aspects of behavior are being recorded will not be provided to those being observed, and 3) data collected during the first fifteen minutes of the observation will be discarded, as this period of observation will serve as an acclimation period for the deli worker and data collector. At minimum, the observations in this study are about 60 minutes, and research suggests that longer observations allow time for the observed to revert to more natural behavior over the course of the observation (Gall, Borg, and Gall, 1996).

The interview data collected for this study may be influenced by social desirability bias- the tendency for people to report greater levels of socially desirable behavior (such as safe food preparation practices) than they actually engage in, or to report their best behavior rather than their typical or worst behavior. Although it is difficult to eliminate this bias altogether, it can be limited by ensuring respondents that the information they report will be anonymous, which we will do (Leary, 2004).

The fact that deli managers will help select the deli worker to be interviewed may also introduce bias, as management may select deli workers that they believe are knowledgeable about food safety. However, we feel this selection technique is necessary to increase management and deli worker participation.

We will only interview managers and workers that speak English well enough to be interviewed in English. The use of this criterion may introduce bias, as non-English speakers may have different food safety knowledge and practices than English speakers, but the resources are not available to include non-English speaking workers in the study. Currently, one of our EHS-Net sites is conducting a study in which food safety practice data will be collected from both Spanish-only speaking food service managers and workers and English-speaking food service managers and workers; the results from this study may give us a better understanding of how data from EHS-Net food service studies may be impacted by the restriction of participants to English speakers.

Any presentation of data from this study will acknowledge these potential biases and include a discussion of how they impact data interpretation.

3. Methods to Maximize Response Rates and Deal with Nonresponse

We will engage in several activities designed to maximize response rates. First, all recruiters will receive training on the recruiting process. Second, multiple attempts will be made to contact potential respondents. Specifically, recruiters will make 10 attempts over 5 days to get a participation response from establishments they have not been able to contact, and 5 attempts over 5 days to get a participation response from establishments that have not provided a response (e.g., ‘call back later’). Third, recruiting scripts will emphasize two issues that have been shown to increase response rates—the anonymous nature of the data collection and the importance of the respondents’ participation in the study. The most recent EHS-Net data collections, conducted in food service establishments, used these techniques and their response rates approached 80% (Kirkland et al., 2009; Sumner et al., 2011).

We will also attempt to determine if retail delis participating in this study differ systematically from non-participating retail delis. To do this, we will compare the ownership (i.e., whether a deli is independent or belongs to a corporate chain) characteristics between respondents and non-respondents. If significant differences are found, any presentation of the data from this study will include a discussion of these differences and how they may impact data interpretation.

4. Test of Procedures or Methods to be Undertaken

The data collection materials and methods were based on those used in other previous, successful EHS-Net studies (Brown, Le & Ripley, 2012; Green et al., 2006; Kirkland et al., 2009; Lee et al., 2004; Sumner et al., 2011). All data collection materials were reviewed and evaluated by key EHS-Net specialists whom are experienced with collecting data for EHS-Net studies. They were also reviewed by CDC EHS-Net personnel with extensive experience in developing and conducting EHS-Net studies. Additionally, all data collection materials will be evaluated in pilot tests with 9 retail delis while we await OMB approval. CDC will provide the EHS-Net sites with a list of randomly selected retail delis for them to recruit for pilot participation. Results from the pilot test will be used to refine the data collection instructions. Given that we are experienced in collecting data from retail food establishments with these types of instruments and methods (this is will be the eleventh study we have conducted in retail food establishments using similar data collection instruments and methods), we are confident that the study is designed well and do not anticipate the need to make changes to the data collection instruments. If we do need to make changes as a result of the pilot, we anticipate that they will be minor. OMB will be notified of any changes to the data collection instruments through the non-substantive change request.

5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following people were primarily responsible for the design, including the statistical aspects, of the data collection and will be primarily responsible for data analysis. Brenda Le is the primary contact for statistical aspects and data collection.

Brenda Le, M.S.

Statistician

Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

770-488-3756

Laura Green Brown, Ph.D.

Behavioral Scientist

Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

770-488-4332

Denita Williams, Ph.D.

Toxicologist

Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

770-448-0704

Personnel in the 6 EHS-Net sites will be responsible for data collection (See table below). In some cases, environmental health specialists from non-EHS-Net sites assist with data collection; these personnel are not included in this table.

Site	Number of Personnel
California Department of Health	1 full-time
Minnesota Department of Health	1 full-time 1 part-time
New York Department of Health	1 full-time 1 part-time
New York City Department of Health and Mental Hygiene	1 full-time
Rhode Island Department of Health	1 full-time
Tennessee Department of Health	1 full-time 1 part-time

References for Part B

Brown, L., B. Le, and D. Ripley. 2012. Restaurant characteristics associated with improper food cooling practices. Poster presented at the International Conference on Emerging Infectious Diseases. Atlanta, GA.

Delea, K., K. Everstine, and E. Coleman. 2010. Restaurant leafy green handling practices. Manuscript in preparation.

Gall, M., W. Borg, and J. Gall. 1996. Educational Research. Longman. White Plains, NY.

Green, L., C. Selman, V. Radke, D. Ripley, J. Mack, D. Reimann, T. Stigger, M. Motsinger, and L. Bushnell. 2006. Food worker hand washing practices: An observation study. J. Food Protect. 69:2417-2423.

Kirkland, E., L. Green, C. Stone, D. Reimann, D. Nicholas, R. Mason, R. Frick, S. Coleman, L. Bushnell, H. Blade, V. Radke, C. Selman, and the EHS-Net Working Group. 2009. Tomato handling practices in restaurants. J. Food Protect. 72:1692–1698.

Leary, M. 2004. Introduction to behavioral science research methods. Allyn and Bacon. Boston, MA.

Lee, R., M. Beatty, A. Bogard, M. Esko, R. Anglulo, C. Selman, and the EHS-Net Working Group. 2004. Prevalence of high-risk egg-preparation practices in restaurants that prepare breakfast egg entre: An EHS-Net study. J Food Protect. 67:1444-1450.

Marcus, R. and C., Monteilh. 2010. Restaurant chicken handling practices. Manuscript in preparation.

Sumner, S., L. Brown, R. Frick, C. Stone, L. Carpenter, L. Bushnell, D. Nicholas, J. Mack, H. Blade, M. Tobin-D’Angelo, K. Everstine, and EHS-Net. 2011. Factors associated with food workers working while experiencing vomiting or diarrhea. J. Food Protect. 74:215–220.

11