SSB EHS-Net Clean

Supporting Statement Part B

Environmental Health Specialists Network (EHS-Net)

Program Generic Package

Revision

OMB No. 0920-0792

OMB Exp. Date: 9/30/2018

Supporting Statement - B

Submitted: June 2018

1. Program Official:

Beth Wittry, M.P.H., R.S.

Environmental Health Officer

Centers for Disease Control and Prevention

National Center for Environmental Health

Environmental Health Services and Practice

2. Water, Food, Environmental Health Services Branch

3. 4770 Buford Highway, NE F – 58

4. Atlanta, GA 30341-3724

5. 404-498-1228 (Phone)

6. 770-488-3635 (Fax)

7. E-mail: [email protected]

Table of Contents

9. B.1. Respondent Universe and Sampling Methods 3

B.2. Procedures for the Collection of Information 5

B.3. Methods to Maximize Response Rates and Deal with No Response 6

B.4. Test of Procedures or Methods to be Undertaken 7

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 11

References 13

List of Attachments

Attachment 1 – Authorizing Legislation

Attachment 2 – 60 Day Federal Register Notice

Attachment 3a – NCEH-ATSDR Research Determination Form

Attachment 3b- Privacy Impact Assessment

Attachment 4 – EHS-Net Publications

Attachment 5 – EHS-Net Manager Recruiting Script

Attachment 6 – EHS-Net Manager Informed Consent

Attachment 7 – EHS-Net Manager Interview

Attachment 8 – EHS-Net Food Worker Recruiting Screener, Consent and Interview

Attachment 9 – EHS-Net Restaurant Observation

Part B. Collections of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

EHS-Net is a collaborative project of the CDC, FDA, USDA, and eight state and local public health departments (currently comprised of: California; Minnesota; New York City; New York; Rhode Island; Tennessee; Southern Nevada Health District, NV; and Harris County, TX.) The respondent universe is comprised of retail food establishments in selected geographical areas within the EHS-Net catchment areas. While the number of areas included in EHS-Net is small, they are demographically diverse and provide good geographical coverage of the U.S. (northeast, mid-west, south, and west). When the sampling methods outlined here for ensuring a representative sample in the current study are used, the results of the collection covered by this OMB package can be used to generalize to the population of retail food establishments in the given EHS-Net site(s).

The respondent universe is all retail food establishments (hereafter referred to as restaurants) in the EHS-Net catchment area. Restaurant lists will be obtained from the restaurant databases maintained by the EHS-Net sites. CDC will use these restaurant lists to generate the sampling frame used to draw the sample for this study.

Each EHS-Net site will typically enroll 50 restaurants per study (Table 1.1.). Enrollment of 50 restaurants per EHS-Net site, totaling 400 restaurants for the entire study, is a reasonable sample size and follows the precedent of previous EHS-Net studies (Green et al., 2006; Kirkland et al., 2009; Sumner et al., 2011). Experience from prior EHS-Net studies also indicates that a sample size of 400 should be sufficient for the analytic purposes, since the analytic parameters are not likely to be considered rare (in distribution) events. Data collected from projects performed under these collections will provide the necessary information for sample size and power calculations for future studies.

1.1 Number of restaurants per current EHS-Net site

Strata (EHS-Net Sites)	Entity	Number of Entities
California	Restaurants	50
Minnesota	Restaurants	50
New York	Restaurants	50
New York City	Restaurants	50
Rhode Island	Restaurants	50
Tennessee	Restaurants	50
Southern Nevada Health District	Restaurants	50
Harris County, Texas	Restaurants	50

A sample size calculation was conducted to estimate the total number of retail food establishments necessary to provide sufficient power to detect group differences in outcomes of interest. The design parameters were set to detect an odds ratio of at least 2.0 or greater (a medium –small effect size) with a .05 Type I error rate, power of .90, and based on the assumption of equivalent group sizes. Calculations established that a total of 372 retail food establishments would need to participate to achieve the desired parameters. With eight EHS-Net sites participating in data collection, each site will need to enroll 47 establishments to meet this requirement. In order to detect a greater odds ratio and establish a stronger power we have increased the total sample size to 400 retail food establishments, with each EHS-Net site enrolling 50 establishments. In addition, this increase will allow for additional statistical designs.

The design is cross-sectional and uses a stratified random sampling plan in which each EHS-Net site serves as its own mutually exclusive stratum. There are two primary reasons for stratifying by EHS-Net site. The first is that food safety regulations vary by jurisdiction. For example, Tennessee state food safety regulations differ from New York state food safety regulations. These regulations can and do greatly influence restaurants’ food safety practices and policies. EHS-Net site/jurisdiction, therefore, poses the largest source of variability from a study design perspective. Thus, it is a critically important factor for stratification. The second reason for stratifying by EHS-Net site is due to practical concerns that limit our ability to stratify on other variables of interest. EHS-Net sites participate in EHS-Net through a cooperative agreement (EH15-001.) The nature of this agreement is such that one site cannot be expected to do a disproportionate amount of work in comparison to other sites (because each site receives relatively equal funding amounts). If we did not stratify by EHS-Net site but by some other factor such as ownership (independently owned or belonging to a corporate regional chain), it is likely that some sites would have to carry a greater burden than other sites in term of recruiting and collecting data in a larger number of restaurants. However, we will be collecting data on these factors of interest and will account for their heterogeneity through statistical modeling. Finally, the need for each site to share an equal burden in data collection is the reason why a fixed-sample allocation method is used for each site (50 establishments per site), instead of a proportionate-sample allocation.

Restaurants will be randomly selected, with equal probability, within their respective EHS-Net site, independent of other sites. This process will give each restaurant in a particular sampling frame the same probability of being selected for study participation. There are three reasons for employing this sampling strategy: reducing sampling error, maintaining equal representation by site, and ensuring generalizability. First, as stated previously, the total target population of restaurants from all EHS-Net sites combined constitutes a highly heterogeneous group. To control for such heterogeneity in the total sample, restaurants will be stratified by EHS-Net site so they can be grouped into more homogeneous strata and then sampled within stratum independently. This reduction in heterogeneity of the total sample will lead to reduction in sampling error, which can improve representativeness of the selected sample and provide estimates (e.g., means) that tend to have less variability than estimates produced from samples that were drawn using the un-stratified, simple random sampling method. Second, with equal allocation of samples (50 restaurants per site), each EHS-Net site will have equal representation in the parameter estimates of the combined sample. An additional benefit is that even sites with small sampling frames will have sufficient data points to support their site-specific analyses. Third, by ensuring that the sampling of restaurants is done by an entity (CDC) separate from the data collectors (EHS-Net sites) and employing a random selection method, we are able to minimize the potential for selection bias. Parameter estimates or study findings obtained from an unbiased study sample could be generalized to the entire EHS-Net target population.

The average response rate across EHS-Net studies that used methods similar to the proposed study is 45% (Brown et al., 2016; Brown et al., 2014; Radke et al., 2016). We expect a similar response rate for the proposed studies performed under this collection. Although this response rate is lower than the 80% response rate mentioned in the OMB guidelines, is a rate that allows us to feel comfortable with our planned analysis (Frey & Oishi, 1995).

2. Procedures for the Collection of Information

As indicated earlier, each EHS-Net site will provide CDC with a list of all restaurants in their catchment area. This list will serve as the sampling frame for the site. CDC will use a random number generator in the SAS statistical program to produce a random sample of restaurants from this restaurant list for each site. As we expect some restaurants will refuse to participate and some will be ineligible to participate, we will select more than the needed number of restaurants--100 restaurants for each site. Once they receive their sample list from CDC, personnel in each site will contact restaurants by telephone to recruit their participation in the study. If the manager is willing to participate, the EHS-Net specialist will arrange a mutually convenient time to conduct the data collection.

In instances where an EHS-Net site is unable to recruit 50 restaurants from the first list of 100 restaurants, CDC will randomly select another group of 50 restaurants for the site to use to recruit additional respondents. Recruitment will be considered complete once data are collected in 50 restaurants. EHS-Net sites will recruit via the telephone and will keep a log of each contact with the restaurants to document participation rates and reasons for refusal and/or ineligibility.

CDC will not know which restaurants on the sample lists participated in the study, and thus will not be able to link restaurant names with any study data. The restaurant identifying information will be maintained by the respective EHS-Net site to facilitate the site visit, but will be destroyed when it is no longer needed. Additionally, on all forms only the specific coded restaurant identifier will be used to minimize a risk of someone inadvertently seeing a completed form and being able to associate it with a specific restaurant.

Data will be collected in the restaurants by the EHS-Net environmental health specialists. For the manager interview portion of the study, the EHS-Net specialist will obtain verbal informed consent and then conduct a face-to-face interview with a manager who has authority over the kitchen and can speak English well enough to complete the interview in English. This interview will include questions on restaurant characteristics, policies, and food safety-related practices. This will take about thirty minutes to complete.

For the food worker survey portion of the study, the data collector will obtain verbal informed consent and then either administer a written/electronic survey or conduct an interview as appropriate for the specific study. The intent of the survey/interview will be to gather information on food worker beliefs, knowledge, and practices related to food safety along with worker demographics.

For the restaurant observation portion, the data collector will observe the kitchen and note the food-safety related infrastructure and the implementation of specific food safety practices.

The data collectors are experienced and knowledgeable in environmental health and food safety and will have received training from CDC on data collection for the studies done under this collection. The EHS-Net administrator in each EHS-Net site and CDC staff will perform quality assurance procedures to check for data entry errors.

Managers’ concerns about the safety practices of their restaurants may result in selection bias- a lower rate of study participation among restaurants with worse or non-existent safety practices compared to restaurants with better safety practices. We have conducted studies using methods similar to those proposed for use in this study in the past, and these studies have found a wide range of food safety practices, including poor ones (Bogard, Fuller, Radke, Selman, & Smith, 2013; Brown, Khargonekar, & Bushnell, 2013; Brown et al., 2012; Coleman, Delea, Everstine, Reimann, & Ripley, 2013; Green et al., 2006; Kirkland et al., 2009; Lee et al., 2004; Sumner et al., 2011). While the potential for selection bias exists, these studies indicate that these biases may be minimal. In addition, the study protocols will incorporate procedures to minimize the potential for and to detect any indication of selection bias. For example, EHS-Net staff will be trained in the recruitment process in order to keep non-response rate as low as possible, which will help minimize selection bias.

The interview and survey data collected for this study may be influenced by social desirability bias- the tendency for people to report greater levels of socially desirable behavior (such as good food safety practices) than they actually engage in, or to report their best behavior rather than their typical or worst behavior. Although it is difficult to eliminate this bias altogether, it can be limited by ensuring respondents that the information they report will be anonymous, which we will do (Leary, 2004).

Because of resource and logistical challenges, we will only include restaurants whose managers speak English well enough to be interviewed in English. The use of this criterion may introduce bias, as non-English speakers may have different food safety knowledge and practices than English speakers. Recruitment data from previous EHS-Net studies indicate that about 4% of restaurants are excluded from our studies because of this criterion. Given this relatively low exclusion rate, we feel confident that this criterion does not limit the usefulness of the data collected from EHS-Net studies.

We will also only be able to interview food workers who can speak English well enough to be interviewed in English. Again, this may introduce bias, as English-speaking workers may differ from non-English-speaking workers. We can find no data indicating how many food workers don’t speak English, but National Restaurant Association data indicates that about a third of food workers speak a language other than English at home. However, in EHS-Net sites with high numbers of Spanish-speaking workers, we will administer written surveys (designed for low literacy readers) in Spanish to food workers. We will work with individual EHS-Net sites and if the resources are present within a site to administer the instruments in alternative languages, we will incorporate those opportunities. These actions will assist in limiting this potential bias.

Any presentation of data from EHS-Net studies will acknowledge these potential biases and include a discussion of how they impact data interpretation. Given that EHS-Net studies provide some of the only data available on real-time food worker food safety practices, these data are valuable to food safety efforts, despite these potential limitations.

B.3. Methods to Maximize Response Rates and Deal with No Response

We will engage in several activities designed to maximize response rates. First, all recruiters will receive training on the recruiting process that will be locally developed by EHS-Net sites. Second, multiple attempts will be made to contact potential respondents. Specifically, recruiters will make 10 attempts over 5 days to get a participation response from establishments they have not been able to contact, and 5 attempts over 5 days to get a participation response from restaurants that have not provided a response (e.g., ‘call back later’). Third, recruiting scripts will emphasize two issues that have been shown to increase response rates—the anonymous nature of the data collection and the importance of the respondents’ participation in the study.

With regards to no response, we will measure and report nonresponse to assess effects on data quality and to inform data users. We will take several steps in this regard. First, we will measure the proportion of the eligible population represented in our studies. Second, if our response rate is below 80%, we will conduct a bias analysis to assess whether the data are missing at random. We will also calculate item (i.e., question) response rates; if any of these are less than 70%, we will conduct an item bias analysis to assess whether the data are missing at random at the item level. The data from these analyses will be used to make decisions on which data to report and publish, and how to interpret published data.

B.4. Test of Procedures or Methods to be Undertaken

The data collection materials and methods are based on those used in other previous, successful EHS-Net studies (Bogard et al., 2013; Brown et al., 2013; Brown et al., 2012; Coleman et al., 2013; Green et al., 2006; Kirkland et al., 2009; Lee et al., 2004; Sumner et al., 2011). All data collection materials will be reviewed and evaluated by key EHS-Net specialists whom are experienced with collecting data for EHS-Net studies. They will also reviewed by CDC EHS-Net personnel with extensive experience in developing and conducting EHS-Net studies.

Data Analysis Plan. A primary purpose of these data collections will be to describe restaurant food handling and food safety procedures and practices, and establishment and manager/worker characteristics. To address this purpose, we will conduct descriptive analyses (frequencies, means, etc.). Table 4.1 contain examples of variables to be included in these analyses. Table 4.2 contains examples of variables used to describe the establishments in which we will collect data and the managers and workers from which we will collect data. Table 4.3 is a table shell that illustrates how we might analyze and present the descriptive data collected from these data collections.

4.1. Example variables measuring food handling/safety procedures and practices

Item Content	Manager interview #
Food handling/safety policies
Does this establishment require PICs or kitchen managers to be food safety certified?	8
Does this establishment have a policy or procedure concerning hand washing?	14
Does this restaurant have a policy that specifies the final cook temperatures for raw animal products?	16a
Does this restaurant have a policy that specifies the temperature at which potentially hazardous foods (or time control for food safety foods) should be cold held?	18
Does this restaurant have a policy that specifies the temperature at which potentially hazardous foods (or time control for food safety foods) should be hot held?	17a
Food handling/safety practices
How do food workers determine the final cook temperature of raw animal foods such as ground beef, chicken, or eggs?	16d
How do food workers determine the temperature of hot or cold held food?	19

4.2. Examples variables measuring characteristics

Item Content	Manager interview #
Establishment characteristics	1
Is this an independent restaurant or a chain restaurant?	2
Approximately how many meals are served here on your busiest day?
Which one of the options below best describes the menu for this establishment- American, Asian, Mexican, Italian, or Other?	3
How many food workers, excluding managers, work in this restaurant?	11
Manager/worker characteristics
Approximately how long have you been employed as the Person-In-Charge- the PIC- or kitchen manager here?	4
Have you, as the PIC or kitchen manager, ever been food safety certified?	8
What is the highest level of formal education you have completed?	29
What is your primary language?	31

4.3- Table Shell: Example descriptive data on food handling/safety procedures and practices

	n	%
Food handling/safety procedures
Does this establishment require PICs or kitchen managers to be food safety certified?
Yes	xx	xx
No	xx	xx
Does this establishment have a policy or procedure concerning hand washing?
Yes	xx	xx
No	xx	xx
Does this restaurant have a policy that specifies the final cook temperatures for raw animal products?
Yes	xx	xx
No	xx	xx
Does this restaurant have a policy that specifies the temperature at which potentially hazardous foods (or time control for food safety foods) should be cold held?
Yes	xx	xx
No	xx	xx
Does this restaurant have a policy that specifies the temperature at which potentially hazardous foods (or time control for food safety foods) should be hot held?
Yes	xx	xx
No	xx	xx
Food handling/safety practices
Method for determining the final cook temperature of raw animal foods such as ground beef, chicken, or eggs
Safe	xx	xx
Unsafe	xx	xx
Method for determining the temperature of hot or cold held food
Safe	xx	xx
Unsafe	xx	xx

A secondary purpose of these data collections is to determine how retail food establishment and manager/food worker characteristics are related to food handling/safety practices. To address this purpose, we will conduct tests for association and logistic regression models. Analysis will involve bivariate tests for association between each individual explanatory (independent) variable and the outcome (or dependent) variables of interest. Odds ratios will be calculated to assess the strength and direction of the bivariate relationships. For those bivariate associations found to be statistically significant at p<.10, the explanatory variables will be used as candidate “predictors” to examine their multivariate relationships with the outcome variables. Multivariable logistic regression will be used to model for the effects that these explanatory variables have in explaining the variations observed in the outcome variables.

Explanatory variables in these analyses include those measuring key establishment and manager/food worker characteristics. Outcome variables include those measuring key food handling/safety practices. Table 4.4 lists example key explanatory variables and key practice outcome variables. Table 4.5 is a table shell that illustrates how we might analyze and present the data examining the relationships between establishment and worker characteristics and key practices.

4.4- Example key explanatory and practice outcome variables

Explanatory variables	Outcome variables
Does this establishment require PICs or kitchen managers to be food safety certified?	Method for determining the final cook temperature of raw animal foods such as ground beef, chicken, or eggs
Is this an independent restaurant or a chain restaurant?	Method for determining the temperature of hot or cold held food
Which one of the options below best describes the menu for this establishment- American, Asian, Mexican, Italian, or Other?
Approximately how long have you been employed as the Person-In-Charge- the PIC- or kitchen manager here?

4.5- Table Shell: Example key explanatory variables associated with the practice outcome variable of whether the method for determining the final cook temperature of raw animal foods is safe

Explanatory variables	Method is safe
	OR (95% CI)	P
Establishment requires certification
No	x.xx (ref)	.xxx
Yes	x.xx
Ownership
Independent	x.xx (ref)	.xxx
Chain	x.xx
Menu
American	x.xx (ref)	.xxx
Non-American	x.xx
Manager length of employment
> xx years	x.xx (ref)	.xxx
> xx years	x.xx

OR=Odds Ratio, P=probability level

4.6- Table Shell: Key restaurant and manager characteristic explanatory variables associated with the practice outcome variables, bivariate analyses

Explanatory variables	Compliance with Food Safety Requirements (OBS: 11-12,14-15,23,25)
	OR (95% CI)	P
Restaurant characteristics
Volume of business (MI11-13)
> xxx	x.xx (ref)	.xxx
< xxx	x.xx
Turnover of staff (MI14-16)
> xx	x.xx (ref)	.xxx
< xx	x.xx
Certified kitchen Manager (MI22-24)
Some hours	x.xx (ref)	.xxx
All hours	x.xx
Not present	x.xx
Industry Segment (MI8)
Fast food	x.xx (ref)	.xxx
Fast casual	x.xx
Fine dining	x.xx
Family style	x.xx
Buffet	x.xx
Other	x.xx
Level of foodhandling (OBS10)
Prep-Serve	x.xx (ref)	.xxx
Cook-Serve	x.xx
Complex	x.xx
Employee characteristics
Employee experience (MI1,5, FWS48,52)
> xx years	x.xx (ref)	.xxx
< xx years	x.xx
Employee certified (MI3, FWS49-51)
Yes	x.xx (ref)	.xxx
No	x.xx
Position in restaurant (MI4, FWS53)
Manager	x.xx (ref)	.xxx
Cook	x.xx
Food preparation	x.xx
Other	x.xx

OR=Odds Ratio, P=probability level, MI = Manager Interview, FWS = Food Worker Survey, OBS = Restaurant Observation Form

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following people were primarily responsible for the design, including the statistical aspects, of the data collection and will be primarily responsible for data analysis. Laura Brown is the primary contact for statistical aspects and data collection.

Laura Green Brown, Ph.D.

Behavioral Scientist

Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

770-488-4332

Adam Kramer, Sc.D., M.P.H, R.S.

Environmental Health Officer

Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

404-498-1228

Rick Hoover, Ph.D.

Behavioral Scientist

Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

706-765-8857

Personnel in the eight EHS-Net sites will be responsible for data collection (See table below for current EHS-Net participant sites). In some cases, environmental health specialists from non-EHS-Net sites will assist with data collection; these personnel are not included in this table.

Site	Number of Personnel
California Department of Health	1 full-time
Minnesota Department of Health	1 full-time 1 part-time
New York Department of Health	1 full-time 1 part-time
New York City Department of Health and Mental Hygiene	1 full-time
Rhode Island Department of Health	1 full-time
Tennessee Department of Health	1 full-time 1 part-time
Southern Nevada Health District	1 full-time
Harris County, Texas	1 full-time

References for Part B

Bogard, A. K., Fuller, C. C., Radke, V., Selman, C. A., & Smith, K. E. (2013). Ground Beef Handling and Cooking Practices in Restaurants in Eight States. Journal of Food Protection, 76(12), 2132-2140. doi:10.4315/0362-028x.jfp-13-126

Brown, L., Hoover, E., Ripley, D., Matis, B., Nicholas, D., Hedeen, N., & Faw, B. (2016). Retail Deli Slicer Cleaning Frequency — Six Selected Sites, United States, 2012. MMWR Morb Mortal Wkly Rep, 65, 306-310. doi: http://dx.doi.org/10.15585/mmwr.mm6512a2

Brown, L., Khargonekar, S., & Bushnell, L. (2013). Frequency of Inadequate Chicken Cross-Contamination Prevention and Cooking Practices in Restaurants. Journal of Food Protection, 76(12), 2141-2145. doi:10.4315/0362-028x.jfp-13-129

Brown, L., Le, B., Wong, M. R., Reimann, D., Nicholas, D., Faw, B., . . . Selman, C. A. (2014). Restaurant manager and worker food safety certification and knowledge. Foodborne Pathogens and Disease, 11(11), 835-843.

Brown, L., Ripley, D., Blade, H., Reimann, D., Everstine, K., Nicholas, D., . . . Quilliam, D. N. (2012). Restaurant Food Cooling Practices. Journal of Food Protection, 75(12), 2172-2178. doi:10.4315/0362-028X.JFP-12-256

Coleman, E., Delea, K., Everstine, K., Reimann, D., & Ripley, D. (2013). Handling Practices of Fresh Leafy Greens in Restaurants: Receiving and Training. Journal of Food Protection, 76(12), 2126-2131. doi:10.4315/0362-028x.jfp-13-127

Frey, J. H., & Oishi, S. M. (1995). How To Conduct Interviews by Telephone and In Person. The Survey Kit, Volume 4: ERIC.

Green, L. R., Selman, C. A., Radke, V., Ripley, D., Mack, J. C., Reimann, D. W., . . . Bushnell, L. (2006). Food worker hand washing practices: an observation study. Journal of Food Protection®, 69(10), 2417-2423.

Kirkland, E., Green, L. R., Stone, C., Reimann, D., Nicholas, D., Mason, R., . . . The, E. H. S. N. W. G. (2009). Tomato Handling Practices in Restaurants. Journal of Food Protection, 72(8), 1692-1698.

Leary, M. R. (2004). Introduction to behavioral research methods. Boston, MA: Pearson/Allyn and Bacon.

Lee, R., Beatty, M. E., Bogard, A. K., Esko, M.-P., Angulo, F. J., Selman, C., & Ehs-Net Working, G. (2004). Prevalence of High-Risk Egg-Preparation Practices in Restaurants That Prepare Breakfast Egg Entrées: An EHS-Net Study. Journal of Food Protection, 67(7), 1444-1450.

Radke, T. J., Brown, L., Hoover, E. R., Faw, B. V., Reimann, D., Wong, M. R., . . . Ripley, D. (2016). Food Allergy Knowledge and Attitudes of Restaurant Managers and Staff: An EHS-Net Study. Journal of Food Protection®, 79(9), 1588-1598.

Sumner, S., Brown, L., Frick, R., Stone, C., Carpenter, L. R., Bushnell, L., . . . Everstine, K. (2011). Factors Associated with Food Workers Working while Experiencing Vomiting or Diarrhea. J Food Prot, 74(2), 215-220. doi:10.4315/0362-028x.jfp-10-108



23