1 - SSB EHS-Net 2021

Supporting Statement – Section B

Environmental Health Specialists Network (EHS-Net) Program

Revision

OMB No. 0920-0792

Expiration Date: 08/31/2021

Submitted: July 20, 2021

Laura Brown, Ph.D.

EHS-Net

Centers for Disease Control and Prevention

National Center for Environmental Health

Emergency and Environmental Health Services

Environmental Health Services Branch

4770 Buford Highway

Atlanta, GA 30341-3724

[email protected]

770-488-4332

Table of Contents

B. Collections of Information Employing Statistical Methods…………………………..	3
1. Respondent Universe and Sampling Methods…………………………………………	3
2. Procedures for the Collection of Information………………………………………….	5
3. Methods to Maximize Response Rates and Deal with Nonresponse………………….	6
4. Tests of Procedures or Methods to be Undertaken…………………………………….	7
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data………………………………………………………………………...	11

B. Collections of Information Employing Statistical Methods

B.1. Respondent Universe and Sampling Methods

EHS-Net is a collaborative project of the CDC, FDA, USDA, and eight state and local public health departments (currently comprised of: Franklin County, OH; Minnesota; New York City, New York; New York; Rhode Island; Tennessee; Southern Nevada Health District, NV; and Harris County, TX). The respondent universe is comprised of retail food establishments in selected geographical areas within the EHS-Net health department jurisdictions. While the number of areas included in EHS-Net is small, they are demographically diverse and provide good geographical coverage of the U.S. (northeast, mid-west, south, and west). When the sampling methods outlined here for ensuring a representative sample in the current study are used, the results of the collection covered by this OMB package can be used to generalize to the population of retail food establishments in the given EHS-Net site(s).

The respondent universe is all retail food establishments (hereafter referred to as restaurants) in the EHS-Net health department jurisdictions. Restaurant lists will be obtained from the restaurant databases maintained by the EHS-Net sites. CDC will use these restaurant lists to generate the sampling frame used to draw the sample for this study.

Each EHS-Net site will typically enroll 50 restaurants per study (Table 1.1.). Enrollment of 50 restaurants per EHS-Net site, totaling 400 restaurants for the entire study, is a reasonable sample size and follows the precedent of previous EHS-Net studies (Green et al., 2006; Kirkland et al., 2009; Sumner et al., 2011). Experience from prior EHS-Net studies also indicates that a sample size of 400 should be sufficient for the analytic purposes, since the analytic parameters are not likely to be considered rare (in distribution) events. Data collected from projects performed under these collections will provide the necessary information for sample size and power calculations for future studies.

1.1 Number of restaurants per current EHS-Net site

Strata (EHS-Net Sites)	Entity	Number of Entities
Franklin County, Ohio	Restaurants	50
Minnesota	Restaurants	50
New York	Restaurants	50
New York City	Restaurants	50
Rhode Island	Restaurants	50
Tennessee	Restaurants	50
Southern Nevada Health District	Restaurants	50
Harris County, Texas	Restaurants	50

A sample size calculation was conducted to estimate the total number of retail food establishments necessary to provide sufficient power to detect group differences in outcomes of interest. The design parameters were set to detect an odds ratio of at least 2.0 or greater (a medium –small effect size) with a .05 Type I error rate, power of .90, and based on the assumption of equivalent group sizes. Calculations established that a total of 372 retail food establishments would need to participate to achieve the desired parameters. With eight EHS-Net sites participating in data collection, each site will need to enroll 47 establishments to meet this requirement. In order to detect a greater odds ratio and establish a stronger power we have increased the total sample size to 400 retail food establishments, with each EHS-Net site enrolling 50 establishments. In addition, this increase will allow for additional statistical designs.

The design is cross-sectional and uses a stratified random sampling plan in which each EHS-Net site serves as its own mutually exclusive stratum. There are two primary reasons for stratifying by EHS-Net site. The first is that food safety regulations vary by jurisdiction. For example, Tennessee state food safety regulations differ from New York state food safety regulations. These regulations can and do greatly influence restaurants’ food safety practices and policies. EHS-Net site/jurisdiction, therefore, poses the largest source of variability from a study design perspective. Thus, it is a critically important factor for stratification. The second reason for stratifying by EHS-Net site is due to practical concerns that limit our ability to stratify on other variables of interest. EHS-Net sites participate in EHS-Net through a cooperative agreement (EH15-001.) The nature of this agreement is such that one site cannot be expected to do a disproportionate amount of work in comparison to other sites (because each site receives relatively equal funding amounts). If we did not stratify by EHS-Net site but by some other factor such as ownership (independently owned or belonging to a corporate regional chain), it is likely that some sites would have to carry a greater burden than other sites in term of recruiting and collecting data in a larger number of restaurants. However, we will be collecting data on these factors of interest and will account for their heterogeneity through statistical modeling. Finally, the need for each site to share an equal burden in data collection is the reason why a fixed-sample allocation method is used for each site (50 establishments per site), instead of a proportionate-sample allocation.

Restaurants will be randomly selected, with equal probability, within their respective EHS-Net site, independent of other sites. This process will give each restaurant in a particular sampling frame the same probability of being selected for study participation. There are three reasons for employing this sampling strategy: reducing sampling error, maintaining equal representation by site, and ensuring generalizability. First, as stated previously, the total target population of restaurants from all EHS-Net sites combined constitutes a highly heterogeneous group. To control for such heterogeneity in the total sample, restaurants will be stratified by EHS-Net site so they can be grouped into more homogeneous strata and then sampled within stratum independently. This reduction in heterogeneity of the total sample will lead to reduction in sampling error, which can improve representativeness of the selected sample and provide estimates (e.g., means) that tend to have less variability than estimates produced from samples that were drawn using the un-stratified, simple random sampling method. Second, with equal allocation of samples (50 restaurants per site), each EHS-Net site will have equal representation in the parameter estimates of the combined sample. An additional benefit is that even sites with small sampling frames will have sufficient data points to support their site-specific analyses. Third, by ensuring that the sampling of restaurants is done by an entity (CDC) separate from the data collectors (EHS-Net sites) and employing a random selection method, we are able to minimize the potential for selection bias. Parameter estimates or study findings obtained from an unbiased study sample could be generalized to the entire EHS-Net target population.

The average response rate across EHS-Net studies that used methods similar to the proposed study is 45% (Brown et al., 2016; Brown et al., 2014; Radke et al., 2016). We expect a similar response rate for the proposed studies performed under this collection. Although this response rate is lower than the 80% response rate mentioned in the OMB guidelines, is a rate that allows us to feel comfortable with our planned analysis (Frey & Oishi, 1995).

B.2. Procedures for the Collection of Information

As indicated earlier, each EHS-Net site will provide CDC with a list of all restaurants in their jurisdiction. This list will serve as the sampling frame for the site. CDC will use a random number generator in the SAS statistical program to produce a random sample of restaurants from this restaurant list for each site. As we expect some restaurants will refuse to participate and some will be ineligible to participate, we will select more than the needed number of restaurants--100 restaurants for each site. Once they receive their sample list from CDC, personnel in each site will contact restaurants by telephone to recruit their participation in the study. If the manager is willing to participate, the EHS-Net specialist will arrange a mutually convenient time to conduct the data collection.

In instances where an EHS-Net site is unable to recruit 50 restaurants from the first list of 100 restaurants, CDC will randomly select another group of 50 restaurants for the site to use to recruit additional respondents. Recruitment will be considered complete once data are collected in 50 restaurants. EHS-Net sites will recruit via the telephone and will keep a log of each contact with the restaurants to document participation rates and reasons for refusal and/or ineligibility.

CDC will not know which restaurants on the sample lists participated in the study, and thus will not be able to link restaurant names with any study data. The restaurant identifying information will be maintained by the respective EHS-Net site to facilitate the site visit but will be destroyed when it is no longer needed. Additionally, on all forms only the specific coded restaurant identifier will be used to minimize a risk of someone inadvertently seeing a completed form and being able to associate it with a specific restaurant.

Data will be collected in the restaurants by the EHS-Net environmental health specialists. For the manager interview portion of the study, the EHS-Net specialist will obtain verbal informed consent and then conduct a face-to-face interview with a manager who has authority over the kitchen and can speak English well enough to complete the interview in English. This interview will include questions on restaurant characteristics, policies, and food safety-related practices. This will take about thirty minutes to complete.

For the food worker survey portion of the study, the data collector will obtain verbal informed consent and then either administer a written/electronic survey or conduct an interview as appropriate for the specific study. The intent of the survey/interview will be to gather information on food worker beliefs, knowledge, and practices related to food safety along with worker demographics.

For the restaurant observation portion, the data collector will observe the kitchen and note the food-safety related infrastructure and the implementation of specific food safety practices.

The data collectors are experienced and knowledgeable in environmental health and food safety and will have received training from CDC on data collection for the studies done under this collection. The EHS-Net administrator in each EHS-Net site and CDC staff will perform quality assurance procedures to check for data entry errors.

Managers’ concerns about the safety practices of their restaurants may result in selection bias- a lower rate of study participation among restaurants with worse or non-existent safety practices compared to restaurants with better safety practices. We have conducted studies using methods similar to those proposed for use in this study in the past, and these studies have found a wide range of food safety practices, including poor ones (Bogard, Fuller, Radke, Selman, & Smith, 2013; Brown, Khargonekar, & Bushnell, 2013; Brown et al., 2012; Coleman, Delea, Everstine, Reimann, & Ripley, 2013; Green et al., 2006; Kirkland et al., 2009; Lee et al., 2004; Sumner et al., 2011). While the potential for selection bias exists, these studies indicate that these biases may be minimal. In addition, the study protocols will incorporate procedures to minimize the potential for and to detect any indication of selection bias. For example, EHS-Net staff will be trained in the recruitment process in order to keep non-response rate as low as possible, which will help minimize selection bias.

The interview and survey data collected for this study may be influenced by social desirability bias- the tendency for people to report greater levels of socially desirable behavior (such as good food safety practices) than they actually engage in, or to report their best behavior rather than their typical or worst behavior. Although it is difficult to eliminate this bias altogether, it can be limited by ensuring respondents that the information they report will be anonymous, which we will do (Leary, 2004).

Because of resource and logistical challenges, we have, in the past, only included restaurants whose managers speak English well enough to be interviewed in English. The use of this criterion may introduce bias, as non-English speakers may have different food safety knowledge and practices than English speakers. Recruitment data from previous EHS-Net studies indicate that about 4% of restaurants are excluded from our studies because of this criterion. Given this relatively low exclusion rate, we feel confident that this criterion does not limit the usefulness of the data collected from EHS-Net studies. However, we plan to attempt to collect data in at least one other language than English in several EHS-Net sites in this data collection cycle.

Any presentation of data from EHS-Net studies will acknowledge these potential biases and include a discussion of how they impact data interpretation. Given that EHS-Net studies provide some of the only data available on real-time food worker food safety practices, these data are valuable to food safety efforts, despite these potential limitations.

B.3. Methods to Maximize Response Rates and Deal with Nonresponse

We will engage in several activities designed to maximize response rates. First, all recruiters will receive training on the recruiting process that will be locally developed by EHS-Net sites. Second, multiple attempts will be made to contact potential respondents. Specifically, recruiters will make 10 attempts over 5 days to get a participation response from establishments they have not been able to contact, and 5 attempts over 5 days to get a participation response from restaurants that have not provided a response (e.g., ‘call back later’). Third, recruiting scripts will emphasize two issues that have been shown to increase response rates—the anonymous nature of the data collection and the importance of the respondents’ participation in the study.

With regards to no response, we will measure and report nonresponse to assess effects on data quality and to inform data users. We will take several steps in this regard. First, we will measure the proportion of the eligible population represented in our studies. Second, if our response rate is below 80%, we will conduct a bias analysis to assess whether the data are missing at random. We will also calculate item (i.e., question) response rates; if any of these are less than 70%, we will conduct an item bias analysis to assess whether the data are missing at random at the item level. The data from these analyses will be used to make decisions on which data to report and publish, and how to interpret published data.

B.4. Test of Procedures or Methods to be Undertaken

The data collection materials and methods are based on those used in other previous, successful EHS-Net studies (Bogard et al., 2013; Brown et al., 2013; Brown et al., 2012; Coleman et al., 2013; Green et al., 2006; Kirkland et al., 2009; Lee et al., 2004; Sumner et al., 2011). All data collection materials will be reviewed and evaluated by key EHS-Net specialists whom are experienced with collecting data for EHS-Net studies. They will also reviewed by CDC EHS-Net personnel with extensive experience in developing and conducting EHS-Net studies.

Data Analysis Plan. A primary purpose of these data collections will be to describe restaurant food handling and food safety procedures and practices, and establishment and manager/worker characteristics. To address this purpose, we will conduct descriptive analyses (frequencies, means, etc.). Table 4.1 contain examples of variables to be included in these analyses. Table 4.2 contains examples of variables used to describe the establishments in which we will collect data and the managers and workers from which we will collect data. Table 4.3 is a table shell that illustrates how we might analyze and present the descriptive data collected from these data collections.

4.1. Example variables measuring food handling/safety procedures and practices

Item Content	Manager interview #
Food handling/safety policies
Does this establishment require PICs or kitchen managers to be food safety certified?	8
Does this establishment have a policy or procedure concerning hand washing?	14
Does this restaurant have a policy that specifies the final cook temperatures for raw animal products?	16a
Does this restaurant have a policy that specifies the temperature at which potentially hazardous foods (or time control for food safety foods) should be cold held?	18
Does this restaurant have a policy that specifies the temperature at which potentially hazardous foods (or time control for food safety foods) should be hot held?	17a
Food handling/safety practices
How do food workers determine the final cook temperature of raw animal foods such as ground beef, chicken, or eggs?	16d
How do food workers determine the temperature of hot or cold held food?	19

4.2. Examples variables measuring characteristics

Item Content	Manager interview #
Establishment characteristics	1
Is this an independent restaurant or a chain restaurant?	2
Approximately how many meals are served here on your busiest day?
Which one of the options below best describes the menu for this establishment- American, Asian, Mexican, Italian, or Other?	3
How many food workers, excluding managers, work in this restaurant?	11
Manager/worker characteristics
Approximately how long have you been employed as the Person-In-Charge- the PIC- or kitchen manager here?	4
Have you, as the PIC or kitchen manager, ever been food safety certified?	8
What is the highest level of formal education you have completed?	29
What is your primary language?	31

4.3- Table Shell: Example descriptive data on food handling/safety procedures and practices

	n	%
Food handling/safety procedures
Does this establishment require PICs or kitchen managers to be food safety certified?
Yes	xx	xx
No	xx	xx
Does this establishment have a policy or procedure concerning hand washing?
Yes	xx	xx
No	xx	xx
Does this restaurant have a policy that specifies the final cook temperatures for raw animal products?
Yes	xx	xx
No	xx	xx
Does this restaurant have a policy that specifies the temperature at which potentially hazardous foods (or time control for food safety foods) should be cold held?
Yes	xx	xx
No	xx	xx
Does this restaurant have a policy that specifies the temperature at which potentially hazardous foods (or time control for food safety foods) should be hot held?
Yes	xx	xx
No	xx	xx
Food handling/safety practices
Method for determining the final cook temperature of raw animal foods such as ground beef, chicken, or eggs
Safe	xx	xx
Unsafe	xx	xx
Method for determining the temperature of hot or cold held food
Safe	xx	xx
Unsafe	xx	xx

A secondary purpose of these data collections is to determine how retail food establishment and manager/food worker characteristics are related to food handling/safety practices. To address this purpose, we will conduct tests for association and logistic regression models. Analysis will involve bivariate tests for association between each individual explanatory (independent) variable and the outcome (or dependent) variables of interest. Odds ratios will be calculated to assess the strength and direction of the bivariate relationships. For those bivariate associations found to be statistically significant at p<.10, the explanatory variables will be used as candidate “predictors” to examine their multivariate relationships with the outcome variables. Multivariable logistic regression will be used to model for the effects that these explanatory variables have in explaining the variations observed in the outcome variables.

Explanatory variables in these analyses include those measuring key establishment and manager/food worker characteristics. Outcome variables include those measuring key food handling/safety practices. Table 4.4 lists example key explanatory variables and key practice outcome variables. Table 4.5 is a table shell that illustrates how we might analyze and present the data examining the relationships between establishment and worker characteristics and key practices.

4.4- Example key explanatory and practice outcome variables

Explanatory variables	Outcome variables
Does this establishment require PICs or kitchen managers to be food safety certified?	Method for determining the final cook temperature of raw animal foods such as ground beef, chicken, or eggs
Is this an independent restaurant or a chain restaurant?	Method for determining the temperature of hot or cold held food
Which one of the options below best describes the menu for this establishment- American, Asian, Mexican, Italian, or Other?
Approximately how long have you been employed as the Person-In-Charge- the PIC- or kitchen manager here?

4.5- Table Shell: Example key explanatory variables associated with the practice outcome variable of whether the method for determining the final cook temperature of raw animal foods is safe

Explanatory variables	Method is safe
	OR (95% CI)	P
Establishment requires certification
No	x.xx (ref)	.xxx
Yes	x.xx
Ownership
Independent	x.xx (ref)	.xxx
Chain	x.xx
Menu
American	x.xx (ref)	.xxx
Non-American	x.xx
Manager length of employment
> xx years	x.xx (ref)	.xxx
> xx years	x.xx

OR=Odds Ratio, P=probability level

4.6- Table Shell: Key restaurant and manager characteristic explanatory variables associated with the practice outcome variables, bivariate analyses

Explanatory variables	Compliance with Food Safety Requirements (OBS: 11-12,14-15,23,25)
	OR (95% CI)	P
Restaurant characteristics
Volume of business (MI11-13)
> xxx	x.xx (ref)	.xxx
< xxx	x.xx
Turnover of staff (MI14-16)
> xx	x.xx (ref)	.xxx
< xx	x.xx
Certified kitchen Manager (MI22-24)
Some hours	x.xx (ref)	.xxx
All hours	x.xx
Not present	x.xx
Industry Segment (MI8)
Fast food	x.xx (ref)	.xxx
Fast casual	x.xx
Fine dining	x.xx
Family style	x.xx
Buffet	x.xx
Other	x.xx
Level of foodhandling (OBS10)
Prep-Serve	x.xx (ref)	.xxx
Cook-Serve	x.xx
Complex	x.xx
Employee characteristics
Employee experience (MI1,5, FWS48,52)
> xx years	x.xx (ref)	.xxx
< xx years	x.xx
Employee certified (MI3, FWS49-51)
Yes	x.xx (ref)	.xxx
No	x.xx
Position in restaurant (MI4, FWS53)
Manager	x.xx (ref)	.xxx
Cook	x.xx
Food preparation	x.xx
Other	x.xx

OR=Odds Ratio, P=probability level, MI = Manager Interview, FWS = Food Worker Survey, OBS = Restaurant Observation Form

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

The following people were primarily responsible for the design, including the statistical aspects, of the data collection and will be primarily responsible for data analysis. Laura Brown is the primary contact for statistical aspects and data collection.

Laura Brown, Ph.D.

Behavioral Scientist

Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

Rick Hoover, Ph.D.

Statistician

Contractor, Centers for Disease Control and Prevention, National Center for Environmental Health

[email protected]

Personnel in an estimated 8 EHS-Net sites will be responsible for data collection.

18