0799_Supporting Statement Part B_2021

0799_Supporting Statement Part B_2021.docx

Survey on the Occurrence of Foodborne Illness Risks Factors in Selected Institutional Foodservice and Retail Food Stores Facility Types

OMB: 0910-0799

Document [docx]

Download: docx | pdf

B. Statistical Methods (used for collection of information employing statistical methods)

Respondent Universe and Sampling Methods

The respondent universe will be 800 respondents as follows:

Respondent Description	Number
Person in charge of randomly selected retail food stores	400
Program director (or designated individual) of the respective regulatory authority over the randomly selected establishments	400
Total	800

A geographical information system (GIS) database containing a listing of businesses throughout the United States is used as the establishment inventory for the data collections. The data were purchased from the Environmental Systems Research Institute (Esri). The healthcare facility data, school data, and retail food store data are the partial of Esri’s USA Business Locations and Business Summary. It is updated annually. The data is stored as GeoDataBase.

The Esri list contains data for the establishment name, location, franchise code, both versions of industry classification codes (SIC system and NAICS), number of employees, and estimated sales volume (expressed in thousands of dollars).

Esri extracts its business data from a comprehensive list of businesses licensed from Data Axle (www.data-axle.com, formerly Infogroup). This business list contains data about 13 million US businesses—including the business name, location, franchise code, industry classification code, number of employees, and sales volume—that is current as of April 2020. Data Axle references several sources including directory listings such as Yellow Pages and business white pages; annual reports; 10-K and Securities and Exchange Commission (SEC) information; federal, state, and municipal government data; business magazines; newsletters and newspapers; and information from the US Postal Service. To ensure accurate and complete information, Data Axle conducts annual telephone verifications with each business listed in the database. We also combined Esri’s datasets with the Homeland Infrastructure Foundation-Level Data (HIFLD, https://gii.dhs.gov/hifld/ ).

We also combined Esri’s datasets with the Homeland Security Infrastructure Program (HSIP) Gold datasets. HSIP Gold is a unified homeland infrastructure geospatial data inventory assembled by National Geospatial-Intelligence Agency (NGA) in partnership with the Department of Homeland Security (DHS) for official use by the Homeland Security and Homeland Defense (HLS/HD) communities.

The addresses of the establishments are geocoded to assign latitude and longitude coordinates. The quality of the local address system varies. Address matching is better in urban areas that use street-level address system than in rural areas. Establishments that cannot be assigned to a census block group are assigned to a census tract or county. We use the geographic codes to do spatial sampling for the risk factor study.

FDA performs a three-tiered filtering process to ensure establishments are correctly classified into the appropriate facility type described in Table 4 and considered eligible to participate in the survey. The filter types include: the subclass the establishment belongs to, the name of the establishment, and keywords. For school establishments, we also use additional filters: school enrollment and whether the school end-grade is 1 or greater than 1. The term “eligible” in this context means that the establishment is contained in the geographic areas from which it is being sampled. Any establishment in the geographic areas can be selected.

TABLE 4 – Description of the Facility Types Included in the Survey

Industry

Segment

Facility Type

Description

Retail Food Stores

Deli Department / Operation

Areas in a retail food store where foods, such as luncheon meats and cheeses, are sliced for the customers and where sandwiches and salads are prepared on-site or received from a commissary in bulk containers, portioned, and displayed. Parts of the deli department/operation may include:

Salad bars, pizza stations, and other food bars managed by the deli department manager,
Areas where meat and poultry products are cooked and offered for sale as ready-to-eat and are managed by the deli department manager.

Efforts are made to exclude establishments from the study that only handle pre-packaged food items or conduct low-risk food preparation activities.

Approximately 23 FDA Retail Food Specialists (Specialists) serve as the data collectors for the study. These individuals possess technical expertise in retail food safety and a solid understanding of the operations within each of the facility types to be surveyed. The Specialists are also standardized by FDA Center for Food Safety and Applied Nutrition (CFSAN) personnel in the application and interpretation of the FDA Food Code. The 400 data collections have been evenly distributed among all available standardized Specialists.

The Specialists are located near major metropolitan areas (i.e. population centers) across the contiguous United States. Population centers usually contain a large concentration of state and local regulatory jurisdictions.

Eligible establishments are randomly selected from among all eligible establishments located within a 175-mile radius of each of the Specialists’ home locations (zip codes). Using the 175-mile radius sampling zones provides a relatively good cross section of urban and rural areas from which to sample the eligible establishments. It also represents a good mix of small, medium, and large regulatory entities having jurisdiction over the eligible establishments. Lastly, it reduces overnight travel and therefore reduces travel costs incurred by the Agency to collect data.

In the interest of cost efficiency, we apply one caveat to the sampling zones as follows. The actual driving distance to a few of the selected establishments may exceed 175 miles to geographic barriers of one form or another. Since travel time is not included in the Specialists’ work plan hours and excessive overnight travel would be financial burden to the Agency, if an establishment on the inventory list exceeds a 175-mile driving distance from the Specialist’s home, as confirmed via Google Maps, the Specialist has the option of requesting a substitute establishment. Specialists are encouraged to still conduct data collections at establishments that may exceed the 175-mile radius by only a few miles (or where travel time is not significantly impacted by the extra distance). When requesting a substitute establishment based on driving distance exceeding 175 miles, the Specialist is to include the Google Map showing the mileage distance from their home to the establishment.

The total number of retail food stores in the database is 171,063, and the total number within the 23 sampling zones is 97,538. This means that the 23 sampling zones contain approximately 57% of all retail food store establishments in the contiguous U.S. If additional FTEs are utilized in the data collection, then an even greater percentage of establishments would be contained within approximately 23 sampling zones.

All analysis reports will clearly indicate that the sample drawn was purposeful and that estimates generated from the study cannot be generalized to the U.S. as a whole.

The following map illustrates the location of FDA’s 23 standardized Specialists and the corresponding 175-mile radius.

FIGURE 1. Location of FDA’s 23 Standardized Retail Food Specialists’ home zip code and the surrounding 175-mile radius (restricted by non-overlap)

A list of Regional Retail Food Specialists who will serve as data collectors by branch, city, state, and zip code can be found in section 5 of this document.

Based on the number of entry refusals from the 2015-2016 Risk Factor Study in the restaurant facility types, we estimate a refusal rate of two percent in the institutional foodservice and retail food store facility types.

Substitute establishments will be selected in cases when an establishment is misclassified, closed, or otherwise unavailable, unable, or unwilling to participate. The Specialists have been provided a substitute establishment list to expedite the process of going back and forth to the Biostatistics Branch. The Specialist must select the establishment from the substitute list in numerical order at it appears on the list.

Procedures for the Collection of Information

In order to obtain a sufficient number of observations to conduct statistically significant analysis, FDA has determined, based on the previous 1998-2008 risk factor study, that approximately 400 data collection inspections of each facility type are needed. This sample size was calculated to provide for sufficient observations to be 95% confident that the compliance percentage is within 5% of the true compliance percentage.

Although many different population parameter estimates will be made using this survey data, the sample size was calculated to ensure that the primary goal of the study was achieved. The required sample size was calculated based on the ten primary data items. Each of the ten primary data items should have a response (IN compliance or OUT of compliance) based on the information statements which are contained within each data item. We expect that all or almost all of the data items will have a response (see B.3). We will have a compliance percentage for the ten primary data items which will simply be the total number of IN compliance observations divided by the total number of IN compliance observations plus the total number of OUT of compliance observations. Therefore, each of the 400 establishments will have 10 observations that will be used to compute the IN compliance percentage for the facility type.

Using data from the previous study the “effective sample size” was calculated as follows:

Where m = 160 responses (10 per establishment) in a geographical area, k= number of geographic areas, and DE is the design effect. .

JMP 10 was utilized to calculate the ICC in EMP-results obtained by the Measurement Systems Analysis platform.

In order to calculate the sample size we needed an estimate of the variance of the proportion, a confidence level. Utilizing the ESS calculated above and estimates for the IN compliance percentages from the previous study, the precision was estimated as follows:

Where Z_.025 = 1.96, P=.71, Q = .29 and n =630 and e is the margin of error. Solving for e gives 3.50%.

Once the data is collected, the observed sample ICC and variance will be used when reporting the results. FDA feels that the sample size will be sufficient to have a margin of error of less than 5% of the estimated proportion of IN compliance observations.

Preparatory Training

Each Specialist will attend a training webinar prior to initiating the data collection. The training will be provided by members of the FDA National Retail Food Team that have been responsible for the design and assessment of all the Retail Food Risk Factor Study elements. The training will cover all the study components with particular emphasis on the data collection protocol and marking instructions for the data collection form.

The training will cover all the following study components:

Data collection protocol (Attachment A)
Marking instructions for the data collection form (Attachment B)
Data collection form (Attachment C)
Industry Introductory Letter (Attachment D)

The data collection form is divided into three sections: Section 1 - Establishment Information; Section 2 - Jurisdiction with Regulatory Authority Information; and Section 3 for tabulating the Specialists’ observations of (a) the food employees’ behaviors and practices related to personal hygiene and food storage, preparation, and service, (b) the industry food safety management being employed, and (c) the frequency of food employee hand washing.

Verification of Eligibility of Randomly Selected Establishments

Specialists will receive a set of institutional foodservice and retail food store facilities within their primary area of responsibility that have been randomly selected for the study by the FDA CFSAN Biostatistics Branch. Prior to conducting the data collection, Specialists will contact the state or local jurisdiction that has regulatory responsibility for conducting retail food inspections for the selected establishment. Specialists will verify that each facility has been properly classified (in the correct facility type category) for the purposes of the study and is still in operation. Specialists will ascertain whether the selected facility is under legal notice from the state or local regulatory authority. If the selected facility is under legal notice, the Specialists will not conduct a data collection in that establishment. The Specialists will remove the establishment from their sample inventory and select another establishment from their list of substitute establishments provided by the FDA CFSAN Biostatistics Branch.

Information Collection Involving Regulatory Authorities

As part of the initial contact with the regulatory authority, Specialists will obtain information from the jurisdiction pertaining to the items listed under the heading “Information on the Regulatory Authority” on the applicable data collection form for the facility type (Attachment C). At that time, Specialists will collect the information for the following data collection fields:

Name of Jurisdiction with Regulatory Oversight.
Enrolled in FDA Retail Food Program Standards.
Jurisdiction Meets Standard 1.
Dates of the Two Most Recent Regulatory Routine Inspections.
Jurisdiction Uses a Grading System.
Type of Grading System.
Jurisdiction’s Program Includes Public Reporting of Inspection Results.
Jurisdiction Has a Mandatory Food Protection Manager Certification Requirement.
Scope of Food Protection Manager Certification Requirement.
Jurisdiction Requires Food Handler Cards.

In addition, the Specialist will complete Section 2 - Jurisdiction with Regulatory Authority Information of the data collection form. Guidance for completing the information fields in this section of the form is included on pages 33-40 of the marking instructions document (Attachment B).

Specialists will extend an invitation to the state or local regulatory authority to accompany him or her on the data collection visit. Should the regulatory authority accept and accompany the Specialist, the Specialist should strongly recommend that the state or local regulatory authority refrain from conducting a regulatory compliance inspection during the data collection visit.

Calibration of Temperature Measuring Devices

Specialists will ensure that thermometers used for each data collection are accurate by verifying their accuracy prior to each establishment data collection visit.

Conducting the Data Collection

Each data collection visit is to be unannounced. The intent is to observe the operation in its normal mode, without special preparation to accommodate the data collection visit.

Upon arrival to the establishment, Specialists will explain to the owner the purpose of the visit. An introductory letter (Attachment D) that explains the purpose of the data collection visit and the study must be used in addition to a verbal explanation. If entry into the selected establishment is denied by the owner or person in charge, Specialists will not conduct a data collection. Specialists will select a new establishment from the substitute establishment list provided by the FDA CFSAN Biostatistics Branch.

After discussing the purpose of the data collection and developing a rapport with the person in charge, Specialist will conduct a quick (two to three minute) walk-though of the establishment’s kitchen as described in the Study Protocol (Attachment A) to determine inspection priorities and flow.

Primary Data Items

Specialists will then make every effort to observe procedures and practices related to data items 1 through 10 (primary data items) in Attachment C. Each of the primary data items has been placed under the appropriate FDA foodborne illness risk factor category which will be used as the key indicators for FDA’s statistical analysis for the study:

Risk Factor –Poor Personal Hygiene

#1 – Employees practice proper handwashing

#2 – Food Employees do not contact ready-to-eat foods with bare hands

Contaminated Equipment / Protection from Contamination

#3 – Food is protected from cross-contamination during storage, preparation, and display

#4 – Food contact surfaces are properly cleaned and sanitized

Improper Holding / Time and Temperature

#5 – Foods requiring refrigeration are held at the proper temperature

#6 – Foods displayed or stored hot are held at the proper temperature

#7 – Foods are cooled properly

#8 – Refrigerated, ready-to-eat foods are properly date marked and discarded within 7 days of preparation or opening

Inadequate Cooking

#9 – Raw animal foods are cooked to required temperatures
#10 – Cooked foods are reheated to required temperatures

Guidance for marking observations of primary data items is provided on pages 81-102 of the marking instructions document (Attachment B).

Other Areas of Interest – Data Items

Specialists will also collect information on data items 11 through 19 are listed under the heading “Other Areas of Interest” in Attachment C. These food safety practices and procedures directly support active managerial control of the foodborne illness risk factor areas addressed under the primary data items:

Data Item #11 – Handwashing facilities are accessible and properly maintained
Data Item #12 – Employees practice good hygiene
Data Item #13 – Consumers are properly advised of risks of consuming raw or undercooked animal foods
Data Item #14 – Time alone is properly used as a public health control
Data Item #15 – Facilities have adequate equipment and tools for ensuring food temperature control and sanitization of food contact surfaces
Data Item #16 – Special processes are conducted in compliance with issued variance / HACCP Plan, when required
Data Item #17 – Food is received from safe sources
Data Item #18 – Toxic materials are identified, used and stored properly
Data Item #19 – Management and food employees are trained in food allergy awareness as it relates to their assigned duties

Guidance for marking observations of other areas of interest data items is provided on pages 103-117 of the marking instructions document (Attachment B).

Information Statements

The Specialists will also collect information related to the “information statements” under most of the data items. These information statements are preceded by a letter for organization purposes and describe a specific observation (food safety practice) associated with the overarching data item under which it is listed.

Documenting Observations of Food Safety Practices

Using the current version of the FDA Food Code, the data collector will determine whether the observations made of the employee food safety practices or behaviors contained in the information statements were IN Compliance, OUT of Compliance, Not Observed (NO), or Not Applicable (NA). The recorded markings of the information statements are then used to determine the compliance status of the corresponding data item.

An observation is based on an evaluation of one or more occurrences of a data item or information statement at an establishment. Specific instructions for marking each data item and information statement are provided in the marking instructions document (Attachment B). The four marking options are defined as follows:

IN – means that all observed occurrences were IN Compliance with the appropriate FDA Food Code provision for the data item or information statement.
OUT – means that one or more of the observations made were OUT of Compliance with the appropriate FDA Food Code provision for the data item or information statement. An explanation of the specific criteria used for determining OUT of Compliance for each data item is to be recorded by the data collector on the data collection form.
NO – means the data item or information statement was Not Observed during the inspection. The NO marking is used when an information statement is a usual practice in the food establishment, but the practice is NOT observed during the time of the inspection.
NA – means the data item or information statement is Not Applicable. The NA marking is used when a data item or information statement is NOT a function of the food establishment.

Quantitative measurements will be made with calibrated thermocouples, heat sensitive tape or maximum registering thermometers, and chemical test strips. Quantitative temperature measurements will be recorded in the food temperature charts provided on the data collection form. Sanitization measurements will be recorded in the comment section for the specific data item observed.

Recording Food Product Temperatures

Specialists will record all food product temperatures measured during the data collection in the charts provided under data items that contain specific product temperature critical limits.

The database that will be used to record the data has been designed to provide a drop down menu for the Food Code Critical Limits for each temperature-based data item. Using the food product temperature entered by the Specialist, the database has been programmed to automatically calculate the difference between the food product temperature recorded by the Specialist and the FDA Food Code critical limit. The database system will then use this information to automatically enter the correct totals in the summary of product temperatures table depicted below. The Specialist will not have to manually complete the product temperature summary tables.

Handwashing Frequency Assessment

Specialists will record all handwashing observations during the regular data collection using the “Handwashing Frequency Assessment” located under data item #1 – Employees practice proper handwashing on the Data Collection Form (Attachment C). Over the course of the data collection visit, the Specialist will record a tally of each time an employee is observed doing the following.

Washing hands properly and when required.
Washing hands improperly.
Failing to wash hand when required.

Guidance for marking the handwashing frequency assessment is provided on pages 79-80 of the marking instructions document (Attachment B).

Assessment of Food Safety Management Systems

In addition to collecting information on compliance with the FDA Food Code, Specialists will obtain information on the extent to which food establishments have developed and implemented food safety management systems. FDA will use this information to examine the correlations, if any, between the degree to which management systems are in place and the control of foodborne illness risk factors.

The Food Safety Management System Assessment will be conducted during the same establishment visit but independent from the determination of FDA Food Code compliance for individual data items.

The assessment of food safety management systems will focus on systems related to the control of the four key foodborne illness risk factors associated with the ten primary data items.

Each randomly selected establishment will have a management system assessment conducted for ONE of the four foodborne illness risk factor areas described above. The FDA CFSAN Biostatistics Branch will randomly select the risk factor area for which a food safety management system assessment is to be conducted for each establishment.

Specialists will evaluate the presence of three key food safety management system elements (procedures, training, and monitoring) for each of the primary data items listed under the assigned risk factor:

Procedures – A defined set of actions adopted by food service management for accomplishing a task in a way that minimizes food safety risks.
Training – Management informs employees what the procedures are and teaches the employees how to carry them out.
Monitoring – Routine observations and measurements made by management to determine if procedures are being followed and maintained.

For each of these three food safety management system elements, the data collector will determine if the information provided by the establishment management adequately addresses the essential critical limits for the assigned risk factor area. A food safety management system assessment questionnaire has been developed for each of the foodborne illness risk factor areas. The questionnaire for each of the risk factor areas requires the Specialist to enter a YES or NO response for the following four statements:

Management is able to describe the critical limits for (the specific risk factor procedure or practice) as they apply to their establishment.
Management is able to describe the steps / tasks (how and when) that are performed to ensure the identified critical limits for (the specific risk factor procedure or practice) are achieved.
Management is able to identify specific employees that have been assigned the responsibility to correctly perform the (the specific risk factor procedure or practice).
Management is able to produce written materials (SOPs; posters; wall charts; wallet cards; etc.) that support the implementation of their (the specific risk factor procedure or practice) within their establishment

Using the food safety management system assessment tool, the data collector will add up the total number of “YES” responses for each of the management system elements (Procedures, Training, and Monitoring). The number of “YES” responses on the assessment tool will determine how to mark the Procedures, Training, And Monitoring sections for the data item on the data collection form.

Guidance for marking the food safety management system assessment (Procedures; Training; and Monitoring) for selected risk factor areas is provided on pages 75-78 of the marking instructions document (Attachment B).

Establishment Information

During the course of the data collection, Specialists will interview the owner/person in charge and make observations to collect information to complete the Establishment Information sections of the Data Collection Form (Attachment D). These sections include:

Establishment Information such as size, capacity and level of activity.
Manager Certification.
Employee Health Policy.

Guidance for completing the information fields associated with these sections of the data collection form is provided on pages 11(health care facilities), 17 (schools), 23 (retail food stores), and 41-54 of the marking instructions documents (Attachment B).

Corrective Actions – Observations that Pose a Significant Public Health Risk

Though industry participation in the Study is voluntary, correction action is to be obtained for observations that pose a significant public health risk. If conditions observed during the data collection visit pose a significant public health risk, the Specialist will discuss the situation with the person in charge and seek to obtain voluntary corrective action. FDA’s experience from data collections performed as part of its previous study indicate that in all but a few instances, industry responded in a cooperative and responsible manner to alleviate potential public health risks.

Should an instance occur where an observation during the data collection poses a significant public health risk and corrective action cannot be voluntarily obtained, the Specialist will contact the appropriate regulatory authority to ensure appropriate corrective actions are taken.

Exit Briefing with Person in Charge

Upon completion of the data collection, the Specialist will conduct an exit briefing with the owner or person in charge to discuss significant findings and answer any questions. No written report is left with the establishment.

Capturing the Data

Data collectors will enter the information into a web-based data platform from secure FDA computers. Before saving a record, the Specialist will conduct a quality assurance check to ensure that all required data entry fields have been completed and are accurate. The program will automatically denote an error if certain fields are inadvertently left blank or if data collection fields are completed in a manner that is inconsistent with the marking instructions for the study. The Specialists will be prompted to correct the data collection error. This quality assurance function will continue automatically until all data entry errors have been rectified.

Methods to Maximize Response Rates and Deal with Non-response

The expected response rate is 98%. The study design includes assignment of substitute establishments to a Specialist when the originally selected establishment is misclassified, closed, or otherwise unavailable, unable, or unwilling to participate. The Specialists have been provided a substitute establishment list to expedite the process of going back and forth to the Biostatistics Branch. The Specialist must select the establishment from the substitute list in numerical order at it appears on the list.

Test of Procedures or Methods to be Undertaken

The procedures were utilized in the 2015-2016 data collection period. No additional testing will be needed for the subsequent data collection.

Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data

Sampling and Statistical Methods/Data Analysis:

Marc Boyer

Mathematical Statistician

FDA/CFSAN/OFDCER/DPHB

5001 Campus Drive

College Park, MD 20740

(240) 402-1686

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Mizrachi, Ila
File Modified	0000-00-00
File Created	2021-08-26