Att 12 - Data Collection Manual

Attachment 12- EHS-Net Listeria Retail Deli Study–Data Collection Manual

I. STUDY DESIGN/METHOD

A. Sampling:

1. Stratified, random sampling

2. 50 retail delis per EHS-Net site

B. Data collection:

1. Data will be collected through: 1) manager interview, 2) manager survey, 3) worker interview, 4) structured observation, and 5) notational observation.

2. Data collection will be anonymous.

II. RECRUITMENT

A. Recruiting Procedure

We expect each EHS-Net site to collect data in 50 retail food establishments with a deli department. Each EHS-Net site will receive a list of randomly selected retail food establishments from CDC and will use this list to recruit retail deli participants for this study. When recruiting, please use the provided recruiting script (Appendix 1) and adhere to the guidelines described below.

1. Do NOT pick and choose retail establishments to contact. Contact all retail food establishments in the order they are listed. This is important; if you pick and choose retail food establishments to contact, then we no longer have a randomly selected sample.

2. Make multiple attempts to contact the retail establishments. Recruiters need to make a concerted effort to establish contact with any retail establishment that they initially tried to but could not make contact. This is important because there may be systematic differences between easy-to-contact retail delis and those that are hard-to-contact. For example, easy-to-contact retail delis may be less busy, better staffed, or better run than hard-to-contact retail delis. As these systematic differences could impact good food safety practices in the retail deli, we want to be sure that hard-to-contact retail delis are represented in our study sample as well as easy-to-contact retail delis. To ensure that every effort is made to contact all retail food establishments in the sample, please do the following:

• Unanswered calls: For unanswered calls, make a minimum of 10 attempts over two weeks to reach the establishment. Also, vary the time and day of the week when attempting to make contact with hard to reach establishments.

• Answered calls: For answered calls where no definitive responses were given with regard to study participation (i.e., Call back later, Not sure, etc.), at least 5 attempts over a minimum of 4 days should be made to re-contact the establishment in order to get a response.

• For retail food establishments with multiple unanswered calls or non-working numbers, consider using internet searches to find a working contact number.

3. Record all call attempts in the provided call tracking log. The Listeria Retail Deli Study-Call Tracking Log, located in Appendix 2, is designed to assist you in keeping track of your calls. The decision to use this log is optional, as CDC will not see this log. Please fill out the call log with the following information:

• Call date

• Call time

• Establishment information

• Attempt number (The number of attempts to contact that particular retail establishment.)

• Result or disposition of the call (e.g., No answer, Nonworking number, or Told to call back)

4. Record the final disposition of the recruitment of all of the retail establishments in the participation log. The Listeria Retail Deli Study-Participation Log, provided in Appendix 3, should be used to keep track of the responses of the retail delis during the recruitment process. Once you have made a final determination about a retail deli, fill out the participation log and send this log, in Excel format, back to CDC when your recruitment is complete.

For all retail delis, fill out the following information in the Excel worksheet:

• Sample Id: Fill in the sample ID as you are calling or recruiting each establishment.

• Response Code: In this column, fill in the response you received from the retail deli using the following codes, CL, I1, I2, I3, I4, P, R, U, and O:

• CL (CLOSED): Use this code if you determine that the retail deli is closed permanently or no longer in operation.

• I1 (INELIGIBLE-- NOT AN EHS-NET RETAIL DELI): Use this code if you determine that the retail deli does not meet the EHS-Net definition of a retail deli. An EHS-Net retail deli is defined as a deli (often located within the deli department of a retail establishment) that prepares and/or serves ready-to-eat foods to customers, where typically, there is a delay between the purchase and consumption of ready-to-eat foods. Restaurant-type delis where consumption of deli products is almost immediately after purchase (e.g., Jason’s Deli and Subway) are not EHS-Net retail delis and thus excluded from this study.

• I2 (INELIGIBLE -NO RETAIL DELI): Use this code if you determine that there is no deli in the retail establishment.

• I3 (INELIGIBLE - LANGUAGE): Use this code if you are able to contact a retail establishment but there are no managers who speak English well enough to schedule a visit or complete data collection.

• I4 (INELIGIBLE – SCHEDULING): Use this code when scheduling conflicts between you and the manager prevents you from visiting the establishment.

• P (PARTICIPATING): Use this code if you are able to contact a retail deli, it meets the EHS-Net definition of a retail deli, and the retail deli agrees to participate in the study.

• R (REFUSED): Use this code if you followed the protocol above for answered calls and made contact with a retail deli, but did not ever get a definitive response on participation. You will also use this code if you do make contact but the manager/owner declines to participate.

• U (UNABLE TO CONTACT): Use this response code if you followed the protocol above for unanswered calls and were not able to make contact with the retail deli. Also use this code if you cannot find a working phone/contact number for the retail establishment.

• O (OTHER): Something that does not fall into any of the other categories.

For any retail deli:

• Reasons for refusal/Comments: For refusing retail delis, enter any information the delis may have provided or that you were able to glean from them during the call on the reasons for refusal. Information on refusal is valuable, so if you do obtain any of this information, please record it. For all other retail delis, provide any comments that you may have obtained during the call.

For participating and refusing delis:

• Ownership: Enter whether the establishment is independently owned or belongs to a corporate chain, based on your knowledge of the retail establishment. If you cannot determine this, enter “Unsure”.

• Communication: Use this column to indicate the level of difficulty you encountered in trying to talk to someone at the establishment who could provide a definitive response about study participation. If you were able to talk to someone at the deli, enter “none” (no difficulty), “a little”, “some”, or “a lot”. For this question, some is more than a little, but less than a lot. This question is intended to be subjective, so use your judgment when answering. If you were not able to talk to someone at the retail deli, leave this cell blank.

• Call No.: Enter the total number of calls you made before you were able to talk to someone at the establishment who could provide a definitive response about study participation. If you did not make any calls (e.g., you knew that the retail deli closed recently), enter “zero”.

5. Stop recruiting once you have recruited and collected data in 50 retail delis. When you have collected data in 50 retail delis, you do not have to collect any more data, even if you have retail establishments on the list that you have not yet contacted.

B. Recruiting Script

When recruiting retail delis for participation, you will need to use the recruiting script entitled Listeria Retail Deli Study- Manager Telephone Recruiting Script. The recruiting script is provided for you in Appendix 1. Although you do not have to use this script word for word, you do need to share all of the information included in the script with the manager you are attempting to recruit.

C. Scheduling

All data collection visits should be scheduled in advance. Schedule visits at a time that will allow you to interview both a deli manager and a worker, as well as observe the deli’s environment and workers performing work-related tasks (e.g., cleaning equipment and serving a customer). You are also strongly encouraged to vary the time of your visits to these retail delis as there may be differences in the behaviors and practices observed during busy hours and those observed during slower hours. For example, the busyness of the deli workers could impact their food safety practices.

III. DATA COLLECTION

A. Preparation

CDC will provide you with a list of retail establishments. Take the data collection instruments (manager interview, manager survey, worker interview, structured observation, and notational observation) to every data collection visit. Do not write any identifying information (e.g., name, address, and phone number) about the retail establishment on any of the data collection forms. All identifying information, including directions to the establishment, should be included in a separate list. This is to ensure that the data collected from this study is anonymous. That is, it should not be possible for anyone to establish a link between the data collected to the respective retail deli.

The data collectors will need to bring additional materials to every site visit. See material checklist below:

• Sanitizing solution test kits

• Calibrated thermometers for taking refrigerator and freezer temperatures

• Wipes for cleaning your thermometers

• Flashlights for inspecting the food slicers

• Watch or any device that indicates time for recording the start and end times during the structured observation and notational observation.

B. Informed Consent

When you arrive at the retail deli, you must obtain the manager’s informed consent before you begin data collection. The script for this is included in the manager’s data instrument, entitled Listeria Retail Deli Study-Manager Informed Consent and Interview. You must also ask the manager for permission to interview a deli worker before you attempt to get the deli worker’s informed consent. In addition, you need to conduct the manager’s interview before asking the manager to complete the manager survey. However, it is up to you to decide the order in which you conduct the interviews and observations. Depending on the situation, for example, after obtaining the manager’s informed consent, you may wish to collect the data in the following order: structured observation, notational observation, worker informed consent and interview, manager interview, and then administer the manager’s survey.

C. Manager Interview

After obtaining informed consent, you will interview the manager who has authority over the retail deli, using the data instrument, Listeria Retail Deli Study-Manager Informed Consent and Interview. Please use the following guidelines.

1. General Guidelines

• For each interview question, you will read the question aloud and mark the appropriate response. Texts that should be read aloud are in bold, while words you should not read aloud are not bolded.

• Answer choices should not be read aloud unless specifically noted or indicated by bolded text.

• In instances where you are required to read the answer choices, do not read the responses of “Unsure” or “Refused.”

• Questions that allow more than one answer or response choices will be followed by instruction that says “Check all that apply.”

• Please note and follow skip patterns. You may skip certain question(s) only if the response option selected by the respondent included an instruction to skip a question (e.g., (Go to Q20)).

• You should attempt to obtain an answer to each question in the interview; however, if you believe that the interviewee is unsure or does not know the answer to a question, the response should be recorded as “Unsure,” and if the manager refuses to answer a question, the response should be recorded as “Refused.”

2. Specific Questions

• Question 1: A chain establishment is defined as one that shares both its name and operations with other establishments. Chain ownership may be private, franchise or corporate.

• Questions 2-5: If managers are unsure, tell them that their best guess based on their experience in the retail deli will suffice.

• Questions 3 and 5: The responses should be estimates of the numbers of customers served on the deli’s busiest and slowest days. Managers do not need to provide the exact number of customers served. Customers may be represented by the number of receipts.

• Question 8: For this question, shift is an indicator of the different times of the day and it does not depend on or reflect what a “shift” would mean to a particular deli. Shifts are defined as: 1^st shift – opening of deli to 10am; 2^nd shift – 10am to 2pm; 3^rd shift – 2pm to 6pm; 4^th shift – 6pm to until deli closes. The manager’s response should reflect the total number of workers who worked during a particular shift or time of day. For example, if a worker works from 11am to 7pm then that one worker would be included in the tally for the second, third, and fourth shifts.

• Question 9: The manager’s response should reflect the age of the physical building that houses the retail establishment and not the age of the deli. For example, if the building is 15 years old but the deli is a new addition that has been constructed within the past 5 years, then 15 years should be recorded as the answer. If the manager cannot determine the exact age, then ask if he/she could estimate the age in 5- or 10-year intervals (e.g., building is 5-15 years old). If age interval cannot be determine, then use other clues such as how long the manager (or others) has worked at the store (same building). If the manager has worked in the store/building for 8 years, then the answer should be recorded as “at least 8 years old”. Be sure to record the exact or estimated years in the appropriate field.

• Question 10: The response should include the time that the manager has worked at the current establishment and any other foodservice establishments prior to the current one.

• Question 11: The response should include the time that the manager has worked at the current establishment as the Person-in-Charge (PIC) or deli manager.

• Questions 9-11: These data should be reported in terms of years and months. If the time is less than 12 months, then record the data in months. Round up the month if necessary. For example, 9.5 months should be recorded as 0 year and 10 months.

• Question 12: The manager’s response should reflect duties or responsibilities associated with the current position only.

• Question 13: The response should be an average number of chubs of deli meat sold per week. Managers do not need to provide the exact number of chubs of deli meat sold per week.

• Questions 14 and 15: The responses should be based on all chubs of deli meats, regardless of type.

• Question 17: Each deli could have different means for recording or tracking the dates of opened chubs so you may receive a variety of responses. Typical responses may include: marking the dates on the wrapping of the chub, tracking dates on paper/notebook, or keeping a record of the dates electronically. Despite the method used, select yes as the answer choice if there is some type of record keeping system in place.

• Questions 20, 21, 22, 22a, 62, and 63: Deli products are foods that are prepared and/or sold in the retail deli as ready-to-eat foods (i.e., deli meats, cheeses, and deli salads). Deli meats and cheeses are often sold in chub form to be sliced to order or for use in other products such sandwiches and wraps. Some examples of deli salads include egg salad, tuna salad, and chicken salad.

• Question 20: The intent of the question is to determine how delis treat deli products that have reached their sell-by date. Do not read the response options out loud. You may need to select more than one response option depending on how the deli treats different expired products. For example, a deli may do one or a combination of these options: discard the products, re-purpose deli products by making them into other combination products, or continue to sell the deli products. This is a stand alone question and is separate from questions 21 and 22.

• Question 21: This question is asking about when the deli actually throws away or discards a deli product, and should be answered, regardless of the response(s) selected in question 20. Products may be discarded before, on the day of, or after the sell-by date. If products are not discarded or thrown away, indicate as such.

• Question 22: This question is asking for the different situation(s) in which combination products are made. For example, in addition to making the combination products on an as needed basis, some delis may make combination products when deli products are about to or have reached their sell-by date. But the question does not necessarily refer only to deli products that have reached their sell-by date. This question should be answered, regardless of the responses selected in question 20 and 21.

• Questions 24 and 26: Regardless whether the deli’s policies (hand washing and glove use) are written or not, these questions must be answered by the deli manager.

• Question 27: Ready-to-eat foods are defined as foods that do not require further cooking or heating for safe consumption (e.g., deli meats, cheeses, rotisserie chicken, deli salads, vegetables, and fruits).

• Question 28: Answer “no” if there is not a policy on preparation of sanitizing solution, or if the deli does not use sanitizing solutions.

• Question 29: Try to lead the deli manager’s response(s) in the direction of the option(s) provided by elaborating further to get those responses. For example, you may need to ask, “Do you use the instructions provided by the manufacturer or do you use instructions provided by your establishment when preparing the sanitizing solution?”

• Question 32: Answer “no” if there is no policy on wet wiping cloths or if the deli does not use wet wiping cloths.

• Questions 35, 37, 39, 41, 42a, and 55a: The answer choices, “At the beginning of each day” and “At the end of each day” refer to when the deli opens and closes, respectively.

• Questions 35 and 35a: In instances where the delis use a combination of their own cleaning and a contractor’s cleaning service for the deli floor, use the “Other” response option to record how often the contractor cleans the floor (Q35) and how (e.g., cleaning method) the contractor cleans the floor (Q35a) .

• Questions 35a and 39a: A high pressure system is defined as a hose with a pistol grip nozzle. A low pressure system is defined as a hose without a pistol grip nozzle.

• Questions 36, 42, 43, 45, 48, 50, 52, 54, and 56. Answer “No” if there is no policy with regard to each of the items or equipment, or if the deli does not use or have the item or equipment in question.

• Questions 45 and 46: A “fully cleaned” slicer is defined as one that has been disassembled into its components where they and cleaned and sanitized and then reassembled for use. You may need to emphasize the definition of “fully cleaned” to the interviewees.

• Question 47, 47a, and 47b: These questions are trying to estimate the number of Hobart 2000 slicers that are still in use at retail delis and how many delis have been contacted by Hobart. The potential issue is a plastic/magnet piece screwed on to the knife cover that may not be sealed very well, resulting in the collection food residue and microbes. If the managers are alarmed by these questions, reassure them that Hobart 2000 slicers (and other slicers) meet NSF design standards and are safe for food use, as long as they follow the instructions for cleaning, sanitizing, and maintenance as directed by the manufacture.

• Questions 55a, 60, and 61: Ready-to-eat foods are defined as foods that do not require further cooking or heating for safe consumption (e.g., deli meats, cheeses, rotisserie chicken, deli salads, vegetables, and fruits).

• Questions 60 and 61: Raw animal products refer to animal products that have not been cooked (e.g., uncooked seafood, chicken, beef, and pork).

• Questions 66a: A valid certificate is one for which the date has not expired.

D. Manager’s Survey

Please use the data instrument entitled Listeria Retail Deli Study-Manager Survey for this part of the data collection. The manager will complete this survey using pen and paper. Immediately after you complete the manager interview, ask them to take the survey. While they are taking the survey, remain nearby, but try to remain unobtrusive. This may be a good time to review your notes. We do not wish to give them the opportunity to look up the answers to the questions on the survey, but we also do not want them to feel that they are being monitored while they are taking the survey. If clarification is needed, these questions are about their retail deli’s policies or standard operating procedures, not FDA’s recommendations. Once the survey is completed, inform the manager that the correct answers, located in Appendix 4 (Listeria Retail Deli Study-Manager Answer Summary), will be provided during the debriefing. Be sure to print Appendix 4 and take it with you during data collection.

E. Worker Informed Consent and Interview

Before you can conduct the worker interview, you will first need to talk with the manager about which deli worker he or she can spare to talk with you for about 10 minutes. The deli worker will need to be someone who can speak English well enough to conduct the interview. Once you have identified the deli worker to be interviewed, attempt to find a place to interview him or her out of the sight and hearing of the manager and other deli workers. This may require that you ask the manager for some privacy. Then read the informed consent script to the deli worker and begin your interview (assuming the deli worker agrees to the interview). If the deli worker does not agree to the interview, you may ask the manager to identify another deli worker whom you could interview. You will need to interview only one worker per deli. Please use the data instrument entitled Listeria Retail Deli Study-Worker Interview for this interview.

1. General Guidelines

• For each interview question, you will read the question aloud and mark the appropriate response. Texts that should be read aloud are in bold, while those that should not be read aloud are not bolded.

• Answer choices should not be read aloud unless specifically noted.

• When reading answer choices, do not read the responses of “Unsure” or “Refused.”

• Questions that allow more than one answer will be followed by instruction that says “Check all that apply.”

• Please note and follow skip patterns. You may skip certain question(s) only if the response option selected by the respondent included an instruction to skip a question (e.g., (Go to Q20)).

• You should attempt to obtain an answer to each question in the interview, however, if you believe that the interviewee is unaware or unsure of the answer to a question, the response should be recorded as “Unsure,” and if the manager refuses to answer a question, the response should be recorded as “Refused.”

2. Specific Questions

• Question 4: The response should include the time that the deli worker has been employed at the current establishment and any other foodservice establishments, including restaurants, commercial establishments, and other delis prior to the current one.

• Question 5: The response should include the time that the deli worker has worked at the current establishment only.

• Questions 4-5: These data should be reported in terms of years and months. If the time is less than 12 months, then record the data in months. Round up the month if necessary. For example, 9.5 months should be recorded as 0 year and 10 months.

• Question 6 and 6a-6e: First locate all the available slicers in the deli and assign each slicer a numerical value from 1 to 5. Keep in mind that the worker interview and the structured observation contain different questions about the same slicers. Thus, the numerical designation of the slicers should be the same for both data collection instruments. For example, slicer number 1 in the structured observation should also be labeled as slicer 1 in the worker’s interview. Next, record the deli worker’s responses to questions 6-6e for each of the respective slicers present. You may need to point to or walk with the deli worker to the specific slicer when you are asking these questions. Also, these questions refer to the tasks that occur during the interviewee’s work shift.

• Questions 6b and 6c: A quick clean or “wipe down” of the slicers, refers to cleaning without disassembling the slicers into its components.

• Question 6d-6e: These questions refer to the process of fully cleaning the slicers. Fully cleaning and sanitizing the slicers consist of disassembling the slicers, cleaning and sanitizing the parts, and reassembling the slicers

• Questions 7: If the worker is unsure on how to answer the question, provide context by asking if inspections are done during a quick wipe down or full cleaning. Also, prompt the worker to estimate the frequency of inspections (occasionally or each time) when slicers are cleaned.

• Question 10: If the worker responded that he/she report the problem to the manager, ask the worker the whether the broken/missing seal was fixed/replaced or if he/she knew what the manager did with the worker’s report of broken or missing seals.

• Question 11: A common use area is one that is available for use by the deli staff to perform any work-related task. For example, a countertop or table used for preparing deli salads and assembling deli sandwiches would be considered a common use area. Also, remind the worker that chicken salad may include rotisserie chicken that are prepared at the deli.

• Question 11a: Try to lead the deli worker’s response(s) in the direction of the option(s) provided by elaborating further to get those responses. For example, you may need to ask, “Do you assemble the salads at the retail deli or do they come pre-made?”

• Question 13: Once the interview is completed, inform the worker that the correct answers to questions 13a-13e, located in Appendix 5 (Listeria Retail Deli Study-Worker Answer Summary), will be provided during the debriefing. Be sure to print out Appendix 5 and take it with you during data collection.

F. Structured Observation

For this portion of the data collection, you will need to make direct observations of the deli’s environment and its workers. In order to answer the questions, you will need observe the deli’s environment and workers, take temperatures, test sanitizer solution, inspect food slicers, and sketch a layout of the retail deli. Please use the data instrument entitled EHS-Net Listeria Retail Deli Study-Structured Observation for this part of the data collection.

1. General Guidelines

• During the structured observation, attempt to remain as unobtrusive as possible and keep interaction with workers to a minimum.

• The questions in this section may require you to make specific and general observations about the deli workers as they are working to prepare food, handle utensils, clean surfaces, etc.

• Do not read the questions aloud. Questions are bolded for easy distinction against the response choices.

2. Specific Questions

• Questions 3 and 3a: Locate all available sinks in the deli and assign each sink a numerical value from 1 to 5. Observe and record the different tasks that are being performed at each sink. Also, indicate whether there are soaps and/or hand drying supplies at or near each sink. Hand drying supplies may include paper or cloth towels or electric hand dryers.

• Question 4: Use the thermometer that you have brought with you to measure the temperatures for the display cases, refrigerators, and freezers that are being used by the deli department. If the display case is fully separated or partitioned into smaller compartments where there is no air flow between the compartments then you’d need to take the temperature for each of the compartments in the display case. Deli products are foods that are prepared and/or sold in the retail deli as ready-to-eat foods (i.e., deli meats, deli cheeses, and deli salads). Deli meats and cheeses are often sold in chub form to be sliced to order or for use in other products such as sandwiches and wraps. Some examples of deli salads include egg salad, tuna salad, and chicken salad.

• Questions 4 and 5: Refrigerators and freezers that store deli products may not be located within the deli areas; you may need to go outside of the deli area to find them.

• Questions 5a, 5b, 7, and 8: Ready-to-eat foods are defined as foods that do not require further cooking or heating for safe consumption (e.g., deli meats, cheeses, rotisserie chicken, deli salads, vegetables, and fruits).

• Questions 5a, 5b, and 8: Raw animal products refer to animal products that have not been cooked (e.g., uncooked seafood, chicken, beef, and pork).

• Question 6: You may need to ask the deli manager to see the records that indicated that the temperatures of incoming foods were taken and recorded as they were delivered to the retail deli.

• Questions 8 and 8a: These questions rely on your experience as inspectors to make the determination about the potential for direct cross contamination from raw animal products. This include observing practices and behaviors, such as preparation and handling of raw products and associated equipment, to determine if there is evidence of direct cross contamination to other food/surface/equipment.

• Question 11: Using the sanitizer test kits that you brought to the retail deli, measure the concentration of one of the sanitizing solutions that is currently in use in the retail deli. If there is more than one sanitizing solution (e.g., buckets) in use, randomly select one solution to test for its concentration. You may need to ask a deli employee to point out all of the sanitizing solutions that are in use at the present time. You should select “Yes” only if the sanitizing solution is at the proper concentration according to your test kit. If the sanitizing solution is outside the range of the proper concentration (e.g., too low or too high), it is not at the proper concentration and you should select “No”. Select “Could not observe” if you cannot find a sanitizing solution container in the deli; it may be that the solution is stored in another area.

• Questions 15a-15e: At this point, if you have not assigned each slicer in the retail deli a numerical value from 1 to 5, please do so. If each slicer already has a numerical designation from the worker’s interview, please use the same numerical designation here. That is, each slicer should have the same numerical value assignment in both the structured observation questionnaire and the deli worker questionnaire. Information on the make, model, and year of manufacture of the slicers should be displayed on the slicers. If you could not observe the make, model, or year on the slicers then mark as such below each response choices.

• Questions 15a1-15e1: For each available slicer, observe the entire slicer and mark any issues that you see. Here, we are trying to gauge the overall condition of the slicers.

• Sketching the deli area: Use the generic sketch provided to draw the layout of the deli area. Keep in mind that the top of the sketch represents the front area or customer service area of the deli. Appendix 6 (Example of Sketching of the Deli Area) provides a sample sketch of a deli area.

  • Be sure to record the Establishment ID in the top left corner of the sketching form so that the sketch could be linked to the rest of the data for each retail deli.

  • For items 1-11, use geometric shapes such as ovals, squares, and circles to depict the actual items in the deli. For each geometric shape drawn, be sure to label the item being depicted.

  • If there is more than one of the same item, be sure to label them as such. For example, if there are 3 sinks in the deli area, then each geometric shape should be labeled as sink #1, sink #2, and sink #3 on the sketch. Numerical assignment of the items (e.g., sinks and slicers) in the sketch should be the same as those assigned in the other data instruments (e.g., structured observation and deli work interview).

  • These items do not need to be drawn to scale but the sketch should accurately reflect the relative placement of these items within the deli.

  • For entry and exit points, please draw arrows to indicate directions of traffic. If a particular opening is for entry traffic only, then indicate this with an arrow pointing into the deli area. Draw an arrow pointing outside of the deli area for openings that only allow exiting traffic. For two-way traffic, use two arrows to indicate both entry and exiting traffic.

  • For items such as refrigerators and freezers that may be stored in a location separate from the deli-proper area, be sure to indicate as such by drawing these items outside of the provided generic box.

G. Notational Observation

To complete the notational observation component, you will observe 1 to 3 deli workers performing work-related tasks (e.g., slicing meat for a customer). Keep in mind that you need to record 100 observations per deli worker. Do not record over 100 actions from the same person. For example, if an establishment has 3 deli workers present, 100 actions from each worker should be observed for a total of 300 actions. However, if an establishment only has 2 deli workers present, 100 actions from each should be reported for a total of 200 observations. These observations should take 30 to 60 minutes per worker depending on how busy the deli is at the time of your visit. Please use the data instrument entitled Listeria Retail Deli Study-Notational Observation to complete the notational observation component of this study.

1. Using the Code Key

A code key is included in the data instrument, Listeria Retail Deli Study-Notational Observation, to assist you in recording all of the observed actions on paper as quickly as possible. It is a list of shorthand codes that was used in previous notational observations. Here are your options with regard to the code key:

• You can use the original code key as it is provided.

• You can develop your own supplemental code key. This should be a short set of codes to be used as a supplement to the original code key.

• You can develop your own comprehensive code key. This should be a comprehensive set of codes that you would use exclusively for the notational observation.

• If you decide to develop your own set of codes, supplemental or comprehensive, you will need to:

  • Notify CDC of your decision.

  • Have a code key in place prior to pilot data collection. As you collect the pilot data, you may need to edit these codes along the way in order to suit your purpose.

  • Send the final, completed list of codes to CDC as soon as you are finished with pilot data collection.

Regardless of your decision, make sure there is an official code key in place and that you are familiar with these shorthand codes, prior to the start data collection.

2. Notational Observation of deli worker(s)

Select one worker in the deli who is beginning to perform a task (e.g., serving a customer or cleaning and sanitizing equipment) and for whom you can observe actions unobtrusively. Start your observations when the deli worker begins the task.

• Use the Practice Notational Observation Form to record the first 10 to 15 minutes of actions.

  • For the first worker observed, use the first 10 to 15 minutes of the notational observation to get used to observing the deli worker’s actions and recording these actions with the shorthand codes.

  • You do not need to do the practice notation on subsequent workers. However, you should consider bringing at least one Practice Notational Observation Form for each worker just in case you do need to practice on multiple workers.

  • Once you feel comfortable with the practice portion of the notational observation, you may begin with the actual data collection.

• Use the Notational Observation Form for data collection on the first deli worker.

  • Fill out the Date, Page number, Establishment ID, and Deli worker number.

  • Record the time that you start observing the deli worker under the Action No. column by action number 1.

  • Focus on hand actions. Do not record “feet actions” or actions performed by foot such as operating foot pedals at sinks or trash cans.

  • After recording the first 20 actions, use a new sheet to record the next 20 actions. Again, fill out the Date, Page number (e.g., Page #: 2), Establishment ID, and Deli worker number. Each worker should have at least 5 sheets of recorded actions, totaling 100 sequential actions observed. Each line on the Notational Observation Form represents one action. For example, if a deli worker touches her cell phone, opens the deli case, picks up a chub, puts it on the food prep table, grabs some deli tissue and slices ham onto the deli tissue, we would expect the sequences of actions to resemble those depicted in Table 1.

Table 1. Example Notational Observation Form

Action No.	Action Sequence	Notes
1-TIME: 12:30 pm	TCH CP
2	OPN CAS
3	PUP CHB
4	PON FPT
5	GB DT
6	SLI HM on DT

• If the worker leaves the observation area and you can no longer see or follow him/her (e.g., worker enters the restroom), enter “Break” in the Notes column, in the same row that the last action was recorded. Resume your observation when the worker returns to a point where you can observe him or her.

• If the worker’s shift ends before you complete your observation, enter “Worker Off” in the Action No. column, leave all other columns blank, and end observation. Be sure to record the time that the observation ended in the Action No. column.

• If the worker moves around the deli, continue to observe him or her to the extent that you can without being obtrusive. For example, you may try to pick a place to stand that allows you to see the employee working in several different places. If the worker moves from one place to another permanently and you cannot observe him or her in the new place, then follow the worker to the new place. Do not follow workers to the restroom. We are trying to remain as unobtrusive as possible.

• Once you are finished with observing and notating 100 actions for a particular worker, record the end time at the bottom of the Notational Observation Form You will only need to do this on the last sheet or Page #5.

• Repeat these steps for each additional deli worker being observed, for up to 3 deli workers.

• You will need to use separate notational observation forms for each worker being observed. Also, each worker will require multiple notational observation forms. We have provided you with one set of the notational observation forms which includes the instructions, the Practice Notational Observation Form, the Notational Observation Form, and the code key. You will need 1-3 copies of the Practice Notational Observation Form and a minimum of 5 copies of the Notational Observation Form for each worker observed. You should bring at least 15 copies of the Notational Observation Form to each establishment in case you can observe 3 deli workers.

• As you fill in each notational observation form, number the sheets consecutively in the space provided so that you can keep track of the pages.

H. Debrief

At the end of your visit, talk with the managers briefly about what you saw. Offer both the manager and worker the correct answers to the surveys, and ask if they have any questions. Mention any unsafe practices that you would like to bring to their attention. Thank them for allowing you into the retail deli.

IV. Other

A. Answers to possible questions.

“What are you doing?”

We are trying to learn more about retail delis’ food safety policies and practices. So we are watching deli workers perform work-related tasks, and taking notes on what they do.

“Why?”

In the long run, we hope to figure out how to make it easier for deli workers to do what they need to do.

“What are you going to do with your notes?”

They will be entered into a computer and analyzed along with other responses. We are not collecting any information that might identify you in any way.

B. Data Entry

1. General Data

Use EHSNIS (Environmental Health Specialist Network Information System) to enter the data collected from the following data instruments, Listeria Retail Deli Study-Manager Informed Consent and Interview, Listeria Retail Deli Study-Manager Survey, Listeria Retail Deli Study-Worker Interview, and Listeria Retail Deli Study-Structured Observation (except for the deli sketch). When you enter data into the information system, please keep in mind the following:

• Questions that are marked with asterisks are required questions and must be answered in order for you to save the forms (data).

• Be mindful of Automatic Disconnect Time (at top left of form) to avoid losing data that you have entered. Each page will disconnect after 45 minutes of inactivity.

• For questions that allow you to answer or enter the data as free text, please keep the responses short and concise, do not use hard returns, and do not use symbols (non-alphanumeric).

• For questions that use the “Check all that apply” format, you need to select a response during data entry for each of the options presented. For each option that was checked in the data instrument, select the “Yes” radio button. For each option that was not checked in the data instrument, select the “No” radio button.

2. Structured Observation Data– Sketch of the Deli Area

When entering the data from the structured observation instrument in EHSNIS, you will receive an Evaluation ID for each retail deli. This Evaluation ID should also be recorded in the top right corner of the deli sketch form. It is very important that you record the Evaluation ID on the sketch as it will be used to link the sketch to the rest of data from the data instruments for each retail deli. Create a digital scan of your sketches of the deli area from each of the establishments and send them to CDC via email.

3. Notational Observation Data

Use the provided Excel worksheets to enter the data collected from the Listeria Retail Deli Study-Notational Observation data instrument. This process requires that you transcribe your shorthand of the observed actions (from the Notational Observation Form) into full text in the Excel worksheet. You can enter the data manually or use the translation macros provided:

• Data entry using the translation macros – Use the Excel worksheet entitled Notational Observation Data Entry-Translator if you would like to perform data entry using the Excel macros to translate your shorthand into full text. Please follow these steps:

  • Step 1: Enter the Evaluation Id. For each recorded action, enter the EHSNIS-generated evaluation identification number under the Evaluation Id column.

  • Step 2: Enter the deli worker number. For each recorded action, enter the deli worker number (e.g., 1-3) under the Deli Worker Number column.

  • Step 3: Enter the action number. The Action Number column identifies the observed actions in sequential order (e.g., 1-100), and it should correspond with the Action No. column in the Notational Observation Form.

  • Step 4: Enter the shorthand codes for each observed action. Under the Write the Code column, enter the exact shorthand (from the code key) of each of the corresponding observed actions that was recorded in the Notational Observation Form (Action Sequence column).

  • Step 5: Translation into full text. After each entry of your shorthand codes under the Write a Code column, press Enter and a translation of the shorthand codes into full text will automatically populate in the corresponding cell under the column Get a Translation.

  • Step 6: Notes. Transcribe any corresponding notes from the Notation Observation Form into the Excel worksheet under the Notes Column.

  • Additional comments:

    • For the preposition on, you need to type it as “on” and do not use a shorthand code.

    • You can use any word that is not a code (e.g., #1) and it will be used without correction. That is, if you type “#1” in the Write the Code column, it would populate as “#1” in the Get a Translation column.

    • Entries are not case sensitive.

    • You can use up to 8 codes per entry in the Write the Code column.

• Manual data entry – Use the Excel worksheet entitled Notational Observation Data Entry-Manual if you would like to enter your data by manually entering the shorthand codes and their corresponding full text translation. Please follow these steps:

  • Step 1: Enter the Evaluation ID. For each recorded action, enter the EHSNIS-generated Evaluation Identification number under the Evaluation ID column.

  • Step 2: Enter the deli worker number. For each recorded action, enter the deli worker number (e.g., 1-3) under the Deli Worker Number column.

  • Step 3: Enter the action number. The Action Number column identifies the observed actions in sequential order (e.g., 1-100) and it corresponds with the Action No. column in the Notational Observation Form.

  • Step 4: Enter the shorthand codes for each observed action. Under the Code column, enter the exact shorthand (from the code key) of each of the corresponding observed actions that was recorded in the Notational Observation Form (Action Sequence column).

  • Step 5: Translation into full text. For each of the corresponding shorthand codes, translate and enter the full text under the column Translation.

  • Step 4: Notes. Transcribe any corresponding notes from the Notation Observation Form into the Excel worksheet under the Notes column.

• Once you are finished with data entry and the data have been reviewed by your administrator, please send the Excel worksheets back to CDC. Be sure to send the data intermittently as you are done with data entry. Do not wait until you are finished with 50 establishments before sending the data to CDC.

Appendix 1: Listeria Retail Deli Study-Manager Telephone Recruiting Script

Hello this is __________ with the _________ Health Department. We are working with the Centers for Disease Control and Prevention (CDC) on a research project designed to help us better understand retail delis food safety policies and practices. Your deli was chosen at random to be in this project. The study involves an interview with a manager and a food worker, and an observation of the deli. Your participation is voluntary, but we would really appreciate your participation, as it will help us understand the difficult issues retail delis face. If you don’t want to be part of the study or if you change your mind later, nothing will happen to you. Whether you are part of the study or not will not affect your deli’s score on any health inspection. Also, if you decide to participate, your name and your establishment’s name will not be recorded on the data collection form, nor will they be included in any reports.

Having said that, I need to let you know that at any time during the survey if I see something that is an imminent health hazard, such as no power, no water or sewage on the floor, I will need to stop the study and report the problem to the local agency that has jurisdiction over your grocery store.

Could we schedule a time to come out and speak with a deli manager, a worker, and observe your deli for a while?

􀂅 No

􀂅 Yes

YES NO

Great! I’ll visit your deli for about an hour and fifty minutes. Keeping in mind that a manager responsible for supervising the deli and a worker who handles food, operates and clean the deli’s food slicers, and who speaks English needs to be available for the interview, when would be a convenient time to schedule the visit? Date: Time: _________________________ Please call me if you need to cancel or re-schedule, or if you have any questions. If you have any questions at a later time, you can contact: (Local Contact Name and Telephone Number).

I understand. Thank you for your time.

Public reporting burden for this collection of information is estimated to average 3 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: CDC/ATSDR Information Collection Review Office, MS D-74; 1600 Clifton Road NE, Atlanta, Ga. 30333; ATTN: PRA (0920-0792)

Appendix 2: Listeria Retail Deli Study-Call Tracking Log

Call Tracking Log
Date	Time	Establishment. & Phone	Attempt #	Result/Disposition of Call

Appendix 3: Listeria Retail Deli Study-Participation Log

	Instructions
	For all retail delis in which contact is attempted: Enter the participation code in Question 1.
	For any retail deli: Enter any information on refusals or comments in Question 2.
	For participating retail delis: Enter the participation code, any comments, and the answers to Questions 3-5.
	For refusing retail delis: Enter the participation code, any information on reasons for refusal, and the answers to Questions 3-5.
	To be answered for all retail delis.		To be completed if you have any information on reasons for refusals or to make comments	To be answered for all participating and refusing retail delis.
		CL=Closed I1=Not an EHS-Net retail deli I2=Ineligible - no deli in est. I3=Ineligible- Language I4- Scheduling P=Participating R=Refused U=Unable to contact O=Other	Reasons for refusal / Comments	Retail deli's ownership type: Independent/Chain	Difficulty communicating: None/A little/ Some/A lot	Number of calls before answer: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, etc.
Obs	Sample ID	1. Response Code	2.Reason for refusal	3. Ownership	4. Communication	5. Call No.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

Appendix 4: Listeria Retail Deli Study-Manager Answer Summary

1. What should you do to ensure that the sanitizing solution you are using on a food prep surface will work properly?

Answer: When mixing the solution, use a test kit to check its concentration. Using a test kit to ensure that the sanitizer solution is at the concentration recommended by the manufacturer is the best way to ensure that the solution will work properly.

2. What are the correct steps used to properly sanitize a prep table?

Answer: Wash, rinse, sanitize, air dry. These cleaning and sanitizing steps are recommended by the FDA and will help ensure clean food surfaces.

3. Common bacteria often found in food can grow well at what temperature range?

Note: There will be multiple answers to this question as some EHS-Net sites have different guidelines regarding the range of the temperature “danger zone.” Each EHS-Net site is listed beside the correct answer for their jurisdiction.

Answer:

• Between 41°F and 135°F (5°C and 57°C) (California and Rhode Island)

• Between 41°F and 140°F (5°C and 60°C) (Minnesota, New York City, and Tennessee)

• Between 45°F and 140°F (7°C and 60°C) (New York)

The temperature range that best supports the growth of bacteria is known as the temperature danger zone. The length of time a food remains in this temperature range greatly impacts the amount of bacterial growth.

4. What is the proper procedure for washing your hands?

Answer: Wet hands with warm water. Apply soap. Vigorously scrub hands and arms for 10-15 seconds. Rinse hands. Dry hands with clean paper or cloth towel. These hand washing steps are recommended by the US Food and Drug Administration (FDA) and will help ensure clean hands.

5. Deli workers should change their gloves:

Answer: All of the above. Changing gloves after they become dirty or torn, taking out the trash, and handling raw meat and before handling ready-to-eat foods will help ensure you don’t contaminate food or objects with dirty gloves.

6. Deli workers must wash their hands before they start work and after:

Answer: All of the above. Washing hands after using the restroom, sneezing, coughing, or using a tissue, and handling raw meat, poultry or seafood will help ensure you don’t contaminate food or objects with dirty hands.

7. When storing foods you should always place ready-to-eat foods that do not need additional cooking before eating on the lower shelves and raw animal products on the higher shelves.

Answer: False. To avoid contact between ready-to-eat foods and fluids from raw animal products, ready-to-eat foods should always be stored above raw animal products.

8. When a deli worker is sick with certain symptom(s), which of the following symptom(s) should you, the manager, be concerned about with regard to excluding that worker from handling food?

Answer: Vomiting and diarrhea. These are common symptoms of many illnesses caused by germs in food. If the deli worker has these symptoms, he/she could potentially transmit these germs to customers through food.

Appendix 5: Listeria Retail Deli Study-Worker Answer Summary

1. One of the best ways to dry your hands after hand washing is with clean paper towels.

Answer: Yes. The US Food and Drug Administration (FDA) says that clean paper towels, clean/unused cloth towels, or hand drying devices should be used for drying hands following a hand wash.

2. Germs grow best in the temperature “danger zone” between 41°F and 135°F.

Note: There will be multiple answers to this question as some EHS-Net sites have different guidelines regarding the range of the temperature “danger zone.” Each EHS-Net site is listed beside the correct answer for their jurisdiction.

Answer: Yes.

• Between 41°F and 135°F (5°C and 57°C) (California and Rhode Island)

• Between 41°F and 140°F (5°C and 60°C) (Minnesota, New York City and Tennessee)

• Between 45°F and 140°F (7°C and 60°C) (New York)

The temperature range that best supports the growth of bacteria is known as the temperature danger zone. The length of time a food remains in this temperature range greatly impacts the amount of bacterial growth.

3. For a proper hand washing, deli workers must scrub their hands and arms for about 4 or 5 seconds.

Answer: No. The FDA says that food workers should scrub their hands and arms for at least 10-15 seconds to properly clean.

4. Deli workers should wash hands between glove changes.

Answer: Yes. The FDA says that food workers should wash their hands in between glove changes to ensure clean hands.

5. It’s okay to use hand sanitizer instead of washing your hands when the deli is busy.

Answer: No. According to FDA’s Food Code (2009), hand sanitizer should be applied only to hands that have been properly cleaned.

Appendix 6: Example of Sketching of the Deli Area

19