SHI_Part A_final_4_16_19

Consumer Research on the Safe Handling Instructions Label for Raw and Partially Cooked Meat and Poultry Products and Labeling Statements for Ready-to-Eat and Not-Ready-to-Eat Products

OMB No. 0583-NEW

Supporting Statement

A. Justification

A.1. Circumstances Making Collection of Information Necessary

Safe handling instructions are required if the meat or poultry component of a product is raw or partially cooked (i.e., not considered ready-to-eat [RTE]) and if the product is destined for household consumers or institutional uses (9 CFR 317.2(l) [meat] and 9 CFR. 381.125(b) [poultry]). The U.S. Department of Agriculture, Food Safety and Inspection Service (USDA, FSIS) established the Safe Handling Instructions (SHI) label for raw and partially cooked meat and poultry products in 1994 (54 FR 14528). Consumer focus groups were conducted to inform the design of the SHI label (Teague & Anderson, 1995; Teague & Anderson, 1993). Since that time, the required design of the SHI label has not been changed.

In response to inquiries from consumer groups and other stakeholders for more information about potential changes to safe handling instructions requirements, FSIS gathered input from members of academia, industry, and consumer stakeholders in November 2013. FSIS presented these suggestions to the National Advisory Committee on Meat and Poultry Inspection (NACMPI) in January 2014. When the SHI label was developed in 1994, minimum internal temperature requirements for determining doneness varied by product. Given product and label size limitations and varying endpoint temperatures, FSIS concluded that “Cook Thoroughly” was the only simple, single statement appropriate to use for all products (54 FR 14538). FSIS now recommends four minimum internal temperatures: one for all poultry (165°F), one for ground meat (160°F), one for all whole-muscle meat (145°F and hold for 3 minutes), and one for fish (145°F). With only four temperature recommendations, the information could be more easily incorporated into the SHI label. Other possible changes to the SHI label include incorporating updated icons and providing a web link or phone number for more information (NACMPI, 2014; Murphy-Jenkins, 2014).

The NACMPI Subcommittee on Food Handling Labels recommended that FSIS pursue changes in the existing SHI label and conduct consumer research to determine the effectiveness of any revisions to the SHI label (NACMPI, 2014). In November 2014, FSIS conducted a strategic planning session to elicit input from FSIS senior leadership on potential revisions to the SHI label, the impact any revisions may have on consumers and industry, and pitfalls to consider. The findings from this session underscored the need to conduct consumer research to determine consumers’ reactions to the current SHI label and potential revisions.

In 2015, FSIS conducted six consumer focus groups (OMB No. 0583-0166; 11/30/2017) to evaluate understanding of the current SHI label and responses to possible revisions. The focus groups revealed that consumers would find certain revisions to the SHI label useful. Participants suggested changes to improve comprehension and adherence to recommended safe handling practices (e.g., add recommendation to use a food thermometer and endpoint temperatures for different cuts of meat and poultry) (Cates et al., 2016).

Additionally, although FSIS has issued guidance to the industry on the labeling of uncooked boneless, breaded chicken products that may appear RTE because of their cooked appearance (USDA, FSIS, n.d.), there have been reports of illnesses associated with these products even when the labels follow the guidance. In May 2016, the National Chicken Council (NCC) submitted a petition requesting that FSIS establish regulations for the labeling and validated cooking instructions for not-ready-to-eat (NRTE) stuffed chicken breast products. In their petition, the NCC also suggested that research be conducted to examine consumers’ handling of NRTE stuffed chicken breast products as well as their understanding of relevant labeling statements and validated cooking instructions. The American Frozen Food Institute, an industry trade association, and the Safe Food Coalition, a coalition of consumer advocacy organizations, submitted comments in support of the petition (NCC, 2016; American Frozen Food Institute, 2016). Prior to this petition and comments, during the March 2016 NACMPI meeting, the committee reviewed and discussed whether FSIS should pursue proposing mandatory features on the label of processed NRTE products that may appear to be fully cooked (e.g., are breaded or have grill marks). The committee recommended that FSIS require statements such as “Raw, “Uncooked,” or “Ready to Cook” on the labels of raw products that may appear ready-to-eat so it is clear that these products require cooking to a proper internal temperature before eating (USDA, FSIS, 2016). The committee also recommended that FSIS conduct consumer research to understand the optimal messaging and design of packaging to ensure consumers properly understand that NRTE products that may appear to be fully cooked need to be cooked for lethality. The committee stated that such labeling may help consumers properly distinguish between NRTE products, which require a lethality step, and RTE products, which do not require a lethality step; thus, the committee stated that this labeling may help consumers safely prepare NRTE products. Specifically, the committee suggested that FSIS conduct consumer research to evaluate the effectiveness of possible locations for point of purchase labeling information and various color options, fonts, and other display options.

To assess whether revisions are needed to the SHI label required on all raw and partially cooked meat and poultry products and to evaluate the ability of consumers to properly discern between NRTE and RTE products and how labeling for these products can be improved, FSIS is requesting approval for a new information collection to conduct consumer behavior research. This research will include a web-based experimental study and a behavior change study, which includes three components: an observational meal preparation experiment, an eye-tracking study, and in-depth interviews (IDIs). The research will help inform whether potential revisions to the current SHI label are needed and assess whether a label revision would be likely to improve consumer behaviors related to safely preparing raw and partially cooked meat and poultry products. The study will also collect information on consumer use and understanding of labeling for RTE and NRTE meat and poultry products.

Specifically, the consumer research will provide empirical evidence as to whether consumers' visual attention is greater for a revised SHI label compared with the current SHI label and whether a revised SHI label results in greater adherence to certain safe handling practices (e.g., thermometer use and handwashing) that food safety experts have ranked as most important to public health compared with the current SHI label. As further described in Section A.16, if the results of the consumer research suggest that a revised SHI label could improve consumer safe handling practices relative to the current SHI label, then analysis will be conducted to estimate the benefits of the revised label. The results of the consumer research will be used in a predictive model to estimate the potential changes in foodborne illness that might occur as a result of improved consumer safe handling practices associated with a revised SHI label. The value of the benefits from estimated potential reductions in foodborne illness from use of the revised label will be compared with the cost to industry to voluntarily update their product packaging to incorporate the new label. Because it is not known what percentage of industry would voluntarily adopt the new label or the percentage of consumers who would change their behavior, we will conduct analyses for different levels of adoption (e.g., 25%, 75%, and 100). If the results of this analysis show that the potential for benefits in predicted reduction of foodborne illness exceed the cost to industry, FSIS plans to allow regulated establishments to update their product packaging to use the revised label (e.g., through the issuance of guidance documents) or continue to use the current SHI label.

A.2. How, by Whom, and Purpose Information Is to Be Used

F SIS has contracted with RTI International and its subcontractor, North Carolina State University (NCSU), to conduct the consumer behavior research. The primary objective of this research is to collect information to determine possible revisions to the current SHI label (see insert) that will improve consumers’ adherence to recommended food safety practices for raw and partially cooked meat and poultry products. A secondary objective is to evaluate the ability of consumers to properly discern between NRTE and RTE products and how labeling for these products can be improved. To address the study’s objectives, the study team will undertake the following consumer research:

• Conduct a web-based experimental study to assess consumer salience or attention to 27 SHI labels with alternative formats, icons, and messages designed for specific safe handling instructions. Appendix Q describes the process used to develop the revised SHI labels and presents the 27 SHI label options to be tested in the web-based experiment. The results of the web-based experimental study will be used to identify the three best-performing label options to test in the subsequent behavior change study (see Figure A‑1).

• Conduct a behavior change study to assess and compare consumer behavior in response to the current SHI label (control) and three alternative SHI labels. Participants in the behavior change study will take part in the following three activities:

  • An observational meal preparation experiment while wearing a mobile eye-tracking device in a test kitchen to evaluate the effectiveness of alternative SHI labels relative to the current SHI label on consumers’ adherence to recommended safe handling instructions.

  • An eye-tracking study to obtain quantitative data to measure visual salience in response to the current and alternative SHI labels using mock food packages (i.e., stimuli).

  • IDIs to gather information on participants’ knowledge and perceptions regarding their handling of RTE and NRTE meat and poultry products, in particular their ability to properly discern between RTE and NRTE products and to ensure that NRTE products that may appear to be ready-to-eat are thoroughly cooked.

Figure A‑1 shows each stage of the study and the decision criteria for moving from each stage of the study. Each stage of the study is described in more detail below.

Figure A-1. Stages and Decision Criteria for Consumer Research on the Safe Handling Instructions (SHI) Label

Conduct Web-Based Experimental Study (n = 3,600 respondents; recruited using national nonprobability-based sample from opt-in web panel)	Test 27 SHI label options (3 formats x 3 instruction sets x 3 icon sets) (see Appendix Q for information on how the labels were developed). Use statistical analysis to identify the three label options that best capture respondents’ attention and the three rationale statements that best convey the importance of complying with the SHI label (see Appendix R). Experts in risk communication will assign one of the three rationales to each of the three SHI labels with the highest visual salience score to create the final three labels for testing in the behavior change study.
Conduct Behavior Change Study (n = 480; conduct study in four locations in different parts of country to provide geographic diversity and reach Spanish-speaking individuals; recruit participants using convenience sampling (social media, community outreach, and outbound telephone recruiting from market research databases)
(1) Meal preparation experiment	Test three SHI label options (treatment) and the current SHI label (control) for a total of four experimental conditions. Conduct statistical analysis comparing the label adherence scores for the four conditions to identify the one label that best leads to adherence with recommended safe handling practices for handwashing, using a food thermometer, keeping raw meat/poultry separate from RTE foods, and cleaning/sanitizing kitchen surfaces and equipment (see Appendix S).
(2) Eye tracking study	Test the same three SHI label options (treatment) and the current SHI label (control) (i.e., four experimental conditions). Conduct statistical analysis to identify the one label that best captures respondents’ attention (see Appendix S).
(3) In-depth interviews	Provide qualitative information for interpreting results of statistical analyses.
Determine Optimal Label Redesign	Use multivariate analysis to examine association between attention to SHI label (from eye-tracking study) and label adherence scores (from meal preparation experiment) to determine optimal design for SHI label revision.  This analysis will provide empirical evidence on which, if any, of the three SHI label options best captures attention and leads to adherence to safe handling practices compared with the current SHI label.
Conduct Cost-Benefit Analysis	Conduct cost-benefit analysis if results of consumer research suggest that a revised SHI label would improve safe handling practices relative to current SHI label. Use results from meal preparation experiment (i.e., adherence to recommended practices) in predictive model (using the Food and Drug Administration’s FDA-iRisk platform) to estimate potential changes in foodborne illness as a result of revised SHI label (see section A.16). Compare the value of the benefits from predicted reductions in foodborne illness with the cost to industry if it voluntarily adopts the revised SHI label.
Final Decision	If the value of the benefits from reduction in foodborne illness exceeds the cost to the industry, FSIS plans to allow regulated establishments to update their packaging to use the revised label (e.g., through the issuance of guidance documents) or continue to use the current SHI label.

Web-Based Experimental Study

RTI will subcontract with Lightspeed (www.lightspeedresearch.com), a provider of a non-probability-based, opt-in national online consumer research panel to program the survey instrument and administer the data collection for the web-based experimental study.¹ We are using nonprobability sampling because it is not feasible to draw a random probability-based sample of the entire population given time and cost considerations. Section B.1 describes the limitations of the study recruitment procedures, specifically that inferences cannot be made to the U.S. population based on the results of the study. Respondents will be selected from Lightspeed’s consumer panel, which consists of approximately 1.5 million adults who are double opted-in. The double opt-in process is as follows. First, a prospective respondent clicks on a link from a panel ad and is directed to the panel registration survey. Second, the prospective panelist must complete the panel registration survey (which collects demographic and household information) and must pass several validation checks (e.g., verify postal address) and agree to the website’s Terms and Conditions and Privacy Policy to become panel members.

The study population for the web-based experimental study is the U.S. general population of adults (aged 18 or older) who are members of the Lightspeed panel. Although some subpopulations are at greater risk of contracting foodborne illness (e.g., young children, the elderly, pregnant women), the SHI label is intended to reach the general population with information on safe handling practices; thus, the study is not limited to at-risk populations and will include a diverse sample of participants who vary on specific demographic characteristics as described in more detail below.

Lightspeed will set inbound quotas to obtain a diverse sample with respect to education, age, race, and English vs. Spanish speaking across all study participants. Previous research suggests that consumer safe handling practices vary based on certain demographics such as education, age, race, and ethnicity (see, for example, Patil, Cates, & Morales, 2015; Kosa et al., 2019; Quinlan, 2013; and Redmond & Griffith, 2003). Selected panelists will receive an email invitation (Appendix A) and interested panelists will be screened for eligibility. The study will include 3,600 participants with approximately n = 133 exposed to each of 27 potential SHI labels created by fully crossing the three primary study features—label shape, safe handling instruction text, and safe handling icons—each of which will have three options.² Appendix Q describes the process used to develop the proposed revised SHI labels and presents the 27 SHI label options to be tested in the web-based experiment. All of the revised SHI label options provide information on recommended temperatures for FSIS-regulated products. The study will take up to 20 minutes to complete and be available in English and Spanish (see Appendix B). To encourage response, up to three e-mail reminders will be sent to nonrespondents (see Appendix C).

The primary aim of the experimental study is to test 27 mock SHI labels that vary visual design elements to determine which labels are most salient to consumers. Label salience (i.e., participant degree of attention to the label) will be assessed using a limited-time exposure approach with cued recall questions. The proposed experimental design with random assignment to one of 27 experimental condition will provide an unbiased assessment of which among the 27 labels are more likely to attract consumer attention. The data from the experimental study will be analyzed to identify the three SHI labels that best attract respondents’ attention for further testing in the behavior change study. Additionally, the experimental study will provide information on the top three performing rationale statements for testing in the behavior change study (as described in Appendix R).

Behavior Change Study

We are using nonprobability sampling for the behavior change study because it is not feasible to draw a random probability-based sample of the entire population given time and cost considerations. Section B.1 describes the limitations of the study recruitment procedures, specifically that inferences cannot be made to the U.S. population based on the results of the study. The data collection will be conducted at test kitchens, each similar in design and layout, in four locations across the country: (1) Wake, Orange, and/or Durham Counties, NC; (2) Brazos County, TX; (3) Yolo County, CA; and (4) Providence County, RI. The four locations were selected to provide geographic diversity with data collection in three of the four Census regions. The Census regions are groupings of states and the District of Columbia that subdivide the United States for the presentation of Census data and are often used in other studies to ensure geographic diversity. Additionally, areas of the United States that have a relatively large percentage of Hispanics were selected so that the percentage of Spanish-speaking participants in the study sample is equal to that of the U.S. population (13% of U.S. adults speak Spanish at home).³ The North Carolina counties (South Atlantic Division, South region) were selected because NCSU has three test kitchens located in this area. The Texas (West South Central Division, South region) and California (West region) locations were selected because of the availability of suitable test kitchens and to reach Spanish-speaking individuals (25.1% Hispanic for Brazos County, TX, and 31.5% for Yolo County, CA). Providence County (Northeast region) was selected to reach an urban population in the Northeast because of the availability of a suitable test kitchen, and to reach Spanish-speaking individuals (22.1% Hispanic).⁴

Within each of the four locations, study participants will be randomly assigned to one of four conditions: three alternative SHI labels or the current SHI label as a control. A total of 480 adults will participate in the behavior change study; of these, 360 will participate in North Carolina and 120 will participate in the other three locations (40 per location). Most of the data collection will take place in North Carolina because of the greater availability of test kitchens, the logistics of conducting this type of data collection, and budgetary constraints. Because we are using an experimental design (i.e., random assignment to one of four study conditions) with the aim of estimating causal effects, our study will have a high degree of internal validity but may not be as generalizable to the broader U.S. population.

Completing all components of the behavior change study (observational meal preparation experiment, eye-tracking study, and IDIs) will take between 2 and 2.5 hours. Convenience sampling will be used to recruit participants for the behavior change study. Quotas will be set to obtain a diverse sample of the U.S. general population with respect to education, age, race, ethnicity, and whether there is a child aged 0 to 17 years in household.⁵

Study participants will be recruited in the four locations using convenience sampling by posting ads in social media outlets (see Appendix D), sending emails to Expanded Food and Nutrition Education Program participants (North Carolina location only) (see Appendix E), and posting notices about the study in various locations (see Appendix F). If necessary, additional outreach may be conducted with hard-to-reach populations such as Hispanics, older adults, and high-school educated individuals (e.g., reaching out to parents and guardians of the Juntos program in North Carolina [program that helps Latinos have more success in middle and high school] and community groups who work with cooperative extension programs). Additionally, the study team will work with local market research facilities in each location to use outbound recruiting to recruit individuals with specific demographics that may be challenging to recruit using social media (e.g., individuals with a high school education or less and older adults). Market research facilities maintain databases of people who have expressed interested in participating in research, including information on demographic characteristics, so that targeted recruiting can be conducted. RTI/NCSU has used similar recruiting methods for conducting observation studies for FSIS (OMB No. 0583-0169: In-Home Food Safety Behaviors and Consumer Education: Annual Observational Study, expiration 6/30/2020) and has enrolled study participants with diverse demographic characteristics (e.g., for a recent observational study 14% of participants were Hispanic or Latino, 25% were 50 years or older, and 24% had a high school diploma/GED).

Recruitment materials will direct prospective participants to either call or email a study team member to be screened for eligibility or access a web link that will host the screening questionnaire (see Appendix G). Study enrollment will include contact by phone (see Appendix H) to schedule an appointment with individuals who meet the eligibility criteria followed by a confirmation email or letter (up to three emails) (see Appendix I) and a reminder call 1 or 2 days before the scheduled appointment (see Appendix J).

Upon arrival at the test kitchen, participants will provide informed consent (Appendix K) and be fitted with the mobile eye-tracking device. Participants will then watch an instructional video with information on the study and listen to additional instructions provided by a study team member including a walk-through of the test kitchen (see Appendix L). Participants will be provided with the recipes and ingredients needed to prepare three dishes: (1) gluten-free pasta and meatballs using frozen, preformed raw meatballs (2) gluten-free pasta and meatballs using raw ground beef, and (3) cherry tomato garnish. Participants will be informed the dish is for an individual who is on a gluten-free diet to avoid eliciting socially desirable behavior. Both the packages of the raw ground beef and the raw, frozen, preformed meatballs will bear the assigned SHI label. Video recording of food handling and meal preparation will begin as soon as the participant enters the test kitchen and will end after the participant leaves the test kitchen. Participants’ cleaning and sanitizing of equipment and environment before and after meal preparation will also be recorded. Meal preparation is expected to take 50 to 80 minutes to complete.

Trained coders will watch the videos and code behaviors using an observation rubric (see Appendix M) to evaluate participants’ adherence to the safe handling instructions listed on the SHI label (e.g., use a food thermometer and wash hands after handling raw meat). The coded data will be used to calculate a label adherence score (see Appendix S). The videos captured by the mobile eye-tracking device will be reviewed to determine (1) the frequency and amount of time for viewing the SHI label and other labeling information during meal preparation and (2) if a thermometer was used, the measured endpoint temperature of the meat product.

In addition to wearing the eye-tracking device during the meal preparation experiment, participants will complete an eye-tracking study after meal preparation is complete. The eye-tracking study will provide quantitative data on several visual and attentional processes related to consumer interaction with and use of labeling on RTE and NRTE products. The eye-tracking study will address two primary research questions:

• Which version of the SHI label is most often attended to when consumers look at a busy food package?

• Can consumers properly distinguish between RTE and NRTE products?

The eye-tracking study will collect data from all 480 individuals participating in the meal preparation experiment, with 120 participants viewing each of the SHI labels (current SHI label or one of three alternatives as assigned for the meal preparation experiment). Participants will view four NRTE products (two of which appear RTE) that bear the assigned SHI label (and other required labeling statements), as well as two RTE products bearing the required labeling statements (for a total of six mock products or stimuli). Data collectors will use a script to direct participants’ attention to each product and to complete several tasks to determine which version of the SHI label is most often attended on a meat and poultry package and to assess whether participants can properly distinguish between RTE and NRTE products that appear to be ready to eat (see Appendix N for the script). This study component will take up to 30 minutes to complete. The eye-tracking data will be used to measure several potential outcomes, including visual observations of the area of interest (AOI), such as time to first view and total view time, distribution of attention, and percentage of attention.

The final component of the behavior change study will be the IDIs, which will provide context to the quantitative data and elaborate the process underlying the role of labeling and food safety messaging on cooking practices. Data collectors will use a semistructured interview guide to conduct the interviews, which will take up to 30 minutes to complete (see Appendix N). The interview will include follow-up questions to the meal preparation experiment to assess reasons for following or not following the recommended safe handling instructions and a set of questions to understand how participants determine whether a meat or poultry product needs to be cooked for lethality.

Statistical analyses comparing the label adherence scores for the control (i.e., current SHI label) and treatment groups (three alternative SHI label versions) will be conducted to identify the label that may most effectively lead to consumers following the safe handling practices on the label. The results of this analysis, along with findings from the eye-tracking study and the IDIs, will provide information on whether revising the SHI label would improve consumers’ safe handling practices when preparing raw or partially cooked meat or poultry products. The Agency will use the findings of this consumer research, along with findings from a cost-benefit analysis, to determine if revisions to the current SHI label are needed

A.3. Use of Improved Information Technology

Web-Based Experimental Study

The experimental study will use web-based data collection in lieu of in-person data collection, which will greatly reduce the burden on participants because they will not be required to travel to a central location to complete the study. This approach will also expedite the timeliness of data collection because a web-based study will take up to 4 weeks to administer versus up to 4 to 5 months for an in-person study. Furthermore, the use of web-based data collection with participants located throughout the United States will allow the study to reach a more diverse study population than would otherwise be possible using an in-person approach and will be significantly less costly to implement.

Behavior Change Study

Most participants will be recruited via social media and have the option to complete a web-based questionnaire for screening, which is less burdensome and more cost-effective than requiring all prospective participants to call research staff to be screened for eligibility. Prospective participants who complete the web-based questionnaire and who meet the eligibility requirements for study participation will still need to be contacted via phone by research staff to schedule an appointment for completing the study.

As part of the behavior change study, all participants will wear a mobile eye-tracking device to collect information on consumers’ attention to product labels during the observational meal preparation experiment. All participants will also wear the eye-tracking device while interacting with the study stimuli (mock meat and poultry products) to collect quantitative data on visual salience for the current vs. alternative SHI labels. This technology is nonintrusive (i.e., similar to wearing glasses) and allows participants to interact with the experimental product stimuli freely.

A.4. Efforts to Identify and Avoid Duplication

FSIS is currently conducting an observational study (OMB No. 0583-0169: In-Home Food Safety Behaviors and Consumer Education: Annual Observational Study) to evaluate its communication and outreach efforts on consumer behaviors. Although the methods for this study are similar to the proposed observational meal preparation experiment component of the behavior change study, the outcomes of the two studies are different. Thus, the findings from the annual observational study cannot be used to answer the research questions of interest in the proposed information collection. Based on a review of the current literature, the Agency concluded that the existing knowledge base on the SHI label does not meet the Agency’s informational needs.

A.5. Methods to Minimize Burden on Small Business Entities

No small businesses will be involved in this collection.

A.6. Consequences of Less Frequent Data Collection

This is a one-time data collection. Without this study, FSIS will not have the needed information to assess whether revisions are needed to the SHI label required on all raw and partially cooked meat and poultry products and if revisions are required for the labeling of NRTE products that appear to be fully cooked. The lack of information would impede the Agency’s ability to provide consumers with more useful and actionable information on how to safely prepare raw and partially cooked meat and poultry products at home. Such information could lead to improved consumer practices and thus potentially help reduce foodborne illness in the United States.

A.7. Special Circumstances Relating to the Guidelines of 5 CFR 1320.5 that Would Cause the Information Collection to be Conducted in a Manner:

• requiring respondents to report information to the Agency more often than quarterly;

• requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

• requiring respondents to submit more than an original and two copies of any docu­ment;

• requiring respondents to retain records, other than health, medical, government contract, grant-in-aid, or tax records for more than 3 years;

• in connection with a statistical survey, that is not designed to produce valid and reli­able results that can be generalized to the universe of study;

• requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

• that includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

• requiring respondents to submit proprietary trade secret or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information's confidentiality to the extent permitted by law.

This information collection fully complies with 5 CFR 1320.5(d) (2). There are no special circumstances associated with this information collection that would be inconsistent with the regulation.

A.8. Consultations with Persons Outside the Agency

In accordance with the Paperwork Reduction Act, FSIS published a 60-day notice requesting comments regarding this information collection request (83 FR 34101; 07/19/2018) and received a total of two comments. The comment from the Center for Foodborne Illness Research & Prevention (CFI) and the Consumer Federation of America (CFA) was generally supportive of the information collection and identified specific information to include on a revised SHI label. Many of the modifications requested by CFI and CFA will be considered in the design of the alternative labels to be tested in the web-based experimental study (e.g., providing end-point temperatures, including a website address for more information, and providing instructions to use a thermometer to verify the product has reached the recommended internal temperature). The results of the web-based experimental study and the behavior change study will be used to inform whether a revised SHI label results captures consumers’ attention and results in greater adherence to certain safe handling practices (e.g., thermometer use and hand washing) compared with the current SHI label.

Additionally, the CFI and the CFA encouraged FSIS to quickly design and implement rules that effectively prevent consumers from confusing raw and cooked products. The proposed study will also collect information on consumer understanding of labeling on RTE and NRTE products and assess whether consumers understand the difference between RTE and NRTE products, specifically to ensure proper cooking of NRTE products. The Agency will use the results of this research to determine whether modifications are needed to current labeling requirements and guidelines for NRTE products that are raw but specifically appear to be fully cooked.

The comment from the North American Meat Institute (NAMI) was generally supportive and identified two concerns with the information collection. First, NAMI noted that if the study shows negligible change in consumer behaviors, a labeling change is not justified and the current SHI label should not be amended. Following the completion of the consumer research, the Agency will use the study results in a cost-benefit analysis to assess whether the benefits from revising the SHI label (i.e., the potential reduction in foodborne illness) exceeds the cost to industry to revise the label. The second concern noted was that the Agency should allow flexibility, especially with regard to incorporating minimum internal temperatures, and consider the feasibility of labeling options when developing the consumer research. The label designs to be tested in the consumer research will be limited to options that are considered realistic for industry to implement so as to not place undue burden on industry.

The National Agricultural Statistics Service also reviewed the information collection and made supportive comments.

A.9. Payments to Respondents

Web-Based Experimental Study

To encourage panelists to participate in surveys, Lightspeed offers its panel members reward points. Upon completing a survey, points are deposited immediately into a panelist’s account. The number of points awarded is based on survey length, complexity, and incidence rate. For this study, respondents will receive 100 points. Panelists may redeem their accumulated points for online gift certificates, merchandise, and PayPal gift card deposits.

Behavior Change Study

We understand that the OMB guidance about incentives for participation in research is based on the principles of the 2006 memo “Guidance on Agency Survey and Statistical Information Collections.” We propose providing each participant a $100 gift card and a small gift (food thermometer valued at $5.38 and magnet valued at $0.23) to maximize the show rate for the behavior change study and to improve data quality. Additionally, participation in the behavior change study will require substantial commitment and investment of time on the part of the participant, in that they must make a commitment to attend the study at a certain time on a specific date. Participation also requires participants to travel to a designated location, with the average commuting time in the United States metropolitan areas estimated at about 26.1 minutes (U.S. Census Bureau, 2017) and may also require that the participant obtain child care for a fee. Thus, providing incentives has long been considered a standard practice in conducting consumer research.

Table A‑1 provides a breakdown of the cost to participate in the behavior change study by whether the participant has a child(ren) requiring child care for a paid fee. Although the cost to participate varies depending on whether paid child care is needed (from $28.45 to $85.53), we propose to offer all participants the same incentive amount ($100) to avoid introducing selection bias that might occur by offering different incentive amounts to individuals with and without children in their households.

The proposed $100 incentive amount is in line with the industry standard. These industry-standard stipends help ensure that respondents can be recruited efficiently and ensure their arrival and participation in the study. These standards also exist to provide fair compensation for costs incurred by participants while participating in the study (i.e., travel and child care expenses). In addition to covering reasonable costs of participation, payment to participants is necessary to ensure that enough respondents from the target population participate in the study. Payment to participants encourages potential participants to agree to allocate their time to the study and maintain that commitment on the day of the research.

Table A-1. Estimated Cost to Participants of Taking Part in the Behavior Change Study by Households with and without Children

Cost Component	Estimated Number of Units	Unit Cost	Total Cost
Households with Children
Cost to travel to/from test kitchen	52.2 milesa,b	$0.545/milec	$28.45
Cost of child care during travel time (1 hour round trip) and attending study (15 minutes before appointment to park, up to 2.5 hours for the study, and 15 minutes after study to checkout and return to vehicle)	4.0 hours	$14.27/hourd	$57.08
Total			$85.53
Households without Children
Cost to travel to/from test kitchen	52.2 milesa,b	$0.545/milec	$28.45
Total			$28.45

^a Source: https://www.census.gov/library/visualizations/interactive/travel-time.html

^b The average commute in a U.S. metropolitan areas is an estimated 26.1 minutes to a designated location. Assuming participants travel 60 miles per hour, the total number of roundtrip miles is 52.2 miles.

^c Source: http://www.gsa.gov/portal/content/100715

^d Source: https://www.care.com/c/stories/2423/how-much-does-child-care-cost/

Offering no incentive or a smaller incentive could potentially exclude sections of the population who cannot participate in the study, either because of the cost of child care and/or travel or the cost of missing work. Excluding sections of the population would limit the information that would be gained through the study and potentially bias the information needed to address the research questions of interest, thus negatively affecting data quality. Moreover, the $100 incentive payment proposed is consistent with what OMB has approved for other consumer food safety studies, when adjusted for the estimated participant burden, for example, OMB No. 0583-0169: In-Home Food Safety Behaviors and Consumer Education: Annual Observational Study; OMB No. 0583-0166: Professional Services to Support Requirements Gathering Sessions for Safe Food Handling Instructions (SHI); OMB No. 0583-0141: Consumer Research, Assessing the Effectiveness and Application of Public Health Messages Affecting Consumer Behavior Regarding Food Safety; and OMB No. 0584-0173: Food Safety Behaviors and Consumer Education: Focus Group Research.

We anticipate that without the gift card incentive and gift, we would need to screen more people to achieve the desired cooperation rate. The current estimated annualized burden for the participant screening is about 8 minutes (0.133 hours) for the study. Without any incentive, we expect that twice the number of individuals would need to be screened so that the total burden for screening would be doubled (452 vs. 226 hours). The cost to respondents and the federal government would increase accordingly.

A.10. Assurance of Confidentiality

The privacy of study participants will be ensured by using an independent contractor to collect the information, by enacting procedures to prevent unauthorized access to participant data, and by preventing the public disclosure of the responses of individual participants.

Web-Based Experimental Study

As part of their registration process, Lightspeed requires panelists to agree to their privacy policy, which includes privacy standards, rights, and information usage. A link to Lightspeed’s privacy policy is always included in study email invitations, is accessible via their panel website, and can be found at http://www.lightspeedresearch.com/privacy-policy/. Lightspeed complies with the research industry standards from the following organizations: the European Society for Opinion and Market Research, Insights Association (formally the Council of American Survey Research Organizations and MRA), Advertising Research Foundation, American Marketing Association, Market Research Society, and Association of Market and Social Research Organisations.

Lightspeed has in place physical, electronic, and managerial procedures to ensure its networks and applications are highly secure and client data are protected. Physical security features include entrances requiring security clearances, secure smart card access, on-site security officers, video surveillance, generator-backed power supply, fire suppression systems with early warning smoke detection, and an HVAC-controlled environment.

Lightspeed uses several layers of network security to prevent unauthorized network access to systems and data. Antivirus software is installed on all servers and workstations. Internet security is provided by the following layered network access architecture: multilayer firewall architecture; data center systems managed via private, firewalled, backend access; a variety of threat monitoring, detection, and intrusion prevention systems (IPS) measures deployed throughout the network; and automated monitoring of network and server performance for WAN, LAN, and production servers.

All data are secured on database servers that only reside on private, backend servers that are behind layered firewall architecture. Data are never stored on a public network or outside the data tier. Relational database management systems (RDBMS) access is strictly controlled and limited to only a few authorized users whose access is limited to the minimum necessary to accomplish administrative tasks. Web and application servers communicate with the RDBMS only via a private network segment with a multilayer firewall architecture in place. Access control is provided to secure data directories. All client-specific data are stored in restricted access data directories controlled by access control lists.

Lightspeed will not share personal information regarding panel members with any third party without the participant’s permission unless it is required by law to protect their rights or to comply with judicial proceedings, court orders, or other legal processes. RTI and FSIS will not have access to panel members’ personal information. No personally identifying information will be included in the data files delivered to the Agency.

Information regarding informed consent, including assurances of data privacy and security, will be provided on the first screen of the survey (see Appendix B). RTI’s Institutional Review Board (IRB) reviewed and determined the study is exempt from IRB review (see Appendix O). A Privacy Impact Analysis (PIA) is not required by FSIS or RTI International (the primary contractor on this project).

Behavior Change Study

The only information in identifiable form (IIF) that will be obtained are the participants’ names, phone numbers, and email or mailing addresses for scheduling the appointment for the behavior change study, mailing confirmation letters, and making reminder phone calls. NCSU will maintain this IIF information. These personal identifiers will not be linked to participant data and will not be shared with FSIS or RTI.

Participation in the behavior change study is voluntary, and participants will be advised that their responses will be treated in a secure manner and will not be linked to their names. The digital video and audio tapes will be stored on a password-protected share drive, accessible only to project staff.

Assurances of data privacy and security are documented in the informed consent form (see Appendix K). NCSU’s and RTI’s IRBs reviewed and approved the study protocol and instruments (see Appendix P). A PIA is not required by FSIS or RTI International (the primary contractor on this project).

A.11. Justification for Questions of Sensitive Nature

Participants in the web-based experimental study and behavior change study will not be asked any questions that are personal or sensitive in nature. For both the web-based experimental study and the behavior change study, participants will be asked if they or any household members have been diagnosed with cancer, diabetes, or other conditions that weaken the immune system. Individuals will not be asked for a specific diagnosis. Immunocompromised individuals are considered at risk for foodborne illness; thus, it is important to collect information on their or their caregivers’ food handling behaviors

A.12. Estimates of Respondent Burden

The total estimated burden for the web-based experimental study is 3,623.3 hours, and the total estimated burden for the behavior change study is 1,491.9 hours, for a total of 5,115.2 hours (see Table A‑2). The annualized cost to all respondents for the proposed information collection is $92,687.42 (5,115.2 x $18.12 per hour) (the 2017 U.S. median hourly wage rate⁶).

Web-Based Experiment Study

The total burden for the web-based experimental study is 3,623.3 hours. To achieve 100 completed surveys during the pretest, Lightspeed will send email invitations to 1,700 randomly selected panel members. To achieve 3,600 completed surveys during the full-scale study, Lightspeed will send email invitations to 70,000 randomly selected panel members. The invitation email for the pretest and the full-scale survey is expected to take 2 minutes to read (0.033 hour). The survey is expected to take 20 minutes to complete (0.333 hour).

Table A-2. Estimated Annual Reporting Burden

Portion of Study	Appendix(s) for Data Collection Instrument or Form		No. of Respondents	Annual Frequency per Response	Total Annual Responses	Hours per Response		Total Hours
Web-Based Experimental Study
Pretest invitation	A		1,700	1	1,700	0.033 (2 min.)		56.7
Pretest	B		100	1	100	0.333 (20 min.)		33.3
Survey invitation	A		70,000	1	70,000	0.033 (2 min.)	2,333.3
Survey	B		3,600	1	3,600	0.333 (20 min.)	1,200.0
Total							3,623.3
Behavior Change Study
Recruitment information	D, E, F		—	—	—	—	—
Screening questionnaire	G		1,695	1	1,695	0.133 (8 min.)	226.0
Appointment phone script, confirmation emails, reminder phone script	H, I, J		565	1	565	0.117 (7 min.)	65.9
Consent form and video		K, L	480	1	480	0.167 (10 min.)			80.0
Meal preparation, eye-tracking, and IDI		L, N	480	1	480	2.333 (140 min.)			1,120.0
Total									1,491.9
Total									5,115.2

Behavior Change Study

The total burden for the behavior change study is 1,491.9 hours. The study will be advertised via social media, emails, and postings in grocery stores and supplemented with outbound recruiting. Prospective participants will complete a screening questionnaire by phone or via a web link to determine eligibility. We estimate that 1,695 individuals will complete the screener and 565 (33%) will be eligible and subsequently contacted by phone to schedule an appointment. Of these, we estimate that 480 (85%) will participate in the behavior change study. Each screening is expected to take 8 minutes (0.133 hour). It is expected to take each participant a total of 7 minutes (0.117 hour) to read or listen to each appointment call/confirmation email/reminder call. It is expected to take each participant 10 minutes (0.167 hour) to read the informed consent form and watch the instructional video and up to 140 minutes (2.333 hours) to complete the behavior change study, which includes an observational meal preparation experiment (50 to 80 minutes), an eye-tracking study (30 minutes), and an IDI (30 minutes).

A.13. Capital and Start-Up Costs and Subsequent Maintenance

No capital, start-up, operating, or maintenance costs are associated with this information collection.

A.14. Annual Cost to Federal Government

The estimated total cost to the federal government for this information collection is $1,478,415. The costs arise from the time spent by the contractor to develop the study design and materials, collect the data, analyze the data, and prepare and deliver a final report.

A.15. Reasons for Changes in Burden

This is a new information collection.

A.16. Tabulation, Analysis, and Publication

The planned schedule for the information collection survey is shown in Table A‑3. Once OMB approval is received, we will begin the data collection activities for the web-based experimental study. The contractor will provide FSIS a report that summarizes the study methods and results within 45 days of completion of the data collection. Appropriate statistical analyses will be used to analyze the survey data and identify the three label designs that best capture respondents’ attention and the three rationale statements that best convey the dangers of foodborne illness. Appendix R provides the analysis plan for the web-based experimental study and describes the development of the final three labels for testing in the behavior change study.

Table A-3. Project Schedule

Date	Activity
Within 15 days following OMB approval	Begin data collection for web-based experimental study
Within 60 days following OMB approval	Complete data collection for web-based experimental study
Within 90 days following OMB approval	Complete summary report on web-based experimental study
Within 30 days following report on web-based experimental study	Begin data collection for behavior change study
Within 180 days following report on web-based experimental study	Complete data collection for behavior change study
Within 240 days following report on web-based experimental study	Complete summary report on behavior change study

Within 30 days of providing the web-based experimental study summary report, data collection activities for the behavior change study will begin. The contractor will provide FSIS with a report that summarizes the study methods and results within 90 days of completing the data collection. For the meal preparation experiment, the contractor will conduct statistical analyses comparing the label adherence scores for the control (current SHI label) and three treatment groups to identify the label that may most effectively lead to consumers following the safe handling practices on the label. All eye-tracking data will be reviewed, coded, and processed to produce eye-tracking metrics for each AOI, including total time spent viewing each AOI, which will be used in additional statistical analyses and to create heat maps and gaze plots. Finally, data from the IDIs will be analyzed using QSR International’s NVivo 11 qualitative data analysis using a thematic content analysis approach. Appendix S provides the analysis plan for the three components of the behavior change study.

Using the findings from the behavior change study, we will construct regression models to examine the association between attention to SHI labels (from the eye-tracking study) and the label adherence scores (from the meal preparation experiment). These data will provide empirical evidence on which, if any, of the three SHI label options best captures respondents’ attention and is most likely to encourage consumers to follow recommended safe handling practices compared with the current SHI label.

If the results of the consumer research suggest that a revised SHI label could improve consumer safe handling practices relative to the current SHI label, then a cost-benefit analysis will be conducted. The results of the meal preparation experiment (i.e., use of recommended practices) will be used in a predictive model developed by risk assessment scientists at NCSU to estimate the potential changes in foodborne illness as a result of a revised SHI label, and the value of the benefits from predicted reductions in foodborne illness will be compared with the cost to industry if they voluntarily adopt the revised SHI label. The FDA’s FDA-iRISK risk assessment platform (FDA, 2018) will be used to construct the predictive model, which has been used extensively by FDA and undergone external peer review (FDA, 2015). The FDA-iRISK model will be used, via scenario analysis, to estimate the likely impact of the revised SHI label on disease burden metrics (e.g., disability-adjusted life-years), a proxy for human disease risk. The predictive modeling will include three steps:

Step 1: Populate FDA-iRISK with input values (and associated distributions) for the most important FSIS-regulated hazard–commodity pairs subject to SHI labeling using data from the peer-reviewed literature, government-supported databases (e.g., the National Center for Health Statistics’ National Health and Nutrition Examination Survey), internal data maintained by FSIS, relevant data collected in FSIS’s ongoing observational meal preparation experiments (study results have been publicly presented and manuscripts are in draft form), and alternative estimates of the proportion of consumers likely to change their behavior (or benefit from a food preparers’ change in behavior). Thus, the model will rely on the best data sources available and assessed for robustness.

Step 2: Based on the four SHI instructions in the revised label tested in the behavior change study estimate the impact of specific behavioral change on the FDA-iRISK outputs by scenario analysis.

Step 3: Translate the results from Steps 1 and 2 to provide food attribution values for inclusion in the cost-benefit analysis.

At this time, the Agency does not plan to revise the current SHI regulations to require industry to use a revised label; instead, FSIS would give regulated establishments the option to continue to use the current SHI label or modify their packaging to use the revised label (e.g., through the use of guidance documents), if the results of the cost-benefit analysis show that the benefits in predicted reductions of foodborne illness exceed the cost to industry.

Dissemination of the study results may include internal briefings, presentations, and reports and posting on FSIS’s website and potentially a manuscript for publication in a peer-reviewed journal.

A.17. OMB Approval Number Display

The OMB approval and expiration date will be displayed on all materials associated with the study. No exemption is requested.

A.18. Exceptions to the Certification

There are no exceptions to the certification.

References

American Frozen Food Institute. (2016, August 17). Letter to USDA, FSIS on petition number 16-03; Petition to establish regulations for the labeling and validated cooking instructions for not-ready-to-eat stuffed chicken breast products that appear ready-to-eat. Retrieved from https://www.fsis.usda.gov/wps/wcm/connect/802587a4-f31f-4293-bad6-5eafa120ef49/16-03-Support-Ltr-081716.pdf?MOD=AJPERES

Cates, S. C., Kosa, K. M., & Muth, M. K. (2016). Professional services to support requirements gathering sessions for safe food handling instructions: Focus group research. Final Report (Contract No. AG-3A94-13-0003, Order AF3A94-K-14-0060). Prepared for U.S. Department of Agriculture, Food Safety and Inspection Service, Office of Public Affairs and Consumer Education. Research Triangle Park, NC: RTI International.

Kosa, K. M., Cates, S., Brophy, J. E., Godwin, S., Chambers, D., & Chambers, E. (2019). Older adults and parents of young children have different handling practices for raw poultry. Journal of Food Protection, 82(2), 200-206.

Murphy-Jenkins, R. (2014, January). Food safety handling labels. Washington, DC: USDA, FSIS. Retrieved from http://www.fsis.usda.gov/wps/wcm/connect/9adeea60-4d52-4eb6-a15a-8e15b6873192/NACMPI-Food-Safety-Handling-Labels-010714.pdf?MOD=AJPERES

National Advisory Committee on Meat and Poultry Inspection. (2014). Report and recommendations based on FSIS proposed questions. Subcommittee on Food Safety Handling Labels. Washington, DC: NACMPI. Retrieved from http://www.fsis.usda.gov/wps/wcm/connect/70d59a26-32bb-4d25-9050-462b0b0cacb5/Food-Safe-Handling-Labels-NACMPI.pdf?MOD=AJPERES

National Chicken Council. (2016, May 24). Letter to USDA, FSIS on petition to establish regulations for the labeling and validated cooking instructions for not-ready-to-eat stuffed chicken breast products that appear ready-to-eat. Retrieved from https://www.fsis.usda.gov/wps/wcm/connect/e63c4755-2965-4872-b834-fcd798f58488/16-03-National-Chicken-Council.pdf?MOD=AJPERES

Patil, S., Cates, S., & Morales, R. (2005). Consumer food safety knowledge, practices, and demographic differences: Findings from a meta-analysis. Journal of Food Protection, 68(9), 1884–1894.

Quinlan J .J. (2013). Foodborne illness incidence rates and food safety risks for populations of low socioeconomic status and minority race/ethnicity: A review of the literature. Int J Environ Res Public Health, 10(8), 3634–52.

Redmond, E. C., & Griffith, C. J. (2003). Consumer food handling in the home: a review of food safety studies. Journal of Food Protection, 66, 130–161.

Teague, J. L., & Anderson, D. W. (1993). Consumer focus groups on safe-handling labels for the pathogen reduction effort. Final report prepared for USDA, Food Safety and Inspection Service. Research Triangle Park, NC: RTI International.

Teague, J. L., & Anderson, D. W. (1995). Consumer preferences for safe handling labels on meat and poultry. The Journal of Consumer Affairs, 29(1), 108–127.

U.S. Census Bureau. (2017). Average one-way commuting time by metropolitan areas. Retrieved from https://www.census.gov/library/visualizations/interactive/travel-time.html

U.S. Department of Agriculture, Food Safety and Inspection Service. (n.d.). Labeling policy guidance uncooked, breaded, boneless poultry products. Retrieved from https://www.fsis.usda.gov/wps/wcm/connect/6d7b7f70-e11b-4861-adc8-6f3269c3eeec/Labeling_Policy_Guidance_Uncooked_Breaded_Boneless_Poultry_Products.pdf?MOD=AJPERES

U.S. Department of Agriculture, Food Safety and Inspection Service. (2016). NACMPI Reports—2016. Retrieved from https://www.fsis.usda.gov/wps/portal/fsis/topics/regulations/advisory-committees/nacmpi-reports

U.S. Food and Drug Administration (FDA). (2018). FDA-iRISK version 4.0. Retrieved from https://irisk.foodrisk.org/

U.S. Food and Drug Administration (FDA). (2015). Peer review report: External peer review (letter) of FDA-iRISK model and associated library of commodity/hazard combinations—Chemical hazards. Retrieved from https://www.fda.gov/downloads/scienceresearch/specialtopics/peerreviewofscientificinformationandassessments/ucm452654.pdf

1 Part B.1 provides additional information on the Lightspeed panel, including the demographics of the panel and information on the limitations of the panel and the interpretability of the results.

2 Part B.1 provides the power analysis that was conducted to determine the sample size.

3 U.S. Census Bureau. (n.d.). 2010-2014 American Community Survey 5-year data profiles. Retrieved from https://www.census.gov/acs/www/data/data-tables-and-tools/data-profiles/2014/

4 Percentage of population that is Hispanic available at http://www.city-data.com/.

5 Part B.1 provides additional information on the limitations of using convenience sampling and the interpretability of the results.

6 Bureau of Labor Statistics, U.S. Department of Labor. Occupational Employment Statistics. Accessed 5/31/2018, [https://www.bls.gov/oes/current/oes_nat.htm]

9