Examining Consumer Responses to Restaurant Menu Labeling Requirements
OMB No. OS-0990-XXXX
Supporting Statement – Section A
Submitted: October 13, 2015; revised March 28, 2016, revised April 13, 2016
Program Official/Project Officer
Amber Jessup, Ph.D.
Senior Economist
U.S. Department of Health and Human Services
Office of the Assistant Secretary for Planning and Evaluation
200 Independence Avenue SW, Washington DC 20201
(202) 690-6621
The Assistant Secretary for Planning and Evaluation (ASPE) seeks approval from the Office of Management and Budget (OMB) under the Paperwork Reduction Act of 1995 to implement the study described below to address questions about consumer responses to restaurant menu labeling requirements. This is a new information collection request.
As part of Section 4205 of the Patient Protection and Affordable Care Act, the FDA recently published a rule requiring that calorie information be included on restaurant menus. The final rule includes a regulatory impact analysis as well as references to previous studies. There are several issues that remain unresolved. Most studies evaluated menu labeling as if it were a weight loss intervention, rather than to assess whether it allows consumers to make better choices in accordance with their preferences. Some customers may want to buy larger, rather than smaller, meals. Contradictory claims about likely effects may reflect differences in how labeling interacts with restaurant type and context as well as consumer preferences (the relative importance of “good value”, “healthfulness”, portion size”, etc) and this project can help to disentangle those interactions. This information is important to design effective consumer education and outreach programs.
Under Contract Number contract number HHSP23320095649WC (TO number HHSP23337037T), ASPE seeks to understand how consumer choices in the presence or absence of calorie information vary by consumer preferences, sociodemographics, and restaurant type.
The proposed study is expected to provide timely and valuable feedback that will inform consumer education and outreach about menu-labeling, which should in turn inform Americans’ dietary behavior. The intent of the study is not to use the information to modify any parts of the final rule.
We evaluate the potential effect that the calorie labeling may have on consumer choices when ordering at restaurants through an internet-based experiment. By using a simulation, a discrete choice type experiment incorporating a range of restaurant types through an online-interface, we can capture some of the heterogeneity in the potential impacts of calorie labels on consumers’ choices. Our survey will provide the first comparable data on how informed consumer choice may vary across different restaurant types, including fast food, fast casual, sit down restaurants, and also include settings such as movie theaters and ice cream parlors.
The study builds on a large literature that was reviewed as part of the project development to assure there is no duplication. The most recent systematic review was in the Journal of the Academy of Nutrition and Dietetics (Sinclair et al., 2014). Other relevant publications include Lee-Kwan et al. (2014), Auchincloss et al. (2013); Breck et al. (2014); Bowers et al., (2014). The 2014 systematic review concludes that contextual or interpretive nutrition information on menus better assists consumers in the selection and consumption of fewer calories (Sinclair et al., 2014), although that conclusion is contradicted by one experimental trial (Hammond et al., 2013). As we show in the Table in Part B, the results from previous studies are far from conclusive. Most studies were statistically underpowered and had no precision, so both large effects in either direction or no effects are consistent with the data. The only hypothesis typically tested was whether menu labeling reduces portion sizes ordered on average, rather than whether labeling aids consumers in making choices in line with their preferences. A new working paper that has not been peer-reviewed argues that weight status is main determinant in whether consumers react to labeling requirements (Deb and Vargas, February 2016). Many consumers may not wish to restrict calories and so lower calorie items may not be more appealing or some consumers may prefer menu items that have the lowest cost per calorie. Contradictory findings across studies may be due to contextual factors that were not comparable (i.e. restaurant type) or to differences in consumer preferences that were not studied. One hypothesis to that consumers are more likely to respond to calorie labeling in situations that are not traditional restaurants and mainly offer “discretionary calories”, such as ice cream parlors or movie theaters. The survey includes questions about how frequently consumers dine in different types of restaurants to control for restaurant preferences.
The two most similar studies to the proposed project were Hammond et al. (2013) and Roseman et al. (2013). Hammond et al (2013) had 150 responses for each experimental menu (for a total of 600 choices), as did Roseman (for a total of 300 choices). In contrast, ASPE expects about 10,000 choices in this project. The prior studies were pure convenience samples, studied on single restaurant type, and Roseman used a text description of food, rather than a visual representation of a menu.
The data collection techniques used are Multimode Interviewing Capability implemented on American Life Panel to field this experiment. ASPE is working with RAND to gather this information. MMIC is a comprehensive information system developed by RAND and used to manage the whole data collection process from questionnaire design, sample management, and fieldwork monitoring to final dataset production. MMIC provides a powerful software that allows questionnaires with visual aids and randomizations of questions and the order they are displayed, making it the ideal environment for this project. Together with the American Life Panel, this provides a robust and established platform without creating additional burdens on businesses or households. The sample size is discussed in Part B of this supporting statement.
This is a new information collection, and to our knowledge, there is no similar information or data gathered or maintained by the U.S. government. The menu labeling survey consists almost entirely of items developed to answer our study questions. Though it is possible that other entities could conduct a similar survey, there is no standardized instrument that currently exists.
No small businesses will be involved in this data collection.
This request is for a one time data collection.
There are no special circumstances associated with this information collection request. This request fully complies with the regulation 5 CFR 1320.5 and will be voluntary.
The 60-day Federal Register notice was published on Friday, July 10, 2015, Volume 80, Number 132. One letter with comments was received on September 10, 2015 from the Academy of Nutrition and Dietetics. Most of the comments were about menu labeling generally, the need for education campaigns, and alternative approaches to menu labeling, rather than about this proposed data collection.
The Academy questions how FDA will utilize the collected information, given that it has promulgated a final rule. Does FDA propose to update the final rule, or instead issue guidance to aid food retailers and end users in compliance and use?
Response (this response was provided by FDA): This information does not influence rule making or lead to changes in the rule as mentioned in the purpose section of this document. Instead, the study is aimed at providing information that may be used by FDA and other public and private entities to develop effective consumer education and public outreach materials that are specific to the type of menu information required by the final rule. FDA understands that the study is the first investigation that focuses on the required format of calorie labeling. Previous studies of consumer effects of menu labeling either used researcher-developed formats or formats specified by various state and local jurisdictions; most of these formats were different from the format required by the final rule. The study also examines various types of restaurants at the same time, rather than one or two particular types only, as with previous research. Furthermore, this study takes into account the frequency of visits to different types of restaurants and individuals’ existing preferences toward menu choices at the various types of restaurants. For these reasons, the information will fill an important gap in the existing literature of consumer impacts of menu labeling, i.e., the information is more relevant to the market status that will prevail after the menu labeling rule takes effect, offers a broader view of how consumers may react to calorie information in different contexts or types of restaurants, and provides insights into how consumers’ existing habits and preferences related to eating-out may affect their responses to calorie information on menus. The new information, together with results from additional research that FDA may be conducting, can be used to inform the design of education and outreach materials and campaigns aimed at increasing motivation and reducing hindrance to use calorie information on menus. FDA notes that this study is aimed at enhancing the beneficial effects of menu labeling on consumers. The study is not designed to address any issues related to industry compliance with the final rule.
Incorporate and evaluate current research in developing study
RAND comprehensively reviewed the literature to assure that this data collection fills an existing void, including the references in the regulatory impact analysis. Rand has worked with NYC on the MenuStat database of calorie information to inform our parameters. The survey will also be pre-tested in order to ensure that it is clear to users.
Ensure Consistency in Nutrition Education Information: FDA should make every effort to make the nutrition information provided on display and available upon request conforms as much as possible with the Nutrition Facts panel so that consistent nutrition education information can be provided.
This comment is unrelated to this specific project and about future outreach/education programs.
Consumer education campaigns should primarily target consumers who are least likely to understand and use the label, including low-income and low-education consumers, who are more likely to suffer from many obesity- and nutrition-related chronic diseases.
Information gathered from this survey will be used to inform consumer education campaigns, but this survey is not an educational tool.
Survey respondents should be able to view sample menu labels to assess clarity of the information presented, readability of the label, and utility of the information.
The comment is unrelated to the project. We study the effect of the FDA rule across different restaurant settings and consumers.
We recommend including individuals with low vision and individuals with medical needs, who comprise a substantial subset of the population.
Panelists are selected to be representative of the general population, so it includes individuals with specific needs. However, the first priority is to have sufficient precision to say something about the general population, rather than any narrow subgroup. Oversampling specific subgroups to have enough statistical power to say something specific for them is theoretically possible, but would increase total costs dramatically and is not possible with the available budget. While there is no disagreement about the desirability, for this particular subgroup, a focus group approach would probably be better suited.
Vary menu information and font sizes
This is beyond the scope of the project. We are not developing different labeling approaches to test comprehension, but only have menus consistent with the FDA rule. We can vary the saliency of calorie labels, but do not introduce different approaches (traffic light) or contextual statements beyond the requirements of the FDA rule.
Allow for data collection in areas with limited internet availability
The ALP was chosen for this research partially because it has focused since the beginning to create a panel that is not subject to the technology bias. Internet access or computer ownership was not a prerequisite, but rather the ALP made internet available to potential participants whenever needed and through differences means, including through TV/cable or providing computers.
Develop a mobile app for consumer reporting on menu confusion. With today’s simple-to-use technology, consumers could send pictures of menus that are (a) particularly easy to use, (b) confusing to consumers, or (c) do not meet guidelines
Unrelated to the project, but a suggestion for future outreach, education initiatives.
Respondents will receive payment in the usual amount for ALP surveys through MMIC, about $10. Because respondents are part of an existing panel and regularly receive payment for completing surveys, they would be unlikely to complete a survey without this incentive. Our response rate calculations are based on past experience with the ALP where respondents were given such an incentive.
Part B discusses the ALP survey. in this section we focus on privacy and data security.
Because we are using the ALP, personally identifiable information will be collected and identifiers such as age, gender, and state are available to the RAND team for subgroup analyses. Survey respondents have been assured that their information will be used for research purposes only and that no identifiable information will be released. Please see appendix D for a description of the RAND ALP data security protection plan. All processes have been approved by the RAND Internal Review Board (IRB).
Survey data is made available on the ALP Data Pages, where researchers are required to agree to the Conditions of Use for ALP Public and Preliminary Release Data Files before allowed access. The data made available through these pages contains no contact information, and responses are assigned an arbitrary id. An application server with a database contains the Respondent Portal and Survey Code, which is what ALP respondents interact with. The survey data is stored on the local database. This application server also contains the ALP Administration Pages, but these pages are only available from the RAND intranet by those people who manage the panel.
A different application server with a database hosts the ALP Data Pages. This database contains the Public Release data for the ALP, along with registration information for the researchers who have access to this data.
Any electronic transmission and sharing of individually identifiable data will be encrypted. This procedure will prevent anyone without permission from accessing and entering the data system. Access to personally identifiable data and raw survey data shall be limited to the minimum number of individuals necessary to achieve the approved purpose and to those individuals on a need-to-know basis only.
The survey does not include any questions of a sensitive nature.
Based on survey testing done by the RAND team and their colleagues, this survey will take approximately 20 minutes to complete per respondent. 2,850 individuals participating in the American Life Panel will be surveyed in 2016 and we expect 2,000 to complete surveys in addition to the 100 test surveys we will collect before the full fielding.
These burden and pace estimates are based on RAND’s testing of this survey. The annual total burden hours are estimated to be 693 hours. The annual total cost burden is estimated to be $15,772.68, based upon mean hourly wages from the “National Compensation Survey: All United States December 2009 – January 2011,” U.S. Department of Labor, Bureau of Labor Statistics.
Table A-12: Estimated Annualized Burden Hours and Costs to Respondents
Type of Respondent |
No. of Respondents |
No. of Responses per Respondent |
Average Burden per Response (in hours) |
Total Burden Hours |
Hourly Wage Rate |
Total Respondent Costs |
ALP Panel Members |
2,000 |
1 |
0.33 |
660 |
22.76 |
$15,021.60 |
ALP Panel members (pre-test) |
100 |
1 |
.33 |
33 |
|
751.08 |
TOTALS |
2,100 |
2,100 |
|
693 |
|
$15,772.68 |
There will be no direct costs to the respondents other than their time to participate in the data collection.
The total cost to the Federal Government is $150,180.
Staff (FTE) |
Average Hours per Collection |
Average Hourly Rate |
Average Cost |
Data collection costs (MMIC) |
2 |
23.10 |
$97,000 |
Social Science Analyst, GS 15 (Analysis and project management costs) |
0.38 |
63.31 |
$53,180 |
Estimated Total Cost of Information Collection |
$150,180 |
||
This is a new information collection request.
The fundamental analysis of this survey data is straightforward as this is a randomized experiment: simple descriptive statistics, possibly stratified by subgroups, will provide unbiased and internally valid results; sampling weights will be used to generalize the results. However, much deeper statistical models may be used to analyze discrete choices, starting from standard economic models (such as multinomial or nested multinomial models) to models incorporating possible violations of classic economic models (e.g. attribute-non-attention).
Project Timeline: Assuming OMB approval of our survey, the plan is to pretest the survey during the week of May 1, 2016 and field the final survey during the week of June 1, 2016. ALP surveys usually stay in the field for about four weeks and we expect survey fielding to end within four weeks. The draft field report for ASPE would then be delivered on or around 8/15/16 and the final field report would be delivered on or around 11/15/16.
Publication of Results: There will be a report to ASPE on the findings of this analysis and additional analysis may be submitted to peer reviewed journals upon ASPE approval.
We are not requesting an exemption.
There are no exceptions to the certification. These activities comply with the requirements in 5 CFR 1320.9.
LIST OF ATTACHMENTS – Section A
Note: Attachments are included as separate files as instructed.
Survey Instrument
Recruitment and Reminder emails
References
Data Security Protection for the RAND American Life Panel
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| File Modified | 0000-00-00 |
| File Created | 2021-01-24 |