NEW Supporting Statement Healthy Symbols - Part B 2022 Final

United States Food and Drug Administration
Quantitative Research on a Voluntary Symbol
Depicting the Nutrient Content Claim “Healthy” on Packaged Foods
OMB Control No. 0910-NEW
SUPPORTING STATEMENT
Part B.

Statistical Methods

1. Respondent Universe and Sampling Methods
Survey. FDA proposes the use of an online (i.e., web-based) panel to conduct the consumer
survey (see Appendix F for survey). The online survey will seek a panel-based (non-probability)
sample of 1,000 U.S. adults ages 18+ that self-identify as primary food shoppers (i.e., they report
they are responsible for at least 50% or more of the grocery shopping in their household). The
contractor has demographic information for all panel members and will use this pre-existing data
to select participants for this study based on the eligibility criteria. A reasonable degree of
diversity in key demographic characteristics such as age, gender, education, and race/ethnicity is
expected.
The sample of individuals invited to participate in the survey will be balanced to the U.S.
population, based on demographic information (age, gender, educational attainment, geographic
region, etc.) collected by the U.S. Census Bureau through the Current Population Survey. Once
the data are collected, the sample will be weighted on the same demographic variables to adjust
the sample to U.S. adult population and correct any potential differential responses by
demographic groups. While the results will be weighted to the U.S. adult population, FDA does
not intend to generalize the results to the overall population or produce precise estimates of
population parameters using the survey.
To support implementation of the consumer online survey, FDA and its contractor will retain
Ipsos, a vendor that maintains a preexisting representative national panel.
The overall sampling frame for the online consumer survey is the Ipsos Online Panel. The Panel
is an actively managed research access panel that uses multi-source recruitment to maintain a
representative base of respondents. It includes individuals who have volunteered to take part in
market research and is extensively profiled to efficiently target respondents.
Ipsos employs a survey router (brand name is Cortex) to manage a sample so that it can be
balanced to the US population. Ipsos follows best-in-class principles to measure and report any
selection bias that arises from the use of a router, including:
•

Designated team to manage and monitor the router, made up of Ipsos sampling and
methodology experts;

1

•
•
•
•
•
•
•
•
•
•

Router management team has final authority over which studies are in the router and
any prioritization decisions;
Restricted impact of highly-targeted studies;
Random and priority reallocation is balanced;
Respondents allowed to opt-out of the screening process with no penalty;
Diverse, large number of studies maintained in the router;
Limited number of custom screening questions allowed;
All router studies are reviewed by the router team and removed from the router if
necessary;
Key metrics monitored multiple times daily to identify potential performance issues;
Supplier traffic is monitored daily to ensure consistency; and
Representativeness and consistency of the router population to the general population
is monitored.

Experimental Study. Experimental study participants will be drawn from a panel maintained by
Prodege. Consumers are invited to join the Panel directly through Prodege's network of portal
sites and complete a double opt-in registration process with multiple verification steps including
CAPTCHA, IP Address verification, and mobile device reputation check. Currently, Prodege's
Panel has over 100 million participants worldwide.
The participant universe for this study is U.S. residents who are 18 or older of Prodege’s online
Consumer Panel.
Participants in the cognitive interviews will be recruited from a commercial database of residents
in the Washington, D.C. metropolitan area. We will recruit approximately 15 to 20 participants
to make sure at least 9 of them will show up for the interviews (see Appendix A for cognitive
interview screener and guide).
The current target sample size for the experimental study is 5,000. A quota will be developed
prior to sampling so that the overall sample of panelists who are sent invitations to participate in
the study are reflective of the Panel in gender, age, education, and race/ethnicity, i.e., outboundbalanced (see Appendix B for e-mail invitation to panel members). The planned balancing
categories are: (a) gender: female and male, (b) age: 18-34, 35-54, and 55+, (c) education: highschool graduate or less and one year or more college education, and (d) race/ethnicity: nonHispanic white and other.
We will test hypotheses related to between-label differences (see Appendix C for study design).
We will impose no a priori direction of differences, if any (i.e., we assume all tests are twotailed). The target sample size will yield enough observations to provide adequate power to
identify 4-way interactions of a medium size (see Appendix D for experimental study instrument
and Appendix E for mock food labels).
The agency does not intend to generate nationally representative results or precise estimates of
population parameters from the experimental studies. The studies will not utilize probability
sampling. Despite the attempt to match between the study’s sample and the participant universe

2

in four demographic characteristics, matching is used solely to produce a sample with a
reasonable degree of diversity in key demographic characteristics.
•

The strength of experimental studies lay in their internal validity. As discussed in the
following sections, the agency has taken commonly accepted measures to enhance
internal validity of the study. Examples of these measures include random
assignment of participants and conditions, counterbalancing condition assignments
within the sample, and use of comparison conditions and relevant covariates.

2. Procedures for the Collection of Information
Survey and Experimental Study. For the cognitive interviews, the contractors will use a
telephone invitation to recruit participants (Appendix A). The recruitment will target for
diversity in participants’ gender, age, race and education. Eligible participants will complete the
draft questionnaire on a personal computer. Then, a moderator will debrief the participant about
how certain questions were interpreted and the process by which responses were selected.
Survey. The survey will be administered online to a proprietary consumer panel. Through the
sampling process, panelists are pre-selected to answer a certain survey; the surveys are not “open
access” (i.e., respondents are not self-selected), and respondents do not know the survey content.
Panelists receive an invitation email with the following information:
•
•
•
•
•
•

Survey information (end date, survey number, survey duration, number of
incentive points)
A unique URL that provides access to the questionnaire
Physical address for Ipsos
Member support email address
Link to privacy policy
Opt-out information

To ensure high quality, data will be subjected to validation techniques, such as disallowing outof-range values.
Experimental Study. The contractor will utilize an online consumer panel owned by Research
Now to select study participants. Panel members will be invited by email (Appendix B) to a
contractor website to complete the study online in one session. After a brief introduction to the
study, each participant will be shown a single product label and will be asked to evaluate the
product based on measures of effectiveness such as perceived healthiness, contributions to a
healthy diet, believability, and trustworthiness (See Appendix D for questionnaire and Appendix
E for mock-up labels). We estimate that it will take participants about 15 minutes to complete the
full study.
The study employs a full factorial design (See Appendix C for spreadsheet visual display of
conditions).

3

The food labels in the study resemble those of real foods but are mock-ups developed for study
purposes; these are: a breakfast cereal, a frozen meal, and a can of soup.
Table 3. Structure of experimental study (See Appendix D for questionnaire)
Questionnaire
Section
Topic
A
Covariates: Affect, Past Label-reading, Behavior, self-reported
literacy, self-reported health
B
Single Product/Symbol Evaluation (Random assignments to a label
image; respond to measures of symbol effectiveness):
Healthfulness perceptions; believability; label perceptions; symbol
perceptions; effects perceptions; purchase intention; manipulation
check
C
Covariates: Dietary interests/nutrition motivation-self-efficacy
D
Beliefs about product category healthfulness
E
Food Shopping and Label reading
F
Demographics
3. Methods to Maximize Response Rates and Deal with Non-response
Survey. Methods for maximizing response rates includes the following:
• Targeted recruitment through various “wide net” methodologies (e.g., email
..campaigns, affiliate networks, banner ads, text ads, search engine, co-registration,
..offline-to-online, specialized websites);
• Use of a point system to incentivize panelists, along with sweepstakes draws.
..Points systems are recognized as being the best in class in online market research,
..as they are a neutral system which does not skew the participation of specific
..groups of people;
• Continuously testing new recruitment sources and methods; and
• Use of an internal data quality process that incorporates data quality checks at the
..survey level to reduce or eliminate random responding, illogical or inconsistent
..responding, overuse of item non-response, and too rapid survey completion.
We anticipate a 75% completion rate for the survey. We do not anticipate challenges with nonresponse, given the broadly defined eligibility criteria for the survey. However, in the event of
non-response challenges, we will send reminders to non-respondents, and monitor and correct
any potential non-response biases.
Experiment. Our experience with online experimental studies suggests that about 15% of those
who are sent invitations will complete a study. The agency will implement several procedures to
maximize participation. We will conduct cognitive interviews and pretests to help improve

4

understandability of the questionnaire, to reduce participant burden, and to enhance interview
administration.
In addition, the contractors will: (1) identify FDA as the sponsor of the study and state the
purpose of the study in their invitation and reminder to encourage participation; (2) provide an
email address and a toll-free number for prospective participants to inquire about the authenticity
of the interview and other questions; and (3) monitor all interviews and sample assignment and
solve any problems daily throughout the course of the collection of information.
4. Test of Procedures or Methods to be Undertaken
For both the survey and the experiment, FDA plans to perform tests to minimize collection
burden on participants and improve quality of collected information. Before the online survey is
implemented, a contractor will pilot test the instrument(s) and method of data collection. Lessons
from the pilot test will be identified, and changes as necessary will be incorporated into the
instrument and method. All survey pilot tests will involve no more than nine individuals.
For the experimental study, the first test consists of cognitive interviews; the primary purpose of
these interviews is to understand the thinking processes that participants use to answer the survey
questions.
The second test is a field pretest focusing more on the length of the questionnaire and participant
burden. The contractor will administer the full questionnaire to 180 adult members of the
Research Now web-based consumer panel shortly after satisfactory revisions following the
cognitive interviews.
Some fine-tuning of the data collection activity may result from the cognitive interviews and/or
the pretest, but substantive changes are not expected. This proposed information collection
requests OMB approval for the pretest in combination with the main collection of information.
We will inform OMB of any changes to the survey procedures or data collection instruments
with a final version before actual data collection begins.
5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing
Data
Survey. Christina Nicols, MPH, MS, MS
Senior Vice President, Strategic Planning, Research & Evaluation
Hager Sharp
Experiment. The contractor, Fors Marsh, will collect the information on behalf of the Agency.
Panne Burke is the Senior Study Director and Miriam Eisenberg Colman, the Senior Researcher
at Fors Marsh, is the project lead. Ron Vega, Senior Scientist at Fors Marsh, was consulted on
the statistics and study design.

5

Drs Linda Verrill and Fanfan Wu, FDA Social Scientists, are the FDA PI’s and will oversee all
aspects of the data collection. They will also conduct the data analysis and information
dissemination.

6