NFP Mod SS Part B updated clean 7 27 11

Experimental Study of Nutrition Facts Label Formats

experimental study

OMB No. 0910-NEW

SUPPORTING STATEMENT

PART B

B. Statistical Methods

1. Respondent Universe and Sampling Methods

1a. Respondent universe

The respondent universe of this study is Synovate’s online Consumer Opinion Panel (“ePanel”).  U.S. Consumers who are 18 or older are invited to join the ePanel primarily through an affiliate marketing program.  Select web sites, portals and Internet Service Providers partner with Synovate to promote ePanel membership through targeted email campaigns as well as placement of banner and pop-up advertisements. Consumers may also join ePanel through referrals from existing ePanel members and re-enlistment of former members. Currently, ePanel has over 2.5 million participants.

1b. Sampling methods

Cognitive Interviews

Respondents in the cognitive interviews will be recruited from a commercial database of residents in the Washington, D.C. metropolitan area. We will recruit approximately 15 to 20 respondents to make sure at least 12 of them will show up for the interviews. Six of the interviews will be conducted prior to OMB approval and 6 after OMB approval (see Appendix A for screener).

Experimental Study

The target sample size is 10,000.  A quota will be developed prior to sampling so that the overall sample of panelists who are sent invitations to participate in the study are reflective of the ePanel in gender, age, education, and race/ethnicity, i.e., outbound-balanced (see Appendix B for invitation email).  The planned balancing categories are: (a) gender: female and male, (b) age: 18-34, 35-54, and 55+, (c) education: high-school graduate or less and one year or more college education, and (d) race/ethnicity: non-Hispanic white and other.  For a given quota, Synovate will send email invitations to randomly selected panelists who qualify for the quota until the quota is filled and each of the predetermined experimental conditions is filled with the predetermined cell size. 

The agency does not intend to generate nationally representative results or precise estimates of population parameters from the experimental studies. The studies will use a convenience samples rather than probability samples. Despite the attempt to match between the study’s sample and the respondent universe in four demographic characteristics, matching is used solely to produce a sample with a reasonable degree of diversity in key demographic characteristics.

Rather, the strength of the experimental studies lies in its internal validity, on which meaningful estimates of differences across experimental conditions can be produced and generalized. As discussed in the following sections, the agency has taken commonly accepted measures to enhance internal validity of the study. Examples of these measures include random assignment of respondents and conditions, counterbalancing condition assignments within the sample, and use of comparison conditions and relevant covariates.

2. Procedures for the Collection of Information

2.1 Statistical methodology for collection and sample selection

Cognitive Interviews

The contractors will use a telephone invitation (Appendix A) to recruit respondents. The recruitment will target for diversity in respondents’ gender, age, race and education. They will also be required to meet other eligibility conditions (see Appendix A). Eligible respondents will complete the draft questionnaires either on pencil and paper (first 6 cognitive interviews to be completed prior to OMB approval) or on a personal computer by herself/himself (second set of 6 cognitive interviews after OMB approval). Then, a moderator will debrief the participant about how she/he interpreted certain questions and the process by which she selected her/his responses.

Experimental Study

Members of the contractor’s online consumer panel will be invited by email (Appendix B) to a Synovate Website to complete the study online in one session. Each respondent will perform a single product evaluation for one product category (frozen entrée, or chips) and one of two types of product choice assessments (See Appendix C for questionnaire, Appendix D for mock-up labels and, Appendix E for nutrition profiles). They will be presented with either a product choice assessment for a pair of products from another product category with the same NF label design (B1), or a product choice assessment comparing a single column format (with one or two servings per container) to a dual column format or a single column format with one serving per container to a single column format with two servings per container (B2). We estimate that it will take respondents about 15 minutes to complete the study.

Table 1. Structure of experimental study

Section	Topic
Task 1 (Section A on the questionnaire)	Judgment of a single product and its label, including (1) overall healthiness, (2) overall purchase intent (3) level of calories and of total fat, sodium, sugars, Vitamin A, Vitamin C, calcium, and fiber in the one serving of the product; (4) ability to find calories, total fat, and fiber per serving and for the entire package; (5) how many servings of the product a person would need to get all the Vitamin A for a day and how many servings the product would provide someone with a full day’s worth of saturated fat and sodium. Time to complete each item in Section A will be recorded.
Task 2A or 2B(order of Task 1 and Task2a/2bwill be random-ized between respondents) (Section B of the questionnaire)	Which product in the pair would be bought as a healthier product for the family; which product has the fewest calories per serving and per total package. Time to complete each item in section B will be recorded.
C	There is no section C
D	Consumption and familiarity of products in study
E	Food label use
F	Dietary awareness, knowledge, and interests
G	Cosmetic Adverse Reporting System familiarity
H	Demographics

2.2 Estimation Procedure

The study will employ a 10 (labeling condition) x 2 (product category) x 2 (nutritional profile per category – one fixed as healthier than the other). The ten labeling schemes are for a two serving product: 1) current NF label control, 2) removing calories from fat, 3) calories enlarged, 4) changed wording for serving size declaration, 5) dual column version 1, 6) dual column version 1 without calories from fat, 7) dual column version 2 without calories from fat, and 8) just dual column for calories and not other nutrients, 9) removing calories from fat but instead of two servings per package it will only have a single, large serving, and 10) calories enlarged but instead of two servings per package will only have a single, large serving per package. The two product categories are frozen entrees and a “Grab bag” sized bag of chips. Within each product category there are two products, one that is healthier and one that is less healthy. Task one is a judgment is about a single product (Section A in Table 1) and task 2 is a choice between two products (healthy and unhealthy profiles) with the same label from the same product category (B1 in Table 1) or between two products (healthy and unhealthy) with different labels schemes (B2 in Table 1) and task two. The order of task one and task two will be randomized.

Each respondent will be randomly assigned to an experimental condition. A condition is a combination of the three factors described above (labeling, product category, and nutrition profile). They will not see the front of the package or any product claims.

2.3 Degree of accuracy needed for the purpose described in the justification

The key experimental hypotheses concern the effects of changes to the NF panel on how respondents perceive the healthfulness of products, their purchase intention, and how well they can use the labels to perform certain tasks such as determining how many calories are in one serving and in the total package. We will also measure the amount of time it takes to complete each task.

The study is designed to test hypotheses related to between-label and between-product type differences in label comprehension, product choices, perceived levels of nutrients, product and labeling perceptions, and time spent on experimental tasks. The target sample size is 10,000 participants, which provides approximately 1,000 observations for each of the ten labeling conditions (nine labeling schemes and control) across task 1. The sample size also allows for approximately 250 observations for each combination of product and labeling format (4 x 10) for the first task and for the two-product choice tasks. The sample size was selected so the study would have over 80% power to detect a small effect size (Ref 1) between experimental conditions on the key dependent measures (e.g., mean number of correct responses out of 10 comprehension questions and probability of making a correct product choice) using a two-tailed test with a significance level of  = .05, after controlling for multiple covariates (e.g., education, age, gender, etc.). FDA believes that the sample sizes will also offer a similar power to detect interactive effects of a medium size (Ref 2).

2.4 Use of specialized sampling procedures

No specialized sampling procedures are required.

2.5 Use of periodic data collection cycles to reduce burden

This is a one-time data collection.

3. Methods to Maximize Response Rates

Our experience with online experimental studies suggests that about 15% of those who are sent invitations will complete a study. The agency will implement several procedures to maximize participation. We will conduct cognitive interviews and pretests to help improve understandability of the questionnaire, to reduce participant burden, and to enhance interview administration (see Appendix M for cognitive interview invitation; the cognitive interview will use the draft questionnaire). We will keep the study questionnaire at a reasonable length to minimize breakoffs.

In addition, the contractors will (1) identify FDA as the sponsor of the study and state the purpose of the study in their invitation and reminder to encourage participation (see Appendix F for reminder); (2) provide an email address and a toll-free number for prospective participants to inquire about the authenticity of the interview and other questions; and (3) monitor all interviews and sample assignment and solve any problems daily throughout the course of the collection of information.

4. Tests of Procedures or Methods

FDA plans to perform two tests to minimize collection burden on respondents and improve quality of collected information. The first test is the cognitive interview; the primary purpose of these interviews is to understand the thinking processes that respondents use to answer survey questions.

The second test is field pretests focusing more on the length of the questionnaire and respondent burden. The contractor who is responsible for the data collection will administer the full questionnaire to 150 adult members of Synovate’s web based consumer panel shortly after OMB approval of the collection of information. This is the same web-based consumer panel from which the control group will be selected.

5. Individuals Involved in Statistical Consultation and Information Collection

The contractor, Synovate, Inc. will collect the information on behalf of the Agency. Valerie Fuller DiPaula, Ph.D. is the Senior Study Director and project lead at Synovate, Inc. Analysis of the information will be conducted primarily by the FDA Project Officer, Amy Lando, MPP, telephone 301-436-1996.

References:

(1) Cohen, J. “A Power Primer.” Psychological Bulletin 112(1): 155-9, 1992.

(2) Montgomery, A.A., T.J. Peters, and P. Little. “Design, Analysis and Presentation of Factorial Randomized Controlled Trials.” BMC Medical Research Methodology 3: p26-5, 2003.

5