Final - OMB Justification

United States

Department of

Agriculture

Center for

Nutrition Policy

& Promotion

3101 Park

Center Drive

Alexandria, VA

22302-1500

Memorandum

To:	Julie Wise, OMB Desk Officer, Office of Management and Budget
Through:	Rachelle Ragland Greene, FNS Information Clearance Officer
	Ruth Brown, OCIO Desk Officer
From:	Robert Post, Deputy Director
Re:	Under Approved Generic OMB Clearance No. 0584-0523
	Request Approval to Perform Survey Research to Inform Public Communication about the 2010 Dietary Guidelines for Americans (DGA)

_____________________________________________________________________

The Center for Nutrition Policy and Promotion of the USDA is requesting approval for formative research under Approved Generic OMB Clearance No. 0584-0523

The following information is provided for your review:

• Title of Project: Formative Research to Inform Public Communication about the 2010 Dietary Guidelines for Americans (DGA)

The research for Task 1 includes an online survey among consumers to test messages and graphic images pertaining to the DGA.

• OMB Control Number: 0584-0523, expiration date 07/31/2012

Public affected by this project: Individuals/Households. Respondent universe includes adult consumers and children ages 10-13. As the DGA is intended for “healthy Americans two years of age and older,” we are including a general sample of 2,000 adults, which will include a diverse mix of demographic factors. We also want to be certain that the DGA consumer messages resonate among adults with lower education levels and lower income levels. Therefore, we have included a sample of 1,000 adults with less than a high school degree, and 1,000 adults under 185% of the federal poverty level (FPL).¹ In addition, we have included a sample of 500 children ages 10-13, as we believe this age range represents a critical time when children are beginning to make their own food choices independent of their parents. During mid-elementary through early middle school, children are beginning to make more food purchasing decisions. In talking with our library staff, they indicated that most of the nutrition education interventions in the literature include this age group (8-13) rather than younger and older children. Therefore, more information is available about the impact of nutrition education on this audience.

A separate data tabulation and summary will be developed for each of the following segments:

• General adult population ages 18+

• Adults under 185% of the federal poverty level

• Adults with less than high school graduate education

• Children ages 10-13

The samples are outlined as follows:

General adult population ages 18+	2,000
Adults under 185% of the federal poverty level	1,000
Adults with less than high school graduate education	1,000
Pretest among adults	25
Children ages 10-13	500
Pretest among children 10-13	25
Parental consent for ages 10-13	525
Total Respondent:	5,075

• Time needed per Response:

Online survey among adult consumers (general population, 185% FPL and low-education adults, and pre-test): Based on the assumption that the average length of the survey will be .4175 hours per adult participant, the burden hours associated with 4,025 respondents is 1,680 hours (.4175 x 4,025 = 1,680 hours).

Responding to invitation to adults for web-based message testing survey: Based on the assumption that the average length of the invitation response process will be .05 hours per adult participant, the burden hours associated with 4,025 respondents is 201.25 hours (.05 x 4,025 = 201.25 hours).

Responding to invitation to children for web-based message testing survey: Based on the assumption that the average length of the invitation response process will be .05 hours per child, the burden hours associated with 525 respondents is 26.25 hours (.05 x 525 = 26.25 hours).

Online survey and pre-test among children 10-13: Based on the assumption that the average length of the survey will be .25 hours per child participant, the burden hours associated with 525 respondents is 131.25 hours (.25 x 525 = 131.25 hours).

Parental consent: Based on the assumption that the average length of the parental consent process will be .05 hours per parent, the burden hours associated with 525 respondents is 26.25 hours (.05 x 525 = 26.25 hours).

The total burden for this project is 2,065 burden hours.

Table 1. Respondent Burden Estimate
Respondent group	Estimated number of respondents	Minutes per respondent	Total respondent burden
Screening* for adults for web-based message testing	0	0	0
Responding to invitation to adults for web-based message testing survey	4,025	3	201.25 hours
Web-based message testing among adults, including pre-test	4,025	25	1,680 hours
Screening* for children for web-based message testing	0	0	0
Parental consent for children ages 10-13	525	3	26.25 hours
Responding to invitation to children for web-based message testing survey	525	3	26.25 hours
Web-based message testing among children ages 10-13, including pre-test	525	15	131.25 hours
Total	5,075	N/A	2,065 hours

*using existing web panel – eligible pool of participants is pre-identified from the database, with no screening necessary

• Project Background, Objective, Methodology and Design, Data Analysis, and Outcome.

Background:

Center for Nutrition Policy and Promotion (CNPP) is responsible for communicating to consumers and professionals about the Dietary Guidelines for Americans (DGA). An updated version of the DGA will be released in 2010. DGA-related nutrition concepts and messages will be communicated to consumers through MyPyramid and other nutrition education programs, tools and on-line resources. The proposed research is intended to inform CNPP communication efforts to raise consumer understanding and application of the DGA.

The messages and graphic images we are testing were informed by a series of 14 focus group sessions conducted throughout the month of October, 2010. The sessions included a total of 112 adults. The focus groups were held in four markets across the U.S.: Bethesda, MD; Sacramento, CA; Chicago, IL; and Philadelphia, PA, and included adults with a diverse range of demographic factors. We tested initial draft messages and graphic images in these sessions, and made changes to them based on the feedback we received. The focus group sessions were approved under Approved Generic OMB Clearance No. 0584-0523.

Objectives:

• Test DGA messages and related graphic images among consumers on the clarity and effectiveness of messages, the effectiveness and appeal of creative designs, and other key factors.

Methodology, Design and Data Analysis:

We will conduct online testing of the “near-final” messages and creative elements for DGA 2010 to test them among a sample of 4,025 adult consumers (including pre-test participants) and 525 children ages 10-13 (including pre-test participants).

For the adult participants, we have developed two separate surveys containing similar messages with different wordings. We will test each version of the survey with half of the adult samples. We believe this approach will enable us to choose the wordings that resonate best among our audiences. Also, the two separate versions of the adult survey contain two sets of graphic images that vary slightly. As with the variations in messages, we believe this approach will enable us to choose the combination of graphic elements that presents the concept of healthy eating most clearly.

Procedures for Information Collection:

The survey will be 100% online. Participants will be selected from a pre-existing online Non-Volunteer Access Panel. The Panel consists of a sample of about 50,000 U.S. residents, ages 18 and older, including cell phone-only households. Children ages 10-13 are recruited from the households of Panel members, and their parents or legal guardians provide consent for their participation. The contractor will send an email to parents asking for their consent (see Attachment B). After consent is granted, the child will receive an email invitation to take the survey. If the child does not have an email address through which to receive the survey, the survey link will be emailed to the parent/legal guardian, who will be asked to hand off the survey to the child. The parent/legal guardian will be able to see what questions are being asked of their child.

The contractor has demographic information for all Panel members, and will use this pre-existing data to select participants for this study. This eliminates the need for screening participants. Panel members who meet study criteria will receive a standard email invitation to participate (see Attachment A). As Panel members, they will have already received a copy of the Privacy and Terms of Use Policy (see Attachment C), which outlines confidentiality, use of the data collected, voluntary participation, and the reassurance that there are no penalties for opting out of the survey.

Target Sub-Groups:

• As the DGA is intended for “healthy Americans two years of age and older,” we are including a general sample of 2,000 adults, which will include a diverse mix of demographic factors. We also want to be certain that the DGA consumer messages resonate among adults with lower education levels and lower income levels. Therefore, we have included a sample of 1,000 adults with less than a high school degree, and 1,000 adults under 185% of the federal poverty level (FPL).² In addition, we have included a sample of 500 children ages 10-13, as we believe this age range represents a critical time when children are beginning to make their own food choices independent of their parents. The Panel is well-suited to meet the needs of the study, as it reduces reporting burden on the U.S. public by re-using previously consented sample and by elimination of re-asking previously asked survey questions. The Panel also supports the targeting of populations that can be more difficult to reach, such as low-education adults, and it includes sample coverage of non-Internet households.

A separate data tabulation and summary will be developed for each of the following segments:

• General adult population ages 18+

• Adults under 185% of the federal poverty level

• Adults with less than high school graduate education

• Children ages 10-13

Response Rate:

The vendor’s best practices for maximizing the within-panel survey cooperation rate will be employed to yield a survey completion rate of 75% or higher.

• Pre-notification email sent to sampled respondents 2-3 days before sending the actual survey invitation;

• Two to three weeks of survey data collection field period;

• Include cash-equivalent incentives of $5 for longer surveys (25 minutes or longer); and 

• Use of email reminders and telephone-based reminder calls for non-responders.

As reported in Public Opinion Quarterly in the Callegaro and DiSogra (2008) article³, the AAPOR cumulative response taking into account all phases of panel recruitment, profiling, and survey participation is approximately 10%.  The covered population for the panel recruitment is approximately 97% of the U.S. population as a result of the use of the ABS sampling. The uncovered population consists of persons with post office boxes and rural route addresses; business and institutional addresses (i.e., dormitories, nursing homes, group homes, jails, etc.); military housing; and multi-dwelling residential structures that have only a single address (called a drop point address) and for which there is no unit-level identifying information (mail is internally distributed).  

Incentives for Participants and Burden:

Participants receive an incentive for participating in the panel. In addition, toward the end of the field period, an email will be sent to the remaining non-responders promising a $5 cash-equivalent incentive for survey participation, in order to maximize the survey cooperation rate. The parent or legal guardian that is a member of the KN panel and related to the child study participant will be given points equivalent to five dollars in value. These points can be later redeemed by the panelists for cash or various other incentive choices.

Steps are taken to ensure that panel members are not overburdened with survey requests. The primary sampling rule is to assign no more than one survey per week to members. This level of survey frequency helps to keep panelists engaged as part of the panel.

Non-response Analysis:

The vendor’s non-response bias measurement includes:

• Statistical comparison of demographic and household characteristics in two groups: the sample frame from which the panel recruitment sample was sourced; and actual survey participants, using ancillary data on person-level and household-level characteristics that are available for units from the sample frame (for the general sample of 2,000 adults).

• Statistical comparison of demographic and household characteristics of the panelists participating in the survey versus the non-responders to identify any systematic within-panel self-selection effects.

• Benchmarking panel survey estimates by placing benchmarking survey questions on the panel survey instrument, and comparing the survey estimates to benchmarks from gold-standard surveys (e.g., NHIS, GSS, SIPP, etc); the selection of the survey questions should be informed by a theory that the survey measures are related to the study topics of interest (e.g., political ideology measure in a study on attitudes towards an government regulation of an environmental good).   

Confidentiality, Privacy and Opt-In Details:

Participation in research is voluntary at the time that respondents are asked to join the panel, at the time they are asked to participate in a survey, and at the time they answer any given question in a survey. Panelists are not coerced to participate in any research and are not removed from the panel as a result of failure to participate in any particular survey project or program of studies. Panelists have the option to ‘opt-out’ of the panel at any time by notifying the contractor. The contractor maintains a toll-free phone number and its own call-center panel management facility to receive requests for information and action from panelists.

The survey data are anonymized prior to the actual collection of the survey data.  The survey data and personally identifying information are never matched or collected simultaneously.  The project research staff at Knowledge Networks do not have access to the PII of KN panelists. Surveys are self-administered and accessible any time of day for a designated period. Participants can complete a survey only once. Members may leave the panel at any time. All Panelists, when joining the panel, are given a copy of the Privacy and Term of Use Policy (see Attachment C). The privacy terms are also available electronically at all times to panelists via the Panel Member website.

For surveys conducted among children ages 10-13, the contractor obtains informed consent forms from parents or legal guardians (see Attachment B).

Test of procedures: For the consumer survey, the contractor will conduct a pre-test among 25 adult Panel participants and 25 child participants using the approved survey instrument. The pretest will field the survey to research subjects and collect feedback on any questions or confusions the respondent wants to report regarding the survey instrument. The vendor collects that information via open-ended questions at the end of the online pretest interview (Attachment F). We will refine the survey questionnaires as needed based on this feedback. After a successful pre-test, the contractor will field the survey over approximately a two-week period among participants.

Consultant for Statistical Aspects of the Survey:

The contractor will perform all statistical analysis of the survey responses and provide banner tables as a summary of the research. The contractor project lead will be J. Michael Dennis, Ph.D., and he will be assisted by Joe Garrett:

J. Michael Dennis, Ph.D.

Senior Vice President, Government & Academic Research

Knowledge Networks, Inc.

1350 Willow Road, Suite 102

Menlo Park, California 94025-1516

(650) 289-2160

(650) 289-2001

J. Michael Dennis, the Senior Vice President for Government & Academic Research at KN, has completed human subjects training from the NIH, University of California (Irvine), and Stanford University.

Joe Garrett

Vice President, Government and Academic Research

Knowledge Networks, Inc.

12070 Rose Hall Drive

Clifton, VA 20124

Office: 703-830-0613   

Cell: 703-371-9327

Outcome:

Preliminary and final summary reports will be provided for all research components. A final summary report of the surveys will be both descriptive and quantitative in nature and will discuss the limitations of the study design. We will provide OMB with a copy of the preliminary and final reports.

Findings will be used to modify messages and graphic images. Although the Agency does not intend to publish its findings, the Agency may receive requests to release the information (e.g., Congressional inquiry, Freedom of Information Act). The Agency will disseminate the findings when appropriate, strictly following the Agency's "Guidelines for Ensuring the Quality of Information Disseminated to the Public", and will include specific discussion that the estimates are not nationally representative or statistically valid.

• Federal Costs: $311,748.22

• Study Start Date: November 22, 2010

• Formative Research Tools:

  1. Appendix A – Sample invitation to take the survey (attached)

  2. Appendix B – Informed Consent Parental Permission form for minors (attached)

  3. Appendix C – Privacy Statement for Participants (attached)

  4. Appendix D v1 – Questionnaire for adults (attached)

Appendix D v2 – Questionnaire for adults (attached)

5. Appendix E – Questionnaire for children (attached)

6. Appendix F – Pretest questions at end of survey (attached)

1 185% of the federal poverty level for a family of four is $40,792.50.

2 185% of the federal poverty level for a family of four is $40,792.50.

3 Callegaro, Mario and Charles Disogra (2008). Computing Response Metrics for Online Panels. Public Opinion Quarterly. 72(5) pp. 1008-1031.