OMBPartA Rev 6-7-10 w_edit

The Food and Nutrition Service Evaluation of the Fresh Fruit and Vegetable Program

SUPPORTING STATEMENT FOR

“The Food and Nutrition Service Evaluation of the Fresh Fruit and Vegetable Program”

Tracy K. Palmer

Special Nutrition Evaluation Branch

Office of Research and Analysis

Food and Nutrition Service

US Department of Agriculture

3101 Park Center Drive

Alexandria, VA 22302

Phone: 703-305-2126

Fax: 703-305-2576

Email: [email protected]

Contents

Part A: Justification 1

A.1 Explanation of Circumstances That Make Collection of Data Necessary 1

A.2 How the Information Will Be Used, By Whom, and For What Purpose 4

A.3 Uses of Improved Information Technology to Reduce Burden 9

A.4 Efforts to Identify and Avoid Duplication 10

A.5 Efforts to Minimize Burden on Small Businesses or Other Entities 10

A.6 Consequences of Less Frequent Data Collection 10

A.7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations 10

A.8 Federal Register Comments and Efforts to Consult with Persons Outside the Agency 11

A.9 Payments to Respondents 12

A.10 Assurance of Confidentiality 12

A.11 Questions of a Sensitive Nature 14

A.12 Estimates of Respondent Burden 15

A.13 Estimates of Other Annual Costs to Respondents 15

A.14 Estimates of Annualized Government Costs 17

A.15 Changes in Hour Burden 17

A.16 Time Schedule, Publication, and Analysis Plans 17

A.17 Display of Expiration Date for OMB Approval 19

A.18 Exceptions to Certification Statement 19

Part B: Statistical Methods 20

B.1 Respondent Universe and Sampling Methods 20

B.2 Procedures for the Collection of Information 24

B.3 Methods to Maximize Response Rates and Deal with NonResponse 29

B.4 Tests of Procedures or Methods to Be Undertaken 31

B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 36

Appendix A: Overview of Data Collection and Analysis

Appendix B: Recruitment Letters and Consent Forms

Appendix C: Survey of State Child Nutrition Agencies

Appendix D: Survey of School Food Authorities

Appendix E: Survey of School Principals

Appendix F: School Food Environment Assessment

Appendix G: School Foodservice Manager Interview and Survey

Appendix H: Teacher Survey

Appendix I: Parent Survey

Appendix J: Self Administered Student Questionnaire and Food Diary

Appendix K: Public Comments and Comments from Consultants Received

Part A: Justification

A.1 Explanation of Circumstances That Make Collection of Data Necessary

The Food, Conservation, and Energy Act of 2008, also known as the Farm Bill (Pulblic Law 110-246) expanded the Fresh Fruit and Vegetable Program nationwide and directed the Food and Nutrition Service (FNS) of the U.S. Department of Agriculture (USDA) to conduct an evaluation resulting in a report to Congress in September 2011. To meet the Congressional deadline, this OMB clearance package requests clearance for the data collection activities planned for Spring and early Fall of 2010. This section contains an overview of the Fresh Fruit and Vegetable Program and discusses why the Food and Nutrition Service Evaluation of the Fresh Fruit and Vegetable Program is needed.

Background

The Fresh Fruit and Vegetable Program (FFVP), administered by the Food and Nutrition Service (FNS) of the U.S. Department of Agriculture (USDA), has the broad goal of fighting childhood obesity and improving overall diet quality by teaching children healthier eating habits. Increased consumption of fruits and vegetables has been found to reduce long-term obesity risk.¹ Experimental interventions involving the addition of fruits and vegetables to the diet have demonstrated short-term effectiveness in reducing body weight in some cases, particularly when paired with reduction of dietary fat and/or overall caloric intake, although it is difficult in these studies to identify the influence of increased fruit and vegetable intake relative separately from that of overall dietary change.² Additionally, the relationship between increased fruit and vegetable consumption and reduced risk of heart disease and some cancers has long been established.³ Initiatives to improve health through school-based programs promoting fruit and vegetable consumption among children are found not only in the U.S. but in other countries as well.⁴

Specific objectives of the FFVP are: to create healthier school environments by providing healthier food choices; to expand the variety of fruits and vegetables children experience; to increase children’s fruit and vegetable consumption; and to make a difference in children’s diets to impact their present and future health. Schools selected to participate in the FFVP are reimbursed for providing fresh fruits and vegetables free to students during the school day, outside normal school breakfast and lunch meals. Schools with higher proportions of low-income students, who may be at higher risk for overweight and obesity, are given the highest priority for selection into the FFVP.⁵

The FFVP began as the Fruit and Vegetable Pilot Program, authorized by Congress under the Farm Security and Rural Investment Act of 2002 (P.L. 107-171). The pilot provided $6 million for the 2002-2003 school year to purchase fresh fruits and vegetables in 100 elementary and secondary schools in four States, and seven schools in the Zuni Indian Tribal Organization (ITO) in New Mexico. The purpose of the pilot was to identify best practices for increasing fresh fruit and vegetable consumption among students, and to determine feasibility and students’ interest.

Over the three-year period that followed, legislation established the FFVP as a permanent program under the National School Lunch Act (NSLA) and funded its expansion to schools in 9 additional States and two additional ITOs. The 2008 Consolidated Appropriations Act (H.R. 2764) again amended the NSLA, expanding the FFVP nationwide. This legislation provided $9.9 million in additional funds for program operations in the 2008-2009 school years. Most recently, the Food, Conservation and Energy Act (Farm Bill) of 2008 (P.L. 110-246) amended the NLSA by adding section 19, the Fresh Fruit and Vegetable Program. This legislation permanently authorized the program in selected elementary schools nationwide (including all 50 States, the District of Columbia, Guam, Puerto Rico, and the Virgin Islands) and provided a significant funding increase, beginning with $40 million in FY2009.

Authority for the Study

The Food, Conservation, and Energy Act of 2008 established a legislative requirement for an evaluation of the impact of the Fresh Fruit and Vegetable Program (FFVP) by amending Section 19(h) of the Richard B. Russell National School Lunch Act, as amended through P.L. 110-246, October 1, 2008. The objective of the evaluation, as specified in the legislation, is “a determination as to whether children experienced, as a result of participating in the program—

(A) increased consumption of fruits and vegetables;

(B) other dietary changes, such as decreased consumption of less nutritious foods; and

(C) such other outcomes as are considered appropriate by the Secretary.”

The evaluation, developed by the Food and Nutrition Service, and its contractor, Abt Associates Inc., explored many potential research designs before proposing the one described in this document. Some good designs were infeasible due to constraints of program structure and the need to meet the Congressional deadline. Some of the challenges faced in developing the evaluation design, place limitations on the study’s findings. Challenges include:

(1) Identifying short-term versus long-term impacts. The deadline for the Congressionally mandated report implies that our study can only measure short-term outcomes, or those occurring while the program is underway. During that period, fruits and vegetables will be distributed to children during the school day. Children will also receive nutrition education messages about fruit and vegetable consumption both directly through classroom lessons and indirectly as a result of the distribution itself. Our study will not measure long-term impacts. Over the longer-term, although the direct impact of being provided with fruits and vegetables on a daily basis will end, acclimation to fruits and vegetables and the direct and indirect nutrition education may remain. The relative size of short-term and long-term impacts is unclear. It is possible that acclimation and nutritional education impacts will grow as children age and gain more control over their food consumption. On other hand, it is possible that impacts will shrink, both because of the elimination of any direct impact of fruit and vegetable distribution and because of the attenuation of nutritional education messages. Since our study only measures short-term outcomes, it will not be informative about these important medium and long-term outcomes.

(2) Program maturity and dosage. The FFVP is a relatively new program, having only been operating nationwide for two years. The program has expanded rapidly; funding has increased approximately 50 percent each year. Most schools and students participating in the impact evaluation will be in their first year of FFVP participation. It is possible that longer exposure would lead to larger student impacts. Similarly, we will measure outcomes in the fall. It may be that schools will improve their programs as their experience increases. Since we are measuring impacts early in the school year and when the program is relatively new, we will not capture impacts that occur with longer exposure to the program or as schools refine their programs in response to lessons learned operating the FFVP.

(3) Selecting a design for the impact analysis with high internal validity, i.e. finding a control group that will yield convincing estimates of causal impacts. As discussed in Part B, we selected Regression Discontinuity (RD) as the most appropriate design for this evaluation. Students in schools participating in the FFVP are compared to students in comparable schools—where comparable is defined as schools with a similar percentage of students approved for free/reduced price lunches (FRSL)—that applied, but are not participating in the program due to funding limitations. This will yield estimates of program impacts at the funding cutoff, which is appropriate for answering the policy question of whether to expand the FFVP. One potential limitation of RD is that it will not provide estimates of the average impact among all children if that impact varies substantially with schools’ percentage of students approved for free/reduced price lunches.

(4) Selecting the most appropriate methodology to collect dietary intake from elementary school students. Collecting information on dietary intake of elementary school children is central to estimating the FFVP’s impact. As discussed in detail in Part B, dietary intake data will be collected using a diary-assisted 24-hour recall methodology, which FNS believes will collect the most accurate information with the least burden on student respondents. While no dietary intake methodology will yield error-free measures of nutritional intake, there is no a priori reason to believe that reporting error would differ across intervention and comparison schools and thus impact estimates will be unaffected. Thus, the study will measure impact accurately but comparison of study means and distributions to findings of other studies may be affected.

(5) Sample size limitations. Congressional funding limitations on the evaluation affect the size of the school/student sample and thus affect the precision with which impacts can be estimated. As discussed in detail in Part B, the sample size will allow us to detect differences in mean fruit and vegetable intake of approximately one quarter cup or larger between FFVP participants and non-participants. The study sample size does not allow us to detect smaller differences. In addition, the study sample size will only allow us to detect relatively large impacts within and across subgroups of the population.

(6) Estimating impacts for a new program with a relatively modest per student expenditure ($50-$75 during the school year). As a result of these features the study will estimate impacts for the program as currently designed and implemented. For example, if this evaluation does not find a significant impact of the FFVP on dietary intake, it is possible that a program that provided a higher per student fruit and vegetable expenditure would find an impact.

Study Objectives

The Food and Nutrition Service Evaluation of the Fresh Fruit and Vegetable Program has two major objectives: 1) to examine how the Fresh Fruit and Vegetable Program is currently being implemented, and 2) to estimate program impacts on participating students.

FNS has broken down these broad objectives into specific research questions that can be grouped into six broad research categories:

1. Description of participating schools

2. Description of participating children

3. Description of FFVP implementation

4. Impacts of the FFVP on school environment

5. Main impacts of the FFVP on children’s consumption of fresh fruits and vegetables, children’s consumption of other foods, levels of selected nutrients consumed by children, and children’s attitudes towards fresh fruits and vegetables

6. Contextual effects on impacts of the FFVP associated with nutrition education and FFVP delivery

Detailed research questions are presented in Appendix A, Exhibit A-1 (first column).

A.2 How the Information Will Be Used, By Whom, and For What Purpose

This section of the supporting statement provides an overview of the research design and data collection efforts planned to meet the overall objectives of the Food and Nutrition Service Evaluation of the Fresh Fruit and Vegetable Program.

Overview of the Research Design

The evaluation design includes two components, an Impact Study and an Implementation Study, reflecting the two main evaluation objectives. A stratified probability sample of 16 States will serve as the first stage sample for both the Impact and Implementation Studies.

After reviewing—and rejecting—several alternative research designs,⁶ the Impact Study will use a regression discontinuity (RD) design which will produce unbiased estimates of the impact of the program. Therefore, the Impact Study requires samples of students in schools that participate in the FFVP and eligible schools that do not participate in the program. However, the nature of the RD design is such that the schools (and students) included in the sample are not nationally representative. By contrast, the Implementation Study is intended to provide detailed information on how the FFVP is implemented in participating schools across the country, which requires a nationally representative sample of participating schools. Thus, we will supplement the sample of participating schools used in the Impact Study with a randomly selected sample of participating schools that were not included in the Impact study. The sampling design is described in more detail in Section B.1.

Congress’s primary interest is in the impact that the FFVP is having on students’ diets—this means the evaluation needs to collect 24-hour dietary recalls from students. Because the program is only in elementary schools, administering 24-hour recalls is more complex than with adult respondents. To address this, trained data collectors will introduce the diary to students during a motivational instruction period. The results of the 24-hour recall provide the core data for the evaluation. However, to analyze it we need to know a variety of things, like how the program is implemented, and what role nutrition education plays in the FFVP. The evaluation must also explore the interaction between the fresh fruits and vegetables in the FFVP and the fruits and vegetables served in the National School Lunch Program (NSLP) and the School Breakfast Program (SBP), which is one reason why school food service managers will be interviewed. Whatever the relationship among all these variables, we will need to determine what is attributable to the FFVP, which is why RD design was selected for this evaluation.

The broad goals of this study require both observation of school environments and assembling information from a wide variety of information sources, including States, school principals, School Food Authorities, school food service managers, teachers, parents, and students. For each of these sources, Exhibit A-2 (see Appendix A) shows which mode(s) of data collection is planned, the sample size and the topics covered. This evaluation relies on five main methods of data collection:

• Web surveys⁷ of State Child Nutrition Agencies, directors of School Food Authorities, and school principals associated with sampled schools;

• Interviews with school food service managers from sampled Impact Study schools;

• Short cover-sheet questionnaires for teachers of Impact Study sampled school classrooms, and for parents of students selected to participate from those classrooms;

• Interviews with students including diary-assisted 24-hour recalls using a modified version of the methodology developed for USDA’s Automated Multiple-Pass Method (AM/PM ) interview; and

• Observations of school food environments.

Data for the impact and implementation components will be collected in 16 States, though some State-level implementation data will be collected from respondents in all 54 participating State Agencies. Our general approach to collecting information from each source and our objectives for each, especially the central task of obtaining student dietary recalls is described below. We proceed in this description sequentially through the four sampling stages: States, schools, classrooms, and students. Recruitment letters are provided in Appendix B and data collection instruments are provided in Appendices C through I of this package.

States: The State Child Nutrition Agencies in the 16 selected States will be our source for data on schools that apply for the FFVP and those that are selected, both in SY2009-2010 and in SY2010-11. These State Agencies will also provide FFVP claims data from sample schools, including food costs, labor costs, other operating costs, and school administrative costs. All 54 State Child Nutrition Agencies will be included in a web survey to collect data on FFVP implementation (see Appendix C). Collection of data from all States will permit a complete national description of State-level FFVP characteristics and a comparison of the 16 States selected for the Impact Study with the rest of the nation. Topics will include: the FFVP selection process, criteria, and results; State FFVP policies, recommended practices, curricula and materials, and oversight; purchasing cooperatives, farm-to-school, and other programs for purchase of foods for the FFVP; non-Federal partnerships at the State level; and statewide FFVP implementation costs (including State administration).

School Food Authorities: Most information about the implementation of the FFVP in the sampled schools will be collected from their respective SFAs via a web survey, which the respondents can answer at a time convenient for them while referring to their written records (see Appendix D). General topics addressed in this survey will include satisfaction with the FFVP and specific foods offered, how fruits and vegetables are distributed through the FFVP (methods and locations, days per week, time of day, duration, frequency), types of FFVP foods offered during the preceding three months, roles of non-Federal partners at the district level, changes to the SBP and NSLP menus, SBP and NSLP meal counts during the preceding three months and the same period in SY2007-2008 (before FFVP implementation), and changes in foods offered and sales in elementary schools by school food service outside the SBP and NSLP (including a la carte, vending machines, and other venues outside the cafeterias).

School principals: Through a web survey, this group will be asked about their satisfaction with the FFVP, roles of non-Federal partners at the school level, nutrition education and promotion by non-food service personnel, and changes in food sources competing with school food service, such as vending machines, bake sales, and school stores (see Appendix E).

School environments: The context in which fruits and vegetables are offered to children as well as the schools’ overall food environment may influence students’ consumption. During our visits to the schools, we will observe and describe the school food environment with respect to types of foods and beverages available to students at school, presentation of fruits and vegetables for the FFVP, presentation of fruits and vegetables at school meals, and nutrition education and promotion (see Appendix F).

School food service managers: We will conduct brief, in-person interviews with school food service managers during which we will collect essential information for the day of the student dietary recalls: SBP and NSLP menus, with preparation information for fruits and vegetables offered during school meals; and fruits and vegetables offered through the FFVP. In addition, school food managers will be asked to complete a short questionnaire to rate their own and their staff’s satisfaction with the FFVP and with specific fruits and vegetables offered (see Appendix G).

Teachers: Teachers in sampled classrooms will be asked how favorably they view the FFVP, and how favorably they view specific fruits and vegetables offered through the FFVP. These questions will appear on a cover sheet included with the set of parental consent forms that teachers will be asked to distribute and collect (see Appendix H).

Parents: Parents of students keeping food diaries will be asked about their child’s eating habits and their satisfaction with the FFVP. These questions will be sent home with consent forms for parents to return to the school to be collected (see Appendix I).

Students (with input from parents/caretakers): Data on dietary intake of elementary school students is central to responding to the questions Congress posed concerning the impact of the FFVP on students’ consumption of fruits and vegetables and other less nutritious foods. We will collect these data from 4^th, 5^th and 6^th grade students using a diary-assisted 24-hour recall, as we believe this methodology will produce the most accurate information with the least burden on student respondents. In this section, we describe the various methods that have been used to assess dietary intakes and discuss the reasons (including validity testing, accuracy in assessing nutrient intakes, and use by researchers in other studies of elementary school age children) why the diary-assisted 24-hour recall is the most appropriate methodology for this study. We then provide additional details about how the methodology will be implemented and discuss enhancements that will reduce reporting error and burden, thus improving overall data quality.

Methods to Assess Dietary Intakes

Numerous methods have been used to assess dietary intakes of children, including food frequency questionnaires, dietary records, and 24-hour recalls. Different dietary assessment methods are appropriate for different purposes. The selection of the most appropriate method requires consideration of factors salient to this particular study, including respondent maturity, respondent burden, time and cost, and importantly, the research questions and study design.

Food Frequency Questionnaires. A food frequency questionnaire (FFQ) can be used to rank an individual's usual intake over a period, usually of one week to one year, but point estimates of intake for both foods and nutrients are unsatisfactory.⁸ One source of inaccuracy is from an incomplete listing of all possible foods. This is particularly problematic with culturally, socioeconomically or geographically diverse groups whose intake patterns can vary substantially from each other. Another source of inaccuracy involves estimation of usual serving size. While estimating the amount consumed is challenging with all dietary assessment tools, it is especially complicated with the FFQ because either a respondent is asked to estimate an average for foods that may have highly variable portion sizes across eating occasions or an average serving size is imputed that may not match an individual's intake. A third source of inaccuracy involves conceptualizing the frequency of intake of different foods over relatively long periods of time (e.g., week, month or year). Because of these errors inherent in the food frequency approach, it is generally inappropriate to use FFQ data to estimate quantitative parameters, such as the mean and variance, of a population’s usual dietary intake.⁹

The results of using an FFQ are particularly disappointing for youth. For example, a short FFQ (10 questions) was validated against a 7-day diet record in a sample of 51 Dutch adolescents.¹⁰ The mean intakes were higher on the FFQ than the diet record (50 percent higher for fruit and 29 percent higher for vegetables for boys, 101 percent higher for fruit and 56 percent higher for vegetables for girls). Spearman correlations were 0.56 for fruit and vegetables together, 0.64 for fruit, and 0.22 for vegetables. In another study, fourth and fifth grade students demonstrated acceptable reliability in completing diet records (previously validated against school meal observations), as well as weekly and monthly fruit and vegetable FFQs. However, validity was unacceptable for both the weekly and monthly FFQ versions due to significant over-reporting.¹¹ The authors concluded the diet record procedure is preferred over the FFQ procedure for collecting self-reported fruit and vegetable consumption data among children.

The Dietary Record. The dietary record involves a participant (or proxy) recording the portion size and description of all foods and beverages consumed daily over an extended period of time, usually 3-7 days. The longer the recording period, generally the more representative the record is of “usual” intake and the more accurate in quantifying foods or nutrients consumed intermittently. The record is subsequently reviewed by a trained interviewer to obtain clarification and ensure record completeness. Recording foods as they are consumed reduces errors of omission and commission and facilitates a more complete description of each item. Further, the measurement of amounts consumed provides more accurate portion sizes than if a respondent must recall portion sizes of foods previously eaten.¹² While the dietary record method is considered the gold standard of accuracy for the quantitative assessment of intake, dietary records require considerable participant literacy and motivation and can result in participants altering their eating habits for ease of recording.¹³ Underestimation of energy intake by subjects from youth to adults, and among both sexes has been documented using all forms of dietary assessment methods, and is more common among overweight subjects.¹⁴ Because of respondent burden, concern about underreporting with increasing days of records, and concern about age-appropriateness, the dietary record method was not selected for the FFVP evaluation of student intakes.

The 24-hour Dietary Recall. Compared to a dietary record, the 24-hour dietary recall method provides a shorter term assessment of dietary intake. Data on intake is obtained during an interview (usually in-person so that portion size models can be used). Because only a single day of dietary intake is assessed, the interview is relatively short and does not require the participant to be as skilled or motivated as for the dietary record method. Because of reduced burden, response rates are higher and results are thus more representative of the sample population. Because of the immediacy of the recall period, respondents are generally able to recall most of what they consume.¹⁵ In addition, interviewers can be trained to capture the details necessary so that any foods eaten can be researched later by the coding staff and coded appropriately. Interviewers can also probe for details on specific foods of interest to the research project.

One drawback of the 24-hour recall is the reliance on respondent’s memory to describe all foods and drinks consumed over the period. A second drawback relates to the representativeness of typical eating patterns. A single 24-hour recall may not be typical of the “usual” intake of an individual. With our regression discontinuity study design, however, a simple comparison of mean intake across students in FFVP schools and students in non-FFVP schools is sufficient to obtain valid estimates of the impact of FFVP. However, such an approach would not allow us to obtain precise estimates of the differences in the distributions of population subgroups or the prevalence of inadequate intake in the study population. Because of day-to-day variation in individual intake, the distribution of intakes estimated from a single day of recall data tends to be wider than the true distribution of “usual” intake over a longer time period, resulting in biased prevalence estimates. One way to deal with this shortcoming is to conduct a second 24-hour recall on a nonconsecutive day for all respondents or a subsample of respondents. This strategy allows use of standard statistical dietary assessment methodology¹⁶ to estimate the distribution of usual intake for the study population, yielding valid estimates of the prevalence of inadequate intakes and tests for differences in population subgroups.

Validity and Age Appropriateness of the Diary-assisted 24-hour Recall. For this evaluation of the FFVP, we have chosen to use a diary-assisted 24-hour recall to collect dietary intake data from students. In this context, the diary is used as a tool to assist a full 24-hour recall interview which will be conducted after completion of the diary by a trained Field Interviewer. On the surface the food diary booklet appears to be difficult for 4th-6th graders to complete. Along with OMB, some outside reviewers also had questions about the method. FNS has confidence in this method because students first participate in a motivational instruction session. In addition, teachers will be encouraged to incorporate it as a learning experience. The diary serves as an important memory aid, thus helping overcome one drawback of the 24-hour interview. Most significantly to FNS’ thinking, research has shown that a completed diary is an effective memory aid during the recall interview for children as young as 3rd grade.¹⁷ The diary is presented in Appendix J.

The basis for the FFVP diary was a diary protocol developed, piloted, and validated in the multisite National Heart, Lung, and Blood Institute Growth and Health Study (NGHS). This study is the largest longitudinal study of low-income African American and White children’s diet with 2,379 girls measured at baseline and annually for the next ten years. Girls, aged 9 and 10 years at baseline, were instructed in keeping 3-day food records consisting of 2 weekdays and a weekend day using a booklet very similar to that proposed for this study. Each girl was given measuring cups, spoons, and the same illustrated written materials included in the FFVP diary, to aid in quantifying amounts consumed. Trained nutritionists reviewed records with the girls within two weeks after they were completed to elaborate on food preparation, brand names and portion sizes. The superiority of this method was demonstrated in a validation study comparing both foods and nutrients from direct observation with data collected from 3 day food diaries, 24-hour recalls, and a food frequency questionnaire in 9 and 10 year old low income children.^18,19 Others have also found a diary-assisted 24-hour recall to provide valid assessment of intakes in children as young as the 3rd grade.²⁰ Harnack and colleagues used this method to assess intervention bias in 5th graders’ dietary reporting.²¹ Others have used a diary-assisted recall to validate a food questionnaire for 11 year olds.²²

In order to estimate usual intake, we are proposing to conduct a second 24-hour recall on a 10 percent subsample of students. We considered conducting the second recall on all students, but rejected the idea because within the fixed budget appropriated by Congress, the cost of a second recall would significantly reduce the student sample size and the number of sites where measurements could be made. These reductions would have restricted the ability to conduct subgroup analyses and diminished the face validity of the study. As discussed below, incorporating food frequency questions into model estimation will help reduce bias.

Conducting the Diary-Assisted 24-Hour Recall

The Diary-Assisted 24-Hour Recall method involves three steps:

• An early morning group motivational instruction to train students to keep diaries of their food and beverage intake;

• Students keeping the diaries with input from their parents/caregivers about foods eaten at home and at other venues; and

• An individual appointment to complete a 24-hour recall interview.

This section discusses each of these steps.

Motivational Instruction for Students. Students will attend a 40-minute group motivational instruction session with a trained Field Interviewer. The training session will be conducted using NGHS protocols to motivate and engage students in the process while they are learning in an age-appropriate classroom setting what the process entails. Field Interviewers will receive extensive training prior to conducting the sessions. During the motivational instruction session, students will be shown how to record in the food diary, and to measure foods using measuring cups and the portion size diagrams.

Further, during the motivational training, students will be instructed to obtain assistance from parents (or other household adults) on recording home prepared/supplied foods (and the diary contains information to help parents/caregivers). Studies comparing 24-hour recalls by parents with direct observation of children's food intake have shown that parents can reliably report their children's intake of foods eaten at home.²³ However, parents have limited ability to report on their children's intake of foods eaten outside of the home (Livingstone 2000). The diary-assisted 24-hour recall has the advantage of allowing for input of both the child for foods and beverages eaten away from home and parents for foods consumed at home.

Completing the Diary. Students will be instructed to begin recording information about their food and beverage intake commencing immediately following the instruction session. They will be asked to write down everything they eat and drink for the rest of the day, including food at school, in after-school programs, at home, at friends’ houses, and restaurants. Students will be asked to bring the completed diary to school with them the following morning.

Multiple Pass Interview Method. To enhance the validity of the diary-assisted recall in the FFVP evaluation, we will employ the multiple pass methodology²⁴ (Bliss 2004, Raper 2004, McBride, 2001, Moshfegh 2008) to improve recall. This standardized USDA interview protocol is used in the National Health and Nutrition Examination Survey (NHANES) to assess dietary intakes with children as young as 9 years of age as the primary respondent (with assistance from a parent or other proxy if needed) and with children as young as 12 years of age as their own respondent. The recall interview will be conducted with respondents regardless of whether they complete and/or return their food diary. While it is preferable to begin with a completed diary, training for diary keeping, together with prior notice of the recall, enhances memory. For students who bring their completed diary to the interview, we will use an adaptation of the USDA method which involves skipping the first step (the Quick List) of collecting the 24-hour recall, since the diary contains the information that would be obtained from the Quick List. For students who do not bring back a completed diary, the complete USDA method will be used.

Briefly, in the first pass (for students with a completed diary), Field Interviewers will probe for details on the diary regarding any forgotten foods and beverages consumed during the prior 24 hours, the time of food consumption, the meal occasion (breakfast, lunch, dinner or morning, afternoon or evening snack), and where the food was eaten and obtained (home, school, friends' home, fast food, restaurant, other). For the FFVP study, information on where food was obtained and eaten will be essential for accurately coding foods and capturing which fruits and vegetables are obtained from school.

In the second pass, Field Interviewers will probe for details about each food and beverage on the diary including: descriptive information about each food (e.g., type, flavor, brand name, main ingredients of mixed foods) using a list of standard NHANES probes; additions to and/or accessory food items; and the amount consumed (using appropriate portion size models and standardized probes). Specific probes will be used to assess the various forms of fruits and vegetables (e.g. fresh, frozen) consumed. In the final and third pass, Field Interviewers will read aloud to the student the list of foods and details collected to ensure completeness and accuracy of the information. At this point, probes are used one final time to encourage the students to recall snack items or other foods or beverages that they may initially have forgotten to mention. The use of this method will ensure that the time burden during the interview is minimized.

Enhancements to the Collection of Dietary Information Using the Diary-Assisted Recall

Our data collection methodology includes two enhancements that are designed to both reduce burden on students and reduce reporting error, thereby improving data accuracy. First, we are collecting detailed information on school meals and snacks directly from the School Food Service Manager. Second, we are collecting limited food frequency information from students to better estimate usual intake.

School Foodservice Manager Interview. The primary purpose of the School Foodservice Manager Interview is to gather important descriptive information about the foods and beverages served to students as part of school meals and snacks that may be recorded in the students' food diaries. This information will be gathered from the school foodservice manager to both reduce burden on students in describing the foods eaten at school and to provide accuracy and consistency of information. Field Interviewers will receive training on what to record about the school foods served for the meals and snacks that students may record in their diary (e.g., description, brand name, product code, the portion size, how it was prepared, nutrient information for packaged foods, recipe number for USDA recipes, whether it is a USDA commodity food, condiments available, and types of milk served and carton colors for each milk type). Field Interviewers will also be trained to use the School Foodservice Manager Interview data and menu for probing students during the recall interview. For example, a student could report that he had “a macaroni or spaghetti thing” with “some red sauce” on it at lunch at school. The Field Interviewer would then check which of the items on the school menu match this description in order to clarify what was eaten.

Student Questionnaire. As discussed above, food frequency questionnaires (FFQ) do not provide accurate estimates of quantitative parameters such as the mean and variance of a population’s dietary intake. They can, however, be incorporated into models for estimating usual intakes of fruits and vegetables based on 24-hour recall data, potentially increasing precision and thereby improving the power to detect relationships between dietary intake and other variables.

Food frequency questions included on the student questionnaire are derived from two sources. Questions on the frequency of consumption of fruits and vegetables were taken from the Centers for Disease Control’s Youth Risk Behavior Surveillance System (YRBSS) questionnaire. We selected these items for several reasons:

• their widespread use;

• previous validity testing, albeit in adolescents and not among elementary school children;

• no other validated instruments were available for use with younger children; and

• 4th - 6th grade students on whom we pilot tested the tools found the questions easy to complete.

Outside reviewers suggested we also include questions about frequency of snack foods and beverages, to compare with estimates we obtain from the 24-hour diary-assisted recalls. We could find no questions which had been validity tested in the age group of interest. However we did find such a questionnaire for older children, the Beverage and Snack Questionnaire (BSQ).²⁵ We selected questions from the BSQ and included them in our revised student questionnaire. We improved the layout and the look of our questionnaire and included clip art pictures to aid younger students in responding to the questions.

The questionnaire (included in Appendix J) will be completed by students at home, where they will have the opportunity to ask parents or others in the household for assistance as necessary.

Use of the Information

The information gathered in the data collection activities described above will be used by FNS to determine if students at FFVP schools have higher fresh fruit and vegetable consumption than students at non-participating schools, and whether FFVP induced other dietary changes such as decreased consumption of less nutritious foods among students. The data collection described in this document is essential for assessing the impacts of the FFVP and will provide FNS with descriptive information about the schools and students that participate in the FFVP as well as how the program has been implemented at schools in all States. There is currently no other national effort that can address the research objectives of the proposed study.

A.3 Uses of Improved Information Technology to Reduce Burden

The study strives to comply with the E-Government Act of 2002 (Public Law 107-347, 44 U.S.C. Ch 36) by using web-based surveys for State Child Nutrition Agencies, School Food Authorities, and school principals to facilitate convenient, timely, and accurate collection of information. Respondents will be able to complete and submit the web-based survey on-line. Features of the web-based surveys that are intended to reduce burden and increase item response include: closed-ended questions with specific answer categories; skip pattern questions that only some respondents need answer; highlighted “key words” that respondents can click on to see standard definitions; and a “contact us” icon respondents can use to send an e-mail question to the survey director. The web-based surveys are designed so that respondents can complete part of the survey, check their records or research answers to questions if they are not sure, and then complete the survey at a later time. The web-based surveys also allow for multiple authorized respondents to access the survey. An agency-specific username and password will be required each time the survey is accessed. The use of on-line survey technology will allow the study contractor to track completed questionnaires in real time, and eliminates coding errors that might be introduced during data entry. FNS estimates approximately 10 percent of respondents will use the web-based survey. The web address will be provided once the contractor has finalized the design of the website.

One-on-one data collection is considered necessary with Impact Study subjects (students) as a design feature of the study. In-school interviews with school food service managers are designed to be short and focused, as are paper and pencil questionnaires for parents and teachers. Administrative data such as FFVP claims data from sample schools in the 16 States will be collected from their respective State agencies thus reducing the burden on the schools and SFAs.

Sample tracking will be managed by our Field Management System (FMS), a major application composed of a set of interrelated applications that control all aspects of sampling, data collection, data cleaning, and delivery of survey data.

A.4 Efforts to Identify and Avoid Duplication

The data requirements for the evaluation have been carefully reviewed to determine whether the needed information is already available. Efforts to identify duplication included a review of FNS reporting requirements, State administrative agency reporting requirements, and special studies by government and private agencies. It was concluded that no existing data sources can provide data needed to answer the study’s research questions.

A.5 Efforts to Minimize Burden on Small Businesses or Other Entities

Information being requested or required has been held to the minimum required for the intended use. Although smaller State agencies and SFAs are involved in this data collection effort, they deliver the same program benefits and perform the same function as any other State agency and SFA. Thus, they maintain the same kinds of information on file. FNS estimates that one percent of our respondents are small entities, approximately 40 respondents.

A.6 Consequences of Less Frequent Data Collection

The data collection for the proposed study will be conducted one time only. Without this effort, FNS will not have the data necessary to estimate program impacts on participating students, or to examine how the Fresh Fruit and Vegetable Program is currently being implemented, which will be used to produce the required report to Congress and inform future program decisions.

A.7 Special Circumstances Requiring Collection of Information in a Manner Inconsistent with Section 1320.5(d)(2) of the Code of Federal Regulations

In order to obtain valid estimates of the usual nutrient intake in a study population, the generally accepted technique is to have a second 24-hour recall from the same respondents.^26,27 Therefore, the study design requires a second record-assisted 24-hour recall to be collected for 10 percent (614) of the students in the sample on a nonconsecutive day in the week directly following the first round of data collection.

There are no other special circumstances. The collection of information is conducted in a manner consistent with guidelines in 5 CFR 1320.5.

A.8 Federal Register Comments and Efforts to Consult with Persons Outside the Agency

An announcement was published in the Federal Register on September 1, 2009, and specified a 60-day period for comment ending on November 2, 2009 (Federal Register Volume 74, No. 168, Page 45159). A copy of the Federal Register Notice is provided in this package. Comments from consultants and public comments received by FNS are included in Appendix K.

Consultations Outside the Agency

Abt Associates Inc., the evaluation contractor hired by FNS, contributed to the design of the Food and Nutrition Service Evaluation of the Fresh Fruit and Vegetable Program, including survey instrument design. Key Abt Associates Inc. staff included: Susan Bartlett (Project Director), Jacob Klerman (Director of Analysis), Michael Battaglia (Sampling Statistician). In addition, FNS has consulted with staff at the USDA Center for Nutrition Policy and Promotion, USDA Agricultural Research Service, Centers for Disease Control and Prevention, and the NIH National Cancer Institute.

Comments were received from outside consultants including Sonia Kim of the Centers for Disease Control and Prevention, Laura Leviton and Punam Ohri-Vachaspati of the Robert Wood Johnson Foundation and Lorelei DiSogra of the United Fresh Produce Association. The comments in their original format are included in Appendix K. In addition, comments from these outside reviewers, sorted by topic area are also provided in Appendix K with responses from Abt Associates. Many of the comments included specific comments on instruments and a number of comments focused on three main issues, which required further discussion: 1) Dose response or Full vs. Partial Implementation of FFVP, 2) Age appropriateness of the Instruments, and 3) Representativeness of the sampled schools. Abt prepared a memo responding to each of these issues, which is also included in Appendix K.

The information collection has been reviewed by the National Agricultural Statistics Service of USDA (NASS) with special reference to the statistical procedures. A report was prepared by NASS and the issues raised were reviewed. The recommendations were taken into consideration and modifications were introduced accordingly.

A.9 Payments to Respondents

Each of the sampled schools will designate a school liaison to assist with the data collection effort in their respective school by distributing study packets to sampled classrooms, collecting parent consent forms and completed questionnaires, distributing and collecting teacher questionnaires, securing a suitable location within the school for interviewing students, and escorting students to and from the assisted 24-hour recall session with data collectors. An incentive payment of $100, based on expected staff time costs, will be provided to each school to improve response rates. Students who complete their 1-day food diary will receive a small novelty item as a gift for their participation with a per-item dollar value of $5.00. Web survey respondents will be sent a mouse pad or other office supply item, valued at no more than $2.00, with the study name and URL to encourage completion of the survey.

A.10 Assurance of Confidentiality

The individuals participating in this study will be assured that the information they provide will not be released in a form that identifies them. No identifying information will be attached to any reports or data supplied to USDA or any other researchers. The only exception to this is that data provided by State Child Nutrition Directors may be presented in a way that identifies specific States.

Abt Associates Inc. has extensive experience in data collection efforts requiring strict procedures for maintaining the confidentiality, security, and integrity of data. Specific data handling and reporting procedures will be employed to maintain the privacy of survey participants and composite electronic files. These data handling and reporting procedures include requiring all project staff, both permanent and temporary, to sign a confidentiality and nondisclosure agreement. In this agreement, staff pledges to maintain the confidentiality of all information collected from the respondents and will not disclose it to anyone other than authorized representatives of the evaluation, except as otherwise required by law.

In addition, Abt Associates Inc. has established a number of procedures to ensure the confidentiality and security of electronic data in their offices during the data collection and processing period.  A system of record notice (SORN) titled USDA/FNS-8 FNS Studies and Reports published in 56 FR 19078 on April 25, 1991 discusses the terms of protections that will be provided to respondents.

Specific precautions will also be taken for the web-based surveys. Surveys will use session cookies to allow respondents to move from page to page without reentering identifying information. The surveys will not use persistent cookies to track respondents’ browsing habits. Upon completion of the surveys by all State agencies, the website will be dismantled.

Institutional Review Board

Abt Associates maintains its own Institutional Review Board (IRB), which serves as the organization’s administrative body that conducts prospective reviews of proposed research and monitors continuing research for the purpose of safeguarding research participants’ rights and welfare. All research involving interactions or interventions with human subjects is within the purview of the Abt IRB. Abt Associates’ IRB is the local agent responsible for ensuring that the organization’s research: 1) meets only the highest ethical standards; and 2) receives fair, timely, and collegial review by an external panel. Abt Associates’ IRB currently holds a federal-wide assurance (FWA) of compliance from the U.S. Department of Health and Human Services’ Office of Human Research Protections (DHHS/OHRP). The FWA covers all federally supported or conducted research involving human subjects. All study materials and instruments for the Fresh Fruit and Vegetable Program evaluation were submitted and approved by Abt’s IRB prior to the pretest activities. Final instruments and other materials have been submitted to Abt’s IRB who will review and approve data collection activities for the full study.

A.11 Questions of a Sensitive Nature

There are no personally sensitive questions contained in the State, SFA, principal, school food service manager, teacher, or student data collection instruments.

One item in the parent survey could qualify as a sensitive issue. This item asks parents to report whether their child was approved for free or reduced price meals at school. This information will be needed for the analysis. Parents who participated in the pre-test were interviewed briefly after completing the survey to discuss any difficulties or concerns with the survey. No respondent indicated an unwillingness or discomfort in providing this information.

A.12 Estimates of Respondent Burden

Exhibit 1 shows sample sizes, estimated burden, and estimated annualized cost of respondent burden for each part of the data collection and for total burden. Burden for pretest and planned data collection are both included.

Exhibit 1: Estimated Respondent Burden
Respondent		Estimated No. Respondents	Responses Annually Per Respondent	Total Annual Responses	Estimated Avg. # of Hours Per Response	Estimated Total Hours	Average Hourly Wagea	Annualized Cost of Respondent Burden
Students – record-assisted dietary recalls	Pretest	9	1.00	9.0	1.2500	11.3	$5.00	$56.50
	Completed	6,144	1.00	6,144.0	1.0833	6,655.8	$5.00	$33,279.00
	Attempted	1,536	1.00	1536.0	0.0500	76.8	$5.00	$384.00
Students –second dietary recall (10% subsample)	Pretest	0	1.00	0.0	0.5830	0.0	$5.00	$0.00
	Completed	614	1.00	614.0	0.5833	358.1	$5.00	$1,790.50
	Attempted	154	1.00	154.0	0.0500	7.7	$5.00	$38.50
Parents	Pretest	5	1.00	5.0	0.0500	0.3	$20.62	$6.19
	Completed	6,144	1.00	6,144.0	0.0833	511.8	$20.62	$10,553.32
	Attempted	1,536	1.00	1,536.0	0.0500	76.8	$20.62	$1,583.62
Teachers	Pretest	2	1.00	2.0	0.0833	0.2	$37.07	$7.41
	Completed	691	1.00	691.0	0.0833	57.6	$37.07	$2,135.23
	Attempted	77	1.00	77.0	0.0500	3.9	$37.07	$144.57
School Food Service Mangers	Pretest	3	1.00	3.0	0.6667	2.0	$16.99	$33.98
	Completed	243	2.00	486.0	0.3333	162.0	$16.99	$2,752.38
	Attempted	13	2.00	26.0	0.0500	1.3	$16.99	$22.09
School Principals	Pretest	9	1.00	9.0	0.5000	4.5	$46.97	$211.37
	Completed	704	1.00	704.0	0.3333	234.6	$46.97	$11,019.16
	Attempted	112	1.00	112.0	0.0500	5.6	$46.97	$263.03
State CN Directors	Pretest	4	1.00	4.0	1.0000	4.0	$35.50	$142.00
	Completed	49	2.00	98.0	0.3333	32.7	$35.50	$1,160.85
	Attempted	5	2.00	10.0	0.0500	0.5	$35.50	$17.75
State CN Directors (Administrative Data)	Pretest	0	2.0	0.0	0.5000	0.0	$35.50	$0.00
	Completed	16	2.00	32.0	0.5000	16.0	$35.50	$568.00
	Attempted	0	2.00	0.0	0.0500	0.0	$35.50	$0.00
School Food Authority (SFA) Directors	Pretest	9	1.00	9.0	1.0000	9.0	$24.66	$221.94
	Completed	704	1.00	704.0	0.5000	352.0	$24.66	$8,680.32
	Attempted	112	1.00	112.0	0.0500	5.6	$24.66	$138.10
Totals		18,895		19,221.0		8590.1		$75,209.80

^a Sources for Pretest and Planned Data Collection: Students: The value of the novelty item planned for student gifts for their participation.

Bureau of Labor Statistics, National Compensation Survey, 2008. August, 2009, Bulletin 2720. (http://www.bls.gov/ncs/ncswage2008.htm):

Parents: Average Hourly Earnings of Civilian Workers. Teachers: Average hourly earnings of elementary and middle school teachers, except special education.

School Food Service Managers: Average hourly earnings of First-line supervisors/managers, food preparation and serving workers,

School Principals: Average hourly earnings of education administrators, elementary and secondary school.

State CN Directors: Average hourly earnings of State and local government administrative services managers.

School Food Authority (SFA) Directors: Average hourly earnings of State and local government food service managers.

A.13 Estimates of Other Annual Costs to Respondents

There are no capital, start-up, or annualized maintenance costs associated with this data collection for respondents.

A.14 Estimates of Annualized Government Costs

This includes the costs associated with the contractor conducting the project and the salary of the assigned FNS project officer. The cost to the Federal government for the all tasks associated with the Food and Nutrition Service Evaluation of the Fresh Fruit and Vegetable Program (FFVP) is $2,649,962. This cost includes all study tasks, including design, pretests, sample selection, recruitment, information collection, analysis and report writing.

A.15 Changes in Hour Burden

This is a new collection of information. This program change will add 8590 burden hours to the OMB collection inventory.

A.16 Time Schedule, Publication, and Analysis Plans

Study Schedule

The schedule shown in Exhibit 2 lists the expected period of performance for the data collection and reporting. Most data collection activities will be completed during the spring of 2010 before the end of the school year. The second round of the State web-based survey is expected to take place in the fall of 2010.

Exhibit 2: Anticipated Schedule for Data Collection and Reporting
Activity	Time Schedule
Collect data (State CN Director, School principal and SFA Director web surveys; student record-assisted recalls, school food service manager interviews, parent and teacher surveys. administrative data from States)	March 2010 – October 2010
Create data files and conduct analysis	August 2010 – November 2010
Prepare final report Final Briefing for FNS	October 2010 – March 2011 April 2011

Analysis Plan

The main lines of analysis follow the six broad research categories outlined in section A.1. Each of the research questions within these categories is explicitly or implicitly associated with one or more outcome measures to be analyzed. Our approach, including data sources for each question and the planned analyses are summarized in Exhibit A-1 in Appendix A.

Analysis Methods

Tabulations and cross-tabulations: The descriptive research questions will be addressed by tabulations or cross-tabulations. For example, simple tabulations will be used to characterize schools that applied for the FFVP. Cross-tabulations will be used to examine outcomes by school characteristics, and to compare groups, for example to characterize students who participate in the FFVP never, sometimes, or usually.

Econometric models: The main impacts of the FFVP on student outcomes will be estimated through multivariate econometric modeling, incorporating the regression discontinuity design described in Section B. Because students are clustered within schools, we will view the data as multi-level and will estimate the models using SAS SURVEYREG.

Exploratory regression models: The contextual effects of nutrition education and FFVP delivery systems will be estimated through exploratory regression models. After conducting simple comparisons of outcomes for schools grouped by intensity of nutrition education and by types of fruits and vegetables offered, we will perform multivariate analyses controlling to the extent possible for school characteristics such as racial and ethnic composition, poverty, and location.

Publication of Study Results

The study’s findings will be presented in a report that will undergo peer review. Once final, FNS will make the final report and executive summary available on its web site. Findings may also be published in one or more journals.

A.17 Display of Expiration Date for OMB Approval

All data collection instruments for the Food and Nutrition Service Evaluation of the Fresh Fruit and Vegetable Program (FFVP) will display the OMB approval number and expiration date. The OMB approval number and expiration date will be displayed on the initial screen of the web surveys.

A.18 Exceptions to Certification Statement

There are no exceptions to the Certification for Paperwork Reduction Act (5 CFR 1320.9) for this study.

1 K. He, F.B. Hu, G.A. Colditz, J.E. Manson, W.C. Willett, and S. Liu, Changes in intake of fruits and vegetables in relation to risk of obesity and weight gain among middle-aged women, International Journal of Obesity, 28, 1569-1574, 2004.

2 B.J. Rolls, J.A. Ello-Martin, and B. Carlton-Tohill, What can intervention studies tell us about the relationship between fruit and vegetable consumption and weight management? Nutrition Reviews, 62, 1-17, 2004; S.A. Jebb, Dietary strategies for the prevention of obesity, Proceedings of the Nutrition Society, 64(2), 217-227, 2005; B. Carlton-Tohill, Fruits and vegetables consumption and body weight management, Acta Horticulturae, 744, 39-46, 2007.

3 K.A. Steinmetz and J.D. Potter, Vegetables, fruit, and cancer prevention: a review, Journal of the American Dietetic Association, 96, 1027-1039, 1996; E. Riboli and T. Norat, Epidemiologic evidence of the protective effect of fruit and vegetables on cancer risk, American Journal of Clinical Nutrition, 78(3 Suppl), 559S-569S, 2003; L. Dauchet, P. Amouyel, S. Hercberg, and J. Dallongeville, Fruit and vegetable consumption and risk of coronary heart disease: A meta-analysis of cohort studies, Journal of Nutrition, 136(10), 2588-2593, 2006.

4 J. de Sa and K. Lock, Will European agricultural policy for school fruit and vegetables improve public health? A review of school fruit and vegetable programmes, European Journal of Public Health, 18(6), 558-568, 2008.

5 U.S. Department of Agriculture, Food and Nutrition Service, Fresh Fruit and Vegetable Program Handbook, August 2008. Retrieved from http://www.fns.usda.gov/cnd/ffvp/Resources/
FFVPhandbookFINAL.pdf

6 Other research designs considered were random assignment and pre-and post comparisons between treatment and comparison schools over time. A random assignment approach was not feasible for this study due to USDA’s FFVP Regulations that allow States the choice of which schools will have FFVP, rather than allowing the evaluator to make the determination. Additionally, it appeared that many States would choose the schools before the start of the evaluation. There was also a concern about the legality of withholding FFVP from eligible schools that were randomly assigned to the control group. The pre/post design was not used because it would require larger sample sizes, the requirement of the pre-FFVP measurement would roughly double the data collection costs, and seasonality of diet would require that optimally “before” and “after” measurements be made in the same month in successive school years. This would push the project timelines out and possibly set FNS up for late delivery of the congressionally mandated final report.

7 Development of web surveys will begin after pre-testing is completed. The URL will be provided once the web surveys are developed and tested.

8 Thompson FE, Subar AF. Dietary assessment methodology. In: A.M. Coulston, C.L. Rock and E.R. Monsen, Editors, Nutrition in the Prevention and Treatment of Disease. 2nd ed, Academic Press, San Diego, CA. 2001.

9 Thompson and Subar, Dietary assessment methodology.

10 Van Assema P, Brug J, Ronda G, Steenhuis I, Oenema A. A short Dutch questionnaire to measure fruit and vegetable intake: relative validity among adults and adolescents. Nutr Health. 2002;16:85-106.

11 Domel SB, Baranowski T, Davis H, Leonard SB, Riley P, Baranowski J. Fruit and vegetable food frequencies by fourth and fifth grade students: validity and reliability. J Am Coll Nutr. 1994;13:33-9.

12 Thompson and Subar, Dietary assessment methodology.

13 Rockett HR, Berkey CS, Colditz GA. Evaluation of dietary assessment instruments in adolescents. Curr Opin Clin Nutr Metab Care. 2003;6:557-62.

14 Bratteby LE, Sandhagen B, Fan H, Enghardt H, Samuelson G. Total energy expenditure and physical activity as assessed by the doubly labeled water method in Swedish adolescents in whom energy intake was underestimated by 7-d diet records. Am J Clin Nutr. 1998;67:905-11.

15 Thompson and Subar, Dietary assessment methodology.

16 Carriquiry AL. Estimation of usual intake distributions of nutrients and foods. J Nutr. 2003;133:601S-8S.

17 Lytle LA, Nichaman MZ, Obarzanek E, Glovsky E, Montgomery D, Nicklas T, Zive M, Feldman H. Validation of 24-hour recalls assisted by food records in third-grade children. The CATCH Collaborative Group. J Am Diet Assoc. 1993;93:1431-6.

18 Crawford PB, Obarzanek E, Morrison J, Sabry ZI. Comparative advantage of 3-day food records over 24-hour recall and 5-day food frequency validated by observation of 9- and 10-year-old girls. J Am Diet Assoc. 1994;94:626-30.

19 Many publications have been based on these data, including: Crawford PB, Obarzanek E, Schreiber GB, Barrier P, Goldman S, Frederick MM, Sabry ZI. The effects of race, household income and parental education on nutrient intakes of 9- and 10-year-old girls: NHLBI Growth and Health Study. Ann Epidemiol. 1995;5:360-8; McNutt SW, Hu Y, Schreiber GB, Crawford PB, Obarzanek E, Mellin L. A longitudinal study of dietary practices of black and white girls 9 and 10 years old at enrollment: The NHLBI Growth and Health Study. J Adolesc Health. 1997;20:27-37; Striegel-Moore RH, Thompson D, Affenito SG, Franko DL, Obarzanek E, Barton BA, Schreiber GB, Daniels SR, Schmidt M, Crawford PB. Correlates of beverage intake in adolescent girls: the National Heart, Lung, and Blood Institute Growth and Health Study. J Pediatr. 2006;148:183-7; Wang MC, Crawford PB, Moore EC, Hudes M, Sabry ZI, Marcus R, Bachrach LR. Influence of adolescent diet on quantitative ultrasound measurements of the calcaneus in young women. Osteoporos Int. 1999;9:532-5; and Ritchie LD, Spector P, Stevens MJ, Schmidt MM, Schreiber GB, Striegel-Moore RH, Wang M, Crawford PB. Dietary patterns in adolescence are related to adiposity in young adulthood: An analysis of data from the longitudinal NHLBI Growth and Health Study of Black and White females. J Nutr 2007;137:399-406.

20 Lytle et al., Validation of 24-hour recalls.

21 Harnack, L, Himes J, Aniker J, Clay T, Gittelsohn J, Jobe J, Ring K, Snyder P, Thompson J Weber J. Intervention-related Bias in Reporting Food Intake by Fifth-Grade Children Participating in an Obesity Prevention Study. Am J of Epid. 2004:160:1117-1121.

22 Moore L, Tapper K, Dennehy A, Cooper A. Development and testing of a computerized 24 –h recall questionnaire measuring fruit and snack consumption among 9-11 year olds. Eu J Clin Nutr. 2005:59:809-816.

23 Klesges RC, Hanson CL, Eck LH, Durff AC. Accuracy of self-reports of food intake in obese and normal weight individuals: effects of parental obesity on reports of children’s dietary intake. Am J Clin Nutr. 1998;48:1252-6; Eck LH, Klesges RC, Hanson CL. Recall of a child’s intake from one meal: are parents accurate? J Am Diet Assoc.1989;89:784-7; Basch CE, Shea S, Arliss R, Contento IR, Rips J, Gutin B, Irigoyen M, Zybert P. Validation of mothers’ reports of dietary intake by four to seven year-old children. Am J Public Health. 1990;80:1314-7; and Baranowski T, Sprague D, Baranowski JH, Harrison JA. Accuracy of maternal dietary recall for preschool children. J Am Diet Assoc. 1991;91:669-74.

24 Bliss RM. Researchers produce innovation in dietary recall. Agric Res. 2004;52:10-12; Raper N, Perloff B, Ingwersen L, Steinfeldt L, Anand J. An overview of USDA’s Dietary Intake Data System. J Food Comp Anal 2004;17:545-55; McBride J. Was it a slab, a slice, or a sliver? High tech innovations take survey to new level. Agric Res. 2001;49:4-7; and Moshfegh AJ, Rhodes DG, Baer DJ, Murayi T, Clemens JC, Rumpler WV, Paul DR, Sebastian RS, Kuczynski KJ, Ingwersen LA, Staples RC, Cleveland LE. The US Department of Agriculture Automated Multiple-Pass Method reduces bias in the collection of energy intakes. Am J Clin Nutr. 2008;88:324-32.

25 Neuhouser ML, Lilley S, Lund A, Johnson DB. Development and validation of a beverage and snack questionnaire for use in evaluation of school nutrition policies. J Am Diet Assoc. 2009;109:1587-92.

26 S. M. Nusser, A. L. Carriquiry, et al., A semi-parametric transformation approach to estimating usual nutrient intake distributions, J. Am. Stat. Assoc., 1996; 91: 1440-1449.

27 Institute of Medicine (IOM). Dietary Reference Intakes: Applications in dietary assessment. Washington, DC: National Academies Press; 2000.