Appendix K Comments

Appendix K: Public Comments and Comments from Consultants Received

Sonia Kim, PhD
DNPAO, CDC
Review of USDA OMB FFVP Evaluation Study materials

Goals of the project:
- There are currently 2 stated goals of the project: assess impact and
implementation.
- Suggestion: add a 3rd goal: determine the reach or coverage of the FFVP.
o Rationale
 The Background section implies that lower-income students are a
priority for the program; therefore it is important to know how well
this target population is actually being reached.
 This information is especially relevant considering the large
increases in funding that will occur by 2012 and that the number of
students served by the FFVP will most likely increase as well.
 Additionally, states will most likely be interested in this information.
o Method
 Using the State Child Nutrition Agency Survey, collect the
following information: (some or all of this information is already
asked)
 Total number of eligible schools
 Of the eligible schools how many apply
 Of the eligible schools that apply, how many receive
funding
 Since the Implementation sample will be nationally representative,
would these numbers be nationally representative as well?
 The states will be interested in their own data and would find it
useful to do state-by-state comparisons also. Thus, could this data
also be collected from all 54 state agencies?
Samples
- Will the implementation and impact data be representative at the state level for
the 16 State Agencies and/or 54 State Agencies?
- It would be useful to be able to link this FFVP data collected to CDC’s youth
behavior data, such as YRBSS and School Health Profiles that are collected by
the Division of Adolescent and School Health (DASH)
(http://www.cdc.gov/healthyYouth/profiles/);
(http://www.cdc.gov/HealthyYouth/yrbs/index.htm)
- For the Impact study: I understood that eligible schools that participated in FFVP
will be compared will other eligible schools that did participate. Is this correct?
- “State Cutoff”
 The definition of this phrase is not clear. On pg. 17 “The FFVP
legislation and FNS regulations require States to give FFVP
funding to the poorest schools, as measured by the percent of
students eligible for free and reduced price school lunches. RD
estimates the causal impact of the FFVP by comparing schools
directly above and below the cut-off for funding.”






This implies that the “State Cutoff” is a demarcation for eligibility.
Thus comparing schools on either side of the cutoff implies that
one group is eligible, while the other is not.
Does FNS have an eligibility cut-off? Does the State Cutoff refer to
a specific criteria set by the state? I.e. by FNS standards the
schools could be considered eligible, but by the state’s standards
they are not eligible? OR because there is a narrow free/reduced
price window, comparing above and below the cutoff results in the
comparison of very similar schools (even though technically one
group is eligible and one is not)? This distinction should be made
more clear.
The definition of “State Cutoff” should be made clearer in the
diagram on pg. 19.

Impact data
- Nutritional status
 On pg. 4, it states that the impact study will look at children’s
nutritional status. Is this information being collected?
- Willingness to try new fruits and vegetables
 This is an important part of attitudes. There is one question about
it (pg. 8, q 16 of the self-administered survey). Other questions
could be added for more depth on this issue. (Alice Ammerman of
UNC has done work on this topic.)
- Increased fruit and vegetable consumption
 Pg. 8 states that the information will be used to determine whether
“…the FFVP increased fruit and vegetable consumption…”
 Is there a pre/post design?
 If not, it is more appropriate to say that the data will be
used to determine if students at FFVP schools have higher
fruit and vegetable consumption than students at nonparticipating schools.

Review of Evaluation Plan: Evaluation of the Fresh Fruit and
Vegetable Program
Laura C. Leviton and Punam Ohri-Vachaspati
The Robert Wood Johnson Foundation
September 16, 2009
Thank you for the opportunity to review and comment on this evaluation
proposal. This is an important program, very much in line with the
needs of our own organization to enable the prevention of childhood
obesity. We have some general comments on the aims and design, and
then we would like to turn our attention to the measurement and data
collection issues, where we have most of our suggestions for
improvement.
General Comments on the Aims and Design
The aims of the study are clear and the proposed design, the use of
regression discontinuity, is excellent. The OMB Clearance Package is
correct that this is a design that is as rigorous as a randomized
experiment under the conditions described in the evaluation plan. A
great many tests and comparisons of the two methods have been
conducted, especially in the school environment. The use of schools
as the unit of assignment is intelligent and appropriate, and the
proposed hierarchical analysis is also highly appropriate. The
proposed data collection of a larger group of schools, to assess
implementation is also important and appropriate.
We have three major concerns and suggestions. Two of them concern
measurement and data collection, and one is analytic. We address the
specifics of measurement in the next section. In summary the two
concerns are:
1. The need to assure comparability of measures with existing high
quality surveys, specifically SNDA III and the Bridging the Gap
surveys of school policies and implementation. Both these groups
have studied food access, availability and consumption issues
affecting school children. Also consult the NCI Measures of Food
Environment website (https://riskfactor.cancer.gov/mfe) - a
compilation of studies investigating community-level measures of the
food environment, including school food environment. This will
allow for use of tested measures that have been and are being used
for tracking changes in perceptions, behaviors, physical
environments, and policies in the school food setting.
Because the proposed surveys will be conducted only once, we
desperately want and need a basis of comparison. This is in line
with the stated aims of NIH and CDC to develop common measures of
policy and environmental factors contributing to the epidemic of
childhood obesity. In general, we will want a basis of comparison
and a context to interpret the results obtained. It would be sheer
folly not to have comparability where feasible. This does not
duplicate efforts in any sense, if that needs to be explained to
OMB; the purpose of the study remains the same and the data
collection is indispensable to do what needs to be done.

1

The instruments for school administrators and food service clearly
borrowed extensively from SNDA III; however, Abt Associates should
take a fresh look at the content of SNDA III in any case and contact
the developers at Mathematica.
We urge them to employ questions that are comparable to the Bridging
the Gap survey. This is an annual survey of a representative sample
of 500 to 700 school districts and schools (elementary, middle and
secondary). The surveys can be found at
http://www.monitoringthefuture.org/ and/or
http://www.impacteen.org/aboutus.htm or by contacting the principal
investigators:
Frank Chaloupka,
[email protected]
Department of Economics
U. Illinois Chicago
Room 558, M/C 275
1747 West Roosevelt Road
Chicago
IL
60608
Voice: 312-413-2287
Fax: 312-355-2801
Lloyd Johnston, [email protected]
Survey Research Center
University of Michigan Institute for Social Research
426 Thompson Street, Room 2324
Ann Arbor
MI
48106-1248
Voice: 734-763-5043
Fax: 734-936-0043
1. The problems of response rate, missing data, and age
inappropriate questions. The finest design in the world cannot
overcome problems that limit the accuracy of data collection.
Impairing the accuracy of data will, in turn, guarantee a no
effect conclusion. Error in measurement introduces noise in that
which we want to analyze. Unfortunately, with the current
instruments and data collection plan, the evaluation is certainly
headed toward a no-effect conclusion. We realize Abt is not
permitted to pilot test the instruments before OMB approval, but
really, given the experience to date in surveying school
administrators, this is worrisome.
a. The instruments for administrators are unnecessarily awkward
to use, impose a large response burden where it is not
necessary, and will therefore impair both response rate and
accuracy.
It is a fundamental principle of survey research
that increased response burden will increase error and missing
data. See details below.
b. Response rate for administrators will be a serious problem
even if the surveys are made more user friendly. The
incentives described will not be sufficient to guarantee the
response rate that Abt is targeting, based on our recent
experience using the web to collect data from school
personnel. There was no description in the OMB package of how
Abt proposes to ensure the response rate they need. In any
revision, it will be important for USDA to know in detail, how
Abt plans to follow up with administrators and food service
personnel to get them to respond. There should be a specific
and ample line item in their budget for labor to convert non2

respondents into respondents. The data collection component
is the biggest budget item for any evaluation project such as
this, and it is the one area that a low bid on the RFP will
impair the most. It will be important to make sure that labor
is assigned to the specific task of assuring the necessary
response rate.
c. The instruments for children are not age appropriate and there
are better instruments available for both the family surveys
and 24 hour recalls. Fourth and fifth graders will not be
able to respond to some of these issues in the formats
provided.
2. Greater analytic attention, and data collection where possible, to
consumption of less nutritious foods. The entire rationale for the
program rests on the assumption that increased fruit and vegetable
consumption will lead to decreased consumption of less nutritious
foods. The background section lists other causal relationships
(e.g. cancer incidence) as purely secondary to the epidemic of
childhood obesity. The rationale is in the legislation as described
page 3. Yet the analytic plan has a fatal flaw. There is no direct
evidence that increasing F&V consumption in children will cause
decreased consumption of calorie dense foods of limited nutritional
value. Yet increasingly there is evidence (e.g. Gortmaker and Wang,
Sturm) that we will only prevent childhood obesity by decreasing the
consumption of calorie dense food of limited nutritional value.
Yet the data collection and analytic plan do not pay sufficient
attention to this issue. The 24 recalls will yield some of the
information, but the self-administered student instrument should
th
address this in depth — see Exhibit A-2, 7 page. Regardless of
decisions about the self-administered
instrument, analytic questions
th
about this issue, as seen on 7 page of Exhibit A-2, should take
higher priority! Unlike data collection, adding another analysis
costs very little, and could tell us so much.
Specific Suggestions on the Aims and Design
1. The statement of aims for the program is very clear on page 2 of
the OMB Clearance Package. However, the introductory statement
under Background, page 1, is not. The program is about so much
more than teaching healthier eating habits, and in fact this
statement is misleading. We would urge you to take another look
at the expanded statement on page 2 to restate the first sentence
under Background, page 1.
2. On page 3, first full paragraph, it is important to estimate the
number of children served by the program, and if possible, the
numbers at each stage of program expansion.
3. On page 6, in smaller districts the school food authorities may
be the same individuals as the school food managers. How will
this be addressed? Remember response burden!
4. On page 7, if there is enough variation in nutrition education
then analysis examining the dose of nutrition education on
outcome variables. This could be addressed on Exhibit A-2,
second to last page.
5. In Exhibit A-2 Topic area 2, another question worth considering
would be “ h ow are the FV presented – whole vs cut-up vs other
creative ways? Does acceptability vary by what is offered, how,
and where for different age, gender, and ethnic groups? ”
3

6. On the last page of Exhibit A-2, what about examining changes in
NSLP based on consumption of F&V after participation in FFVP?
From dietary recalls, can be easily analyzed.
General Comments on the Instruments:
1. Given the length of the school administrator surveys, all
instruments should be reviewed with regard to their utility in
addressing specific research questions. Some suggestions on
simplifying and cutting back the survey are included under
specific surveys.
th
th
2. The self administered survey and the food diary for 4 – 6
graders included in the package seem quite advanced and beyond
th
th
the reading and comprehension levels of many 4 and 5 grade
students. This will make data erroneous for large portions of
the respondents. Suggestions for alternate measures are provided
under specific surveys.
3. Given the current debate over the role of healthy fruit and
vegetable consumption vs the role of energy dense food
consumption to maintain energy balance, and the fact that the
legislation in place specifically requires that the program be
evaluated with regard to its effect on consumption of other
foods, consider adding questions on energy dense foods in
children’s survey.
4. Respondents for SFA’s and School Food Managers may be the same
individual in many small to medium school districts.
Specific Comments:
State Child Nutrition Agency
1. Given that the elementary schools can be different combinations
of grades (k-4, k-6, k-8 etc), it would be good to know the grade
levels in schools selected in the different states. Schools may
choose different implementation strategies based on the age of
children. This info can be obtained here, from the principal or
SFA.
2. Question 5, 7 – need to define what does satisfactory school
wellness policy mean – is it the presence of a policy? Level of
implementation? Or some type of scoring?
3. Check options for Question 9 – one date and month option for 0910 but open date and month for 10-11?
4. In question 12 give examples for options like Implementation
plans, nutrition education (may want to include things like
number of hours and frequency, partnerships)
5. Question 16 might consider adding promotional materials and
education materials as options

SFA Survey
2. In large school districts, SFA would have to consult with
individual schools to answer some of these questions – it might
be helpful to acknowledge that upfront. It also makes the time
for administration longer than what is specified in the OMB
package.
4

3. This is a large module and the response burden would be high.
Also, for a number of questions, it is unlikely that the SFA
would have the level of detail for individual schools that is
being asked (see comments below). Might consider adding a don’t
know option.
4. F2 – as it reads now, you will not know if any changes took place
in the prior years especially for schools that have had the
program for a few years?
5. For Q M1 – please check if the schools are used to reporting
average number of meals served per day or total number of meals
per month – it would make it easier for them to report along the
same lines for this survey.
It is our experience that these
individuals report average meals per day. If so, then the burden
of calculating the total meals falls to the respondent— w hich is
contrary to the principles behind paperwork reduction, and also
will greatly increase the probability of an erroneous answer!
For an on-line survey it should be exceedingly easy to ask the
respondent how they usually report this information – then
present a skip out to the format that they generally employ —
daily average or monthly total. Knowing the number of school
days in the month, let the computer calculate the total, for
those individuals that report a daily average. These and similar
issues are so important to the accuracy and completeness of
survey responses – it is very surprising that Abt did not address
this given the size of the firm and their assumed experience. We
realize they cannot pilot test the instruments, but really, given
the experience to date in surveying school administrators, this
is worrisome.
6. Questions M2, M2b, M2d, M2e – If the SFA is reporting for all
schools in the FFVP, the changes are likely to vary from school
to school - for example changes in 3-8grade schools may be
quite different from changes in k-3 schools etc. Asking for each
school may be quite cumbersome - but you will not get useful
information by lumping all the schools together. Again, a skip
out pattern could be used — specify each of the schools in the
sample, then query the SFA as to whether changes are similar for
next school in the list. If so, they can skip out to the next
named school — if not, they can fill in the necessary information.
This reduces response burden in a way that is consistent with web
survey, but minimizes useless error.
7. It would be good to cut down on the number of items asked in q
M2e – not sure how useful is the bread stick category, I would
also consider consolidating all types of cookies and frozen
desserts – the low-fat options are still loaded with sugar.
8. Q M2e – separate soda pop and fruit drinks as categories
9. M3 – SFA may not know of all the USDA programs the school has
participated in – for example, Extension staff often make contact
with the school principals to set up classes for SNAP ed or EFNEP
and the SFA is often not aware of it. It may be better to limit
to types of USDA activities that the SFA is promoting in these
schools.
10.
M5 – if the school is getting the snack from the parents,
food bank or a local store donation, the SFA may not be aware of
it. Given the limited staffing in most SFA’s, I would be
surprised if they can accurately give you details on freq and
timing for snacks that are not coming through them. It would be
helpful if you split this question and ask the details only for
those snacks that are provided through SFA and just ask about the
5

SFA’s awareness of other types of snacks that may be offered to
children.
11.
S2 – this is a time consuming question and I am not sure
about the value of asking this question, is it addressing a
specific research question? Asking for so much information might
jeopardize response rate or provide unreliable data.
Instead
for implementation why not just ask – i. distribution method (by
grade level), ii. times of day when offered, and iii. common FV
offered.
12.
S4 – not sure if Very Poor quality should be worded
differently otherwise you may not get many people checking that
category for the SFA survey.
School Foodservice Manger
1. In some schools the person filling out the SFA form may be the
same as the one filling this one. This is of concern for
response burden.
2. Page 9 – may want to replace the word serving with portion – to
avoid confusion with USDA servings.
3. P 11 – consider adding “ I think students eat less (or more)
vegetables at lunch time since FVPP started ” and “ I think
students eat less (or more) fruit since the FVPP started ” .
School Principal
1. Principals may need to consult with other staff to answer these
accurately – may be good to say that upfront so they are
prepared. OR give don’t know as an option.
2. Do you want to know if the district / school wellness policy is
in place and being implemented? Either here or in SFA or both?
3. Would they count PSA’s and interactive displays as nutrition
education activities?
4. For N1 – have to say nutrition education or promotion activities
occurred at SCHOOL during the week for at least some classes…
5. Move the statement “ If you do not have access to this
information check here to before the table.
6. Ref period in N1a different from reference period in N4.
7. Questions like N2c, may need a “ don’t know ” option. Other
options of interest may be Choose healthy beverages, choose
healthy snacks
8. Q N2e – Add Students, Volunteers as options
9. Q N3a, N4a – add options as for N2c
10.
Q N6 Change first column heading to Type of occasion /
venue. Add another column in the table for “ N o food offered at
this venue/occasion ”
11.
Q C2 – clarify the difference between school food service
run and school run operations. Suggest following wording:
Compared to the 2007-2008 school year, would you say your school
now serves more, less, or about the same amount of the following
types of foods in school-operated venues – those that are not run
by school food service?
12.
Q C2 – separate soda pop and fruit drink categories; for
skim and 1% milk address if it includes flavored milk
13.
QC2 – the distinction between the first and last column
headings is not clear
14.
QC2 – recommend consolidating all types of cookies, chips,
and ice-creams – low fat versions can still be high in calories
and sugar
15.
QF2 – may consider adding Farm to school as a partnership?
6

16.
O – consider adding “ I think students eat less (or more)
vegetables at lunch time since FVPP started ” and “ I think
students eat less (or more) fruit since the FVPP started ” .
Teacher survey
1. consider adding “ I think students eat less (or more) vegetables
at lunch time since FVPP started ” and “ I think students eat
less (or more) fruit since the FVPP started ” .
Food Record
1. This record would be helpful in obtaining 24 hour recall from
young children, however, the format; the description guide; and
the volumetric and size assessment visuals seem very advanced for
th
th
4 and 5 graders level of comprehension and reading abilities.
th
th
Use of fractions and decimals will also be beyond many 4 and 5
graders.
2. The researchers may want to look at the methodology used for SNDA
III studies http://www.fns.usda.gov/ora/MENU/Published/CNP/FILES/SNDAIIIInstruments.pdf
Self Administered Student Questionnaire
1. Many of the questions in this instrument seem much more complexth
and advanced than the comprehension and reading level of many 4
th
and 5 graders. These include Hispanic and Race questions, NCI
Fruit and Vegetable screener; complex format for questions 12,
13d (skip patterns).
2. Will these questions ever be read to the children – reading
comprehension in some schools may be a challenge.
th
3. Recommend looking at the SPAN survey validated for 4 graders and
also includes questions on energy dense foods
http://www.sph.uth.tmc.edu/catch/catch_em/4th%20SPAN%20Eng%20v8.p
df
4. Q 13 b – would be interesting to add “ I do not like the fruits
and vegetables that are offered ” and “ I do not like how the
school offers fruits and vegetables, for example, are they cut
up, whole, or in a bag, etc”
5. A four point scale may be more than children can discern on – a
three point scale may be more appropriate.
6. This instrument needs questions that will assess dietary changes
related to consumption and preferences for less nutritious,
energy dense foods. See questions in SPAN survey above.
School Food Environment Assessment
1. Section A, Q 1 – Instructions need to include the possibility
that the students may already in the classroom and FV may arrive
there.
2. Q 13 need to define the three options for staff attire – is it
cleanliness or creativity or both
3. Q 15, 16, 18 – for good inter-rater reliability define the
categories clearly – what is meant by most, some, little?
4. Section B – similar comments as in Section A.
5. For vegetables served at school lunch will fresh include – salad,
cut up, steamed, stir fried? etc.
6. Page 9 – Q 17 typo – replace fruit with vegetable.

7

7. Page 10 – Q D1. For ease of data entry draw a line from column
location to column number
8. Page 10 Q2 Juice (50%) should be listed as Juice Drink . Not
sure of the Water or sparkling water with juice category – isn’t
that same as 50% juice drink or is that something schools make
and sell?
9. Consolidate low fat and regular cakes etc.
Parent
1. Q 6 add option “ d id not apply ”
2. Q8 – will not allow making a distinction between USDA lunches and
a-la-carte or other competitive source lunches.

8

NASS Comments
OMB Docket for the Food and Nutrition Service: Evaluation of the Fresh Fruit and Vegetables Program
The OMB package for the Food and Nutrition Service’s evaluation of the Fresh Fruit and Vegetable
Program (FFVP) was prepared by Abt Associates, who designed and will also administer the multiple
surveys comprising the complex evaluation. The package is comprehensive, and generally wellpresented, but there is a notable omission: no questionnaire was included for any of the component
surveys; appendices C through I are blank. Although Appendix A contains a helpful overview of the
planned data collection, outlining outcome measures and planned analyses (Exhibit A-2), we could not
specifically review question sets for the Survey of State Child Nutrition Agencies (C), the Survey of
School Food Authorities (D), the Survey of School Principals (E), the School Food Environment
Assessment (F), the School Food Service Manager Interview (G), the Teacher Survey (H), or the Student
Self-Administered Questionnaire (I). Within the text of the document, item A.8 understandably contains
blanks (since the Federal Register’s announcement of the impending evaluation had not appeared at the
time the version of the docket sent to us was completed).
The surveys constituting the FFVP evaluation fall into two groups: those targeting the impact of the
program on the participating schools and their students (impact study), and those focusing on the
implementation of the program (implementation study). The main feature of the impact study is a
survey based on a regression discontinuity design covering elementary schools in 16 states (with selection
of thirteen states by region--two from the Northeast, three from the Midwest, six from the South, and two
from the West-- based on PPS sampling where the measure of size is the number of elementary school
students attending schools where at least 50 percent of the students participate in the National Free or
Reduced Lunch Program (FRLP), and including California, Florida, and Texas as certainty states).
Within the selected states, sample elementary schools are selected by a PPS scheme, some slightly above
their state’s cutoff for the Fresh Fruit and Vegetable Program, hence eligible and participating in the
program, and some slightly below their state’s cut-off , hence not participating in the program. FFVP
eligibility criteria vary by state—only elementary schools may participate, and those schools with the
highest percentages of low income students for their states have the highest priority for inclusion. The
total school sample for all 16 states was designed to contain 128 responding FFVP-participating schools,
scoring above the state-specific FFVP cut-offs, and 128 responding schools falling slightly below them
(and not participating). Within the selected schools, one classroom will be randomly selected from the
fourth, one from the fifth, and one from the sixth grades represented in the school, and within each of the
three classrooms selected, ten students will be drawn into a stratified cluster sample, along with their
teacher. An 80% response rate is posited, yielding a 24-student sample per school. These students will be
trained to complete a 24-hour food diary for one specific day, with the assistance of a caregiver; they will
also be interviewed by a survey specialist on their food consumption and nutritional attitudes, on the diary
due-date, the day after the diary date. Because FFVP-eligibility scores of the surveyed schools, whether
they are program participants or not, are similar within their state, all falling close to the state cut-off ,
program participation can be considered random within this population, regression equations can be run,
and outcomes for the FFVP-participating students may be attributed to their program participation, once
allowance is made for demographic and “school-environmental” covariates in the regression equation.
SAS Proc SurveyReg will be used to account for the sample design in the development of regression
equations, but the possible regression models have not been specifically described (possibilities are

sketched out in Exhibit A-2) Carrying out the impact survey plan and obtaining valid data requires
official input on state FFVP cutoffs, on schools applying for the program, with school demographics,
school environmental characteristics, and precise school scores obtained on the poverty characteristic
used to test for eligibility within the sample state. Abt plans to obtain these data from the state Child
Nutrition Agencies (CNAs) for each state included in the surveys. Through an additional web survey,
Abt plans to obtain further data on the FFVP from School Food Authorities (those entities legally
responsible for administering the FFVP and other federal school programs, at school district level) to have
their assessment of the FFVP, details of the program administration, foods offered, any FFVP-related
changes in the School Breakfast Program or the National School Lunch Program. Additional validating
data on the FFVP food items served, their scheduling and venues, will be obtained through a short
interview with Food Service Managers at the selected schools. There are also visits by trained observers
checking the physical environment of the FFVP schools, the set-up and conditions for distribution of the
FFVP fruits and vegetables, and the presentation of nutritional information in the schools, completing the
School Food Environment Assessment cited in Attachment F. The teachers of the students in sample
contribute their own data through a short, self-administered survey, distributed with the student diary
forms, and the principals of the sampled schools are asked to complete a web survey on their school’s
FFVP.
Note that the three main subsidiary surveys--of CNAs, School Food Authorities, and Principals-contributing to the impact study also supply data for the implementation study, intended to provide
national estimates of program implementation procedures by FFVP-participating schools. The regression
study, with its small sample of schools all selected close to the state cut-off scores for program
participation, cannot be generalized to the whole set of FFVP schools. For the implementation study, in
addition to the 128 FFVP-participating schools included in the regression sample, an additional 560
participating schools will be included, with the goal of providing at least 448 additional FFVPparticipating (and responding) schools (yielding a total of 576 FFVP schools), assuming an 80%
response rate at school level. Sampling details for the additional 560 schools are not given (the
documentation does state that, for generalizability, all FFVP schools in the continental U.S. will have a
positive selection probability). It is clear from the description and Appendix A that sampling stops at
school level for this survey. Analysis to be performed for the implementation study is not described in
detail; estimates are to be descriptive in nature, “consisting primarily of proportions.” According to
Appendix A, the school-level data on FFVP implementation will be obtained from the three subsidiary
surveys feeding the regression study, which will be extended to include the additional 560 FFVP schools
selected and all 54 state Child Nutrition Agencies. The surveys involved are all internet surveys, and
web-based surveys are known to have particular unit nonresponse issues; whole unit response rates for
these three (including the state CNA survey) may easily sink below 80%, according to Don Dillman.
Item nonresponse is also highly probable, and will have to be dealt with.
From B.3 in the packet, it is clear that considerable thought has been given to maximizing response and
gaining student and school support for the impact study. The importance of gaining the support of the
state agencies is acknowledged. Certain measures have been taken toward these goals: schools and
students will receive modest incentives for their participation; a study liaison will be designated to visit
the classrooms and deliver study packets, and reminder letters will be provided to be sent home to
caregivers whose child’s food diary isn’t turned in on the due date. It should also be noted that some
preliminary testing has been carried out: student/parent/teacher/food service manager parts of the impact

survey were pretested by an Abt associate in two California elementary schools, in a small test involving
nine students, their parents, two teachers and a food service manager (the method of selection is not
stated). In a follow-up session, some difficulties were noted, suggestions were made for improving these
instruments, and certain questions were revised. However, the California pretest may not be conclusive:
it is no easy task for ten to twelve-year-olds in schools with high poverty rates to provide reliable,
informative survey data, even with the assistance of caregivers—and it is not clear that they will be able
to do so. In any case, more information is needed. The questionnaires should be included in the packet,
because question sequence and skip patterns for the surveys influence response patterns. Obtaining
complete, valid data from CNAs and School Food Authorities is particularly crucial to the success of the
project.

Evaluation of the Food and Nutrition Service’s Fresh Fruit and Vegetable Program: Additional Materials
Additional documentation for the Evaluation of the Food and Nutrition Service’s Fresh Fruit and
Vegetables Program (FFVP) by Abt Associates includes seven questionnaires from the surveys associated
with the program, and the text of the food environment assessment carried out separately by trained
observers, as well as revised versions of Parts A and B of the docket, with further details on the pretesting
phase of the program. The children’s survey and food diary, the survey of parents, and the teachers’
survey seem unproblematic. The Children’s Food Diary, illustrated and provided with measuring tools,
appears to be easy for the preteens to interpret and complete, mitigating some concerns about nonresponse
from elementary school students participating in the survey. The teachers’ and parents’ surveys also seem
readable and easy to follow. One suggestion: since the evaluation includes schools with fairly high
proportions of low income students in Pacific coast and Southwestern states, it could be very helpful--and
beneficial for the Fresh Fruit and Vegetables Program--to provide a Spanish translation of the parents’
survey (costs may be prohibitive). The School Principal’s Survey, web-based, with different branches
for principals of FFVP-participating schools and those whose schools are not in the program, appears
well-designed and should be an effective instrument for data gathering.
The following are our suggestions for the State Child Nutrition Agency survey, the School Food
Authorities’ (SFA) survey, and the survey of food service managers. The web-based survey of State
Child Nutrition Agencies requires an intensive data-gathering effort from respondents, which could result
in considerable item nonresponse. In Section D, collecting and reporting of various expense items from
FFVP Schools, a URL needs to be provided for the agency’s claim form and instructions. In Section F,
the detailed listing of FFVP expenses for the period from July 1, 2009 through June 30, 2010, should not
be visible to respondents before the fall of 2010. Could it be cited in the State Child Nutrition Agency
survey and transmitted separately at a later date? In Section C, “Non-Federal Partnerships,” question 17.b
needs a link, “name” probably, as in the school principals’ questionnaire, to tie the partnerships from 17.a
to one of the columns of functions listed for them in 17.b. Since in the tabular presentation of 17.b, there
are only four partner columns, the last sentence of the text of 17.b should read, “An additional sheet is
provided…if you have more than 4 major partners” [Not “5”, a carry-over from the school principals’
questionnaire where the corresponding item had five columns ]. For clarity, the 17.b partnership types
could be qualified as “major type of partner” in the opening sentence as well. It might be simpler for
respondents to have the “four additional partners” item follow 17.b directly, instead of placing it at the

end of the questionnaire (this may be an automated skip pattern already built into the web survey).
Respondents of the School Food Authorities’ survey (also web-based) have an intensive data-gathering
task as well—requiring SFA director co-operation, school district-level data, and finally, school-level data
for one FFVP-participating school, and one non-FFVP school in the SFA’s district. To cite an example
from the FFVP school-related section, it may not be realistic to ask these respondents to attempt to gauge
the popularity of each fruit or vegetable item served during the elementary school reference week (a task
better suited to the food service managers, who are surveyed separately by personal interview, or to
teachers, who are in direct contact with their students on a daily basis). Finally, in surveying the food
service managers, interviewers should avoid survey terminology and use common English for effective
communication with their target respondents. Thus for question 1, I would suggest, “For what day did the
students list their school lunch in their food diaries?” (with no mention of “survey reference” days).

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue
Goals of the Project 

Item

Reviewer
Sonia Kim 
(CDC)

Sonia Kim 
(CDC)

REVISION
Comments
FNS 
Abt 
Outside Reviewer Comments
There are currently 2 stated goals of the project: assess impact and implementation. Suggestion: Yes ‐ May require  Basic data will be collected to address this 
add a 3rd goal: determine the reach or coverage of the FFVP. Rationale: 1) The Background
an additional data  objective. See below. See also Memo 10/21/09
section implies that lower-income students are a priority for the program; therefore it is important to collection survey
know how well this target population is actually being reached. 2) This information is especially
relevant considering the large increases in funding that will occur by 2012 and that the number of
students served by the FFVP will most likely increase as well. 3) Additionally, states will most
likely be interested in this information.
Data on eligible schools, how many apply, and 
Method for adding a 3rd goal: Using the State Child Nutrition Agency Survey, collect the following  Yes‐‐Will need 
information: (some or all of this information is already asked)                                                                      information on all  how many are funded will be collected from all 
FFVP schools to 
54 states to provide national totals without 
• Total number of eligible schools
find out frp%, 
sampling error.  Number of days per week will 
• Of the eligible schools how many apply
number of days per  be collected from the nationally representative 
• Of the eligible schools that apply, how many receive funding
sample of SFAs.  Cost data will permit 
week operating, 
‐Since the Implementation sample will be nationally representative, would these numbers be 
calculation of cost per child per serving day.  
intensity, and 
nationally representative as well?
average cost per  Number of servings is not tracked and would 
‐The states will be interested in their own data and would find it useful to do state‐by‐state 
not be feasible to collect.  
serving. 

comparisons also. Thus, could this data also be collected from all 54 state agencies?
Samples

Sonia Kim 
(CDC)

Will the implementation and impact data be representative at the state level for the 16 State
Agencies and/or 54 State Agencies?

Sonia Kim 
(CDC)

It would be useful to be able to link this FFVP data collected to CDC’s youth behavior data, such as
YRBSS and School Health Profiles that are collected by the Division of Adolescent and School
Health (DASH) (http://www.cdc.gov/healthyYouth/profiles/);
(http://www.cdc.gov/HealthyYouth/yrbs/index.htm)

Sonia Kim  For the Impact study: I understood that eligible schools that participated in FFVP will be compared
will other eligible schools that did participate. Is this correct?
(CDC)
Recompeting of schools will pose a major problem.
Joe 
Yes
Thompson 
(RWJF)

Abt responses to FNS Review OMB VERSION.xls

State survey data will be collected from all 54 
State agencies.  Implementation data will be 
collected from a nationally representative 
sample of X SFAs.
The food frequency questions on the self‐
administered student questionnaire are 
adapted from the frequency instrument from 
the YRBS, and may be used to compare 
descriptive information about reported 
frequency of fruit and vegetable consumption 
in our sampled students to YRBS summary 
statistics. Similarly, questions about foods for 
sale in schools in the SFA and school principal 
surveys used food categories adapted from the 
categories used in the School Health Profiles, 
and may be compared similarly in some cases.
Clarified in OMB package
The study will be representative of schools 
selected for and participating in SY 2009‐2010.  
Prior participation will be identified.

Page 1

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

Impact Data

Comparability of the 
Proposed Instruments

Problems of response 
rate, missing data, and 
age inappropriate 
questions

REVISION
Item
Reviewer
Comments
"State Cutoff"  Sonia Kim  The definition of this phrase is not clear. On pg. 17 “The FFVP legislation and FNS regulations
require States to give FFVP funding to the poorest schools, as measured by the percent of
pg. 17
(CDC)
students eligible for free and reduced price school lunches. RD estimates the causal impact of the
FFVP by comparing schools directly above and below the cut-off for funding.”
"State Cutoff" Sonia Kim 
(CDC)
"State Cutoff" Sonia Kim 
(CDC)

This implies that the “State Cutoff” is a demarcation for eligibility. Thus comparing schools on either
side of the cutoff implies that one group is eligible, while the other is not.
Does FNS have an eligibility cut-off? Does the State Cutoff refer to a specific criteria set by the
state? I.e. by FNS standards the schools could be considered eligible, but by the state’s standards
they are not eligible? OR because there is a narrow free/reduced price window, comparing above
and below the cutoff results in the comparison of very similar schools (even though technically one
group is eligible and one is not)? This distinction should be made more clear.

"State Cutoff"  Sonia Kim 
pg. 19
(CDC)

The definition of “State Cutoff” should be made clearer in the diagram on pg. 19.

Sonia Kim 
Nutritional 
(CDC)
Status pg. 4
Willingness to  Sonia Kim 
try new fruits  (CDC)
and 
vegetables pg. 
8
Increased fruit  Sonia Kim 
and vegetable  (CDC)
consumption 
pg. 8

On pg. 4, it states that the impact study will look at children’s nutritional status. Is this information
being collected?
This is an important part of attitudes. There is one question about it (pg. 8, q 16 of the selfadministered survey). Other questions could be added for more depth on this issue. (Alice
Ammerman of UNC has done work on this topic.)

FNS 

Abt 
Clarified in OMB package

Clarified in OMB package
Clarified in OMB package

Clarified in OMB package

No

No. wording changed.

Yes (agree 
w/importance)

Additional questions have been added to the 
student questionnaire on willingness to try, and 
preferences for particular fruits and vegetables

Pg. 8 states that the information will be used to determine whether “…the FFVP increased fruit and
vegetable consumption…”

Text changed to suggested text.

The need to assure comparability of measures with existing high quality surveys, specifically SNDA III and
the Bridging the Gap surveys of school policies and implementation. Both these groups have studied food
access, availability and consumption issues affecting school children. Also consult the NCI Measures of
Food Environment website (https://riskfactor.cancer.gov/mfe) - a compilation of studies investigating
community-level measures of the food environment, including school food environment.

SNDA III and earlier FNS studies were reviewed 
in instrument development.  School 
environment instrument has been used in 
previous CWH studies.

We urge them (Abt) to employ questions that are comparable to the Bridging the Gap survey. This is an
Laura 
Leviton and  annual survey of a representative sample of 500 to 700 school districts and schools (elementary, middle and
Punam Ohri‐ secondary).
Vachaspati 
(RWJF)

FFVP schools are by definition atypical; 
comparisons to the general population of 
schools are not a stated priority for the 
evaluation.  Relevant comparisons are between 
FFVP schools and eligible non‐participating 
schools.  We did not have time to review the 
BtG instruments and still meet our schedule.

The instruments for administrators are unnecessarily awkward to use, impose a large response burden where Yes (agree)
Laura 
Leviton and  it is not necessary, and will therefore impair both response rate and accuracy. It is a fundamental principle of
Punam Ohri‐ survey research that increased response burden will increase error and missing data.
Vachaspati 
(RWJF)

Instruments have been simplified to reduce 
burden. This issue will be revisited after the 
pretest.

Laura 
Leviton and 
Punam Ohri‐
Vachaspati 
(RWJF)

Abt responses to FNS Review OMB VERSION.xls

Page 2

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer
Laura 
Leviton and 
Punam Ohri‐
Vachaspati 
(RWJF)

Comments
Response rate for administrators will be a serious problem even if the surveys are made more user friendly.
The incentives described will not be sufficient to guarantee the response rate that Abt is targeting, based on
our recent experience using the web to collect data from school personnel. There was no description in the
OMB package of how Abt proposes to ensure the response rate they need. In any revision, it will be
important for USDA to know in detail, how Abt plans to follow up with administrators and food service
personnel to get them to respond.
The instruments for children are not age appropriate and there are better instruments available for both the
Laura 
Leviton and  family surveys and 24 hour recalls. Fourth and fifth graders will not be able to respond to some of these
Punam Ohri‐ issues in the formats provided.
Vachaspati 
(RWJF)

FNS 

FNS will obtain 
instruments 

Abt 
Incentives will be reassessed after the pretest.

Memo prepared addressing this concern.  
Methods used successfully for many years. Pilot 
showed reasonable quality data from children 
grades 4‐6 and high response rate 10/11.

Greater analytic
Background 
attention, and data
Section
collection where
possible, to consumption
of less nutritious foods

Laura 
Leviton and 
Punam Ohri‐
Vachaspati 
(RWJF)

There is no direct evidence that increasing F&V consumption in children will cause decreased consumption
of calorie dense foods of limited nutritional value. Yet increasingly there is evidence (e.g. Gortmaker and
Wang, Sturm) that we will only prevent childhood obesity by decreasing the consumption of calorie dense
food of limited nutritional value. Yet the data collection and analytic plan do not pay sufficient attention to
this issue. The 24 recalls will yield some of the information, but the self-administered student instrument
should address this in depth—see Exhibit A-2, 7th page. Regardless of decisions about the self-administered
instrument, analytic questions about this issue, as seen on 7th page of Exhibit A-2, should take higher
priority! Unlike data collection, adding another analysis costs very little, and could tell us so much.

Memo addressed this issue. No valid 
questionnaire for this age group but selected 
questions on FQ snack foods and beverages 
from BSQ‐ 24 hr data will give better estimates 
for the group on intake of these foods than any 
FFQ

General Comments

pg. 1 & 2

Laura 
Leviton and 
Punam Ohri‐
Vachaspati 
(RWJF)

The statement of aims for the program is very clear on page 2 of the OMB Clearance Package. However, the
introductory statement under Background, page 1, is not. The program is about so much more than teaching
healthier eating habits, and in fact this statement is misleading. We would urge you to take another look at
the expanded statement on page 2 to restate the first sentence under Background, page 1.

Overall goals and objectives clarified

pg. 3

On page 3, first full paragraph, it is important to estimate the number of children served by the program, and
Laura 
Leviton and  if possible, the numbers at each stage of program expansion (reach/coverage issue- within schools).
Punam Ohri‐
Vachaspati 
(RWJF)

pg. 6

On page 6, in smaller districts the school food authorities may be the same individuals as the school food
Laura 
Leviton and  managers. How will this be addressed? Remember response burden!
Punam Ohri‐
Vachaspati 
(RWJF)

Instruments are designed with minimal overlap 
between SFA and school food manager 
questions.  School food manager instrument 
will only be used in impact sample schools; data
collectors will be instructed to skip overlapping 
questions that have been answered as part of 
the SFA questionnaire.

pg. 7

On page 7, if there is enough variation in nutrition education then analysis examining the dose of nutrition
Laura 
Leviton and  education on outcome variables. This could be addressed on Exhibit A-2, second to last page.
Punam Ohri‐
Vachaspati 
(RWJF)

See memorandum on dose response analysis.

Abt responses to FNS Review OMB VERSION.xls

Yes

See above (row 8)

Page 3

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Appendix A. Overview of  Exhibit A‐2 
Data Collection and 
Topic Area 2
Analysis

Exhibit A‐2 
(last page)

Reviewer
Lorelei 
DiSogra 
(UFPA)
Laura 
Leviton and 
Punam Ohri‐
Vachaspati 
(RWJF)

Comments
Dose response is critical; how are you going to deal with schools who offer fruits and vegetables 
infrequently?  Also, want to capture doses given in the past and current. 

Yes

In Exhibit A-2 Topic area 2, another question worth considering would be how are the FV presented – whole Yes
vs cut-up vs other creative ways? Does acceptability vary by what is offered, how, and where for different
age, gender, and ethnic groups?

On the last page of Exhibit A-2, what about examining changes in NSLP based on consumption of F&V after
Laura 
Leviton and  participation in FFVP? From dietary recalls, can be easily analyzed.
Punam Ohri‐
Vachaspati 
(RWJF)

DRAFT OMB PACKAGE
DATA COLLECTION INSTRUMENTS
General Comments

FNS 

Outside Reviewer Comments
Given the length of the school administrator surveys, all instruments should be reviewed with regard to their
Laura 
Leviton and  utility in addressing specific research questions. Some suggestions on simplifying and cutting back the
Punam Ohri‐ survey are included under specific surveys.
Vachaspati 
(RWJF)

Abt 
See memorandum on dose response analysis.

Form in which FFV are served captured by 
Environment Assessment observations; analysis 
will relate this to participation and satisfaction 
measures as appropriate. However, note that 
inference from this analysis will necessarily be 
correlational, not causal; schools that spend 
significant time and effort on FV presentation 
may differ in unobservable ways from schools 
that do not.

Yes‐ need to make 
sure that student 
participation in 
NSLP and SBP on 
the same day as 
FFVP

Will have information on student receipt of 
NSLP lunch. Will examine differences in counts 
of NSLP lunches between FFVP and non‐FFVP 
schools. Can consider further exploration if 
large differences in counts.

Yes

Instruments have been simplified to reduce 
burden. This issue will be revisited after the 
pretest.

Laura 
Leviton and 
Punam Ohri‐
Vachaspati 
(RWJF)

The self administered survey and the food diary for 4 th – 6th graders included in the package seem quite
advanced and beyond the reading and comprehension levels of many 4 th and 5 th grade students. This will
make data erroneous for large portions of the respondents. Suggestions for alternate measures are provided
under specific surveys.

See memo addressing this issue. Methods have 
been successfully used in long term large scale 
NIH studies.

Laura 
Leviton and 
Punam Ohri‐
Vachaspati 
(RWJF)

Given the current debate over the role of healthy fruit and vegetable consumption vs the role of energy dense
food consumption to maintain energy balance, and the fact that the legislation in place specifically requires
that the program be evaluated with regard to its effect on consumption of other foods, consider adding
questions on energy dense foods in children’s survey.

Analysis of 24‐hour recall will provide 
information on consumption of energy dense 
foods. Questions on frequency of consumption 
of energy dense foods added to student 
questionnaire to improve estimates of usual 
intake

Abt responses to FNS Review OMB VERSION.xls

Page 4

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer
Comments
Respondents for SFA’s and School Food Managers may be the same individual in many small to medium
Laura 
Leviton and  school districts.
Punam Ohri‐
Vachaspati 
(RWJF)

FNS 
Yes

Lorelei 
DiSogra 
(UFPA)

Important to obtain monthly FFVP menus‐‐this will provide information on variety of fruits and vegetables 
being offered.  Important to know if they are having problems with offering variety since some may have a 
negative perspective on vegetables. 

Lorelei 
DiSogra 
(UFPA)

Want to capture who the "champion" is for the program, who is driving it? This is important because it varies  Yes
from school to school and can impact implementation.  
Is there a "champion", and why was that person chosen?  As an open‐ended question.  

Lorelei 
DiSogra 
(UFPA)
Guenther, 
Patricia 
(CNPP)
Sonia Kim 
(CDC/DNPA
O)

Yes

Cycle of menus will vary. We will obtain 3 
months of detail on food purchases from FFVP 
school in the impact sample.  All sample SFAs 
will be asked both what they offered in the 
reference week and what other F/V they 
offered during the year.
Question added to State survey.

Yes (Can list in 
Re: "why" ‐ Not feasible to collect consistent 
report as opposed  and usable responses with a self‐administered 
to analyzing)
web survey.  Interviews would be needed; not 
in scope.

Principals and Superintendents know all about the benefits the FFVP has on the school environment and 
students. There is no survey for the Superintendent which is usually one of the champions or driving forces 
for the program. 
recommendations for fruit and vegetable intake are no longer expressed in terms of “servings”, but rather in Yes
“cups.”

Not enough information on Waste: How much of the purchased food is being thrown away by the food
service staff? This question should be asked at least of the School Foodservice Manager and School Food
Authority Director. If possible, perhaps some observations of students could be added. A question could be
added to the Self Administered Student Questionnaire (see detailed comments).
Sonia Kim  Not enough information on Staff burden: In a lot of schools, foodservice staff is being cut and have more
(CDC/DNPA than their share and cannot handle the load they currently have. Information should be collected from the
School Foodservice Manager, the School Food Authority Director, and the School Principal. Related, what
O)
unmet needs are being communicated by the foodservice workers, the principal, or the district (e.g. training,
supplies, better source of fruits and vegetables?
Sonia Kim  Not enough information on Distribution process: This information is asked in the "SFA Survey" and thus I 
(CDC/DNPA think at the district level. It will be answered separately for each school, but the Principal and School 
O)
Foodservice Manager may have more accurate information. In addition, each of them should be asked about 
their satisfaction with the distribution method.

Abt 
Instruments are designed with minimal overlap 
between SFA and school food manager 
questions.  School food manager instrument 
will only be used in impact sample schools; data
collectors will be instructed to skip overlapping 
questions that have been answered as part of 
the SFA questionnaire.

Will be reported appropriately in analysis.

Questions added to SFA, principal, and school 
FS manager surveys

Abt has added several questions to the SFA 
Survey, the Principal Survey, and the school FS 
manager to assess the degree to which the 
FFVP represents a burden on school or district 
staff
SFA survey instructions clarified so R's know to 
consult with school‐level personnel if needed. 
Principal survey includes Q on adequacy of 
kitchen facilities.

PARENT SURVEY
Outside Reviewer Comments

Abt responses to FNS Review OMB VERSION.xls

Page 5

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

Q6

Q 6 add option “did not apply”
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

Left as is, but added question about 
whether child eats FFVP snacks

Q8

Q8 – will not allow making a distinction between USDA lunches and a-la-carte or other competitive
Laura 
Leviton and  source lunches.
Punam 
Ohri‐
Vachaspati 
(RWJF)

Comment does not seem applicable to 
questions

SCHOOL ENVIRONMENT ASSESSMENT
Outside Reviewer Comments
Section A, Q 1 – Instructions need to include the possibility that the students may already in the
Section A, Q1 Laura 
Leviton and  classroom and FV may arrive there.
Punam 
Ohri‐
Vachaspati 
(RWJF)

Yes

DONE

Q13

Q 13 need to define the three options for staff attire – is it cleanliness or creativity or both
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

Yes

DONE

Q15, 16, 18

Q 15, 16, 18 – for good inter-rater reliability define the categories clearly – what is meant by most,
Laura 
Leviton and  some, little?
Punam 
Ohri‐
Vachaspati 
(RWJF)

Yes

DONE

Abt responses to FNS Review OMB VERSION.xls

Page 6

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Section B

pg. 9, Q17

Reviewer

Comments

Section B – similar comments as in Section A.
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

FNS 

Yes

Abt 

DONE

For vegetables served at school lunch will fresh include – salad, cut up, steamed, stir fried? etc.
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

FRESH will be defined in trainings to 
include no processing except for cutting, 
slicing (e.g., yes to green salad, no to 
steamed)

Page 9 – Q 17 typo – replace fruit with vegetable.
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

DONE

Page 10 – Q D1. For ease of data entry draw a line from column location to column number
pg. 10, QD1 Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

DONE

pg. 10 Q2

50% juice may be available in schools so 
left as separate item

Page 10 Q2 Juice (50%) should be listed as Juice Drink . Not sure of the Water or sparkling water
Laura 
Leviton and  with juice category – isn’t that same as 50% juice drink or is that something schools make and sell?
Punam 
Ohri‐
Vachaspati 
(RWJF)

Abt responses to FNS Review OMB VERSION.xls

Page 7

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Q1

Q6

Q19

Title of 
Survey
pg. 11, D1
pg. 12, D2 

Abt 

We decided not to consolidate low fat and 
regular cakes, etc because of nutritional 
differences between them; we will 
however train data collectors to ask about 
this if they are unable to tell based on 
looking at the product

Consolidate low fat and regular cakes etc.
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)
Lorelei 
DiSogra 
(UFPA)

Expand to collect other data‐ need to go broader, it is too brief and has potential to get more 
information since a live person is there. 

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)
CDC/DASH

This can vary, need to clarify‐‐‐be more specific on what to expect and what photographs you want. Yes

Unclear what additional information is 
required; Did add some questions but must 
also be mindful of time needed to collect 
additional data
Will include in instructions to data collector

Staff needs to be specified i.e.‐ teacher, principals, etc..

Yes

DONE

Get monthly FFVP menus

Yes

DONE

Are you interested in type of payment system used?

No

NO, not relevant to aims to know about for 
other school foods & FFVP is free

CDC/DASH Number of what?  A la care locations? Students served? Individual vending machines?  Vending 
machine locations?
CDC/DASH "Energy and sports drinks…" It would be more meaningful to have these items listed separately.

Yes

DONE
Left as is because distinction is not relevant 
in elementary school settings.

SCHOOL FOODSERVICE MANAGER INTERVIEW
Outside Reviewer Comments
In some schools the person filling out the SFA form may be the same as the one filling this one. This
Laura 
Leviton and  is of concern for response burden.
Punam 
Ohri‐
Vachaspati 
(RWJF)

Abt responses to FNS Review OMB VERSION.xls

Instruments are designed with minimal 
overlap between SFA and school food 
manager questions.  School food manager 
instrument will only be used in impact 
sample schools; data collectors will be 
instructed to skip overlapping questions 
that have been answered as part of the SFA 
questionnaire.

Page 8

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

pg. 9

Page 9 – may want to replace the word serving with portion – to avoid confusion with USDA
Laura 
Leviton and  servings.
Punam 
Ohri‐
Vachaspati 
(RWJF)

DONE

pg. 11

P 11 – consider adding “I think students eat less (or more) vegetables at lunch time since FVPP
Laura 
Leviton and  started” and “I think students eat less (or more) fruit since the FVPP started”.
Punam 
Ohri‐
Vachaspati 
(RWJF)

DONE

Lorelei 
DiSogra 
(UFPA)
pg. 9
Lorelei 
DiSogra 
(UFPA)
pg. 9, Q2
Lorelei 
DiSogra 
(UFPA)
pg. 9, Q3 and  Lorelei 
4
DiSogra 
(UFPA)
pg. 10
Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)
Q23
Lorelei 
DiSogra 
(UFPA)
Q24
Lorelei 
DiSogra 
(UFPA)
Q24
CDC/DASH

Ask fo rmonthly FFVP menu‐ menus for all months

DONE

Add to instructions "as part of the FFVP" after "separate from school meals"

DONE

Most often delivered to classroom for students to pick up on way out to recess.  

DONE

"servings" ‐ clarify in packages, in paper bowls, etc…

DONE

I think the students benefit from the FFVP.  The Fresh F&V students receive in the FFVP may be the  FNS noted 
only fresh f&v they eat.  
comment

DONE

Other questions to consider:  Has the FFVP influenced what frut and/or vegetable you serve in 
school lunch?  Has the FFVP resulted in studenst taking/eating more fruits and/or vegetables in 
school lunch?  
Same as Q21

Yes

DONE

DONE

Same as Q22

DONE

Same as Q22

DONE

Abt responses to FNS Review OMB VERSION.xls

Page 9

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Q29

Q30

Reviewer

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

Comments

FNS 

Abt 

Add "fresh" before "vegetables"

DONE

"Foodservice staff"‐‐what about other school officials: principals, teachers, nurse…?

Left as is, because school foodservice 
manager likely to know best what school 
foodservice staff are doing; principal best 
to ask about what all school staff are doing.

TEACHER SURVEY
Outside Reviewer Comments
consider adding “I think students eat less (or more) vegetables at lunch time since FVPP started” and Yes 
Laura 
Leviton and  “I think students eat less (or more) fruit since the FVPP started”.
Punam 
Ohri‐
Vachaspati 
(RWJF)

2 questions added

FOOD DIARY
Outside Reviewer Comments
This record would be helpful in obtaining 24 hour recall from young children, however, the format; Critical
Laura 
Leviton and  the description guide; and the volumetric and size assessment visuals seem very advanced for 4th and
Punam 
5th graders level of comprehension and reading abilities. Use of fractions and decimals will also be
Ohri‐
beyond many 4th and 5th graders.
Vachaspati 
(RWJF)
The researchers may want to look at the methodology used for SNDA III studies Laura 
Leviton and  http://www.fns.usda.gov/ora/MENU/Published/CNP/FILES/SNDAIII-Instruments.pdf
Punam 
Ohri‐
Vachaspati 
(RWJF)
CDC/DASH Where to include vitamin water?
Food 
Description 
Guide‐ 
Soda/Sparkli
ng Water
SELF‐ADMINISTERED STUDENT QUESTIONNAIRE

The diary is used as a tool to raise 
awareness, and promote memory and 
accuracy in portion estimation for the 24 hr 
recall conducted on the second day. 
Children are trained in how to record, and 
do so relatively well. Probes are included 
for the parents.
snda iii used split 24 hr recall 1/2 with 
parents at home‐ requires two contacts 
with students. Not feasible within the 
design, resources of this study.

DONE

Outside Reviewer Comments

Abt responses to FNS Review OMB VERSION.xls

Page 10

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

Many of the questions in this instrument seem much more complex and advanced than the
Laura 
Leviton and  comprehension and reading level of many 4th and 5th graders. These include Hispanic and Race
Punam 
questions, NCI Fruit and Vegetable screener; complex format for questions 12, 13d (skip patterns).
Ohri‐
Vachaspati 
(RWJF)

See memo for responses to all comments‐ 
YRBS questions used  for F V, and BSQ 
quesitons added for snacks and beverages. 
To only be used to cross check the 
estimates from 24 hr recall data, not as 
another source of point estimates.

Will these questions ever be read to the children – reading comprehension in some schools may be a
Laura 
Leviton and  challenge.
Punam 
Ohri‐
Vachaspati 
(RWJF)

No will be self admin, children do well with 
it.

Recommend looking at the SPAN survey validated for 4th graders and also includes questions on
Laura 
Leviton and  energy dense foods http://www.sph.uth.tmc.edu/catch/catch_em/4th%20SPAN%20Eng%20v8.pdf
Punam 
Ohri‐
Vachaspati 
(RWJF)

Spans asks about yesterday, not FFQ. 24 hr 
data more useful for yesterdays intake of 
snacks and bevs. Have used SPANS format 
to tailor the questionnaire to younger 
children.

Q 13 b – would be interesting to add “I do not like the fruits and vegetables that are offered” and “I
Laura 
Leviton and  do not like how the school offers fruits and vegetables, for example, are they cut up, whole, or in a
bag, etc”
Punam 
Ohri‐
Vachaspati 
(RWJF)

Concepts captured in other questions.

A four point scale may be more than children can discern on – a three point scale may be more
Laura 
Leviton and  appropriate.
Punam 
Ohri‐
Vachaspati 
(RWJF)

Will reassess after pretest.

Abt responses to FNS Review OMB VERSION.xls

Page 11

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

pg.5, 13a

CDC/DNPA This questionnaire assumes the student knows about the FFVP.  Is it reasonable to assume that 
O
students will be able to distinguish among fruits and vegetables from breakfast, lunch, vending, 
stores, after‐school programs?
pg. 5, Waste CDC/DNPA Add "If you take the free fruit and vegetable snack, do you usually eat the whole snack? (Or do you  Define whole?
O
throw some of it away.)
pg. 7, Q14
CDC/DASH Consider open‐ended option for students, too…If you could change the FFVP, what change would 
you make?
SCHOOL PRINCIPAL SURVEY
Outside Reviewer Comments
Principals may need to consult with other staff to answer these accurately – may be good to say that
Laura 
Leviton and  upfront so they are prepared. OR give don’t know as an option.
Punam 
Ohri‐
Vachaspati 
(RWJF)
Do you want to know if the district / school wellness policy is in place and being implemented?
Laura 
Leviton and  Either here or in SFA or both?
Punam 
Ohri‐
Vachaspati 
(RWJF)
Would they count PSA’s and interactive displays as nutrition education activities?
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

Abt responses to FNS Review OMB VERSION.xls

Abt 

Analysis of 24‐hour recall data will be used 
to assess changes. Some questions are 
added to survey to improve usual 
estimates of energy dense food intake.

This instrument needs questions that will assess dietary changes related to consumption and
Laura 
Leviton and  preferences for less nutritious, energy dense foods. See questions in SPAN survey above.
Punam 
Ohri‐
Vachaspati 
(RWJF)

Yes

Will be discussed in motivational 
instruction session.
DONE
DONE

Abt has added specific instructions to 
principals to consult with other staff when 
necessary to answer questions, rather than 
just say "don't know".  Need for "don't 
know" as an answer option for questions 
will be reviewed after the pretest. 
SFA and principal surveys include questions 
on most important elements of school 
wellness policy:  foods offered in school 
meals, competitive foods, and nutrition 
education/promotion.

As in prior FNS nutrition ed. studies, these 
are considered indirect education.

Page 12

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

For N1 – have to say nutrition education or promotion activities occurred at SCHOOL during the
Laura 
Leviton and  week for at least some classes…
Punam 
Ohri‐
Vachaspati 
(RWJF)

We considered this comment and left the 
question as worded.

Move the statement “If you do not have access to this information check here to before the table.
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

We considered this comment and left the 
question as worded.  We want to 
encourage the principal to get the answer.

Ref period in N1a different from reference period in N4.
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

N1a combined with N1 covers 4 weeks; N4 
covers all 4 weeks together.

Questions like N2c, may need a “don’t know” option. Other options of interest may be Choose
Laura 
Leviton and  healthy beverages, choose healthy snacks
Punam 
Ohri‐
Vachaspati 
(RWJF)

List of messages expanded. We want to 
encourage the principal to get the answer.

Q N2e – Add Students, Volunteers as options
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

We considered this comment and left the 
question as worded.

Abt responses to FNS Review OMB VERSION.xls

Page 13

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

Q N3a, N4a – add options as for N2c
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

We considered this comment and left the 
question as worded.  We want to 
encourage the principal to get the answer.

Q N6 Change first column heading to Type of occasion / venue. Add another column in the table for
Laura 
Leviton and  “No food offered at this venue/occasion”
Punam 
Ohri‐
Vachaspati 
(RWJF)

Rows are defined by time, not venue, so 
header left as is. Column for "not 
applicable" covers "not applicable"

Q C2 – clarify the difference between school food service run and school run operations. Suggest
Laura 
Leviton and  following wording: Compared to the 2007-2008 school year, would you say your school now serves
more, less, or about the same amount of the following types of foods inschool-operated venues –
Punam 
those that are not run by school food service ?
Ohri‐
Vachaspati 
(RWJF)

We considered this comment and left the 
question as worded.  If there is confusion in 
the pretest, we will clarify.

Q C2 – separate soda pop and fruit drink categories; for skim and 1% milk address if it includes
Laura 
Leviton and  flavored milk
Punam 
Ohri‐
Vachaspati 
(RWJF)

Existing categories were based on a 
previously‐validated instrument, so we did 
not make this change.

QC2 – the distinction between the first and last column headings is not clear
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

Wording for colum headings has been 
changed to clarify intent.

Abt responses to FNS Review OMB VERSION.xls

Page 14

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

QC2 – recommend consolidating all types of cookies, chips, and ice-creams – low fat versions can
Laura 
Leviton and  still be high in calories and sugar
Punam 
Ohri‐
Vachaspati 
(RWJF)

Existing categories were based on a 
previously‐validated instrument, so we did 
not make this change.

QF2 – may consider adding Farm to school as a partnership?
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

Messages (N2a etc.) include "Where FFV 
come from".  Partners include farmer's 
markets, produce associations/commodity 
groups.  "Farmer's market" category 
intended to include local farmers; to be 
clarified in final version.

O – consider adding “I think students eat less (or more) vegetables at lunch time since FVPP started” Yes
Laura 
Leviton and  and “I think students eat less (or more) fruit since the FVPP started”.
Punam 
Ohri‐
Vachaspati 
(RWJF)

We ask the most knowledgeable people 
this question:  the school food service 
managers and the SFA director.

pg. 3, N2c &  CDC/DNPA Education messages.  The ones detailed in the "State Child Nutrition Agency Survey" on pg. 5, ques 
pg. 4, N3
O
13a were better and more comprehensive ("role of fresh fv in a complete diet, where fresh fv come 
from, trying new foods, variety, etc)
pg. 5, N7
CDC/DNPA "Advisory/Policy group of parents or community members…" What if the group is comprised 
O
mainly of teachers and staff?  Shouldn't this count?
pg. 1

CDC/DASH How long after the reference week will this survey be given? There is concern about accurate
recollection of the data.

N1

CDC/DASH Is there a reason there is no row for afterschool?

Abt responses to FNS Review OMB VERSION.xls

Education messages now match longer list 
of answer options from State Child 
Nutrition Agency Survey.
Teachers/staff was added to the list of 
individuals who could make up the advisory 
group for this question.
The reference week will be the last full 
school week before the week when the 
survey is completed.  This will be specified 
in the instructions.  
Intent of the question is nutrition 
education conducted during school hours 
only, which could potentially be tied to the 
FFVP in FFVP schools.

Page 15

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

N2d

CDC/DASH Will principals actually know this information being asked?

Abt has added specific instructions to 
principals to consult with other staff when 
necessary to answer questions, rather than 
just say "don't know".  Need for "don't 
know" as an answer option for questions 
will be reviewed after the pretest. 

C2

Wording for colum headings has been 
changed to clarify intent.

C2

CDC/DASH "Stopped offering this food after 2007-2008"--Would something in this column also qualify in the
“This food not offered in 2007-2008 or now column”? Should the “or now” be deleted from the first
column?
CDC/DASH "Food category"--Add energy drinks as a separate item (e.g., Red Bull)

O Q10

CDC/DASH Consider adding this item to the other surveys (parent, foodservice manager, etc)

pg. 1

Lorelei 
DiSogra 
(UFPA)

Don't think survey should start w/ tedious questions on nutrition education.  Start with overarching  Yes
questions about success. 

Abt responses to FNS Review OMB VERSION.xls

Abt felt it was unlikely that these drinks 
would be offered to elementary school 
students, and furthermore that 
respondents would not react well to being 
asked. Also, the existing categories were 
based on a previously‐validated 
instrument, so we did not make this 
change.
See above.
First section of survey is questions to be 
answered by all principals.  Objective, 
descriptive data are the primary objective 
of the survey.  Satisfaction questions have 
been left at the end of the survey, since it 
is Abt's feeling that questions about the 
program's success should best be asked 
after the principal has been prompted in 
the rest of the survey to think through 
various aspects of the FFVP and related 
programs at the school.  Opinion questions 
are easier to answer and therefore good to 
place at the end after respondent has 
answered harder questions.  Introduction 
and recruiting materials will motivate 
respondents; placing opion questions at 
the end will encourage respondents to 
"stick with it".

Page 16

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

pg. 1

Q. N6

Q. N7

Reviewer

Comments

FNS 

Abt 

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

Start survey with QO, so that it begins with a focus on FFVP, the other questions seem tedious.

See response above

I think principals will have trouble answering this level of detail about nutrition education.

Yes

Abt has added specific instructions to 
principals to consult with other staff when 
necessary to answer questions, rather than 
just say "don't know".  Need for "don't 
know" as an answer option for questions 
will be reviewed after the pretest. 

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

Too limiting.  What about whole grains, more f&v

Yes

List of messages expanded. 

Nothing about wellness policies…changes in teacher's lounge, birthday parties, holiday 
parties….impact of FFVP on wellness policies (nutrition/physical activity)

Yes

SFA and principal surveys include questions 
on most important elements of school 
wellness policy:  foods offered in school 
meals, competitive foods, and nutrition 
education/promotion.

Q. C1

Lorelei 
DiSogra 
(UFPA)

Will they know this?  Level of detail requires asking someone else.

Abt has added specific instructions to 
principals to consult with other staff when 
necessary to answer questions, rather than 
just say "don't know".  Need for "don't 
know" as an answer option for questions 
will be reviewed after the pretest. 

Q. C2

Lorelei 
DiSogra 
(UFPA)

What about a‐la‐carte lines?  2007‐2008 is too many years back.  

Q. F2

Lorelei 
DiSogra 
(UFPA)

Needs to be broader to capture all partnerships‐‐other State Agencies‐‐Ag, Health & other Fed 
programs for collaborations.

Yes

A la carte lines are part of cafeteria and are 
run by food service.  2008‐09 is too late for 
baseline because some schools will have 
had FFVP.  We need a consistent baseline. 
We will see in the pretest if recall is a 
problem.
We avoid duplication by asking about state‐
level partnerships in State survey, district‐
level partnerships in SFA survey, and school‐
level partnerships here.

Q. O5

Lorelei 
DiSogra 
(UFPA)

We should offer daily

FNS noted 
comment

Abt responses to FNS Review OMB VERSION.xls

No response required.

Page 17

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Q. O5

Reviewer

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

Comments

Add question (not legible)

FNS 

Abt 

FNS will ask 
Lorelai

Need to understand the role of principal‐‐they know about impact on kids, family, parents, schools, 
teachers. 

Principal questions are within scope of 
what we expect to be usual knowledge.

SFA SURVEY 
Outside Reviewer Comments
In large school districts, SFA would have to consult with individual schools to answer some of these
Laura 
Leviton and  questions – it might be helpful to acknowledge that upfront. It also makes the time for administration
longer than what is specified in the OMB package.
Punam 
Ohri‐
Vachaspati 
(RWJF)

Abt has added specific instructions to SFA 
survey to consult with other staff when 
necessary to answer questions, rather than 
just say "don't know".  Need for "don't 
know" as an answer option for questions 
will be reviewed after the pretest. 

This is a large module and the response burden would be high. Also, for a number of questions, it is
Laura 
Leviton and  unlikely that the SFA would have the level of detail for individual schools that is being asked (see
comments below). Might consider adding a don’t know option.
Punam 
Ohri‐
Vachaspati 
(RWJF)

Abt has worked to streamline and 
reorganize the instrument overall in order 
to address burden concerns. Need for 
"don't know" as an answer option for 
questions will be reviewed after the 
pretest. 

F2

F2 – as it reads now, you will not know if any changes took place in the prior years especially for
Laura 
Leviton and  schools that have had the program for a few years?
Punam 
Ohri‐
Vachaspati 
(RWJF)

This is true. Survey is focused on current 
year.  Asking about prior years would add 
burden.

M1

For Q M1 – please check if the schools are used to reporting average number of meals served per day
Laura 
Leviton and  or total number of meals per month – it would make it easier for them to report along the same lines
for this survey. It is our experience that these individuals report average meals per day. If so, then
Punam 
the burden of calculating the total meals falls to the respondent—which is contrary to the principles
Ohri‐
Vachaspati  behind paperwork reduction, and also will greatly increase the probability of an erroneous answer!
For an on-line survey it should be exceedingly easy to ask the respondent how they usually report this
(RWJF)
information – then present a skip out to the format that they generally employ—daily average or
monthly total. Knowing the number of school days in the month, let the computer calculate the total,
for those individuals that report a daily average. These and similar issues are so important to the
accuracy and completeness of survey responses – it is very surprising that Abt did not address this
given the size of the firm and their assumed experience. We realize they cannot pilot test the
instruments, but really, given the experience to date in surveying school administrators, this is worris

Our experience is based on reporting 
requirements.  School meal counts are 
rolled up to monthly basis for claims. 
Question is asked of SFA where claims are 
prepared. This is consistent with prior 
studies for FNS such as School Lunch and 
Breakfast Cost Study.

Abt responses to FNS Review OMB VERSION.xls

Page 18

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

M2

Questions M2, M2b, M2d, M2e – If the SFA is reporting for all schools in the FFVP, the changes are
Laura 
Leviton and  likely to vary from school to school - for example changes in 3-8grade schools may be quite
different from changes in k-3 schools etc. Asking for each school may be quite cumbersome - but
Punam 
you will not get useful information by lumping all the schools together. Again, a skip out pattern
Ohri‐
Vachaspati  could be used—specify each of the schools in the sample, then query the SFA as to whether changes
are similar for next school in the list. If so, they can skip out to the next named school—if not, they
(RWJF)
can fill in the necessary information. This reduces response burden in a way that is consistent with
web survey, but minimizes useless error.

Questions moved to school‐specific 
module.

M2e

It would be good to cut down on the number of items asked in q M2e – not sure how useful is the
Laura 
Leviton and  bread stick category, I would also consider consolidating all types of cookies and frozen desserts –
the low-fat options are still loaded with sugar.
Punam 
Ohri‐
Vachaspati 
(RWJF)

Existing categories were based on a 
previously‐validated instrument, so we did 
not make this change.

M2e

Q M2e – separate soda pop and fruit drinks as categories
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

Existing categories were based on a 
previously‐validated instrument, so we did 
not make this change.

M3

M3 – SFA may not know of all the USDA programs the school has participated in – for example,
Laura 
Leviton and  Extension staff often make contact with the school principals to set up classes for SNAP ed or
EFNEP and the SFA is often not aware of it. It may be better to limit to types of USDA activities
Punam 
that the SFA is promoting in these schools.
Ohri‐
Vachaspati 
(RWJF)

Programs specified are the ones of most 
interest. Principal survey will pick up school‐
level partnerships with EFNEP/CES.

M5

M5 – if the school is getting the snack from the parents, food bank or a local store donation, the SFA
Laura 
Leviton and  may not be aware of it. Given the limited staffing in most SFA’s, I would be surprised if they can
accurately give you details on freq and timing for snacks that are not coming through them. It would
Punam 
be helpful if you split this question and ask the details only for those snacks that are provided through
Ohri‐
Vachaspati  SFA and just ask about the SFA’s awareness of other types of snacks that may be offered to children.
(RWJF)

Question moved to school‐specific module. 
SFA encouraged to contact school if 
needed to complete this module.  Will 
revisit whether principal response needed 
after pretest.

Abt responses to FNS Review OMB VERSION.xls

Page 19

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

S2

Reviewer

Comments

FNS 

Abt 

We are trying to reconstruct the menu for 
the week and link foods to grades served. 
We have simplified so that we ask for the 
list of FFV offered by all distribution 
methods in the two time periods.  We will 
revisit this after we get information from 
the pretest on the burden.

S2 – this is a time consuming question and I am not sure about the value of asking this question, is it
Laura 
Leviton and  addressing a specific research question? Asking for so much information might jeopardize response
rate or provide unreliable data. Instead for implementation why not just ask – i. distribution
Punam 
method (by grade level), ii. times of day when offered, and iii. common FV offered.
Ohri‐
Vachaspati 
(RWJF)

We will revise if needed based on pretest 
feedback.

S4

S4 – not sure if Very Poor quality should be worded differently otherwise you may not get many
Laura 
Leviton and  people checking that category for the SFA survey.
Punam 
Ohri‐
Vachaspati 
(RWJF)

pg. 2

CDC/DNPA Add "Source of fruits and vegetables"
O

pg. 3

CDC/DNPA Separate "Local grocers and stores" from "Farmers' markets" and "other food distributors."  [This 
O
question was asked on other surveys as well, e.g. "School principals survey."]

Abt has made this change.

pg. 18

CDC/DASH Please clarify what "per class" means

Intent is to capture the average time that 
an individual student has access to FFV.  
Will clarify if needed after pretest.

What can be 
done to get at 
local sources fro 
produce…i.e. 
locally grown?

Abt responses to FNS Review OMB VERSION.xls

This question is primarily intended to give 
us a sense of quantifiable changes in FFVP 
implementation over time (e.g., more or 
less distribution methods, more or less 
nutrition education activities.) Changes in 
source could be along several dimensions; 
a separate question would be needed to 
sort this out. The RFP did not contain any 
research questions relating to sources of 
fruits and vegetables. For these reasons, 
and because of burden considerations, we 
did not add this.

Page 20

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

pg. 1

Lorelei 
DiSogra 
(UFPA)

To assess variety‐‐ask for FFVP monthly/yearly menus.  Many of these SFA staff are very involved in  Yes
implementation of FFVP, select all FF&V, teaching nutrition ed, marketing the FFVP to other 
schools, etc…

Cycle of menus will vary. We will obtain 3 
months of detail on food purchases from 
FFVP school in the impact sample.  All 
sample SFAs will be asked both what they 
offered in the reference week and what 
other F/V they offered during the year.

pg. 1, F

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

Opening paragraph doesn't make sense

Questionnaire has been revised with 
clearer instructions.

"Fruit and Vegetable distribution methods"‐‐what does this mean?  Not clear to me. 

Added examples of distribution methods 
(e.g. kiosks, classroom) to clarify this point.

F2

F3

F4a & F4b

Lorelei 
DiSogra 
(UFPA)
F5
Lorelei 
DiSogra 
(UFPA)
M2a & M2b Lorelei 
DiSogra 
(UFPA)
M2d
Lorelei 
DiSogra 
(UFPA)
M5
Lorelei 
DiSogra 
(UFPA)
S1
Lorelei 
DiSogra 
(UFPA)
S4 & S5
Lorelei 
DiSogra 
(UFPA)

This question needs to also include State government partners and other federal programs.  Not 
capturing full picture.  

Yes

change "promoted" to provided/served and add another question "…within the next few weeks."

Include "f. Our SFA has other eligible schools that would like to participate in FFVP."

We avoid duplication by asking about state‐
level partnerships in State survey, district‐
level partnerships in SFA survey, and school‐
level partnerships here.  We expect 
partnerships with federal agencies to occur 
at the state level.
Language has been changed to respond to 
this request.

No‐ this raises a 
different issue

Remove "dried" under vegetables category

Will do after pretest.

Suggest this question be clearer

Wording for colum headings has been 
changed to clarify intent.

Place "FF/V" before the word "snacks" in the opening paragraph and under "a".

We will revise if needed based on pretest 
feedback.

Need to define kiosk

Definition has been added.

Make sure to put "Fresh"  before "fruits or vegetables" in opening paragraph.  Replace 
"distributed" with "offered/provided/served"

We will revise if needed based on pretest 
feedback.

Abt responses to FNS Review OMB VERSION.xls

Page 21

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

S4

S6

S6

Reviewer

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

S6a

Lorelei 
DiSogra 
(UFPA)
O
Lorelei 
DiSogra 
(UFPA)
O Q2
Lorelei 
DiSogra 
(UFPA)
O Q8
Lorelei 
DiSogra 
(UFPA)
O
Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)
STATE CHILD NUTRITION AGENCY SURVEY

Comments

FNS 

Abt 

What ? best describes?  This ? changes in all surveys?  (not legible)

What about series of questions about/capturing was this FF/V new to students.  First time 
trying/their response.

Not identified as a research priority; not 
advisable to lengthen instrument.

Do we need this questions?

Useful to understand reasons for choices of 
foods offered; could drop to cut burden.

Change "distribution" to "Program"

"Distribution" used to link to preceding 
questions; "program" is broader and could 
be interepreted to refer to promotion 
activities etc.
Not identified as a research priorty; not 
advisable to lengthen instrument.

add an option:  Some new FF&V offered this week.  (One issue not identified is that many don't 
know about wide variety available…even apples‐‐value added, etc…

Yes

I don't think this question should be last.  

Question is no longer placed last in 
instrument.

What is the point?  Ate or participated?  

Focus is what they eat.

Need to clarify question

We will revise if needed based on pretest 
feedback.

This list needs to be expanded to capture other positive/negative opinions

Questions have been expanded.  Questions 
on application process and challenges 
added. 
Not identified as a research priorty; not 
advisable to lengthen instrument.

Other possible questions:  How involved are you in the implementation of the FFVP in schools in 
your district?  What role do they play?

Outside Reviewer Comments
Given that the elementary schools can be different combinations of grades (k-4, k-6, k-8 etc), it
Laura 
Leviton and  would be good to know the grade levels in schools selected in the different states. Schools may
choose different implementation strategies based on the age of children. This info can be obtained
Punam 
here, from the principal or SFA.
Ohri‐

Yes

Grade levels of schools in sample will be 
identified from CCD.

Vachaspati 
(RWJF)

Abt responses to FNS Review OMB VERSION.xls

Page 22

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

Q5‐7

Question 5, 7 – need to define what does satisfactory school wellness policy mean – is it the presence
Laura 
Leviton and  of a policy? Level of implementation? Or some type of scoring?
Punam 
Ohri‐
Vachaspati 
(RWJF)

State defines what is satisfactory. Asking 
for this definition would increase burden.

Q9

Check options for Question 9 – one date and month option for 09-10 but open date and month for 10Laura 
Leviton and  11?
Punam 
Ohri‐
Vachaspati 
(RWJF)

Corrected.

Q12

In question 12 give examples for options like Implementation plans, nutrition education (may want to
Laura 
Leviton and  include things like number of hours and frequency, partnerships)
Punam 
Ohri‐
Vachaspati 
(RWJF)

We will revise if needed based on pretest 
feedback.

Q16

Question 16 might consider adding promotional materials and education materials as options
Laura 
Leviton and 
Punam 
Ohri‐
Vachaspati 
(RWJF)

We will revise if needed based on pretest 
feedback.

pg. 9

CDC/DNPA Also ask if the following is being collected from FFVP schools: requests/problems/needs from the 
O
school related to FFVP (e.g. for training, supplies, change in State policies).

pg. 10

CDC/DNPA The salary for a full‐time FFVP coordinator in the State was asked, but not if there is a full‐time 
O
FFVP coordinator in the State. If not, who is in charge of administering the program, and what % of 
time does this represent?
CDC/DNPA Add training and education expenses here. 
O

pg. 11

B. 12

CDC/DASH Any difference between farm‐to‐café and farm‐to‐school?

Abt responses to FNS Review OMB VERSION.xls

Yes

Added (now Q18)

Revised per FNS comment

Do not expect states to report this 
separately; if applicable will be identified 
as component of admin expense
Both terms used (now q14)

Page 23

FNS Evaluation of the Fresh Fruit and Vegetable Program - Reviewer Comments (OVERALL)

DRAFT OMB PACKAGE
Issue

REVISION
Item

Reviewer

Comments

FNS 

Abt 

F4

CDC/DASH Are you referring to the F3 period?

Q11

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

Questions included (Q10, q11)
Add questions:  g) Less schools applied for the FFVP than the SA expected; h) Do all eligible schools  Yes, to the 
know about the availability of FFVP; i) Is the SA satisfied w/the # of school application s for FFVP  concept, not the 
received
wording
"Serving fruits and vegetables"‐‐frequency‐times per week of offering FFVP (times/wk). 
"Distribution methods, time of day, portion 
sizes"
Goal setting to reach # of F/V servings recommended by DG's

Yes

Covered by "role of ffv in complete diet"; 
also a topic more suited to adults or teens 
than elementary students.

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

feedback to  schools to improve implementation

Yes

Feedback would be part of listed activities.

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

Include produce companies/produce growers/farmers

Lorelei 
DiSogra 
(UFPA)
Lorelei 
DiSogra 
(UFPA)

Does this apply to applications for school year 10‐11?

Yes ‐ reference clarified

Add same questions suggested in Q11

Same Q asked for both years.

Q12

Q13a

Q14

C Q15

C Q15a

Q18

F4

F5

Yes ‐ reference clarified

Expand question to include other State agencies (Ag/Health) that may be federally in‐State funded  Yes
to capture full picture.  

Include types of F/V served/offered each month.

Abt responses to FNS Review OMB VERSION.xls

Yes

Covered by" City, County, State, or Tribal 
government agency (e.g. health 
departments, agriculture departments, 
etc.)"
We will revise if needed based on pretest 
feedback.
Captured by " Food purchase cost detail by 
item or category" but could add this as 
separate category for states that get 
menus but don't get/save detail of food 
costs by food item.

Page 24

Dr. Robert C. and Veronica Atkins

Center for Weight and Health
UNIVERSITY OF CALIFORNIA, BERKELEY
College of Natural Resources & School of Public Health
BERKELEY • DAVIS • IRVINE • LOS ANGELES

TO:

•

RIVERSIDE

•

SAN DIEGO

•

SAN FRANCISCO

SANTA BARBARA • SANTA CRUZ

Tracy Palmer, FNS, USDA

FROM: Karen Webb, Lorrene Ritchie and Pat Crawford
DATE: October 16, 2009
RE:

FNS re age appropriateness of student questionnaire and diary
assisted 24 hr recall

The purpose of this memorandum is to respond to reviewers’ comments about the age
appropriateness of the student diary and the student questionnaire. The following information
may be useful in understanding and placing confidence in the diary assisted recall method and
the student questionnaire. Small modifications have been made to the student questionnaire to
address one of the reviewer’s comments as described below.
We understand that a food diary and a prompt list may appear to be difficult for 4th-6th graders.
However, the method in this context is used as a tool to assist a full 24 hr recall interview on the
following day. The basis for the diary assist was a diary protocol developed, piloted, and
validated in the multisite NHLBI Growth and Health Study. This study is the largest
longitudinal study with low income African American and White children’s diet with 2,379
children measured at baseline and annually for the next ten years. The superiority of this method
was demonstrated in a validation study comparing food diaries, 24 hr recall and food frequency
in 9 and 10 year old low income children (Crawford et al., 1994). There are many publications
based on these data (see reference list). A script for student training and practice session, and a
detailed protocol and coding manual for administration of the diary and the modified multiple
pass 24 hr recall will be used to train and oversee data collectors and coders. The investigators
have extensive experience in using these forms of dietary assessment in large scale studies.
It appears that the comments about the age inappropriateness of the self administered student
questionnaire related mostly to the use of YRBS questions on frequency of consumption of fruit
and vegetables. We included food frequency questions as a cross check on the usual intakes of
fruits and vegetables, in comparison to that reported on the 24 hr data. The reason for selecting
YRBS questions is their widespread use, and previous validity testing, albeit in adolescents and
not among elementary school children. However, we could find no questionnaire with validated
questions on fruit and vegetable intakes, so we retained these questions in the student
questionnaire and will specifically assess children’s understanding of the questions in the pilot.
Reviewers suggested we include questions about frequency of snacks and beverages, which we
can do as a check on how they compare with estimates we obtain from the 1-day diary assisted

recalls. Again, we could find no questions about these which had been validity tested in the age
group of interest. However we did find such a questionnaire for older children, the BSQ, and we
have selected questions from that questionnaire and included them in our revised student
questionnaire. We improved the layout and the look of our questionnaire and included pictures
along the lines of the SPANS questionnaire recommended by Laura Leviton, and we will pilot
our revised questionnaire to assess understanding of content with low income 4th-6th graders. It
is notable that all tools identified, including the SPANS questionnaire ask about food intake
“yesterday.” The diary assisted 24 hr recall will capture food intakes more accurately than a
short questionnaire. While it may be possible to develop a simple food frequency questionnaire
for elementary aged children for use in this study, it would have unknown validity, so we have
chosen to supplement our dietary data with selected YRBS and BSQ questions, both of which
have been validity tested with diverse, albeit older youth.
Selected references using food diary method in elementary aged children from the NHLBI
Growth and Health Study:
1. Crawford PB, Obarzanek E, Morrison J, Sabry ZI. Comparative advantage of 3-day food
records over 24-hour recall and 5-day food frequency validated by observation of 9- and 10year-old girls. J. Am. Diet. Assoc., 94:626-630, 1994.
2. Crawford PB, Obarzanek E, Schreiber GB, Barrier P, Goldman S, Frederick MM, Sabry ZI.
The effects of race, household income and parental education on nutrient intakes of 9- and 10year-old girls: NHLBI Growth and Health Study. Annals of Epidemiology 5(5):360-368, 1995.
3. McNutt SW, Hu Y, Schreiber GB, Crawford PB, Obarzanek E, and Mellin L. A longitudinal
study of dietary practices of black and white girls 9 and 10 years old at enrollment: The NHLBI
Growth and Health Study. J. Adol. Health, 20:27-37, 1997.
4. Striegel-Moore R, Morrison JA, Schreiber G, Schumann BC, Crawford PB, Obarzanek E.
Emotion induced eating and sucrose intake in children: The NHLBI Growth and Health Study.
Intl. J. of Eating Disorders. 25:389-398, 1999.
5. Wang MC, Crawford PB, Moore EC, Hudes M, Sabry ZI, Marcus R, Bachrach LR. Influence
of adolescent diet on quantitative ultrasound measurements of the calcaneus in young women.
Osteoporosis International. 9:532-535, 1999.
6. Ritchie LD, Spector P, Stevens MJ, Schmidt MM, Schreiber GB, Striegel-Moore RH Wang,
Crawford PB. Dietary patterns in adolescence are related to adiposity in young adulthood: An
analysis of data from the longitudinal NHLBI Growth and Health Study of Black and White
females. J Nutr 2007;137:399-406.
7. Striegel-Moore RH, Thompson D, Affenito SG, Franko DL, Obarzanek E, Barton BA,
Schreiber GB, Daniels SR, Schmidt M, Crawford PB. Correlates of beverage intake in
adolescent girls: the National Heart, Lung, and Blood Institute Growth and Health Study. J
Pediatr. 2006 Feb;148(2):183-7.
Validation of the BSQ, the Beverage and Snack Questionnaire:
Development and Validation of a Beverage and Snack Questionnaire for Use in Evaluation of
School Nutrition Policies Marian L. Neuhouser, Sonya Lilley, Anne Lund, Donna B. Johnson
JADA, September, 2009 pages 1587-1592

memorandum
Social and Economic Policy

Date

October 20, 2009

To

Tracy Palmer, Ted Macaluso

From

Susan Bartlett, Jacob Klerman

Subject

Fresh Fruits and Vegetables Program Study: Number of Days of Operation per Week

The purpose of this memorandum is to respond to the first section of a memorandum from Tracy
Palmer dated October 8, on variations in impacts with number of days per week that fresh fruits
and vegetables are made available. Our response has three elements:




We suggest that examining variation in number of operating days will not contribute to
understanding the impacts of the FFVP.
We suggest a slight change in the central research question and a corresponding change
in the data collection.
We describe how variations in impacts with numbers of days per week (and timing of
environment changes) could be measured—with the conclusion that this would delay the
study and would have substantial effects on the budget if statistical power was to be
maintained at or near current levels.

Implications of Variation in Numbers of Days of Operation for Quality of Offerings

The section of the FNS memorandum was entitled “Dose-Response or Full vs. Partial
Implementation of FFVP”. While we understand the interest of FNS and child nutrition advocates
in variations in impacts, this nomenclature is seriously misleading. We wish to emphasize that
variation in number of days per week does not correspond to either dose-response or full vs.
partial implementation. With a fixed per-student budget, a school that offers fresh fruits and
vegetables five times per week is offering less expensive selections than a school that offers them
three times a week. The portions must be smaller, or the items must be of lower quality, or they
must be cheaper types of produce. The choice made by the school regarding number of days per
week presumably reflects their judgment of how best to spend the limited budget so as to have the
maximum impact on children, balancing considerations of the size and appeal of the individual
offerings and the frequency of reinforcement of the healthy eating message.
We think that advocates may be discounting the budget constraint, and may be mentally
comparing offerings of the same qualities three vs. five times per week. If, however, the claim is
that FFVP is more effective if the fixed budget is spread over five days per week, despite the
contrary choices made by schools, we would like clarification of this hypothesis.

1

Implications of Variations in Number of Days of Operation for Specification of the
Research Questions

The research questions in the RFP were written to address the impacts of the program on days on
which it was offered. A central impact question in the Statement of Work was:
To what extent does children’s consumption on school days of fresh fruits and vegetables
change on days in which the FFVP provides fresh fruits and/or vegetables to children?
Our study is currently designed to answer this question; we collect dietary recall data for students
for days on which fresh fruits and vegetables were offered.
However, having carefully considered issues about days per week of distribution, we urge FNS to
re-consider and modify this stated goal of the study. Measured outcomes in participating schools
on a day in which FFVP was offered are likely to be larger in schools that implement fewer rather
than more days per week. This is true because schools offering fewer days per week will be able
to offer larger or better portions on those days.
We therefore suggest that FNS modify the research questions to focus on the overall effects of
FFVP on children throughout the school week. If FNS makes that modification, we would visit
schools on a random day of the week rather than on a day that the program was necessarily in
operation. Note that this change would decrease the reported impact of the program relative to the
current design. To understand this, suppose that all schools operate the program either 3 or 5 days
per week. If, as currently planned, we visit schools only on days of operation, our treatment group
will include both children who are getting the benefits of 1/5 of their school’s weekly FFVP
budget and children who are getting the (presumably greater) benefits of 1/3 of the school’s
weekly budget. If instead we visit schools on a random day, the benefits received by children in
the treatment group will be independent of the frequency choice made by their schools, because
40 percent of the children from 3-day-a-week schools will be sampled on days of non-operation.
In our judgment this is the correct way to measure the impact of FFVP. We are happy to discuss
this issue further.
Implications of Variations in Days of Operation and Timing of Environmental Changes for
Subgroup Analyses

In its memorandum, FNS stated that the two subgroups of greatest interest were:
1. Participating schools that implement the FFVP more frequently (4 or 5 days a week)
versus those that implement less frequently (1 to 3 days a week).
2. Schools that implemented policies for a healthier school food environment prior to
application vs. schools that implemented policies for a healthier school food environment
after their entry into the program (or after a comparable time point for non-accepted
applicants) vs. schools that did not implement policies for a healthier school food
environment either before or after.

2

These process-based subgroups are substantially more challenging for estimation of variations in
impacts than subgroups that are based on fixed characteristics of schools or students (e.g.
racial/ethnic composition). The challenge arises from the consideration that the same factors that
dictate schools’ decisions in these arenas (such as the quality of the pre-existing nutrition
program, or the attitudes of the principal, the teachers, and the parents) could also affect student
outcomes in the presence or absence of FFVP.
With regard to number of days of operation, we cannot know how many days the comparison
schools would actually have operated the program if they had been selected. By studying their
applications, we can however learn the number of days they planned to operate. We could
therefore define subgroups of both treatment and comparison schools based on “more frequent
planned operation” (4 or 5 days per week) and “less frequent planned operation” (1, 2, or 3 days
per week). Potential drawbacks and limitations of this procedure are as follows:











We would want to select approximately equal numbers of schools above and below the
cutoff for each subgroup. States do not however attempt to balance selected schools on
this consideration. We might find that schools around the cutoff are disproportionately in
one group or the other, requiring us to go a considerable distance from the cutoff to make
up our sample.
The statistical power of our overall estimates would be reduced because of the need for
disproportionate sampling and greater distance from the cutoff.
We would need to review many hundreds of applications in the 16 States, both accepted
and rejected, to perform the classifications. This will take both substantial calendar time
and project resources.
The result will be a subgroup comparison based on planned days of operation, not actual
days of operation. We are uncertain as to the strength of the relationship between
“planned” and “actual”. This relationship could be measured ex post for the treatment
group schools.
The implications for sample size are considerable. A recent presentation by Klerman (and
earlier papers in the biomedical literature by Rothwell and Wang) imply that detecting
even moderate sized-differential impacts requires very large samples (typically four times
as large as for detecting main impacts; slightly smaller if the sample is highly clustered;
larger if the sample is highly imbalanced in the dimension of interest). If subgroup
analysis is now a primary interest of FNS, we would advise considering quadrupling the
sample size, probably by adding States.
Schools that choose less frequent operation probably differ in important ways from
schools that choose more frequent operation. Hence even though we will have valid
impact estimates for the two subgroups, it requires a leap of faith to attribute the
differential impacts to days of operation. Our conclusion would be descriptive of the
impacts for the two groups of schools. It would not be prescriptive, in the sense that
imposing a requirement of a particular frequency on schools would change impacts.

Similar considerations arise regarding timing of changes in the healthier school food
environment. Again, our comparisons would be based on schools’ planned changes, as reported
in their applications. We would need to review substantial numbers of applications to assign the
groups. The need for three balanced subgroups would have greater deleterious effects on the

3

statistical power for measuring main effects than two subgroups. If effects were needed for both
types of subgroups, we would need to balance over six categories (2 × 3). Also, the results would
be strictly descriptive.
It is not that subgroup analyses per se are difficult or impossible. The difficulty is with processbased subgroups, which (a) require time-consuming analysis of the applications, (b) can only be
analyzed with respect to planned rather than actual values, and (c) can only yield descriptive
results. These issues do not arise with regard to fixed school characteristics. Sample size
considerations are however relevant for all subgroup analyses. Our original proposal proceeded
on the assumption that USDA was interested in sub-group analyses, but that they were not the
primary focus of the study (i.e., USDA did not have funds sufficient to power the study to detect
all but the largest sub-group impacts).
Since our last conference call, we have given careful consideration to several other strategies
which would not require considerable additional data collection. We have concluded that those
other strategies would face severe threats to their internal validity and would not yield believable
causal inferences. The underlying differences between schools that offer fruits and vegetables
more versus fewer days per week would comprise an intractable source of selection bias. Given
FNS's need for a study that will stand up to the scrutiny of the research community, we concluded
that those methods were not worthy of further investigation.

4

memorandum
Social and Economic Policy

Date

October 20, 2009

To

Tracy Palmer, Ted Macaluso

From

Susan Bartlett, Jacob Klerman

Subject

Fresh Fruits and Vegetables Program Study: Participating and Nonparticipating
Schools

The purpose of this memorandum is to respond to the third section of the October 8th
memorandum from Tracy Palmer. This section was entitled “Representativeness of the Sample”
and deals with comparisons of school characteristics of various groups of schools. We would like
to clarify the analyses we have planned to answer the research question posed. We believe FNS
has a slightly different understanding about the analyses we are intending to perform.
Planned Analyses

One of the research questions posed by FNS in the RFP was:
“For the School Years 2009-10 and 2010-11, what are the characteristics of the schools that
were selected in each State to participate and how do they compare to those that were not
selected? To other schools in the State or district?”
Our approach to answering this question is summarized in the Exhibit 1 on the following page.
This analysis would be performed for each of the 16 study States and combined across all 16
States. The six columns in the exhibit refer to:
(1) all elementary schools in the State, according to the Common Core of Data (CCD);
(2) all elementary schools that are eligible according to the CCD, i.e. in which at least 50
percent of students are eligible for free/reduced price meals1;
(3) all elementary schools that are eligible according to the CCD, in districts that had at least
one eligible applicant for FFVP;
(4) all elementary schools that applied for FFVP (and were eligible);
(5) all elementary schools that are eligible according to the CCD, in districts in which at
least one school participates in FFVP;
(6) elementary schools participating in FFVP.
1

The CCD does not indicate whether a school participates in the NSLP. However, since over 90 percent of
public school districts do participate, this should not substantially affect the comparisons.

1

Exhibit 1: Planned Comparisons of Participant and Nonparticipant Schools
All elementary schools that are …
In
Eligible In districts
Eligible
In districts
State (2)
which have
applicants
which have
(1)
eligible
(4)
FFVP
applicants
participants
(3)
(5)
Percent
NA
NA
NA
NA
free/reduced
price, on
applications
Percent
free/reduced
price, in CCD
Demographic
characteristics,
from CCD
School
characteristics,
from CCD

FFVP
participants
(6)

Comparison of the first two rows for eligible applicants and for schools selected to receive FFVP
(columns 4 and 6) will provide some guidance on how well the CCD measures current school
characteristics.
For rows (2) through (4), this tabulation will tell us:
1. How eligible schools compare with all schools statewide with regard to poverty and
demographics (columns 1 and 2): What does the eligibility screen accomplish? Are
eligible schools notably poorer and otherwise different from other schools in the state?
2. How eligible applicants compare with all eligible schools in their districts (columns 3 and
4): Who chooses to apply? Are the schools that apply in each district drawn from among
the poorer eligible schools, or are they otherwise different?
3. How FFVP schools compare with all eligible schools in their districts (columns 3 and 6):
How do participants compare with eligible nonparticipants? Are the schools that
participate in each district poorer or otherwise different from other eligible schools?
4. How FFVP schools compare with eligible applicants (columns 4 and 6): What does the
selection process from among applicants accomplish? Are the schools that participate in
general poorer or otherwise different from those that apply?
5. How FFVP schools compare with all other schools in their districts (columns 5 and 6):
What is the final result of the selection process? Are the schools that participate in each
district notably poorer and otherwise different from other schools in those districts?
In addition, for all 54 “states”, we will collect some numerical information on the application
process: number of eligible schools, number of schools applying, number of schools selected, and

2

limited information about the characteristics of the schools selected (% with FSL 60-75 and
above 75).
Merging with the CCD

Our planned analysis requires that we merge the data we have received from States covering all
schools that applied for FFVP in the 16 States (and were eligible). This match will support
tabulations both for each State and for all 16 States combined. Furthermore, we will include a
narrative discussion of how the state's approach to selecting schools affected the characteristics of
the schools actually selected (assuming we resolve confidentiality issues).
Merging State lists with the Common Core Data (CCD) allows us to characterize schools by
student demographics such as race and ethnicity, and by school characteristics such as highest
and lowest grade served. FFVP eligibility (based on percent of students eligible for free and
reduced price meals) is however time-dependent, and the information in the CCD will not be as
current as the data on FFVP applications.
We also note that it is quite time intensive to match schools from State lists to the CCD. State
lists include only school name and district. We will need to sort the CCD by State and district and
then proceed to do the matches manually.
FNS Memorandum

The language in the FNS memorandum that differs from our plan is as follows:
FNS believes this data provides the FRP numbers for the schools in the State and that
under the contract Abt will inform us (for the 16 States) on how many schools in the
State meet the FFVP FRP cutoff, of those how many applied, and of those how many are
funded.
To get consistent counts for these would require that the States provide us with current
information on percent free/reduced price for all schools meeting the 50 percent free/reduced
price cutoff, regardless of whether they applied. With this information we could fill in column
(2) in the first row of Exhibit 1. This data request would increase State burden. Our plan, in
contrast, compares the State counts of applicants and participants (columns 4 and 6 in Exhibit 1)
with CCD estimates of number of eligible schools.
Our current plan includes only simple counts beyond the 16 selected states. If FNS wants, Abt
would be willing to cost out an implementation analysis for all 54 “states”. An analysis like that
would include requesting lists of schools from every states (eligible, applying, selected) as well as
process information (i.e., how was the selection done). This would support 54 "case studies".
We look forward to guidance from FNS on this issue.

3