NASS Comments -1

OMB Docket for the Food and Nutrition Service: Evaluation of the Fresh Fruit and Vegetables Program

The OMB package for the Food and Nutrition Service’s evaluation of the Fresh Fruit and Vegetable Program (FFVP) was prepared by Abt Associates, who designed and will also administer the multiple surveys comprising the complex evaluation. The package is comprehensive, and generally well-presented, but there is a notable omission: no questionnaire was included for any of the component surveys; appendices C through I are blank. Although Appendix A contains a helpful overview of the planned data collection, outlining outcome measures and planned analyses (Exhibit A-2), we could not specifically review question sets for the Survey of State Child Nutrition Agencies (C), the Survey of School Food Authorities (D), the Survey of School Principals (E), the School Food Environment Assessment (F), the School Food Service Manager Interview (G), the Teacher Survey (H), or the Student Self-Administered Questionnaire (I). Within the text of the document, item A.8 understandably contains blanks (since the Federal Register’s announcement of the impending evaluation had not appeared at the time the version of the docket sent to us was completed).

The surveys constituting the FFVP evaluation fall into two groups: those targeting the impact of the program on the participating schools and their students (impact study), and those focusing on the implementation of the program (implementation study). The main feature of the impact study is a survey based on a regression discontinuity design covering elementary schools in 16 states (with selection of thirteen states by region--two from the Northeast, three from the Midwest, six from the South, and two from the West-- based on PPS sampling where the measure of size is the number of elementary school students attending schools where at least 50 percent of the students participate in the National Free or Reduced Lunch Program (FRLP), and including California, Florida, and Texas as certainty states). Within the selected states, sample elementary schools are selected by a PPS scheme, some slightly above their state’s cutoff for the Fresh Fruit and Vegetable Program, hence eligible and participating in the program, and some slightly below their state’s cut-off , hence not participating in the program. FFVP eligibility criteria vary by state—only elementary schools may participate, and those schools with the highest percentages of low income students for their states have the highest priority for inclusion. The total school sample for all 16 states was designed to contain 128 responding FFVP-participating schools, scoring above the state-specific FFVP cut-offs, and 128 responding schools falling slightly below them (and not participating). Within the selected schools, one classroom will be randomly selected from the fourth, one from the fifth, and one from the sixth grades represented in the school, and within each of the three classrooms selected, ten students will be drawn into a stratified cluster sample, along with their teacher. An 80% response rate is posited, yielding a 24-student sample per school. These students will be trained to complete a 24-hour food diary for one specific day, with the assistance of a caregiver; they will also be interviewed by a survey specialist on their food consumption and nutritional attitudes, on the diary due-date, the day after the diary date. Because FFVP-eligibility scores of the surveyed schools, whether they are program participants or not, are similar within their state, all falling close to the state cut-off , program participation can be considered random within this population, regression equations can be run, and outcomes for the FFVP-participating students may be attributed to their program participation, once allowance is made for demographic and “school-environmental” covariates in the regression equation. SAS Proc SurveyReg will be used to account for the sample design in the development of regression equations, but the possible regression models have not been specifically described (possibilities are sketched out in Exhibit A-2) Carrying out the impact survey plan and obtaining valid data requires official input on state FFVP cutoffs, on schools applying for the program, with school demographics, school environmental characteristics, and precise school scores obtained on the poverty characteristic used to test for eligibility within the sample state. Abt plans to obtain these data from the state Child Nutrition Agencies (CNAs) for each state included in the surveys. Through an additional web survey, Abt plans to obtain further data on the FFVP from School Food Authorities (those entities legally responsible for administering the FFVP and other federal school programs, at school district level) to have their assessment of the FFVP, details of the program administration, foods offered, any FFVP-related changes in the School Breakfast Program or the National School Lunch Program. Additional validating data on the FFVP food items served, their scheduling and venues, will be obtained through a short interview with Food Service Managers at the selected schools. There are also visits by trained observers checking the physical environment of the FFVP schools, the set-up and conditions for distribution of the FFVP fruits and vegetables, and the presentation of nutritional information in the schools, completing the School Food Environment Assessment cited in Attachment F. The teachers of the students in sample contribute their own data through a short, self-administered survey, distributed with the student diary forms, and the principals of the sampled schools are asked to complete a web survey on their school’s FFVP.

Note that the three main subsidiary surveys--of CNAs, School Food Authorities, and Principals--contributing to the impact study also supply data for the implementation study, intended to provide national estimates of program implementation procedures by FFVP-participating schools. The regression study, with its small sample of schools all selected close to the state cut-off scores for program participation, cannot be generalized to the whole set of FFVP schools. For the implementation study, in addition to the 128 FFVP-participating schools included in the regression sample, an additional 560 participating schools will be included, with the goal of providing at least 448 additional FFVP-participating (and responding) schools (yielding a total of 576 FFVP schools), assuming an 80% response rate at school level. Sampling details for the additional 560 schools are not given (the documentation does state that, for generalizability, all FFVP schools in the continental U.S. will have a positive selection probability). It is clear from the description and Appendix A that sampling stops at school level for this survey. Analysis to be performed for the implementation study is not described in detail; estimates are to be descriptive in nature, “consisting primarily of proportions.” According to Appendix A, the school-level data on FFVP implementation will be obtained from the three subsidiary surveys feeding the regression study, which will be extended to include the additional 560 FFVP schools selected and all 54 state Child Nutrition Agencies. The surveys involved are all internet surveys, and web-based surveys are known to have particular unit nonresponse issues; whole unit response rates for these three (including the state CNA survey) may easily sink below 80%, according to Don Dillman. Item nonresponse is also highly probable, and will have to be dealt with.

From B.3 in the packet, it is clear that considerable thought has been given to maximizing response and gaining student and school support for the impact study. The importance of gaining the support of the state agencies is acknowledged. Certain measures have been taken toward these goals: schools and students will receive modest incentives for their participation; a study liaison will be designated to visit the classrooms and deliver study packets, and reminder letters will be provided to be sent home to caregivers whose child’s food diary isn’t turned in on the due date. It should also be noted that some preliminary testing has been carried out: student/parent/teacher/food service manager parts of the impact survey were pretested by an Abt associate in two California elementary schools, in a small test involving nine students, their parents, two teachers and a food service manager (the method of selection is not stated). In a follow-up session, some difficulties were noted, suggestions were made for improving these instruments, and certain questions were revised. However, the California pretest may not be conclusive: it is no easy task for ten to twelve-year-olds in schools with high poverty rates to provide reliable, informative survey data, even with the assistance of caregivers—and it is not clear that they will be able to do so. In any case, more information is needed. The questionnaires should be included in the packet, because question sequence and skip patterns for the surveys influence response patterns. Obtaining complete, valid data from CNAs and School Food Authorities is particularly crucial to the success of the project.