FFVP-CFD_Final OMB_Part_B_AG-3198-S-14-0028.(2014-06-30)

Evaluation of the Pilot Project for Canned, Frozen, or Dried Fruits and Vegetables in the Fresh Fruit and Vegetable Program

Contract Number:
AG-3198-S-14-0028

OMB Supporting Statement
Part B: Statistical Methods

June 30, 2014

Project Officer: Allison Magness

Office of Policy Support

Food and Nutrition Service/USDA

Telephone 703-305-2098

This page has been left blank for double-sided copying.

CONTENTS

Part B: Statistical Methods

B.1. Respondent universe and sampling methods 1

B.2. Procedures for the collection of information 6

1. Design rationale 6

2. Sampling methodology 9

3. Expected precision 11

4. Sample weighting, nonresponse adjustments, and variance estimation 14

B.3. Methods to maximize response rates and deal with nonresponse 16

B.4. Tests of procedures or methods to be undertaken 16

B.5. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data 17

This page has been left blank for double-sided copying.

Part B: Statistical Methods

B.1. Respondent universe and sampling methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection methods to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The respondent universe or target population for the Food and Nutrition Service (FNS) evaluation of the Canned, Frozen, or Dried (CFD) Fruits and Vegetables pilot project of the Fresh Fruit and Vegetable Program (FFVP will consist of low-income elementary schools serving grades 4 through 6 that apply and are selected by FNS to participate in the pilot program (FFVP-CFD) during the 2014–2015 school year (SY). At this time, the universe of such schools is unknown; it will be determined by the end of summer 2014.¹ As a result, two sampling plans, Plan A and Plan B, are presented to encompass the range of possibilities that result. Under Plan A, if more than 100 schools are approved, the study will select a probability sample of up to 100 schools to participate in the evaluation. Under Plan B, if 100 or fewer schools are approved for the pilot, the evaluation will conduct a census of these schools; for this discussion we assume 75 schools will be included under Plan B although the actual number maybe larger or smaller. Within the selected schools, we will collect data from principals, food service managers (FSMs), classroom teachers, students (surveys and dietary recall), and students’ parents, as outlined in Exhibit B.1.

The FFVP-CFD pilot evaluation is designed to collect data from a variety of sources, so we can fully understand (1) the impact of the pilot program on schools and students; (2) how schools, food service staff, and (as appropriate) School Food Authorities (SFAs) implement the pilot program; and (3) the challenges and benefits the program provides. Under Plan A, the study design for the FFVP-CFD evaluation uses a multilayered, five-stage design to sample the appropriate entities for data collection: (1) States and SFAs; (2) schools and school FSMs; (3) classrooms, their teachers, and FFVP snack food service observation on program days; (4) classroom students and their parents; and (5) for a subset of the selected classroom students, dietary recall and plate waste. This five-stage design closely follows the stages used for the national evaluation of the Fresh Fruit and Vegetable Program (Bartlett et al. 2013). Exhibit B.1 presents a summary of the proposed FFVP-CFD design under Plan A. Exhibit B.2 also provides the proposed sampling plans under both Plan A and Plan B (described in more detail in section B.2). As noted, the sampling plan will consist of a five-stage design if a State/SFA and/or school selection stage is required under Plan A; otherwise, if the number of participating schools is fewer than or equal to 100, the study will conduct a census of these schools (Plan B – 75 schools) and the first stage of selection will begin with stage 3, the selection of classrooms from participating schools. Because, the FFVP snacks are generally served in the classroom (or outside of it, but to the children in that classroom), the design for both plans includes a classroom selection stage to limit the costs of having to place observers in several places throughout the day, following the prior FFVP evaluation procedures.

Exhibit B.1 also presents the expected response rates for each of the five sample components under Plan A (see Appendix B for additional details). The expected response rates are based on the prior FFVP evaluation (Bartlett et al. 2013), required participation by the approved schools, prior experiences conducting similar school nutrition studies, and the proposed data collection strategy which is designed to closely achieve the OMB response rate standard of 80 percent or higher across the data collection entities. The expected response rates range from 80 percent for the parent survey and 80 percent for the student survey and in-school dietary recall to 97 percent for the school FSM survey.

Exhibit B.1. Summary of data collection components, sample sizes, and response rates (Plan A)

Data collection component	Initial sample each period		Methods	Expected response rates
	Fall period 2014	Spring period 2015
State Child Nutrition Director Survey	54	--	All 54 States and territories	80% State Child Nutrition Directors
SFA Director Survey	50a	50a	For up to10 States, five per State	80% SFA Directors
School menu and nutrition assessments FSM survey and Principal survey	100 schoolsa	100 same schoolsa	Up to 100 schools in up to 10 Statesa. Repeat data collection in spring.	100% school participationb 80% FSM 80% principal
Classrooms: observation and teacher surveys	250	250	Two to three classrooms selected per school. Same classrooms used in both periods.	100% classroom 80% teachers
Parent and student surveys	5,000 parents 3,750 students	5,000 parents 3,750 students	Conducted on one day in each classroom for all classroom students	80% parents 95% students
Total student diary, in-school recall and plate waste collections (students)	3,300 (3,000)	3,300 (3,000)	Select 12 students from each classroom (24 to 36 per school). To the extent possible, the same students will be interviewed in the fall and spring. Second-day recall for 10 percent of students.	95% students

Note: All data collection components are proposed to be pre-post with the same students and schools.

^a Assuming Plan A in Exhibit B.2 is exercised, consisting of a sample of up to 100 schools selected, two each from five SFAs in up to 10 States.

^bAs part of the approval process schools must participate in the research.

Exhibit B.2. Summary of sampling steps and data collection components (Plans A and B)

Based on input from the Child Nutrition Directors for States selected for the pilot evaluation and school administrative officials (for example, principals and school FSMs), the study will build a sampling frame of pilot schools for the study, lists of classrooms for selected schools, and lists of students for selected classrooms. Depending on the number and the SFA membership of the pilot schools, and from which sources the data required are available (for example, school lists, school menus, student lists), under Plan A, the design may select five SFAs per State and two schools per selected SFA, to create a sample of up to 100 schools for stages 1 and 2 of the sample design. Under Plan B, these two stages of selection will be eliminated if a census of the participating schools is conducted. For each selected school, the contractor (Mathematica Policy Research, Inc.) will identify a target week to conduct the school and student data collection.

In subsequent stages, using the sampling plan, the contractor will select 2 or 3 classrooms from each selected school (250 classrooms for Plan A; 188 for Plan B) to observe the FFVP-CFD snacks served. For stage 4, the design will select on average (12 students under Plan A; 16 under Plan B) (with parental consent) from each selected classroom so that, overall, the evaluation will collect a single in-school day dietary recall [Plan A: for 24 to 36 students per school (30 on average. Plan B: for 32 to 48 students per school (40 on average)] in each of the selected schools over the designated week (10 percent of these students will also receive a recall on a second day), to yield up to 3,300 recall observations in each period (6,600 total fall and spring; 3,300 students per period) under both Plan A and Plan B. The selected classrooms and their selected students will be randomly assigned to ten daily interviewer slots consisting of a combination of each day of the week (five days, Monday - Friday) by two interviewer assignments (10 per week²). On a program day, the designated interviewer will observe the snack in the classroom assigned to their slot, will collect and measure the plate waste of the students in that classroom selected for recall on that day, and will conduct the dietary recall with these students by the end of the day or the next morning. On a nonprogram day, the interviewer will conduct the dietary recall with the students assigned to their slot.

The assignment of selected classrooms and students to the day-by-interviewer slots was designed to achieve multiple statistical and operational objectives. First, this approach creates a consistent workflow for the interviewers to save field labor costs. Second, it reduces burden on students; and third, it limits the degree to which the students’ and school staff’s school day is disrupted, while at the same time allowing the study to prepare consumption estimates that would be representative of the entire target week (and, separately, the program days in the week). Moreover, this approach permits field staff to observe FFVP snacks across all days in the week which increases the precision in the estimates considerably over a design that selects one day per week to conduct the recalls. This is due to the fact that most if not all students will receive the same snack items on a given day; but will tend to receive different food items over the course of the week.

B.2. Procedures for the collection of information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection

• Estimation procedure

• Degree of accuracy needed for the purpose described in the justification

• Unusual problems requiring specialized sampling procedures

• Any use of periodic (less-frequent-than-annual) data collection cycles to reduce burden.

1. Design rationale

The FFVP-CFD pre-post evaluation design and the associated sample sizes were chosen to maximize the statistical precision in the study estimates (for example, comparisons of the consumption levels and nutritional intake between the fall FFVP standard program and spring pilot FFVP-CFD implementation), considering operational, burden, and pilot program implementation factors, plus the fact that the study is limited to a fixed level of funding designated by Congress. Exhibit B.3 shows the sample sizes for Plan A and Plan B. Under Plan A, up to 100 schools will be sampled; under Plan B, a census of 75 schools is assumed.

Exhibit B.3. Minimum detectible differences in cups in fruit (F) and vegetable (V) consumption between fall and spring under two sampling plans for the FFVP-CFD program. Plan A: for a sample of 100 schools. Plan B: for a census of 75 schools.

Plan	Total students receiving recalls	Entire sample	50 percent subgroup of students (e.g., males)	Program day intake	For a given meal (30 to 40 percent of the daily consumption) or food type (e.g., F or V separately)
A.100 sampled schools (30 dietary recalls per school)	3,000	0.16	0.18	0.17		0.08
B. Census of 75 schools (40 dietary recalls per school)	3,000	0.08	0.09	0.08		0.04

A major consideration in selecting the design was whether to use the proposed pre-post design where the pre-collection under the standard FFVP occurs in the fall and the post-collection under the FFVP-CFD pilot in the spring. Such a pre-post design may introduce a seasonal effect based on differences between the fresh fruits and vegetables available between the two periods and other seasonal or timing changes in student activity, climate, or program implementation not directly attributable to the transition to the pilot program. Because of this concern, the study examined different approaches and evaluated their strengths and weaknesses. In the end, the pre-post design was found to be the best option, but features were added to the pre-post methodology to reduce any seasonal effects to the extent possible and to alleviate or control for these effects, while keeping the cost of the study within the resources available and the schedule set by Congress for the evaluation.

Other options are (1) using a design in which a control group of nonpilot schools (not implementing the pilot use of CFD fruits and vegetables in both the fall and spring) is added to both the fall and spring periods, or (2) limiting the evaluation to the spring period and comparing the pilot schools to a control group of nonpilot schools. Adding a nonpilot school control group to both periods would enable the study to conduct a differences-in-differences analytic approach to account for the timing and seasonal effects; a comparison (difference) of the control group results from the fall to spring would measure the impact of a change in the season without executing the pilot FFVP-CFD program. Another option is to conduct the evaluation of the pilot FFVP-CFD program in the fall of the next school year (or in addition to a spring collection), to yield a fall-to-fall evaluation. Unfortunately, all these approaches either add substantial costs to the study or require a reduction in the sample sizes to offset these costs, which would greatly reduce the expected statistical precision of the estimates of the pilot impact. Moreover, the best use of a control group of schools would be to match nonpilot schools to the pilot schools based on location, food service patterns, and student body demographic and social-economic characteristics. However, because the pilot schools may be unique in many ways, and may consist of all elementary schools in a given SFA, it might not be possible to match nonpilot schools to the pilot schools sufficiently to achieve an effective control group; this, in turn, would introduce new errors into the measurements from the differences between the control group and pilot schools. In addition, the nonpilot schools will have less motivation to participate in the research. Because the participating schools have agreed to participate in this research study as part of the FNS pilot approval process, we expect the participating school cooperation rate to approach 100 percent. In addition, a fall-to-fall comparison would delay the project evaluation, and many of the students will have moved away or changed schools (as they advance a grade), preventing us from being able to interview the same students in all periods (a key feature of the design to be discussed next).

Because these options were not feasible, to alleviate the potential seasonal effects under the pre-post approach, the study was enhanced to do the following three things:

1. Resample the same students in both periods, to the extent possible, to eliminate differences in factors associated with their food consumption between the fall and spring periods (student tastes, body size, and general activity levels) that would influence the results if different students were interviewed in each period.

2. Conduct in-school dietary recalls on program and nonprogram days on every day of the target week and spread the evaluation of schools over 10 to 12 weeks in each of the fall and spring periods, which increases the diversity of the fruits and vegetables being served and observed in both periods. This, in turn, reduces differences in the type of fruits and vegetables offered between the fall and spring that are seasonally based (for example, fresh apples in the fall versus fresh peaches in the spring), rather than those changes that result from the implementation of the pilot program. The pilot program gives the school the option to change the format of the food (fresh to CFD, such as fresh peaches to canned peaches), which may also affect the type of food offered and the portion size that is not seasonally related (for example, ½ cup of fresh apple slices to 2/3 cup of canned pineapple). Ultimately, with the widespread availability of fresh fruits and vegetables throughout the year, the seasonal effect on the foods offered may be minimal; however, whatever effect it has will be reduced by spreading the school collection over 10 to 12 weeks in each period and observing snacks over a full week in each school. Likewise, this approach allows for making estimates of the complete impact of the program during a typical school week.

3. Use a regression-based analysis to control for (as desired) differences in the types of fruits and vegetables being offered (fruit vs. vegetable, popular/common vs. unique/new), the portion size given, and whether the FFVP snack was served immediately after, or before, a physical activity, along with any other situations that may change between the periods that are not a direct result of the pilot implementation.

These design enhancements will help achieve a high level of statistical accuracy so we can effectively judge the impact and implementation factors associated with the pilot for future policy decisions.

2. Sampling methodology

The study plan assumes that some level of sampling may be needed to select schools within each of the States designated by FNS; however, the number of States and elementary schools that apply for the pilot might be small enough that a census would be conducted. It is also possible that the study will sample SFAs first and then sample schools from the selected SFAs, considering whether (1) the data required for sampling are available from the State or SFA; (2) menus and food purchasing (for example, what fruits and vegetables will be offered) are SFA-based or school-based functions; and (3) we want to tailor the school sample based on various SFA-level characteristics potentially including urbanicity (e.g. city, suburban, town, rural), enrollment size, year school began participation in the FFVP, and percent of students approved for free or reduced-price lunch. The final strategy will be determined when the list of States and schools participating in the pilot are known.

Under Plan A, we will sample five SFAs per State, followed by a selection of two schools from each selected SFA to reach up to 100 schools studied. We plan to use probability-proportionate-to-size (PPS) selection methods, coupled with stratification, to select SFAs and schools. These methods assign a size measure (MOS) to each SFA and/or school based on the number of students they contain so that larger SFAs/schools have a greater chance of selection than smaller ones. This process also leads to a “self-weighting design,” in which a consistent number of students are selected per school; this improves the statistical precision in the estimates. The PPS selection process, combined with the use of explicit stratification, also allows us to base the MOS on a set of factors, so that SFAs and/or schools with traits of interest are sampled at a higher rate to ensure that a sufficient sample is obtained to support subgroup-level inferences. We will determine the subgroups of interest in the next few months, as we design the final methodology once the pilot schools are selected.

For the proposed design, we assume the oversampling for subgroup analysis will impart a design effect of 1.5 in developing estimates of the minimum detectable differences (MDDs) that the sample should support for the change in the cups of FVs consumed between the fall and spring periods. The actual precision in the estimate would be higher or lower, depending on how much subgroup targeting is desired to conduct subgroup analysis. For example, we may stratify SFAs and/or schools in each State by urban, suburban, or rural location, and take rural schools at a different sampling rate. To draw the school sample, we will obtain from the State or SFA, along with the Common Core of Data (CCD), a list of the schools in the pilot CFD program that have students in any grades 4, 5, and 6. Under Plan B, we assume a census of 75 schools will be conducted.

Within the selected schools, we will pick two to three classrooms, on average, to participate in the study, either at random or using PPS methods. If the snack is not served in the classroom, we will sample the point-of-service (kiosk, cafeteria, hallway, and playground) locations where the snacks are distributed during the target week for FFVP snack observation and collect plate waste from the students assigned to the interviewer’s daily slot.

The contractor will collect parent surveys from parents of each selected student as part of the consent process. For the student in-school diary recall (and plate waste) assessments, they will obtain a list of the students with consent in the fall in each selected classroom (assuming about 21 students in each). They will select an average of (30 Plan A; 40 Plan B) students across the two or three classrooms, and assign the selected classrooms and students to one of 10, five day a week by two interviewer slots as noted. They will also select additional students to serve as replacements for a student who is absent or otherwise cannot participate on the day(s) assigned, or who does not participate in either the fall or spring period.

3. Expected precision

For each of the data collection components, we estimated the 95 percent confidence half-interval associated with estimates for the fall and spring (pre- and post-) periods and the MDD in the FV cup portion consumed between the fall and spring periods at 80 percent statistical power under the two design Plans A and B. For the student, parent, principal, teacher, and FSM surveys, we based the precision measures on a binary variable with a 50 percent mean (for example, a response to a question such as, “What percent of the time do you eat the vegetables offered?”), and for the dietary assessments, in terms of the FV cup equivalent consumed. We also examined the precision estimates for a 50 percent subgroup (for example, males only) for a given meal, and for separate fruit and vegetable consumption estimates.

The expected sampling precision estimates account for various aspects of the proposed design. Under Plan A, the staged and clustered nature of the design (for example, schools within SFAs, classrooms within schools) creates some redundancy in the responses, because the responses at each level of the sample design tend to be similar among other respondents who are members of the same cluster; this, in turn, reduces the effective sample sizes and precision levels. For SFAs, the level of redundancy as measured by the intra-class correlation (ICC) would be as high as 0.10, because schools within the same SFA of the same grade span tend to serve the same menus over the week, which would have a substantial influence on what the student sees and consumes. This factor is not required under the census plan, Plan B. Furthermore, whenever some overlap occurs in the program FV snacks offered each day in the selected target week (for example, the school serves fresh apple slices as the snack throughout the week), this decreases the precision level for both plans. Although such overlap is expected, we assume there will be enough variability in the FFVP items offered over the week to warrant a full-week collection of data on student FFVP and FFVP-CFD snack consumption to learn about consumption levels across different types and forms of fruits and vegetables. Finally, we must consider how much consumption will vary from student to student, based on interest, taste, age, gender, body size, and length of time since breakfast or lunch, as well as competition of snack time with other activities (such as a physical activity break). Under both Plan A and Plan B, we assumed that the distribution of cup equivalents of fruits and vegetables consumed by students has a standard deviation of 0.30 cups for a given meal or fruit (F) versus vegetable (V) separate daily consumption. For both plans, we also assumed that, for the full daily consumption of FVs, the standard deviation would be 0.60 cups. Using data from the third School Nutrition Dietary Assessment Study (SNDA-III), we found a standard deviation in the daily values of 0.56 for FVs to support this assumption.³

Exhibit B.3 presents estimates of the MDDs at 80 percent power for the student’s in-school day recalls for Plan A and Plan B. Plan A in row 1 assumes a sampling of up to 100 schools will be conducted from a larger population of participating schools as proposed. Plan B assumes that only 75 schools are selected for the pilot program and that a census of the schools will be obtained but a larger sample of students is collected per classroom to keep the total number of in-school recalls under both plans the same. The two plan scenarios as noted provide the range in the potential sample designs and their related precision levels that could occur in the actual implementation.

As Exhibit B.3 shows, by conducting a census of all pilot schools under Plan B, thus eliminating the sampling variation from SFA and school sampling, we can increase the precision in the estimates considerably over the values under a 100-school sample plan. Under Plan B, assuming a census of 75 schools is conducted using a sample size of 3,000 students per period, the entire sample MDD drops to 0.08 instead of 0.16 under a sample of 100 schools from a larger population. The information from Exhibit B.3 indicates that, for Plan A, the study is expected to:

• Measure a change in school day weekly consumption between the fall and spring period across the sample of 3,000 students per period of 0.16 cups at 80 percent power.

• Measure for a 50 percent subgroup (such as males or females) a change in school day weekly consumption between the fall and spring of 0.18 cups at 80 percent power.

• Measure on program days, expected to be limited to 2,200⁴ students per period (800 on non-program days), a change in school day consumption between the fall and spring of 0.17 cups at 80 percent power.

• Detect for a given meal (30 to 40 percent of the daily consumption) or food type (for example, fruit or vegetables separately) a change in school day consumption between the fall and spring of 0.08 cups at 80 percent power.

• Measure (not shown), for a general 50 percent characteristic (such as, “What percent of the time do you eat the vegetables offered at home?”) from the parent and student surveys, the percentage to within 3.6 percentage points and detect changes in these percentages between the fall and spring of 6.5 percent at 80 percent power. Likewise, for teacher and classroom measurements, we expect to be able to measure such a percent to within 4.5 percentage points and to detect changes between fall and spring of 8.1 percent.

Overall, the results show that, at the sample sizes proposed under each plan, the level of precision will be sufficient to detect meaningful impacts from the pilot program for the overall estimates and select key subgroups. The actual precision levels may be lower or higher, based on the final sample design and considering the survey nonresponse adjustments.

4. Sample weighting, nonresponse adjustments, and variance estimation

For estimation purposes, sampling weights reflecting the overall probabilities of selection and differential nonresponse rates will be attached to each data record providing usable SFA data. The study will create a base weight under a 100-school sampling plan for the classroom-level data equal to the reciprocal of the probability of selecting the SFA for the study, the school within the selected SFA, and the classroom within the selected school. For student and parent surveys, we will adjust these base weights further to account for nonconsent and nonresponse within cells consisting of students and/or parents expected to be homogeneous in response propensity. To determine the appropriate adjustment cells, we will conduct a nonresponse bias analysis (even though the response rates are expected to be 80 percent or higher) to identify characteristics of students and parents that are correlated with nonresponse. The potential set of predictors to be used to define the adjustment cells will come from the school rosters in each selected school and characteristics of the school from the school sampling frame. Within these cells, a weighted response rate will be computed and applied to the base weights to obtain the corresponding nonresponse-adjusted weights. Similarly, for the in-school dietary recall estimates, we will adjust the base weights for student selection for the in-school recall from the consented students in the classroom and for student recall nonresponse.

To properly reflect the complex features of the sample design, we will use jackknife replication to calculate standard errors of the survey-based estimates. Under the jackknife replication approach, 100 or more subsamples, or “replicates,” will be formed in a way that preserves the basic features of the full sample design. A set of weights (referred to as “replicate weights”) will then be constructed for each jackknife replicate. Using the full sample weights and the replicate weights, estimates of any survey statistic can be calculated for the full sample and for each of the jackknife replicates. The variability of the replicate estimates is used to obtain the variance of the survey statistic. The replicate weights can be imported into variance estimation software (such as SAS, SUDAAN, WESVAR) to calculate standard errors of the survey-based estimates. In addition to the replicate weights, stratum and unit codes will be provided in the data files to permit calculation of standard errors using Taylor series approximations if desired. Although replication and Taylor series methods often produce similar results, jackknife replication has some advantages in reflecting statistical adjustments used in weighting such as nonresponse and poststratification (see, for example, Rust and Rao 1996).

B.3. Methods to maximize response rates and deal with nonresponse

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Achieving the specified response rate involves using the procedures described next to locate the sample members to secure participation. We expect that 80 percent or higher of the State Child Nutrition Directors, sampled SFA Directors, and FSMs will complete the data collection.

Here, we describe procedures to be followed to maximize the number of sample members who complete the survey:

• The letters inviting SFA Directors and State Child Nutrition Directors to participate in surveys will be carefully developed to emphasize the importance of this study and how the information will help FNS better understand and address current policy issues related to Special Nutrition Program operations.

• Designated FNS regional staff will serve as regional study liaisons and be kept closely informed of the project so that they will be able to answer questions from SFAs and States and encourage participation.

• The contractor will have a toll-free number and study email address so that States, SFAs, and schools can receive assistance with completing the requirements for the evaluation study.

• We will follow up by telephone with all sampled SFA and school food service staff who do not complete the survey within a specified period and urge them to complete the survey. At that point, if FSMs, Child Nutrition Directors, or SFA Directors prefer to complete the survey or remaining sections of the survey by telephone, an interviewer will administer the survey or remaining parts of it by telephone.

B.4. Tests of procedures or methods to be undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of test may be submitted for approval separately or in combination with the main collection of information.

The data collection will rely largely on instruments and individual items that have been fielded in previous studies. Therefore, pretesting with more than nine respondents will not be necessary.

B.5. Individuals consulted on statistical aspects and individuals collecting and/or analyzing data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The information will be collected and analyzed by Mathematica Policy Research, Inc. The sampling procedures were developed by Michael Sinclair (telephone (202): 552-6439) of Mathematica. The sampling plans were reviewed internally by John Hall (telephone (609) 275-2357, senior fellow at Mathematica. Brent Farley (telephone (202) 720-3489) of the National Agricultural Statistics Service (NASS) has also reviewed this supporting statement and provided comments that have been incorporated.

References:

Bartlett, S., L. Olsho, J. Klerman, et al. “Evaluation of the Fresh Fruit and Vegetable Program (FFVP): Final Evaluation Report.” Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service, 2013, Exhibit A.1, page 160.

Rust, K.F., and J.N.K. Rao. “Variance Estimation for Complex Surveys Using Replication Techniques.” Statistical Methods in Medical Research, vol. 5, 1996, pp. 283–310.

1 In the FFVP evaluation (Bartlett et al. 2013), the report indicates that “in the study year, 2010–2011, 4,950 schools participated in FFVP nationwide, serving an estimated 1.9 million students. FFVP allocations ranged between $50 and $75 per student. In 2010–2011, the minimum number of schools funded in a State was 41 and the maximum number was 209. The majority of States funded between 50 and 100 schools.” As noted only a subset of these schools are expected to apply and be approved to participate in the pilot FFVP-CFD program.

2 Two interviewers will be working in the school each day. Interviewer one will work slot 1 on Monday and Interviewer two will work slot 2 on Monday. Likewise on Tuesday, interviewer one will work slot 3 and interviewer 2 will work slot 4 and so on. The selected classrooms and their students will be assigned at random to these 10 slots over the week to create a representative sample of the students and the meals they receive in school over the target week.

3 Data from the 2010-2013 FFVP evaluation provides an estimate of the standard deviation in FV consumption of 0.66; since the results are similar to SNDA (i.e., 0.56), we use the value of 0.60 for our calculations in Exhibit B.3.

4 In the prior FFVP evaluation, the study found that the program snack was served about 3.7 of the 5 days of the week. So of the 3,000 single day dietary recalls, we expected 3.7/5 or 74 percent of the them to be conducted on program days (0.74 times 3000 is equal to 2,200).