NASS Comments

FNS – SNAP ED Wave II (OMB #0584 – 0554) Comments – Jan 2011

This is a review of survey details contained in OMB docket for proposed study for FNS.

RTI response to comments provided in red.

Sample Design-INN-

The population of interest is parents and caregivers of third grade students attending eligible schools in four Iowa school districts.

To clarify, the population of interest is third grade students attending eligible schools in four Iowa school districts. Data on this population will be gathered through surveys of children’s parents and/or guardians about their dietary behavior in the family home. We revised the text in Part B and Appendix E to reflect this clarification.

Districts involved are purposively selected based on on-going relationship with INN researchers. Schools have not yet been selected. Schools considered for possible selection for the study must meet certain specified eligibility requirements. Selected schools will be assigned to a treatment group based on the districts in which they reside, so there is no random component in the assignment of schools to treatments. This could be considered a weakness in the design, but might be due to practical considerations. No blocking factor is introduced via the design making it more difficult to separate treatment effects from district effects, if they exist.

The use of a county-wide intervention (e.g., the social marketing program) limits the opportunity to randomize schools within county while maintaining a design in which children and parents from control/comparison schools would not be influenced by the intervention. We revised the text in Appendix E to discuss the need for a quasi-experimental design, rather than a fully randomized design.

District effects on the outcome should be reduced somewhat by special efforts made in the school selection phase. Stratification and school matching analysis applied to available administrative data will be used to balance the samples across the two treatment groups and the control group so that the selected schools in the three groups are as similar as possible with respect to these data. Once selected for inclusion in the study, pre-intervention data will be obtained and analyzed to determine if significant differences are apparent across the three school groups with respect to key study variables. Variables for which the pre-intervention data show significant difference among the three school groups will be used to compensate for these differences in the outcome analysis phase.

Sample Design- UKCES-

The population of interest is parents and caregivers of first through third grade children in eligible elementary schools in two counties in Kentucky.

To clarify, the population of interest is first through third grade students attending eligible schools in two Kentucky school districts. Data on this population will be gathered through surveys of children’s parents and/or guardians about their dietary behavior in the family home. We revised the text in Part B and Appendix E to reflect this clarification.

Elementary schools in two counties were eligible for consideration to participate in the study. Basis for county selection was not specified. Schools in the two selected counties having fewer than 40 students in grades 1 through 3 were eliminated from participation consideration due to per-school sample size requirements. It appears that all remaining elementary schools in the two counties participated in the study and no sampling of schools was conducted, although it is not completely clear to this reviewer.

The reviewer’s interpretation is correct. After removing schools with fewer than 40 students, all remaining schools were included in a simple random selection process. Random selection was conducted for each county separately with the aim of including a similar number of schools from each county. We revised the text in Appendix E to provide additional clarification on this point.

Within each county, schools were matched using an algorithm involving the percentage of the schools’ students that received subsidized lunches and school population size. Each school in the county was paired with another school in the county that was most similar to it, based on the algorithm. One school in each pair was randomly assigned to the control group, and the other was then assigned to the single treatment group. This permits the use of a blocking factor that can separate outcome effects that are simply attributable to differences in county characteristics from those caused by the treatment.

Sample Design- MSUE –

Study population is adults aged 60 years and older who attend one of 29 senior centers throughout the state of Michigan.

The study excludes very small centers (less than 30 residents) and centers that provide two or more meals per day to its residents. Centers were placed in strata based on geographical location and further stratified based on number of meals provided by the center. Very large centers were placed in a large-center stratum that was further stratified by the number of meals served. Random assignment to either the control or the treatment group was made within the strata. The sample design does a good job of distributing size, geographic location and number of meals served per week uniformly between treatment and control.

The sample design is a stratified cluster design and there is no mention of any sub-

sampling of individual seniors within the cluster (senior-center).

We do plan to conduct subsampling of individual seniors within centers. We revised Appendix E to reflect the fact that sub-sampling will occur within larger centers (those serving 100+ seniors). Sub-sampling will help maintain balance and prevent larger centers from exerting undue influence over the impact analyses.

Sample Size Determination-INN and UKCES

Standard procedures are used based on estimates of standard deviations obtained from a previous study. Intra-cluster correlation coefficients observed in similar studies are used as a proxy for the current study and factored in to sample size calculations. A detection level of a .30 standard deviation change with Type I and Type II error rates of .05 and .20 respectively for a two-tailed test are specified. This reviewer questions the necessity of a two-tailed test as it seems reasonable that the effect of the intervention, if it exists, would only be positive. A one tailed test would yield greater power for detecting a strictly positive shift, resulting in a smaller required sample size. The researchers are probably being statistically conservative in this respect, and there might be other concerns known to the researchers that require such an approach.

It is common among health prevention programs to apply a two-tailed test to assess intervention impacts. While a one-tailed test would yield greater power, we must consider that secular phenomena (e.g., extra-programmatic influences) could lead to reduction in children’s consumption of healthy foods. We added a footnote to Appendix E to explain the use of a two-tailed test.

One note concerning the UKCES sample design sample size determination. This study involves multiple grade levels within the same school, and therefore family cluster effects might exist. The intra-cluster correlation at the family level for families with more than one child in the first through third grade levels might be rather high, requiring an increase in sample size to achieve stated precision goals. This is not much of a concern with the INN study as it only involves the third grade and the number of families with multiple children in the same grade level would be small.

We have considered the family cluster problem and plan to include only one child per family. This will be done post-hoc through a process that randomly selects one child from the multi-child family to serve as the index child for the purpose of analyses. We added text to Appendix E to explain the approach we will use to address this concern

Sample Size Determination-MSUE.

The same basic approach is used as is used in the INN and UKES studies. Roughly the same proxy intra-cluster correlation was used. This reviewer thinks that using the same value of intra-cluster correlation for the MSUE study might result in it being underestimated, since the individuals that make up the cluster (senior center) reside at the same location and individuals might be more apt to affect one another’s attitude toward the study. This could result in sample sizes being smaller than necessary to achieve the levels of precision desired. This is a caution and not a recommendation.

Our exclusion of assisted living facilities includes all housing centers. Our description in Part B and Appendix E has been revised so that this point stands out more clearly. Accordingly, there are no residential centers within our sampling frame and we believe the proxy ICC is an acceptable estimate.

Compensating for Attrition-INN, UKCES, MSUE:

Attrition is defined as parents/caregivers, or senior center residents (as the case may be) failing to report post-intervention data after having reported pre-intervention data. It is quite possible that drop-outs are dissimilar to completers with respect to key study variables. It is important to ascertain whether evidence of possible differences exist between those that drop out of the study and those that complete the study.

Attrition bias will be addressed by analyzing the pre-intervention data and then looking for key study variables that are good at differentiating between completers and drop-outs.

If significant differences are observed in these predictors between completers and drop-outs, these variables can be used to adjust out some of the bias that could otherwise distort the results.

Analysis-INN, UKCES, MSUE

Explicit details of the proposed analysis is not given, rather general descriptions of important factors that will be considered in model development are mentioned. Generalized-linear and mixed-effects models will be used to analyze the results. These models are flexible and the researchers appear sensitive to including important variables in the models.

Other Comments:

INN and UKCES-

One area of concern this reviewer has is that one of the main treatments of interest (i.e. classroom education) for these two studies is applied to the students (although the students are given some education materials to take home for the parents/caregivers), the effect of the treatment seems largely to be measured by the change in the behavior of the parent/caregiver. This could be a little problematic, assuming that control over what the children eat at meal-times is likely to be in the control of the parent/caregiver. The INN study does include a community-wide social marketing campaign that could directly affect the attitude of the parents and mitigate this concern somewhat. The questionnaires were not reviewed by this reviewer in any detail but it was apparent that some of the data that will be collected is related to the children’s attitude with respect to fruits and vegetables in their diet—as reported by the parent/caregiver. These data would be important to understand the effect of the treatment on the children.

As previously stated, our descriptions of the target populations have been revised to emphasize the fact that our primary interested is in changes in children’s dietary behavior, not parent’s dietary behavior. We are collecting data from parents – and perhaps this is where the confusion arose – to carry out an independent evaluation that does not introduce reactivity or unnecessary burden through surveying the same individuals twice (i.e., by the implementing agency and FNS through the independent evaluation).

Matt Fetter,

Mathematical Statistician

USDA/NASS

January 4, 2011