NASS Comments

Appendix J.

Comments and FNS Responses

NASS Review of OMB Documents on SNAP-ED Evaluation Study

I. Study Purpose

The purpose of this study is clearly explained and all terminology and project descriptions are accurately defined.

II. Sample Selection

Overall, the sample selection methodology used is appropriate for two of the four projects that are being evaluated. The CNNS and University of Nevada appear to use sufficient randomization in the sample selection and the sample should be representative of the target populations.

My only concern with the NYSDOH sample design is that the researchers should verify the distribution of the 156 CACFP centers between the two defined strata (NYC boroughs and the remainder of the state of NY). Your proposal indicates that you will sample six centers from each stratum. This procedure is fine assuming that your population of the 156 CACFP centers is equally distributed between NYC boroughs and the rest of the state (i.e. there exists approximately 78 centers in each strata). If for example, 75% of these centers are located outside of the NYC boroughs and the remaining 25% fall within the boroughs, then I would recommend sampling 9 centers from outside NYC boroughs and 3 centers from within the boroughs. This will reflect the 75/25 percent population distribution in your sample. This will ensure that both strata populations are accurately represented in your study. I would recommend investigating how many centers fall within each stratum and have your sample follow a similar distribution.

FNS Response: FNS has adopted the selection of pairs of centers proportional to the size of the stratum from which they are drawn. As a consequence, 5 pairs will be drawn from the upstate New York area and 7 pairs will be selected from New York City.

My concern with the PSU sample design is that participation in the study is limited to only English literate women and women with internet access. This has the potential to bias your results if non-English literate women or women without internet access make up a significant portion of the SNAP eligible women in the targeted counties. Further analysis to verify that the target population of women, ages 18 to 45, and residing in one of the 34 selected counties does not have a significant number of non-English literate women or women without internet access would be beneficial to ensure an unbiased study.

FNS Response: Because Pennsylvania State University’s target population is English speaking, literate women with internet access, we do not believe that a bias is introduced if we limit our inferences to this population. We will not generalize to non-English speaking, illiterate women without internet access.

That said, English is the preferred language of 98.5% of SNAP eligible household in selected un-served and under-served PA counties. Also, based upon formative research conducted by principal investigator, Dr. Barbara Lohse, 80% of low-income people interviewed had access to the internet at home or another location (e.g., school, work or a library).

III. Data Collection Instruments

Concerning the data collection instruments for impact evaluations, I would recommend a slight revision of the questions that currently ask in a yes/no/maybe format for information on whether or not a child ate or drank certain foods during the prior week. Since the current responses for these questions have the following response options,

1. No

2. Yes, sometimes

3. Yes, often

4. Yes, everyday

I feel the responses need to be better defined. Two respondents may interpret “sometimes” and “often” differently causing a bias in your results. I would recommend changing the question wording to ask for “How many days in the past week did your child….. “ and then the responses could be as follows…

1. None

2. 1-3 days

3. 4-6 days

4. Everyday

The number of days for responses 2 and 3 can be modified from this example to what the researchers feel is a good breaking point for differentiating between “sometimes” and “often”.

FNS Response: After conducting additional cognitive pretests, the NASS suggestion has been adopted and the instruments changed accordingly.

IV. Data Collection Procedures

Data collection procedures are defined clearly for each project. Since most of the projects include both mail and phone follow ups, it will be critical to have an efficient system in place (if one is not already developed) to monitor the receipt of mail returns prior to contacting by phone to avoid unnecessary respondent burden if a respondent has already submitted the survey by mail.

FNS Response: As a matter of standard operating procedure, the Research Triangle Institute has a sophisticated receipt control system in place.

Also, in the impact questionnaire, there are a couple of references that a picture will be included displaying a serving size or a cup of vegetables. If these visual aides are critical for the respondent to see to accurately answer the question, consider the possible negative impact of a respondent not being able to view these pictures during a phone follow up interview. I’m not sure how critical the pictures are to accurately answer the questions, but just something I wanted to include here.

FNS Response: Cognitive pretesting has indicated that the pictures are helpful to respondents. It is true, but also unavoidable, the secondary/follow-up data collection modes, such as phone follow-up, will not benefit from the pictures. However, the mode of data collection will be included as a controlling covariate.

V. Statistical Methods

The statistical methods section is complete and thorough. The algorithm used to match similar schools in the CNNS project is a good approach. Also, the statistical methods proposed for each of the four projects all seem appropriate.

VII. Overall Project Design

The supporting documents for this study are very well written. All aspects of the study have been covered and appear to have a solid plan in place. From a statistical point of view, if the items noted above concerning the sample design and questionnaire are further analyzed and corrected where necessary, this should be a successful study.

0