BBBB_NASS commentsEdited

Appendix BBBB:
National Agricultural Statistics Service Comments

NASS Review of OMB Documents

FNS WIC Nutrition Study

I. Study Purpose and Objectives

The proposal clearly states the purpose of the study. The objectives define what the study is supposed to accomplish, and also establish benchmarks that can be used to carry out and evaluate the study.

II. Sample Design

The design of the samples fall into these categories:

• Phase I sample is designed appropriately to “provide a comprehensive nationally representative description of WIC nutrition education” and to provide a basis for Phase II sampling:

• Stage 1(Local Agencies): The sample design will accomplish the task of selecting the required number of local agencies for Phase 1. The use of a stratified sampling design is useful in separating the agencies by type and allowing for a more efficient design. The use of a probability proportional to size design in the stratum with “All Other Local Agencies” will allow the inclusion of the agencies with the greatest impact (because of size) in the sample. The selection of a reserve sample will allow the required sample size

to be achieved in case of nonresponse.

• Stage 2(Site): A systematic random sample will achieve the goal of having an adequate range of case loads (with geographic and nutrition education delivery mode diversity) in the sample from the agencies selected in stage 1.

• Site Sub-sample: In-depth telephone interviews with respondents to this sample will provide the information needed to describe the nutrition education process, and provide information for Phase II sampling.

• Phase II sampling objectives are stated to “to capitalize on the variability of WIC nutrition education to enable using a dose-response design. At the same time, we plan to employ a sampling procedure that would be scalable to a national study that yields nationally representative estimates”. The study will accomplish these objectives by

• Utilizing a process in site selection (from Phase I interview sites) that ensures diversity in geographic location, population characteristics, and type of nutrition education.

• Creating a multiple component index of dosage of nutrition education to use in ranking and selecting a subset of sites from the Phase I site interviews.

III. Data Collection Instruments

The data collection instruments are exhaustive and well designed. They should

achieve the data collection goals of the study.

IV. Data Collection Procedures

The use of web based surveys in Phase I – stage 1 is cost efficien,t since the respondents are supervisors/administrators and the response rate should be high for this population. Using telephone interviews for Phase I – stage 2 is appropriate in order to get the more detailed information that is needed.

Collection of the WIC participant data in three stages (base, interim and final) is appropriate for the estimation methods (such as difference-in-difference estimates) that will be used for the study.

The data collection plan for Phase II is thorough. It is well thought-out, and

should achieve the study objectives.

V. Statistical Methods

The statistical methods outlined in the proposal are suitable for the objectives of the study and for the structure/characteristics of the data:

• To accomplish study objectives:

• Segmentation of WIC participants into three sub populations will provide information on how programs affect each group.

• Analyses/comparisons of the data for the base, interim and final periods will allow researchers to assess the impact of the nutrition education programs over time. Difference-in-difference estimates will depict how recipients of higher dosage respond differently from recipients of lower dosage.

• Supplementing the quantitative data with qualitative data (focus groups, etc.), will provide an enhanced assessment of the education process and its impact.

• To evaluate/accommodate the data structure:

• Nonresponse bias analysis (Phase I) will determine the difference between respondents and nonrespondents. This will assist in the interpretation of the study results.

• Analysis of Participant surveys data over time will allow testing to determine if the data adheres to the model(s) assumptions. It will also assist in the selection of the appropriate estimation techniques, depending on the results of the tests.

• Attrition analysis to examine the differences between those who completed a stage of the study, and those that did not, will help to determine if the results of the study are valid for the members of the WIC population that are like those that did not complete the study.

• Mixed effects models is appropriate for the analyses given that the measurements on the participants will clearly be clustered:

Local agency → WIC site → WIC participant

The data will also be correlated because of repeated measurements on WIC participants. The mixed effects models allow the researchers to model the form of the correlation. These models are also appropriate because they allow for the real possibility of attrition of the WIC participants (given that the data is missing at random).

• Generalized linear mixed models and logistical mixed models give the researchers tools with a lot of flexibility for statistical estimation from complex data that may not follow the expected assumptions.

• The proposal specifies techniques appropriate for the estimation of standard errors from complex data.

VI. Overall Project Design

This study is well designed. The methods used should provide a good understanding of WIC nutrition education programs and their impact. The data collection procedure and the correspondence to the participants should be effective in obtaining the data required. The sampling methodology is sound and the statistical methods take into account the objectives of the project and the complexity of the data.