Attachment G: NASS Technical Comments

Attachment G – NASS Technical Comments

NASS Review of OMB Control Number 0584- NEW
Review of Child Nutrition Data & Analysis for Program Management
July 8, 2016
The purpose and eight research objectives of this study are clearly explained in the
“USDA_Review of Child Nutrition Part A”.
The description in this paper and web interview scripts are very clear and follow
Dillman’s principle. For Example:
1. Construction of respondent-friendly web questionnaires. These Web surveys
include clear descriptions of computer actions at point of use, and also include
the option to skip questions. By respondent-friendly, you may reduce the
occurrence of sample survey errors through improvement of the motivational
aspects of responding, as well as the technical user-interface between computer
and respondent.
2. Use of lightly shaded color as background fields on which to write all questions
provides an effective navigational guide to respondents. When shaded
backgrounds fields are used, identification of all answer spaces in white helps to
reduce non‐response.
3. Placement of the instructions exactly where that information is needed and not at
the beginning of the questionnaire; placement of special instructions inside of
question numbers and not as freestanding entities, etc.

The survey methodology is correct:
1. A census will be used for State agencies and the District of Columbia because
their MIS are unique and would not be properly represented through a sample.
2. A proportional stratified random sampling design will be used to select a
nationally representative sample of SFAs. The sample will be representative in
terms of the seven FNS regions and three size of SFA. Proportionate

stratification provides equal or better precision than a simple random sample of
the same size, and gains in precision accrue to all survey measures.
3. The levels of precision is at a high standard (All public SFAs: ±5.0% at the 95%
level of confidence; and Sub-groups: between ±7.0% and 10.0% at the 95% level
of confidence).
4. The statistical analysis methods are appropriate.
a. Using descriptive statistics (such as mean, median, and standard
deviation) for continuous variable (e.g., cost of developing an electric
MIS);
b. Testing the difference between subgroups will employ Chi-square (Χ2)
tests for categorical variables, and
c. Using one-way ANOVA tests for continuous variables.

5. The methods to adjust for non-respondent bias is correct. Propensity Modeling
is an increasingly popular method for adjusting for nonresponse; that is, creating
a logistic regression model that predicts the likelihood of response versus nonresponse. This model makes use of any and all available and relevant data.
After finding out the final set of covariates for the non-response model (i.e., the
variables that are strong predictors of non-response), the model is then applied
to the responders, and a log probability of responding is generated for each case.
The weighting adjustment factor is then calculated as the inverse of this
probability. The adjusted sampling weight will be assigned to each responding
SFA in data analyses to make the sample representative of all SFAs in the seven
FNS regions.
6. The way to maximize response rates is thoughtful, such as effective recruitment,
a detailed followed–up plan and an effective Web survey design.

SUGGESTIONS

1. Consider adding the appropriate question numbers in Attachment B.12 – Web
State Survey (Web Screenshots), so that they match the question numbers in
Attachment B.10 – Paper State Survey.
2. Consider adding the question numbers in Attachment B.13 – Web School Food
Authority Survey (Web Screenshots), so that they match the question numbers in
Attachment B.11 – Paper School Food authority Survey.
3. Consider changing the format for choosing items and answers for question 7 in
attachment B.11 (page 6 in paper version):
o It may be better to use the format below, Figures 1 and 2, so respondents
with multiple modules can select more than one choice.
4. The MIS could have two possible outcomes that is equal to 1 if the SFA uses an
MIS for any of its functions and zero otherwise. It is correct to use logistic
regression for data analysis when MIS is used as the dependent variable.
o Please be aware of that the proportion odds assumption is a special case
of the parallel lines assumption when LINK=LOGIT. If the score chi-square
for testing the proportional odds assumption is highly significant, this
indicates that the cumulative logit model might not adequately fit the data.
If it is the case, then an alternative model to use may be the generalized
logit model with the LINK=GLOGIT option.
o When the data sets are too small or when the event occurs very infrequently
or when some of the cells formed by the outcome and categorical predictor
variable have no observations, the maximum likelihood method may not work
or may not provide reliable estimates. Exact logistic regression provides a way
to get out these difficulties. What it does is to enumerate the exact distributions
of the parameters of interest, conditional on the remaining parameters.
o In linear regression, the significance of a regression coefficient is assessed by
computing a t test. In logistic regression, there are several different tests
designed to assess the significance of an individual predictor, most notably the
likelihood ratio test and the Wald statistic.

Figure 1

Initial
Development
and
Implementation
Costs

Upgrade Costs (If
non-routine
upgrade[s]
occurred after
initial
implementation)

Applications Scanner

Online Application

Eligibility Determination

 No direct cos t to s tate
s chool nutrition agency
(upgrades are m ade by
s tate IT s taff)

 No direct cos t to s tate
s chool nutrition agency
(upgrades are m ade by
s tate IT s taff)

 No direct cos t to s tate
s chool nutrition agency
(upgrades are m ade by
s tate IT s taff)

 Les s than $50,000

 Les s than $50,000

 Les s than $50,000

 $50,000 - $99,000

 $50,000 - $99,000

 $50,000 - $99,000

 $100,000 - $299,999

 $100,000 - $299,999

 $100,000 - $299,999

 $300,000 - $499,999

 $300,000 - $499,999

 $300,000 - $499,999

 $500,000 - $699,999

 $500,000 - $699,999

 $500,000 - $699,999

 $700,000 - $999,999

 $700,000 - $999,999

 $700,000 - $999,999

 $1,000,000 -$4,999,999

 $1,000,000 -$4,999,999

 $1,000,000 -$4,999,999

 $5,000,000 or m ore

 $5,000,000 or m ore

 $5,000,000 or m ore

 No upgrades purchas ed

 No upgrades purchas ed

 No upgrades purchas ed

 No direct cos t to SFA
(upgrades are m ade by
SFA IT s taff)

 No direct cos t to SFA
(upgrades are m ade by
SFA IT s taff)

 No direct cos t to SFA
(upgrades are m ade by
SFA IT s taff)

 Les s than $50,000

 Les s than $50,000

 Les s than $50,000

 $50,000 - $99,000

 $50,000 - $99,000

 $50,000 - $99,000

 $100,000 - $299,999

 $100,000 - $299,999

 $100,000 - $299,999

 $300,000 - $499,999

 $300,000 - $499,999

 $300,000 - $499,999

 $500,000 - $699,999

 $500,000 - $699,999

 $500,000 - $699,999

 $700,000 - $999,999

 $700,000 - $999,999

 $700,000 - $999,999

 $1,000,000 -$4,999,999

 $1,000,000 -$4,999,999

 $1,000,000 -$4,999,999

 $5,000,000 or m ore

 $5,000,000 or m ore

 $5,000,000 or m ore

 Don’t know

 Don’t know

 Don’t know

Figure 2

Initial
Development
and
Implementation
Costs

Upgrade Costs (If
non-routine
upgrade[s]
occurred after
initial
implementation)

Applications Scanner

Online Application

Eligibility Determination

 No direct cos t to s tate
s chool nutrition agency
(upgrades are m ade by
s tate IT s taff)

 No direct cos t to s tate
s chool nutrition agency
(upgrades are m ade by
s tate IT s taff)

 No direct cos t to s tate
s chool nutrition agency
(upgrades are m ade by
s tate IT s taff)

 Les s than $50,000

 Les s than $50,000

 Les s than $50,000

 $50,000 - $99,000

 $50,000 - $99,000

 $50,000 - $99,000

 $100,000 - $299,999

 $100,000 - $299,999

 $100,000 - $299,999

 $300,000 - $499,999

 $300,000 - $499,999

 $300,000 - $499,999

 $500,000 - $699,999

 $500,000 - $699,999

 $500,000 - $699,999

 $700,000 - $999,999

 $700,000 - $999,999

 $700,000 - $999,999

 $1,000,000 -$4,999,999

 $1,000,000 -$4,999,999

 $1,000,000 -$4,999,999

 $5,000,000 or m ore

 $5,000,000 or m ore

 $5,000,000 or m ore

 No upgrades purchas ed

 No upgrades purchas ed

 No upgrades purchas ed

 No direct cos t to SFA
(upgrades are m ade by
SFA IT s taff)

 No direct cos t to SFA
(upgrades are m ade by
SFA IT s taff)

 No direct cos t to SFA
(upgrades are m ade by
SFA IT s taff)

 Les s than $50,000

 Les s than $50,000

 Les s than $50,000

 $50,000 - $99,000

 $50,000 - $99,000

 $50,000 - $99,000

 $100,000 - $299,999

 $100,000 - $299,999

 $100,000 - $299,999

 $300,000 - $499,999

 $300,000 - $499,999

 $300,000 - $499,999

 $500,000 - $699,999

 $500,000 - $699,999

 $500,000 - $699,999

 $700,000 - $999,999

 $700,000 - $999,999

 $700,000 - $999,999

 $1,000,000 -$4,999,999

 $1,000,000 -$4,999,999

 $1,000,000 -$4,999,999

 $5,000,000 or m ore

 $5,000,000 or m ore

 $5,000,000 or m ore

 Don’t know

 Don’t know

 Don’t know