Attachment G - Ssummary of study findings for OMB

Summary of Studies Exploring Potential Technical Enhancements to the Methods USDA Uses to Measure Household Food Security in the United States

Mark Nord

April 19, 2012

This document summarizes research by USDA’s Economic Research Service (ERS) exploring five issues identified by the Committee on National Statistics (CNSTAT) panel that reviewed the methods USDA uses to measure household food security in the United States. This summary is provided in response to OMB’s request in the authorization of the Current Population Survey Food Security Supplement for 2009-2012 (OMB Control No. 0536-0043):

“Within 30 days of completion of the peer review, ERS will present to OMB the results of the ERS-[Iowa State University] cooperative agreement work that explores five different alternatives to the current food insecurity classification scheme. OMB will schedule a subsequent meeting to discuss ERS’ findings as to the implications of these results for whether, and if so how, the survey should be changed or the data analyzed differently.”

Background:

In 2003-06 an expert panel convened by the Committee on National Statistics (CNSTAT) of the National Academies conducted a thorough review of the food security measurement methods. USDA requested the review by CNSTAT to ensure that the measurement methods USDA uses to assess households’ access—and lack of access—to adequate food and the language used to describe those conditions are conceptually and operationally sound and that they convey useful and relevant information to policy officials and the public. The panel, which included economists, sociologists, nutritionists, statisticians, and other researchers, recommended that USDA continue to measure and monitor food insecurity regularly in a household survey, affirmed the appropriateness of the general methodology currently used by USDA, and suggested several ways in which the methodology might be refined (National Research Council, 2006).

USDA made several of the suggested changes that did not require confirmatory research:

• In 2006 (the report on the 2005 data), USDA began using new labels to describe ranges of severity of food insecurity, removing the word “hunger” from the label for the more severe range.

• In the December 2006 survey, USDA changed the wording of the resource-constraint (e.g., “because there wasn’t enough money for food”) of several of the food security questions to standardize the wording across questions.

• In the December 2007 survey, USDA re-ordered the food security questions so that all the child-referenced questions are grouped together following the adult-referenced questions.

• ERS commissioned two studies of the duration or recurrence of food insecurity over five or more years (Wilde et al., 2011; Ryu and Bartfeld, 2011).

The CNSTAT panel also recommended that USDA consider several technical enhancements to the statistical methodology used to assess items and scales and to classify households as to food security status. ERS studied five of the most salient potential technical enhancements recommended by the Panel:

1. Incorporate frequency-of-occurrence information of all items for which it is available into the main measure using polytomous IRT models rather than the current dichotomous model.

2. Treat items with frequency follow-up questions appropriately, for example, as a single ordered polytomous item rather than as two independent questions.

3. Allow item discrimination parameters to differ from item to item when indicated by the relevant data.

4. Fit models that allow for different item parameters for households with and without children…in order to study the possibility and effects of differential item functioning.

5. Develop a new classification system that reflects the measurement error inherent in latent variable models. This can be accomplished by classifying households probabilistically along the latent scale.

ERS conducted the initial research and produced a research paper on each topic. Dr. Amy Froelich, a statistician at the Iowa State University who specializes in the relevant Item Response Theory (IRT) statistical methods, reviewed the five research papers under a cooperative research agreement funded by ERS. FED conducted further research and revised the papers based on Dr. Froelich’s recommendations. The revised papers were then reviewed by Dr. Matthew Johnson, a nationally recognized expert on these measurement methods at the Columbia University Teachers College. At the recommendation of Drs. Johnson and Froelich, ERS undertook a sixth study to examine effects of three of the potential enhancements in combination.

ERS plans to publish the compiled papers after one further round of peer review, and revisions if needed, to meet ERS standards of scientific quality and independent review. The working title of the report, which will be published on the ERS web site, is Assessing Potential Technical Enhancements to the U.S. Household Food Security Measures. The prepublication draft of the report is available on request from ERS; contact Mark Nord ([email protected] 202-694-5433).

Results of the six studies are summarized as follows:

1. It may not be appropriate to incorporate frequency-of-occurrence information of all items for which it is available into the main measure using polytomous IRT models. Frequent or persistent food insecurity appears to represent a somewhat distinct dimension from the greatest severity experienced at any time during the year. Furthermore, a polytomous scale would provide small to minimal gains in measurement precision as evidenced by its failure to reduce estimation errors on prevalence rates of food insecurity and very low food security. An alternative way to incorporate information on frequency and duration of food insecurity into the measurement process may be to report the prevalence of frequent/persistent food insecurity and very low food security based on a separate scale comprising the standard food security questions recoded to reflect only the most frequent occurrence of each condition or behavior.

2. Frequency-of-occurrence follow-up questions that are included in the measure should be modeled along with their base items as ordered polytomous items rather than as two independent questions. ERS has adopted this methodology as recommended by the CNSTAT panel. When modeled as trichotomies, the three items in the current measures that have such follow-ups fit the Rasch measurement model well enough that they do not substantially distort measurement in the adult scale and child scale. In the household food security scale that combines adult and child items, the trichotomous items do not fit well and introduce slight distortions into the measure.

3. Allowing item discrimination parameters to differ from item to item would improve measurement precision only slightly and would make prevalence statistics less transparent than those based on the single-parameter model. Although a two-parameter logistic (2PL) model may contribute to item and scale assessment, a measure based on the 2PL model would provide negligible improvement in the applications of most importance and visibility in the Federal Government’s food security monitoring system—the calculation of USDA’s published food security prevalence statistics. A measure based on a 2PL model would be a weighted combination of responses and would be much more difficult to explain to the policy officials, the media, and the public.

4. The extent of differential item function (DIF) between households with and without children is not great enough to substantially distort comparisons of prevalence. Differential item function (DIF) between households with and without children was found to be substantial for only one item, “We couldn’t afford to eat balanced meals.” Some DIF also exists between Hispanic and non-Hispanic households. However, the practical implications of the extent of DIF that exists are modest, and correcting for it would not change assessments of inter-group differences in any important ways.

5. A classification system that reflects the measurement error inherent in latent variable models by classifying households probabilistically would reduce the extent of error in prevalence estimates, but would not change patterns of prevalence across subpopulations or over time. With the important exception of the difference between households with and without children (which may be addressed in other ways), probabilistic assignment of food security status would have minimal effects on comparisons of prevalence rates of food insecurity over time or across subpopulations. However, prevalence rates based on probabilistic assignment would very likely have smaller standard errors than those based on the current method.

6. Assessment of three potential technical enhancements in combination (the polytomous model, the 2PL model, and probabilistic assignment of food security status) did not substantially alter conclusions from the separate assessments of each potential enhancement (as described above). Gains in measurement reliability would be almost completely a result of the probabilistic assignment of food security status. Adding the other two potential enhancements would neither improve measurement reliability nor change prevalence rates of subpopulations of primary policy interest to any substantively important extent.

Literature Cited

National Research Council. Food Insecurity and Hunger in the United States: An Assessment of the Measure. Committee on National Statistics, Panel to Review the U.S. Department of Agriculture’s Measurement of Food Insecurity and Hunger, Gooloo S. Wunderlich and Janet L. Norwood (eds.). Washington, DC: The National Academies Press, 2006.

Ryu, Jeong-Hee, and Judi Bartfeld, “Long Term Food Security Patterns and Children’s Cognitive and Health Outcomes,” final report on ERS-funded cooperative research project, Washington, DC: Economic Research Service, U.S. Department of Agriculture, 2011.

Wilde, Parke E., Mark Nord, and Robert E. Zager, “In longitudinal data from the survey of Program Dynamics, 16.9% of the U.S. population was exposed to household food insecurity in a five-year period,” Journal of Hunger and Environmental Nutrition 5(3): 380-398, 2010.

3