Attachment K FCSM Presentation

Attachment K
2023 Federal Committee on Statistical Methodology Research and
Policy Conference Presentation “Addressing Nonresponse Bias in
Food Security Measures Using Weighting Adjustments”

Addressing Nonresponse Bias in Food Security Measures Using
Weighting Adjustments
Jonathan Eggleston (U.S. Census Bureau), Matthew P. Rabbitt
(Economic Research Service), David C. Ribar (Georgia State University),
Alisha Coleman-Jensen (Economic Research Service)
FCSM, Session B-2
Thursday, October 26, 2023
Any opinions and conclusions expressed herein are those of the authors and do not represent the views of the U.S. Census Bureau. The Census
Bureau has ensured appropriate access and use of confidential data and has reviewed these results for disclosure avoidance protection (Project
7531994: CBDRB-FY23-0524).

1

Survey Non-Response
• Response rates for the CPS and other
surveys plummeted during COVID-19
and have generally fallen
• CPS module response rates have also
decreased
• Non-response may lead to nonrepresentative samples and affect
estimates of food insecurity and
other outcomes
• Data available within the CPS to
adjust for non-response are limited
2

60.0
Jan-21

Sep-20

May-20

Jan-20

Sep-19

May-19

Jan-19

Sep-18

May-18

Jan-18

Sep-17

May-17

Jan-17

Sep-16

May-16

Jan-16

Sep-15

May-15

Jan-15

Sep-14

May-14

Jan-14

Sep-13

May-13

Jan-13

Sep-12

May-12

Jan-12

Sep-11

May-11

Jan-11

CPS Response Rates

100.0
95.0

90.0

85.0

80.0

75.0

70.0

65.0

Source: BLS
3

CPS Food Security Supplement
• Focus on CPS Food Security
Supplement
• Sponsored by the Economic
Research Service-USDA and fielded
by the Census Bureau as an annual
supplement to the December CPS
• Source for federal statistics on
household food security in U.S.
• Food security: access at all times to
enough food for an active, healthy
life for all household members
4

Non-response in the Food Security Supplement
•

Recent research on the CPS Annual Social and Economic
Supplement finds lower income people are less likely to respond
(Rothbaum & Bee, 2020)

•

Differential non-response affects estimates of poverty rates and
other income distribution statistics

•

Not fully addressed by standard weighting procedures

•

Because of the relationship between income and food security,

•
•

Differential non-response likely occurs in the Food Security Supplement
And may affect estimates of food security
5

This paper
• Applies the methodology developed in research on the basic monthly
CPS to improve the weighting correction for non-response
• Utilizes data from multiple administrative data sources including IRS
1040 and 1099 data, SSA benefit data, earnings data from the
Longitudinal Employer-Household Dynamics, state public assistance
records, and other sources
• Develops and calibrates new weights from these sources
• Applies the weights and compares estimates based on the standard
weighting methodology and this new methodology
6

Weighting Overview

7

What Do Survey Weights Do?
•

Simple overview: weights increase or decrease the “importance” of
individual respondents to make the responding sample look more
like the target population

•

For example, if older individuals are more likely to respond to a
survey than younger individuals, we would give older individuals
lower weight values and younger people higher weight values to
make the sample (hopefully!) more representative

8

Overview of CPS-FSS’s Weighting Algorithm
1. Household Noninterview Adjustment
•

Using microdata on both respondent and nonrespondent households,
distribute the weights of nonrespondents to the respondent households with
similar characteristics

2. Second Stage Adjustment
•

Adjust CPS-FSS weighted counts of age, race, Hispanic origin, and sex to
independent Census population estimates at national and state levels

•

Also adjust CPS-FSS metro status and income statistics to same measures
calculated on the Basic Monthly CPS

9

Overview of CPS-FSS’s Weighting Algorithm
• Current Household Noninterview Adjustment
• Adjustment based only on geography (state and metropolitan
status)
• Concern: this may not fully account for economic characteristics
that influence response

• Calibration Step
• While had adjustment for income, target based on CPS basic
respondents, which may be nonrepresentative
10

Administrative Data
•

•

Add the following administrative data to CPS-FSS’s weighting algorithm
•

IRS 1040 and 1099

•

SSA program benefit data

•

Demographic data from 2010 Census and SSA

•

Industry data from the Census Business Register

•

Third-party home value data

•

Quarterly earnings data from the Longitudinal Employer-Household Dynamics (LEHD)
program

•

State SNAP/TANF/WIC Data

Have these data not only for many respondents, but also for CPS-FSS and CPS
Basic nonrespondents
11

Modifications to Weighting Algorithm
• Replace geography-based noninterview adjustment with one based on IRS
microdata and other administrative data
• Create new cells with CART (Classification and Regression Tree)
• Run model of household food security on administrative data in order to create cells that
have a good correlation with our key outcome of interest
• Estimate model on respondents. Apply model output to both respondents and
nonrespondents to create noninterview adjustment cells.

• Add administrative data to calibration step as well
• Use same CART model. Target is predicted probabilities of food security status,
where the administrative data are the inputs for creating the predicted probabilities

12

Strength and Weaknesses of Administrative
Data
• Administrative data includes information that should be highly related to food
security
• IRS Income
• SNAP and WIC receipt

• Nevertheless, we don’t actually observe food security in the administrative data
• If there are additional factors correlated with both food security and response
even after controlling for these observables, some nonresponse bias will remain
• E.g. Don’t observe expenditures. Differences in expenditures decisions for a given level
of income and SNAP benefit amounts could affect food security, but could also be
correlated with behavioral differences that influence whether someone responds to a
survey
13

Results-Household Food Insecurity

14

Change in Food
Percent Food Insecure
(Low or Very Low Insecurity Estimates
with Using
Security) Production
Estimate Administrative Data

All households
With children < 18 years old
____With children < 6 years old
____Married-couple families
____Female head, no spouse
____Male head, no spouse
With no children < 18 years
____More than one adult
____Women living alone
____Men living alone
With elderly
____Elderly living alone

2019
10.54%
13.65%
14.47%
7.54%
28.73%
15.37%
9.27%
6.72%
13.05%
12.84%
7.22%
8.70%

Source: 2019 and 2020 CPS-FSS + Administrative Data

2019
0.20%
0.41%
0.20%
0.17%
0.95%
-0.42%
0.10%
0.13%
0.30%
-0.37%
0.24%
0.32%

2020
0.36%
0.49%
0.31%
0.43%
0.28%
0.29%
0.31%
0.24%
0.40%
0.29%
0.36%
0.46%
15

Change in Food
Percent Food Insecure
(Low or Very Low Insecurity Estimates
with Using
Security) Production
Estimate Administrative Data
White, non-Hispanic
Black, non-Hispanic
Hispanic
Other, non-Hispanic
Under 1.00 Poverty Line
Under 1.30 Poverty Line
Under 1.85 Poverty Line
1.85 and over Poverty Line
Income unknown
Northeast
Midwest
South
West

2019
7.93%
19.07%
15.63%
9.47%
34.86%
33.02%
27.65%
5.08%
8.38%
9.60%
10.53%
11.19%
10.16%

Source: 2019 and 2020 CPS-FSS + Administrative Data

2019
0.22%
0.25%
0.17%
-0.01%
-0.28%
-0.31%
-0.33%
0.04%
-0.08%
0.30%
0.17%
0.25%
0.06%

2020
0.32%
0.46%
0.45%
0.21%
0.19%
-0.01%
-0.05%
0.13%
0.32%
0.32%
0.39%
0.33%
0.42%
16

Results-Child Food Insecurity

17

Source: 2019 and 2020
CPS-FSS + Administrative
Data
All households
____With children < 6 years old
____Married-couple families
____Female head, no spouse
____Male head, no spouse
White, non-Hispanic
Black, non-Hispanic
Hispanic
Other, non-Hispanic
Under 1.00
Under 1.30
Under 1.85
1.85 and over
Income unknown
Northeast
Midwest
South
West

Change in Food
Percent Food Insecure
Insecurity Estimates with
(Low or Very Low Security)
Using Administrative
Production Estimate
Data
2019
7.04%
6.65%
3.40%
16.89%
7.02%
5.25%
13.54%
8.28%
5.35%
20.91%
19.88%
17.23%
2.36%
5.53%
7.35%
7.38%
6.77%
6.96%

2019
0.36%
0.42%
0.12%
1.01%
-0.46%
0.18%
0.98%
0.49%
0.22%
0.61%
0.33%
0.13%
0.09%
0.22%
0.38%
0.10%
0.48%
0.36%

2020
0.12%
-0.09%
0.08%
0.03%
-0.09%
0.14%
0.19%
-0.03%
0.01%
-0.32%
-0.18%
-0.14%
0.17%
0.31%
-0.01%
-0.08%
0.15%
0.32%

18

Conclusion
•

Adding administrative data to CPS-FSS’s weighting algorithm results
in a modest change in food security estimates
•

Shifts largely from change in estimates of the income distribution

•

Change in estimates larger in 2020 compared to 2019 for household food
security

•

Pattern reversed/not as consistent for child food security

19

My Contact Information
Jonathan Eggleston
Senior Economist
Survey Improvement Technical Lead
Survey and Economic Research Group
Center for Economic Studies
U.S. Census Bureau
Office: 301.763.2357
[email protected]
20