Attachment I Nonresponse Bias Report

Attachment I
U.S. Census Bureau Report “Evaluating Nonresponse Bias in the
2022 Food Security Supplement to the Current Population Survey”

October 18, 2023
MEMORANDUM FOR

Michelle Ver Ploeg
Chief, Food Assistance Branch
U.S. Department of Agriculture Economic Research Service

Through:

Kyra M. Linse
Survey Director, Current Population & American Time-Use Surveys
Associate Directorate for Demographic Programs

From:

Anthony G. Tersine, Jr.
Chief, Demographic Statistical Methods Division

Subject:

Evaluating Nonresponse Bias in the 2022 Food Security
Supplement to the Current Population Survey

The purpose of this memorandum is to report on analysis of various nonresponse estimates
computed for the 2022 Food Security Supplement to the Current Population Survey and to
provide nonresponse bias analysis tables for those estimates.
If you should have any questions about this document, please contact Emily Hood at 301-7630284 or [email protected] or David Hornick at 301-763-4183 or
[email protected].
The U.S. Census Bureau reviewed this data product for unauthorized disclosure of confidential
information and approved the disclosure avoidance practices applied to this release. CBDRBFY22-POP001-0145.
Attachment: Evaluating Nonresponse Bias in the 2022 Food Security Supplement to the Current
Population Survey
cc:
Alisha Coleman-Jensen
Tim Marshall
Michael Brennan
Greg Weyland
Lorelei De Vos
Sunhak Kim

(USDA ERS)
(ADDP)
(DSD)

David Hornick
Emily Hood
Rebecca Hoop
Jana Hatch
Weimin Zhang
Kiauna Womack
Tim Trudell

(DSMD)

DRB Clearance Number CBDRB-FY23-POP001-0162
census.gov

[This page is intentionally left blank]

DRB Clearance Number CBDRB-FY23-POP001-0162

Demographic Statistical Methods Division
Sample Design and Estimation

Evaluating Nonresponse Bias in the 2022 Food
Security Supplement to the Current Population
Survey
October 18, 2023

Rebecca Hoop
Weimin Zhang
James Farber, ADC
David Hornick, Lead Scientist
Emily Hood, Supervisor

DRB Clearance Number CBDRB-FY23-POP001-0162

Table of Contents
Executive Summary ...................................................................................................... 1
1. Introduction .......................................................................................................... 2

2.

1.1

Overview of the Current Population Survey .................................................................... 2

1.2

Overview of the 2022 Food Security Supplement to the Current Population Survey ...... 3

1.3

Discussion of Nonresponse in the 2022 Food Security Supplement to the Current
Population Survey............................................................................................................ 3

Methodology ........................................................................................................ 5
2.1

Data ................................................................................................................................. 5

2.2

Weights ........................................................................................................................... 5

2.3

Universe for the Estimates .............................................................................................. 6

3.

Limitations ............................................................................................................ 6

4.

Response Rates ..................................................................................................... 6

5.

Respondent Distributions .................................................................................... 133

6.

Conclusions......................................................................................................... 19

7.

References........................................................................................................ 200

List of Tables
Table 1: 2022 Food Security Unit Response Rates ........................................................................ 7
Table 2: Response Rates for December 2022 Current Population Survey Households ................. 8
Table 3: Response Rates for 2022 Food Security Supplement Households ................................. 10
Table 4: Response Rates for 2022 Food Security Supplement Households for Characteristics
Only Available for Responding Current Population Survey Households........................ 12
Table 5: Respondent and Nonrespondent Distributions for December 2022 Current Population
Survey Households...................................................................................................... 144
Table 6: Respondent and Nonrespondent Distributions for 2022 Food Security Supplement
Households ................................................................................................................. 166
Table 7: Respondent and Nonrespondent Distributions for 2022 Food Security Supplement
Households for Characteristics Only Available for Responding Current Population
Survey Households........................................................................................................ 18

DRB Clearance Number CBDRB-FY23-POP001-0162

Executive Summary
This report presents results of an analysis of various nonresponse estimates computed for the
2022 Food Security Supplement (FSS) to the Current Population Survey (CPS). The sample
included all households who completed a CPS interview. A nonresponse bias analysis was
conducted to determine whether nonresponse among different demographic groups may have
biased estimates. We investigated overall response rates, demographic subgroup response
rates, and demographic respondent and nonrespondent distributions.
Our key findings for the 2022 FSS are:
•

The CPS household weighted response rate was 70.91 percent. The FSS household
weighted response rate was 76.27 percent. Thus, the overall weighted response rate for
the FSS sample was 54.08 percent.

•

For the CPS household estimates, excluding blanks (no responses), there are significant
differences in the response rates for each of the variables that we investigated except
gender of reference person. Excluding the blanks and missing values, one of the largest
differences in response rates is seen for the age of reference person. Excluding blanks,
each of the variables we investigated, except type of living quarters, principal city status,
and urban/rural status, have significant differences in the respondent distributions. The
largest differences between respondent and nonrespondent distributions are within
race, gender, Hispanic origin, and age of reference person.

•

For the FSS household estimates, excluding blanks, there are significant differences in
the response rates and respondent distributions for each of the variables that we
investigated except for type of living quarters, principal city status, and urban/rural
status. Region had one of the largest differences in response rates. 1 The largest
difference between respondent and nonrespondent distributions is within age of
reference person.

•

For FSS household estimates for characteristics only available for CPS respondents,
there are significant differences in the response rates for each of the variables that we
investigated except family income and child(ren) present. There are significant
differences in the respondent distributions for each of the variables that we investigated
except child(ren) present. The largest difference in respondent and nonrespondent
distributions is seen within family income.

The largest difference in response rates for region is not significantly different than the largest difference in
response rates for race of reference person.

1

DRB Clearance Number CBDRB-FY23-POP001-0162

1

1.

Introduction
The Office of Management and Budget (OMB) provides guidelines for conducting a
nonresponse bias study when the expected unit response rate of a survey is below 80
percent (OMB, 2006). The Current Population Survey (CPS) household response rates
have historically been above 80 percent, 2 but the overall supplement response rates
(which are the product of the CPS household and Food Security Supplement (FSS)
household response rates) are below this threshold.
This document provides results from our evaluation of nonresponse in the 2022 FSS to
the CPS. Its purpose is to determine the existence of potential nonresponse bias in the
2022 FSS.
1.1

Overview of the CPS
The monthly CPS collects primarily labor force data about the civilian
noninstitutional population living in the United States. The institutional
population, which is excluded from the population universe, is composed
primarily of the population in correctional institutions and nursing homes (98
percent of the 4.0 million institutionalized people in Census 2010). Interviewers
ask questions concerning labor force participation about each member 15 years
old and over in sample households. For December 2022, the week containing the
twelfth of the month is the interview week. The week containing the fifth is the
reference week (i.e., the week about which the labor force questions are asked).
The CPS uses a multistage probability sample based on the results of the
decennial census, with coverage in all 50 states and the District of Columbia. The
sample is continually updated to account for new residential construction. When
files from the most recent decennial census become available, the Census
Bureau gradually introduces a new sample design for the CPS.
Every ten years, the CPS first-stage sample is redesigned 3 reflecting changes
based on the most recent decennial census. In the first stage of the sampling
process, primary sampling units (PSUs) 4 were selected for sample. In the 2010
sample design, the United States was divided into 1,987 PSUs. These PSUs were
then grouped into 852 strata. Within each stratum, a single PSU was chosen for
the sample, with its probability of selection proportional to its population as of

2
3
4

During the COVID-19 pandemic, data collection faced extraordinary circumstances that impacted response
rates.
For detailed information on the 2010 sample redesign, please reference Bureau of Labor Statistics (2014).
The PSUs correspond to substate areas (i.e., counties or groups of counties) that are geographically
contiguous.

DRB Clearance Number CBDRB-FY23-POP001-0162

2

the most recent decennial census. In the case of strata consisting of only one
PSU, the PSU was chosen with certainty.
1.2

Overview of the 2022 Food Security Supplement to the Current Population
Survey
In December 2022, in addition to the basic CPS questions, interviewers asked
supplementary questions of how much households spent for food, their use of
Federal and community food assistance programs, and whether they were able
to afford enough food. The universe for this supplement is households eligible
for the basic CPS. This supplement allows for proxy response. However, if at all
possible, interviewers should interview the person within the household who is
responsible for buying or preparing food for the household. Households with
incomes below 185 percent of the poverty threshold are asked all supplement
questions, whereas households with incomes over 185 percent of the poverty
threshold are asked only a few questions, unless their answers identify them as
“food insufficient or experiencing some degree of food hardship” and make
them eligible for the entire supplement (U.S. Census Bureau, 2019 or U.S. Census
Bureau, 2022a).
The key estimates include:
• Concern about food adequacy
• Money for food
• Access to food
• Use of emergency food
• Food assistance program participation
• Food intake reductions or hunger
Key domains, or characteristics for which the key estimates are created, include:
• Households
• Families
• Household Composition
• Unrelated Individuals
• Family Income

1.3

Discussion of Nonresponse in the 2022 Food Security Supplement to the
Current Population Survey
Some degree of nonresponse bias and variance is a normal feature of almost all
statistical surveys. The FSS produces food security estimates using the answers
from responding households and persons. These food security estimates will be
biased if answers from respondents differ from the potential answers of

DRB Clearance Number CBDRB-FY23-POP001-0162

3

nonrespondents. The magnitude of the bias is a function of the response rate
and differences between respondents and nonrespondents.
There were two ways that a household could be a nonrespondent to the FSS:
• The entire household did not respond to CPS (the occupants were not
found at home after repeated calls or are unavailable for some other
reason).
• The household responded to CPS but did not respond to the FSS
interview.
Because the FSS is directly linked to the CPS response rate, the CPS and FSS
attempt to minimize nonresponse bias by increasing response rates and
adjusting weights for potential differences between respondents and
nonrespondents. We try to increase response rates within CPS by conducting
personal visit interviews for new and returning sample units, mailing advance
letters for all sample units, providing a Spanish language questionnaire for
potential respondents who do not speak English, allowing interpreters for
potential respondents who do not speak English or Spanish, training field
representatives to gain respondent cooperation, allowing proxy respondents in
special circumstances, and mailing follow-up letters to nonresponding
households. We also help minimize nonresponse bias by reducing respondent
burden for the FSS by limiting the length of the survey.
We reduce the effects of respondent/nonrespondent differences through
noninterview weighting adjustments. These adjustments group respondents and
nonrespondents into adjustment cells, and the weights of the nonrespondents
are reallocated to the respondents within the adjustment cells.
CPS noninterview adjustment cells are formed by noninterview cluster (NICL)
and central city status. The NICLs are created based on sample PSUs that are
similar in metropolitan status and population size within the same state (U.S.
Census Bureau, 2006). Metropolitan status is defined as metropolitan or
nonmetropolitan. Within metropolitan PSUs, a further breakdown into “central
city” and “not central city” is defined. This results in 127 NICLs and 214
adjustment cells. These variables were chosen for the noninterview adjustment
cells because they are thought to be correlated with the CPS variables of
interest.
FSS noninterview adjustment cells are defined to be the same as the CPS
noninterview adjustment cells.

DRB Clearance Number CBDRB-FY23-POP001-0162

4

Despite the measures taken to reduce nonresponse bias, there is likely still some
amount of nonresponse bias that we cannot correct without knowing the food
security of the nonrespondents.
2.

Methodology
2.1

Data
The data for this nonresponse bias analysis are from the December 2022 FSS to
the CPS and the December 2022 CPS. The U.S. Census Bureau conducts the CPS
every month, although this file has only December data. The December survey
uses two sets of questions, the basic CPS and a set of supplemental questions.
The CPS, sponsored jointly by the Census Bureau and the U.S. Bureau of Labor
Statistics, is the country’s primary source of labor force statistics for the entire
population. The U.S. Department of Agriculture, Economic Research Service
sponsors the supplemental questions for December.
For a small number of variables, we had complete household information for all
sample households, including respondents and nonrespondents. These variables
were primarily limited to geographic and sampling data. There are also some
variables with partial information for the nonrespondents. Normal CPS
processing uses previous responses to demographic questions (when available)
and does not re-ask those that are unlikely to change from interview to
interview. Any variables that have never been answered are imputed using the
hot deck imputation method. Hot deck imputation assigns a value collected for a
person with similar characteristics to the missing value. Where possible,
nonrespondent actual values were used as opposed to edited or imputed values
in the comparison to respondents. The two exceptions are for tenure and
presence of children, because these characteristics were only available for CPS
respondents.

2.2

Weights
In the detailed weighting process for the CPS, base weights were adjusted with
the weighting control factor, which accounts for subsampling in the field but
does not include any nonresponse/noninterview or population coverage
adjustments. This subsampling-adjusted base weight is the weight used
throughout this report for household calculations for CPS.
When computing rates and distributions for FSS households, the FSS base
weights, which are the noninterview-adjusted weights from CPS, were used.
Note that FSS base weights are higher than CPS base weights because they
include the CPS noninterview adjustment, which inflates weights back up to the
eligible weighted CPS household sample.
DRB Clearance Number CBDRB-FY23-POP001-0162

5

All numbers presented in the report are weighted unless otherwise noted. All
estimates in this report are rounded according to Census Bureau Disclosure
Review Board policies.
2.3

Universe for the Estimates
We analyzed nonresponse for households. Since one person in each household
responded for the entire household, our analysis focused on household
nonresponse within reference person demographics, housing unit (HU)
characteristics, and geography.

3.

Limitations
There are some limitations to this analysis which may affect the results. In particular:
1. Using past data to assign subgroup variables to nonrespondents is not necessarily
accurate for households. Due to inmovers and outmovers, it is possible for
demographic variables that we get from past data to be out of date. However, we do
not believe our results need to be 100 percent accurate to show major differences
between respondents and nonrespondents. This assumes that the demographics of
neighborhoods do not change much in one and a half years.
2. Nonrespondents for CPS are never given the opportunity to respond to the FSS.

4.

Response Rates
The response rates tell us the percentage of eligible sample cases that responded to the
CPS and the FSS. It is useful to compare response rates for different subgroups to
understand the magnitude of potential biases.
We produced weighted and unweighted response rates for the 2022 FSS by key domains
and variables. The overall FSS response rate is the product of CPS household response
and FSS household response rate.
Response rates are defined as:
𝑅𝑅𝑅𝑅 =

∑𝑖𝑖∈𝑠𝑠 𝑤𝑤𝑖𝑖 𝑅𝑅𝑖𝑖 𝐷𝐷𝑖𝑖
∑𝑖𝑖∈𝑠𝑠 𝑤𝑤𝑖𝑖 𝐷𝐷𝑖𝑖

where:
𝑤𝑤𝑖𝑖 = the appropriate weight (1 if unweighted) for the response rate calculation
𝑅𝑅𝑖𝑖 = the response indicator (1 for respondents, 0 for nonrespondents)
𝐷𝐷𝑖𝑖 = the domain indicator (1 if within domain of interest, 0 otherwise)
𝑠𝑠 = the set of all eligible households
DRB Clearance Number CBDRB-FY23-POP001-0162

6

Eligible households are all sample housing units (HUs) that did not receive Type B or
Type C (out-of-scope) outcome codes. Persons within group quarters (GQs) are treated
as individual HUs. The CPS interview data contains all eligible and non-eligible HUs, and
the FSS interview data contains only eligible HUs to the CPS.
For the December 2022 CPS, there were approximately 59,500 occupied HUs eligible for
the household analysis. Of the 59,500 occupied households, 42,000 were interviewed.
Of the 42,000 households that were interviewed for CPS, 32,000 also responded to the
FSS.
Table 1 shows that the weighted percentage of households where at least one person
responded to CPS is 70.91 percent. From those responding households, 76.27 percent of
the weighted households responded to FSS. This results in an overall weighted FSS
response rate of 54.08 percent.
Table 1: 2022 Food Security Unit Response Rates
Response Category

Count

Weighted
Sum*

Unweighted
Response
Rates

Weighted
Response
Rates

Sampled CPSA Households
68,500 141,700,000
Eligible CPSA Households
59,500 123,900,000
CPSA Household Response
42,000
87,850,000
70.43%
70.91%
Food Security Households
42,000 123,900,000
Food Security Household Response
32,000
94,480,000
76.42%
76.27%
Overall Food Security Response
53.82%
54.08%
Source: U.S. Census Bureau internal data from December 2022 Current Population Survey.
A
CPS: Current Population Survey
* May not sum to totals due to rounding. For Current Population Survey (CPS) households, CPS household
weights prior to noninterview adjustments were used. For Food Security Supplement (FSS) households, the
FSS base weights, which are the noninterview-adjusted weights from CPS, were used. Note that FSS base
weights are higher than CPS base weights because they include the CPS noninterview adjustment, which
inflates weights back up to the eligible weighted CPS household sample. The CPS Household Response row
and Food Security Households row are the same set of households but are presented twice to show the
difference in weights used.

Table 2 shows weighted response rates for all CPS households by domain. The standard
error column shows the standard error of the response rate. Standard errors are
conditional on the sample and represent expected variability in the response process,
rather than traditional sampling error. Replicate weights were used to calculate the
standard errors to account for the sample design. The CPS uses the successive
difference replication method to calculate replicate weights. For detailed information on
variance estimation, please reference U.S. Census Bureau (2019).

DRB Clearance Number CBDRB-FY23-POP001-0162

7

Table 2: Response Rates for December 2022 Current Population Survey Households
Characteristic
Type of Living Quarters
Housing Unit
Non-Housing UnitA
BlankB
Principal City Status
Principal City within CBSA/MSAC
Residual within CBSA/MSAC
Outside of a CBSA/MSAC
Region
Northeast
Midwest
South
West
Urban/Rural Status
Urban
Rural
Missing
Race of Reference Person
White Only
Black Only
Asian Only
Other Race/Two or More Races
Blank
Gender of Reference Person
Male
Female
Blank
Hispanic Origin of Reference Person
Hispanic
Non-Hispanic
Blank
Age of Reference Person
15-29
30-39
40-49
50-59
60-69
70+
Blank or Less than 15
Overall

Unweighted
Households*

Weighted
Households*

Weighted Response
Rate (%)

Standard
Error (%)

Significance
Grouping×

56,500
2,600
100

118,700,000
4,924,000
225,500

70.85%
75.66%
1.347%

0.2641%
1.024%
1.336%

B
A
C

19,000
29,000
11,500

40,170,000
66,520,000
17,190,000

67.77%
71.52%
75.92%

0.4505%
0.3036%
0.8058%

C
B
A

10,000
11,500
22,500
15,000

21,530,000
27,080,000
47,730,000
27,550,000

67.12%
73.04%
69.84%
73.64%

0.5963%
0.5554%
0.4543%
0.4591%

C
A
B
A

45,000
13,500
850

98,160,000
23,900,000
1,826,000

69.98%
75.36%
63.13%

0.2834%
0.5492%
1.992%

B
A
C

36,500
5,300
2,300
1,400
14,000

75,280,000
11,770,000
5,320,000
2,328,000
29,190,000

88.74%
83.40%
87.22%
85.18%
15.79%

0.1915%
0.5677%
0.7687%
1.149%
0.4162%

A
C
AB
BC
D

24,000
24,500
11,000

50,080,000
51,040,000
22,770,000

86.85%
86.89%
0.071%

0.2553%
0.2605%
0.0277%

A
A
B

5,900
41,500
12,000

14,410,000
85,120,000
24,360,000

84.51%
88.19%
2.525%

0.5435%
0.1893%
0.2018%

B
A
C

4,900
7,500
7,400
7,800
8,800
9,700
13,000
59,500

10,240,000
15,840,000
15,780,000
16,590,000
18,060,000
19,900,000
27,480,000
123,900,000

81.83%
84.90%
85.54%
87.51%
90.18%
93.46%
11.37%
70.91%

0.5959%
0.4904%
0.4462%
0.4153%
0.3996%
0.3052%
0.3581%
0.2567%

E
D
D
C
B
A
F

Source: U.S. Census Bureau internal data from December 2022 Current Population Survey.
A
Non-Housing Units include quarters within rooming or boarding homes; non-permanent units in transient
hotels, motels, etc.; unoccupied sites for mobile homes, trailers, or tents; group quarters in school
dormitories; and other units that are not defined to be housing units.
B
Blank indicates that the living quarters type was either not identified or was identified with an invalid
code.
C
CBSA/MSA: Core-Based Statistical Area/Metropolitan Statistical Area
* May not sum to totals due to rounding. For weighted percent of total sample, reference Table 5.
×
Within each characteristic, response rates identified with the same letter are not significantly different at
the α=0.10 level. A indicates the highest response rates, B indicates the next highest rates, etc. P-values
were adjusted for multiple comparisons within each demographic characteristic using the Tukey-Kramer
method (NIST/SEMATECH, 2013).

DRB Clearance Number CBDRB-FY23-POP001-0162

8

Excluding the blanks and missing values, one of the largest differences in response rates
for the CPS subgroups is 11.64 percent, seen for the age of reference person, where age
group 70+ has a response rate of 93.46 percent versus 81.83 percent for age group 1529. Additionally, households in living quarters that are non-HUs have a higher response
rate than households in HU living quarters. Households outside of a core-based
statistical area/metropolitan statistical area (CBSA/MSA) have the highest response rate
within principal city status, west has one of the highest response rates among the
regions, 5 rural has a higher response rate than urban, White only has a higher response
rate than Black only and other race/two or more races, 6 non-Hispanic has a higher
response rate than Hispanic, and households with reference person aged 70+ has the
highest response rate among the age groups.
The response rate for blanks within the demographic subgroups is low because these
demographic items are collected during the interview, resulting in a large portion of the
household nonrespondents falling within these blank categories instead of the
categories where they belong. Any household with a blank value within the
demographic subgroups above indicates that the household has not previously
responded to the CPS or never provided responses to those demographic questions in
previous interviews. The nonresponse in the non-blank demographic categories is from
households which had previously responded to the CPS and provided a valid response
(non-blank) within the demographic category.
Table 2 shows standard errors which facilitate hypothesis testing of differential
response rates. However, the practical significance of response rate differences is
usually driven more by the magnitude of the difference. Therefore, excluding blanks, if
the nonrespondents are different from respondents, age of reference person has the
most potential for bias.
Table 3 shows weighted response rates for all FSS households by domain.

5
6

The response rate for west is not significantly different than the response rate for midwest.
The response rate for White only is not significantly different than the response rate for Asian only.

DRB Clearance Number CBDRB-FY23-POP001-0162

9

Characteristic

Table 3: Response Rates for 2022 Food Security Supplement Households
Unweighted
Weighted
Weighted
Standard
Households*
Households*
Response
Error (%)
Rate (%)

Type of Living Quarters
Housing Unit
Non-Housing UnitA
BlankB
Principal City Status
Principal City within CBSA/MSAC
Residual within CBSA/MSAC
Outside of a CBSA/MSAC
Region
Northeast
Midwest
South
West
Urban/Rural Status
Urban
Rural
Missing
Race of Reference Person
White Only
Black Only
Asian Only
Other Race/Two or More Races
Blank
Gender of Reference Person
Male
Female
Blank
Hispanic Origin of Reference Person
Hispanic
Non-Hispanic
Blank
Age of Reference Person
15-29
30-39
40-49
50-59
60-69
70+
Blank or Less than 15
Overall

Significance
Grouping×

40,000
1,900
<15

118,900,000
5,032,000
S

76.30%
75.44%
S

0.2928%
1.216%
S

A
A

13,000
20,500
8,500

40,000,000
66,660,000
17,220,000

75.91%
76.33%
76.86%

0.4966%
0.4257%
1.062%

A
A
A

6,800
8,400
15,500
11,000

21,550,000
27,080,000
47,710,000
27,550,000

76.70%
79.70%
71.83%
80.25%

0.6502%
0.6298%
0.5459%
0.5249%

B
A
C
A

31,500
9,900
550

97,960,000
24,320,000
1,605,000

76.60%
75.08%
74.25%

0.3134%
0.7257%
2.249%

A
A
A

32,500
4,400
2,000
1,200
1,900

93,600,000
14,350,000
6,646,000
2,780,000
6,514,000

77.73%
71.07%
75.50%
77.65%
66.95%

0.3389%
0.8129%
1.043%
1.481%
1.143%

A
B
A
A
C

20,500
21,000
<15

61,300,000
62,570,000
22,580

75.82%
76.72%
29.99%

0.3474%
0.3700%
17.77%

B
A
C+

4,900
36,500
250

17,460,000
105,600,000
833,600

74.63%
77.02%
15.87%

0.5733%
0.3206%
2.286%

B
A
C

4,000
6,300
6,300
6,800
7,900
9,100
1,400
42,000

11,880,000
19,130,000
19,070,000
20,470,000
22,920,000
26,050,000
4,367,000
123,900,000

77.08%
77.46%
76.11%
77.84%
78.70%
77.74%
40.61%
76.27%

0.7438%
0.5924%
0.7014%
0.5813%
0.5847%
0.5692%
1.628%
0.2916%

A, B
A, B
B
A, B
A
A, B
C

_

Source: U.S. Census Bureau internal data from December 2022 Current Population Survey.
A
Non-Housing Units include quarters within rooming or boarding homes; non-permanent units in transient
hotels, motels, etc.; unoccupied sites for mobile homes, trailers, or tents; group quarters in school
dormitories; and other units that are not defined to be housing units.
B
Blank indicates that the living quarters type was either not identified or was identified with an invalid code.
C
CBSA/MSA: Core-Based Statistical Area/Metropolitan Statistical Area
* May not sum to totals due to rounding. For weighted percent of total sample, reference Table 6.
+
Exercise caution: The sample size is extremely small, leading to unreliable estimates.
×
Within each characteristic, response rates identified with the same letter are not significantly different at
the α=0.10 level. A indicates the highest response rates, B indicates the next highest rates, etc. P-values

DRB Clearance Number CBDRB-FY23-POP001-0162

10

were adjusted for multiple comparisons within each demographic characteristic using the Tukey-Kramer
method (NIST/SEMATECH, 2013).
S The estimate is withheld because estimate did not meet publication standards. Significance grouping is not
applicable for these cases.

For the FSS household estimates, excluding the blanks, there were no significant
differences among the type of living quarters, principal city status, or urban/rural status
response rates. Excluding the blanks and missing values, one of the largest differences in
response rates for the FSS subgroups is 8.42 percent, 7 seen for region, where the west
has a response rate of 80.25 8 percent versus 71.83 percent for the south. Additionally,
households with a female reference person have a higher response rate than
households with a male reference person, and households with a non-Hispanic
reference person have a higher response rate than households with a Hispanic
reference person.
Again, although Table 3 shows standard errors which facilitate hypothesis testing, the
practical significance of response rate differences is driven more by the magnitude of
the difference than the sample size. Therefore, excluding blanks, if the nonrespondents
are different from respondents, region and race of reference person has the most
potential for bias.
Table 4 shows weighted response rates for all FSS households by domain for
characteristics that were only available for CPS respondents.

7
8

The largest difference in response rates for region is not significantly different than the largest difference in
response rates for race of reference person.
The response rate for the west is not significantly different than the response rate for the midwest.

DRB Clearance Number CBDRB-FY23-POP001-0162

11

Table 4: Response Rates for 2022 Food Security Supplement Households for Characteristics Only Available for
Responding Current Population Survey Households
Characteristic

Unweighted
Households*

Weighted
Households*

Weighted
Response
Rate (%)

Standard
Error (%)

Significance
Grouping×

Tenure (Edited)
Owned or Mortgage
29,000
83,910,000
77.07%
0.3391%
A
Rented for Cash
12,500
38,670,000
74.46%
0.4805%
B
No Cash Rent
450
1,307,000
78.01%
2.251%
A, B
Family Income
Less than $10,000
1,400
4,174,000
79.71%
1.347%
A
$10,000-$19,999.99
2,600
7,502,000
82.79%
0.9227%
A
$20,000-$29,999.99
2,800
8,093,000
81.49%
0.8902%
A
$30,000-$39,999.99
3,200
9,301,000
82.96%
0.8203%
A
$40,000-$49,999.99
2,500
7,128,000
84.70%
0.8346%
A
$50,000-$59,999.99
2,600
7,458,000
82.67%
0.8948%
A
$60,000-$74,999.99
3,400
96,910,000
83.49%
0.7827%
A
$75,000-$99,999.99
4,200
12,120,000
83.38%
0.7016%
A
$100,000-$149,999.99
4,800
14,320,000
83.83%
0.6580%
A
$150,000+
5,500
17,120,000
83.35%
0.6057%
A
Blank or Don’t Know
1,800
5,253,000
51.96%
1.444%
B
Refused
7,200
21,730,000
51.72%
0.7109%
B
Household Type
Husband/Wife Primary Family
20,000
58,550,000
76.71%
0.3702%
A
Unmarried Householder
6,600
20,170,000
74.55%
0.6054%
B
Primary Family
Primary Individual
15,500
45,030,000
76.55%
0.4703%
A
Group Quarters with Family
<15
24,400
87.62%
12.20%
A, B+
Group Quarters without
30
112,400
38.33%
14.93%
B
Family
Child(ren) Present (Edited)
No
32,500
95,660,000
76.38%
0.3381%
A
Yes
9,400
28,230,000
75.90%
0.5117%
A
Measure of Labor Force Participation Status of Reference Person
Employed
24,500
72,750,000
76.60%
0.3578%
B
Unemployed
700
2,090,000
80.66%
1.714%
A
Not in Labor Force
16,500
48,440,000
75.89%
0.4461%
B
Blank
250
599,100
50.90%
4.588%
C
Overall
42,000
123,900,000
76.27%
0.2916%
Source: U.S. Census Bureau internal data from December 2022 Current Population Survey.
* May not sum to totals due to rounding. For weighted percent of total sample, reference Table 7.
+
Exercise caution: The sample size is small, leading to unreliable estimates.
×
Within each characteristic, response rates identified with the same letter are not significantly different at
the α=0.10 level. A indicates the highest response rates, B indicates the next highest rates, etc. P-values
were adjusted for multiple comparisons within each demographic characteristic using the Tukey-Kramer
method (NIST/SEMATECH, 2013).

For the FSS household estimates of characteristics only available for CPS respondents,
excluding the blanks, refusals, and GQ groups, there were no significant differences in
response rates within the family income and child(ren) present categories. For the
DRB Clearance Number CBDRB-FY23-POP001-0162

12

tenure category, owned or mortgage has a higher response rate than rented for cash,
but neither are significantly different from the response rate for no cash rent.
Husband/wife primary family households and primary individual households have
response rates that are not significantly different, but both are higher than the response
rate for unmarried householder primary family households. For the measure of labor
force participation category, unemployed had a higher response rate than either
employed or not in the labor force. 9
Again, although Table 4 shows standard errors which facilitate hypothesis testing, the
practical significance of response rate differences is driven more by the magnitude of
the difference than the sample size. However, among the categories in Table 4, the
magnitudes of the differences are not statistically different. Therefore, it is difficult to
identify which category has the most potential for bias. The measure of labor force
participation category has a difference in response rates of 4.77 10 percent, where
unemployed has a response rate of 80.66 percent versus 75.89 percent for not in the
labor force. 11
5.

Respondent Distributions
Respondent and nonrespondent distributions show the relative percent of members of
a domain subset within respondents and nonrespondents separately. This is different
than the response rates, which are the relative percent of respondents within the
different domain subsets. We used chi-square tests to determine if the respondent and
nonrespondent distributions differed.
Respondent distributions are defined as:
𝑅𝑅𝑅𝑅 =

∑𝑖𝑖∈𝑠𝑠 𝑤𝑤𝑖𝑖 𝑅𝑅𝑖𝑖 𝐷𝐷𝑖𝑖
∑𝑖𝑖∈𝑠𝑠 𝑤𝑤𝑖𝑖 𝑅𝑅𝑖𝑖

This definition assumes the same eligibility criteria, weights, and indicators as the
response rate calculations in the previous section. Nonrespondent distributions use the
same formula, but with the 𝑅𝑅𝑖𝑖 variable indicating nonrespondents instead of
respondents. The chi-square test statistics were calculated using replicate weights to
account for the sample design.
Table 5 shows the percent of total sample distribution as well as comparisons of
respondent and nonrespondent distributions for CPS households within the different
domain subgroups.
9
10
11

The response rates for employed and not in the labor force are not significantly different.
The largest difference of 4.77 percent for measure of labor force participation was not significantly different
from the largest differences for family income, tenure, or household type.
The response rate for not in the labor force is not significantly different than the response rate for employed.

DRB Clearance Number CBDRB-FY23-POP001-0162

13

Table 5: Respondent and Nonrespondent Distributions for December 2022
Current Population Survey Households
Characteristic
Type of Living Quarters
Housing Unit
Non-Housing UnitA
BlankB
Principal City Status
Principal City within CBSA/MSAC
Residual within CBSA/MSAC
Outside of a CBSA/MSAC
Region
Northeast
Midwest
South
West
Urban/Rural Status
Urban
Rural
Missing
Race of Reference Person
White Only
Black Only
Asian Only
Other Race/Two or More Races
Blank
Gender of Reference Person
Male
Female
Blank
Hispanic Origin of Reference Person
Hispanic
Non-Hispanic
Blank
Age of Reference Person
15-29
30-39
40-49
50-59
60-69
70+
Blank or Less than 15
Overall

Percentage
of Total
Sample*

Household
Respondent
Percentage*

Household
Nonrespondent
Percentage*

Chi-Square
Statistic (df)

P-value

95.84%
3.975%
0.1821%

95.76%
4.240%
0.003459%

96.06%
3.326%
0.6175%

32.43%
53.70%
13.88%

30.99%
54.15%
14.86%

35.93%
52.58%
11.49%

92.97 (2)

< 0.0001

17.38%
21.86%
38.53%
22.24%

16.45%
22.51%
37.94%
23.10%

19.64%
20.26%
39.94%
20.16%

88.16 (3)

< 0.0001

79.24%
19.29%
1.474%

78.19%
20.50%
1.312%

81.79%
16.34%
1.868%

96.29(2)
[74.47(1)

< 0.0001
< 0.0001]

60.77%
9.500%
4.294%
1.879%
23.56%

76.04%
11.17%
5.281%
2.257%
5.245%

23.52%
5.423%
1.888%
0.9576%
68.21%

21,300 (4)
[114.9 (3)

< 0.0001
< 0.0001]

40.42%
41.20%
18.38%

49.50%
50.48%
0.01841%

18.28%
18.57%
63.14%

30,780 (2)
[0.0135 (1)

< 0.0001
0.9074]

11.63%
68.71%
19.67%

13.86%
85.44%
0.7002%

6.192%
27.90%
65.91%

8.266%
12.79%
12.73%
13.39%
14.58%
16.06%
22.18%
100%

9.538%
15.31%
15.36%
16.52%
18.54%
21.17%
3.557%
100%

5.165%
6.638%
6.332%
5.748%
4.921%
3.611%
67.59%
100%

198.3 (2)
[18.48(1)

< 0.0001
< 0.0001]

21,980 (2)
[49.37 (1)

< 0.0001
< 0.0001]

22,460 (6)
[455.0 (5)

< 0.0001
< 0.0001]

Source: U.S. Census Bureau internal data from December 2022 Current Population Survey.
A
Non-Housing Units include quarters within rooming or boarding homes; non-permanent units in transient
hotels, motels, etc.; unoccupied sites for mobile homes, trailers, or tents; group quarters in school
dormitories; and other units that are not defined to be housing units.
B
Blank indicates that the living quarters type was either not identified or was identified with an invalid code.
C
CBSA/MSA: Core-Based Statistical Area/Metropolitan Statistical Area
* May not sum to totals due to rounding.
[] The values within brackets are the chi-square statistic, df, and p-value when the blanks/missings are
excluded from the chi-square test.

DRB Clearance Number CBDRB-FY23-POP001-0162

14

The chi-square tests for CPS households showed significant differences (at the α=0.10
level) between respondent and nonrespondent distributions for all variables. Simply
looking at the distributions for the race, gender, Hispanic origin, and age of reference
person, you can tell that there are large differences between the respondent and
nonrespondent distributions, which corresponds to the magnitude of the chi-square test
statistics (21,300, 30,780, 21,980, and 22,460, respectively). However, when you
exclude the blanks from the chi-square test, the gender of the reference person no
longer has a significant difference between the two distributions. Note: The chi-square
tests only indicate that the distributions of respondents and nonrespondents differ but
do not necessarily indicate a nonresponse bias problem. These differences will only
cause bias if the respondents and nonrespondents report differing rates of food
security.
Even though there are significant differences between the respondents and
nonrespondents, the differences might not be large enough to cause meaningful
differences in estimates. Furthermore, weighting adjustments might also minimize the
impact of some differences. Because the CPS noninterview adjustments take NICL and
central city status into account, the principal city status and region differences may be
reduced within those adjustments.
Table 6 shows the percent of total sample distribution as well as comparisons of
respondent and nonrespondent distributions for FSS households within the different
domain subgroups.

DRB Clearance Number CBDRB-FY23-POP001-0162

15

Table 6: Respondent and Nonrespondent Distributions for 2022 Food Security Supplement Households
Characteristic
Type of Living Quarters
Housing Unit
Non-Housing UnitA
BlankB
Principal City Status
Principal City within CBSA/MSAC
Residual within CBSA/MSAC
Outside of a CBSA/MSAC
Region
Northeast
Midwest
South
West
Urban/Rural Status
Urban
Rural
Missing
Race of Reference Person
White Only
Black Only
Asian Only
Other Race/Two or More Races
Blank
Gender of Reference Person
Male
Female
Blank
Hispanic Origin of Reference Person
Hispanic
Non-Hispanic
Blank
Age of Reference Person
15-29
30-39
40-49
50-59
60-69
70+
Blank or Less than 15
Overall

Percentage
of Total
Sample*

Household
Respondent
Percentage*

Household
Nonrespondent
Percentage*

95.94%
4.062%
0.002948%

95.98%
4.018%
0.000%

95.78%
4.203%
0.01242%

32.29%
53.81%
13.90%

32.14%
53.85%
14.01%

17.39%
21.86%
38.51%
22.24%

Chi-Square
Statistic (df)

P-value

N (2)
[0.5148 (1)

N
0.4731]

32.77%
53.67%
13.55%

0.7262 (2)

0.6955

17.49%
22.84%
36.27%
23.40%

17.07%
18.69%
45.72%
18.51%

159.0 (3)

< 0.0001

79.07%
19.63%
1.296%

79.41%
19.33%
1.261%

77.98%
20.62%
1.406%

5.382 (2)
[3.871 (1)

0.0678
0.0491]

75.55%
11.58%
5.365%
2.244%
5.258%

77.00%
10.79%
5.311%
2.285%
4.615%

70.91%
14.12%
5.538%
2.113%
7.322%

158.2(4)
[77.59 (3)

< 0.0001
< 0.0001]

49.48%
50.50%
0.01822%

49.19%
50.80%
0.007167%

50.41%
49.54%
0.05376%

13.19 (2)
[4.564 (1)

0.0014
0.0326]

14.10%
85.23%
0.6729%

13.79%
86.07%
0.1400%

15.07%
82.54%
2.385%

9.591%
15.44%
15.39%
16.52%
18.50%
21.03%
3.525%
100%

9.693%
15.68%
15.36%
16.86%
19.09%
21.43%
1.877%
100%

9.261%
14.66%
15.49%
15.43%
16.61%
19.72%
8.823%
100%

528.7 (2)
[14.35 (1)

< 0.0001
0.0002]

765.9 (6)
[10.54 (5)

< 0.0001
0.0613]

Source: U.S. Census Bureau internal data from December 2022 Current Population Survey.
A
Non-Housing Units include quarters within rooming or boarding homes; non-permanent units in transient
hotels, motels, etc.; unoccupied sites for mobile homes, trailers, or tents; group quarters in school
dormitories; and other units that are not defined to be housing units.
B
Blank indicates that the living quarters type was either not identified or was identified with an invalid
code.
C
CBSA/MSA: Core-Based Statistical Area/Metropolitan Statistical Area
* May not sum to totals due to rounding.
N Estimate is not available.
[] The values within brackets are the chi-square statistic, df, and p-value when the blanks/missings are
excluded from the chi-square test.

DRB Clearance Number CBDRB-FY23-POP001-0162

16

The chi-square tests for FSS households showed significant differences (at the α=0.10
level) for the distributions of all variables except type of living quarters, principal city
status, and urban/rural status. Simply looking at the distributions for age of the
reference person, you can tell that there are large differences between the respondent
and nonrespondent distributions, which correspond to the magnitude of the chi-square
test statistic (765.9).
As mentioned for CPS household respondent distributions, the chi-square tests only
indicate that the distributions of respondents and nonrespondents differ but do not
necessarily indicate a nonresponse bias problem. Furthermore, weighting adjustments
might minimize the impact of some differences. Because the FSS noninterview
adjustments take NICL and central city status into account, the principal city status and
region differences may be reduced within those adjustments.
Table 7 shows the percent of total sample distribution as well as comparisons of
respondent and nonrespondent distributions for FSS households within the different
domain subgroups for characteristics that were only available for CPS respondents.

DRB Clearance Number CBDRB-FY23-POP001-0162

17

Table 7: Respondent and Nonrespondent Distributions for 2022 Food Security Supplement Households for
Characteristics Only Available for Responding CPS Households
Percentage of
Household
Household
Chi-Square
Characteristic
Total
Respondent
Nonrespondent
P-value
Statistic (df)
Sample*
Percentage*
Percentage*
Tenure (Edited)
Owned or Mortgage
67.73%
68.45%
65.43%
25.03 (2)
< 0.0001
Rented for Cash
31.22%
30.47%
33.60%
No Cash Rent
1.055%
1.079%
0.9773%
Family Income
Less than $10,000
3.369%
3.521%
2.881%
$10,000-$19,999.99
6.055%
6.573%
4.391%
$20,000-$29,999.99
6.533%
6.980%
5.094%
$30,000-$39,999.99
7.507%
8.166%
5.391%
$40,000-$49,999.99
5.754%
6.390%
3.709%
$50,000-$59,999.99
6.020%
6.525%
4.396%
2,718 (11)
< 0.0001
$60,000-$74,999.99
7.823%
8.563%
5.443%
[17.02 (9)
0.0485]
$75,000-$99,999.99
9.780%
10.69%
6.850%
$100,000-$149,999.99
11.56%
12.71%
7.878%
$150,000+
13.82%
15.10%
9.695%
Blank or Don’t Know
4.240%
2.889%
8.584%
Refused
17.54%
11.89%
35.69%
Household Type
Husband/Wife Primary Family
47.26%
47.54%
46.38%
Unmarried Householder Primary
Family
16.28%
15.91%
17.46%
25.05 (4)
< 0.0001
Primary Individual
36.35%
36.48%
35.92%
{18.58 (3)
0.0003}
Group Quarters with Family
0.01968%
0.002260%
0.01027%
Group Quarters without Family
0.09077%
0.04561%
0.2359%
Child(ren) Present (Edited)
No
77.21%
77.32%
76.86%
0.6441 (1)
0.4222
Yes
22.79%
22.68%
23.14%
Measure of Labor Force Participation Status of Reference Person
Employed
58.73%
58.99%
57.90%
Unemployed
1.687%
1.784%
1.375%
53.95 (3)
< 0.0001
Not in Labor Force
39.10%
38.91%
39.73%
[6.955 (2)
0.0309]
Blank
0.4836%
0.3228%
1.000%
Overall
100%
100%
100%
Source: U.S. Census Bureau internal data from December 2022 Current Population Survey.
Note: The chi-square test for household type could not be calculated due to a frequency of zero within the
Group Quarters with Family cell.
* May not sum to totals due to rounding.
[] The values within brackets are the chi-square statistic, df, and p-value when the blanks and refusals are
excluded from the chi-square test.
{} The values within braces are the chi-square statistic, df, and p-value when combining the groups, Group
Quarters with Family, and Group Quarters without Family.

The chi-square tests for FSS household estimates of characteristics only available for CPS
respondents showed significant differences (at the α=0.10 level) for the distributions of
all variables except child(ren) present. Simply looking at the distributions for family
DRB Clearance Number CBDRB-FY23-POP001-0162

18

income, you can tell that there are large differences between the respondent and
nonrespondent distributions, which correspond to the magnitude of the chi-square test
statistic (2,718). Even after excluding the blanks and refusals, there is still a significant
difference between the respondent and nonrespondent distributions for family income.
As mentioned previously, the chi-square tests only indicate that the distributions of
respondents and nonrespondents differ but do not necessarily indicate a nonresponse
bias problem. Furthermore, weighting adjustments might minimize the impact of some
differences.
6.

Conclusions
This analysis found evidence of potential nonresponse bias for both CPS and FSS
households. For CPS, there is potential nonresponse bias for all investigated
characteristics except possibly for gender of reference person. For FSS, there is potential
nonresponse bias for most investigated characteristics except type of living quarters,
principal city status, or urban/rural status.
Excluding the blanks and missing values, one of the largest differences in response rates
for the CPS subgroups is 11.64 percent, seen for the age of reference person, where age
group 70+ has a response rate of 93.46 percent versus 81.83 percent for age group 1529. For respondent and nonrespondent distributions within CPS households, the largest
differences are seen within race, gender, Hispanic origin, and age of reference person.
Excluding the blanks and missing values, one of the largest differences in response rates
for the FSS subgroups is 8.42 percent, 12 seen for region, where the west has a response
rate of 80.25 13 percent versus 71.83 percent for the south. For respondent and
nonrespondent distributions within FSS households, a large difference is seen within age
of reference person.
Among the estimates for the FSS households for characteristics only available for CPS
respondents, one of the largest differences between the respondent and
nonrespondent distributions is seen within family income.
Using the information learned from this analysis, discussions should be had with the
sponsor regarding enhancements to the weighting process. The findings suggest that
research could be done into the possible inclusion of other geographic and demographic
characteristics into the household noninterview adjustments for the FSS. Research could
be conducted into whether the nonresponse adjustment should include the geographic
and demographic characteristics that were investigated in this report to determine if

12
13

The largest difference in response rates for region is not significantly different than the largest difference in
response rates for race of reference person.
The response rate for the west is not significantly different than the response rate for the midwest.

DRB Clearance Number CBDRB-FY23-POP001-0162

19

they can help reduce the nonresponse bias. Some other potential characteristics that
may be related to food security to consider including in the noninterview adjustment
may be block and/or tract planning database variables, which would include geographic
and demographic variables based on the location of the sampled household, such as
percent of population that is Hispanic, percent of HUs where no one lives regularly
(vacant HUs), percent of population that is below the poverty level. Note: planning
database variables are estimates using American Community Survey or Census data.
Please reference U.S. Census Bureau (2023) for additional information on the planning
database.
7.

References
Bureau of Labor Statistics. (2014). “Redesign of the Sample for the Current Population
Survey.” http://www.bls.gov/cps/sample_redesign_2014.pdf
NIST/SEMATECH. (2013). “NIST/SEMATECH e-Handbook of Statistical Methods.”
http://www.itl.nist.gov/div898/handbook/prc/section4/prc471.htm
Office of Management and Budget. (2006). “Standards and Guidelines for Statistical Surveys.”
https://georgewbushwhitehouse.archives.gov/omb/inforeg/statpolicy/standards_stat_surveys.pdf
U.S. Census Bureau. (2006). Current Population Survey: Design and Methodology. Technical
Paper 66. Washington, DC: Government Printing Office. https://www2.census.gov/programssurveys/cps/methodology/tp-66.pdf
U.S. Census Bureau. (2019). Current Population Survey: Design and Methodology. Technical
Paper 77. Washington, DC: Government Printing Office. https://www2.census.gov/programssurveys/cps/methodology/CPS-Tech-Paper-77.pdf
U.S. Census Bureau. (2022a). “Current Population Survey Food Security Supplement.”
https://www.census.gov/data/datasets/time-series/demo/cps/cps-supp_cps-repwgt/cpsfood-security.html
U.S. Census Bureau. (2022b). “December 2020 Food Security Technical Documentation.”
https://www2.census.gov/programs-surveys/cps/techdocs/cpsdec21.pdf
U.S. Census Bureau. (2023). “Planning Database (2015, 2016, 2018-2022).”
https://www.census.gov/data/developers/data-sets/planning-database.2019.html
All online references last accessed on August 29, 2023.

DRB Clearance Number CBDRB-FY23-POP001-0162

20