Assessing the Effect of Calibration on Nonresponse Bias in the 2008 ARMS Phase III Sampel Using Census 2007 Data

Assessing the Effect of Calibration on Nonresponse Bias in
the 2008 ARMS Phase III Sample Using Census 2007 Data
Morgan Earp1, Jaki McCarthy2, Eric Porter3, & Phillip Kott4
1

National Agricultural Statistics Service (NASS), 3251 Old Lee Highway Room 305,
Fairfax, VA 22030
2
NASS, 3251 Old Lee Highway Room 305, Fairfax, VA 22030
3
NASS, 1400 Independence Avenue SW, Washington, DC 20004
4
NASS, 3251 Old Lee Highway Room 305, Fairfax, VA 22030

Abstract
The USDA’s National Agricultural Statistics Service (NASS) conducts the annual
Agricultural Resource Management Survey (ARMS). ARMS is a detailed economic
survey suffering from relatively low response rates for a federal survey. To adjust for
ARMS nonresponse bias, coverage and measurement errors, NASS uses calibration
weighting. Prior research using 2002 Census of Agriculture data, available for 2005-2006
ARMS samples, indicated that calibration decreased nonresponse bias except in two
cases; however, because not all ARMS calibration targets were collected on the 2002
Census of Agriculture, NASS did not fully replicate the ARMS calibration process. This
study replicates prior research for the 2008 ARMS using the 2007 Census of Agriculture
data, which includes equivalent variables for all ARMS calibration targets thus allowing
NASS to assess fully the effectiveness of ARMS calibration.
Key Words: Nonresponse; Bias; Calibration; Response Rate

1. Introduction
Survey nonresponse happens; the question is, how do we address it? In 2003, the Federal
Government’s Federal Committee on Statistical Methodology (FCSM) formed a
subcommittee of the Interagency Council on Statistical Policy (ICSP) representative
nominees to update Federal standards for statistical surveys. This Subcommittee on
Standards for Statistical Surveys concluded that in order to ensure the quality, objectivity,
utility, and integrity of Federal Government data, nonresponse bias should be assessed
when surveys exhibit insufficient response rates. Under the guidance of the FCSM and
ICSP, ICSP representatives recommended Federal survey standards and guidelines to the
Executive Office of the President’s Office of Management and Budget in 2004. After
public review, the Executive Office of the President ultimately released the Office of
Management and Budget Standards and Guidelines for Statistical Surveys on September
22, 2006.
The United States Department of Agriculture’s (USDA) National Agricultural Statistics
Service (NASS) along with several other federal statistical agencies helped develop the
OMB’s new standards and guidelines for statistical surveys. This paper focuses
specifically on Standard 3.2. Standard 3.2 addresses response rates and analysis of
nonresponse bias, requiring that “Agencies must appropriately measure, adjust for, report,
and analyze unit and item nonresponse to assess their effects on data quality and to

inform users” when survey response rates fall below 80 percent. (Office of Management
and Budget, 2006, p. 14).
In 2005 and 2006, the Agricultural Resource Management Survey Phase III (ARMS III)
response rate fell below the OMB response rate threshold of 80 percent listed in
Guideline 3.2.9, and as a result NASS conducted two independent analyses of
nonresponse bias (Earp, McCarthy, Schauer, & Kott, 2008; Earp et al., 2009). Both
assessments were done using the Census of Agriculture 2002 data as a proxy; However,
since the key variables of interest included expenditures which were not included in the
2002 short form, the analysis was limited to only those responding to the 2002 Census of
Agriculture long form. Furthermore, the analysis was also limited by the fact that not all
of the calibration targets used for ARMS III adjustments were collected on the 2002
Census of Agriculture. This report assesses the effectiveness of calibration as a tool for
reducing nonresponse bias to insignificant levels, using the 2007 Census of Agriculture
data as a proxy. Unlike the 2002 Census of Agriculture, the 2007 Census of Agriculture
consisted of one form that included expenditure items as well as all items necessary to
replicate fully the ARMS III 2008 calibration process.
The ARMS is conducted in three phases. Phase I screens for potential samples for Phases
II and III. Phase II collects data on cropping practices and agricultural chemical usage,
while Phase III collects detailed economic information about the agricultural operation,
as well as information about the operator’s household. Phase III is the only phase of the
2006 ARMS with response rates lower than 80 percent.
Due to lower response rates with the ARMS Phase III, the potential for nonresponse bias
is greater there. NASS weights the ARMS Phase III respondent sample in such a way that
estimated variable totals for a large set of items match “targets” determined from other
sources. This is done through a weighting process called “calibration.” Calibration is the
process of adjusting survey weights so that certain targets are met. NASS uses official
estimates of farm numbers; corn, soybean, wheat, cotton, fruit and vegetable acreage; egg
and milk production; and cattle, hog, broiler, and turkey inventories as calibration targets.
For example, after calibration, the calibration-weighted sum of the survey data will equal
the NASS estimate for corn acreage. In addition to reducing confusion in the user
community that might result from NASS releasing alternative estimates for the same
totals, calibration weighting produces ARMS Phase III estimates with generally lower
variances and reduces nonresponse biases. This report describes an ongoing research
effort aimed at measuring the potential for nonresponse bias in the ARMS Phase III and
the success or failure of calibration in removing it.
Nonresponse bias is very difficult to measure directly. Fortunately, an indirect measure of
nonresponse bias is available for assessing ARMS III using an operation’s Census of
Agriculture data as a proxy.
The Census of Agriculture is a mandatory collection of data from all known agricultural
operations. NASS has data from the Census on items of interest for many of the ARMS
nonrespondents; however, the Census itself is incomplete. An estimated 16.24 percent of
all farms were missing from the 2007 Census Mailing List, and 14.65 percent of farms on
the List failed to respond to the Census (USDA, 2007, Table A). Moreover, 5.67 percent
of the operations sampled for ARMS III could not be matched to 2007 Census records.
Nevertheless, by comparing the 2007 Census values of ARMS III respondents to the full
sample of ARMS III respondents as a whole, we can measure the difference between the

average ARMS III respondent and the average of the full sample without any
nonresponse adjustment. Additionally, this analysis intends to measure the reduction of
that difference from using a calibration-weighting process similar to the one used for the
2008 ARMS.
Although the 2007 Census data do not perfectly match the 2008 ARMS Phase III data,
they are moderately to highly correlated (see Appendix Tables A-2 & A-3). The present
evaluation will effectively compare 2008 ARMS Phase III survey respondents to
nonrespondents using their 2007 Census data.
2. Method
Our analytical data set consists of census values for farms sampled for the ARMS III that
responded to the 2007 Census of Agriculture.
The base sampling weight for a farm in our analytical data set was its ARMS III sample
weight before calibration multiplied by its Census weight. Each ARMS III responding
farm was calibrated to produce weighted totals for the calibration variables that were
equal to the base-sampling-weighted totals computed from both respondents and
nonrespondents. The calibration variables used were inventory/acreage numbers for
cattle, corn, cotton, pigs, soybeans, wheat, fruit, vegetables, broilers, and turkeys. We
used all the target variables, plus egg and milk production, in calibrating the ARMS III
data.
As in the operational program, the ARMS III respondent subset was calibrated
independently in 20 regions. These included the 15 leading cash receipt states (Arkansas,
California, Florida, Georgia, Illinois, Indiana, Iowa, Kansas, Minnesota, Missouri,
Nebraska, North Carolina, Texas, Washington, and Wisconsin). The remaining 33 states
(Alaska and Hawaii are not sampled for the ARMS) were grouped using the five
production regions: 1) Atlantic, 2) South, 3) Midwest, 4) Plains, and 5) West (Figure 1).

Figure 1. ARMS III Estimation Regions

Our analysis focuses on 17 specific (non-calibration) variables collected on both the
ARMS and the Census:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
15.
16.
17.

Total Acres
Total Sales
Acres Rented
Cropland Acres
Total Production Expenses
Crop Expenses
Seed Expenses
Fertilizer Expenses
Chemical Expenses
Livestock Purchases
Feed Purchases
Hired Labor Expenses
Machinery and Equipment Value
Government Payments
Operator’s Age
Operator’s Race
Farm Type.

These variables were also included in a similar analysis for the 2005 and 2006 ARMS
Phase III (Earp et al., 2008 & Earp et al., 2009).
Letting yr denote the base-sample or calibrated-sample mean among the ARMS
respondent subset for a study variable, and yt denote the corresponding base-sample
mean among the entire matched sample, it is a simple matter to compute the relative bias
of the former with respect to the latter, relBias =

yr

yt
yr

. The statistical significance of

this value is much harder to assess since the samples on which
complex and overlapping.

yr

and

yt

are based are

Fortunately, we can easily test the persistence or absence of a systematic bias across the
20 regions. To this end, we compute the following measure of bias of an ARMSrespondent mean (before or after calibration) with respect to the Census mean in every
region:
M = log( y r )
= log

yr

= log 1

log( y t )

yt

yr

yt
yt

yr

yt
yt

This measure is conveniently symmetric, log( yt ) log( yr ) [log( yr ) log( yt )] while
retaining the scale-invariance property of the relative bias (i.e., multiplying the reported
item value on each farm by a fixed factor does not affect the overall relative bias).
The bias measure M for a study variable in a region can be treated as an independent
random variable. The null hypothesis of no bias (again, either before or after calibration)
can be tested against an alternative hypothesis of a persistent bias (p %) across all the
regions. The conventional t test based on the 20 observations (one per region) is
asymptotically normal under both the null and alternative hypotheses. We follow the
standard practice of approximating the distribution of this test statistic with a Student’s t
having 19 degrees of freedom. This may lead to liberal inferences (the inappropriate
rejection of the null hypothesis when it is true) because the M-values for the study
variable may not be normally distributed with a common variance across regions.
Nevertheless, by taking logs we create a test statistic that is more nearly normal and
homoscedastic than absolute biases would be.
A sign and a signed-rank test of the 20 paired observations for a study variable before and
after calibration was conducted. The sign test is not as powerful as the other two tests
(i.e., it more often fails to find that M is significantly different from 0 when, in fact, there
is a persistent bias across the regions), but it assumes neither that M is normal nor
homoscedastic. The signed-rank test assumes the latter, but not the former. We include all
three analyses in our results for completeness.
3. Results
Table 1 provides a summary of the results. All estimates exhibited significant bias before
calibration; however, after calibration the following estimates no longer exhibited
significant bias: Total Sales, Acres Rented, Cropland Acres, Total Production Expenses,
Cropland Expenses, Seed Expenses, Chemical Expenses, Livestock Purchases, Feed
Purchases, Hired Labor Expenses, Government Payments, and Farm Type. This finding
is consistent with the results of the 2005 and 2006 analyses for the estimates where Feed
Expenses, Total Production Expenses, Seed Expenses, Livestock Purchases, Cropland
Expenses, and Hired Labor Expenses were shown to no longer exhibit significant bias
after calibration (Earp et al., 2008 & Earp et al., 2009).
In over 70 percent (12/17) of the study variables exhibiting persistent biases using the
base sample weights, calibration weighting was able to reduce the bias so that it was no
longer significantly different from zero using a t-test with p < .05. The rate of bias
elimination was slightly lower than found in 2005 to 2006 (Earp et al., 2008 & Earp et
al., 2009); however, in previous years our proxy data was limited since expenditure data
was only collected from a subset of Census 2002 records. Furthermore, the following
new calibration targets were used in this analysis that were not used in previous analyses:
hay acreage, rice acreage, peanut acreage, sugarcane/sugar beet acreage, tobacco acreage,
nursery/floriculture acreage, cattle on feed inventory, milk production, egg production,
number of farms by eight economic classes, number of farms by nonestimate states, and
total number of farms. Although egg and milk production were technically used to create
calibration weights for the 2005 and 2006 ARMS III respondents, these data were not
collected on the Census 2002 and therefore could not be included in our previous
analyses. All of the other new calibration targets were added after 2006 to improve
further our nonresponse and undercoverage adjustments.

All of these variables show a significant reduction in bias levels using a paired t-test.
After calibration, five study variables had significant remaining bias. This result varied
from 2005 and 2006, where for each year only one estimate still had significant
remaining bias after calibration: fertilizer expense in 2005 and total sales in 2006 (Earp et
al., 2008 & Earp et al., 2009). While Fertilizer Expenses exhibited significant bias after
calibration in 2005, it did not in 2006, but did again in 2008. Total Sales no longer has
significant remaining bias after calibration in 2008. The following four estimates that did
exhibit significant bias after calibration in 2005 or 2006 continued to do so in 2008: Total
Acres Operated, Fuel and Oil Expenses, Machinery and Equipment Value, and
Operator’s Age. We explored the bias levels of all five estimates still exhibiting
significant bias after calibration at the regional level to determine if calibration performed
better or worse in certain regions. Overall nonresponse bias levels significantly decreased
for all estimates after calibration adjustment; however, at the regional level nonresponse
bias levels did increase after calibration adjustment in the South Region for Total Acres
Operated and Machinery Equipment Value, and in California, Indiana, and the Atlantic
Region for Operator’s Age.
As in 2005 and 2006, the estimated bias of livestock purchases remains the largest among
the study variables. Using only the base-sampling weights, this bias was highly
significant using all three test statistics. After calibration, although still large in
magnitude, the estimated bias was reduced to statistical insignificance in terms of all the
tests. For this variable, calibration continues to reduce the bias significantly, if not
completely.
4. Discussion
ARMS data are used by farm organizations, commodity groups, agribusiness, Congress,
State Departments of Agriculture, and the USDA. The USDA uses ARMS data to
evaluate the financial performance of farms and ranches, which influences agricultural
policy decisions. The Department also uses Phase III data for objective evaluation of
critical issues related to agriculture and the rural economy. Due to the broadness of the

ARMS Phase III data user community and the survey’s impact on agricultural
policy, it is crucial that the calibration process effectively adjusts for nonresponse
bias. Assuming that the adjustment process is even more effective than
demonstrated here using the actual ARMS III data, it appears that NASS is
appropriately addressing the issue of nonresponse bias in ARMS Phase III
through the calibration process. Furthermore, NASS has expanded the number of
calibration targets used since 2006 to include hay acreage, peanut acreage, rice
acreage, sugarcane/sugar beet acreage, tobacco acreage, nursery/floriculture
acreage, cattle on feed inventory, number of farms by states for which separate
estimates are not produced, total number of farms, and number of farms by eight
economic classes.
Using the 2007 Census data as a proxy for 2009 ARMS III data, we demonstrated that
although significant bias was exhibited using just the base sample weights, it was reduced
to insignificant levels for the following 12 estimates through calibration weighting: Feed
Purchases, Total Production Expenses, Total Sales, Seed Expenses, Livestock Purchases,
Cropland Expenses, Hired Labor Expenses, Chemical Expenses, Acres Rented, Cropland
Acres, Government Payments, and Farm Type. Although the magnitude of the relative
bias of the mean estimate remained high for livestock purchases using the calibrated

weights, calibration reduced the magnitude of this bias to statistical insignificance (see
Table 1).
While calibration appears to be less effective than demonstrated in previous years (Earp
et al., 2008 & Earp et al., 2009), the analysis of the ARMS III 2008 sample was more
thorough, since the 2007 Census collected data on egg and milk production and
expenditure items from all farming operations, as opposed to just a subset, as was done
on the 2002 Census. Unlike previous years, we were able to replicate fully the 2008
ARMS III calibration process, using all 2008 ARMS III sampled operations that
completed the 2007 Census.
Limitations of this analysis include the inability to: 1) assess farms not covered or
responding to the 2007 Census of Agriculture; 2) correlate items between the 2008
ARMS III and the 2007 Census; and 3) recognize localized biases in the ARMS data
(tests were limited to persistent biases across regions).
Knowing that the analyzed data come from the 2007 Census and not from the 2008
ARMS Phase III survey does not limit, but strengthens the analysis. It allows us to focus
entirely on the impact of the nonresponse per se.

Table 1: Mean Comparisons and Indicated Biases for Matching Records Using Base Sampling Weights versus Calibrated Weights

Table 1 (Cont.): Mean Comparisons and Indicated Biases for Matching Records Using Base Sampling Weights versus Calibrated Weights

5. References
Earp, M.S., McCarthy, J.S., Shauer, N.D., & Kott, P.S. (2008), Assessing the Effect of
Calibration on Nonresponse Bias in the 2005 ARMS Phase III Sample Using Census
2002 Data. Research and Development Division Staff Report RDD-08-01, United
States Department of Agriculture, National Agricultural Statistics Service.
Earp, M.S., McCarthy, J.S., Schauer, N.D., & Kott, P.S. (2008). Assessing the Effect of
Calibration on Nonresponse Bias in the 2005 Agricultural Resource Management
Survey Phase III Sample Using Census 2002 Data. In JSM Proceedings, Government
Statistics Section. Denver, CO: American Statistical Association.
Earp, M.S., McCarthy, J.S., Schauer, N.D., & Kott, P.S. (2008). Assessing the Effect of
Calibration on Nonresponse Bias in the 2006 ARMS Phase III Sample Using Census
2002 Data. Research and Development Division Staff Report RDD-08-06, United States
Department of Agriculture, National Agricultural Statistics Service.
Kott, P.S. (2005), “Using Calibration Weighting to Adjust for Nonresponse and Coverage
Errors” Survey Methodology 32: 133-142.
Hopper, R. (2007). 2006 Agricultural Resource Management Survey (ARMS) Phase III
Survey Administration Analysis. Census and Survey Division Staff Report SAB-0714, United States Department of Agriculture, National Agricultural Statistics Service.
United States. Department of Agriculture. (2007). 2007 Census of Agriculture, Vol. 1,
Appendix A. Washington, DC: U.S. Department of Agriculture.
United States. Department of Education. 2003. National Center for Education Statistics
Statistical Standards. Washington, DC: U.S. Department of Education.
United States. Executive Office of the President. 2006. Office of Management and Budget
Standards and Guidelines for Statistical Surveys. Washington, DC: U.S. Executive
Office of the President.

6. Appendix

Table A-1: Census 2007 and ARMS Phase III 2008 Variable Correlations with Outliers
r2

.89603
( n = 22,720 )

.80287

Scatter Plots
Census

Total Acres Operated

r

Total Sales

.70927
( n = 21,987 )

.50306

Census

ARMS PHASE III

Acres Rented

.75192
( n = 21,276 )

.53654

Census

ARMS PHASE III

Cropland Acres

.88253
( n = 22,258 )

.77886

Census

ARMS PHASE III

Total Production Expenses

.87044
( n = 22,720 )

.75767

Census

ARMS PHASE III

Seed Expenses

.39646
( n = 22,720 )

.15718

Census

ARMS PHASE III

Fertilizer Expenses

.79022
( n = 22,720)

.62455

Census

ARMS PHASE III

Chemical Expenses

.73953
( n = 22,720 )

.54690

Census

ARMS PHASE III

Crop Expenses

.68397
( n = 22,720 )

.46781

Census

ARMS PHASE III

Livestock Purchases

.68277
( n = 22,720 )

.46618

Census

ARMS PHASE III

ARMS PHASE III

.86243
( n = 22,720 )

.74379

Census

Feed Purchases

Hired Labor Expenses

.81630
( n = 22,657 )

.66635

Census

ARMS PHASE III

Fuel & Oil Expenses

.91400
( n = 22,720 )

.83540

Census

ARMS PHASE III

Machinery & Equipment

.73324
( n = 22,720)

.53764

Census

ARMS PHASE III

Government Payments

.62929
( n = 12,890 )

.39601

Census

ARMS PHASE III

Operator’s Age

.34654
( n = 22,720 )

.12001

Census

ARMS PHASE III

Farm Type

.73019
( n = 22,720 )

.53318

Census

ARMS PHASE III

ARMS PHASE III

1. All correlations were significant at the .05 level.
2. Correlations were only estimated for ARMS respondents.
3. Outliers were flagged using DFFITS, Cook’s D, and studentized residuals and are shown in red.

Table A-2: Census 2007 and ARMS Phase III 2008 Variable Correlations without
Outliers
r2

.94389
( n =22,398)

.89093

Scatter Plots
Census

Total Acres Operated

r

Total Sales

.76027
( n =21,290)

.57801

Census

ARMS PHASE III

Acres Rented

.89592
( n =20,977)

.80267

Census

ARMS PHASE III

Cropland Acres

.96328
( n =21,401)

.92791

Census

ARMS PHASE III

Total Production Expenses

.89508
( n =22,367)

.80117

Census

ARMS PHASE III

Seed Expenses

.72660
( n =22,526)

.52795

Census

ARMS PHASE III

Fertilizer Expenses

.85389
( n =22,007)

.72913

Census

ARMS PHASE III

Chemical Expenses

.83845
( n =22,141)

.70300

Census

ARMS PHASE III

Crop Expenses

.87174
( n =22,309)

.75993

Census

ARMS PHASE III

Livestock Purchases

.59306
( n =22,583)

.35172

Census

ARMS PHASE III

ARMS PHASE III

.86233
( n =22,382)

.74361

Census

Feed Purchases

Hired Labor Expenses

.89342
( n =22,356)

.79820

Census

ARMS PHASE III

Fuel & Oil Expenses

.86510
( n =22,020)

.74840

Census

ARMS PHASE III

Machinery & Equipment

.75614
( n =21,978)

.57175

Census

ARMS PHASE III

Government Payments

.76001
( n =12,054)

.57762

Census

ARMS PHASE III

Operator’s Age

.63436
( n =20,765)

.40241

Census

ARMS PHASE III

Farm Type

.90607
( n =21,149)

.82096

Census

ARMS PHASE III

ARMS PHASE III

1. All correlations were significant at the .05 level.
2. Correlations were only estimated for ARMS respondents.