ARMS Progress Report

ARMS Progress Report
USDA’s National Agricultural Statistics Service
and Economic Research Service
Respond to Recommendations by the
Agricultural Resource Management Survey Review Panel

July 2015

National Agricultural
Statistics Service

United States
Department
of
Agriculture

Economic
Research
Service

NASS and ERS Respond to
ARMS Review Panel Recommendations

In 2008, the Committee on National Statistics (CNSTAT) of the National Research
Council released the findings and recommendations of an independent review of
USDA’s Agricultural Resource Management Survey in Understanding American
Agriculture.1 The review was requested by the National Agricultural Statistics
Service and the Economic Research Service as part of a program of continuous
improvement for ARMS.
Senior executives at the National Agricultural Statistics Service (NASS) and the
Economic Research Service (ERS), two USDA agencies that jointly manage the
Agricultural Resource Management Survey (ARMS), reviewed the recommendations
and developed an implementation strategy based on a cost/benefit analysis. NASS
and ERS continually reevaluate resources and priorities and will continue to respond
to the recommendations as resources allow.
The ERS/NASS ARMS Steering Committee was formed based on the numerous
recommendations from this review. The committee, whose members are ARMS
managers and specialists from both agencies, meets monthly to discuss survey
issues and solutions.
RECOMMENDATIONS and RESPONSES
In the pages that follow, the review panel’s recommendations are presented as
they appeared in the report along with actions NASS and ERS have taken in
response as of February 2014. Updates will be issued at least annually, and more
frequently as progress warrants, and will be posted to the “Independent Reviews”
box on the NASS Surveys Web page.
Data Integration and Relevance
CNSTAT Recommendation 2.1: The Natural Resources Conservation Service
[NRCS], NASS, and ERS should engage in a focused research and testing program
and use experience with integrating the Conservation Effects Assessment Project
and ARMS to assess the feasibility of integrating ARMS with other surveys and data
sources.
NASS/ERS Response: The team formed to examine ARMS/census of
agriculture integration succeeded in improving the joint ARMS/census data
set for 2012. After an extensive effort that looked at all data items common
to ARMS and the census, the team proposed ways to align the concepts and
questions asked on the two data collection instruments. The
recommendations were accepted by NASS and ERS senior management, and
1

National Research Council (2008). Understanding American Agriculture: Challenges for the Agricultural Resources
Management Survey. Panel to Review USDA’s Agricultural Resource Management Survey. Committee on National
Statistics, Division of Behavioral Social Sciences and Education. Washington, DC: The National Academies Press.

Revised April 2015

Page 3

NASS and ERS Respond to
ARMS Review Panel Recommendations

resulted in a shortened form for 2012. The link with the Conservation Effects
Assessment Project was facilitated at the time by new funding from the
Natural Resource Conservation Service (NRCS). The current work with NRCS
on watersheds does not easily fit into the context of the ARMS program.
Status: Completed. Integration with census was of high importance and
has been completed for the 2012 Census of Agriculture. No other integration
is relevant in the current NASS program. If additional funding is received for
other survey programs that address ARMS-related content, NASS will pursue
opportunities for survey integration.
CNSTAT Recommendation 2.2: In preparation for funds becoming available for a
longitudinal design of ARMS, ERS and NASS should systematically conduct research
and explore the need for and feasibility of obtaining panel data from ARMS.
Furthermore, as a test of the power of such information, more use should be made
of the existing longitudinal microdata from the census of agriculture. One possible
approach would be to create a pseudopanel of such data. Another would be to
make a retrospective link between the census of agriculture and a year of ARMS.
NASS/ERS Response: ERS has done some research on linking ARMS and
census in the context of measuring structural change in lieu of pursuing a
panel data collection. Additional work will be primarily done by ERS with
access to NASS ARMS and census data. NASS recently obtained access to
census of agriculture data prior to 1974; once NASS finishes converting them
into a compatible file format, they will be available for ERS research in
structural change.
Examples of ERS research projects using linked census records and research
projects that link ARMS and census of agriculture records include:
Linked Census Records:
T. Kirk White, B, Kirwan, and Y. Uchida. "Aggregate and Farm-level
Productivity Growth in Tobacco: Before and After the Quota Buyout". Amer.
J. Agricultural Economics, forthcoming (2012).
Weber, Jeremy, and Nigel Key. “How Much Do Decoupled Payments Affect
Production?” Amer. J. Agricultural Economics 94 (2012).
Hoppe, Robert, J. MacDonald, and P. Korb. “Small Farms in the United
States: Persistence Under Pressure.” USDA Economic Research Service
Economic Information Bulletin No. EIB-63. 2010.
O’Donoghue, Erik, Michael Roberts, and Nigel Key. “Did the Federal Crop
Insurance Act Alter Farm Enterprise Diversification?” J. Agricultural
Economics 60 (2009).

Revised April 2015

Page 4

NASS and ERS Respond to
ARMS Review Panel Recommendations

Key, Nigel, and Michael Roberts. “Government Payments and Farm Business
Survival.” Amer. J. Agricultural Economics 88 (2006).
Linked ARMS and Earlier Census Records:
Kirwan, Barrett. “The Incidence of Agricultural Subsidies on Farmland Rental
Rates.” J. Political Economy 117 (2009).
Status: Completed for post-1974 Census of Agriculture data. Linkage
has been completed between the current ARMS sample and previous
censuses of agriculture back through 1974. Linked data sets are available,
with approved written agreements, for ongoing analyses by ERS and
academic researchers using the NORC Data Enclave at the University of
Chicago. Once census data prior to 1974 are available in a compatible file
format, additional linkages will be made available to researchers.
Survey Management
CNSTAT Recommendation 3.1: The ARMS program should have structured
mechanisms in place for stakeholder feedback and discussion on ARMS, beyond
what is currently done, such as organized stakeholder forums, with some obligation
to respond. Specifically, USDA should solicit input in developing the survey from
stakeholders from within USDA and from other government agencies, universities,
professional associations, and the private sector.
NASS/ERS Response: An ARMS data users’ conference was held in
conjunction with the February 2009 Ag Outlook Forum. A webinar was
conducted in spring 2009 and a data users forum was held at the Agricultural
and Applied Economics Association (AAEA) in August 2009. The NASS LongRange Planning Team requested input from data users in the agricultural
community during 2009. Stakeholders provided significant input both before
and after the chemical use component of ARMS was reinstated in summer
2009. NASS continually seeks input from data users at various trade
association meetings, often setting up forums at those meetings to discuss
surveys relevant to the stakeholder group. Comments on the three phases of
ARMS are also accepted at NASS annual Data Users’ Meetings.
The ARMS briefing room on the ERS Web site provides an opportunity for
stakeholder feedback regarding data characteristics, use of the information
for statistical purposes, and questionnaire content. ERS receives 40 to 60
inquires annually from this facility. Feedback is systematically reviewed by
the ERS/NASS ARMS Steering Committee.
In 2011, an external panel of experts in farm financial analysis was
assembled to conduct a comprehensive review of the ARMS process for
constructing financial statements and to provide recommendations regarding
Revised April 2015

Page 5

NASS and ERS Respond to
ARMS Review Panel Recommendations

possible changes to questionnaire content, variable summarization methods,
and data collection procedures. The external panels received briefings on the
ARMS process, asked questions of the process, and then met to discuss
possible recommendations. An AAEA symposium was held to further vet
recommendations. A special issue of the Agricultural Finance Review is in
progress to publish the outcome of the review and panel recommendations
and the ERS/NASS ARMS Steering Committee will develop a response to the
panel.
ERS staff contacts academic animal scientists and economists, extension
staff, other government agencies, and commodity groups during the
development of community of practice livestock versions, and solicits their
advice on pressing issues and specific question formulations. The efforts have
been expanded and systematized since the ARMS review.
Status: Completed. NASS will conduct an ongoing annual program to
solicit stakeholder input on the ARMS three-phase program and report to the
Office of Management and Budget (OMB) in its Paperwork Reduction Act
(PRA) submissions. The special issue of the Agricultural Finance Review

was published in July of 2012 and is available via Emerald Subscription
http://www.emeraldinsight.com/journals.htm?issn=0002-1466.

CNSTAT Recommendation 3.2: The NASS Advisory Committee on Agriculture
Statistics should expand its scope to include an annual review of ARMS.
NASS/ERS Response: The NASS Advisory Committee is organized to review
ARMS content and methodology. In addition, a subgroup was formed in
February 2008 to strengthen outreach efforts. The 2010 Advisory Committee
meeting was not held because it had not yet been re-chartered by USDA. The
ARMS was reviewed at the 2012 Advisory Committee meeting in March.
Status: Completed. NASS will continue to conduct a review of the ARMS
program at annual meetings of the NASS Advisory Committee on Agriculture
Statistics and report to OMB in its PRA submissions.
CNSTAT Recommendation 3.3: ERS and NASS should establish an ongoing,
jointly sponsored, and appropriately funded methodology research and
development program. Such a program should provide adequate resources to
support current and future research, development, and statistical analysis needs
throughout the implementation of ARMS and to assess and manage the quality of
the data. If new funds cannot be obtained, funds from existing programs must be
reallocated.
NASS/ERS Response: In FY 2009 – FY 2011, NASS redirected funds to invest
$1.2 million in a cooperative agreement with the National Institute of
Revised April 2015

Page 6

NASS and ERS Respond to
ARMS Review Panel Recommendations

Statistical Sciences to address three high-priority projects. Two had
particular impact on ARMS – one looked at a multivariate approach to
imputation for the ARMS Phase III data (to which ERS staff contributed); the
other investigated the large difference between farm numbers in the 2007
June Area Survey (JAS) and the 2007 Census of Agriculture (JAS farm
numbers are the sample control for ARMS estimates). Both projects involved
NASS staff working with academics, doctoral candidates, and recent postdocs in an effort to bring in technical expertise that NASS did not have. Both
projects were successfully completed in June 2011. NASS is in the process of
developing implementation plans for the multivariate imputation; changes
have already been made in operational procedures relevant to the JAS.
NASS has additionally developed cooperative agreements with the University
of Florida (Malay Ghosh and Linda Young), Iowa State University’s Center for
Survey Statistics and Methodology (Sarah Nusser, Jae, Kwan Kim, Cindy Yu,
and Zhengyuan Zhu), the Joint Program in Survey Methodology (Frauke
Kreuter), Washington State University (Don Dillman and Dana Moore),
University of Nebraska (Jolene Smyth and Kristen Olson), American
Statistical Association Research Fellowship (Partha Larhiri), and others to
continue to bring in academic experts to enhance research contributions and
to develop NASS staff. In 2010-11, NASS successfully recruited six doctorallevel mathematical statisticians or survey methodologists to build a stronger
research base in the organization; the agency expects to hire three more in
2012. These researchers are working on a number of projects to strengthen
NASS’s general foundation for statistical and survey research. Some are
specifically assigned to ARMS-related research.
In April 2012, NASS and ERS finalized a joint multi-year (2012-2016) ARMS
Research Plan.
Status: Completed. See the ARMS Research Plan. Update as of May
2014: NASS processed both 2011 and 2012 ARMS Phase III data through
both Iterative Sequential Regression (ISR) Imputation Methodology and
operational imputation methodologies. Summarized results under these
methodologies were compared, with favorable results. A similar comparison
is currently being conducted using 2013 ARMS Phase III data. Assuming
favorable results are again obtained, ISR imputation methodology will be
operational for the 2014 ARMS III data.
Update as of March 2015: ERS has conducted research on use of
multivariate imputation for missing items that are not imputed by NASS.
Initial results, for farm debt, were presented at the 2014 AAEA meetings
(See: http://ageconsearch.umn.edu/handle/169401). A team consisting of
two recently hired ERS economists and two North Carolina State faculty are
currently working under a cooperative agreement to develop procedures for
household items that can be effectively implemented in the survey process.

Revised April 2015

Page 7

NASS and ERS Respond to
ARMS Review Panel Recommendations

ERS is also embarked upon a project to assess non-response bias in the
household section of ARMS. Farms that do not respond to the household
section of ARMS do provide demographic and other household information on
the Census of Agriculture, and the project will use that information to
evaluate systematic differences in household attributes among respondents
and non-respondent to the ARMS household section.

CNSTAT Recommendation 3.4: NASS and ERS should commit resources to
developing a five-year plan tied to the census of agriculture for ARMS content,
coverage, and methodology. The agencies should develop measures to control
changes during the five-year period to minimize disruptions to the time series of
the core content in ARMS.
NASS/ERS Response: The content of Version 5 of the questionnaire for
ARMS Phase III (commonly referred to as the CORE version), which is
designed for mail, has been stable for almost ten years. Solicitation for input
to the content for the 2012 Census of Agriculture was done in 2009/2010 and
ERS took the opportunity to respond. A large ERS-NASS effort examined the
content of both data collections to align similar data items. These changes
have been made in the 2012 Census of Agriculture and will be made in the
2012 ARMS.
The time series disruptions after the 2007 Census of Agriculture were
primarily due to a shift in the number of farms from previously published
estimates. The June Area Survey is the primary survey source for
establishing the annual number of farms estimates. To address this issue,
NASS now: 1) puts increased emphasis during enumerator training on
screening the JAS frame tracts for agricultural production, 2) provides
additional administrative information to the enumerators that may be useful
during screening, and 3) provides additional time for screening data
collection. NASS has also implemented list frame maintenance procedures
that will facilitate better tracking of changes over time to the list frame
records. This will enhance the coverage and quality of the NASS frame for
all NASS surveys including, in particular, ARMS and the census of agriculture.
See responses to recommendations 5.1, 5.2, 5.3, 5.4, 5.5, 6.3, 6.4, 6.5, 6.6,
6.7, 6.9, 7.3 and the ARMS Research Plan for information on methodology
research.
Status: Completed. See the ARMS Research Plan. Update as of May
2014: After the 2007 Census, a Farm Number Research Project was
conducted that focused on identifying why the farm number indications from
the June Area Survey were low. A key findings from the project indicated
Revised April 2015

Page 8

NASS and ERS Respond to
ARMS Review Panel Recommendations

that our field enumerators were not following the instructions consistently
across all states in the screening of tract operators. As a result Agricultural
tracts were being coded up as Non-Ag Tracts. Several national training
schools were held to re-iterate the June Area Frame methodology and the
proper procedures needed to be followed for screening area segments to
identify tract operators. As a result of the training the number of agricultural
tracts has increased over the last several years which in turn led to an
increase in the number of farms direct expansions.
The largest undercount is for the number of farms. As measured in the 2012
Census 12.3 percent of the total adjustment is from undercoverage, while
only 3.4 percent of the Land in Farms adjustment is from
undercoverage. The JAS is critical in the measurement of the coverage for
the Census of Agriculture and for several other agriculture surveys. Major
research efforts have been conducted by the Research and Development
Division during the past four years to understand and model the undercount,
misclassification and non-response. As a result of the research, new
methodologies to adjust farm counts based on a Capture-Recapture
methodology have been implemented for the Census of Agriculture. The
methodology encompasses four sources of error, non-response, imputation,
misclassification and coverage. The work is now being extended into the
estimation process for the annual number of farms publications.

Sample and Questionnaire Design
CNSTAT Recommendation 4.1: The methodology research and development
program the panel recommends should systematically (1) evaluate current
instruments and practices, (2) collect data that inform both the revision of existing
items as well as the creation of new items, (3) test revised instruments before they
are put into production, (4) use experimental control groups to evaluate the
differences between the old and new questionnaires, (5) improve understanding of
respondent record-keeping practices and their effect on survey quality, and (6)
designate a subsample of the existing ARMS sample for research and testing
purposes. Key parts of this work would best be conducted in a cognitive or usability
laboratory facility. It would be enabled by obtaining a generic clearance from the
Office of Management and Budget for testing of all phases of the survey to allow for
broader cognitive testing, evaluate the quality of data reported in response to each
question, and evaluate the impact of mode of data collection across the three
phases.
NASS/ERS Response: NASS now has an OMB-approved generic clearance
docket (OMB Control # 0535-0248), which is used to do testing and
evaluation of NASS questionnaires. A variety of testing methods, including
cognitive testing, focus groups, split sample field tests, etc., are used to test
Revised April 2015

Page 9

NASS and ERS Respond to
ARMS Review Panel Recommendations

ARMS and other NASS surveys. NASS does not plan to create a cognitive
laboratory facility due to the geographic dispersion of farm operators needed
for testing. As is typical in establishment surveys, most testing is conducted
with onsite visits. NASS is using the OMB-approved generic clearance docket
to evaluate current instruments and practices (item 1) and to test revised
instruments before they are put into production (item 3).
NASS conducted an extensive analysis of imputation for the 2007 Census of
Agriculture and then used the analysis to inform questionnaire design for the
2012 Census of Agriculture and the 2012 ARMS, which have many of the
same questions (item 2). See also the response to Recommendation 3.4.
NASS currently uses an experimental control group to evaluate differences
between data reported on ARMS mail and field versions to determine whether
less detailed information obtained on mail can substitute for the
disaggregated detail on the field version (for the 2011 data). NASS will
continue to use the experimental design approach to assess questionnaire
differences (item 4).
NASS has hired an individual with prior experience with agricultural data to
lead a project on designing data collection methods for large and complex
operations across its surveys, including ARMS. This could involve
examination of respondent record-keeping practices, but past research in this
area was not productive (item 5).
A subsample of ARMS for research and testing purposes will be considered
when there are sufficient new ARMS initiatives to justify this mode of testing
(item 6).
Update as of February 2014: ERS and NASS have research underway to
study the challenges posed for USDA statistical survey programs by changing
technology and farm structure, particularly the shift of production to larger
and more complex farms. A case study was developed and presented to a
workshop of large commercial farmers, held at UC/Davis in March of 2013.
It identified the key issues and barriers in eliciting response from large and
complex producers and alternative treatment paradigms. The presentation
allowed ERS and NASS staff to engage with large and complex producers, to
market our surveys to them, and to engage them in discussions on ways to
elicit data more efficiently. Staff will continue these discussions with
producers from the workshop, and we intend to explore further workshop
opportunities in other programs.

Revised April 2015

Page 10

NASS and ERS Respond to
ARMS Review Panel Recommendations

Status: Completed. The ARMS Research Plan identifies the components of
this recommendation that are of most importance in the next 5 years.
CNSTAT Recommendation 4.2: ERS and NASS should improve the consistency of
variables across ARMS versions and over time.
NASS/ERS Response: NASS developed and expanded the Questionnaire
Repository System that allows for improved standardization of variables
across ARMS questionnaire versions and over time. Consistent master
variable names are shared across questionnaire versions. The same master
variable names that are used to generate questionnaires are used to
populate the Data Warehouse. This also facilitates re-use of these master
variable names from year to year, enabling researchers to consistently query
using the same names.
ERS provides metadata and other documentation that informs ARMS data
users on constructing variables across questionnaire versions. See, for
example, this list of variables and the Phase III summary listing and
description of classification variables in Attachment B.
Status: Completed. Current ARMS metadata are recorded in the NASS
Questionnaire Repository System as part of the agency's operational
efficiency measures. ERS provided historical information.
CNSTAT Recommendation 4.3: NASS and ERS should explore the collection of
auxiliary information on a formal basis, as well as feasibility of enriching the ARMS
data files with information from administrative data sources, geospatial data, and
the like.
NASS/ERS Response: ERS and NASS are participating in an OMB-led
initiative to incorporate selected administrative data into surveys, and will
evaluate opportunities with regard to current ARMS questions. NASS is a key
participant in a USDA effort to synchronize reporting of administrative
(program) data for the Farm Service Agency (FSA), the Risk Management
Agency (RMA), and the Natural Resources Conservation Service (NRCS) that
is seeking common definitions and reporting. The NASS role has been to
inform the data development process. Ultimately the administrative data will
be of more value for developing agricultural production and conservation
statistics – several components addressed by ARMS. NASS has also made
progress in developing the Cropland Data Layer (CDL) using geospatial data
that provide end-of-season crop acreage estimates, with staff researching
the development of yield estimates for major commodities. These data could
feed into the ARMS database.
Status: In progress. As administrative data from the USDA project become
available, NASS will assess their use in ARMS. NASS continues to assess
applications of its geospatial data for its survey programs, including ARMS
Revised April 2015

Page 11

NASS and ERS Respond to
ARMS Review Panel Recommendations

and will conduct relevant research as opportunities become available. See
the ARMS Research Plan.
Data Collection
CNSTAT Recommendation 5.1: ARMS should use automated means to collect
paradata on interviewer assignments to cases, the relationship between the
interviewer and the sample farm respondent (i.e., whether they know each other),
demographic characteristics of the interviewer and the characteristics of the sample
farms for nonrespondents that are coordinated with information obtained for
respondents, either through the interview or interviewer observation. These
paradata could be used to determine the need for additional research on the impact
of the relationship between the interviewer and the respondent on the quality of
answers. This data collection can best be facilitated using computer-assisted
technologies.
NASS/ERS Response: The use of paradata in managing the respondentinterviewer interaction is best accomplished using computer-assisted
technologies. NASS initiated an operational efficiency in FY 2010 to pilot the
use of computer-assisted reporting in the field using personal enumeration
devices. The Apple iPad was selected for this purpose, using wireless
broadband transmission. Prototypes were developed; as of mid-2012field
offices in 18 states are equipped with iPads. Initially the iPads are being used
for questionnaires available with Web instruments. Once iPads are
implemented in all states, it will be feasible to use paradata for managing
field interviewing. During this implementation time-period, ARMS instruments
will be designed for access on the iPad. NASS is also developing a system to
facilitate the use of paradata on the iPADs. This can include scores from
recently developed ARMS nonresponse propensity models. See
recommendations 5.5 and 6.3.
Status: In progress. NASS has begun to develop the systems that will
facilitate the use of paradata, designing the systems specifically for this use.
As systems are implemented, paradata will be used in managing the
interview process. See the ARMS Research Plan. . Update as of May 2014:
All Field Offices are equipped with CAPI instruments but updates are needed
to the system to allow for a more interview friendly application to complete
the complex ARMS surveys. Less than 2% of the 2014 ARMS survey was
completed using the CAPI instrument.

CNSTAT Recommendation 5.2: NASS should systematically explore the
consequences of interviewer departures from standardization in the interview. To
facilitate this, NASS should collect paradata on the frequency with which
interviewers follow the order of the questionnaire, read questions as worded,
Revised April 2015

Page 12

NASS and ERS Respond to
ARMS Review Panel Recommendations

provide clarification, and similar indications of departures from standardized
procedures.
NASS/ERS Response: To analyze departures from standardization by
interviewers, the ARMS interviews would have to be recorded and analyzed.
NASS currently does not have the systems in place to allow this, but is
working with the Census Bureau to obtain the use of the computer audiorecorded interview (CARI) system developed by RTI International under
contract to the Census Bureau. Once the system is available, it should be
feasible to implement it on any NASS surveys collected by field enumerators
on an iPad.
In the interim, NASS and ERS invest in an annual national workshop for the
field statisticians that focus on standard data collection, edit, and analysis
procedures. Participants at the national workshop in turn hold local
workshops where standard procedures are taught to the interviewers. These
workshops provide a platform to strengthen the standardization efforts and,
in turn, result in improved data quality through standardized editing. Future
costs analysis will also be improved through standard data collection
procedures.
Status: In progress. See ARMS Research Plan for research NASS and ERS
will conduct prior to implementing CARI. In the interim, NASS continues to
instruct NASDA Supervisors to monitor data collection procedures and reinterview a small percentage of respondents for quality check purposes.
NASDA Supervisors must complete the quality check forms and return them
to the NASS Regional Field Office for review. Update as of May 2014: The
ARMS User’s Guide is published and available on the ERS website. In 2014,
NASS conducted a proof of concept project using the Census Bureau’s CARI
system for the Agricultural Labor Survey. Although the project showed that
a CARI system can provide beneficial information about the quality of
questionnaire instruments and interviewer behavior, that particular CARI
system does not fit into NASS’s information technology infrastructure. NASS
is currently looking at other CARI systems that may fit our needs and
integrate with our call centers’ telephone systems.

CNSTAT Recommendation 5.3: NASS should use available analytic tools, for
example, cognitive interviews, interviewer debriefing, recording and coding of
interviews, and re-interviews, to investigate the quality of survey responses.
NASS/ERS Response: Enumerator training and quality assurance follow-ups
have been expanded.

Revised April 2015

Page 13

NASS and ERS Respond to
ARMS Review Panel Recommendations

NASS plans to initiate methodological research on total survey error once a
PhD statistician is hired with this competency. The Research and
Development Division has added a PhD statistician whose background is in
quality control. One of his assignments is to assist in setting up quality
control measures for our data collection process, especially with respect to
the newly established NASS National Operations Center (NOC), which began
collecting data during the last quarter of 2011 on a limited number of
surveys.
In FY 2013, NASS expects to use the NOC in lieu of the Census Bureau’s
National Processing Center to process the ARMS data. Thus NASS will
implement quality procedures developed for the NOC. Once CARI is
operational and NASS has developed a computer-assisted interview
instrument for ARMS, it will be feasible to record and code interviews. This
may not occur until after the 2012 Census of Agriculture. See also the
response to Recommendation 5.2.
To complement the longer-term CARI solution and supplement cognitive
interviews, NASS will utilize interviewer debriefing and training for field staff
and interviewers to investigate and address the quality of survey responses.
Status: In progress. Staff are in place to address this recommendation as
the required systems become operational. See the ARMS Research Plan.
Update as of June 2015: In May 2014, NASS trained field staff in several
states on cognitive interviewing methods and procedures. Having a trained
cadre of cognitive interviewers makes conducting cognitive interview projects
more cost and time effective. So far, these trained staff have conducted
cognitive interviews for several NASS surveys, and will likely be used for
ARMS cognitive interviews in the future. Additional NASS field staff will be
trained on cognitive interviewing in July 2015.
CNSTAT Recommendation 5.4: NASS should move to computer-assisted
interview and possibly Web-based data collection, after research and testing to
determine possible effects of the collection mode on the data. Computer-assisted
personal interviews and Web-based data collection will provide opportunities to
increase timeliness, improve data quality, reduce cost, and obtain important
paradata.
NASS/ERS Response: Web-based data collection is available to about one
half of the ARMS sample nationally. NASS utilized Morae usability testing
software to test computer- based instruments. Implementation and testing of
computer-assisted personal interviewing (CAPI) began in fall 2009 with the
assignment of a CAPI project manager. Due to the complex nature of ARMS,
with numerous tables and interrelated instrument designs, CAPI
implementation will be incremental over the next few years.

Revised April 2015

Page 14

NASS and ERS Respond to
ARMS Review Panel Recommendations

Status: In progress. NASS is in the process of putting this
recommendation in place with its operational efficiency measures. The
application of CAPI to ARMS is addressed in the ARMS Research Plan.
Update as of June 2015: Completed. Web-based and CAPI data collection
was available for all the 2014 ARMS sample and will be available in all future
ARMS surveys.
CNSTAT Recommendation 5.5: NASS and ERS should develop a program to
define metadata and paradata for ARMS so that both can be used to identify
measurement errors, facilitate analysis of data, and provide a basis for
improvements to ARMS as part of the broader research and development program
the panel recommends.
NASS/ERS Response: Start-up activities on defining and using paradata have
begun with a preliminary literature search to determine state of the art,
current applications, and general trends. This information will be used to
inform decisions about how to organize this effort.
Research has been directed to developing predictive models to identify
operations highly likely to be non-respondents in ARMS and other surveys.
These models use census of agriculture data as a proxy for ARMS
respondents and non-respondents. During the 2011 ARMS, NASS is collecting
information on how the data from the predictive models can be used in ARMS
data collection. The potential impact of identified subsets of these likely nonrespondents on non-response bias has been evaluated and results have been
documented in research reports and external conference presentations.
In the future, expanded use of CAPI data collection on iPads and
development of an ARMS CAPI questionnaire should facilitate the capture of
additional paradata both directly and from interviewer observation. However,
current ARMS data collection does not include the routine capture of
paradata. Ultimately, NASS hopes to be able to use paradata to reduce nonresponse bias.
Interim and complementary responses to recommendations 5.2 and 5.3 will
be employed to provide a basis for improvement to ARMS until the longerterm solution can be implemented.
Status: In progress. Technology is being developed to move this initiative
forward. Until those systems are in place the use of paradata will not be very
effective. Testing will continue on how to best use the information from the
predictive models in data collection. See the ARMS Research Plan.

Revised April 2015

Page 15

NASS and ERS Respond to
ARMS Review Panel Recommendations

Nonresponse, Imputation, and Estimation
CNSTAT Recommendation 6.1: NASS should routinely report ARMS case
dispositions linked across survey phases to provide the foundation for appropriate
response rate calculations for Phases II and III.
NASS/ERS Response: This information has been reported in past years for
specific commodity versions of the ARMS Phase III survey in an internally
published document. This information will be expanded to cover all versions
of ARMS and will also be included in the Methodology and Quality Measures
document released to the public in August 2012 and annually thereafter.
The Farm Production Expenditures report published in August 2011 contains
a statement and link to additional information on survey methodology and
quality measures. Quality metrics include sample size, response rates,
coefficients of variation, and percent of estimate from respondents.
Status: Completed. ARMS case dispositions linked across survey phases
have been compiled and maintained within NASS since 2006. Starting in
August 2012, these tables will be included in the ARMS Phase III Survey
Methodology and Quality Measures document that is published along with the
Farm Production Expenditures report. These documents will be available to
the general public through the NASS Web page.
CNSTAT Recommendation 6.2: All published ARMS response rates for Phase II
and III should be calculated to reflect the nonresponse from the preceding
phase(s).
NASS/ERS Response: A new method of calculating the response rates to
reflect the nonresponse from previous phases will be developed. NASS has
always reported response rates for each individual phase of ARMS
independently, but this new method will provide a response measure that
covers all three phases. This information will be considered for inclusion in
the Methodology and Quality Measures document released to the public in
August 2012.
Status: Completed.
Update May 2014: NASS has calculated multiple “cumulative” response
rates and will evaluate those response rates and will be considered for
publication (pending any unforeseen issues) in the August 2014 Farm
Production Expenditures Methodology and Quality Measures Document.
Update June 2015: NASS will publish a “cumulative” response rate to the
data user’s manual on http://www.max.gov

Revised April 2015

Page 16

NASS and ERS Respond to
ARMS Review Panel Recommendations

CNSTAT Recommendation 6.3: The nature of the ARMS nonresponse bias should
be a key focus of the research and development program the panel recommends.
This research and development program should focus initially on understanding the
characteristics of nonrespondents.
NASS/ERS Response: NASS has and will continue to explore nonresponse
bias using predictive models built using census of agriculture data with the
current year ARMS data. These analyses have evaluated bias for key survey
estimates and the effect of NASS weighting procedures on bias. These
studies of nonresponse bias and additional analysis of respondent incentives
have been conducted and results have been published. NASS continues to
assess nonresponse bias. Studies to date have shown that current NASS
weighting procedures reduce or eliminate bias for most key survey items,
although, as described below in research conducted by ERS, the impact of
nonresponse adjustment on estimates is sizeable for some measures.
In 2008, research projects were completed to examine reasons for
nonresponse in Phase III of the 2006 ARMS. Studies were completed in
Washington and Louisiana, providing an opportunity to examine regional
differences. Item nonresponse tabulations are routinely circulated among
ARMS managers, and summary analyses are disseminated through a survey
research Web page).
Research is currently underway to evaluate the use of the information from
predictive models in data collection both to increase response and to
decrease non-response bias. Based on the results of current research, future
activities may focus on ways to use the nonresponse models to supplement
the current ARMS weighting procedures.
ERS research on nonresponse bias has focused on economic variables that
influence nonresponse, and the effects of nonresponse on economic analyses
using ARMS data. The research uses census responses from ARMS
nonrespondents, and finds that farm size plays an important role in
nonresponse. Accounting for nonresponse has very minor effects on most
coefficients analyzed in several econometric papers, but important (50%100%) impacts in a few. Moreover, standard econometric corrections for bias
do not work in all cases of concern. Continuing work aims to isolate the types
of measures for which bias will be important.
Status: Completed. Update February 2014: NASS is currently using the
nonresponse propensity models to identify likely non respondents. This
information is utilized when assigning data collection methods. Targeted
methods used for operations that were identified as likely non respondents
included in-person recruitment by more experienced NASS staff and
interviewers, providing publications and brochures, a drop off/pick up
Revised April 2015

Page 17

NASS and ERS Respond to
ARMS Review Panel Recommendations

methodology and emphasizing data uses that apply to specific types of
operations.
CNSTAT Recommendation 6.4: The research and development program should
continue NASS’s work on both public relations and incentives, and it should do so
with a focus on nonresponse bias, not simply nonresponse rate.
NASS/ERS Response: Much work has been done over the years on targeting
public relations materials toward specific groups in ARMS—in some cases,
those with historically low response rates. For several years, monetary
incentives have been used and researched for the ARMS core sample with
some success in incremental response rate increases. However, with the use
of nonresponse propensity scores (see recommendation 6.3), we plan to
utilize the incentive funds to conduct more targeted nonresponse avoidance
activities in lieu of its use to manage the debit cards.
Nonresponse propensity models can be used to identify highly likely nonrespondents before data collection begins. The models developed by NASS
identify multiple subgroups of highly likely non-respondents according to
farm production or operator characteristics. This will allow NASS to alter data
collection procedures, develop targeted publicity materials and incentives, or
alter interviewer assignments in focused nonresponse avoidance. Likely nonrespondents are currently being identified and a split sample experiment is
being conducted to evaluate whether response rates can be improved for
these operations. Similar to prior research, the impact of these nonrespondents on bias in key survey estimates will also be included. Results
may indicate that some groups of likely non-respondents have greater
impact on data quality and these would be the focus of future efforts.
Status: Completed. Update February 2014: The models are being
utilized in planning data collection and for follow-up non respondents.
CNSTAT Recommendation 6.5: The research and development program should
analyze whether there are differences in ARMS unit and item nonresponse rates
between census and non-census years, with an eye toward deciding whether
making ARMS mandatory would improve data quality.
NASS/ERS Response: The Research and Development Division performed a
detailed analysis of the item nonresponse rates for the 2006 and 2007 ARMS
Phase III. The report summarizing the analysis, published June 2012, looks
at item nonresponse in two different ways to account for the fact that
collection procedures at the time did not permit differentiating between valid
zeros, zeros that are imputed by an analyst, or zeros that were filled in by
data entry staff when no value was available during keying. In addition, a
change rate was calculated to examine the total number of changes to an
item. The report contains these three calculations for all variables collected in
ARMS Phase III and identifies the problematic items.
Revised April 2015

Page 18

NASS and ERS Respond to
ARMS Review Panel Recommendations

A relatively small number of items did not meet the OMB threshold. However,
the items that fell short were consistent across years. Most of these items
dealt with landlord and contractor expenses, values that may not be readily
available (or available at all) to the respondent (the operator). Some
manually imputed items were imputed one hundred percent of the time,
while one machine-imputation-eligible item, landlord’s property tax expense,
was imputed over half the time. The analysis also discovered several dozen
items that always get zero responses and many more that get only a few
responses. These variables are being or have been addressed by the
NASS/ERS Steering Committee in questionnaire design and editing
procedures; they will be evaluated annually as part of post-data-collection
and summary evaluation procedures. At this time, the Committee believes
ARMS should remain a voluntary survey.
Status: Completed. There is no current initiative or external effort to
evaluate mandatory reporting (nor is there expected to be) on ARMS.
CNSTAT Recommendation 6.6: The research and development program should
examine how questionnaire design and interviewing changes could reduce item
nonresponse.
NASS/ERS Response: Questionnaires are routinely pretested to ensure that
respondents can understand and answer ARMS items. In addition, field office
staff submit comments and suggestions for changes after data collection that
are used in the design of subsequent ARMS questionnaires and data
collection procedures.
Many questions that were the focus of testing and redesign on the 2012
Census of Agriculture also appear on ARMS questionnaires. Work on the
census has been done to identify the items with the most nonresponse and
this was used to determine the areas of the questionnaire that were the
focus of redesign. Item nonresponse in the 2012 Census of Agriculture will be
compared to the 2007 Census to determine the impact of those changes.
Status: Completed. Update February 2014: The testing and redesign of
questions from the 2007 Census were integrated into the ARMS
questionnaire. NASS and ERS continue to evaluate and make adjustments as
needed.
CNSTAT Recommendation 6.7: NASS and ERS should consider approaches for
imputation of missing data that would be appropriate when analyzing the data
using multivariate models. Methods for accounting for the variability due to using
imputed values should be investigated. Such methods would depend on the
imputation approach adopted.
NASS/ERS Response: NASS and ERS staff have worked together over the
past two years, and with academia as members of a cooperative research
Revised April 2015

Page 19

NASS and ERS Respond to
ARMS Review Panel Recommendations

venture with the National Institute of Statistical Sciences (NISS), to develop
an improved, multivariate approach to imputation for the ARMS Phase III
data. They have developed new imputation methodology that will be
incorporated operationally in the near future. The imputation procedure
samples imputations from a joint model that is constructed from a sequence
of conditional regression models known as iterative sequential regression.
The procedure is conducive for high dimensional problems since it allows for
flexible selection of a predictor function in each conditional model while
maintaining a valid joint distribution. The procedure will jointly impute for
more than 150 ARMS variables using models that were created using a
combination of economic expertise and automated variable selection
procedures. The product of the research is a system written in R
programming language that will incorporate multivariate imputation for key
ARMS variables into the NASS processing system.
ERS has developed a cooperative agreement with researchers at North
Carolina State University, to assist us in developing analyses and routines in
support of the implementation of improved imputation methodologies for the
variables that ERS imputes in the ARMS survey. These variable tend to be
non-negative, clustered at zero, and highly skewed, and are therefore not
directly amenable to the sort of iterative procedures being implemented by
NASS for imputation. However, there are a set of transformations that may
allow us to follow the NASS approach. ERS will engage with the NC State
team during the Spring of 2013.
Status: Research completed. R routine is being incorporated into the
NASS processing system. Update as of February 2014: Testing is
expected to occur in 2014 with implementation in 2015. Update as of June
2015: Multivariate imputation was used operationally for the 2015 survey.
CNSTAT Recommendation 6.8: All missing data that are imputed at any stage in
the survey should be flagged as such on files to be used for analysis.
NASS/ERS Response: After the ARMS review, NASS initiated tracking of all
item imputation computed by the machine imputation process. Any
manual imputation currently done by a field office statistician is not
traceable. If a field office statistician makes an update to the questionnaire
before the data are entered into our system, that update cannot be
differentiated from a value reported by the respondent. NASS business
processes are being updated and it is anticipated that in the future original
data will be captured before analyst review, which will result in all changes to
the data being captured. This will occur as NASS moves to using Apple iPads
for field interviewing, and as editing in CAPI becomes the same as editing by
field office statisticians through training and supporting documentation.
Forms returned by mail are scanned and keyed at a processing center so no
editing occurs by a field office statistician.

Revised April 2015

Page 20

NASS and ERS Respond to
ARMS Review Panel Recommendations

Status: Partially completed. System changes for machine imputation
have been made to respond to this recommendation. Training procedures
have been enhanced to emphasize appropriate use and notation of manual
imputation. Development of a CAPI instrument for the Apple iPad is planned
for 2015. See the ARMS Research Plan. Update as of June 2015:
Completed. As of 2015, NASS systems allow for the tracking of the origin of
the “current value” on the dataset for ARMS. This allows us distinguish
between reported, updated, imputed, and edited values on the dataset
moving forward.
CNSTAT Recommendation 6.9: NASS and ERS should provide more clarification
and transparency of the estimation process, specifically the effect of calibrations on
the assignment of weights and the resulting estimates.
NASS/ERS Response: NASS has assessed the impact of calibration weighting
used for nonresponse adjustment on nonresponse bias for several years
using census of agriculture data. These analyses show that calibration
substantially reduced bias for most key ARMS estimates. NASS specialists
have conducted seminars at ERS on the subject of calibration.
The ARMS Phase III Methodology and Quality Measures document published
for the first time for the 2010 survey contains a table that displays the
percent of the survey estimate that came directly from the respondents. The
converse of that number is the percent of the estimate that resulted from
weight adjustments due to calibration, indicating the impact that calibration
has on the survey estimates. Also included in the document are overall
survey response rates and the coefficients of variation for each published
estimate. Each provides the data user with a level of quality and precision in
the ARMS Phase III estimates. The document is available to the public at the
same time as the Farm Production Expenditures publication. Prior to this
complete quality measures and methodology document, the coefficients of
variation for the national estimates were included in the annual publication
since the 2008 survey release.
ARMS data summaries made available on the ERS Web site include a
measure of statistical reliability for each variable presented. In addition, the
site provides survey documentation including enumerator manuals, survey
procedures, data dictionary, and other reference material. A data user's
guide is under development, chapters of which are available upon request.
Status: Completed. Update as of February 2014: See quality measures
on the NASS and ERS Web sites. Update as of May 2014: The ARMS User’s
Guide is published and available on the ERS website.

Revised April 2015

Page 21

NASS and ERS Respond to
ARMS Review Panel Recommendations

Methods of Analysis
CNSTAT Recommendation 7.1: NASS should continue to provide sampling
weights with the ARMS data set, combined with replication weights for variance
estimation.
NASS/ERS Response: A complete set of weights that include the calibrated
weights, version specific weights, and masked weights along with their
respective sets of replicate weights have been made available for many years
and this practice will continue.
Status: Completed.
CNSTAT Recommendation 7.2: NASS and ERS should continue to recommend
the design-weighted approach as appropriate for many of the analyses for users of
ARMS data and as the best approach for univariate or descriptive statistics.
NASS/ERS Response: NASS always recommends using the design-weighted
approach when data users attempt to utilize micro level ARMS data in other
research and analysis. All data requests are accompanied by an explanation
of the weights we recommend and why utilizing another weighting method
may not be accurate in representing the total population of farms at the
state, regional, or national level.
Status: Completed.
CNSTAT Recommendation 7.3: NASS should investigate and implement
improvements to the current jackknife replicates to make them more useful for the
types of analyses performed by users in ERS and other organizations. In particular,
NASS should increase the number of replicates and apply bounds to the magnitude
of the weight adjustments.
NASS/ERS Response: Based on research it conducted on this issue, NASS
has taken several steps to improve the jackknife replicate method of variance
estimation. Specifically: increasing the number of jackknife replicates from
15 to 30, limiting the magnitude of weight adjustments in calibration and in
the creation of replicate weights, posting a document describing methods
that may be employed to sharpen analyses derived from the method, and
undertaking research on improvements to the current method, with a
particular focus on the applicability of the estimator for analyses of
subsamples of the database.
Status: Completed.

Revised April 2015

Page 22

NASS and ERS Respond to
ARMS Review Panel Recommendations

CNSTAT Recommendation 7.4: NASS and ERS should investigate the feasibility
of providing sufficient information on the design and nonresponse characteristics of
ARMS, in order to perform design-based statistical analysis without using the
replicate weights and to allow users to incorporate design and nonresponse
information in model-based analyses.
NASS/ERS Response: Documentation is planned that will serve as a guide for
data users. (See response to recommendation 7.6.) A section of the guide
will address alternative statistical procedures reflecting the improvements in
imputation methods and use of replicates currently being investigated.
Status: In progress. NASS and ERS will report on the status of this
documentation to OMB in its PRA submission. This status document will be
updated annually or more frequently as progress warrants. It will be posted
to “the Independent Reviews” box on the NASS Surveys Web page. Status
Change: Completed. Update as of February 2014: ARMS User’s Guide is
in pre-publication phase at ERS and should be published online by May 2014.
CNSTAT Recommendation 7.5: ERS should build an enhanced level of in-house
survey statistics expertise, in cooperation with NASS. The specialized expertise in
both econometrics and survey statistics needed to accomplish this is currently not
present in ERS and is likely to require a significant effort in recruiting and training.
NASS/ERS Response: ERS has hired new staff with survey research expertise
and is continuing to recruit with econometric and survey research expertise
in mind. Since the ARMS review, ERS has hired two economists with
extensive survey experience or training, who have devoted part of their time
to survey research efforts (on imputation and on nonresponse). ERS has
hired a third economist with extensive survey experience to work on
database development and integration at ERS.
NASS has hired five doctoral-level survey statisticians and one survey
methodologist. NASS has also used a PhD economist in ongoing research
projects.
Status: Initial effort completed. ERS and NASS will continue to build on
this resource in accordance with other agency priorities. Update as of June
2015: Completed. NASS and ERS completed a joint training in June of 2015
for ERS researchers.
CNSTAT Recommendation 7.6: ERS and NASS should collaborate on writing a
Guide for Researchers for performing multivariable analyses using data from
complex surveys, particularly data from ARMS. In areas in which expertise is not
available for writing parts of such a guide, expertise should be sought from the
statistics and economics community, especially those with experience in the
analysis of survey data from complex survey designs.

Revised April 2015

Page 23

NASS and ERS Respond to
ARMS Review Panel Recommendations

NASS/ERS Response: ERS now provides copies of the interviewer’s manual,
as well as copies of the questionnaires, for each year of ARMS as part of the
documentation on the public Web site. ERS also provides all users with a set
of file documents on variable listings and definitions, estimation procedures,
and the structure of financial accounts. An ARMS User’s Guide, which
combines existing documentation memos and programs with new material
and an annotated table of contents along with an executive summary in an
organized framework, is being developed by ERS. Completed chapters are
already offered to data users. Chapters are posted on the ERS intranet site
for easy access by ERS staff, and are provided directly to external data
users, who receive some chapters when they first inquire about data access
and the others once they are granted access.
In addition the DaTUM committee identified three types of users: 1) The
casual user who wants to simply know what the survey is and wants access
to basic public data; 2) the advanced user of public data who digs deeper
into the data and studies more about the survey; 3) the researcher (both at
ERS and outside of ERS) that is authorized to use record level data. As a
departure from earlier work shown to OMB, the team now is working toward
a substantial update to the website to address the needs of users 1 and 2.
Mockups should be available for internal ERS review by the end of April 2013.
The website will provide a basic overview of the survey and a clear pathway
to the data for type 1 users and point to the large amount of publicly
available information for type 2 users. The type 3 user will have all the public
website material available and will be provided with additional resources that
address all in-depth issues of record level data use at the passwordprotected site Max.gov. New documents are planned where needed and the
team maintains its goal of launch no later than December 2013.
Status: Completed. Update as of February 2014: ARMS User’s Guide is
in pre-publication phase at ERS and should be published online by May 2014.
Update as of May 2014: The ARMS User’s Guide is published and available
on the ERS website.
Dissemination
CNSTAT Recommendation 8.1: ERS should continue to improve the ARMS Web
tool by providing summaries on more variables and more subsets from ARMS, and
to improve the ARMS extranet Web tool by adding the ability to link over years and
to more sophisticated models.
NASS/ERS Response: An ERS-NASS team is preparing a research and
documentation Web-based search tool that will enable interested users to
Revised April 2015

Page 24

NASS and ERS Respond to
ARMS Review Panel Recommendations

locate and download ERS and NASS survey research papers and annual
ARMS metadata summaries. ERS recently added a new set of tables detailing
participation in government programs to the Web tool. Additional tables are
under consideration.
Status: Initial effort completed. NASS and ERS will report on the Web
tool in its PRA submission to OMB.
CNSTAT Recommendation 8.2: USDA should consider extending the availability
of ARMS microdata through the Census Bureau research data centers to increase
access opportunities for using additional data sets and enabling researchers to
match ARMS files with other data sets.
NASS/ERS Response: ERS and NASS have joined the NORC Data Enclave
program at the University of Chicago. The Data Enclave expands ARMS
access opportunities for qualifying researchers in controlled on-campus
environments. It provides a confidential, protected environment within which
authorized researchers can access sensitive microdata remotely from their
offices, an approach that combines good researcher access with researcher
training and administrative support.
Currently 18 researchers representing 15 academic institutions are using the
Data Enclave to accomplish their research, increasing the value added of the
ARMS data collection through high-quality analysis, deeper insights into key
issues, and by tapping into a broader analytic community. These researchers
are presenting their findings at conferences and publishing them in
proceedings and journals. They are able to address questions at a more local
level than can be done directly at ERS. Participants have achieved greater
efficiency and lower costs by not having to undertake the time and expense
of travel to USDA offices, and the support burden on these offices has been
reduced. The Data Enclave is better suited than the Census Bureau research
centers for ARMS data. Researchers are enthusiastic about how their
analyses are facilitated, enabling them to collaborate with ERS in a more
productive way.
Status: Completed. NASS and ERS determined that the NORC Data
Enclave program better suited the agencies’ and their researchers’ needs and
have been developing this access mechanism.
CNSTAT Recommendation 8.3: ERS should provide more training for new data
users, including developing a data user manual, which also includes the
recommended guide on statistical estimation, and offering training workshops.
NASS/ERS Response: In 2010, ERS had an agency-wide two-day
comprehensive training for ARMS users including participation from NASS
and the Bureau of Economic Analysis. The workshop covered the uses of the
survey, its components, the links between the survey’s goals and
Revised April 2015

Page 25

NASS and ERS Respond to
ARMS Review Panel Recommendations

questionnaire design, and technical features of designing the survey,
developing a research database, and analyzing the data. Topics included
survey design and sample selection, weighting and calibration, data editing
and imputation, inference with complex survey data, and the creation of farm
income and wealth accounts from raw data. Another comprehensive training
is planned for 2013.
Status: Training will be offered based on demand and resources.
NASS and ERS will report on training as applicable in its PRA submission.
Update as of May 2014: The ARMS User’s Guide is published and available
on the ERS website.
Update as April 2015: ERS will be again be conducting a formal ARMS
training workshop, focused on the needs of new users, in June of 2015, with
presentations from NASS and ERS staff. The workshop will be aligned with
material from the ARMS User’s Guide, completed in 2014. ERS intends to
post powerpoint presentations and record sessions, so that others can use
the material.

CNSTAT Recommendation 8.4: Database management practices should include
a system for managing and reporting errors found by users, for ensuring the
consistent labeling of the codes for raw variables, and for using consistent names of
the ERS-created summary variables over time.
NASS/ERS Response: ERS maintains the capability to receive email
suggestions and notices regarding the ARMS data tool available on its home
page. Responses are reviewed by staff. An email address and telephone
number are provided for a member of the agency’s team for specific
questions regarding access, special tabulations, or other questions regarding
access and use of the data. See the ARMS Briefing Room.
Status: Completed.

Revised April 2015

Page 26

Attachment A – ARMS Research Plan

National Agricultural Statistics Service
and Economic Research Service
Agricultural Resource Management
Survey
Research Plan
FYs 2012 - 2016

June 2012

Attachment A – ARMS Research Plan

Timeline
2012
•

•
•
•
•
•

Testing – Annual questionnaires are routinely pretested to ensure that respondents can
understand and answer ARMS items. For 2012, conduct cognitive tests for soybeans and
wheat.
Nonresponse bias analysis – Annual studies are now part of the operational process by
our methods staff; in 2012, test 2011 data.
Test iterative sequential regression imputation methodology.
Large and complex farm project – Initial planning has begun and continues.
ERS and NASS will collaborate on research to explore the implications of expanding the
value codes used in ARMS.
ERS and NASS will analyze differences in questionnaire reporting for specific items
related to mode of data collection (mail versus personal interview).

2013
•

•

•
•

•

•

Large and complex farm project -- Begin testing for ARMS and census.
o Update February 2014: Pilot procedures for this project were developed and an
internal steering committee was formed.
Testing – Annual questionnaires are routinely pretested to ensure that respondents can
understand and answer ARMS items. For 2013, conduct cognitive tests for vegetables,
rice, and peanuts.
o Update February 2014: Testing was moved from Research and Development
Division to the Standards and Survey Development Methodology Branch thus
making it operational.
Nonresponse bias analysis – In 2013, evaluate bias in 2012 data.
Parallel test iterative sequential regression imputation methodology.
o Update February 2014: Parallel testing is being conducting for the 2014 survey
and will be operational in 2015.
Begin computer audio-recorded interviewing (CARI) system development, integration,
and testing. \
o Update February 2014: NASS started exploring the feasibility of adoption of
CARI and tested it on small surveys. Not operational at this time.
Develop three-phase response rate for ARMS.

2014
•
•

Begin animated graphical Internet displays for ARMS work.
Nonresponse bias analysis – In 2014, test 2013 data.

Attachment A – ARMS Research Plan

•
•
•

Initiate research on linking ARMS data to administrative data available through USDA’s
Acreage/Crop Reporting Streamlining Initiative (ACRSI).
Complete historic census data conversion for complex analysis.
Implement iterative sequential regression imputation methodology.
o Update June 2014: Parallel test was conducted with iterative sequential
regression imputation methodology and will be implemented for the ARMS 2015
survey.

2015
•
•

•

•

Begin to implement computer assisted personal interviewing (CAPI) for ARMS
questionnaires with table in Blaise IS (Internet Services) software for 2015 data year.
Testing – Annual questionnaires are routinely pretested to ensure that respondents can
understand and answer ARMS items. For 2015, the questions/commodities to be
cognitive tested are still to be determined. Conduct census and ARMS evaluation for
census year.
Assess the coordination effort to synchronize ARMS questions with the 2012 Census of
Agriculture report form. Use data from both the census of agriculture and ARMS to
determine edit and imputation rates and evaluate nonresponse.
Nonresponse bias analysis – In 2015, test 2014 data year.

2016
•
•

•
•

Use CARI for quality control in ARMS.
Testing – Annual questionnaires are routinely pretested to ensure that respondents can
understand and answer ARMS items. For 2016, the questions/commodities to be
cognitive tested are still to be determined.
Nonresponse bias analysis – In 2016, test 2015 data.
Automate collection of ARMS paradata.

Attachment A – ARMS Research Plan

Plan
1. Sample and Questionnaire Design
•

Solicit stakeholder feedback
o Prior to each reference period for data collection, NASS and ERS will engage
stakeholders for input on the ARMS three-phase program. Resulting changes will
be reported to the Office of Management and Budget (OMB) in its Paperwork
Reduction Act (PRA) submissions. Public comments will be solicited from the
Federal Register Notice, at NASS’s annual Data User’s Meeting, through the ERS
ARMS Briefing Room, and other events and means. Internal comments are
solicited through a formal request and response system and evaluated by the
ERS/NASS ARMS Steering Committee.

•

Test questionnaires
o Questionnaires are routinely pretested to ensure that respondents can understand
and answer ARMS items. Questions are considered for evaluation and redesign
each year. Questions can change depending on agricultural policies and structural
changes. In addition, field office staff submits comments and suggestions for
changes using E-2 forms after each survey data collection period. This
information is used to modify and make design changes to subsequent ARMS
questionnaires and data collection procedures. The need for testing each year
depends on content, timing, complexity, and resource constraints.
When NASS conducts large-scale tests, we use an OMB-approved generic
clearance docket (OMB Control # 0535-0248) to do testing and evaluation of
NASS questionnaires. In years when only minor changes are made to any of the
questionnaires, testing is limited to nine or fewer cognitive type interviews and is
not submitted to OMB for approval. A variety of assessment methods, including
cognitive testing, focus groups, split sample field tests, etc., are used to test
ARMS and other NASS surveys. An experimental control group is used to
evaluate differences between mail and field collected responses. Item nonresponse
and survey design are examined. Varied data collection methods are evaluated for
large and complex operations. The geographic dispersion of farm operators limits
the use of cognitive laboratory testing. As is typical in establishment surveys,
most testing is conducted with onsite visits. The OMB-approved generic
clearance docket provides a venue to evaluate current instruments and practices
and to test revised instruments before they are put into production.
Prior to the 2012 Census of Agriculture, extensive coordination was done to
synchronize questions on ARMS with the relevant census of agriculture
questions. After the 2012 Census of Agriculture, NASS will assess these common
questions using both census and ARMS data to determine edit and imputation
rates and evaluate nonresponse in 2015.

Attachment A – ARMS Research Plan

•

Develop CAPI instrument for Apple iPad (field data collection)
o Research and Development Division (RDD) is developing a data collection
instrument utilizing Blaise IS software for the area data collection in 2012. The
software is currently a beta version. Once this software is in production version
and NASS has tested questionnaires with table format questions, NASS will begin
full development of CAPI ARMS questionnaires. The plan date is 2015 for 2014
data.

•

Use administrative data in lieu of collecting data
o NASS is one of the four agencies involved with the Acreage/Crop Reporting
Streamlining Initiative within USDA. ACRSI is establishing data standards to be
used for the annual acreage reports collected by the Farm Service Agency (FSA)
and the Risk Management Agency (RMA). The data collected under ACRSI
standards will be available to NASS; however, these data will not be
fundamentally different from what’s currently available from FSA and RMA. The
challenge in using the data will be in mapping FSA data to specific NASS
operations sampled for ARMS. The research required to link FSA data to ARMS
operations will begin by April 2014.

•

Analyze detail data to determine what questions work using CARI (2016) and paradata
research related to the implementation of CARI.
o RDD is currently investigating a beta version of CARI software. Once the Census
Bureau accepts the software, NASS will integrate it into its system. NASS will
begin to design procedures to select portions of CARI interviews for question
review or interviewer coding as soon as software is available for research. Work
is expected to be completed in 2016.

2. Data Collection
•

Use of paradata
o NASS does not yet have systems in place for the automated collection of paradata
on interviewer assignments, interviewer characteristics, and their possible impact
on data quality in the interview. Systems to allow this should be in place in 2015.
NASS currently has little social science staff expertise to explore the impact of
characteristics of interviewer and respondents. NASS expects to hire more
research staff with this background and experience and to begin research in this
area in 2016.

•

Large and complex farm project
o After hiring a new staff member, RDD began initial planning in 2012 to
investigate alternative methods for collecting data on large and complex
agricultural operations. Initial implementation of any program for data collection
from impact operations will begin with a small pilot set of operations. These
operations will be selected by field office directors and other NASS staff. Indepth interviews and reviews of their relevant records, operating structures, and

Attachment A – ARMS Research Plan

contact information could be conducted. Because all ARMS samples are
coordinated with the census of agriculture in the census year, data collection for
the pilot operations in 2013 will include ARMS and census data.
•

Quality – Design quality control procedures with CARI for interview verification
o Once CARI software is in place in NASS, ongoing review of ARMS interviews
will be possible. Samples of interviews can be captured with CARI software and
interviewer and respondent behavior can be coded and analyzed. This will allow
evaluation of both questionnaires and interviewers. We expect this to begin in
2016.

•

Quality – Develop three-phase response rate
o The Statistical Methods Branch will use the 2011 ARMS survey data to derive
and compute a multi-phase response rate that will accurately reflect the
nonresponse from each preceding phase of the ARMS program. This new
computation will also be tested on the 2009 and 2010 ARMS survey data. All
testing will be completed by January 2013. Once successful tests have been
executed for 2009 through 2011, this program will be implemented into the
operational program for the 2012 ARMS survey cycle. Results will be published
August 2013 in the 2012 ARMS Quality Measures and Methodology document,
published annually.

•

Quality – Conduct nonresponse bias analyses
o Nonresponse bias analysis has been developed by research staff, and is now
integrated into ongoing post-data-collection activities. Bias analysis can be
conducted annually.

3. Nonresponse, Imputation, and Estimation
•

Incorporate multivariate imputation into the edit/imputation system
o Iterative Sequential Regression (ISR) imputation methodology. NASS will test
ISR in 2012 after the 2011 ARMS Phase III data are processed. Parallel test ISR
imputation methodology in 2013. Operational use of ISR is expected to start in
2014.

4. Analysis of Complex Systems, Data Preparation
o NASS obtained historic census of agriculture data files from the Census Bureau
for the years 1964, 1969, 1974, and 1978. Before use, these data files need to be
converted into formats that are readable by modern processing systems.
Unfortunately, the conversion process is complicated by two facts: (a) the same
conversion process is not workable for all years, and (b) the record layouts for the
historic files are not always available. Work to convert the historic data files to
the extent possible continues. However, data will probably not be recovered for
all years or for all states. Also, once recovered, data may not be able to be
mapped to more recent censuses and ARMS operations because of differences in

Attachment A – ARMS Research Plan

operation identification numbers. RDD expects to complete the process to convert
the historic data files (to the extent possible) and map data to operations (also to
the extent possible) by January 1, 2014.
5. Dissemination
o Work has begun on providing animated graphical displays of data for NASS’s
Web site. Once the technology has been adopted, ARMS Web-animated graphical
displays will be a high priority. Work on ARMS is expected to begin in 2014.
6. Data User Resources
o

ERS is developing an ARMS User’s Guide. A topics-based outline is nearly
Complete; ERS will evaluate whether to post the outline to its website.

7. Staff Development
o Managers in NASS and ERS will continue to support the Joint Program on
Statistical Methodology or other professional associations and encourage staff
involvement in the program to enhance staff skills.

Attachment B – ARMS Summary and Variables

ARMS

Listing and Description of Farm Business and Farm Operator Household
Summary and Classification Variables, 1991-2010

[Type text]

Page 8