NEW - ARMS 3 Economic Surveys 2023-2025 - PartB - 04-20-2023

Supporting Statement – Part B

AGRICULTURAL RESOURCE MANAGEMENT PHASE 3 ECONOMIC SURVEYS

OMB No. 0535-NEW

The information that is provided in this supporting statement serves as an overview of the sampling, statistical methodology, weighting of data for non-response, methods for increasing response rates, measurements for accuracy, testing of instruments, etc. The more detailed information for the ARMS Phase 3 can be found in the attachments to this submission.

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection has been conducted previously, include the actual response rate achieved during the last collection.

Respondent Universe: The ARMS Phase 3 is screened using the ARMS 1 / Integrated Screening Survey (ISS) included in OMB Control Number 0535-0218. The purpose of screening is to identify records for sampling. The target population for ARMS is the official NASS farm population with the exclusion of institutional farms, approximately 2 million operations. The target population for the ARMS Production Practice & Cost and Returns (PPCR) survey for targeted commodities is operations that produce the commodity of interest. The respondent universe for the Contractor Expense Surveys comes from known contractors on our list frame. No screening to identify contractors is necessary.

Sampling: The ARMS is a multiple frame survey using a list frame of farms identified on the NASS list frame and a complementary area frame. Sample list strata are developed using major categories, a combination of targeted crops, livestock, fruit, vegetables, and horticulture. Strata are developed by grouping operators by the total value of sales followed by the presence of the targeted commodity. The list is an efficient sampling frame because it contains most of the farms with the largest production and economic activity. The area frame, stratified by land use, provides the completeness missing from the list. The multiple frame expansions are unbiased and more precise than expansions which could be obtained using one frame alone.

The list classification process is very extensive, examining many crop and livestock control data values. After classification is completed, list records are partitioned into size groups based on qualifying control data for the current year commodities, type of farm, and estimated size. Operations will always belong to one of the standard expenditure strata and are eligible to also belong to a targeted commodity stratum.

The area frame sample consists of a subset of respondents from the June Agricultural Survey (JAS), included in OMB Control Number 0535-0213. During the JAS, farm operators associated with the land area segments are classified according to whether they overlap with the list frame. Those farm operators not eligible for selection on the list are eligible for selection in the ARMS area sample.

Sequential Interval Poisson (SIP) sampling is used for selecting the PPCR and Cost & Returns Report (CRR) surveys in ARMS. In SIP, the sampling probabilities were defined to ensure each operation was in one and only one sample. The probabilities of selection can be based on any type of probability scheme. The SIP procedure was used to minimize overlap with the previous year’s ARMS sample as well as the current year’s quarterly Agricultural Production Surveys samples included in OMB Control Number 0535-0213.

Response Rates: Following are average response rates for all survey phases based on the last three survey cycles.

NASS continues its efforts at reducing respondent burden while improving response rates. With the combined work of our Research and Development Division, Methodology Division, Census and Survey Division, Public Affairs Office, and our Training Group, NASS is looking at what factors work for some surveys and not others. Through the use of project management techniques and building on to lessons we have learned from previous surveys and the Census of Agriculture we are able to make changes to internet versions of questionnaires to make them more user friendly, combining smaller surveys so that we can reduce the frequency of contacting the farmers, and improve on our sampling of farmers.

The NASS Public Affairs Office (PAO) promotes NASS survey efforts and educates respondents about the need and uses for the data they are asked to provide. This group has developed survey-specific materials enumerating the benefits and uses of the data gathered from the economic surveys. PAO works with data users and industry leaders to provide concrete examples of instances where the data that respondents provide are used to service the respondents. They are also actively publicizing survey activities by generating and distributing news reports and drop-in articles for industry publications and news outlets.

2. Describe the procedures for the collection of information including:

• statistical methodology for stratification and sample selection,

• estimation procedure,

• degree of accuracy needed for the purpose described in the justification,

• unusual problems requiring specialized sampling procedures

Agricultural Resource Management Survey (ARMS) - The annual surveys collect production practices and cost of production data on selected commodities and also detailed whole farm financial information from a representative sample of farms and ranches from across the country. To accomplish this, the ARMS are conducted in three data collection phases. In many ways, the three phases can be viewed operationally as independent surveys. However, the power of the ARMS design is that data across phases are related and can be combined and analyzed.

The ARMS Phase 1 (included in OMB Control Number 0535-0218) is conducted from May through July, and it collects general farm data such as crops grown, livestock inventory, and value of sales. The Integrated Screening Form is used in years that NASS will be conducting the Vegetable Chemical Use Survey (included in OMB Control Number 0535-0218). The integrated form will also be used for the ARMS Phase 3 surveys. These data are used to qualify or screen farms for these surveys to make sure the samples are as accurate as possible.

The ARMS Phase 2 (included in OMB Control Number 0535-0218) is conducted from September through December. Phase 2 collects data associated with agricultural production practices (field operations, pest management practices, etc.), resource use (pesticide applications, fertilizer and nutrient application, types of equipment used, etc.), and variable costs of production for specific commodities.

The ARMS Phase 3 is conducted from December through April following the survey reference year to enable collection of full year financial data. This phase collects whole farm finance and operator characteristics information. Samples of the Phase 3 advance letters, respondent booklets, sample questionnaires, and telephone quality control sheets are attached. Some of these respondents will be asked to complete a commodity-specific report to obtain financial, resource use, and cost of production data for the selected commodity and the entire farming operation. It is vital that operators who are selected for both the second and final phase complete both phases, so that we can collect data for the entire crop production process (physical activities and financial costs). Data from both phases provide the link between agricultural resource use and farm financial conditions; this is a cornerstone of the ARMS design. The commodity-specific versions consist of the Core questionnaire with appropriate customization of questions with a general scope as shown in example crop and livestock questionnaires.

As questionnaires are updated each year to accommodate changes in the farming conditions for that year or for a particular commodity, the final versions will be submitted to OMB as they become available.

Contractor Expense Surveys - These surveys are used to collect average contractor expenses for the five commodity groups (layers, turkeys, hogs and pigs, broilers and starter pullets, and processed vegetables). The sample size will vary from State to State, dependent upon the number of contractors and the amount of influence they each have on the market in their respective States.

NASS Regional Field Office staff will receive a Survey Administration Manual which provides detailed aspects of the survey data collection and editing process. Field enumerators in each State will be given an Interviewer’s Manual.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Based on previous studies, NASS feels that the best tool for increasing response rates is improving the training of our field enumerators. Enumerators who are better prepared to answer questions raised by respondents and to inform respondents of the importance of the data and how it will be used have had the best success rates. These data are very important to both the farming community as well as external data users (politicians, educators, banking industry, farm supply companies, etc.).

Research: Findings from the 2015 NASS research report, Targeted Data Collection Efforts for the 2013 ARMS III, are summarized in the next three paragraphs:

In 2012, the Research and Development Division (RDD) of National Agricultural Statistics Service (NASS) conducted a study examining targeted data collection techniques for the Agricultural Resource Management Survey Phase III (ARMS 3) for likely nonrespondent operations identified by classification trees. Due to a variety of reasons, the study was not conducted as anticipated in 2012. RDD conducted the study again in 2013 using a subset of states to assess whether specific data collection techniques can be implemented and whether they improve response rates.

Likely nonrespondent operations were identified using classification trees and a measurement of the impact the operation has on the calibration targets for ARMS 3, so that targeted data collection could be used to increase response rates for those operations. The targeted data collection techniques included providing enumerator incentives and using supervisory enumerators and/or regional office directors and State Statisticians to make in-person contacts with highly likely nonrespondents.

Because this was largely an exploratory study, our sample sizes are too small to demonstrate statistical differences in response rates. Overall, NASS found a meaningful difference in response rates (7%) for the treatment group above the control group, although it was not statistically significant. Working with the smaller number of states, we were able to obtain much needed buy-in for the study, something that was missing from the 2012 study.

RESEARCH: Findings from the 2022 NASS research report, Reassessing the Effect of Calibration on Non-response Bias in the 2020 ARMS Phase III Sample Using Census 2017 Data, are summarized in the next two paragraphs.

Records sampled for the 2020 ARMS Phase III were matched with those from the 2017 Census of Agriculture, and means of census data were calculated for matching records which had also provided 2017 expenditure data for the census. Nonresponse bias in ARMS data was assessed, using census data as a proxy, in terms of the degree to which the mean based on all sample cases versus respondent cases differed. Three means were computed and compared across 20 regions in order to assess relative bias: 1) the mean of all matching cases using base sampling weights, 2) the mean for only matching ARMS respondents using the same base sampling weights, and 3) the mean for matching ARMS respondents using the sampling weights as adjusted through calibration.

Using 16 “study variables,” relative bias of means was assessed using a variation of the formula provided by OMB in Guideline 3.2.9. Although significant biases were exhibited in all 16 study variables using the 2020 ARMS III base sampling weights, the 2020 ARMS III calibration weights were able to reduce the bias so that it was no longer significantly different from zero (p < .05) in 13 of the 16 study variables. This study suggests that the process of calibration is an effective tool in reducing nonresponse bias levels.

The following are non-response adjustment for ARMS 3.

Non-response is taken into account in the ARMS 3 sample allocations by State. ARMS 3 sets a target for positive usable responses by State and ARMS Region; the sample size is adjusted based on a 5-year historical response rate to achieve the targeted usable rates.

Unit non-response in the ARMS 3 weights, for all versions, is adjusted by using calibration. The calibration process modifies the survey weights so that certain targets are met. NASS uses official estimates of farm numbers, corn, soybean, wheat, cotton, hay, peanuts, rice, sugar, tobacco, fruit, and vegetable acres as well as cattle, cattle on feed, milk production, hogs, broilers, eggs, and turkeys as calibration targets.

Item non-response for the ARMS 3 surveys is dealt with by using machine imputation. Data collectors do not impute item values in the field. About 300 survey variables that are critical to NASS analysis and/or ERS work are imputed using usable data from current survey respondents. A multivariate imputation scheme is used, and covariates come from several different components of the questionnaire including but not limited to region, economic sales class, type of farm, acreage and production expenses. Imputed item values are flagged for data users, and the algorithms for imputation are described in technical documents that will be available to data users.

Contractor expenses are imputed from the data collected with the Contractor Expense Survey.

NASS deploys several data collection evaluation and monitoring tools. These tools enhance NASDA data collection tactics producing maximum positive impact on coverage/calibration. Real-time NASDA data collection oversight allows for flexibility of collection tactics and an improvement in survey quality metrics. Over several years, results from these survey monitoring tools are blended into a cohesive ARMS 3 data collection strategy, tailored to each State.

4. Describe any tests of procedures or methods to be undertaken.

NASS uses an OMB-approved generic clearance docket (OMB Control # 0535-0248), to conduct testing and evaluation of most NASS questionnaires. In this PRA approval request, NASS is including an allowance to conduct a total of 50 cognitive test interviews (annually) on the various questionnaires included in this docket. If a different method of testing is necessary or a larger sample is needed, NASS will submit a request using the generic clearance docket (0535-0248). The generic testing docket allows for a variety of testing methods, including cognitive testing, focus groups, split sample field tests, etc., that can be used to test ARMS and other NASS surveys. NASS does not plan to create a cognitive laboratory facility due to the geographic dispersion of farm operators needed for testing. As is typical in establishment surveys, most testing is conducted with onsite visits.

NASS plans to conduct cognitive interviews for all major changes that are proposed for the ARMS Phase 3 surveys. These interviews would address specific questions and sections to assess modified content and formatting.

Web-based data collection is available for respondents for the ARMS 3 survey samples nationally. Additionally, Computer Assisted Personal Interviewing (CAPI) began in fall 2009 and is available for all enumerators to use.

In the past few years, NASS has collected a large portion of the ARMS Phase 3 sample through the mail and Internet. NASS will continue to improve the questionnaire to make it more user-friendly for self-administered data collection. This will be done by reviewing data from previous years and conducting cognitive testing and usability testing to meet this goal.

Data from the Contractor Expense Survey that is included in this approval request, is used to impute data for operations that could not or would not provide expenses that were conducted under a contract agreement.

Response improvement techniques will continue to be researched and tested to improve response rates in the area of questionnaire improvement, respondent relationship building, and soft refusal conversion techniques.

In 2016 NASS implemented methodology to identify the 100 most impactful respondents in terms of their contribution to the final estimates. NASS Survey Administrators have recommended our Regional Field Offices to contact these operations in person to collect the data. This practice was suspended for crop year 2020 and 2021 due to the COVID-19 pandemic. NASS plans to resume this practice depending on the status of the pandemic.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The sampling plans are developed by NASS. Questionnaire design, data edit, and initial summarization will also be completed by NASS, and for the ARMS surveys, these tasks will be done in consultation with ERS.

Survey design and methodology are determined by the Summary, Estimation, and Disclosure Methodology Branch, Statistics Division; Branch Chief is Jeff Bailey, (202) 690-8141.

Sample sizes for each State are determined by the Sampling, Editing, and Imputation Methodology Branch, Methods Division; Branch Chief is Mark Apodaca, (202) 690-8141.

Data collection is carried out by NASS Regional Field Offices (RFOs). The Western Field Operation’s Director is Troy Joshua, (202) 720-8220. The Eastern Field Operation’s Director is Jody McDaniel, (202) 720-3638.

The NASS survey statisticians in Headquarters listed below are responsible for coordination of sampling, questionnaires, data collection, and other Field Office support. Branch Chief is Gerald Tillman, (202) 720-3918; Section Head is Torey Lawrence (202) 720-5921.

The NASS commodity statisticians in Headquarters listed below are responsible for national summaries, analysis, and publication. Branch Chief is Tony Dorn, (202) 720-6146, Section Head is Bruce Boess (202) 720-4447.

Survey	Survey Statisticians Census and Survey Division, Survey Administration Branch, Environmental and Economic Surveys Section	Commodity Statisticians Statistics Division, Environmental, Economics and Demographics Branch, Economic, and Environmental and Demographics Sections
ARMS Phase 3	Pam Coleman (202) 720-6564 Jeff Lemmons (202) 690-3692	Stephanie Brennan (202) 690-0392
Contractor Expense Survey	Pam Coleman (202) 720-6564 Jeff Lemmons (202) 690-3692		Stephanie Brennan (202) 690-0392

April 2023

8