Testing Plan for Acreage and Land Use Sections of Census

1/4/2013

TESTING PLAN FOR ACREAGE AND LAND USE SECTIONS OF THE CENSUS REPORT FORM

INTRODUCTION AND BACKGROUND

The census of agriculture includes two sections in which detailed questions are asked about land ownership and usage. Historically respondents have had many problems reporting the items in these two sections.

The first section, Section 1 – Acreage, aims to determine the acreage of the entire operation by asking detailed questions about land owned, land rented from others, and land rented to others. Respondents are asked to perform a simple mathematical calculation to determine the total number of acres operated. A separate question in this section asks about number of acres on a fee per-head or animal unit month (AUM) basis. This land should not be included as part of the land rented or leased from others, or as part of the total acres operated. Another question asks to report the acreage in the principal county and in all other counties. The principal county is the county where the largest part of the value of the agricultural products sold from the farm or ranch were raised or produced. If the operation is located in more than one county, the names of the additional counties and/or States and the number of acres in each additional county should be reported under “Other County Name(s)”.

The second section, Section 2 – Land, aims to determine how the operated acreages were used or distributed among prelisted items during the census year. The land use items are arranged in what is normally considered a decreasing order of economic importance. These items are: cropland harvested, cropland on which all crops failed, cropland in cultivated summer fallow, cropland idle, permanent pastures, woodland pastures, other pastures, woodland not pastured, and other land. Each acre is to be reported only once in this section, even though the land may have been used for more than one purpose. Respondents are asked to add all these individual items to determine total acres. The total acres figure has to equal the figure calculated on the previous section, Section 1 - Acreage. A common error in reporting is that the values reported in Section 1, Total Acres (K046) is not equal to Section 2, Total Acres (K798) figure. Throughout the years the wording of these questions and the location of these two sections have changed significantly.

2007 Census of Agriculture

I. Imputation Rates Analysis

This analysis identified item codes with high imputation rates for the 2007 Census of Agriculture. Item codes with imputation rates of 30 percent or higher were identified for possible changes to question wording and/or questionnaire formatting. The 30 percent imputation rate criterion was decided based on the Standards and Guidelines for Statistical Surveys developed by the Office of Management and Budget (OMB) to ensure consistency among and within statistical activities conducted across the Federal Government. These guidelines suggest that if the item nonresponse rate is less than 70 percent, the agency should conduct an item nonresponse analysis to determine if the data are missing at random at the item level for at least the items in question. Data for this analysis were provided by the Census Statistics Section of the Census Planning Branch. The numbers were generated by running BRIO queries on the Wip_production.oce database.

Imputation was defined as any automated change at a record level where a missing or zero value for an item code was replaced with a positive value. This included donor imputation, when a donor value from a different record is used to supply the missing value; and other computer generated changes from missing or zero to a positive value such as calculating a missing sum from other values provided on the report form.

Tables 1 and 2, show imputation rates for individual item codes in sections 1 and 2 of the questionnaire. The headings for the tables represent the following:

Total Count: Total number of records with Current Data > 0

No changes: Reported Data = Current Data

All Imputations: Reported Data<= 0, Current Data > 0

Deletions: Reported Data > 0, Current Data<= 0

Other Changes: Other changes performed by the edit or an analyst such as changes to positive values and deletions performed by analysts.

Section 1: Acreage in 2007

Table 1: Acreage in 2007

Item Code	Description	Total Count	No Changes	All Imputations	Deletions	Other Changes
			Percent	Percent	Percent	Percent
43	Land owned	1,433,705	93.6	1.7	0.0	4.7
44	Land rented from others	517,870	94.7	1.3	0.5	3.5
45	Land rented to others	231,673	87.0	1.9	5.0	6.1
46	Total Acres	1,690,599	80.4	5.5	1.0	13.1
53	Owned Land rented to others	238,910	69.8	9.3	11.3	9.6
56	Acres in Principal County	1,523,825	71.2	12.7	0.0	16.1

No imputation problems with this section.

Section 2: Land

Table 2: Land

Item Code	Description	Total Count	No Changes	All Imputations	Deletions	Other Changes
			Percent	Percent	Percent	Percent
787	Cropland Harvested	1,019,301	63.1	11.8	3.7	21.4
790	Cropland Failed Crops	76,520	52.1	7.9	14.4	25.5
791	Cropland in Summer Fallow	66,029	40.6	11.7	19.9	27.8
1062	Cropland idle	272,175	53.3	24.8	4.2	17.8
796	Permanent Pasture	813,586	62.5	7.5	4.4	25.7
794	Woodland Pastured	278,234	64.0	8.6	5.1	22.3
788	Cropland for Pasture	325,820	36.9	29.2	11.8	22.1
795	Woodland not pastured	450,580	64.2	8.8	4.4	22.6
797	All Other Land	825,236	68.5	10.9	3.8	16.8
798	Total Acres	1,523,825	72.0	17.5	0.0	10.4

There were 406,625 respondents (26.7%) who reported different values on the total land item codes K046 and K798.

Imputation rate for cropland for pasture was 29.2 percent.

II. Classification Tree Analysis

Classification tree (aka decision tree) analysis was used to identify farm attributes such as farm size, farm type, and geographical region, linked with errors in Sections 1 and 2 of the 2007 Census of Agriculture report form. This analysis used unedited and edited figures from 529,588 records from the 2007 Census of Agriculture. The sample was comprised of all records from Arizona, Arkansas, California, Colorado, Florida, Georgia, Iowa, Minnesota, New York, North Carolina, Pennsylvania, South Dakota, Utah, Washington, and Wisconsin. This sample included operations from all 6 geographical regions, 16 farm types, and 13 farm size classifications covered by the census.

The first type of error investigated was when total land acreage reported in Section 1 did not match its subparts (K046 K043+K044-K045). The figure reported for total acres (K046) differed 26 percent of the time from the sums calculated based on the reported subparts. They differed 1.4 percent of the time by at least 10 acres.

The second type of error investigated was when the total land acreage reported did not match its subparts by land use (K798 K787+K790+K791+K1062+K796+K794+K788+K795+K797). The subparts did not add to the reported sum 38.6 percent of the time. They differed about 5 percent of the time by at least 10 acres.

The third type of error looked at inequalities between K046 and K798. The values reported on these two keycodes were different 27.8 percent of the time. This error rate was reduced to 7.6 percent when only including differences of at least 10 acres.

Results from the classification tree analysis did not provide a clear pattern of farm attributes related to errors in Sections 1 and 2. Each investigation had its own pattern of associated farm characteristics more prone to errors.

2010 Census Content Test

Analysis of the 2010 Census Content data indicated that out of approximately 31,000 farms about 1,370 (4.4 percent) had total acres in Section 1 (K046) not equal to the sum of the parts (land owned and land rented from others minus the land rented to others). Also, 1,561 farms (5 percent) had total acres in Section 1 (K046) not equal to total acres in Section 2 (K798). Analysis of the pasture questions in Section 2 indicated that 3,038 farms (9.8 percent) reported the same number of acres for both permanent pasture (K796) and woodland pastured (K794); 2,897 farms (9.3 percent) reported the same number of acres for both permanent pastures (K796) and other pastures (K788); and 4,727 (15.2 percent) farms reported the same number of acres for both woodland pastured (K794) and other pastures (K788). This information indicates that respondents are confused about how to report pasture.

General Plan

The format and wording of Sections 1 and 2 were changed significantly for the 2010 Content Test and the 2012 Census. For Section 1, the wording description of each item was reduced and bullets were used to highlight important information. Also, response boxes were used for people to write down the individual pieces that go into the calculation of total acres. For Section 2, changes were made to the wording for pasture items, and instructions were provided for respondents to verify that the total acres reported in Section 2 were equal to the total acres reported in the Section 1. Follow up with respondents was never conducted after the 2010 Content Test to verify that the changes were working as intended. These changes were incorporated into the 2012 Census of Agriculture report form. Conducting a test of these two sections is necessary to verify that respondents are correctly reporting their data in these two sections.

The proposed testing plan will follow up on a subsample of respondents who reported errors in Sections 1 and/or 2 of the 2012 Census of Agriculture report form. Testing will involve 25 cognitive interviews conducted by Research and Development Division (RDD) staff and 175 follow up interviews conducted by National Operations Center (NOC) staff. These interviews will ask respondents to review their answers, provide more detail on what was included in their answers, and attempt to resolve any errors in their reports. From this we hope to gain an understanding of how respondents interpreted and responded to the survey questions and further insight into the sources of errors in data reporting. These findings will then be used to develop further testing for the 2015 Census Content Test.

Cognitive Interviews: Spring 2013

Several rounds of cognitive interviews will be conducted with a small number of respondents. Staff from Census and Survey Division (CSD) will select a sample of respondents to the 2012 Census of Agriculture who misreported items in the Section 1 and/or Section 2, such as summation discrepancies within the section, items left blank, and summation discrepancies between the two sections. CSD staff will also select a sample of respondents who exhibited no obvious reporting errors in these sections to verify that the information was reported correctly. Staff from CSD and RDD will select the names of respondents to be interviewed. Field offices will assist in making the necessary arrangements to contact respondents and arrange the time for the interviews. Using this sample, staff from RDD will conduct 1-2 rounds of cognitive interviews with 25 respondents. Respondents will be mailed a cover letter indicating that they will be called by a NASS representative to discuss their answers (see appendix A), as well as a copy of their 2012 Census of Agriculture form, which they will be asked to refer to during the interview. RDD staff and field office staff who are trained to conduct cognitive interviews will administer scripted and emergent probe questions designed to elicit information on respondents’ comprehension of and response to questions in sections 1 and 2. Probing on other sections of the Census form may occur if this proves to be insightful in resolving reporting errors in Sections 1 and 2. Probe questions will include things like, how did you interpret this question, tell me what you included in your response, did you notice the list of includes and excludes? Interviews will be conducted in person or over the telephone. Interviews will last 1 hour and may be audio recorded with respondents’ permission.

NOC Follow Up Interviews – Summer 2013

The NOC will be used to conduct a 175 follow up interviews. As with the cognitive interviews, respondents will be mailed a cover letter indicating that they will be called by a NASS representative to discuss their answers, as well as a copy of their 2012 Census of Agriculture form, which they will be asked to refer to during the interview. Telephone enumerators at the NOC will be trained to administer a series of structured probes in a scripted re-interview regarding respondents’ answers to questions in sections 1 and 2 of the Census form. Structured probes will be less open-ended than the ones used in the cognitive interviews and will include things like, do you own any land that was not included in item 1? Did you have land any land used on a per-head or animal unit month (AUM) basis? Was this land included or excluded from your calculation of item 2? Interviews will last 1 hour and may be audio recorded with respondents’ permission.

Appendix A: Cover Letter

March XX, 2013

Dear Farmer or Rancher:

The National Agricultural Statistics Service (NASS) is reviewing the 2012 Census of Agriculture report form that you recently completed, and we need your help. We want to ensure our respondents understand the questions in the same way. We would like to review your responses to the Acreage and Land sections of your 2012 Census of Agriculture Census report form and ask you some additional questions. We are enclosing your responses to these two sections. You may receive a call from NASS staff to discuss your answers. The interview should last between 30 to 60 minutes.

Employees of the National Agricultural Statistics Service (NASS) are required by Federal law to keep your answers confidential. Only sworn NASS employees can see your report, and NASS will use your answers for statistical purposes only. The census of agriculture is required by law (Title 7, U. S. Code), but we are asking for your voluntary participation in this project.

If you have any questions or need help, please call 1-888-xxx-xxxx (toll-free). Thank you for your cooperation in this important endeavor.

Sincerely,

Cynthia Clark

Administrator

Enclosures