Analysis Plans for Census of Ag Content Test

Analysis plans for pre-2012 Census of Agriculture Content Test

Contact: Jaki McCarthy, Research and Development Division/NASS 703-877-8000, ext. 142.

The first step in the Content Test was an extensive review of the data collected during the 2007 Census of Agriculture. The number of times each item on the questionnaire was edited or imputed (including the source of the edit: automated process or by a NASS analyst) was reviewed. In addition, comments from NASS field offices, NASS analysts and subject matter experts and the record of calls on the toll free help telephone line were also reviewed. This was used to identify the most problematic items on the 2007 form.

Based on this review, initial revisions to the form were made for testing. In addition, proposed new content was added to the form. Phase 1 of testing for these forms was cognitive interviews. Approximately 50 cognitive interviews were conducted with subsets of the form to ensure that all sections of the form were tested. Problems identified in these interviews lead to additional revisions to the form designed for the first mail out test. Of course, the small number of cognitive interviews and the fact that they do not recreate actual census data collection procedures does not provide enough cases to determine whether questions will work as intended for most respondents.

Phase 2, the first mail-out test, is intended to include more respondents to ensure that the questionnaire is collecting data accurately under more realistic field conditions. The mail out is scheduled for late January/early February 2010. The sample has been selected to ensure that all sections of the form will be completed by some respondents, and also includes subsamples specifically included to examine difficult questions or sections based on known reporting errors in the 2007 Census of Agriculture. During Phase 2 we are also testing strategies to increase online reporting for the web based version of the questionnaire. Since the 2007 Census of Agriculture was the first to include an on-line reporting option, it is an important goal for NASS to improve the on-line form and increase the number of reports obtained on-line for 2012.

Edit and imputation rates for the individual questionnaire items from the mail-out sample will be compared to the 2007 data review to determine if fewer edits and imputations were necessary. A subsample of records who reported inconsistent data within their forms in 2007 will be specifically included. This will include respondents who reported different total number of acres in their operations in separate sections of the questionnaire, and those who reported land enrolled in government programs greater than the amount of acres in their operation. These will be reviewed to determine if the new form prompted more internally consistent reports from these respondents.

In addition, a subsample of mail- out respondents will be re-contacted for follow up cognitive interviews. These interviews will include in-depth questions to verify that key data items have been interpreted correctly and accurately reported. These interviews will include questions from the Phase 1 cognitive interviews with any additional questions added based on a review of the data reported by Phase 2 mail respondents. For example, respondents may be asked to verify whether they are reporting in accordance with NASS official definitions or to explain discrepancies in their reports in open ended questions.

The only split sample of the paper form to be mailed will compare two separate versions of the horse questions. In each strata of the phase two sample half will receive one version of the form and half will receive the other. Some operations in all strata are expected to report horses on their operations. Collecting information on farm operation horses has been a long standing problem, and both versions of the questions are new to the form. The decision on which of these is working better will be made based on review of the data reported and information collected in the follow up cognitive interviews.

In addition, we plan to include split sample experiments focused on the on-line questionnaire in a two by two design within the 2000 operations sampled for the on-line reporting test. These experiments are designed to test alternative methods to prompt on-line response (as opposed to mailed back response). The sample size for the split panel test was determined by using an extremely conservative estimate of the characteristic of interest being 50 percent. Each panel was assumed to have the same variance and the sample size calculation presumed a 95-percent confidence level for each panel. Taking into consideration the fact that one-sided hypothesis testing will be utilized (i.e., only examining if the response rate is higher for the experimental group) and the desire to maintain a margin of error of less than 3 percent, a panel sample size of 1000 records was determined after adjusting by an overly cautious response rate of only 75%. One subsample will receive a pre-survey letter inviting them to complete the form on-line with their access code. This group will NOT receive the paper questionnaire in their initial mailing. A control group will also get their survey access code, but will receive it in the package containing the paper questionnaire. An additional split sample will divide the operations into two subgroups, one given a specific response deadline date, one without. Response rates and speed of returns for the groups will be compared. In addition, respondent reactions to the on-line form will be solicited in the follow up interviews.

Based on the above analysis, additional revisions to the form may be made following Phase 2 of the tests. Any substantial changes will be subject to additional cognitive testing.

Following completion of Phases 1 and 2, Phase 3 of the test will be a larger mail-out, scheduled for January 2011. This mail-out can be considered a pilot test of the 2012 Census of Agriculture, using procedures and conditions similar to the operational Census operations. This Phase of the test will be used primarily to test data collection and processing procedures; few changes to the questionnaire are expected after this test. However, review of the quality of the data collected in Phase 3 will be similar to that conducted prior to and during Phase 2. Sample specifications for Phase 3 will be finalized after Phase 2 analysis, as any problems identified may suggest subgroups that should be included at higher rates in that sample. The Phase 3 sample specifications will be forwarded once they are finalized.

January 2010