Analysis Plans for 2017 Census of Agriculture Content Test

Analysis plans for 2015 Census of Agriculture Content Test

Contacts: Dan Beckler, Methodology Division/NASS 202-720-2248 and Jaki McCarthy, Research and Development Division/NASS 703-877-8000, ext. 142.

The first step in the 2015 Census of Agriculture Content Test was an extensive review of the data collected during the 2012 Census of Agriculture. The number of times each item on the questionnaire was edited or imputed (including the source of the edit: automated process or by a NASS analyst) was reviewed. In addition, comments from NASS field offices, NASS analysts and subject matter experts and the record of calls on the toll free help telephone line were also reviewed. This was used to identify the most problematic items on the 2012 form.

Based on this review, initial revisions to the form were made for testing. In addition, proposed new content was added to the form. This included an additional battery of questions about the role of decision makers on the operations recommended by a panel of outside experts convened to help improve data on women and beginning farmers. Initial cognitive interviews and usability testing will be conducted under the Generic Testing Docket (OMB Number 0535-0248). Cognitive interviews will be conducted with subsets of the form to ensure that all sections of the form are tested and to finalize new content. Usability testing will focus on the web based version of the questionnaire. Goals for the usability test are to improve the user experience for those respondents completing the form on-line form and to increase the number of reports obtained on-line for 2012.

Problems identified in the initial cognitive interviews and usability interviews will be used to make additional revisions to the form for Phase One, the mail out test. Phase One is intended to include enough respondents to ensure that the questionnaire is collecting data accurately under more realistic field conditions. The mail out is scheduled to begin in December 2015 with mail, Internet, and telephone followup-up data collection continuing into 2016. During telephone follow-up, phone interviewers will be monitored for quality assurance and to assess any problems with the telephone instrument. The sample for the mail out test will be selected to ensure that all sections of the form will be completed by some respondents, and also to include subsamples specifically included to examine difficult questions or sections containing known reporting errors in the 2012 Census of Agriculture. Four questionnaire versions will be tested in the mail out. The versions will differ in the order of the sections, formatting of the commodity sections, and may include different versions of selected questions based on the results of the cognitive testing.

Phase One will also include an additional pre-survey mailing for a small subset of cases to collect contact information for larger operations on a contact card. Response rates in the full mail out will be compared for those sample units that do or do not return the contact information card. If the response rate for cases that did not return the contact card is significantly lower, this may be useful in the census to target records for more resource intensive data collection efforts or earlier nonresponse follow up.

Response rates for all versions of the form will be compared. Edit and imputation rates for selected questionnaire items from the mail-out sample will be compared to determine if some versions of the form require fewer edits and imputations. The item missing rates and the number of items reported will also be compared to assess whether some versions of the form prompt underreporting.

Phase Two will involve up to 400 cognitive interviews and usability interviews, conducted in two rounds. A subsample of mail-out respondents will be re-contacted for follow up cognitive or usability interviews. These interviews will include in-depth questions to verify that key data items have been interpreted correctly and accurately reported. These interviews will include questions from any previous cognitive interviews and usability testing, with additional questions added based on a review of the data reported by Phase One respondents. For example, respondents may be asked to verify whether they are reporting in accordance with NASS official definitions or to explain discrepancies in their reports in open ended questions.

Based on the above analysis, additional revisions to the form may be made. Any substantial changes will be subject to additional cognitive testing.

Phase 3 of the test will be a smaller mail-out of up to 15,000 operations, scheduled for January 2017. This mail-out will be focused on testing alternative versions of the online form. Alternative methods of instructing respondents how to access the form, format of questions, and navigation through the form will be tested. The sample will be drawn to include respondents so that all sections of the form will be tested.

June 2015