Analysis Plans for 2020 Census of Agriculture Content Test

Analysis plans for 2020 Census of Agriculture Content Test

Contacts: Jaki McCarthy, Methodology Division/NASS 202-690-2389 and Dan Beckler, Methodology Division/NASS 202-720-2248.

The first step in the 2020 Census of Agriculture Content Test was a review of the data collected and procedures used during the prior Census of Agriculture (2017). The number of times each item on the questionnaire was edited or imputed (including the source of the edit: automated process or by a NASS analyst) was reviewed. In addition, comments from NASS field offices, NASS analysts and subject matter experts and the record of calls on the toll free help telephone line were also reviewed. This was used to identify the most problematic items on the 2017 form.

Based on this review, initial revisions to the form were made for testing. In addition, proposed new content was added to the form. This included new items for precision agriculture practices, new commodities, and changes to the questions on internet access. Changes were also made to the web based questionnaire intended to make it easier for respondent to report their information and potentially pre-fill some information obtained in previous contacts. Initial cognitive interviews and usability testing will be conducted under the Generic Testing Docket (OMB Number 0535-0248). Cognitive interviews will be conducted with subsets of the form to ensure that all sections of the form are tested and to finalize new content. Usability testing will focus on the web based version of the questionnaire. Goals for the usability test are to improve the user experience for those respondents completing the form on-line form, evaluate the use of previously reported data provided to respondents and to increase the number of reports obtained on-line for 2022.

Problems identified in the initial cognitive interviews and usability interviews will be used to make additional revisions to the form for Phase One, the mail out test. Phase One is intended to include enough respondents to ensure that the questionnaire is collecting data accurately under more realistic field conditions. The mail out is scheduled to begin in December 2020 with mail, Internet, and telephone follow-up data collection continuing into 2021. During telephone follow-up, phone interviewers will be monitored for quality assurance and to assess any problems with the telephone instrument. The sample for the mail out test will be selected to ensure that all sections of the form will be completed by some respondents, and also to include subsamples specifically included to examine difficult questions or sections containing known reporting errors in the 2017 Census of Agriculture. One questionnaire version will be tested in the mail out.

Phase One will include several tests to evaluate changes to the format, content and timing of contacts. These tests are intended to evaluate if these changes will accelerate responses, increase response rates, or reduce costs. Additional tests will evaluate the use of previously reported data provided to respondents in the web based questionnaire.

Phase Two will involve up to 200 cognitive interviews and usability interviews, conducted in two rounds. A subsample of mail-out respondents will be re-contacted for follow up cognitive or usability interviews. These interviews will include in-depth questions to verify that key data items have been interpreted correctly and accurately reported. These interviews will include questions from any previous cognitive interviews and usability testing, with additional questions added based on a review of the data reported by Phase One respondents. For example, respondents may be asked to verify whether they are reporting in accordance with NASS official definitions or to explain discrepancies in their reports in open ended questions.

Based on the above analysis, additional revisions to the form may be made. Any substantial changes will be subject to additional cognitive testing.

Phase 3 of the test will be a smaller mail-out of up to 15,000 operations, scheduled for January 2022. This mail-out will be focused on testing alternative versions of the online form. Alternative methods of instructing respondents how to access the form, format of questions, and navigation through the form will be tested. The sample will be drawn to include respondents so that all sections of the form will be tested.

June 2020