Non-Response Bias Approach

USDA - NASS

Nonresponse Bias Plan for Field Crops Terms of Clearance

12/11/2012

Nonresponse Bias in the Field Crops Docket

The National Agricultural Statistics Service (NASS) conducts surveys in order to prepare national, State, and county estimates of crop and livestock production, disposition, prices, and related environmental and economic factors. This document addresses the Field Crops docket Terms of Clearance.

TERMS OF CLEARANCE: In accordance with 5 CFR 1320 and prior terms of clearance (2009), the information collection is approved for three years conditional on the following: 1) a plan to evaluate the extent of nonresponse and nonresponse bias for probability sample surveys, censuses, and coverage surveys with an observed response rate of less than 80% will be received by OMB no later than six months from the date of this approval; 2) the complete evaluation will be received prior to the next request for clearance for this docket.

The following documentation serves as an overview of NASS’s plan to address the potential for nonresponse bias for surveys in the Field Crops docket as spelled out in the OMB Terms of Clearance above. In response to the Terms of Clearance, this plan will address nonresponse bias issues for surveys with a less than 80% response rate (unacceptable range according to the OMB Standards and Guidelines for Statistical Surveys) in the Field Crops docket.

Phase I

For the 13 surveys in the Field Crops docket that have a response rate below 80%, NASS will determine whether data are available to calculate weighted item response rates that include a measure of size to account for the unit’s relative importance in the estimates. Attachment A shows the 2011 unit response rates for surveys in the Field Crops docket.

Phase 2

Weighted response rate data are available

Calculate weighted item response rates for key items for surveys where the unit response rate is less than 80%. Many establishment surveys use weighted item response rates because a small number of extremely large operations may dominate an industry. Attachment B defines the weighted and unweighted response rates.

• Unweighted response rate is in the acceptable range (i.e., 80% or more) – NASS will document this and provide the information to OMB.

• Unweighted response rate is in the unacceptable range, but the weighted item response rate is in the acceptable range (i.e., 70% or more) – NASS will document this and provide the information to OMB.

• Both unweighted and weighted response rates are in the unacceptable range – NASS will document this and proceed to Phase 3.

Weighted response rate data are not available

For some surveys, data are not available to calculate weighted item response rates. This plan includes the steps NASS will take to obtain the data needed to calculate these metrics for future data collections.

• NASS is currently making updates to its systems to obtain the data needed to calculate weighted item response rates for all future surveys.

• NASS will establish a standard reporting mechanism for Field Offices to provide this information to Headquarters in a consistent, usable manner.

• NASS will modify its aggregates database and estimation systems to have Field Offices submit response rates and commodity weighted response rates with each estimate. Eventually, all surveys will be standardized and in a centralized environment, which will allow NASS to pull the data automatically.

• NASS will provide Field Offices with the tools and training necessary to collect, capture, and report this data for all surveys.

• For probability-based surveys, NASS will create a timeline for the standardization of the survey and the comprehensive plans for collecting the necessary data.

• For opinion-based surveys (e.g., Crop Weather), NASS will document the procedures undertaken to select a qualified sample, train respondents, monitor and validate responses, etc.

• NASS will calculate weighted response rates for future data collections in the Field Crops docket once these data are available and will evaluate the need for a nonresponse bias study. NASS will provide this information to OMB.

Phase 3

As mentioned in Phase 2, NASS will provide documentation of the response rates to OMB if (1) the unweighted unit response rates are in the acceptable range or (2) the unweighted unit response rates are in the unacceptable range, but the weighted item response rates are in the acceptable range. Under these circumstances, NASS will not conduct a nonresponse bias study since response rates are in the acceptable ranges according to the OMB Standards and Guidelines for Statistical Surveys.

When unweighted response rates and weighted item response rates are both in the unacceptable range, NASS will examine potential nonresponse bias and provide the findings to both OMB and the public. The size of the survey and the use of the estimates will be taken into account prior to conducting these studies to allocate resources effectively. In some cases, such as for opinion-based or small surveys, NASS may provide a justification statement to OMB documenting the findings and explaining the statistical soundness of the data.

Attachment C contains a flowchart of the nonresponse bias approach.

Improving response rates is an ongoing goal and constant challenge for establishment surveys. NASS is currently working on establishing best practices for handling impact (i.e., those that have the most impact on estimates) farming establishments. These practices include building rapport, educating operators, conducting outreach, coordinating data collection contacts, developing data collection plans, and implementing follow-up strategies. NASS will continue to work on ways to improve survey response rates and is committed to providing timely, accurate, and useful statistics in service to U.S. agriculture.

Terminology – Response rate definitions

Unweighted (Unit) response rate = (Questionnaires Completed by Respondents /(Selected Sample – Office Holds))*100

Weighted item response rates (excerpt from OMB - STANDARDS AND GUIDELINES FOR STATISTICAL SURVEYS)

PHASE I PHASE 2 PHASE 3

Unweighted response rate is in the acceptable range, Create documentation statement.

No bias study is needed. Provide documentation statement to OMB explaining situation(s).

.

Determine if data to calculate weighted and unweighted response rates are available in Headquarters (HQ).

Conduct nonresponse bias study. Provide the findings to OMB.

For probability-based surveys, document plans to gather data for future analysis and a projected time line for completion.

Provide justification statement to OMB documenting the findings and the statistical soundness of the data.

Unweighted response rate is below 80% but the weighted item response rate is in the acceptable range. Create documentation statement.

Data are not available in Field Offices. States will be given instructions on how to obtain this data for the future.

Data are available in Field Offices and needs to be provided to HQ.

Data are available in HQ. Calculate weighted item response rates for key items.

Provide documentation to OMB of the survey procedures & statistical soundness of the data.

Unweighted and weighted response rate are in the unacceptable range.

For opinion-based surveys, create documentation of survey practices.

OR

OR

OR

NO

YES

OR

OR

Provide justification statement to OMB explaining situation. Examine potential nonresponse bias for future collections.