0243 - 2015 Census Content Test - Supporting Statement B

Supporting Statement

2015-2016 CENSUS OF AGRICULTURE CONTENT TEST

OMB No. 0535-0243

B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS

1. Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection has been conducted previously, include the actual response rate achieved during the last collection.

Up to 50 cognitive interviews and usability interviews will be conducted under the Generic Testing Docket (OMB Number 0535-0248) prior to the work described here. Those interviews will be conducted with operations selected from NASS’s list frame based on characteristics about their operation including geography, the types and amounts of commodities grown, size of the operation, and other characteristics.

The Census of Agriculture Content Test described in this docket will consist of three phases. Phase One will consist of a mail out test with telephone follow-up, using a stratified random sample of 50,000 farm and ranch operations from a universe of approximately 2.2 million agricultural operations. Telephone follow-up contacts will be monitored for quality assurance and to assess any problems with the telephone instrument. Stratification will be used to ensure sufficient coverage of various sizes, locations, and types of agricultural operations. The sample will be divided into control and treatment groups to test alternative versions of the questionnaires. Sampled operations will be able to respond with either the supplied paper report form (via mail) or through an Internet version of the questionnaire. A minimum 50 percent response is expected, which will adequately support the test design.

Phase Two will consist of up to 400 randomly selected agricultural operations from a universe of approximately 2.2 million agricultural operations. Sample operations will be asked to participate in cognitive interviews or usability interviews. The sample will consist of some agricultural operations that completed the questionnaire in Phase One, as well as some additional operations. The cognitive interviews and usability interviews conducted with Phase One respondents will be used to improve the overall 2017 Census of Agriculture questionnaire by allowing NASS to follow-up with respondents to explain unusual responses and to ascertain their comprehension of the questions. The remainder of the cognitive interview sample will be randomly selected operations that meet size and type criteria. A minimum 80 percent response is expected, which is adequate for cognitive interviews.

Phase Three will consist of a stratified random sample of 15,000 farm and ranch operations from a universe of approximately 1.5 million agricultural operations believed to have Internet access. Stratification will be used to ensure sufficient coverage of various sizes and types of agricultural operations. The sample will be divided into control and treatment groups to test alternative versions of the on-line questionnaires and methods to increase on-line response. A minimum 50 percent response is expected, which will adequately support the test design.

2. Describe the procedures for the collection of information including:

• statistical methodology for stratification and sample selection,

• estimation procedure,

• degree of accuracy needed for the purpose described in the justification,

• unusual problems requiring specialized sampling procedures

The sample sizes of 50,000 for Phase One will be sufficient to test key differences between five treatment groups with 95 percent power. Similarly, the sample size of 15,000 for Phase Three will be sufficient to test key differences between two treatment groups with 95 percent power. This is based on an anticipated 50 percent response rate for each Phase. The following analyses supports these sample sizes:

A two-sided multivariate hypothesis test (H₀ : p_i = p_j vs. H₁: p_i ≠ p_j, for sample size n_i =n_j = n) was used to compute treatment group sample sizes. To conduct m tests and attain an experiment-wise significance level alpha, then – using the conservative Bonferroni adjustment – the required sample size is:

Where:

n_i , n_j : treatment group sample size.

p_i, p_j: response rate.

σ_i² , σ_j² : variance of the response rate.

m: m pair tests

α: Type I error.

β: Type II error.

∆ : Margin of error (P_i-P_j).

i, j : 1,2,3, 4,5 for Phase One and 1,2 for Phase Three

A binomial approximation of the normal variance (based on an estimated minimum response rate ( of 50 percent) was used to calculate a common variance ( .

Phase One

Using α, power (1- β) and ∆ equal to 0.01, 0.95 and 0.05, respectively, to test differences between five treatment groups (m=5), resulted in a sample size of 11,211 per group. Therefore, for the five treatment groups planned for Phase One, the total sample size needed is approximately 50,000.

Phase Three

Using α, power (1- β) and ∆ equal to 0.01, 0.90 and 0.05, respectively, to test differences between two treatment groups (m=2), resulted in a sample size of 7,446 per group. Therefore, for the two treatment groups planned for Phase Three, the total sample size needed is approximately 15,000.

The sample size for the cognitive interviews and usability interviews includes cognitive interviews for several purposes from fall 2015 through approximately fall 2017. First, there will be a small number (20 – 30 interviews) used between now and October to make final changes to the questionnaire used for Phase 1 (although most of the cognitive interviews for this purpose will be conducted under the Generic Testing Docket (OMB Number 0535-0248)). Second, there will be several targeted cognitive interviews conducted as follow-up interviews to Phase 1 (approximately 150 interviews). These interviews will target respondents who had problems with the paper or Internet version of the questionnaire (determined by inconsistent or unexpected responses) or reported differently than expected (determined by comparing reported data with list frame control data). Third, usability interviews will be conducted prior to and after the Internet test (approximately 150 interviews). The Internet instrument will be very different from previous versions of the Internet questionnaire as it will be much more dynamic.

Due to the length of the questionnaire and the extensive nature of some of the probes used for some sections and some operations, the cognitive interviews target only specific sections of the questionnaire. This requires more cognitive interviews overall, but allows us to limit the interviews to 1.5 – 2 hours each.

3. Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

Because results are not intended for publication, the anticipated response rate of 50 percent is sufficient to provide sound basis for determining final 2017 Census of Agriculture instrument design.

4. Describe any tests of procedures or methods to be undertaken.

The methodology for this information collection is based on previous content tests and follows accepted practices for surveys of this nature.

5. Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The Census of Agriculture Content Test methodology and procedures are overseen by the Research and Development Division; Chief Cognitive Research Methodologist is Jaki S. McCarthy, (703) 877-8700, and Methodology Division, Standards and Survey Development Methodology Branch Chief, Dan Beckler, (202) 720-8858.

The sample size is determined by the Sampling, Editing, and Imputation Methodology Branch, Methodology Division; Branch Chief is Mark Apodaca, (202) 720-5805.

Data collection is carried out by NASS National Operations Division (NOD); the NOD Director is Joseph Prusacki, (314) 595-9501.

The overall Census of Agriculture program is overseen by the Census Planning Branch, Census and Survey Division; Branch Chief is Chris Messer, (202) 690-8747.

.

June 2015

Revised August 2015