Census of Ag Enumerator and Filed Office Pretesting Guide

Census of Agriculture

Enumerator and Field Office

Pretesting Guide

1. GENERAL

Basically, we are attempting to evaluate the content as well as the structure of the instrument. We want to make sure all of the questions:

1. Are asked in a manner that is understandable to the respondent

2. The terms and answer choices are presented correctly

3. That structure of the questions elicit the desired information

Of primary interest are any questions that respondents have general difficulty with or any terms and definitions within a question that are problematic. The key to fixing these situations is determining “why” something is unclear or a problem, and then asking the respondent how they would fix it—e.g. “how can we make this better for you?” Our goal is in pretesting is twofold: to gather feedback on the questions as they are currently presented, as well as solicit suggestions for improvements.

Introducing the Interview

Before the interview begins, explain the interview process to the respondent. The primary objective in a pretest interview is to find out how respondents are interpreting and answering the questions, NOT collecting data. Explain to the respondent that the questions may not be worded or formatted in the best way and that we are interested in any problem areas the respondents have. This is NOT a test of the respondent’s reporting ability. The objective is to make the report form as easy to complete as possible.

Conducting the Interview

Unlike a standard interview you are encouraged to interact with the respondent. Ask the respondent to think aloud as they are completing the questionnaire. Probe the respondent when he or she seems confused about a question. Encourage the respondent to describe their thought process when coming up with the answer.

We must be careful not to lead the respondent or “interpret” items they may be having difficulty on. First, we want to ensure that the question is indeed problematic—not that they are just thinking about their answer. Next, we need to find out what exactly was difficult to answer or not understandable. Finally, we want to determine the respondent’s interpretation of what we are asking and how the respondent would be better able to understand the question. It is often useful to ask the respondent to paraphrase the question, by repeating the question in their own words.

By getting the respondent’s interpretation, we learn how they are hearing what we are asking. A question could potentially be completely misinterpreted from what we intended to ask. Hopefully, this is not the case (or at least a very rare occurrence) but in the event that it occurs, we need to hear how respondents perceive what we are asking and take this into consideration as we revamp a question.

By the same token, positive feedback is also welcome. Anytime a respondent comments on something that is asked well, terms correctly used, usefulness of instructions or definitions, etc. we want to record or note that as well.

Example Questions/Probes to ask during the pre-test interview: Remember that the pre-test is an interactive process so not all questions will need to be asked every time.

• Is there anything about this question that you find confusing?

• What do you think the researcher wants to find out about in this question?

• Is there anything that could be changed to make this question easier to understand?

• Could you please tell me what you were thinking about to come up with your answer?

Throughout the interview you should be keeping notes of when a respondent gives any unexpected responses, such as their own definitions/interpretations, reporting units, etc.

2. SECTION GUIDELINES

The following are questions to ask in the sections specified where we think respondents may have difficulty reporting. You will need to rely on observation and follow up probing of respondents to determine the answers. If respondents appear to have any other problems, please feel free to add additional follow up questions. Don’t assume you know what respondents are doing – ASK THEM.

FACE PAGE:

Did R pause and read the face page? What information did they read?

SECTION 1: Acreage in 2009

Step 1, Land Owned:

Step 2, Land Rented From: any difficulty reporting this item? Did R consider the includes and excludes? Did they have any land rented rent-free? Was it correctly reported?

Step 3, Land Rented To: any difficulty reporting this item? Did R have any land rented rent free? Was it correctly reported?

Step 4, Calculation of Total Acres: Did R complete step 4 correctly? Did R perceive this to be undue burden?

Step 5, AUM question: Did R have AUM land? Are they able to report the number of acres? Did they initially include acres in Box 2? Did they correctly exclude them from the Total Acres in Box 4?

Step 6, If Acres = 0: Did R read bolded instruction to report for Total Acres Operated on rest of the form?

Did you use the instruction sheet?

SECTION 2: Land

Cropland: Did R read list of inclusions?

Pasture: How did R define “cropland used only for pasture or grazing?” How is this different from permanent pasture? How does respondent define “Woodland Pasture” and how is it different from permanent pasture?

Woodland:

Other Land: what was included here? Did R have a farmstead? Was it reported here? Did R have any difficulty reporting Other Land?

Total Acres: were any acres reported more than once in this section? Were acres added correctly? Did box D = box E? If not, what was the discrepancy and what did the R do?

Did the respondent have CRP, WRP or other ag program land? If so, was it reported in this section? In what item?

Ask respondent if any of their land could have been reported in more than one item in this section? If so, where did they report it? How did they decide where to report it?

SECTION 4: CRP and Crop Insurance

If CRP land was reported, where did respondent report the land in section 2?

SECTION 5: Field Crops

Did you report only the amount harvested?

SECTION 6: Grain Storage Capacity

Did you report the total storage capacity or total amount in storage right now?

SECTION 7: Hay and Forage Crops

Question 2: Did you make two or more cuttings from any acres?

How many times did you count these acres?

Did you count production from all cuttings?

SECTION 9: Nursery, Greenhouse, Floriculture, Sod, Mushrooms, Vegetable Seeds, and Propagative Materials

Where you able to report mushroom area using the pre-printed units?

SECTION 12: BERRIES

Question 2: Describe acres, determine if recorded correctly.

SECTION 13: CATTLE AND CALVES

SECTION 14: SHEEP, LAMBS AND GOATS

Was it clear that this was asking about animals OWNED? Were any animals NOT owned included?

SECTION 15: CHICKENS AND TURKEYS

Were the roosters included in this section? Were any poultry other than chickens and turkeys reported in this section?

Determine if value of sales was correctly reported.

SECTION 18: BEES AND CROP POLLINATION

Did respondent include all bees used for crop pollination on this operation?

SECTION 19A (2 versions): EQUINE

If you have more than one operation with equine, alternate the version of the questions used. After the respondent has answered the section, provide them with the alternative version and get their opinion on which they prefer.

Question 1: For any horses not owned on this operation, who owns them?

Question 5: can R easily categorize themselves into one of the given types of operations? Do these categories make sense to them?

SECTION 19B: OTHER POULTRY, LIVESTOCK AND LIVESTOCK PRODUCTS

Were any animals that should have been reported in other sections reported here? (i.e. cattle, chickens, hogs, horses, aquaculture, etc.)

SECTION 20: Production contracts and custom feeding

Question 1: Are respondents familiar with the term “production contracts”?

Question 2: Can respondents provide the information?

Do respondents understand that both corn and soybean contracts were reported in item 2i?

If no value reported, why? If not answered, why?

Question 3: Did respondents know all information? If not answered, why?

SECTION 22: organic agriculture

Can the respondent answer the questions in this section?

Question 4: Did the respondent report only organic sales?

Did the respondent include value added products such as wine, jams, and maple syrup?

SECTION 23: Commodity Credit coorporation Loans (CCC)

Question 1: Is the respondent familiar with CCC loans?

Question 2: Was the respondent able to answer the question?

Did the respondent include money paid back from loans received prior to 2009? Did the respondent consider the “includes”?

Question 3: Did the respondent only report amount paid back in 2009?

Did the respondent have access to the information?

SECTION 24: Federal and State agricultural program payments

Question 1: Did the respondent consider the “excludes”?

Question 2: Did the respondent have any difficulty reporting in the categories as defined?

Did the respondent understand the differences among the categories?

Did the respondent have access to the information?

SECTION 25: Income from farm-related sources

General: Was the respondent able to report these values before taxes and expenses?

SECTION 26: Production Expenses

Questions 1, 2: Were you able to break out questions 1 and 2?

Question 7: What did you include for utilities?

SECTION 27: Farm Labor

Question 3: What is included in this number?

Were migrant workers in item 3 also included in the item 1 number of hired workers (if not contract labor)?

If contract labor is used, are you able to report number of laborers?

SECTION 28: FERTILIZERS AND CHEMICALS APPLIED

Verify that they reported for nematicide applications and not just planting nematode resistant crops.

Was defoliant question correctly reported, especially for cotton growers.

SECTION 29: MARKET VALUE OF LAND, BUILDINGS, MACHINERY, AND EQUIPMENT

Was the value of land and buildings rented to others included in value of land and buildings owned?

SECTION XX: Land Use/Practices

General observations about problems with questions, terminology, ability to report acres?

Ask respondent to paraphrase the questions – i.e. what do you think this question is asking?

How does respondent define a conservation easement?

SECTION 34: Operator Characteristics

General observations on problems with disability questions?

Year began operating: Was the difference between the two questions (year began operating this operation vs year began operating any operation) clear?