NASS Review

Review of the Survey of Hemp Producers and Production Trends

OMB NO. 0581 – NEW

June 11, 2021

Review was conducted by Mike Jacobsen - Methodology Division

and

David Hancock – Agricultural Statistician - NASS – OMB Clearance Officer

Comments and Questions on the Domestic Hemp Production Program Request for Approval

Statement A

I have two main concerns with the information presented in Statement A. First, I am not clear on how the survey instrument was created. The statement does not provide any information on how the questions were selected and tested. I want to know if producers can provide the information requested in the survey and this means whether the survey was tested on producers. Second, I see no particular use for the collected data. The current statement does not provide a defined output for the collected data other than it being included in a report to Congress. However, the statement says the survey will collect information on current costs, production practices and marketing practices. It is not clear on how this information will be useful to the public in a Congressional report.

Pg 1 Question 2: What is the final product that will come out of this data collection effort? Is it a written report or set of data tables? How will this data be used? Who will use this data? This seems to be in conflict with the response to Question 16 where it is stated only Congress will see this data in a final report.

Pg 4 Question 4: Some more clarity here would be helpful to see how this differs from other data collection efforts.

Pg 7 Question 8: What type of testing was done to design the instrument? Were there field tests or cognitive interviews? How many producers provided feedback on the instrument? How different were the responding producers?

Pg 12 Question 12: Why does this number differ from what was put in the 60-day FR Notice?

Pg 14 Question 16: Why is this data collection needed if it will not be disseminated? The instrument contains many questions that would be useful to the hemp industry. So, why is it not being made available to the industry?

Statement B

This statement is very vague in describing the survey methodology. While it describes how the data will be collected (and I have some questions on the sampling and collection procedures), it provides no information on how the data will be handled after data collection. I have many questions on whether the data will be reviewed and errors fixed, especially with the heavy use of skip logic in the survey. I do not see any plan on how to handle nonresponse (both unit- and item-level). Furthermore, I do not understand if this data collection is intended to be made representative of the population of US hemp growers. If so, there needs to be some discussion on how coverage issues will be addressed, especially if a state list is found to not accurately cover hemp growers. Finally, the level of accuracy for the estimates are not clear, leading me to wonder what is considered a precise estimate from the data.

Pg 1 Question 1:

• What is the definition of a “producer”? Is it what they produce? How much they produce? How much they plant?

• How many acres they plant? How will the data quality of the state lists be assessed? How will the records with poor or missing contact data on the lists be handled? How will you choose between multiple records in a list that have the same contact information?

• Would there be a possibility of a State Dept. of Ag not providing a list? If so, how would data collection for that state be handled?

• What other potential list sources have been considered? Would AMS have a list that could be used?

• How will duplicates in the state lists be handled? The survey instrument asks producers to report on operations outside of their state of identification. The way this is worded suggests that a respondent may receive multiple surveys since they could be identified on multiple lists. Major quibble is who exactly will be sending out the survey. Will the Depts. of Ag have these capabilities? Or are you sending it out through a group like NASDA?

Pg 2 Question 2A:

• The difference between this question and the one above is this pertains to final selected group or producers who will actually receive the survey. More detail is needed here on how you prepare the sample after you have compiled the Depts. Of Ag lists. What is the census sample file? What do the final eligible census producers look like in the file? Who do you expect to include and not include in the final file to be loaded into the data collection software?

Pg 3 Question 2B:

• So, what will be done with the data after it is collected to make it useful for inference? How will producer nonresponse be handled? Is there a plan to determine if a record is a valid producer if they have not responded? How will records who could not be reached be in the census sample file be accounted for in the final estimates?

• What is the data collection timeframe? When does the survey start and when does it end? When will data collection start and end? Before, during or after the planting season?

• What tool will you use to develop the instrument? SurveyMonkey? Something else?

• How will the emails be sent out? I have seen data collection issues pile up due to emails not be sent out at the appropriate time because of technological issues.

• What will be the plan if 75% response rate is not achieved after all of the above efforts have been worked? Will you look at potential nonresponse bias (i.e., perform a nonresponse bias assessment)?

Pg 3/4 Question 2C:

• How will standard errors be calculated? What is considered an accurate estimate? What is the level of granularity required for this survey? State level? County level?

• Could there be a potential event (i.e., a pandemic) that would require you to recontact producers? You cannot predict the future but it would be helpful to have a plan B for continuing necessary data collection.

Pg 5 Question 4: How will the skip logic be controlled during data collection? What happens if the respondent does not follow the skip logic? How will the data be corrected after data collection? Is there a plan for data cleaning?

Survey Instrument

I appreciate that the survey instrument has been reviewed by experts but the version that was submitted suggests that it has not been tested. This is especially true with the presence of skip logic. There are several nonexistent questions that the skip logic directs the respondent to answer (see question B8 directing the respondent to a nonexistent B9 for an example). Some of the skip logic directions are too precise (i.e., page numbers included) and can cause issues if the survey is updated. Finally, the skip logic is not present for all response items in some questions (see “Don’t Know” on question C4 as an example), which will lead to respondent confusion. I also have questions about some of the questions concerning the response items in the questions and how respondents could provide answers.