NASS Review of Part B SuptStmt 6-17-19

Local Food Directories and Survey

OMB NO. 0581-0169

Reviewers:

• Linda J Young, Chief Mathematical Statistician and Director, Research & Development Division

• Ben Reist, - Deputy Dir Science & Planning, Research & Development Division

• Mark Apodaca. Sampling, Editing, and Imputation Methodology Branch Chief, Methodology Division

• Ken Pick, Survey Methodologist, Methodology Division

• Gerald Tillman, Chief Survey Administration Branch, Census and Survey Division

• David Hancock, NASS OMB Clearance Officer, Methodology Division

General Comments:

Part A

In item number 8, AMS has listed the primary NASS people they have consulted with on this joint survey. AMS will need to include any data users or industry people they consulted with as well.

Part B

In item number 1 NASS has made several text suggestions which are included in these revised paragraphs:

The respondent universe for the National Farmers Market Managers Survey is mangers of US farmers’ markets in business in 2019. For this study, a farmers’ market is a collection of two or more farm vendors selling agricultural products directly to customers at a common, recurrent physical location. This study will utilize capture-recapture methodology to ensure complete coverage of this population. Two list sources will be used. One list will be the Farmers Market Directory maintained by the Agricultural Marketing Survey (AMS). AMS annually sends reminder emails to market managers of the over 8,700 farmers markets listed in the Directory that have valid email addresses, State Departments of Agriculture, and State Farmers Market Associations. The Farmers Market Coalition also assists the Department with the annual update push each year by encouraging its members to list and update their listing in the Directory. The AMS Public Affairs Office writes blogs and tweets about the importance of listing and updating existing listings in the Directory. Additionally, AMS conducts internet searches to determine whether markets are still in operation and to collect current email addresses for these farmers markets where possible. Afterwards, AMS sends requests to markets with outdated listings to update their listings.

The second list, consisting of about 16,000 records, has been obtained by web scraping. Record linkage has identified about 400 records on the AMS list that are not on the web-scraped list; the remaining records on the AMS list are not on the web-scraped list. The approximately 7,500 additional records on the web-scraped list are not on the AMS list. Unlike the AMS list, these 7,500 records only on the web-scraped list have not been confirmed to be associated with active farmers’ markets.

The anticipated response rate is 60%.

This response rate will still need attention now that we are adjusting for coverage. All farmers markets identified by AMS will be included in the mail out. Potential farmer markets identified on the open source frame and not on the AMS List of farmer markets are eligible to be selected. Or vice versa if we use the AMS list to measure the incompleteness of the open source list.

In one of the early documents (notes from a meeting between AMS and NASS) it stated that the “sample” would not exceed 10,000. Is this still the case? I think the cost estimates used 10,000.

Will need to add in the details similar to what was included in the JARP OMB Part B (0535-0140).

In item number 2, NASS made numerous text changes that are included in the following paragraphs.

AMS made every effort to gather a complete listing of farmers markets in the U.S. by employing a web scraping service, MACE, to identify farmers markets that advertise their operation on the internet but do not appear on any federal or state listing of farmers markets. AMS contacted state farmers market associations and State Departments of Agriculture to obtain their current listings of farmers markets in their state. In addition, AMS requested farmers market managers that list their markets in USDA’s National Farmers Market Directory to update their listing so that the most accurate contact information of farmers markets is available for the survey. The survey will be mailed twice to survey respondents. Each mailing of the questionnaire will include a cover letter and instructions of how the questionnaire can be completed online. Farmers’ market managers will have the option of responding via mail, on-line, phone enumeration, and/or field enumeration.

The key estimates for this survey are the number of farmers’ markets in the U.S., in each U.S. Census Bureau Region, in each U.S. Census Bureau Division, and each urbanicity category. The sample will be designed using a 2 percent coefficient of variation (CV) target for all key estimates assuming a 60% response rate and that 70% of the farmers’ markets on the frames will be currently active farmers’ markets. To achieve these CV targets, a stratified systematic sample of 10,000 farmers’ markets will be drawn from the union of two lists mentioned in B1. Strata will be defined using the following variables:

• Farmers’ Market Directory indicator

• Web-scraped list indicator

• U.S. Census Bureau Division

• Three level urabanicity category based on the 2013 Rural-Urban Continuum Codes

Within strata a systematic sample will be drawn using the following sort variables:

State

• 2013 Rural-Urban Continuum Code

• County

• City/Town

Based on the availability of additional variables at the time of sampling, more sort variable may be utilized.

The sample will be allocated based on the CV targets. In addition, the sample will be allocated to insure an adequate number of respondents for each list membership combination to adjust for coverage (i.e. directory only, web-scrape only, on both lists). Sample size for each stratum will be adjusted to account for differential nonresponse and out-of-scope rates (e.g. out-of-business farmers’ markets) between strata. Nonresponse and out-of-scope records are particularly a concern for the web-scraped list since these markets might not have a prior relationship with AMS or USDA, and these records have not been confirmed to be associated with active farmers’ markets.

Unit nonresponse will be accounted for using a standard nonresponse weighting adjustment. In addition, a capture-recapture approach will be used to account for under coverage of the union of the two frames.

AMS worked with USDA’s National Agriculture Statistics Service (NASS) to develop the survey questionnaire. NASS will administer the survey.

The key estimates mentioned in paragraph 2 will still need to be confirmed.

The term urbanicity in paragraph 2 above will need to be defined. You may want to include a mapping of Rural-Urban Continuum Code to Urban, Suburban, Rural.

A possible suggestion:

1. Metro (1,2,3), non-metro but metro adjacent (4,6,8), and non-metro and not metro adjacent (5,7,9).

2. Metro (1,2,3), Urban non-metro (4,5,6,7), and non-metro completely rural (8,9).

In item number 3 NASS has made several text suggestions which are included in this revised paragraph:

The survey questionnaire will be mailed twice to farmer’s market managers. Each mailing of the questionnaire will include a cover letter and instructions of how the questionnaire can be completed online. Farmer’s market managers will have the option of responding via mail or on-line. Phone and limited field enumeration will be used for mail/web non-respondents. These procedures are expected to maximize the response rate. Data collection may be challenging since farmers markets locations change frequently, and farmer’s market management has a high turnover rate. We will strive to achieve a 60 percent response rate.

In item number 4 NASS has made a couple minor changes which are included in this revised paragraph:

NASS conducted cognitive testing on the questionnaire with prospective respondents to evaluate the time necessary to complete the questionnaire and to assess the instrument for understanding clarity. It is estimated that the questionnaire will take 28 minutes to complete.

In item number 5 NASS has made a several changes which are included in this revised paragraph:

everale minor changes oned in paragraph 2 will still need to be confirmed.
ollowing paragraphs.
should pMS will need to includeThe AMS/MSD employee who will coordinate this data collection with NASS is Edward Ragland (202-690-1327).

Several NASS units contributed in the development of this project:

• Linda Young, Director, Research and Development Division: (202) 690-1401

• Ben Reist, - Deputy Dir Science & Planning, Research & Development Division: (202) 720-0324

• Mark Apodaca, Chief, Sampling, Editing, and Imputation Methodology Branch: (202)720-4008

• Ken Pick, Survey Methodologist, Methodology Division: (202) 720-5778

• Gerald Tillman, Chief Survey Administration Branch, Census and Survey Division: (202) 720-3895

• David Hancock, NASS OMB Clearance Officer, Methodology Division: (202) 690-2388