Comments from USDA's National Agricultural Statistics Service

APPENDIX G2: NASS COMMENTS AND RESPONSE

FM OMB Package 5-25-11 Review

Nutrition Assistance in Farmers Markets: Understanding Current Operations

Overall

The documentation provided for review was thorough and well written. The survey methodology is sound and seems to address the defined study objectives.

Questionnaire Differences

Two types of differences are present between the farmers market (FM) and direct marketers (DM) questionnaires. These types are differences in response choices and differences in reference areas. These differences could make it difficult to combine and/or compare data from the two versions. In addition these differences could lead to biases in the data should be noted in any results and/or publications.

Response: FNS is interested in examining current operations of nutrition assistance programs. We determined in cognitive testing that direct marketers are an important part of nutrition assistance redemptions but their operations are very different from farmers markets. Since about 40 percent of farmers market type retailers on the STARS database are direct marketers, it was important to capture information about these retailers. Therefore, the differences in topics (reference areas) and response options between the FM and DM surveys is intentional to permit us to elicit information from both groups. Responses to common questions on both surveys may be combined for more power, but we do not intend to compare the responses across the two groups.

Examples of differences:

1. Differences in response choices may bias data. An example is Section E of both forms. Question E2 on the FM version has an additional response choice “Education and training for farmers/food producers” that is not listed on the DM version.

Other examples of these types of differences:

FM A5 and DM F4

FM Section B and DM Section C

2. Differences in reference areas may lead to bias in the data. An example is question A4 of the FM version and question F4 of the DM version. The FM version the respondent is asked to detail the days and hours by month for the farmers market they operate. The DM version the respondent is asked to detail the days and hours by month for the outlet where they redeem the most SNAP benefits. The DM version respondent may or may not be referencing a farmers market.

Other instances these type of differences:

FM A5 and DM F4

FM A12 and DM F14

As noted above, we determined that it was important to capture information on direct marketers because they make up a substantial portion of SNAP redemptions. An objective of the survey is to capture the operations of a specific market location. Since direct marketers may frequent several locations (outlets), we asked them to select an outlet where they redeem the most SNAP redemptions.

Undercoverage

How the estimate for the number of farmers markets per county on the AMS file was calculated will influence how this value can be used in expected undercoverage calculations. If the total number of reported farmers markets divided by the total number of counties, then it is assumed that the non-reporting counties do not have a significant portion of the farmers markets. This implies that the amount of undercoverage is minimal and a reduced sub-study would be sufficient.

If the average is calculated by dividing the total number of reported farmers markets by the total number of reported counties, then it is assumed that the counties that do not have any reported farmers markets do in fact have farmers markets, however the list is incomplete. This implies that further investigation of the undercoverage is warranted and a full sub-study should be conducted.

Our working assumption is described in the 2^nd paragraph above; since we have no reason to assume that non-reporting counties do not have farmers markets.

The AMS list is open for updates and additions for multiple sources and the same farmers market may have been added by different sources. Prior to any calculations, it may be helpful to ensure that the AMS list does not contain any duplication.

This is a good suggestion. We will plan to de-duplicate the AMS directory based on name and address of each market.

The undercoverage of the AMS list is composed to two types of farmers markets: 1) SNAP authorized farmers markets, 2) farmers markets never SNAP authorized. An indication of the proportion of SNAP authorized farmers markets not included on the AMS list can be calculated by comparing the number of SNAP authorized farmers markets on STARS II to those on the AMS list. In part of A.2, Overview of undercoverage sub-study of farmers markets, it is noted that between STARS II and the AMS list about 85% of farmers markets are identified. This first type of undercoverage in conjunction with the average number of reported farmers markets per reported county can provide an indication for determining the expected number of missing farmers markets. Using this indication could provide a solid base for calculating the number of new markets and the sample size for new markets.

As suggested, we calculated the number of SNAP authorized markets on the current AMS directory and found that approximately 20 percent of markets on the STARS database were not on AMS file. However, we believe the difference is much greater then that. For one, we know that STARS is an underestimate of all farmers markets since there are umbrella organizations on the STARS file that represent multiple individual farmers markets that may be listed on the AMS file. If those markets were split out on the STARS file, the difference between the two files would be larger. Second, as suggested above, we understand that there may be some duplication of markets on the AMS file, suggesting that there are fewer markets on the AMS file than first appear. Therefore, there is no reason to suggest that the undercoverage would be less than 85 percent, and in fact there is some evidence to suggest that could be more. Once we get the new, more complete AMS directory (in August 2011), and clean the STARS database to dissagregate the umbrella organizations, we will have better information to calculate the sample size for the undercoverage study.

Sample Sizes

It is unclear throughout the documents what the sample sizes are for this survey. Different values are reported in the following portions of the documents: A.2 (Overview of National Survey of Nutrition Assistance and Farmers/Farmers markets), Table A.12A (Respondent Burden and Cost Estimate), Table B.1 (Sample Sizes for Precision Estimates), Table B.2.3.1 (Samples Sizes, Selection Probabilities and Finite Population Corrections for Minimally Detectable Differences at p₀ = 0.2 and p₀ = 0.5).

In A.2 narrative, the sample size for Farmers Markets is correct; for the direct marketing farmers, this number has been corrected. The sample size has been corrected in Table A.12A. No change was necessary to the sample sizes in Tables B.1 and B.2.3.1.

Focus Group

A portion of the SNAP recipients population may be excluded by the survey methodology and could introduce bias into the survey results. Screening focus group members solely by telephone could exclude SNAP participants that do not have incoming telephone service. Selecting only English speaking SNAP focus group members may reduce the number of potential group members. These conditions should be noted in any results and publications as potential biases to the data.

The overall population of total SNAP participants is not detailed in this document; hence it is difficult to determine if 40 focus group participants will be adequate to account for any geographical differences.

The purpose of the focus group is not to gather nationally representative data, but to provide some context for farmers market operations from a client perspective. This exploratory work will also provide a foundation for follow-on work that is planned for 2012.

Specific Document Comments

FM SSA 11-5-25 (KEK).docx

Citation style should be consistent throughout the document. Use either footnotes or in work citation with the author and year.

Page iv: Appendices B2-B4

Title does not match the attached zip file, B2 is the direct farmer in English.

The mismatch between title and reference in the Appendix has been corrected.

Page 2: “SNAP⁸”

It may be helpful to move this reference to the renaming of food stamps to SNAP to the first instance of the SNAP abbreviation (the end of the previous paragraph).

The reference has been moved, per suggestion.

Page 3: “The goal of the proposed study is to increase FNS’s understanding of the farmers market environment and inform future policy in a manner that attracts a greater numbers of farmers markets and SNAP shoppers.”

The second portion of the goal, to inform future policy, is unclear. Is the goal to influence future policy decisions? Or is the goal to determine factors influencing a farmer’s markets decision to participate in SNAP?

This statement is changed as follows: The goal of the proposed study is to increase FNS’s understanding of the farmers market environment such that it can inform future policy in a manner that attracts a greater numbers of farmers markets and SNAP shoppers.

Page 4: FM and DM

First use of the abbreviation is before the definition.

The abbreviation for FM and DM has been provided before the definition. .

Page 7: sub-study sample

The population for the sample is unclear. Are counties that are on AMS directory included? A more complete undercoverage estimate would be obtained by including counties on and not on the AMS list in the population.

Yes, the sampling frame consists of all 3,143 counties as explained in SSA, page 7 and SSB, page 26.

Page 8: “inform FNS policy decisions”

See Page 3 above

See revision to page 3 sentence.

Page 10: “survey of four strata”

On page 5 there were four groups defined. Are these the four strata referred to on page 10?

We have replaced the term “groups” with strata to avoid confusion due to terminology.

Page 16: “so informed before the either the survey”

First “the” can be deleted.

We have deleted “the” as per suggestion.

Page 19: Table A.12A

Formatting is not consistent across all portions of the table. Business (for profit) Subtotals do not have commas, however All Total has commas.

The formatting has been standardized across the table and commas have been deleted from the All Total cells.

Farms Subtotal vertical divider is missing between Estimated total hours and Annualized cost of respondent burden.

The vertical divider has been added between Estimated total hours and Annualized cost of respondent burden.

Subtotal Annualized cost of respondent burden is to two decimals, all total has zero decimals

Subtotals for individuals or households do not equal the parts for number of respondents and total annual responses. The all total for the same categories does not equal the parts.

Two decimals have been used for the All Total cell. The subtotals for individuals or households has been corrected to equal parts for the respondents and total annual responses. The All total numbers are verified and corrected as needed.

FM SSB 11-5-25 (KEK).docx

Page 26: Undercoverage of AMS file

It is unclear how it was determined that the AMS database should include 2.4 farmers markets per county.

The AMS directory contains approximately two markets per county (6,132 records on AMS directory/3,143 counties in the US = 1.95 markets per county). On the assumption that there is only 85 percent coverage of the total number of markets, the true average number of markets per county is 2.4 (2 markets per county on AMS file/.85% = 2.4 markets per county actual).

FM Appendix B1 FM manager survey 11-5-24.docx

Question 1: How will this questionnaire be used if market manager is not chosen?

(The form is intended for managers of farmers markets.)

The reviewers are correct in pointing out that the form is intended to be completed by farmers market mangers. We note that in some situations a market may not have a formal market manager and a paid staff or volunteer is responsible for providing oversight. We are interested in capturing this information to understand the current operations of farmers markets. We expect these individuals to be able to complete the survey with the same level of accuracy as the market manager. After the data are complete, we will examine the proportion of surveys completed by non-market managers. We will also compare the number of items with missing or don’t know values between the surveys completed by the manager and non-managers. We will treat completed responses as valid and include them in all item and aggregate analysis.

Question 2: Is there a standard unit that the responses should fall in (seasons, years, months)?

We have added the following standard to enable respondents to report information:

___ years and ____ months

Questions A4: The question asks the number of hours the market was open in each month, by day.

The example shows how many hours the market is open on a Saturday. If the market was open for 3 hours each Saturday during the month for 4 Saturdays, should the respondent enter 12 or 3. With minor rewording the respondent will have an easier time reporting total hours for the month or average hours per day.

The question is reworded to obtain average hours per day.

Question A5: In the Direct Marketers version, plants are a separate item. Where would a respondent place herb plants? Reorganizing the order of items listed may make the table easier to complete.

For example:

Fruits

Vegetables

Milk

Cheese

Other dairy products

Fish or Seafood

Meat or Poultry

Bread or rolls

Other baked goods

Juice or cider

Prepared foods

Non-food items (plants, crafts, etc.)

The list has been revised per recommendation.

Question A3: This question asks how many farmers markets the organization/partnership operates. In the following questions the respondent is asked about this farmers market.

Should the respondent be including all the farmers markets that their organization/partnership operate or only the one that was sampled?

With almost every question the respondent is asked about “this” farmers market which refers to the one that was sampled. A3 is simply asking a follow up to A2 If the respondent answered A2 with “operated in partnership” or “operated by another organization.”

Question A12: The question wording may lead to vendors that did not pay a cash fee to skip the question.

There is a response option for this scenario.

Question A13: Does this include credit/debit card processing machines not operated by the farmers market, but by individual vendors?

No, the question is asking only about availability of credit/debit card processing machines operated by the farmers market, so as to understand if the market has access to a credit/debit card machine but does not accept or redeem EBT.

Question B5b: Initial questions screen for farmers markets there were authorized to participate in SNAP, however one of the answers is neither the farmers market nor individual vendors were SNAP authorized. Is this included to ensure accurate responses to question B5?

Yes, this is included to ensure accurate responses to question B5.

FM Appendix B2 direct farmer survey 11-5-24.docx

Question A4: The question is difficult to read and understand.

A possible rewording: In the table below list the number of people, including yourself, that were employed or volunteered on your or the operating farmer's business from October 2010 to September 2011.

This question has been reworded per suggestion.

Question B3: see FM Appendix B1 FM manager survey 11-5-24.docx question A5

This question is revised per recommendation.

Question B4: Should this value include all products types listed in B3? (i.e. should the non-food items be included?)

Yes, it should include value for items included in B3. This clarification, to include food and non-food items, has been added in parentheticals to Question B4.

Question C2: Respondents may have both visited the USDA FNS website and seen the handbook. This would result in both selection boxes being checked.

It is possible to have both boxes checked. The intent for providing these two response options is that while direct marketing farmers may have been to the USDA website, they may not have seen the handbook for farmers markets. We have added an instruction to check all that apply after the question.

Question C6: Respondents that have already noted that they were not authorized to accept SNAP and detailed the main reason are being asked again for the main reason they are not participating.

Skip pattern hat prevents them from answering it twice.

The skip pattern has been corrected and respondents who do are not authorized to accept SNAP and provide a reason will not answer twice.

Question F3: see FM Appendix B1 FM manager survey 11-5-24.docx question A4

The recommended change has been made to Question F3.

Question F4: see FM Appendix B1 FM manager survey 11-5-24.docx question A5

The recommended change has been made to Question F4.

Question F8: Go to destination does not exist. Maybe update to: go to next page.

Instructions have been corrected and respondents are directed to go to next page.

Question F7a: Numbering is out of sync. F8a instead?

Numbering has been corrected.

Question F8a: Numbering is out of sync. F9a instead?

Numbering has been corrected.

Question F14: see FM Appendix B1 FM manager survey 11-5-24.docx question A12

There is a response option for this scenario.

Question F19: Capitalize Samples

“Samples” has been capitalized.

Spanish survey forms Appendix B3-B4 were not available for review.

Spanish survey forms are included in the current submission.

FM Appendix C1 Introductory Survey Participant Email 11-5-24.docx

This document appears to be the same as the letter included in the mail survey packet. Consider modifications to make this document more appropriate for an email message.

Example:

“You can help by ….the enclosed Farmers Market Manager OR the Direct Marketing Farmer survey”.

Perhaps replace enclosed with attached or U.S. mailed.

Recommended changes have been made to the introductory email.

FM Appendix E1 telephone script 11-5-13.doc

Spanish script was not available for review.

Telephone script in Spanish language is included in the current submission.

FM Appendix F1 FG Recruit_Script_Participant 11-05-16.docx

Have other methods of contacting participants of the focus groups been considered? SNAP participants are low-income Americans who may not be easily reached by telephone.

Given the small number of focus groups being conducted and with the knowledge that is sample is not intended to be representative; we will only contact those who can be reached by telephone.

Farmers Market Manager and Direct Market forms are both being created in English and Spanish. Will a significant portion of the SNAP population be excluded when only English speaking respondents are included in the focus groups?

No focus groups will be conducted in Spanish. The proposed focus groups are exploratory in nature, designed to gather limited data on the market features and operations that may facilitate or hinder households form shopping there. A new study, Task 2 under this BPA, will evaluate SNAP participants’ knowledge, attitudes, and behaviors about shopping at local farmers markets. That study will also include a larger number of focus groups, with both English and Spanish speakers.

Question 1: Respondents may be hesitant to confirm the year that they were born over the phone when they are not expecting a call.

We will work with the state office or local agency to obtain information on household composition and demographics; this will enable us directly ask for the adult in the household by name when we conduct the telephone screener, and negate the need to obtain year of birth.

Question 2: May want to consider changing the wording of the introduction paragraphs. Offering a $75.00 incentive prior to the screening questions could lead to respondents assuming they will be paid after answering the questions over the phone.

The introduction paragraph has been revised to clarify that the incentive will be paid after they have completed their participation in focus group discussions.

FM Appendix F3 FG Participant – Demo survey 11-5-11.docx

Questions 2 and 2a: “Go to 7”

This question (7) does not exist on the questionnaire.

This skip pattern has been corrected.

FM Appendix G1 Farmers Market: 60 day FRN Comments Received and Action.docx

Page 3: NYC Health Bucks initiative evaluation, is there any duplication between the surveys that would need to be noted in FM SSA 11-5-25 (KEK).docx in item A.4?

FNS is interested in collecting nationally representative data on Farmers Markets since the operations of farmers markets vary considerably across states and regions. The evaluation of the NYC Healthy Bucks initiative is limited to NYC. Much of what was collect in the NYC study is not generalizable to markets outside of NYC (e.g., built environment). We note this in Supporting Statement A, item A.4.

Westat’s IRB Appendix H2 was not available for review

Westat’s IRB approval letter is included in the current submission.

16