Attachment 4b - TWG ConferenceCallSummary March2012

Attachment 4b – TWG Conference Call Summary March 2012

Technical Working Group Conference Call Summary

FNS Farmers Market Incentive Program Study (FMIPS)

Wednesday, March 14, 2012

 

PARTICIPANTS Alyson Abrami – NY City Health Department, Healthy Bucks Program
Rachel Chadderdon –  Fair Food Network
Anna Curtin – Portland Farmers’ Market
Jezra Thompson  –  Roots of Change
Eric Sean Williams, PhD – Contracting Officer Representative, FNS ORA
Kelly Kinnison, PhD –  Contracting Officer Representative, FNS ORA
Mark Byron- FNS, SFP WIC and Senior Farmers Market Nutrition Program
Melissa King – FM Incentive Programs Project Director
Jane Schulman, PhD –  Corporate Officer
Stacy Miller – Executive Director, Farmers’ Market Coalition
Vibha Vij – Study Manager
Stephen Leard – Research Assistant

 

PURPOSE

The purpose of this meeting was to get feedback from the Technical Working Group (TWG) members on the developed interview to be administered to the organizations regarding their experiences with SNAP incentive programs, and to review the implementation and operational plans for conducting the interviews.

To open the meeting, each of the TWG members related status updates on their respective organizations and incentive programs, focusing on 2011 metrics and opening of new markets and operational channels. An overview of the Task 3 data collection plan was provided, and there was a discussion regarding FNS-collected information that could be exchanged for participating organizations’ self-evaluation data (action item 1). The Task 3 research plan was discussed, and the organizational typology and operational process was clarified further. The group reviewed and provided feedback on the interview instruments, with discussion focused on instrument questions related to potential misuse (action item 2), social web analysis, and additional edits.

SUMMARY

Status updates from TWG members

To open the meeting, each of the TWG members related status updates on their respective organizations and incentive programs, focusing on 2011 metrics and opening of new markets and operational channels.

• The NYC Health Department Healthy Bucks Program saw a 93% redemption rate from its Healthy Bucks program, involving 65 area farmers markets. This number was an increase from 89% the previous year. The Health Department plans to maintain this number of coupons and markets. An independent evaluation of the Healthy Bucks program conducted by the CDC and Abt Associates will be finalized for release in May at the Weight of the Nation conference.

 

• The Fair Food Network Double-Up Food Bucks program recorded an 80% redemption rate from a network of over 70 markets, approximately $600,000. Fair Food Network is also in the process of finalizing results from an independent evaluation, to release in a report within the next few weeks, posted to the Fair Food Network website.

 

• The Portland Farmers’ Market opened a large new market at Portland State University, as well as a successful winter market, to achieve virtually year-round operational status. The Portland Farmer’s Market also submitted paperwork to establish a 5013(c) arm. Another market was added to their incentive program, and the incentive program dollar amount was raised to $7 ($2 increase) at three of seven markets. The Portland Farmer’s Market plans to begin surveying at each of their markets, to identify target populations at each market location, as well as assess what survey method is best to reach these target populations, with a focus on identifying indicators that might be used to facilitate positive changes in behavior. The Portland State University market will not be an incentive site – it is too large to feasibly support an incentive program.

 

• Roots of Change is in its third year as a recipient of a specialty crop block grant. Roots of Change has been working with the California Department of Agriculture to allocate a percentage of its approximately $500,000 in resources towards incentive programs, and has been approved to use 8% for incentive programs (~$40,000). Support will still rely mainly on fundraising efforts. RoC fundraises to cover sponsored markets programmatics, but funding sources available are often in flux, and the markets do a significant amount of fundraising. A new branding effort is underway, with an incentive program, Market Match, being launched in ~120 markets. A new customer survey is being developed, more specific to the RoC mission, with a focus on community development.

 

• The Farmers’ Market Coalition did not have any updates specific to incentive programs, but Ms. Miller described a recent successful webinar on integrity and inspections at farmers markets. Another webinar will be conducted next month focusing on farmers market applications, project evaluation logic models, and planning for outcomes. Ms. Miller distributed an email to the Farmers’ Market consortium list with observations about similarities and differences in farmers market incentive and nutrition programs between markets in the United States and abroad. There is a lot of variety in these programs and how they are administered, and these observations provide an interesting international perspective on how farmers’ markets are developing these programs worldwide.

 

Overview of the Data Collection for Task 3

Ms. King provided an overview of the data collection for Task 3. Data collection for the Farmers Market Operations Study (Task 1, FM Ops) began at the beginning of January and information from the managers responding to this survey will be used to develop the sampling frame for FMIPS (Task 3).

Regarding the typology of the organizations, Ms. King reiterated that this typology will remain highly dynamic and malleable given that there will be overlap across the typologies defined a priori. It is anticipated that the large Type I organizations will be relatively easy to identify in the frame of supporting organizations that result from Task 1. She went on to explain that input from the Type I organizations that are selected and agree to participate will inform and drive the selection of Type 2 and Type 3 organizations.

Telephone interviews will be conducted with multiple members of each participating organization. Anti-fraud Locator EBT Retailer transactions (ALERT) data will also be used in the Task 3 data collection plan. (Example of ALERT audit report – http://www.usda.gov/oig/webdocs/27002-0001-DA.pdf) Organizations that agree to participate will also be asked to supply self-evaluation data.

 

Dr. Williams (FNS) referred to the discussion at the previous TWG meeting, in which some organizations mentioned potential reticence regarding sharing their self-evaluation data without a comparable incentive in return.  He queried the TWG members to find out whether providing participating organizations with information regarding the magnitude of under-counting of SNAP clients shopping at farmers markets when cash is used rather than their EBT card. This information will be available based on data that will come out of Task 2, the SNAP Client Survey. This information could be used by organizations to obtain a more accurate count of the number of SNAP clients frequenting farmers markets.

 

Ms. Chadderdon noted that this information would be useful, but only if it was provided in such a way that it could serve a specific use, such as for the implementation of outreach programs. Ms. King clarified that only SNAP clients will be surveyed in Task 2. The additional information Dr. Williams described providing could demonstrate to organizations the number of people not being reached at a given market, which could then help target outreach efforts – if a large number of SNAP clients are not using a farmers markets, outreach efforts could be targeted towards that population. Dr. Williams explained that this information could be used for a variety of purposes – outreach, better assessments of program effectiveness, and additional data that could be used in applications for funding of various sorts. Markets providing on-time incentive programs or programs with limited budgets might be particularly interested in this information, as was mentioned at a recent consortium meeting. One time incentive programs are intended to encourage return customers, and more accurate information might provide a better picture of how many clients are coming back to the market. Additional information would also be provided on SNAP clients making farmers market purchases but not using their SNAP cards. It was clarified that part of Task 2 is to specifically ask SNAP clients why they are not using incentive program benefits.

 

Mr. Byron expressed interest in information about the reasons WIC incentive program redemption rates are not as high as expected. Ms. Miller agreed the information Dr. Williams described would be exceptionally useful, noting that most organizations at the community level have a mechanism to survey clients that shop at the market, but have less capacity to survey people who are not shopping at the market. She raised two questions: How is this information different from the Task 3 research purpose generally? and How can this difference be targeted to specifically benefit organizations that agree to provide self-evaluation data?

Dr. Williams outlined the three broad observable categories of SNAP clients as they relate to farmers markets:

• Non-shoppers (further sub-divided into SNAP clients who haven’t heard of farmers’ markets and haven’t shopped there, SNAP clients who have heard of farmers’ markets and chosen not to go, and SNAP clients who have visited farmers’ markets but have not returned)

• SNAP clients who shop at farmers’ markets with SNAP

• SNAP clients who shop at farmers’ markets but don’t use their SNAP cards

The last group has not been well-researched, and Dr. Williams noted that it appears to be much bigger than originally thought. Ms. Thompson noted that it would be a great opportunity for participating organizations if coupled with additional data, such as demographics, landscape (retail space, etc), number of farmers’ markets (hours, seasons), and incentives (as well as what kind of incentives). Without these overlays, it would be difficult to utilize new info for effective outreach. Ms. King noted that it might be possible to partially provide this information (e.g. with GIS data), and proposed an action item, for each TWG member to send an email sometime during the week of March 19^th describing what additional information would be ideal relating to the data on SNAP clients who shop at farmers’ markets but don’t use SNAP cards. The group agreed to this action item. In response to a question from Ms. Chadderdon, Dr. Williams clarified that Task 2 does involve an evaluative component in its data collection plan.

 

Discussion of FMIPS Research Plan

Ms. Vij provided an overview of the FMIPS research plan, particularly the pre-testing process and the logistics of selecting organizations and participants, and administering the telephone interviews.

 

Potential pretest participants suggested by Ms. Miller will be contacted and mailed an invitation to participate in the pre-testing process. Five individuals agreeing to participate will then be sent the instruments for review and interviews would be conducted with the five on issues such as interview content, phrasing of questions, and appropriate probes. Based on their feedback and any subsequent revisions, the instruments will be pretested on four more participants. Subsequent to this pretesting, the instruments will be finalized and submitted for OMB clearance, which is required before the study can begin.

 

It was pointed out that the completed interview goals for each organization (15 Type 1, 40 Type 2, 45 Type 3) was only a target estimate. Given the qualitative nature of this part of the research, statistical power and precision are not required, allowing some flexibility on the number of interviews to be completed. Dr. Williams added that it was important to develop these target numbers for the OMB process, which requires an estimate of burden to be calculated. Ms. Vij described the recruitment process, where participating Type 1 organizations will lead to the recruitment of Type 2 organizations, which will lead to the recruitment of Type 3 organizations. This group of approximately 100 organizations, yielding an estimated 210 interviews, includes both executive directors of large organizations, as well as boots-on-the-ground managers at individual farmers' markets.

 

Upon receiving OMB approval, each possible participant will be mailed an invitation to participate, and an interview will be scheduled. The administration of subsequent interviews with other organizations will depend on the suggestions and willingness of the organizations to provide the study with referrals based on other organizations with which they work. Ms. King noted that basing recruitment on pre-existing relationships within and between organizations will hopefully help to endorse the validity of the study an thus maximize organizational participation. She further pointed out that this cluster or social-web design was built into the study as a direct result of feedback from the TWG at the previous meeting. Dr. Schulman described how the goal of this design is to facilitate studying the community in total, with analysis occurring both within and between communities.

 

Both Ms. Abrami and Ms. Curtin discussed concerns with the classification of NYC DMHH and the Portland Farmers' market, respectively, and this led to a discussion of the strengths and weaknesses of the typology, such that some organizations at the Type 3, local level, for example, may not have any clear organizations working “above” them. A consensus was reached that the nature of the relationships was more circular than linear or hierarchical. Ms. King described how the specific aspects of the typology are not the focus of the study, so much as the structure of existing relationships and the determination of that structure; it’s about talking to organizations that work together on incentive programs, or that work closely in any capacity.

Ms. Vij pointed out that the relationships are not one-way, and the flowchart used to represent the study design will be revised to reflect this. Dr. Williams described the goal as trying to be determine generally how the farmers market movement works, - where are the chains, the webs, and the isolates (small organizations working by themselves, with no or weak connections). Ms. Miller offered the Portland Farmers' Market as an example of a Type 3 “isolate,” because it works so independently. Ms. King noted this input, and especially in cases like NYC DMHH special care would be taken to not miss them as a “Type 1” type of organization. Instead she confirmed that the initial selection of sites won’t necessarily be as dependent on size of the organization but a model of operations instead. Ms. Miller and Dr. Williams concluded by saying that Task 1 data will be critical in facilitating a clear path of action for Task 3.

 

Review and Input on Interview Guides

Feedback was requested on protocols and instruments prior to OMB submission. Ms. King reviewed the process of selecting and interacting with organizations. The process would start with the executive director at a Type 1 (of “first”) organization. In some cases, the executive director might be the point person for the incentive program - if she is involved in day-to-day operations, a longer interview can be arranged. The interviews are meant to be open-ended, general questions with several probes.

 

The group reviewed the Type 1 program director interview, which closely resembles the Type 2 and 3 program director interviews. Ms. King asked whether the TWG members considered the questions as lending themselves to the kind of information the study is attempting to elicit. Ms. Thompson thought that the methodology and instruments were successfully collect the right information, and suggested a greater focus on geographical proximity. Particularly with RoC in California, there are some interesting dynamics in populations that are driven by this factor, and the instruments might be adjusted to capture this data.

 

Regarding the fourth question in the interview instruments, related to funding, it was agreed that the question be expanded to look at organizations the participant organization supports, with a subsequent probe to address funding specifically. Ms. Abrami pointed out that there may be wide variability with respect to mission and goals, particularly regarding incentive programs. In response to Ms. King's question of suggestions for reassessing where the probes should be placed and how they should be timed, Ms. Thompson concluded that the instruments are comprehensive and will ultimately achieve the goal,

 

Ms. Curtin pointed out that in question 2, regarding an organization's history with SNAP programs, her reading of the question would likely yield a response driven by numbers and data, how SNAP clients relate to rest of the market population, which might not be the intended goal.

 

There was a discussion about how to best frame instrument questions to capture data related to issues of potential misuse within incentive programs. Dr. Williams described a large public concern with fraud in general, specifically with retail level trafficking the goal of the interview questions was to steer away from a suggestion of enforcement, and more address the current situation within farmers' markets, at the participant level of incentive programs, for example the counterfeiting of vouchers, misrepresentation of beneficiaries, etc. Ms. Miller raised a point of concern that the use of the word fraud implies conscious and significant wrong-doing, but no one would think what happens at a farmers’ market is “fraud,” though specific activities such as trading tokens or redeeming for ineligible foods might occur. Examples would elicit a more accurate response, with the notion that it’s not fraud so much as potential misuse. Positive framing should be emphasized, as well as respect for the participating organization.

 

Ms Miller offered the example question “Does your org have any concerns about the improper use or counterfeit production of incentive tokens?” Ms. Curtin suggested “What measures does your organization have to prevent possible misuse, and why are those measures in place?” Ms. Abrami: “What measures would you suggest to put in place to prevent potential misuse?” Mr. Byron pointed out that the question wording, “particularly vulnerable,” implies bias, whereas most instances of “fraud” in farmers' markets are a result of being unaware of misuse and what constitutes misuse. Dr. Schulman suggested editing “are” to “may be.” There was general agreement that a statement offered by Mr. Byron, mentioning the critical importance of maintaining the integrity of the organization, with the understanding that mistakes might occur, would be a good idea to include. Mr. Byron also suggested a comprehensive list in the question probes would be useful, based on his experience working with farmers’ market. Ms. Thompson thought appropriate probes rather than comprehensive list might work better, as most program directors will be familiar enough with potential misuse. It was decided as an action item that the fraud question would be modified and redistributed to the group for feedback, and the group was thanked for its comments.

 

There was consensus that the question related to challenges faced by an organization did not need additional probes. Suggestions for the self-evaluation question included using “measure” rather than know, “outcomes,” “progress,” or “impact” rather than “success.” It was noted that an additional probe will likely be added including information about what additional data will be provided to a participating organization that offers self-evaluation data, one this additional data is finalized. The social web analysis questions were discussed, including the overlaid social webs between Tasks 1 and 3. Feedback was very positive from the TWG members. The extent that identifying information would be incorporated into the final report and released is currently unclear due to privacy concerns that are currently being addressed. Dr. Kinnison pointed out a lot of that information depends on consent from organizations, and knowing what information would be of interest can be considered while developing the consent progress.

 

Regarding expectations of what organizations would be willing to disclose, including the details of their respective social networks, Ms. Thompson suggested assuming it will be collaborative and beneficial to everyone, while expecting politics to come into play later. This was generally agreed upon. It was noted that being identified and included in a social-web analysis might lead to more funding rather than less, due to proof of support and collaboration. Ms. King pointed out that this issue could be explored in the pre-testing phase. Ms. Curtin raised the question of distributing interview materials to participating organizations in advance of the scheduled interview, and there was consensus among the TWG members that this would be preferred. Ms. King will consult with Dr. Butler about the effect this process step would have on the methodology. The action items were reiterated, and all members were thanked for their comments and participation.

ACTION ITEMS:

• Each TWG member will send an email to Melissa during the week of 3/19 listing additional useful information related to data collected in Task 2 that might be offered to participating organizations who agree to provide self-evaluation data for Task 3.

• The revised instrument question relating to fraud/potential misuse within incentive programs will be circulated to TWG members (including Mark Byron) for feedback and possible additional revisions.