Appendix C. FINI Independent Evaluation SOO

Appendix C
ATTACHMENT I
BPA STATEMENT OF OBJECTIVES
TITLE: EVALUATION OF FOOD INSECURITY NUTRITION INCENTIVES
SECTION C:

STATEMENT OF OBJECTIVES

C.1

GENERAL OVERVIEW.................................................................................................... 2

C.2

BACKGROUND ................................................................................................................ 2
C.2.1 Policy Background ................................................................................................. 2
C.2.2 Research Background ............................................................................................ 5
C.2.3 References.............................................................................................................. 6

C.3

STATEMENT OF OBJECTIVES ...................................................................................... 7
C.3.1
C.3.2

C.3.3

Evaluation Overview ............................................................................................. 7
Key Research Objectives and Questions ............................................................... 8
C.3.2.1
Process Evaluation ............................................................................. 8
C.3.2.2
Outcome Evaluation ........................................................................... 9
C.3.2.3
Comparative Analysis ........................................................................ 9
Evaluation Design Parameters and Constraints ..................................................... 9

1

C.1

GENERAL OVERVIEW

The Agriculture Act of 2014 (P.L. 113-79) authorizes USDA to provide grants to eligible organizations to
design and implement projects to increase the purchase of fruits and vegetables among low-income
consumers participating in the Supplemental Nutrition Assistance Program (SNAP) by providing
incentives at the point of purchase. It also directs USDA to undertake an independent evaluation of each
project using rigorous methodologies capable of producing scientifically valid information regarding their
effectiveness in increasing fruit and vegetable purchases and improving the nutrition and health status of
participating households. Congress provided $100 million over 5 years to fund grants for project
operations and to support the costs of USDA’s administration, monitoring, and evaluation of the grant
program. USDA’s National Institute for Food and Agriculture (NIFA) has invited interested
organizations to submit applications for the fiscal years 2014-15 Food Security Nutrition Incentive (FINI)
Grant Program. Subsequent grant cycles will be funded in fiscal years 2016, 2017, and 2018.
This solicitation and statement of objectives requests qoutations to design, implement, analyze, and report
on the results of the independent evaluation of the FINI Grant Program to measure short-term changes in
fruit and vegetable purchases and consumption and basic measures of nutrition and health status among
SNAP participants, with an initial focus on grants awarded in 2015 and 2016. 1
C.2

BACKGROUND

C.2.1

Policy Background

The Supplemental Nutrition Assistance Program (SNAP) is the foundation of America’s national
nutrition safety net, the nation’s first line of defense against hunger, and a powerful tool to improve
nutrition among those with little income and few assets. In fiscal year 2013, SNAP served 47.6 million
participants in an average month, providing a benefit to purchase food at authorized retailers and markets
across the nation. While a growing body of evidence documents SNAP’s effectiveness in reducing food
insecurity, many low-income Americans still report difficulty putting enough food on the table. At the
same time, many struggle with the consequences of overweight and obesity.
In recent decades, the prevalence of overweight and obesity in the United States has been a subject of
growing concern. In 2009-10, more than one-third of American adults and nearly one in five children
were obese (Ogden, Carroll, Kit, & Flegal, 2014). While the prevalence of overweight and obesity has
stabilized in recent years, many Americans remain at greater risk for adverse health outcomes such as
type 2 diabetes and coronary heart disease and overall morbidity and mortality as a consequence of their
weight (National Heart, Lung, and Blood Institute Obesity Education Initiative, 1998). Although
consuming adequate amounts of fruits and vegetables is widely seen as an essential part of reducing poor
health outcomes, in general Americans eat fewer fruits and vegetables and more fats, added sugars, and
calories than recommended (Krebs-Smith, Guenther, Subar, Kirkpatrick, & Dodd, 2010).
High prices and limited access have been identified by some as barriers to consumption of healthful foods
by low-income and disadvantaged Americans. For example, Drewnowski and Specter (2004) found that
prices are lower for energy-dense, high-calorie foods than for fresh, healthful foods. Ver Ploeg et al.
(2009) described “food deserts,” low-income neighborhoods across the United States in which residents
cannot readily access supermarkets.
Many observers have identified targeted incentive, coupon, or voucher programs as promising approaches
to encourage expenditures on fruits and vegetables (Guthrie et al. 2007; GAO 2008). Financial incentive
1

Future task orders may be issued under this solicitation to evaluate grants awarded in 2017 and 2018.

2

programs are designed to make fruits and vegetables more affordable relative to alternative, less nutritious
choices by affecting relative prices and enhancing individuals’ overall purchasing power. Beyond these
price effects, information effects may include increased awareness about healthful eating and implicit
messages about the importance of healthful foods. Together these mechanisms are expected to influence
behavior by increasing consumption of desired foods and thereby overall dietary quality, ultimately
improving weight status and associated health outcomes [Exhibit 1].
Exhibit 1:

Theoretical Relationships between Nutrition Promotion Approaches and Outcomes
and Impacts

*Reproduced from: United States Government Accountability Office (“Food Stamp Program: Options for Delivering Financial Incentives
to Participants for Purchasing Targeted Foods.” Report to the Chairman, Committee on Agriculture, Nutrition, and Forestry, U.S. Senate;
July 2008).

A variety of organizations have acted in recent years to use financial incentives as a strategy to affect diet
quality among SNAP participants and within low-income neighborhoods. The most common use addedvalue or bonus coupon programs; examples include New York City’s Health Bucks, Wholesome Wave’s
Double Value Coupon Program, Fair Food Network’s Double Up Food Bucks, and Roots of Change’s
California Market Match. The idea behind these initiatives is that adding bonuses to SNAP benefits may
relieve some of the economic pressure that can lead participants to make unhealthy food choices with
limited purchasing resources. Moreover, by restricting the use of the bonus funds to the purchase of
foods at farmers’ markets, these programs assist local farm economies by increasing the demand for fresh
produce. 2 In 2012, four programs – Wholesome Wave, Fair Food Network, Market Umbrella, and Roots
of Change – sponsored or managed 518 farmers’ markets incentive programs that served more than
131,000 SNAP customers and engaged over 4,800 farmers/vendors in 24 states and the District of
Columbia (Community Science, 2013). King et al. (2014) provide an extensive summary of the
characteristics, roles, and relationships of organizations involved in such incentive programs.
2

Fair Food Network’s experience at farmers markets was brought to grocery stores in 2013 through the Double Up
Grocery Store Project in Detroit.

3

The Massachusetts Department of Transitional Assistance, with funding from USDA, recently tested a
different approach in Hampden, MA. Under the Healthy Incentives Pilot, SNAP participants received on
their SNAP Electronic Benefit Transfer (EBT) card an incentive of 30 cents for every dollar they spent on
targeted fruits and vegetables. 3 The incentive was capped at $60 per household per month, a level
sufficiently high that few households reached it.
With an interest in building on and expanding these efforts, the Agriculture Act of 2014 provided USDA
$100 million over 5 years to establish and evaluate the effectiveness of nutrition incentives for fruit and
vegetable purchases among low-income consumers. The primary goal of the FINI Grant Program is to
provide nutrition incentives at the point of purchase to SNAP participants and evaluate their impact on
purchases, consumption, nutrition, and health outcomes. USDA’s National Institute for Food and
Agriculture (NIFA) has invited interested organizations to submit applications for the first grant cycle
(fiscal years 2014 and 2015). 4 USDA will give priority to projects proposed by eligible organizations
that:
•
•
•
•
•
•
•

maximize the share of funds used for direct incentives to participants;
test innovative or promising strategies that would contribute to understanding how best to increase the
purchase of fruits and vegetables by SNAP participants to inform future efforts;
develop innovative or improved benefit redemption systems that could be replicated or scaled;
use direct-to-consumer sales marketing;
demonstrate a track record of designing and implementing successful nutrition incentive programs
that connect low-income consumers and agricultural producers;
provide locally or regionally produced fruits and vegetables, especially culturally appropriate fruits
and vegetables for the target audience; and
are located in underserved communities, particularly Promise Zones and StrikeForce communities). 5

FINI projects are intended to bring together stakeholders from the distinct parts of the food system and to
foster understanding of how they might improve the nutrition and health status of participating
households receiving incentives to purchase fruits and vegetables. FINI projects are also intended to
address the development of effective and efficient technologies for benefit redemption that are replicable
by others.
Applications are invited in each of three categories: (1) FINI Pilot Projects aimed at new entrants
seeking funding for small-scale (less than $100,000 over no more than one year) community-based
projects in the early stages of incentive program development; (2) multi-year, FINI Projects to support
local and state nutrition incentive programs on a moderate scale (less than $500,000 over no more than
four years); and (3) multi-year FINI Large-Scale Projects to support multi-county, state and regional
3

Targeted fruits and vegetables included fresh, canned, frozen, and dried fruits and vegetables without added
sugars, fats, oils or salt, but excluded white potatoes and 100% fruit juice (the same set of fruits and vegetables
eligible for the WIC Fruit and Vegetable Cash Value Voucher).
4

See Food Insecurity Nutrition Incentive (FINI) Grant Program: 2015 Request for Applications issued September
29, 2014 for information on the request for applications for nutrition incentive grants (available on-line
http://www.nifa.usda.gov/funding/cfp/fini.html and at www.grants.gov).
5

Eligible organizations are limited to government agencies and nonprofit organizations, including emergency
feeding organizations; agricultural cooperatives; producer networks or associations; community health
organizations; public benefit corporations; economic development corporations; farmers’ markets; communitysupported agriculture programs; buying clubs; SNAP-authorized retailers; and State, local, or tribal agencies.

4

programs (more than $500,000 over no more than four years). All grantees will be expected to conduct a
self- assessment (the scale and scope of which depends on project type), and multiyear FINI Projects and
Large-Scale Projects are required to cooperate with and contribute to an independent evaluation to
determine the effectiveness of nutrition incentive programs in achieving the legislative goals of increasing
fruit and vegetable purchases and improving the nutrition and health status of participating households.
NIFA anticipates that up to $31.5 million will be distributed as grants in fiscal years 2014 and 2015.6
C.2.2

Research Background

There is a large literature on interventions and policies to increase fruit and vegetable intake. Some
interventions focused on nutrition education or awareness campaigns designed to influence attitudes
toward fruits and vegetables. Other research has explored the effect of prices on fruit and vegetable
spending, and a small number of interventions included a financial incentive component.
There is a growing body of evidence that suggests participation in healthy food incentive programs can
have a positive effect on fruit and vegetable purchases and consumption. The largest and most rigorous
of these studies – USDA’s evaluation of the Healthy Incentives Pilot – used a randomized control trial to
assess the causal impact of a financial incentive on fruit and vegetable consumption by SNAP
participants, and on other key measures of dietary intake. It found that HIP participants consumed almost
a quarter of a cup-equivalent more fruits and vegetables per day than did non-participants, a difference of
26 percent (Bartlett et al., 2014).
Herman et al. (2008) randomly assigned WIC mothers to one of two intervention groups (vouchers for
grocery stores or vouchers for farmers markets) or a control group as part of a program offering fruit and
vegetable vouchers. Fruit and vegetable consumption increased substantially and significantly in both
voucher groups but not in the control group, with changes sustained 6 months after the voucher program
ceased. Average increases in fruits and vegetables were 1.4 servings per participant for the farmers’
market voucher group and 0.8 servings for the supermarket voucher group, where one serving is
equivalent to approximately ½ cup-equivalents of fruit or vegetables.
Another body of research suggests that financial incentives at farmers markets may be an effective
pathway to increased fruit and vegetable purchases. Wholesome Wave (undated) indicates, for example,
that 90 percent of Double Value Coupon Program customers reported that they increased or greatly
increased their consumption of fruits and vegetables and 66 percent of participating farmers reported
increased sales. Similarly, Fair Food Network (2013) indicates that 78 percent of customers reported that
they increased the amount of fruits and vegetables they buy and 83 percent of farmers said they make
more money at the farmers’ markets because of the Double Up Food Bucks program. A cluster
evaluation sponsored by Wholesome Wave, Fair Food Network, Market Umbrella, and Roots of Change
found that more than three-fourths of SNAP recipients reported that they increased their purchase of
produce because of the incentives and that SNAP incentives were a strong factor in their decision to shop
at a farmers’ market. The majority of vendors (at least 64 percent) reported that they sell more produce,
make more money, and have more customers because of the SNAP incentives (Community Science,
2013). In addition, financial incentive programs can be an important draw for low-income shoppers:
Karakus et al. (2014) report that SNAP participants who are aware of financial incentive programs are 40
times more likely to shop at farmers markets.

6

The Act provides a total of $35 million in fiscal years 2014-15, $20 million in 2016, $20 million in 2017, and $25
million in 2018. No more than 10 percent of the total provided each year may be used to pay costs associated with
administering, monitoring, and evaluating each project. USDA has reserved 4 percent of the funds available for
administration and monitoring of the FINI Grant Program.

5

The overarching goal of the independent evaluation of the Food Insecurity Nutrition Incentive Grant
Program is to expand this evidence base by subjecting a broader range of incentive programs in a wider
range of community settings to a more rigorous evaluation to provide policy makers in USDA and
Congress with credible information on which interventions work best.
C.2.3

References

Barley, Z. A., & Jenness, M. (1993). Cluster evaluation: A method to strengthen evaluation in smaller
programs with similar purposes. Evaluation Practice, 14(2), 141-147.
Bartlett, S., Klerman, J., Wilde, P., Ohlso, L., Logan, C., Blocklin, M., Beauregard, M., and Enver, A.
(2014). Evaluation of the Healthy Incentives Pilot (HIP) Final Report. Alexandria, VA: Food and
Nutrition Service, USDA.
Community Science. (2013). SNAP Healthy Food Incentives Cluster Evaluation 2013 Final Report.
Gaithersburg, MD: Community Science.
Drewnowski, A., & Specter, S. E. (2004). Poverty and Obesity: The Role of Energy Density and Energy
Costs. The American Journal of Clinical Nutrition, 79(1), 6-16.
Fair Food Network. (2013). Double Up Food Bucks 2012 Evaluation Report. Ann Arbor, MI: Fair Food
Network.
Government Accountability Office. (2008). Food Stamp Program: Options for Delivering Financial
Incentives to Participants for Purchasing Targeted Food. Washington, DC: U.S. General
Accountability Office.
Guthrie, J., Andrews, M., Frazao, E., Leibtag, E., Lin, B.-H., Mancino, L., et al. (2007). Can Food Stamps
Do More To Improve Food Choices? An Economic Perspective. Washington, DC: Economic
Research Service, USDA.
Herman, D. R., Harrison, G. G., Afifi, A. A., & Jenks, E. (2008). Effect of a Targeted Subsidy on Intake
of Fruits and Vegetables among Low-income Women in the Supplemental Nutrition Program for
Women, Infants, and Children. American journal of Public Health, 98(1), 98-105.
Karakus, M., MacAllum, K., Milfort, R., and Hao, H. (2014). Nutrition Assistance in Farmers Markets:
Understanding the Shopping Patterns of SNAP Participants. Alexandria, VA: Food and
Nutrition Service, USDA.
King, M., Dixit-Joshi, S., MacAllum, K., Steketee, M., & Leard, S. (2014). Farmers Market Incentive
Provider Study. Alexandria, VA: Food and Nutrition Service, USDA.
Krebs-Smith, S. M., Guenther, P. M., Subar, A. F., Kirkpatrick, S. I., & Dodd, K. W. (2010). Americans
Do Not Meet Federal Dietary Recommendations. The Journal of Nutrition, 140(10), 1832-1838.
National Heart, Lung, and Blood Institute Obesity Education Initiative. (1998). Clinical Guidelines on the
identification, Evaluation, and Treatment of Overweight and Obesity in Adults. National Institute
of Health.
Ogden, C. L., Carroll, M. D., Kit, B. K., & Flegal, K. M. (2014). Prevalence of Childhood and Adult
OLbesity in the United States, 2011-2012. Journal of the American Medical Association, 311(8),
806-814.
Ver Ploeg, M., Breneman, V., Farrigan, T., Hamrick, K., Hopkins, D., Kaufman, P., et al. (2009). Access
to Affordable and Nutritious Food—Measuring and Understanding Food Deserts and Their
Consequences: Report to Congress. Washington, DC: Economic Research Service, USDA.

6

W.K. Kellogg Foundation. (2004). W.K. Kellogg Foundation Evaluation Handook. Retrieved July 17,
2014, from www.wkkf.org: http://www.wkkf.org/resource-directory/resource/2010/w-k-kelloggfoundation-evaluation-handbook
Wholesome Wave. (undated). Increasing Food Access and Local Farm Business Nationwide: 2009-2012
Outcomes and Trends Full Report. Wholesome Wave: Bridgeport, CT.
C.3

STATEMENT OF OBJECTIVES

C.3.1

Evaluation Overview

The authorizing statute requires an independent evaluation using rigorous methodologies capable of
producing scientifically valid information regarding the effectiveness of each project in terms of
increasing fruit and vegetable purchases and improving the nutrition and health status of participating
SNAP households. The scope of this study requires a vendor to design, implement, analyze, and report
on the results of the independent evaluation of the FINI Grant Program to measure short-term changes in
fruit and vegetable purchases and consumption and basic measures of nutrition and health status among
SNAP participants.
USDA believes that a non-equivalent comparison group design offers the best balance within the limited
resources provided for evaluation between the statutory requirements for scientific rigor and the limited
administrative and evaluation capacity of some potential grantees. Respondents to this solicitation,
however, are invited to propose alternative designs that are consistent with the Congressional interest in
rigor, the statement of objectives set forth here, and the funding available. The independent evaluation
will rely on a mix of operational and transaction summaries provided by grant sites and surveys or
interviews of consumers, vendors, program administrators, and other stakeholders. The evaluation will
have four main components:
•

a process evaluation to identify barriers and facilitators to implementation of nutrition incentive
programs in order to improve on existing program models inform others wishing to implement similar
interventions,

•

an outcome evaluation to assess the effectiveness of nutrition incentives in increasing fruit and
vegetable purchases and consumption among SNAP participants,

•

a comparative analysis to integrate results across sites and approaches, attempting to answer the
question of what works best in which contexts, and

•

technical assistance to support consistent implementation of evaluation protocols.

All grantees will be required to undertake their own process assessments, and FINI Large-Scale Projects
will be required to undertake their own outcome evaluation. USDA expects the independent evaluation
Vendor to integrate and synthesize the process assessments from each project and supplement the LargeScale Projects’ outcome assessments with an independent outcome and comparative analysis.
The initial focus of the independent evaluation is on the larger FINI Projects and FINI Large-Scale
Projects awarded grants in fiscal years 2015 and 2016, with future task orders to evaluate grants awarded

7

in 2017 and 2018. 7 FINI Pilots – targeted on new entrants to the field – are exempt from the independent
evaluation because these projects are in the earliest stages of development that do not offer a fair test of
their potential effectiveness. The independent Vendor, however, will integrate implementation lessons
revealed in the FINI Pilot Projects self-assessments into the process evaluation.
C.3.2

Key Research Objectives and Questions

The fundamental goal of the independent evaluation of the Food Insecurity Nutrition Incentive Grant
Program is to determine whether nutrition incentives at the point of purchase contribute to increased fruit
and vegetable purchases and consumption among SNAP participants and whether different approaches
are more or less effective than others.
C.3.2.1

Process Evaluation

Research Objective 1: Document each nutrition incentive program by describing its design and
operations to identify (1) barriers and facilitators to implementation and (2) specific lessons learned to
support replication of successful programs.
Research Questions:
• What is the demographic, social, economic, and nutrition context in which each program operates?
• How is each incentive program implemented?
 What form does the incentive take, and how is it distributed to target recipients?
 How are potential vendors recruited and reimbursed?
 How is the program marketed to SNAP participants?
 Is the incentive linked to nutrition education or other wellness activities?
• What challenges are encountered during implementation and how are they resolved?
• Are oversight and monitoring processes adequate to inform program administrators and key
stakeholders?
• What are the implementation and operational costs of the nutrition incentive program?
C.3.2.2

Outcome Evaluation

Research Objective 2: Assess the effectiveness of each Multiyear FINI Project and Multiyear FINI
Large-Scale Project in increasing fruit and vegetable purchases and consumption among SNAP
participants.
Research Questions:
• To what extent do SNAP participants take up the offer of nutrition incentives? To what extent are the
benefits distributed to participants redeemed by farmers or vendors? What factors influence take-up
and redemption rates?
• Does participation in the nutrition incentive program alter:
 Knowledge and attitude about fruits and vegetables?
 Frequency, amount, and type of fruit and vegetable purchases?
 Frequency, amount, and type of fruit and vegetables consumed?
 Perceived health status or well-being?
 Participant shopping patterns?
7

USDA expects some adjustment to the scale, scope and form of the evaluation of grants awarded in fiscal years
2017 and 2018 based on the lessons learned and experience gained from the initial evaluation experience and on
policy interests and research needs at the time future task orders are issued.

8

•
•

How well do participants understand the incentive? What is their opinion of and satisfaction with the
incentive?
Does the nutrition incentive program alter the amount or variety of fruits and vegetables offered in the
community?

C.3.2.3

Comparative Analysis

Research Objective 3: Compare the relative outcomes of different forms of incentives to help determine
the most effective and efficient strategies for using incentives to increase purchase and consumption of
fruits and vegetables among SNAP participants.
Research Questions:
• Are there combinations of program features that have a significant bearing on project outcomes?
• Are there particular roles for non-governmental organizations, states, and the Federal government that
have a significant bearing on project outcomes?
C.3.3

Evaluation Design Parameters and Constraints

Vendors shall include in their quotations a clear study plan reflecting the research objectives and
questions provided above and the design parameters and constraints below. It shall include a detailed
study methodology, sampling procedures, data elements and definitions, data collection and processing
procedures, draft data collection instruments, analytic methods to be used to address study objectives,
plans to integrate results from the process, outcome, and comparative components of the evaluation, and
plans for disseminating the results.
There are a number of unique challenges and expectations that will shape this evaluation:
•

The scale and scope of the evaluation will be shaped by several funding constraints. The Agriculture
Act of 2014 limits the amount of funding available for evaluation and makes it available over the
course of 5 years. The Act provides $35 million in fiscal years 2014-15, $20 million in 2016, $20
million in 2017, and $25 million in 2018. No more than 10 percent of the total provided each year
may be used to pay costs associated with administering, monitoring, and evaluating each project.8
Thus, funding for the evaluation is spread across multiple years and capped at varying levels.
Respondents to this solicitation should structure quotations that are consistent with both the amount
and timing of the funds available. USDA’s goal is to produce the strongest, most rigorous evaluation
possible within the resources provided by Congress.

•

The number of projects awarded under the FINI Grant Program will depend on the level of
community interest, the number of high-quality grant proposals, and the amount of funding available
for distribution across the three grant categories. It is expected that projects will span the country, but
the number subject to evaluation in the first funding cycle will not be known until after awards are
made in or near March 2015. USDA expects to award a second round of grants in fiscal year 2016. 9
For planning purposes, bidders should assume at least 10 FINI Pilots (only technical support
required), 20 FINI Projects, and 5 FINI Large-Scale Projects in each of these two award cycles and

8

USDA has reserved 4 percent of the funds made available to cover the costs of administering and monitoring the
FINI Grant Program.

9

USDA will make additional grant awards when fiscal year 2017 and 2018 funding becomes available.

9

provide contingency plans if the number of grant awards differs substantially from the planning
targets.
•

Many of the selected projects are likely to have limited experience with the design and execution of
rigorous evaluations. While the grant agreements will require cooperation with the independent
evaluation, the independent evaluation Vendor must define reasonable and appropriate expectations
that balance the need to minimize burden on local projects with the need to capture information
critical to the evaluation. In its quotation, Vendors should describe planned technical assistance on
evaluation and data collection issues to individual projects as requested and as needed to ensure
consistent implementation of evaluation protocols, timely reporting of the minimum core data set,
implementation of informed consent, and outreach or marketing to achieve adequate samples. It
should also discuss plans to engage interested stakeholders, challenges that may arise, and how they
will be addressed.

•

FINI projects are likely to vary substantially in scale and maturity, and grantees are likely to vary
considerably in evaluation capacity and expertise. As a consequence, evaluation requirements differ
for each of the three FINI grant categories. All grantees regardless of size will be expected to conduct
a self-assessment of the process of implementing the nutrition incentive projects. FINI Large-Scale
Projects will also be expected to conduct a rigorous outcome-based self-assessment following
guidelines developed by the independent evaluation Vendor in collaboration with grantees to ensure
an appropriate level of comparability of methods, outcomes, and measures. The results of all these
self-assessments will be reported to the independent evaluation Vendor. All FINI Projects and LargeScale Projects are required to cooperate with and contribute to the independent evaluation to
determine the relative effectiveness of the incentive program in achieving the legislative goals of
increasing fruit and vegetable purchases and improving the nutrition and health status of participating
SNAP households. 10 FINI Pilot Projects are exempt from the independent evaluation.

•

FINI Projects and FINI Large-Scale Projects will be required to periodically provide the independent
evaluation Vendor a minimum core data set to ensure common program tracking and enable
meaningful comparisons across all projects. 11 FINI Projects and FINI Large-Scale Projects may
choose at their option to collect additional information beyond the minimum as part of their selfassessments. The minimum core data set includes the following site, project, incentive program
information:




Management or organizational structure of the program
Financial instrument used for SNAP and incentive purchases (tokens, scrip, electronic)
Retail locations, mobile routes, or other pertinent information to understand how the project
improves access to healthy food for underserved, low-income consumers
 Months of operation and operating days and hours
10

Cooperation entails supporting implementation of evaluation requirements (including but not limited to helping
the independent evaluation contractor identify appropriate comparison sites or groups and providing the minimum
core data set described below); meeting periodically with staff from USDA, the independent evaluation contractor,
and other FINI grantees to review project plans, evaluation objectives and methods, data collection and reporting
requirements, and analysis and reporting of results; facilitating access to or providing documentation of project
implementation, operations, costs, and outcomes; and facilitating site visits and interviews with project staff,
partners and program participants.
11

The frequency of data reporting may depend on individual project site capacity. For planning purposes, bidders
should assume that point-of-purchase transaction data will be aggregated by month and reported quarterly. Other
data descriptive of program organization and operations may be reported annually.

10















•

Whether it is a new SNAP incentive program or the continuation, expansion, or modification of
an existing program
Whether program sites accept other nutrition assistance program benefits
Whether program sites collaborate with nutrition education programs or offer other experiential
nutrition education activities
Expenses associated with establishment and operations of the program
Fruit and vegetable products eligible for incentives
Incentive level (ratio and maximum)
Incentive delivery mechanism
Number of SNAP participants (per site/per year)
Dollar value of SNAP purchases (per site/per year)
Number of SNAP transactions (per site/per year)
Dollar value of incentives issued (per site/per year)
Dollar value of incentives redeemed (per site/per year)
Average incentive value redeemed per recipient (weekly/monthly/annually)

The independent Vendor is expected to supplement the minimum core data set with additional data
needed to fully meet the research objectives, including, but not limited to, consumer-based
information on: 12












Gender; age; ethnicity; household size and composition;
Location of residence (e.g. ZIP code);
Whether participant has used SNAP at similar program sites before;
How they learned of the nutrition incentive program;
Whether their purchases of fruits and vegetables increased because of the incentive program,
and if so by how much (e.g., percentage, dollar value);
Whether the type and variety of fruits and vegetables purchased changed, and if so by how
much;
Whether their consumption of fruits and vegetables increased because of the incentive, and if
so, by how much;
Whether the type and variety of fruits and vegetables consumed changed, and if so by how
much;
Motivations for purchasing fruits and vegetables at that location, including price, shopping
experience, freshness or quality of food, selection of foods available, convenience,
participation in nutrition education or other activities at the site; and
Use of any other nutrition assistance program benefits at the site.

To strengthen the outcome measures of fruit and vegetable consumption, the independent Vendor is
expected to make use of a relatively brief food frequency questionnaire (FFQ) or comparable
alternative in surveys of SNAP consumers. 13 The food frequency instrument would be uniform
12

USDA has identified at least four options to define the consumer respondent population: (1) all residents of
neighborhoods in which incentive programs operate, providing a potential measure of community-wide outcomes;
(2) all SNAP participants in neighborhoods in which incentive programs operate, providing an indication of the
incentive’s penetration into target population, and thus overall outcomes; (3) SNAP participants offered an incentive
(at venues that distribute incentives), providing some insight into the reasons that some may choose not to redeem
available incentives and a reasonable measure of outcomes among those who use the incentive; and (4) SNAP
participants who redeem incentives (at venues that accept incentives), providing the most efficient measure of
outcomes among those who use the incentive. Bidders are expected to recommend and justify a target population
based on the overall research objectives and feasibility of implementation.

11

across all sites, fielded by the independent evaluation Vendor, and analyzed according to standard
protocols.
•

The Vendor is also expected to gather data from incentive program administrators, retail and market
operators (including store managers, farmers’ market managers, and farmers/vendors), and local
community organizations to document the evolution of the program and early lessons learned, track
program implementation, and assess adherence to its intended model.

•

Although the FINI Grant Program has long-term objectives related to permanent changes in overall
dietary quality and, ultimately, reductions in obesity status and improvements in overall health,
USDA has determined that it is not feasible to assess the effectiveness of the program in meeting
these goals in depth given the relative newness of the initiative, the short timeframe for the
evaluation, and the available resources. Moreover, past research has demonstrated a link between
increased fruit and vegetable consumption and longer-term health outcomes; if the evaluation
concludes that the FINI Grant Program increases fruit and vegetable consumption, one may infer that
the intervention holds promise for achieving these ultimate goals. The evaluation Vendor will,
however, identify and develop basic measures of nutrition and health status that may be reasonably
captured in a brief consumer survey. 14

•

To conserve limited evaluation resources, the independent Vendor shall use existing data collection
instruments to the maximum extent feasible. If new instruments are deemed necessary, the Vendor
shall note that any pre-test with more than nine (9) respondents may require approval under terms of
the Paperwork Reduction Act, described below. Information collection instruments intended for use
among program participants must be available in both English and Spanish. Vendors should address
how their plans for Spanish language audiences and other non-English speakers as needed.

•

USDA believes that the quasi-experimental comparison design would be strengthened if resources
permit a difference-in-difference analysis with at least one pre- and one post-intervention data
collection. The timing and duration of any pre/post data collections is affected by at least three
factors. First, the start of project operations is likely to vary across sites. Second, some projects
(those that operate in farmers’ markets, for example) are likely to be seasonal and of limited duration,
while others (those in retail stores) are likely to operate throughout the year. Third, it is expected that
any effects of the nutrition incentive on purchases will emerge relatively quickly. Although this
solicitation does not require a pre/post design, bidders are encouraged to consider and propose a
schedule for the timing, duration, and frequency of pre- and post-intervention data collections (or
alternative design that enhances the overall evaluation) that is consistent with overall research
objectives, maximizes efficiency and reduces cost, and accommodates the variation in project
approaches.

•

Past research suggests that the impact of nutrition incentives comparable to those offered to date on
SNAP recipients’ fruit and vegetable purchases is likely to be important but modest. The Vendor

13

There are several validated instruments such as (among others) those developed by the National Cancer Institute
for the National Health and Nutrition Examination Survey
(http://appliedresearch.cancer.gov/archive/usualintakes/FFQ.English.June0304.pdf) and by the Centers for Disease
Control and Prevention for the Behavioral Risk Factor Surveillance System
(http://www.cdc.gov/brfss/questionnaires/pdf-ques/2013%20BRFSS_English.pdf).
14

For examples of potential candidate items, refer to the Behavioral Risk Factor Surveillance System Questionnaire
available http://www.cdc.gov/brfss/questionnaires/pdf-ques/2013%20BRFSS_English.pdf.

12

shall discuss considerations (including resource implications) for the sample sizes necessary overall
and within each project area to detect meaningful differences in fruit and vegetable purchases. USDA
is particularly concerned that the limited evaluation funding provided may support an overall sample
that is large enough to detect moderate differences in fruit and vegetable purchases for the overall mix
of projects considered in 2015 and 2016, but a sample in any given site that is too small to detect any
but the largest changes. This will seriously impair USDA’s ability to address the Congressional
interest in knowing which approaches work best. Bidders are strongly encouraged to propose
alternative strategies that maximize analytic power within the limited evaluation resources available.
•

The Vendor must prepare the required publication notices and clearance package for submittal to the
Office of Management and Budget (OMB) to obtain approval for all data collection activities in all
components of the evaluation subject to the requirements of the Paperwork Reduction Act. The
clearance package must provide an explicit, concise description of the direct links between the study
objectives, research questions, variables, instrument items, data analysis plans and desired products.
The PRA data collection package shall contain copies of all final data collection instruments and a
supporting statement as set forth in the revised Standard Form No. 83a, “Instructions for Requesting
OMB Approval under the Federal Reports Act, as Amended.

•

Vendors are advised that obtaining approval for data collections under the Paperwork Reduction Act
can be a lengthy process. The Vendor should be aware that the OMB data collection package will be
reviewed by multiple groups within USDA. Revisions to the package may be required after each
level of review and will not be complete until it receives official OMB clearance. Past experience has
shown that it can take six to nine months from the first submission of a 60-day Federal Register
notice to final approval of the information collection. Vendors are advised that clearance is often the
primary rate limiting factor in many studies. Preparation of a high-quality initial draft of instruments
and justification statements, followed by timely responses to required revisions, will greatly facilitate
the approval process. For additional information — including detailed guidance, a checklist for final
collection request submissions, and estimated timelines, see the following link
(www.ocio.usda.gov/policy-directives-records-forms/information-collection). Vendors are advised
that clearance package formats that have worked at other Federal agencies may be rejected by USDA.

•

Full conformance with the OMB standards and guidelines for surveys is required. Vendors shall note
that an 80 percent survey response rate is required; if response rates are less, a nonresponse bias
analysis is mandatory. Vendors shall explain and justify their proposed procedures for ensuring that
high response rates are obtained and demonstrate their understanding and acceptance of OMB
standards and guidelines for calculating response rates and actions that are required if response rates
are under 80 percent.

•

The Vendor will be expected to work with USDA and FINI Grant Program sites to develop
procedures and materials to ensure legally effective and prospectively obtained informed consent
from participants in the evaluation sample. The informed consent process will (1) disclose to
potential research subjects information needed to make an informed decision; (2) facilitate the
understanding of what has been disclosed; and (3) promote the voluntariness of the decision about
whether or not to participate in the research.

•

USDA is interested in research of high scientific rigor communicated in a policy relevant fashion.
The Vendor shall propose an approach to document, communicate, and present the results of the
evaluation and supporting analyses to USDA, the broader research and policy field, and the programs
and sites that participate in FINI Grant Program. USDA also has a strong interest in obtaining results
as soon as possible from the initial round of 2015 grant awards and then expanding the evidence base
as the initial projects mature and as subsequent 2016 awards are made. To this end, the Vendor shall
13

prepare and submit preliminary, interim and final reports that consolidate and integrate all
information available from project self-assessments and the independent evaluation after
approximately 18 months, 36 months, and 54 months, respectively, of operation of FINI Grant
Program projects awarded in the initial funding cycle. USDA expects that the preliminary report will
focus largely on early results from the 2015 projects, the interim report will broaden the focus to
include follow-on results from the 2015 projects and early results from the 2016 projects, and the
final report will present follow-on results from the 2016 projects and summarize the conclusions
drawn from both award cycles. The final report shall be subject to peer review requirements for
influential scientific information.
•

The Vendor shall furnish all necessary labor, materials, services, facilities, and otherwise do all things
necessary to execute the scope of work for this project. Within its technical quotation, bidders should
present a set of tasks necessary to achieve the objectives of the evaluation. Within this task structure,
the technical quotation should discuss:













•

future plans to revise the study plan as needed to reflect more detailed information and
understanding of the activities required to meet the goals and objectives of the project than was
reflected in the Vendor’s proposal based on the specifics of the 2014-2015 funded projects;
plans to develop final data collection instruments, administrative record protocols, and requests
for approval under the Paperwork Reduction Act to collect data necessary to support the
evaluation;
recruitment and training of data collectors;
timelines for collecting and processing outcome data from SNAP consumers, market operators
and vendors, and key stakeholders; proposed sources for necessary and relevant data beyond the
minimum core (including specific data items needed; the timing, frequency, and mode of data
collection; and data management and quality control processes); and any potential challenges and
proposed resolutions;
timelines for conducting the process evaluation synthesis; plans to ensure, to the extent feasible,
the consistency of process data gathered; and any potential challenges and proposed resolutions;
timelines for conducting the outcome evaluation and comparative analysis, plans to analyze the
data gathered (including the minimum core data set required of FINI Projects and FINI LargeScale Projects), and any anticipated challenges and proposed resolutions;
plans for preparation, review, and revision of preliminary, interim and final reports and briefings
of USDA policy officials and other interested stakeholders;
plans for creation of public use data files in a machine-readable and open format that is
nonproprietary, publicly available and without restrictions on its use, consistent with OMB
Memorandum M-13-13, Open Data Policy-Managing Information as an Asset.

In general, USDA expects to have multiple opportunities for engagement in the review and comment
on most aspects and products of the independent evaluation, including revised study, data collection
and analytic plans; data collection instruments and requests for PRA clearance; preliminary data
tables and graphics; preliminary, interim, and final reports; briefing materials; and public use data
files. The set of proposed tasks should clearly indicate proposed deliverables and timelines for
USDA review, and comment, and subsequent revisions.

14