SUPPORTING STATEMENT FOR
Healthy Incentives Pilot Evaluation
Kelly Kinnison
Office of Research and Analysis
Food and Nutrition Service
US Department of Agriculture
3101 Park Center Drive
Alexandria, VA 22302
Phone: 703-305-2124
Fax: 703-305-2576
E-mail: [email protected]
Chapter Page
PART A. JUSTIFICATION 1
A.1. Explanation of Circumstances That Make Collection of Data Necessary 1
A.2. How the Information Will be Used, by Whom, and for What Purposes 4
A.3. Uses of Improved Information Technology to Burden Reduction 9
A.4. Efforts to Identify and Avoid Duplication 11
A.5. Efforts to Minimize Burden on Small Businesses or Other Entities 11
A.6. Consequence of Less Frequent Data Collection 12
A.7. Special Circumstances Requiring Collection of Information in a Manner Inconsistent With Section 1320.5(D)(2) of the Code of Federal Regulations 13
A.8. Federal Register Comments and Efforts to Consult With Persons Outside the Agency 14
A.9. Payments to Respondents 15
A.10. Assurance of Confidentiality 16
A.11. Questions of a Sensitive Nature 18
A.12. Estimates of Respondent Burden 20
A.13. Estimates of Other Annual Costs to Respondents 20
A.14. Estimates of Annualized Government Costs 21
A.15. Changes in Hour Burden 21
A.16. Time Schedule, Publication, and Analysis Plans 22
A.17. Display of Expiration Date for OMB Approval 24
A.18. Exceptions to Certification Statement 24
PART B. COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 25
B.1. Respondent Universe and Sampling Methods 25
B.1.1. Respondent Universe 25
B.1.2. Sampling Methods 26
B.1.3. Response Rates and Non-Response Analysis 36
B.2. Procedures for the Collection of Information 39
B.2.1. Statistical Methodology for Stratification and Sample Selection 39
B.2.2. Estimation Procedures 39
B.2.3. Degree of Accuracy Needed for the purpose described in the justification 40
B.2.4. Unusual Problems Requiring Specialized Sampling Procedures 45
B.2.5. Any use of Periodic Data Collection Cycles to Reduce Burden 45
B.3. Methods to Maximize the Response Rates and to Deal With Nonresponse 46
B.4. Test of Procedures or Methods to be Undertaken 49
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 51
Appendixes
A1 Round 1 Advance Letter to Study Participant Survey: English
A2 Round 1 Advance Letter to Parents of Minor Study Participants: English
A3 Round 2 Advance Letter to Study Participant Survey: English
A4 Round 2 Advance Letter to Parents of Minor Study Participants: English
A5 Round 3 Advance Letter to Study Participant Survey: English
A6 Round 3 Advance Letter to Parents of Minor Study Participants: English
A7 Round 1 Advance Letter to Study Participant Survey: Spanish
A8 Round 1 Advance Letter to Parents of Minor Study Participants: Spanish
A9 Round 2 Advance Letter to Study Participant Survey: Spanish
A10 Round 2 Advance Letter to Parents of Minor Study Participants: Spanish
A11 Round 3 Advance Letter to Study Participant Survey: Spanish
A12 Round 3 Advance Letter to Parents of Minor Study Participants: Spanish
B1 Round 1 Study Participant and Primary Shopper Survey: English
B2 Round 2 Study Participant and Primary Shopper Survey: English
B3 Round 3 Study Participant and Primary Shopper Survey: English
B4 Round 1 Study Participant and Primary Shopper Survey: Spanish
B5 Round 2 Study Participant and Primary Shopper Survey: Spanish
B6 Round 3 Study Participant and Primary Shopper Survey: Spanish
B7 In Person Script: English
B8 Voice Mail script: English
B9 Refusal Avoidance and Conversion: English
C Automated Multiple Pass Method (AMPM) Screenshots
D1 Focus Group Participant Telephone Recruitment Script: English
D2 Focus Group Participant Consent Form: English
D3 Focus Group Protocol: English
D4 Focus Group Data Sheet: English
D5 Focus Group Participant Incentive Receipt Form: English
E1 Round 1 Participating Independent Store Survey
E2 Round 2 Participating Independent Store Survey
E3 Round 1 Participating Chain Store Survey
E4 Round 2 Participating Chain Store Survey
E5 Non-Participating Store Survey
E6 Withdrawn Store Survey
E7 Newly Participating Store Survey
E8 Round 1 and Round 2 Store Observation Form
E9 Follow-up Reminder Telephone Script
E10 Retailer Telephone Notification Recruiting Script
F1 Round 1 Massachusetts Executive Interview Guide
F2 Round 2 Massachusetts Executive Interview Guide
F3 Round 3 Massachusetts Executive Interview Guide
F4 Round 1 DTA Staff HIP Interview Guide
F5 Round 2 DTA Staff HIP Interview Guide
F6 Round 3 DTA Staff HIP Interview Guide
F7 Round 1 DTA Technical Team HIP Interview Guide
F8 Round 2 DTA Technical Team HIP Interview Guide
F9 Round 3 DTA Technical Team HIP Interview Guide
F10 Round 1 Local Office Interview Guide
F11 Round 2 Local Office Interview Guide
F12 Round 3 Local Office Interview Guide
F13 Round 1 EBT Contractor (ACS) Interview Guide
F14 Round 2 EBT Contractor (ACS) Interview Guide
F15 Round 3 EBT Contractor (ACS) Interview Guide
F16 Round 1 Integrated Retailers’ Interview Guide
F17 Round 2 Integrated Retailers’ Interview Guide
F18 Round 3 Integrated Retailers’ Interview Guide
F19 Round 1 Third Party Processors (TPPs) Interview Guide
F20 Round 2 Third Party Processors (TPPs) Interview Guide
F21 Round 3 Third Party Processors (TPPs) Interview Guide
F22 Round 1 Community Based Organization Interview Guide
F23 Round 2 Community Based Organization Interview Guide
F24 Round 3 Community Based Organization Interview Guide
G1 Federal Register Notice Comment
G2 National Agricultural Statistics Service Comments
H Technical Work Group (TWG): Evaluation Design and Study Plan
I IRB Approval
J Analysis Plan
This is a new information collection. The Food, Conservation, and Energy Act of 2008, also known as the 2008 Farm Bill (Public Law 110-246), Section 4141(K)(3)(E) authorized funds for pilot projects to evaluate health and nutrition promotion in SNAP to determine if nutrition education and incentives provided to SNAP recipients at the point-of-sale increase the consumption of fruits, vegetables, or other healthful foods. The legislation also provided for an evaluation of the funded pilot project. On the basis of this legislative authority, the U.S. Department of Agriculture (USDA) Food and Nutrition Service (FNS) designed the Healthy Incentives Pilot (HIP) and its evaluation. This OMB package requests clearance for HIP evaluation data collection activities planned for 2011-2012. This section contains an overview of the pilot and discusses the objectives of the evaluation.
Reducing the prevalence of obesity is a key national health objective. In response to the increase in obesity and chronic disease in the United States, the nation’s scientific authorities and policy leaders emphasize the goal of increasing fruit and vegetable intake.1,2 Fruit and vegetable intake reduces the long-term risk of obesity,3 which threatens both health and economic welfare. Fruit and vegetable intake are associated with lower rates and reduced risk of chronic diseases, including the major causes of death in the United States: heart disease and cancer.
Most U.S. adults fail to meet the fruit and vegetable intake goals as specified in the Dietary Guidelines for Americans.2 Only 28.4% of adults meet the fruit guidelines, and only 32.5% of adults meet the vegetable guidelines,4 and these intake patterns have not improved over time. Fruit and vegetable intake shortfalls are largest for low-income Americans and participants in the Supplemental Nutrition Assistance Program (SNAP), formerly known as the Food Stamp Program.
To address this problem, FNS designed the Healthy Incentives Pilot (HIP) which will provide financial incentives to SNAP households to encourage their purchase and consumption of ‘targeted’ fruits and vegetables. The targeted fruits and vegetables are the same foods that are eligible under the national definition for WIC fruit and vegetable cash value vouchers. These foods include fresh, canned, frozen, and dried fruits and vegetables without added salt, sugars, fats, or oils. Fruit juices, mature legumes, and white potatoes are excluded, but yams and sweet potatoes are included. The intention of this incentive is to increase fruit and vegetable consumption both through the standard economic pathway of lower prices and through the implicit message of the importance of fruit and vegetable consumption that is sent by the incentive itself.5,6,7
The pilot will be operated in Hampden County, MA, by the Massachusetts Department of Transitional Assistance (DTA). FNS selected the DTA competitively based on its comprehensive pilot proposal that included very thorough and strong design, implementation, staffing and management plans. Hampden County includes a mix of 27 urban, rural, and suburban cities with approximately 53,000 SNAP households. The majority of recipients are concentrated in the areas of Springfield, Holyoke, and Chicopee. Massachusetts will operate the pilot for 14 months, beginning in late 2011 and concluding in early 2013.
Congress mandated that a rigorous evaluation (using random assignment or other methods that are capable of producing scientifically valid estimates) be conducted to assess the impact of HIP on participants’ intake of fruits and vegetables. This evaluation is designed to answer the following research questions:
What is the impact of HIP on SNAP participants’ consumption of fruits and vegetables, other healthful foods, discretionary calories, and total calories?
What are the factors that influence how HIP impacts participants? Do the effects vary by household demographics, the household food environment, dietary knowledge, attitudes and beliefs, or household shopping patterns? How do participants perceive HIP and did participation affect their intention to purchase and consume fruits and vegetables?
What processes are involved in implementing and operating HIP? How are retailers, participants, and other stakeholders recruited? What are the challenges and lessons for future implementation?
What is the impact of HIP on the Pilot site and its partners? How does HIP affect workload and operations in the SNAP offices? Does HIP affect business for participating and non-participating retailers?
What are the costs associated with HIP implementation – i.e. initial startup, ongoing administration, and the costs of the incentives? What would it cost to implement HIP nationwide?
This section of the supporting statement provides an overview of the HIP as well as the research design and data collection efforts planned to meet the overall objectives of the HIP evaluation.
7,500 SNAP households within Hampden County, MA will be randomly assigned to participate in the pilot. HIP participants will earn an incentive of 30 cents for every dollar spent on targeted fruits and vegetables (TFVs). This incentive will be added to the household’s SNAP benefit account and can be used just like other SNAP benefits. The incentive payment is capped at $60 per household per month, which is a level sufficiently high that few households are expected to be constrained by the cap.8
The primary objective of the evaluation relates to the impact of HIP on SNAP participants. In order to evaluate HIP’s impact, we will randomly select a sample of 5,070 SNAP participants to take part in the evaluation – this includes 2,535 participants from the universe of 7500 HIP demonstration participants and 2,535 SNAP participants from the remainder of the SNAP caseload not receiving HIP to serve as the control group9. Thus, the demonstration and control samples will both be random samples from the common population of SNAP households in Hampden County, making them appropriate for comparison as a way of measuring the demonstration’s impact without risk of selection bias.
Understanding the implementation and costs of HIP and the potential for nationwide expansion—other important objectives of the evaluation—require collecting information from HIP stakeholders, including SNAP retailers(Business-for-profit), DTA State and local officials(State/Local agencies), HIP partners (e.g. Business not-for-profit community groups), and the Electronic Benefit Transfer (EBT) vendor and third-party-processors (Business-for-profit) involved in SNAP redemption. Samples of 75 retailers participating in HIP and a similar number of retailers choosing not to participate will be selected. Approximately 30 other stakeholders will be selected to provide information on HIP implementation and operations. The rest of this section provides an overview of the data collection efforts.
SNAP Participants – sampled SNAP HIP participants and SNAP non-HIP participants will complete three rounds of a computer-assisted telephone interview (CATI) survey at three points in time based on HIP implementation timelines: (1) at baseline prior to HIP implementation/participation, (2) 3 months into HIP implementation/participation, and (3) 11 months into HIP implementation/participation. Participants will be sent an advance letter explaining the study and the importance of their participation (Appendix A1 - Appendix A6), and interviewers will contact them within the next few weeks to conduct an interview.
When possible, the questions on the participant survey have been taken from well-validated existing surveys suitable for the study population, including NHANES10, NCI EATS11, NCI Food Attitudes and Behavior12. The survey is divided into two questionnaires: the Study Participant and the Primary Food Shopper questionnaires. Round specific annotated paper versions of the English surveys are included in Appendix B1 to B3 and Spanish surveys are included in Appendix B4 to B6 respectively. The Baseline Interview will collect information on:
Fruit and Vegetable intake,
Participation in nutrition assistance programs,
Exposure to nutrition education,
Family food environment,
Food preferences and beliefs,
Barriers to consuming fruits and vegetables,
General shopping patterns,
Food expenditures,
Household composition,
Demographics, and
Employment status.
While demographics and participation in nutrition assistance programs will only be collected in Round 1, the other topics listed above will also be included in Rounds 2 and 3 of the interviews. In addition, Rounds 2 and 3 will collect information on experiences with HIP (HIP participants only) and additional detail on dietary intake, using a 24-hour dietary recall protocol (Appendix C). We will use USDA’s Dietary Intake Data System to process, and code the dietary data13. This system includes the AMPM, the Post Interview Processing System (PIPS), and the SurveyNet. The Respondent and Primary Shopper surveys were cognitively tested in English and Spanish.
A sample of HIP participants not involved in the survey data collection will participate in focus group discussions. We will use the focus groups to listen to and gather information from participants regarding the implementation and operation of the pilot. Focus group topics include: how participants learned about HIP and the training they received; perceived changes in food spending and consumption after HIP implementation; and use of HIP benefits, including ease of use, process at checkout, and changes in shopping patterns.
We will conduct two rounds of focus groups, with three focus groups in each round. The first round will be conducted at the same time as the Round 2 participant interviews (3 months post-implementation) and the second group will occur at the same time as the Round 3 interviews (11 months post-implementation). A copy of the discussion guide and related focus group materials are found in Appendix D1 to D5.
Retailers – The retailer survey will collect information regarding issues surrounding the implementation and operation of HIP. A stratified sample of chain retailers, independent retailers and farmers’ markets will complete two rounds of a mail survey. Respondents include both HIP-participating retailers and non-HIP-participating retailers. The first round of the survey will occur prior to HIP implementation and the second round will occur late in HIP implementation (approximately 11 months after HIP begins). Topics covered in the first round include: store characteristics, recruitment to HIP and reasons for participating or not participating, preparations for the HIP demonstration, and fruit and vegetable inventory. The second round will focus on how HIP has affected the retailer, including: problems experienced, changes in stock of fruits and vegetables, and perceived effect on sales.
Store observations in a small sample of retailers (10 observations at three points in time, roughly corresponding to the rounds of the participant survey) will supplement the retailer survey, collecting first-hand data on transaction processes, fruit and vegetable inventory, and other environmental factors. Copies of the survey and observation form are shown in Appendix E1-E8.
Stakeholders – The local and State SNAP officials, local and State partners, EBT vendors and 3rd party processors will complete three rounds of in-person interviews to provide detailed information about the development, implementation, and operation of HIP in Hampden County. The rounds will coincide with the rounds of the participant survey and will focus on the following topics:
HIP implementation and operations,
EBT system modifications for HIP,
Notifying and training retail participants,
Impact of HIP on the State and its partners,
State and local costs of the pilot, and
Feasibility of HIP expansion.
Copy of the stakeholder interview guides are shown in Appendix F1-F24.
In addition to the data collection efforts described above, we will use administrative data provided by FNS and DTA for sampling and analysis purposes. Administrative data will include:
SNAP Case Files—demographic, household, eligibility, recertification and issuance information for persons in households receiving SNAP benefits. The SNAP Case File will be used to identify the sampling frame and conduct random assignment. It will also serve as a source of household characteristics for analysis.
Administrative Costs Files—data on the HIP-related implementation and operating costs for the State and local SNAP agencies, and costs for the EBT vendor and retailers reimbursed by the State.
Daily EBT Transaction Files—data on SNAP issuance, HIP and non-HIP purchase transactions, HIP incentive credits, and other transactions for each EBT card issued to a SNAP household. These files will be used to examine the take-up of the HIP incentive by participants and to examine the impact of HIP on retailer redemptions.
Retailer Files (STARS II and other DTA/FNS provided data)—retailer characteristics to be used for sampling retailers and in analysis of the retailer survey and the transaction data.
The information gathered in the data collection activities described above will be used by FNS to determine if SNAP recipients participating in HIP have higher fruit and vegetable consumption than recipients who did not receive the incentive. The data will also permit analysis of how impacts vary by recipient characteristics. The data collection is also essential for allowing FNS to determine the potential implications of a nationwide HIP-like program. There is currently no other effort that can address the research objectives of the proposed study.
Automation of Participant Data Collection. In compliance with E-Government Act 2002, to reduce burden to the respondent and improve data quality, the HIP participant survey will implement the use of computer-assisted data collection technology. The use of computer-assisted data collection technology reduces the survey completion time – automatic skip patterns that are built-in to the program ensure that respondents are asked only relevant questions, based on their response. The branching and skip patterns applied by the system will also prevent staff from mistakenly skipping sections, omitting questions, or asking the wrong questions during interviews.
Other Data Collections. Stakeholder interviews will be conducted in person as this is considered the best method for collecting data on program implementation and operations. The interviews are designed focus on the evaluation objectives and questions will be asked of the most knowledgeable respondent. The retailer survey is designed as a paper and pencil self-administered questionnaire. This methodology is deemed most appropriate for the respondent population—many small retailers have limited access to computers and limited familiarity with web-based surveys. Sample tracking for these data collections will be managed using an Access database.
24-Hour Dietary Recall Automated Multiple Pass Method (AMPM). 24-Hour Dietary Recall Automated Multiple Pass Method (AMPM). In order to obtain valid estimates of the usual nutrient intake in a study population, participants will complete a 24-hour dietary recall interview (24HR) which utilizes the Automated Multiple Pass Method (AMPM). (The study design requires a second recall to be collected on 10 percent (300) of the sample.) The AMPM is made possible by computer-assisted data collection technology. Trained telephone interviewers at the Westat call center will submit participants’ responses via the internal software program developed by the USDA Agricultural Research Service and Westat (see http://1.usa.gov/j4jf2Z).
FNS estimates 100 percent of these responses will be electronically submitted.
The data requirements for the evaluation have been carefully reviewed to determine whether the needed information is already available. There is no duplication of the data to be collected in this evaluation of the Healthy Incentives Pilot. HIP represents the most dramatic effort to promote fruits and vegetables ever attempted in the 35-year history of the modern SNAP program. Past research suggests that the HIP financial incentive holds promise, but the proposed information collection is required to evaluate the approach.14,15 Information required for analysis that has been determined to be accurately available elsewhere, such as participant status information previously collected, updated, and available in SNAP case files, will be obtained from the existing records and will not be duplicated in this study.
Information being collected has been held to the minimum required for the intended use. A sample of retailers will be asked to complete a self-administered questionnaire. FNS estimates that approximately 60 percent of retailers in Hampden County are small businesses. Therefore, across the two survey rounds, approximately 125 retailer survey respondents will be small businesses. The self-administered surveys are designed to that they can be done at a time most convenient to respondents. If respondents’ request, the survey will be completed by telephone.
This is a single-time study. Respondents to the HIP participant survey will be asked some questions two or three times over the course of the data collection period in order to evaluate the impact of HIP. The first round of interviews will take place prior to the implementation of HIP; questions are designed to capture baseline information (prior to exposure to the incentive). The design involves two rounds of surveys after the pilot begins—Round 2 will occur 3 months post-implementation and Round 3 will occur approximately 11 months post-implementation. In order to evaluate the impact of the pilot on SNAP participants’ consumption of fruits and vegetables and other foods, we need information on a variety of measures (e.g. attitudes and beliefs, exposure to nutrition education, fruit and vegetable spending and intake, family food environment), both before the pilot begins and during pilot operations to measure both shorter-term and longer-term impacts.
Retailers will also be asked some questions (e.g. store characteristics, fruit and vegetable inventory) at two points in time—prior to implementation and late in the implementation period. The rationale for asking questions at two different times is to understand how these factors changed over the course of the pilot.
Requiring respondents to report information to the agency more often than quarterly;
Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;
Requiring respondents to submit more than an original and two copies of any document;
Requiring respondents to retain records, other than health medical, government contract, grant-in-aid, or tax records for more than three years;
In connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of study;
Requiring the use of a statistical data classification that has not been reviewed and approved by OMB;
That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or
Requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.
In order to obtain valid estimates of the usual nutrient intake in a study population, the generally accepted technique is to collect a second 24-hour dietary recall (24HR) from a subsample of the same respondents.16,17 Therefore, the study design requires a second recall to be collected on 10 percent (300) of the sample.
There are no other special circumstances. The collection of information is conducted in a manner consistent with the guidelines in 5 CFR 1320.5
FNS published a notice February 7, 2011 in the Federal Register Volume 76, Number 25, pages 6,597-6,598 and provided a 60-day period for public comments. We received one comment that was not germane to the evaluation. Appendix G1 includes the comment received for the 60 day Federal Register Notice.
The information collection has been reviewed by the National Agricultural Statistics Service of USDA (NASS) with special reference to the statistical procedures. The report prepared by NASS is presented in Appendix G2. We incorporated the comments and suggestions regarding statistical procedures into Part B of the supporting statement.
FNS consulted with Federal government experts:
Margaret Andrews, USDA Economic Research Service, 202-694-5441
Sue Krebs-Smith, National Cancer Institute, 301-496-4766
Alanna Moshfegh, USDA Agricultural Research Service, 301-504-5040
A Technical Work Group (TWG) has been assembled and consulted regarding project design considerations for data collection and analysis. TWG members are listed in Table A8.1. The TWG members participated in a meeting with FNS and the evaluation contractor. Appendix H presents a summary of the issues discussed and feedback received at the meeting; it also presents decisions and the rationale for how input will be used to improve the evaluation. No major changes in the evaluation were recommended.
Name |
Title and Affiliation |
Area of Expertise |
Tom Baranowski (713-798-6762) |
Professor, Department of Pediatrics Baylor College of Medicine |
Experimental research design (community settings) |
Simone French (612-626-8594) |
Professor, Division of Epidemiology and Community Health School of Public Health University of Minnesota |
Incentives to promote healthful food choices |
Joel Gittelsohn (410-955-3927) |
Professor, Department of International Health Johns Hopkins Bloomberg School of Public Health |
Qualitative methods or process evaluation |
David Just (607-255-8048) |
Economist and Associate Professor, Department of Applied Economics and Management Cornell University |
Incentives to promote healthful food choices |
Diane Whitmore Schanzenbach (847-491-3884) |
Associate Professor, School of Education and Social Policy, Northwestern University Faculty Research Fellow, National Bureau of Economic Research. |
Food demand analysis |
Achieving high response rates is critical to data quality. Providing an extrinsic incentive increases cooperation rates, especially in populations defined as being in poverty,18,19 and a monetary incentive even more so.20 An effective incentive can improve initial response rates, which reduce the need for follow-up and decrease survey costs.21 In addition, incentives disproportionately encourage those less interested in the research to participate, thus reducing non-response bias.22 Finally, the research suggests that individuals receiving an incentive are more co-operative in providing contact information that will allow tracking their whereabouts for successive rounds of a survey.23 Based on this research, SNAP participants in the HIP study will receive a monetary incentive after completing each round of the survey: $20 for round 1, $30 for round 2, and $40 for round 3. This differentially higher incentive scale will encourage respondents to provide contact information for subsequent rounds, and will persuade them with a “promise” of a higher incentive, to continue their participation through the three rounds of data collection. Focus group participants will receive a monetary incentive of $75.
No incentive payments are planned for HIP-participating retailers, the EBT vendor, or other stakeholders; their involvement in the HIP evaluation. Participating retailers will sign a Memorandum of Understanding with the DTA that includes agreement to participate in the evaluation. The EBT vendor and most other stakeholders receiving funding from DTA and will be required to participate in the evaluation.
Securing the cooperation of the non-participating retailers and those who dropped out of the pilot may be more challenging than for participating retailers. We will encourage non-participating retailers or those who dropped out of the pilot to cooperate by noting that if a HIP-like program was to be implemented nationwide, retailers might see an increase in sales of fruits and vegetables that earned SNAP participants an incentive. We would also appeal to their interest in improving the health and nutrition of Americans participating in SNAP. Finally, we will offer a modest $40 monetary incentive to the non-participating retailers for completing a survey.
The individuals participating in this study will be notified that the information they provide will not be published in a form that identifies them. No identifying information will be attached to any reports. Identifying information will not be included in the public use dataset. USDA will receive a dataset that includes participant IDs so that USDA can conduct follow-up research in the future using administrative data.
Abt Associates Inc. and Westat have extensive experience in data collection efforts requiring strict procedures for maintaining the privacy, security, and integrity of data. Specific data handling and reporting procedures will be employed to maintain the privacy of survey participants and composite electronic files. These data handling and reporting procedures include requiring all project staff, both permanent and temporary, to sign a confidentiality and nondisclosure agreement (Appendix I3). In this agreement, staff pledges to maintain the privacy of all information collected from the respondents and will not disclose it to anyone other than authorized representatives of the evaluation, except as otherwise required by law.
In addition, Abt Associates Inc. and Westat have established a number of procedures to ensure the privacy and security of electronic data in their offices during the data collection and processing period. A system of record notice (SORN) titled FNS-8 USDA/FNS Studies and Reports in the Federal Register on March 31, 2000, Volume 65, Number 63, and is located on pages 17251-17252 discusses the terms of protections that will be provided to respondents.
Abt Associates maintains its own Institutional Review Board (IRB), which serves as the organization’s administrative body that conducts prospective reviews of proposed research and monitors continuing research for the purpose of safeguarding research participants’ rights and welfare. All research involving interactions or interventions with human subjects is within the purview of the Abt IRB. Abt Associates’ IRB is the local agent responsible for ensuring that the organization’s research: 1) meets only the highest ethical standards; and 2) receives fair, timely, and collegial review by an external panel. Abt Associates’ IRB currently holds a federal-wide assurance (FWA) of compliance from the U.S. Department of Health and Human Services’ Office of Human Research Protections (DHHS/OHRP). The FWA covers all federally supported or conducted research involving human subjects. All study materials and instruments for the HIP evaluation were approved by Abt’s IRB. Copies of the IRB approval letters are in Appendix I1 and I2.
Several questions on the HIP participant survey could be considered to be sensitive to study respondents. These include participant marital status, household participation in nutrition assistance programs, household income, and household food expenditures. In addition, for some participants the enumeration of their household, which requires listing all members living in the household, might be a sensitive area. Finally, when reporting their 24-hour dietary recall, participants are asked to remember “everything they ate or drank yesterday However, because the purpose of the study is to measure the impact of HIP on SNAP households and their quality of life, and such determinations are based on household characteristics (e.g., household size, number of dependents, income,), those questions are necessary to evaluate HIP. All of these questions were cognitively tested and no respondent indicated an unwillingness to answer the question or discomfort with providing a response.
Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated. If this request for approval covers more than one form, provide separate hour burden estimates for each form and aggregate the hour burdens in Item 14 of OMB Form 83-I.
Provide estimates of annualized cost to respondents for the hour burdens for collections of information, identifying and using appropriate wage rate categories.
Table A12.1 shows sample sizes, estimated frequency of responses, estimated annual responses, estimated burden, and estimated annualized cost of respondent burden for each part of the data collection and for total burden.
The estimated annualized cost to respondents for the HIP participant survey and HIP participant focus groups with primary food shoppers is based on the national minimum wage of $7.25 per hour. The estimated annualized cost to respondents for the retailer survey is based on the May 2009 National Occupational Employment and Wage Estimates for the United States, available at www.bls.gov/oes/current/oes_nat.htm, for job category “First-Line Supervisors / Managers of Retail Sales Workers”, occupation code #41-1011, with a median wage of $16.78 per hour. The estimated annualized cost to respondents for the stakeholder interviews (including local and State SNAP officials, local and State partners, EBT vendors/3rd party processors) is based on the May 2009 National Occupational Employment and Wage Estimates for the United States, available at www.bls.gov/oes/current/oes_nat.htm, for job category “Management Occupations”, occupation code #11-0000, with a median wage of $42.95 per hour.
Affected Public |
Respondent |
Number
|
Average Responses Annually Per Respondent |
Total Annual Responses |
Estimated Hours Per Response |
Estimated Total Hours |
Annualized Cost of Respondent Burden |
|
Individual or Households |
SNAP recipients* |
Pretest |
27** |
1.000 |
27 |
0.559 |
15.09 |
$ 109 |
Completed |
2,828 |
2.061 |
5,829 |
.7672 |
4,472.01 |
$ 32,422 |
||
Attempted |
1,212 |
1.720 |
2,085 |
.0521 |
108.63 |
$ 788 |
||
SNAP recipient focus groups |
Completed |
60 |
1.000 |
60 |
1.5000 |
90.00 |
$ 653 |
|
Attempted |
6 |
1.000 |
6 |
0.0500 |
0.30 |
$ 2 |
||
SUBTOTAL |
|
4,133 |
|
8,007 |
|
4,686.03 |
$ 33,974 |
|
Business for profit |
Retailers |
Pretest |
9 |
1.000 |
9 |
0.5000 |
4.50 |
$ 76 |
Completed |
165 |
1.000 |
165 |
0.5758 |
95.01 |
$ 1,594 |
||
Attempted |
42 |
1.000 |
42 |
0.0500 |
2.10 |
$ 35 |
||
EBT
vendors/ |
Completed |
8 |
1.000 |
8 |
1.5000 |
12.00 |
$ 515 |
|
Attempted |
1 |
1.000 |
1 |
0.0500 |
0.05 |
$ 2 |
||
SUBTOTAL |
|
225 |
|
225 |
|
113.66 |
$ 2222 |
|
State and Local agencies |
Local and State SNAP Officials |
Completed |
17 |
1.000 |
17 |
1.4118 |
24.00 |
$ 1,031 |
Attempted |
2 |
1.000 |
2 |
0.0500 |
0.10 |
$ 4 |
||
SUBTOTAL |
|
19 |
|
19 |
|
24.10 |
$ 1035 |
|
Business not for profit |
Local and State Partners |
Completed |
5 |
1.000 |
5 |
1.5000 |
7.50 |
$ 322 |
Attempted |
1 |
1.000 |
1 |
0.0500 |
0.05 |
$ 2 |
||
SUBTOTAL |
|
6 |
|
6 |
|
7.55 |
$ 324 |
|
|
TOTAL |
|
4,383 |
|
8,257 |
|
4,831.34 |
$ 37,555 |
*Figures inclusive of burden for treatment and control groups
** English and Spanish surveys were tested in two rounds; no more than 9 respondents were asked the same question
There are no capital/start-up or ongoing operation/maintenance costs associated with this information collection.
The total annualized cost to the Federal government for all tasks associated with the HIP Evaluation, including the data collection and associated project costs, is $2,993,504 per year. This includes evaluation contractor costs associated with: (1) developing instruments, correspondence and administrative forms; (2) developing sampling plan and sample selection; (3) developing evaluation, data collection and analysis plans; (4) systems programming and testing of the data collection software and tracking systems; (5) interviewer training; (6) data collection; (7) data cleaning and processing; (8) data tabulation and analyses; (9) report writing; and (10) overall project management. It also includes costs of the FNS Project Officer for monitoring the study.
This is a new collection of information and as a result of program changes will add 4,831.34 hours to FNS inventory.
The schedule for the study showing sample selection, beginning and ending dates of collection of information, completion of reports, and publication dates is shown on Table A16.1.
Activity |
Schedule |
Sample selection |
August 2011 |
Round 1 field period: participants, retailers, stakeholders |
September–December 2011 |
Round 2 field period: participants, stakeholders |
February–April 2012 |
Round 3 field period: participants, retailers, stakeholders |
August–December 2012 |
Implementation report |
March 2012 |
Interim report |
November 2012 |
Final report |
September 2013 |
Summary report |
December 2013 |
Annual reports to Congress |
December 2010 December 2011 December 2012 |
The primary purpose of the study is to evaluate the implementation, operation, and impact of HIP. The main lines of analysis follow the five research objectives outlined in section A.2. Exhibit A16.2 presents an overview of the research objectives, data collection activities, and reports that this study will produce for future policy development regarding point-of-purchase financial incentives. Appendix J presents the detailed analysis plan.
Objectives |
Data Sources |
Reports |
1. Assess the causal impact of HIP on participant fruit and vegetable consumption and other key measures of dietary intake. |
Participant survey: R1, R2 Participant focus groups: R1 EBT transactions data |
Interim |
Participant survey: R1, R2, R3 Participant focus groups: R1, R2 EBT transactions data |
Final |
|
2. Identify and assess factors that influence how HIP impacts participants. |
Participant survey: R1, R2 Participant focus groups: R1 SNAP casefile data EBT transactions data |
Interim |
Participant survey: R1, R2, R3 Participant focus groups: R1, R2 SNAP casefile data EBT transactions data |
Final |
|
3. Describe the processes involved in implementing and operating HIP. |
Retailer survey: R1 Retailer observation: R1 Stakeholders*: R1 |
Implementation |
Retailer survey: R1, R2 Retailer observation: R1, R2, R3 Stakeholders*: R1, R2, R3 Participant survey: R2, R3 |
Final |
|
4. Assess the impact on the State SNAP agency, the local SNAP Agency, and partners. |
Retailer survey: R1, R2 Retailer observation: R1, R2, R3 Stakeholders*: R1, R2, R3 EBT transactions data |
Final |
5. Quantify, to the extent possible, the Federal, State, and local administrative and benefit costs of the pilot. |
Retailer survey: R1, R2 Retailer observation: R1, R2, R3 Stakeholders*: R1, R2, R3 EBT transactions data |
Final |
* Includes State and local agency SNAP staff, State and local partners (including community organizations), EBT vendor/Third Party Processors
The study’s findings will be presented in reports that will undergo peer review. Once final, FNS will make the reports available on its web site. Findings may also be published in one or more professional journals and publications intended for general or trade audiences, such as nutrition educators or food retailers.
All data collection instruments for the Healthy Incentives Pilot Evaluation will display the OMB approval number and expiration date.
There are no exceptions to the Certification for Paperwork Reduction Act (5 CFR 1320.9) for this study.
1U.S. Department of Health and Human Services (2000). Healthy People 2010. 2nd Edition, Washington, DC: U.S. Government Printing Office.
2U.S. Department of Agriculture and U.S. Department of Health and Human Services. Dietary Guidelines for Americans, 2010. 7th Edition, Washington, DC: U.S. Government Printing Office, December 2010.
3He, K., Hu, F.B. Colditz, G.A., Manson, J.E., Willett, W.C., and Liu. S. (2004). Changes in intake of fruits and vegetables in relation to risk of obesity and weight gain among middle-aged women. International Journal of Obesity Related Metabolic Disorders 28(12), 1569-74.
4Casagrande, S., Wang, Y, Anderson, C & Gary T. (2007). Have Americans increased their fruit and vegetable intake? The trends between 1988 and 2002. American Journal of Preventive Medicine 32(4), 257-263.
5Wilde, P.E., & Andrews, M. (2000). The Food Stamp Program in an era of welfare reform: Electronic benefits and changing sources of cash income. Journal of Consumer Affairs, 34(1), 31-46.
6Wilde, P & Ranney, C. (2000). The monthly food stamp cycle: Shopping frequency and food intake. American Journal of Agricultural Economics 82, 200-213.
7Wilde, P.E., Troy, L.M, & Rogers, B.L. (2009). Food Stamps and food spending: An Engel Function approach. American Journal of Agricultural Economics 91(2), 416-430, May 2009.
8Based on three simulations using three alternative models of consumer purchasing behavior. Findings indicated that if the statutory cap is set too low, it could limit the impact of HIP on households’ expenditures for target fruits and vegetables. (HIP participants who are not capped face a price for fruits and vegetables that is reduced by 30%, whereas participants who are capped face the original price of fruits and vegetables.) However, if the statutory cap is set too high, it could encourage fraud and cause the total incentive payments to exceed the budgeted amount of $2 million.
9 We expect 20.3% attrition from SNAP between sampling and the start of data collection, yielding 2,020 respondents on SNAP in each group (a total of 4,040 participants, as shown in Exhibit A12.1.). We will attempt to contact these 4,040 respondents to complete the baseline survey.
10 NHANES: OMB No. 0920-0237; EXP DATE 12/ 31/2011
11 NCI EATS: OMB No. 0925-0450; EXP. DATE: 07/31/2000
12 FAB: OMB #0925-0560; Exp. Date: 03/31/2009
13 Raper N, Perloff B, Ingwersen L, Steinfeldt L, Anand J. An overview of USDA’s Dietary Intake Data System. 2004. J Food Composition and Analysis 17: 545-555.
14Andreyeva, T., Long, M.W., & Brownell, K.D. (2010.) The impact of food prices on consumption: A systematic review of research on the price elasticity of demand for food. American Journal of Public Health 100(2), 216-22.
15Government Accountability Office. (2008). Food Stamp Program: Options for delivering financial incentives to participants for purchasing targeted foods (GAO-08-415). Washington, D.C.: GPO. Available: http://www.gao.gov/new.items/d08415.pdf..
16S.M. Nusser, A..L. Carriquiry, et al (1996). A semi-parametric transformation approach to estimating usual nutrient intake distributions, J. Am. Stat. Assoc., 91: 1440-1449.
17Institute of Medicine (IOM). Dietary Reference Intakes: Applications in dietary assessment. Washington, DC: National Academies Press; 2000.
18Singer E. (2002). The use of incentives to reduce non response in households surveys in: Groves R, Dillman D, Eltinge J, Little R (eds) Survey Non Response. New York: Wiley, pp 163-177.
19James T. (1996). Results of wave 1 incentive experiment in the 1996 survey of income and program participation. Proceedings of the Survey Research Section, American Statistical Association., 834-839.
20Groves R, Fowler F, Couper M, Lepkowski J, Singer E. (2009) in: Survey methodology. John Wiley & Sons, pp 205-206.
21Berlin M, Mohadjer L, Waksberg J, Kolstad A, Kirsch D, Rock D, Yamamoto K. (1993). An experiment in monetary incentives. Proceedings of Survey Research Methods Section of American Statistical Association, 393-8.
22Groves R, Singer E, Corning A. (2000). Leverage-saliency theory of survey participation: description and an illustration. Public Opinion Quarterly. 64(3): 299-308.
23Shettle C, Mooney G. (1999). Monetary incentives in US Government surveys. Journal of Official Statistics, 15(2): 231-250.
| File Type | application/msword |
| File Title | Healthy Incentives Pilot Evaluation |
| File Modified | 2011-05-24 |
| File Created | 2011-05-24 |