ARRA II Study -- Part A June Update

Study to Assess the Effect
of Supplemental Nutrition Assistance Program Participation on Food Security in the
Post-American Recovery and Reinvestment Act Environment

Part A

March 8, 2011

Contract Number: AG-3198-D-10-0051 Mathematica Reference Number: 06801.410 Submitted to: Office of Research and Analysis Supplemental Nutrition Assistance Program Program Development Division, Certification Policy Branch USDA Food and Nutrition Service 3101 Park Center Drive, Room 1014 Alexandria, VA 22302 Project Officer: Sarah Zapolsky Submitted by: Mathematica Policy Research 600 Maryland Avenue, SW Suite 550 Washington, DC 20024-2512 Telephone: (202) 484-9220 Facsimile: (202) 863-1763 Project Director: Jim Ohls Deputy Project Director: James Mabli Survey Director: Dawn V. Nelson

Study to Assess the Effect of Supplemental Nutrition Assistance Program Participation on Food Security in the Post-American Recovery and Reinvestment Act Environment Part A March 8, 2011

CONTENTS

A JUSTIFICATION 1

A.1. Circumstances Making the Collection of Information Necessary 1

A.2. Purpose and Use of the Information 2

A.3. Use of Information Technology and Burden Reduction 4

A.4. Efforts to Identify Duplication and Use of Similar Information 5

A.5. Impacts on Small Businesses or Other Small Entities 6

A.6. Consequences of Collecting the Information Less Frequently 6

A.7. Special Circumstance Relating to the Guideline of 5 CFR 1320.5 6

A.8. Comments in Response to the Federal Register Notice and Efforts to Consult Outside Agency 7

A.9. Explanation of Any Payment or Gift to Respondents 8

A.10. Assurance of Confidentiality Provided to Respondents 9

A.11. Justification for Sensitive Questions 10

A.12. Estimates of Hour Burden Including Annualized Hourly Costs 11

A.13. Estimates of Other Total Annual Cost Burden to Respondents or Record Keepers 12

A.14. Annualized Cost to Federal Government 12

A.15. Explanation for Program Changes or Adjustments 12

A.16. Plans for Tabulation and Publication and Project Time Schedule 12

1. Household Telephone Survey Data Analysis 13

2. Current Population Survey Analysis 22

3. In-Depth Interview Analysis 27

4. Project Schedule 28

A.17. Reason(s) Display of OMB Expiration Date Is Inappropriate 29

A.18. Exceptions to Certification for Paperwork Reduction Act Submissions 29

A.19. Customer Service Center 29

B COLLECTIONS OF INFORMATION EMPLOYING STATISTICAL METHODS 29

B.1. Respondent Universe and Sampling Methods 29

B.2. Procedures for the Collection of Information 32

B.3. Methods to Maximize Response Rates and to Deal with Nonresponse 39

B.4. Test of Procedures or Methods to be Undertaken 44

B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 46

References 47

Legislative Authority

APPENDIX A: Telephone Survey - English

APPENDIX B: Telephone Survey – Spanish

APPENDIX C: In-depth Interview Guide

COPY OF FEDERAL REGISTER NOTICE

APPENDIX d: Comments received in response to federal register notice

APPENDIX E: FNS Response to Federal Register comments

APPENDIX F: Respondent Advance Letter—Baseline Telephone Survey (English & Spanish)

APPENDIX G: Respondent Reminder Post Card (English & Spanish)

APPENDIX H: Respondent Advance Letter—Follow-up Telephone Survey (English & Spanish)

APPENDIX I: Respondent Thank You Letter—(English & Spanish)

APPENDIX J: IN-DEPTH INTERVIEW CONSENT FORM

Appendix K: NASS Comments

TABLES

A.1 Estimated Total Annual Hour Burden by Respondent Type 9

A.2 Percentage of Households That Are Food Insecure, by Length of SNAP
Participation 13

A.3 Regression Coefficients of the Effects of SNAP Participation and Household Characteristics on a Household’s Likelihood of Being Food Insecure 15

A.4 Regression-Adjusted Percentages of Households That Are Food
Insecure, by Length of SNAP Participation 16

A.5 Regression-Adjusted Percentages of Households That Are Food
Insecure, by Length of SNAP Participation and by Whether Household
Has Children 19

A.6 Percentage of SNAP Households That Are Food Insecure Before and
After the 2009 ARRA Benefit Increase, by CPS-FSS Sample 22

A.7 Percentage of Households That Are Food Insecure, by SNAP
Participation Status: Changes from 2008 to 2009 24

B.1 Data Collection Assumptions 31

B.2 Minimum Detectable Differences over Time in the
Longitudinal Design 36

B.3 Minimum Detectable Differences over Time in the Cross-Sectional
Design 37

This page has been left blank for double-sided copying.

A. JUSTIFICATION

A.1. Explain the circumstances that make the collection of information necessary. Identify any legal or administrative requirements that necessitate the collection. Attach a copy of the appropriate section of each statute and regulation mandating or authorizing the collection of information.

The Food and Nutrition Service (FNS), U.S. Department of Agriculture (USDA), is requesting approval from the Office of Management and Budget (OMB) to conduct The Study to Assess the Effect of SNAP (Supplemental Nutrition Assistance Program) Participation on Food Security in the Post-ARRA (American Reinvestment and Recovery Act) Environment, which involves a new collection of information. Assisting in the project will be FNS’ contractor, Mathematica Policy Research, a well-known survey and research firm.

The Supplemental Nutrition Assistance Program (SNAP)—the new name for the federal Food Stamp Program—is the largest of the 15 domestic food and nutrition assistance programs administered by USDA’s FNS and is a central component of the Federal nutrition assistance safety net. SNAP provides nutrition assistance benefits and education services to reduce hunger and to improve the health and well-being of low-income individuals and families. Over the past few years, SNAP participation has increased over 26 percent and is now at record high levels with over 20.7 million households across the nation enrolled in the program in January 2011 (USDA, 2011). The prevalence of food insecurity is also at a record high level, at 14.6 percent in 2009, as noted in the latest annual report on household food insecurity (Nord et al, 2010).

Policymakers, advocates, and those administering SNAP have long hypothesized that increasing benefit amounts would reduce food insecurity and, perhaps, draw into the program individuals who might otherwise have been reluctant to apply. A unique opportunity to measure the impact of increased benefits on food insecurity has been presented by the implementation of the American Recovery and Reinvestment Act (ARRA) economic stimulus package of 2009, which temporarily increased the maximum allotments provided to SNAP participants by 13.6 percent, eased eligibility requirements for childless adults without jobs, and provided additional funding to state agencies responsible for administering the program. This evaluation of the effect of SNAP participation on food security will provide new information on the extent to which food insecurity declines with SNAP participation in a post-ARRA environment. This will produce important new evidence on the program’s impact, will inform policy decisions regarding the size of SNAP allotments for eligible households, and inform related operational decisions regarding the likely impact of allotment changes on the propensity to participate.

Legislative Authority. Section 17 [7 U.S.C. 2026] (a)(1) of the Food and Nutrition Act of 2008 provides general legislative authority for the planned data collection. This section authorizes the Secretary of Agriculture to enter into contracts with private institutions to undertake research that will help to improve the administration and effectiveness of SNAP in delivering nutrition-related benefits. More specific legislative authority is found in the ARRA economic stimulus package implemented in April 2009 (Public Law Number 111-5, Section 101(c)(1)), which provides administrative funds to FNS for management and oversight and for managing the integrity and evaluation of the stimulus changes.

A.2. 2. Indicate how, by whom, and for what purpose the information is to be used. Except for a new collection, indicate the actual use the agency has made of the information received from the current collection.

The purpose of this data collection is to allow analysis that will support the following study objectives:

• To determine how, if at all, the prevalence of household food insecurity and amount of food expenditures vary with SNAP participation.

• To determine how, if at all, the observed results vary by key household characteristics and circumstances.

• To determine what factors distinguish between food-secure and food-insecure SNAP households with children.

To meet the first two objectives, FNS will recruit, and conduct a 30-minute, structured telephone interview with two samples of SNAP households – one representative of newly certified households, and the other representative of households which, in their current spell, have participated in the program for six to seven months. Both groups will contain interviews conducted in English and Spanish.

After sampling data is obtained from the states, using the most current address information, advance letters signed by a USDA official that include a $2 prepaid incentive and a promised $20 additional incentive upon completion of the telephone survey will be sent by first class postal mail to convince the sampled households of the value of the baseline survey and the importance of participation. This effort will be repeated for the six-month follow-up interview. The 60 participants in the in-depth interview will also receive a $30.00 post pay incentive for completing the 90-minute interview.

Approximately three days after advance letters are mailed to the sampled households, experienced, trained telephone interviewers will begin contacting the households and conducting interviews using the programmed computer-assisted teleph5one interview (CATI) instrument.

• The first sample will be interviewed twice – once soon after they have been approved to receive benefits to assess food expenditures and food security before SNAP becomes part of the usual family resources, and again (for those who continue to participate in the program) approximately six to seven months later

• The second sample will only be interviewed once.

These data will be used to measure the effectiveness of SNAP in two ways – once by comparing in the same calendar period the food security status of new entrant SNAP households and households that have participated for six to seven months, and again by examining the change in the new-entrant households’ food security between the baseline and follow-up interviews. The former approach allows a cross-sectional comparison that controls for change over time, while the latter allows a longitudinal comparison that controls for differences in the characteristics of the sampled households. These results will be synthesized to provide definitive evidence of the impact of SNAP on household food security.

The third objective will be met by obtaining qualitative data. Specifically, FNS will select a small subsample of each of the two initial samples, choosing households with children, and conduct a 90-minute, in-depth, in-person, unstructured interview with each. The subsample will include households that are food secure, those that have low food security, and those with very low food security. Data from these in-depth interviews will provide detailed information that will help explain the lives and experiences of SNAP participant households and generate important insights into the challenges low-income families face and the coping strategies they use to maintain food security. The interviews will also contribute to an overall understanding of food security, since the insights generated from them are intended to inform the findings from the descriptive and multivariate analyses of the research study.

Complementing the new data collection will be a comparison of food-security data from December 2008 (pre-ARRA) with data from December 2009 and 2010 (post-ARRA), from the Current Population Survey Food Security Supplement (CPS-FSS). This comparison will provide a general picture of the prevalence and characteristics of food-insecure households before and after the ARRA-related benefit increase.

The analysis of the data will be published in a report made available to policymakers and program operators, and to the public through the FNS website.

A.3. Describe whether, and to what extent, the collection of information involves the use of automated, electronic, mechanical, or other technological collection techniques or other forms of information technology, e.g. permitting electronic submission of responses, and the basis for the decision for adopting this means of collection. Also describe any consideration of using information technology to reduce burden.

FNS is committed to complying with the E-Government Act, 2002, to promote the use of technology. The use of information technology will be incorporated into the data collection, and other steps will be taken to reduce respondent burden and improve data quality:

• Computer-assisted telephone interview (CATI) technology will be used to conduct the structured telephone surveys, improving the pace and flow of the interviews and reducing respondent burden.¹

• Administrative record data, when available to the research contractor, will be used to reduce the number of questions asked in the respondent interviews.

• Electronic mail will be used when possible to send reminders and other communications to the sampled households.

• The in-person approach of in-depth interviews² will enhance access to hard-to-reach households, and the conversational exchange will allow us to obtain more detailed information with which to supplement the CATI surveys. Interviewers visiting respondents’ homes will be able to make and record observations about the home environment and the neighborhood, and respondents’ greater ease with being in their own homes for the interviews will reduce nonresponse.

A.4. Describe efforts to identify duplication. Show specifically why any similar information already available cannot be used or modified for use for the purposes described in Item 2 above.

The data requirements for the evaluation have been carefully reviewed to determine whether the needed information is already available. Efforts to identify duplication included a review of FNS reporting requirements, State administrative agency reporting requirements, and special studies by government and private agencies. It was concluded that no existing data sources can provide data needed to answer the study’s research questions. As noted above, the analysis will make use of extant data from the CPS-FSS to the extent feasible, minimizing duplication of data collection. The new data collected for this study are required to understand the circumstances of newly certified SNAP households. Completing ample numbers of surveys within a few weeks of certification is critical to assessing food expenditures and food security before benefits become part of the usual family resources. Although the CPS-FSS instrument includes questions on income, SNAP participation, and food security measures, very few households just entering the SNAP program are represented in the data, which is a key element for the current project.

A.5. If the collection of information impacts small businesses or other small entities (Item 5 of OMB Form 83-I), describe any methods used to minimize burden. No small businesses or other small entities will be involved in this information collection.

A.6. Describe the consequence to Federal program or policy activities if the collection is not conducted or is conducted less frequently, as well as any technical or legal obstacles to reducing burden.

This is a one-time data collection effort in response to a legislative mandate. If the study is not conducted, FNS will not have an effective means of assessing whether, and to what extent, food insecurity changes with SNAP participation or whether food insecurity has declined as a result of increased SNAP benefits in the post-ARRA environment.

A.7. Explain any special circumstances that would cause an information collection to be conducted in a manner:

• Requiring respondents to report information to the agency more often than quarterly;

• Requiring respondents to prepare a written response to a collection of information in fewer than 30 days after receipt of it;

• Requiring respondents to submit more than an original and two copies of any document;

• Requiring respondents to retain records, other than health medical, government contract, grant-in-aid, or tax records for more than three years;

• In connection with a statistical survey that is not designed to produce valid and reliable results that can be generalized to the universe of study;

• Requiring the use of a statistical data classification that has not been reviewed and approved by OMB;

• That includes a pledge of confidentiality that is not supported by authority established in statute or regulation, that is not supported by disclosure and data security policies that are consistent with the pledge, or which unnecessarily impedes sharing of data with other agencies for compatible confidential use; or

• Requiring respondents to submit proprietary trade secret, or other confidential information unless the agency can demonstrate that it has instituted procedures to protect the information’s confidentiality to the extent permitted by law.

There are no such special circumstances. The collection of information is conducted in a manner consistent with the guidelines in 5 CFR 1320.5.

A.8. If applicable, provide a copy and identify the date and page number of publication in the Federal Register of the agency's notice, required by 5 CFR 1320.8 (d), soliciting comments on the information collection prior to submission to OMB. Summarize public comments received in response to that notice and describe actions taken by the agency in response to these comments. Specifically address comments received on cost and hour burden.

Federal Register Notice. In accordance with 5 CFR 1320.8(d), an announcement of FNS’s intention to seek approval to collect this information provided an opportunity for public comment on this study. This announcement was published in the Federal Register, Volume 75, Number 241, pp. 78673 to 78674 and specified a 60-day period for comment ending February 14, 2011. One comment was received in response to this announcement. A copy of the notice is attached; the comment received is attached in Appendix D and the FNS response is attached in Appendix E: FNS Response to Federal Register comments.

Consultation with experts. The design of this study has proceeded through many stages, which involved consulting a wide range of individuals. In addition to the expert study design authors at Mathematica Policy Research (609-799-3535) that we retained for the project, we consulted with outside experts. Mark Nord of USDA ERS (202-694-5433) reviewed all the instruments, Kathy Edin at Harvard’s Kennedy School of Government (617-495-1100) was consulted in preparation of the in-depth interview guide, and Sharyn Lavender of the Statistics Division, NASS/USDA (202-690-0901) reviewed sampling and statistical methodologies for the National Agricultural Statistical Service.

A.9. Explain any decision to provide any payment or gift to respondents, other than remuneration of contractors or grantees.

It is well documented that cash incentives improve survey response, especially among low-income populations (Kovac and Markesich (2003), Nemeth (2009), Singer et al (1999)). In order to maximize response, it is essential to include incentives in this study; therefore, FNS proposes to offer respondents a two-tiered incentive plan: a $2 bill sent with the advance letter to gain attention and interest, followed by a $20 incentive after respondents complete the 30-minute telephone interview. A $30 post-pay incentive will be provided for completing the 90-minute in-depth interview. Respondents will be assured that the receipt of this money will not affect the SNAP benefits they currently receive.³ The $20 and $30 incentives will be given in the form of gift cards, which can be accessed and redeemed more easily and conveniently than checks. This is an especially important consideration for participants, many of whom may not have bank accounts or access to inexpensive check-cashing facilities.

A.10. Describe any assurance of confidentiality provided to respondents and the basis for the assurance in statute, regulation, or agency policy.

FNS will comply with the Privacy Act of 1974. Section 11(e)(8) of the Act and Section 272.1(c) of the regulations limit the use or disclosure of information obtained from applicant households or contained in the case files of participating households to persons directly connected with the administration of SNAP or other federal or federally assisted means-tested programs; persons directly connected with the verification of the immigration status of aliens; the Office of the Comptroller General of the United States for audit and examination authorized by other provisions of law; and to local, state, or federal law enforcement officials for the purpose of investigating an alleged violation of the Food and Nutrition Act or regulations.

In the advance materials and at the start of the telephone interview, respondents will be informed that the information they provide will be kept private and will not be disclosed to anyone but the researchers authorized to conduct the study, except as otherwise required by law. Advance letters (see Appendices F–I) sent to respondents will also tell them that the information being gathered is for research purposes only, and that their identity will not be disclosed to anyone outside of the study. A written consent form will be given to all who participate in in-depth interviews before the interviews commence (see Appendix J).

As with all studies of this nature, and in accordance with the Privacy Act System of Records FNS-8 (FNS Studies and Reports, published at 65 Federal Register 17251-17252), personally identifiable information (PII) obtained during data collection, such as contact information, will be stored in a secure study database on an encrypted secure server. Only those study team members with a viable reason to view the PII will have access to it. No information will be reported by the contractor in any way that permits linkage to individual respondents, and the information will be destroyed once the final study report has been released. Additionally, the contractor requires every employee to sign a pledge to protect the confidentiality of data and respondent identity, the breaking of which is grounds for immediate dismissal and possible legal action.

A.11. Provide additional justification for any questions of a sensitive nature, such as sexual behavior or attitudes, religious beliefs, and other matters that are commonly considered private. This justification should include the reasons why the agency considers the questions necessary, the specific uses to be made of the information, the explanation to be given to persons from whom the information is requested, and any steps to be taken to obtain their consent.

While the questions in the household telephone survey are largely not of a sensitive nature, some households may be reluctant to provide information on household income, food security, mental health and well-being, and the respondent’s body weight and height. Obtaining answers to such questions is, however, essential to the effective execution of this study, for the following reasons:

• Food security, which is the primary outcome measure, serves as the main indicator of household health and well-being. The food security measures we are including in the telephone interview are widely used as indicators of adequate food access in a number of major public national surveys, including the Current Population Survey.

• Income and sources of income are critical background characteristics, both in that they define key subgroups of households, and that they are important control variables in assessment of household outcomes.

• Mental health and well-being measures other than food security measures, such as questions on whether the respondent suffers from depression, will be used to define subgroups when evaluating the effect of SNAP participation on household food security.

• The respondent’s body weight and height information will be used to determine the extent to which obesity affects the relationship between SNAP participation and food security.

All respondents will be informed that they can decline to answer any question that they do not wish to answer, with no negative consequences for not responding.

A.12. Provide estimates of the hour burden of the collection of information. Indicate the number of respondents, frequency of response, annual hour burden, and an explanation of how the burden was estimated.

Table A.1 shows sample sizes and estimated burden and cost for each component of the data collection and overall.

Table A.1. Estimated Total Annual Hour Burden and Annualized Costs by Respondent Type

Respondent Type	Instrument/s	Estimated Number of Respondentsa	Number of Responses per Respondent	Total Annual Responses	Estimated Average Number of Hours per Response	Estimated Total Burden	Estimated Cost of Burdenb
Pretest	Telephone survey (English)	6	1	6	1	6	$43.50
	Telephone survey (Spanish)	6	1	6	1	6	$43.50
	In-depth interview guide (English)	5	1	5	1	5	$36.25
	In-depth interview guide (Spanish)	1	1	1	1	1	$7.25
New Entrants	Baseline only	3,572	1	3,572	0.50	1,786.0	$12,948.50
	Baseline and follow-up	4,001	2	8,002	0.50	4,001.0	$29,007.25
	Baseline and in-depth interview	45	2	90	1.00c	90.0	$652.50
	Attemped Interview	2,282	1	2,282	0.08	183.6	$1,331.10
Current SNAP Participants	Baseline only	3,957	1	3,957	0.50	1,978.5	$14,344.13
	Baseline and in-depth interview	45	2	90	1.00b	90.0	$652.50
	Attempted Interview	1,488	1	1,488	0.08	119.0	$862.75
	Total	15,408		19,499		8,266.1	$59,798.73

^a Assumes 10 percent of the full sample (1,100 of new entrants and 610 of current SNAP participants) will not be contacted due to invalid or incomplete contact information.

^b Assumes Federal minimum wage rate ($7.25 per hour)

^c Average of 1.5 hours for the in-depth interview and 0.5 hours for the baseline interview.

A.13. Provide estimates of the total annual cost burden to respondents or record keepers resulting from the collection of information, (do not include the cost of any hour burden shown in items 12 and 14). The cost estimates should be split into two components: (a) a total capital and start-up cost component annualized over its expected useful life; and (b) a total operation and maintenance and purchase of services component.

There are no direct monetary costs to respondents and no other costs except for their time to participate in the study.

A.14. Provide estimates of annualized cost to the Federal government. Provide a description of the method used to estimate cost and any other expense that would not have been incurred without this collection of information.

The three-year contract cost to the federal government for the study design, data collection, data analysis, and report preparation is $3,999,606, over 3 years. In addition, Federal staff time to direct the work is estimated at roughly 250 hours per year for a GS-15 Supervisory Program Analyst ($61.28 per hour -- $15,320) and 500 hours per year for a GS-13 Social Science Research Analyst ($42.66 per hour -- $21,330) The annualized cost of data collection is thus $1,369,852.

A.15. Explain the reasons for any program changes or adjustments reported in Items 13 or 14 of the OMB Form 83-1.

This is a new information collection effort that will add 8,266.1 burden hours to the FNS inventory as a result of program changes.

A.16. For collections of information whose results are planned to be published, outline plans for tabulation and publication.

The contractor will analyze the household survey data collected as well as data from the CPS-FSS and prepare a report and a briefing for FNS. The report will present findings from descriptive and multivariate analyses of the impact of SNAP participation on the food security of participant households (using the household telephone survey data) and the impact of the April 2009 ARRA SNAP benefit increase on this relationship (using the CPS-FSS). A second report will be based on the information collected in the in-depth interviews. That report will attempt to develop tentative hypotheses about the causes and consequences of food insecurity. In this section we present the analysis plans for each data source and the corresponding project schedule.

1. Household Telephone Survey Data Analysis

In the analysis of the household survey data, we will estimate the impact of SNAP participation on household food security using two samples: a cross-sectional sample, in which outcomes for new entrant households are compared with those for households that have participated for six months; and a longitudinal sample, in which outcomes for the new entrant households are compared with outcomes for those same households six months later. The analytical methods have been selected to overcome challenges specific to each design. For example, the cross-sectional design may suffer from selection bias if households just entering the program are systematically different from those participating for six months. The longitudinal design may suffer from a time confound if changes such as increases in the unemployment rate occur between the baseline and follow-up interviews. In presenting the plans for each analysis, the contractor will address each of these threats to obtaining unbiased estimates of the impact of SNAP participation on food insecurity.

Descriptive analysis. The analysis of impact estimates for both designs will begin with descriptive assessments of food insecurity rates for the groups of new entrant and six-month participant households. We will statistically compare the percentage of households that are food insecure across the two groups (Table A.2). In separate tables, we will also compare the percentage of households that are food insecure with low food security across groups and the percentage of households that are food insecure with very low food security across groups. We will make each of these comparisons using the full samples of SNAP participant households, and we will compare food security estimates for policy-relevant subgroups, including those based on gender, race and ethnicity, age of the head of the household, and the household’s composition and monthly income.

Multivariate analysis—cross-sectional sample. Although comparing food insecurity rates across groups using a tabular analysis provides valuable information about differences in the outcome measure across groups, it is important to perform multivariate regression analyses to account for compositional differences across groups that might bias the impact estimates. In particular, many characteristics might be correlated with both a household’s continuing to participate in the SNAP program through six months and its food insecurity. We will use a logistic regression model that relates the probability of a household’s being food insecure to a variable indicating whether the household has participated in SNAP for six to seven months and to a set of household and state characteristics. The household characteristics will include the household head’s gender, race and ethnicity, age, current employment status, and marital status; household income, composition, and region of residence; and a variable indicating whether the household has participated previously in SNAP. The state characteristics will include economic measures related to the state unemployment rate and wage distribution. We will also include state policy variables that may affect households’ continued participation in the program, such as re-certification periods, simplified income reporting, and broad-based expanded categorical eligibility.

Table A.2. Percentage of Households That Are Food Insecure, by Length of SNAP Participation

	Cross-Sectional Sample				Longitudinal Sample
	Six-Month Participants	New Entrants	Difference		New Entrants at Six Months	New Entrants	Difference
All Households
Gender of Household Head
Male
Female
Race and Ethnicity of Household Head
Non-Hispanic white
Non-Hispanic black
Non-Hispanic other
Hispanic
Age of Household Head
17 and younger
18-24
25-49
50-64
65 and older
Employment Status of Household Head
Employed full time
Employed part time
Unemployed
Out of the labor force
Monthly Income as a Percentage of Poverty
Less than 50%
50% to 100%
More than 100%
Households with Children
Single adult
Multiple adults
Households with Elderly
Elderly living alone
Elderly living with others
Households with persons with disabilities

Source: SNAP Food Security Household Survey 2011.

*Significantly different from zero at the .05 level, two-tailed test.

**Significantly different from zero at the .01 level, two-tailed test.

The key finding from this analysis will be the estimate of the impact of participating in the SNAP program for six to seven months on the likelihood of being food insecure. Our intuitive expectation is that, because we are accounting for differences in household-level characteristics, the effect of selection bias on the impact estimate should be lessened, if not eliminated, relative to the estimate derived in the descriptive analysis. We will determine whether the impact estimate is statistically significant using a two-tailed t-test and standard errors of the regression coefficients that account for the survey’s multi-stage stratified sampling design.

We will present the results of this analysis in several ways, using detailed tables and summary tables. First, we will present a table with the regression coefficients and standard errors (Table A.3) to help the reader understand what variables are used in the regression and how these results translate into the subsequent set of summary tables. Next, we will present regression-adjusted tables of impact estimates that resemble the descriptive tables presented earlier (Table A.4). This table compares the rates of food insecurity across groups after accounting or adjusting for compositional differences across groups. Because we will use logistic regression analysis, the procedure for obtaining the regression-adjusted estimates consists of estimating the regression, using the regression coefficients and variable values for each household in the sample to obtain a predicted probability of being food insecure, and averaging the predicted probabilities to obtain the adjusted (predicted) rate of food insecurity in the sample. By performing these steps assuming all sample members are new entrants and then repeating the procedure assuming all sample members are six-month participants, we obtain two averaged values. The difference between these values is the regression-adjusted estimate of the impact of SNAP on food insecurity.

Table A.3. Regression Coefficients of the Effects of SNAP Participation and Household Characteristics on a Household’s Likelihood of being Food Insecure

		Coefficient			Standard Error
SNAP Participation
Gender of Household Head (male is referent group)
Female
Race and Ethnicity of Household Head (non-Hispanic white is referent group)
Non-Hispanic black
Non-Hispanic other
Hispanic
Age of Household Head (less than 17 is referent group)
18-24
25-49
50-64
65 and older
Employment Status of Household Head (unemployed is referent group)
Employed full time
Employed part time
Out of the labor force
Monthly Income as a Percentage of Poverty (less than 50% is referent group)
50% to 100%
More than 100%
Household Contains Children (referent group is no children)
Household Contains Elderly (referent group is no elderly)
Household Size
State Average Wage
State Unemployment Rate
Region of Residence (western region is referent group)
Northeast
Mid-Atlantic
Midwest
Southeast
Southwest
Mountain Plains
SNAP Policies (referent group is not having a policy)
Broad-based categorical eligibility
Simplified reporting
Positive outreach expenditures

Source: SNAP Food Security Household Survey 2011.

*Significantly different from zero at the .05 level, two-tailed test.

**Significantly different from zero at the .01 level, two-tailed test.

Table A.4. Regression-Adjusted Percentages of Households That Are Food Insecure, by Length of SNAP Participation

	Six-Month Participants			New Entrants		Difference
All Households
Gender of Household Head
Male
Female
Race and Ethnicity of Household Head
Non-Hispanic white
Non-Hispanic black
Non-Hispanic other
Hispanic
Age of Household Head
Less than 17
18-24
25-49
50-64
65 and older
Employment Status of Household Head
Employed full time
Employed part time
Unemployed
Out of the labor force
Monthly Income as a Percentage of Poverty
Less than 50%
50% to 100%
More than 100%
Households with Children
Single adult
Multiple adults
Households with Elderly
Elderly living alone
Elderly living with others
Households with persons with disabilities

Source: SNAP Food Security Household Survey 2011.

Note: We estimated the percentages by first evaluating the prediction equation using sample values of all variables and then averaging the likelihood of being food insecure.

*Significantly different from zero at the .05 level, two-tailed test.

**Significantly different from zero at the .01 level, two-tailed test.

Multivariate analysis—longitudinal sample. The multivariate analysis for the longitudinal sample will be similar to that presented for the cross-sectional sample. That is, we will estimate a logistic regression model that relates the probability of a household being food insecure to a variable indicating whether the household has participated for six to seven months as well as a set of household and state characteristics. The longitudinal impact analysis will differ from the cross-sectional analysis, however, in an important way. There will be two observations per household in the analysis file, one corresponding to the baseline interview and the other to the follow up interview. We will use this feature of the data to eliminate selection bias by including household-level fixed effects in the regression model. Because these variables account for the effect of time-invariant characteristics on the probability of being food insecure, the estimate of the impact of SNAP on food security will not reflect differences in characteristics such as gender of the household head or his or her education (assuming it does not change in the six month period). Standard errors will be adjusted to account for the correlation between the two observations from a given household.

The explanatory variables in the longitudinal analysis will be specified in the regression as levels rather first differences. That is, the household’s amount of income at the baseline and follow-up interview will be included in the regression, rather than the change in income over time. If specification tests indicate that there is value in interacting the time period with a particular household or state characteristic, then we will allow the effect of the variable on the outcome measure to differ across the two periods.

Subgroup analysis. We will obtain impact estimates for key subgroups from multivariate analyses. Since the sample size we chose is large enough to accommodate the analysis of subgroups, we will re-estimate the logistic regression equations for key subgroups and obtain the regression-adjusted impact estimates. For example, we will estimate the equation for households with and without children and obtain the regression-adjusted values (see Table A.5 for an example from the cross-sectional sample). While the regression in Table A.3 presents the impact of SNAP on food security while controlling for differences in characteristics across households, regressions like the one in Table A.5 allow the effect of SNAP on food security to differ by these characteristics. To ensure an adequate level of statistical power in these analyses, we will restrict these analyses to subgroups with sufficiently large sample sizes.

Sampling weights. Sampling weights will be constructed to correct for differences in household selection probabilities and propensities to respond. Using these weights, a weighted distribution of responding households will match the distribution of SNAP participant households in the frame from which the sample was drawn. Sampling weights will be constructed to correct for differences in probability of selection of households and households’ propensity to respond. These weights will restore the distribution of the responding sample to the same proportions as the frame of SNAP participant households from which it was drawn. Without the weights, some groups in the study population would be overrepresented and others underrepresented. Thus, analysis of unweighted data could lead to seriously biased estimates.

Different sets of weights will be constructed for the cross-sectional and longitudinal analyses described above. Each basic weight will be the product of several factors:

• The inverse of the (cumulative) probability of selection to correct for differential chances of selection.⁴

• The inverse of the cooperation rate within a state-level selection stratum if a state does not participate in the study.

• The inverse of the response rate within a response cell to correct for nonresponse at the household level.

• A post-stratification factor within strata of states so that the sum of the weights will equal our best estimate of the population of SNAP participant households at each point of participation (new entrant and at six months).

Table A.5. Regression-Adjusted Percentages of Households That Are Food Insecure, by Length of SNAP Participation and by Whether Household Has Children

	Households with Children					Households Without Children
	Six-Month Participants	New Entrants		Difference		Six-Month Participants	New Entrants		Difference
All Households
Gender of Household Head
Male
Female
Race and Ethnicity of Household Head
Non-Hispanic white
Non-Hispanic black
Non-Hispanic other
Hispanic
Age of Household Head
Less than 17
18-24
25-49
50-64
65 and older
Employment Status of Household Head
Employed full time
Employed part time
Unemployed
Out of the labor force
Monthly Income as a Percentage of Poverty
Less than 50%
50% to 100%
More than 100%
Households with Elderly
Elderly living alone
Elderly living with other adults or children
Households with persons with disabilities

Source: SNAP Food Security Household Survey 2011.

Note: We estimated the percentages by first evaluating the prediction equation using sample values of all variables and then averaging the likelihood of being food insecure.

*Significantly different from zero at the .05 level, two-tailed test.

**Significantly different from zero at the .01 level, two-tailed test.

We will examine the distribution of the weights to detect outliers (unusually large weights) and, if any are found, will trim the weights. The trimmed weights will then be post-stratified again. These sampling weights will be used for both the descriptive and multivariate analyses described above.

Synthesizing Results from Two Designs. For both all households and selected subgroups, we will compare the two sets of impact estimates from the cross-sectional and longitudinal analyses to determine whether the estimates have similar implications. To complement these comparisons, we will also compare the rates of food insecurity (and food insecurity with low and very low food security) among the six-month participant households in the comparison-group design and the longitudinal design to determine whether changes in external factors or the program structure of SNAP might have contributed to differences in impact estimates under each design. This is also a means through which to account for the effects of seasonal changes in food insecurity on measuring outcomes in the longitudinal sample.

2. Current Population Survey Analysis

As a complement to the analysis of the household survey data, we will analyze data from the CPS-FSS collected in December 2008, 2009, and 2010 to investigate these relationships further, with particular attention to the effects of ARRA. It should be noted that this CPS-based analysis does not involve the new data collection covered by this OMB Clearance Document. However, a summary of this analysis helps provide a full context for our plans for the household survey data.

The further research questions of this analysis are as follows:

1. What was the effect of the April 2009 benefit increase on food insecurity rates and food expenditures for SNAP households, and did this effect differ by key household characteristics, including length of SNAP participation spell and benefit amount (relative to the maximum allotment for households of the same size)?

2. In 2008, 2009, and 2010, what was the effect of SNAP participation on household food insecurity rates and food expenditures, and did this effect change from 2008 to 2010?

We will answer question 1 by comparing outcomes (food insecurity rates, food expenditures, and so on) for SNAP households in 2008 with those same outcomes for SNAP households in 2009. We will answer question 2 by comparing outcomes for SNAP households in 2008 with those same outcomes for non-SNAP households in 2008. We will make the same comparisons between SNAP households and non-SNAP households in 2009 and between SNAP households and non-SNAP households in 2010. We will then compare the 2008, 2009, and 2010 estimates to each other.

The 2008, 2009, and 2010 CPS-FSS files can be used either as a set of repeating cross-sections of SNAP participants or, if a subset of each file is used, as a longitudinal sample with interviews of the same households at two points in time.⁵ We plan to use both approaches to examine the questions listed earlier.

Descriptive analysis. The analysis of impact estimates for both the cross-sectional and longitudinal designs will begin with a descriptive assessment of food insecurity rates and food expenditures. To address the first research question, we will statistically compare outcome measures, such as the percentage of households that are food insecure or the average food expenditures across SNAP households before and after the April 2009 benefit increase using the 2008 and 2009 samples. Table A.6 shows how we will display results for the outcome of food insecurity rates. Similar tables will display results for average food expenditures and other outcomes (including the percentage of SNAP households that are food insecure with low food security and the percentage of SNAP households that are food insecure with very low food security). We will make each of these comparisons using the full sample of SNAP participant households as well as policy-relevant subgroups, such as those listed in Table A.6.

Table A.6. Percentage of SNAP Households That Are Food Insecure Before and After the 2009 ARRA Benefit Increase, by CPS-FSS Sample

	Cross-Sectional CPS Sample				Longitudinal CPS Sample
	Before Benefit Increase	After Benefit Increase	Difference		Before Benefit Increase	After Benefit Increase	Difference
All Households
Length of Current SNAP Participation Spell
1 month
3 months
6 months
12 months
Benefit Amount Relative to Maximum Allotment (maximum)
Less than 1/3 of max
Between 1/3 and 2/3 of max
Greater than 2/3 of max
Gender of Household Head
Male
Female
Race and Ethnicity of Household Head
Non-Hispanic white
Non-Hispanic black
Non-Hispanic other
Hispanic
Age of Household Head
17 and younger
18–24
25–49
50–64
65 and older
Employment Status of Household Head
Employed full time
Employed part time
Unemployed
Out of the labor force
Monthly Income as a Percentage of Poverty
Less than 50%
50% to 100%
More than 100%
Households with Children
Single-adult household with children
Multiple-adult household with children
Households with Elderly
Elderly living alone
Elderly living with other adults or children
Households with persons with disabilities

Source: Current Population Survey’s Food Security Supplement from 2008 and 2009.

*Significantly different from zero at the .05 level, two-tailed test.

ARRA = American Recovery and Reinvestment Act of 2009; CPS-FSS = Current Population Survey’s Food Security Supplement; SNAP = Supplemental Nutrition Assistance Program.

While the first research question examines how food security might have changed after the implementation of the ARRA SNAP provisions in April 2009, the second characterizes the relationship between SNAP participation and food security more generally during the 2008–2010

period. To begin answering this second question, we will statistically compare food insecurity rates and average food expenditures across SNAP and non-SNAP households for each CPS supplement (2008, 2009, and 2010) separately. We will define each set of households in various ways. For example, we will compare households that have not participated in SNAP in the previous 12 months with households that have participated at any time in the same period. We will also compare households according to whether they currently participate in the program.

Table A.7 shows how we will display results for the outcome of food insecurity rates using the 2008 and 2009 data. We will construct a similar table using the 2009 and 2010 data. Table A.7 presents the results of statistical tests for (1) the difference in food insecurity rates between SNAP and non-SNAP households in a given year (2008 or 2009), and (2) the difference in this difference across years (2008–2009). The first test answers the first part of research question 2 asking what was the effect of SNAP participation on household food insecurity rates in 2008 and 2009. The second test answers the second part of question 2 on whether this effect changed from 2008 to 2009 (in other words, whether the April 2009 benefit increase changed the way in which food insecurity rates varied with SNAP participation).

Multivariate analysis—cross-sectional sample. The descriptive analyses for research questions 1 and 2 suffer from the following biases. For research question 1, any observed changes from 2008 to 2010 in the food insecurity rates (and other outcomes) for SNAP households might not be due to the April 2009 benefit increase, but rather to simultaneous economic changes, such as increased food prices or unemployment rates. The multivariate analyses can lessen the bias from

Table A.7. Percentage of Households That Are Food Insecure, by SNAP Participation Status: Changes from 2008 to 2009

Cross-Sectional CPS Sample

Ever Participated

Current Participation Status

2008 Supplement

2009 Supplement

2008–2009

2008 Supplement

2009 Supplement

2008–2009

[A] Non-SNAP Households (did not participate in past 12 months)

[B] SNAP Households (participated at any time in past 12 months)

[C] Diff. [A–B]

[D] Non-SNAP Households (did not participate in past 12 months)

[E] SNAP Households (participated at any time in past 12 months)

[F] Diff. [D–E]

[G] Diff. [C–F]

[H] Non-SNAP Households (are not currently participating)

[I] SNAP Households (are currently participating)

[J] Diff. [H–I]

[K] Non-SNAP Households (are not currently participating)

[L] SNAP Households (are currently participating)

[M] Diff. [K–L]

[N] Diff. [J–M]

All Households

Gender of Household Head

Male

Female

Race and Ethnicity of Household Head

Non-Hispanic White

Non-Hispanic Black

Non-Hispanic Other

Hispanic

Other Subgroups of Interest

Longitudinal CPS Sample

All Households

Gender of Household Head

Male

Female

Race and Ethnicity of Household Head

Non-Hispanic White

Non-Hispanic Black

Non-Hispanic Other

Hispanic

Other Subgroups of Interest

Source: Current Population Survey’s Food Security Supplement from 2008 and 2009.

*Significantly different from zero at the .05 level, two-tailed test.

these time confounds by controlling for such environmental changes. For research question 2, SNAP and non-SNAP households potentially differ in other ways that might affect food security status—most notably, in the characteristics that determine whether or not they are participating, such as differences in income and asset levels and in willingness to seek government assistance—so the differences in food insecurity rates between these two types of households provide a biased estimate of the effect of SNAP participation on food security status. Controlling for other household characteristics (in the cross-sectional analysis) and creating a matched sample of households that move from non-SNAP to SNAP over the course of a year (in the longitudinal analysis) will potentially reduce this selection bias.

Multivariate analysis—longitudinal sample. As mentioned previously, the descriptive analysis used to answer research question 2 might suffer from selection bias because households that are more food insecure are also more likely to participate in SNAP. In other words, key observable and unobservable differences occur between SNAP and non-SNAP households for which cross-sectional analysis cannot control. We can use analysis of longitudinal samples to minimize this bias by identifying a set of households that move from not participating to participating in SNAP during the course of a year. Following Nord and Golla (2009), we will estimate the effect of SNAP participation on the likelihood of a household being food insecure (based on a 30-day food security measure) in each year by comparing this likelihood among households in the months just before SNAP enrollment to its incidence among those same households in the months just after enrollment.

3. In-Depth Interview Analysis

Much of the analysis of the in-depth interview data will take place in group meetings attended by all of the interviewers and by the contractor’s senior project staff, some of whom have extensive experience in this kind of analytical work. The interviewers will have their interviewing notes with them, and the senior staff, including consultants Kathy Edin and Sara Greene, as well as key Mathematica staff, including James Mabli and Dawn Nelson, will lead them through structured discussions of their findings, organized by research questions. Past experience in similar work suggests that through this process common themes will emerge, which can be verified during subsequent parts of the discussion. Eventually the findings that emerge from this process will be described and documented by the senior staff who have guided this analysis work, and incorporated into the reporting for the study.

4. Project Schedule

The planned schedule for this study, assuming receipt of OMB clearance by early July 2011, is as follows:

Activity	Schedule
Select sample of states	November 2010
Develop data collection instrument	October 2010 through February 2011
Conduct pre-test of instrument	February 2011 through March 2011
Produce memo on pre-test findings	March 2011
Train data collectors	June 2011 to August 2011
Produce memo summarizing results of training	August 2011
Conduct baseline data collection	August 2011 through November 2011
Conduct follow-up data collection	February 2012 through May 2012
Conduct in-depth interviews	November 2011 through December 2011
Produce report on in-depth interview findings	February 2012
Produce memo containing household survey table set	August 2012
Conduct briefing on findings of household survey at FNS	February 2013
Produce report on findings of household survey and CPS	April 2013
Produce policy brief on findings of household survey and CPS	April 2013

A.17. If seeking approval to not display the expiration date for OMB approval of the information collection, explain the reasons that display would be inappropriate.FNS plans to display the OMB approval number and expiration date of the information collection on all instruments.

A.18. Explain each exception to the certification statement identified in Item 19 "Certification for Paperwork Reduction Act."

There are no exceptions to the certification statement.

A.19. Customer Service Center

This collection is not related to the Customer Service Center.

This page has been left blank for double-sided copying.

References

Bollinger, Christopher and Martin H. David "Estimation with Response Error and Non-response: Food Stamp Participation in the SIPP'' Journal of Business and Economic Statistics, April, 2001, Vol. 19, no. 2, pp. 129-141.

Cody, Scott, Laura Castner, James Mabli, and Julie Sykes. “Dynamics of Food Stamp Program Participation: 2001-2003.” Final Report Submitted to Food and Nutrition Service, USDA, Washington DC, November 2007.

Cohen, Barbara, Mark Nord, Robert Lerner, James Parry, and Kenneth Yang. “Household Food Security in the United States, 1998 and 1999: Technical Report.” E-FAN-02-010, prepared by IQ Solutions and USDA, Economic Research Service, 2002. Available at: http://www.ers.usda.gov/ publications/efan02010/.

Hamilton, W.L., J.T. Cook, W.W. Thompson, L.F. Buron, E.A. Frongillo, Jr., C.M. Olson, and C.A. Wehler. “Household Food Security in the United States in 1995: Technical Report.” Prepared for USDA, Food and Consumer Service. 1997. Available at: http://www.fns.usda.gov/oane/menu/ published/foodsecurity/tech_rpt.pdf/.

Kennedy, Eileen T., James C. Ohls, Steven Carlson, and Katherine Fleming. “The Healthy Eating Index: Design and Applications.” Journal of the American Dietetic Association, vol. 95, no. 10, October 1995, pp. 1103-1108.

Kish, Leslie. Survey Sampling. New York: John Wiley & Sons, Inc., 1965.

Kovac, Martha D., and Jason Markesich, “Pre-Paid Vs. Promised Incentives: Which Works Better for a Telephone Survey of Low-Income Respondents?” Paper presented at the Annual Conference of the American Association of Public Opinion Research, Nashville, TN, May 2003.

Nemeth, P. “Zero-Two-Five: Which Pre-Pay Amount Gets You More for Your Money?” Paper presented at the American Association for Public Opinion Research Annual Conference, Hollywood, FL, May 2009.

Nord, Mark, Alisha Coleman-Jensen, Margaret Andrews, and Steven Carlson. Household Food Security in the United States, 2009. ERR-108, U.S. Dept. of Agriculture, Econ. Res. Serv. November 2010.

Nord, Mark, and Anne Maria Golla. Does SNAP Decrease Food Insecurity? Untangling the Self-Selection Effect. Economic Research Report No. 85, U.S. Dept. of Agriculture, Economic Research Service. Oct. 2009.

Ohls, James, Larry Radbill, and Allen Schirm. “Household Food Security in the United States, 1995-1997: Technical Issues and Statistical Report.” Prepared by Mathematica Policy Research, Inc., for USDA, Food and Nutrition Service. 2001. Available at: www.fns.usda.gov/oane/MENU/ Published/FoodSecurity/FoodSecurityTech.pdf/.

Singer, Eleanor, J. Van Hoewyk, and M.P. Maher. “Experiments with Incentives in Telephone Surveys.” Public Opinion Quarterly, vol. 64, 2000, pp. 171-188.

The American Association for Public Opinion Research. 2009. Standard Definitions: Final Dispositions of Case Codes and Outcome Rates for Surveys. 6th edition. AAPOR.

U.S. Department of Agriculture, Food and Nutrition Service, Office of Research and Analysis, Characteristics of Supplemental Nutrition Assistance Program Households: Fiscal Year 2009, by Joshua Leftin, Andrew Gothro, and Esa Eslami. Project Officer, Jenny Genser. Alexandria, VA: 2010.

Improving public well-being by conducting high-quality, objective research and surveys

Princeton, NJ ■ Ann Arbor, MI ■ Cambridge, MA ■ Chicago, IL ■ Oakland, CA ■ Washington, DC

Mathematica® is a registered trademark of Mathematica Policy Research

www.mathematica-mpr.com

1 See Appendixes B A and CB for the English and Spanish versions of the structured telephone survey.

2 See Appendix D C for the in-depth interview guide.

3 Mathematica will work with FNS to obtain waivers so the monetary incentive does not affect participants’ benefits.

4 The sampling design is described in more detail in section B.1.

5 The latter is possible because of the 4-8-4 sampling structure of the survey, with respondents interviewed for 4 consecutive months, not interviewed for 8 consecutive months, and interviewed for another 4 months. Recent studies have exploited this feature of the survey by linking the files across consecutive years, such as December 2008, 2009, and 2010, to obtain a pseudo-panel data file (Nord and Golla 2009). The main drawback of this approach is that using the matched records results in smaller sample sizes.