081512 FMCS OMB Supporting Statement B

SNAP Participants who used their Electronic Benefit Transfer (EBT) card at a Farmers Market (FM) during the last 12 months;
SNAP Participants who did not use their EBT card at a Farmers Market during the last 12 months.

Details on the respondent universe for each of the above groups are provided below in the context of sampling methods.

B.1.2. Sampling Methods

This study will implement a two stage sampling process. In Stage 1 we will select a nationally representative sample of farmer’s markets with EBT transactions greater than $1,000. In Stage 2 we will select a random sample of SNAP participants who are FM EBT users and FM EBT nonusers from all EBT transactions that were made at any authorized retailer within a predefined area surrounding the selected farmer’s markets.

For stage 1, we plan to use the sample drawn for the FNS study: Nutrition Assistance in Farmers Markets: Understanding Current Operations (OMB #: 0584-0564; Expiration Date November 30, 2014). This study is a survey of FM managers and includes a sample of FMs and direct marketing farmers (DMF) with EBT transactions in the past year (August 2010 through July 2011). We expect to have approximately 469 completed interviews from FMs. We will identify retailers from an extract from the USDA’s Store Tracking and Redemption System (STARS II). This system stores information on retailers that participate in the SNAP program. The STARS II extract will include only those farmer’s markets that redeemed at least $1,000 in SNAP benefits in the 2011 market season; we expect this to be about 249 markets. We do not intend to include markets that will be newly authorized in 2012 because we would like to study the shopping behavior of participants who had certain level of exposure to a FM. To identify SNAP participants, we will take extracts from the USDA’s Anti-Fraud Locator using EBT Retailer Transactions (ALERT). The ALERT system records every transaction where a SNAP/EBT card is used. We will analyze the ALERT data on participants from July 2011 through June 2012. The ALERT dataset includes all EBT transaction records and contains the following data items: store ID, store state, terminal ID, household account number, card account number, transaction date, transaction time, transaction amount, transaction sign, transaction type, response code, available balance prior to transaction, and amount, if the response code indicates that the transaction was accepted. Data items that will be requested from STARS II for the period July 2010 through June 2011 will include: store identifier, store name, address, telephone, business type, number of registers, average gross monthly sales, and monthly food stamp redemption by coupon and by EBT.

An important selection criterion is that the farmers market must have had at least $1,000 of SNAP redemptions in the 2011 FM season. After appropriate adjustments for nonresponse, the sample of 249 eligible FMs produces a nationally representative sample of the SNAP authorized (non-Direct Marketing Farmers) farmer’s markets with more than $1,000 in annual redemptions. Given the limited field period, and the time associated with obtaining participant names and contact information for SNAP recipients from the State/local agencies, we will select a representative subsample of 50-70 FMs, which we project will be located in 10-15 States. The FMs will be selected with probabilities proportionate to a composite measure of size that takes into account the number of EBT transactions at the FM as well as the number of EBT transactions at other stores within the catchment area of the corresponding FM. With proper weighting, the resulting sample of FMs will provide an unbiased representation of the eligible farmer’s markets that redeemed SNAP benefits in 2011 with annual EBT transactions greater than $1,000.

In Stage 2, the areas surrounding the FMs will be defined by a set of ZIP Codes that are in close proximity to each market; we define this to be the “catchment area” for each of the selected farmer’s market. The ALERT data will contain transaction information for each EBT card that was used at retailers in those ZIP Codes during the period from July, 2011 through June, 2012. Using a list of all EBT card numbers that were used in the catchment areas surrounding the targeted FMs during that period, we will request the contact information and individual and household characteristics from the SNAP State office or local agency for each EBT card number. We will work with States to identify their preferred method of data transfer. Westat follows information technology (IT) and systems security policies and best practices, and monitors conformance to these policies and best practices throughout the organization. From the list provided by the State and local agencies we will identify all SNAP clients who have shopped at an authorized retailer in the catchment area; this will serve as the sampling frame for both FM EBT users and nonusers within the catchment areas of the farmer’s markets selected at Stage 1. Once the sampling frame is created, we will randomly select FM EBT users and nonusers for the client survey to meet the analytic objectives of the study. The samples will be stratified to ensure that adequate numbers of households with children under 5 years and seniors over 60 years of age are selected.

Focus Group with SNAP Recipients

We will conduct 12 in-person focus group sessions with English-speaking and Spanish-speaking SNAP recipients to develop a further understanding of shopping decisions of users and non-users of SNAP benefits. The knowledge gained from a focus group will allow for a more in-depth understanding of attitudes than the survey can provide. Westat will identify a purposive sample of three farmer’s markets, selected to provide insights into issues that affect SNAP participants’ FM shopping decisions. Once the three farmer’s market locations are chosen, we will exclude SNAP clients from the sampling frame who were selected for the SNAP Client Survey. SNAP clients who are interested in participating will be asked a series of screening questions to determine if they are eligible for one of the four groups (e.g., English- or Spanish-language; shopper or non-shopper) and are able to attend at the specified time and location. For each scheduled discussion, we will recruit up to 12 participants who meet the criteria. Appendix E1-E12 presents the procedures for recruiting participants and conducting the focus groups.

B.1.3. Response Rates and Sample Size

We will start with initial sample of 4,625 participants and we are expecting completed surveys from a sample of approximately 3,700 SNAP participants (shown in Table 1). This assumes an 80 percent response rate, which will be achieved by multiple mailings and telephone follow-up for non-response. We also assume a 3-month attrition rate of 11% for SNAP participants to estimate the sample draw needed to achieve the target sample size.

Table 1. Sample size

	FM EBT Users (1,700 Completes)		FM EBT Nonusers (2,000 Completes)	TOTAL
	With Incentives	W/O Incentives	FM EBT Nonusers (2,000 Completes)	TOTAL
Sample draw	1,405	983	2,809	5,197
Attrition (11%)	155	108	309	572
Sample	1,250	875	2,500	4,625
Completes (80%)	1,000	700	2,000	3,700

It is important to emphasize that the SNAP participants are considered to be a hard to reach population for studies such as this one. Frequently, the data from State agencies have incomplete, missing, or incorrect information on mailing address and telephone numbers. In addition, SNAP participants are more likely to be using temporary (pre-paid) telephones and they may change numbers more often than the population average. Considering all the information on difficulties of reaching this population, to ensure 3,700 complete interviews, we will also have a reserve sample of 1,295 SNAP participants that will be released if it appears that we will not achieve an 80% response rate with the original release.

B.2. Describe the Procedures for the Collection of Information including:

Statistical methodology for stratification and sample selection,
Estimation procedure,
Degree of accuracy needed for the purpose described in the justification,
Unusual problems requiring specialized sampling procedures, and
Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

B.2.1. Statistical Methodology for Stratification and Sample Selection

As noted above, we will use the Nutrition Assistance in Farmers’ Markets: Understanding Current Operations study (FM Ops) responding farmer’s markets as the Stage 1 sampling frame, which will be explicitly stratified by the census region, urban/rural status, and incentive program status. Within each explicit stratum, we will implicitly stratify the FMs by market size (measured by the redemption value of EBT transactions). This will be accomplished by sorting the FM frame by market size and selecting the samples using systematic sampling with a random start. Implicit stratification will ensure good representation of markets across the sort variables and is likely to improve the precision of survey estimates. In Stage 2, once the frame of participants is created it will be divided into two strata: SNAP participants who redeemed their SNAP benefits at a targeted FM and SNAP participants who did not redeem their benefits at a targeted FM, but who did redeem benefits at a food store in the catchment area.

B.2.2. Estimation Procedures

Sample Weights

Sample weights will be developed for SNAP participants responding to the survey for estimation purposes. Each sampled SNAP participant will be assigned a base weight, therefore, reflecting each SNAP participant’s chance of selection. To compensate for unit nonresponse, a standard approach is to calculate adjustment factors within selected weighting classes, where the adjustment factor is the ratio of the sum of the weights (using the base weights) of both respondents and nonrespondents to the sum of the weights for respondents alone in each weighting class. These factors will be used to inflate the base weights, so that estimates from responding SNAP participants can be used to make appropriate inference to the SNAP participant population. The weighting classes may be the same as the original sampling strata, or they may be defined based on other relevant characteristics that are available for both responding and nonresponding units. Analyses of differential nonresponses (e.g., using a software package such as CHAID) are planned which will help to identify classes with important differences in response propensity, a critical criterion in the formation of useful nonresponse adjustment classes. The sum of the weights of the respondents, after the adjustment, will equal the sum of the weights of the respondents and nonrespondents before the adjustment.

Sampling Error Estimation

When a survey is conducted using a complex sample design, the design must be taken explicitly into account to produce unbiased estimates and standard errors for these estimates. This is accomplished by dividing the complete sample into a number of subsamples known as replicates so that each replicate sample, when properly weighted, will provide appropriate estimates of population characteristics of interest. In general, replicate samples are formed to mirror the original sampling of primary sampling units. In this study, replicate weights using the jackknife methodology will be developed as part of the weighting process to calculate sampling errors of survey estimates and to conduct statistical significance tests of survey findings. WesVar Software for Complex Survey Analysis will be used with replicate weights to take the sample design into account when calculating point estimates, correlation, and regression coefficients and their associated standard errors. A series of jackknife replicate weights will be created and attached to each data record for variance estimation purposes. These replicate weights will then be imported into WesVar to calculate appropriate standard errors for survey-based estimates.

B.2.3. Degree of Precision Needed for the Purpose Described in the Justification

Margin of Error

The SNAP participant sample has been designed to estimate response proportions in each of the four subgroups: (1) EBT users of FMs with incentives, (2) EBT users of FMs without incentives, (3) all EBT users of FMs with and without incentives combined, and (4) FM EBT nonusers who shop within the selected FM catchment areas, within a precision requirement of five-percentage point margin of error at a 95 percent confidence level.

Table 2 provides estimates of the levels of precision to be expected under the proposed design for subgroup sample sizes ranging from 700 (sample size for FM EBT users without incentives) to 2,000 (sample size for FM EBT nonusers), and design effects ranging from 1.10 to 1.50. The design effects reflect the increase in sampling variance due to the variation in sampling rates resulting from disproportionate stratified sampling and clustering of sample SNAP participants. The expected level precision in Table 2 were estimated with effective sample sizes under different design effect assumptions, where the effective sample size is defined as the ratio of the actual sample size to the overall design effect associated with a given estimate. Thus, for a subgroup sample size of 1,000 responding EBT users of FMs with incentives, the margin of error on an estimated 50-percent characteristic (P = 50%) can be expected to range from ±3.3 percent to ±3.8 percent at the 95 percent confidence level depending on the magnitude of the design effect. For a subgroup consisting of 1,700 respondents (all FM EBT users), the corresponding margins of error are expected to range from ±2.5 percent to ±2.9 percent depending on the design effect.

Table 2. Expected 95 percent confidence bounds by subgroup sample size, for selected design effects (DEFF) and prevalences (P)

Subgroup	Sample size*	DEFF = 1.10			DEFF = 1.25			DEFF = 1.50
Subgroup	Sample size*	P = 20%	P = 33%	P = 50%	P = 20%	P = 33%	P = 50%	P = 20%	P = 33%	P = 50%
FM EBT users, w/o incentives	700	±3.1%	±3.7%	±3.9%	±3.3%	±3.9%	±4.1%	±3.6%	±4.3%	±4.5%
FM EBT users, with incentives	1,000	±2.6%	±3.1%	±3.3%	±2.8%	±3.3%	±3.5%	±3.0%	±3.6%	±3.8%
All FM EBT users	1,700	±2.0%	±2.3%	±2.5%	±2.1%	±2.5%	±2.7%	±2.3%	±2.7%	±2.9%
FM EBT nonusers	2,000	±1.8%	±2.2%	±2.3%	±2.0%	±2.3%	±2.5%	±2.1%	±2.5%	±2.7%

* Sample size of completes per subgroup.

Minimum Detectable Difference

In addition to calculating descriptive statistics of the SNAP participant population, comparisons will be made between subgroups. Table 3 provides estimates of the minimum detectable differences (MDD) for two pairs of comparisons of proportions under the proposed design for subgroup sample sizes. The first row in the table shows the MDDs for the comparison between FM EBT users with incentives (n=1,000) and FM EBT users without incentives (n=700); and the second row in the table shows the MDDs for the comparison between FM EBT users (n=1,700) and FM EBT nonusers (n=2,000). The detectable differences are for a one-sided test with significance level of 0.05 and power of 0.80. These values show, for example, that if Deff=1.25, for p₀ = 20% observed in the subgroup of FM EBT users, a value of p₁ = 23.8% or larger observed in the subgroup of FM EBT nonusers would be considered statistically significant. Alternatively, for a value of p₀ = 50% in the FM EBT users subgroup, a value of 54.6% or larger in the FM EBT nonusers subgroup would be considered statistically significant.

**Table 3. Minimum detectable difference* between subgroups for selected design effects (DEFF) and underlying prevalences (P)**

Subgroup sample size 1*	Subgroup sample size 2*	DEFF = 1.10			DEFF = 1.25			DEFF = 1.50
Subgroup sample size 1*	Subgroup sample size 2*	P = 20%	P = 33%	P = 50%	P = 20%	P = 33%	P = 50%	P = 20%	P = 33%	P = 50%
FM EBT users, w/o incentives (n=700)	FM EBT users, with incentives (n=1,000)	5.4%	6.2%	6.4%	5.5%	6.3%	6.6%	6.3%	7.2%	7.5%
FM EBT users (n=1,700)	FM EBT nonusers (n=2,000)	3.6%	4.1%	4.3%	3.8%	4.4%	4.6%	4.2%	4.8%	5.0%

* Calculations assume a one-sided test with a significance level of 0.05 and power of 0.80.

B.2.4. Unusual Problems Requiring Specialized Sampling Procedures

No specialized sampling procedures are involved.

B.2.5. Any use of Periodic (less frequent than annual) Data Collection Cycles to Reduce Burden

The study design requires a one-time data collection from respondents. All data collection activities will occur within a 3 month period.

B.3. Describe Methods to Maximize Response Rates and to Deal with Issues of Non-Response.

The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

By explaining the importance and potential usefulness of the study findings in the introductory letters from FNS, and by implementing a series of follow-up reminders with a final attempt to complete the survey by telephone, we expect to achieve an overall survey response rate of 80%. Specific procedures to maximize response rates include:

A cover letter from USDA/FNS (Appendix A1-A2).
A prepaid incentive included with introductory letter and survey (Appendix B).
A promissory incentive discussed in introductory letter (Appendix A1-A2).
Two Interactive Voice Response (IVR) calls to respondents who have not completed the survey after two weeks of the first and second survey mailing (Appendix D5-D6).
Two data collection modes (mail or telephone) for participants’ convenience
Telephone follow-up interview for non-responder (Appendix D1-D2).
Make up to 9 unsuccessful call attempts to a number without reaching someone before considering whether to treat the case as “unable to contact.”
Implement refusal conversion efforts for first-time refusals and use interviewers who are skilled at refusal conversion and will not unduly pressure the respondent (Appendix D3-D4).
Provide a toll-free number for respondents to call to verify the study’s legitimacy or to ask other questions about the study.
Implement standardized training for telephone data collectors. The interviewer training will focus on basic skills of telephone interviewing, use of CATI platforms for interviews.

B.4. Describe any Test of Procedures or Methods to be Undertaken.

Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The survey instrument has gone under two rounds of cognitive testing with 24 SNAP participants, in which no more than 9 respondents were asked the same question. In the first round, Westat cognitively tested the survey with both users and nonusers for question flow and understandability. Users are defined as those who have been identified to have used their SNAP/EBT card at a specified farmer’s market. Non-users are defined as those who have not been identified to have used their SNAP/EBT card at a specified farmer’s market. Based on the findings from round one, the survey was revised to address concerns regarding the wording of questions or instructions that proved difficult for participants to comprehend. Then, a second round of cognitive testing used the revised survey instruments to ensure that the survey can be administered by participants with relative ease. The cognitive testing was conducted with both English and Spanish speaking participants. Each cognitive interview took approximately 60 minutes and the participants were given $50 as a token of appreciation.

B.5. Provide the Name and Telephone Number of Individuals Consulted on Statistical Aspects of the Design and the Name of the Agency, Unit, Contractor(s), Grantee(s), or Other Person(s) Who Will Actually Collect and/or Analyze the Information for the Agency.

The information has been reviewed by Leanne Tang, 202-720-6957, of the Methods Branch of USDA’s National Agricultural Statistics Service (NASS), with special reference to the statistical procedures. See the NASS comments in Appendix F3.

Additionally, FNS consulted with the following contractors on statistical aspects of the design. Westat will collect and analyze the information on behalf of FNS:

Name	Affiliation	Telephone Number	e-mail
Mustafa Karakus	Project Director, Westat	301-294-2874	[email protected]
Thomas Bosworth	Senior Study Director, Westat	301-610-5542	[email protected]
Cynthia Robins	Senior Study Director, Westat	301-738-5424	[email protected]
John Burke	Senior Study Director, Westat	301-294-2057	[email protected]

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
Author	Sujata Dixit-Joshi
File Modified	0000-00-00
File Created	2021-01-30

081512 FMCS OMB Supporting Statement B