Revised SNAP Online_Supporting Statement Part B 5-10-19 PRAO edits

SUPPORTING STATEMENT - PART B for

OMB Control Number 0584-[NEW]:

Evaluation of Technology Modernization for SNAP Benefit Redemption through Online Transactions for the USDA Food and Nutrition Service

Eric Williams

Social Science Research Analyst

Office of Policy Support – SNAP Research and Analysis Division

USDA, Food and Nutrition Service

3101 Park Center Drive

Alexandria, Virginia 22302

Table of Contents

B.1 Respondent Universe and Sampling Methods 1

B.2 Procedures for the Collection of Information 3

B.3 Methods to Maximize the Response Rates and to Deal with Nonresponse 10

B.4 Test of Procedures or Methods to be Undertaken 10

B.5 Individuals Consulted on Statistical Aspects & Individuals Collecting and/or Analyzing Data 11

B.1 Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The U.S. Department of Agriculture’s Food and Nutrition Service (FNS), selected seven retailers to participate in the SNAP Online Purchasing Pilot in 2017.¹ These retailers will route online Supplemental Nutrition Assistance Program (SNAP) purchases through a designated third-party processor that has the only FNS-approved system for secure on-line entry of personal identification numbers (PINs) for Electronic Benefits Transfer (EBT) transactions. Data will be collected from all organizations involved with the pilots, including the seven retailers, the web service providers used by two retailers, the two EBT host processors, and State SNAP agencies in the eight States in which the pilot will be implemented. There is no statistical sampling required for the evaluation; we anticipate 100 percent participation from all respondents. Exhibit B.1.1 presents the sampling methods and target response rates for each of the respondent subgroups. Exhibit B.2.1 provides more detail on the operational data.

Exhibit B.1.1: SNAP Online Purchasing Evaluation Respondents

Respondent Universe	Respondent Subgroup	Sampling Methods and Target Response Rates	Data Collection Strategies
Pilot retailers	Personnel, responding in their official capacity for the retailer, who handle the fulfillment, shipping, and delivery of SNAP EBT customer orders	Retailers have agreed to participate in the evaluation as a condition of participating in the pilot. Therefore, the team expects a 100 percent response rate.	Semi-structured telephone or in-person interviews
Pilot retailers	Customer service managers, responding in their official capacity for the retailer.	Retailers have agreed to participate in the evaluation as a condition of participating in the pilot. Therefore, the team expects a 100 percent response rate.	Semi-structured telephone or in-person interviews
Pilot retailers	IT personnel responding in their official capacity for the retailer.	Retailers have agreed to participate in the evaluation as a condition of participating in the pilot. Therefore, the team expects a 100 percent response rate.	Semi-structured telephone or in-person interviews Retailer detailed transaction data files Retailer summary transaction data files Retailer store files
Web service providers	Web service provider personnel, responding in their official capacity for the retailer.	Web service providers are under contract to retailers that have agreed to participate in the evaluation as a condition of participating in the pilot. Therefore, the team expects a 100 percent response rate.	Semi-structured telephone or in-person interviews Retailer detailed transaction data files Retailer summary transaction data files Retailer store files
Pilot retailers	Project manager, responding in their official capacity for the retailer.	Retailers have agreed to participate in the evaluation as a condition of participating in the pilot. Therefore, the team expects a 100 percent response rate.	Semi-structured telephone or in-person interviews
Pilot retailers	Other retailer personnel, responding in their official capacity for the retailer.	Retailers have agreed to participate in the evaluation as a condition of participating in the pilot. Therefore, the team expects a 100 percent response rate.	Semi-structured telephone or in-person interviews
Pilot retailers	Personnel who completed the pilot application, responding in their official capacity for the retailer.	Retailers have agreed to participate in the evaluation as a condition of participating in the pilot. Therefore, the team expects a 100 percent response rate.	Semi-structured telephone or in-person interviews
EBT host processors	Management/staff, responding in their official capacity for the EBT host processor.	EBT host processors are contractually required to participate in the evaluation. Therefore the team expects a 100 percent response rate.	Semi-structured telephone or in-person interviews
Third-party processor	Management/staff, responding in their official capacity for the third party processor.	The third-party processor has agreed to participate in the evaluation. Therefore the team expects a 100 percent response rate.	Semi-structured telephone or in-person interviews
State SNAP agency EBT coordinators	EBT coordinator, responding in their official capacity for their organization.	State SNAP agencies have agreed to participate in the evaluation as a condition of participating in the pilot. Therefore, the team expects a 100 percent response rate.	Semi-structured telephone interviews State SNAP case record files

B.2 Procedures for the Collection of Information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection,

• Estimation procedure,

• Degree of accuracy needed for the purpose described in the justification,

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

This section describes the data collection procedures for the SNAP Online Evaluation’s key informant interviews, then operational data. It also describes the analyses using this data. Collectively, our data collection period will run from the time the evaluation team receives OMB clearance to summer 2020.

Key Informant Interviews

Interviews with key informants will be conducted by phone or in person (Appendices B-E). In-person interviews are preferred with small pilot retailers and respondents with a key role in multiple pilots (e.g., third-party processors and EBT host processor representatives). Some questions may be more easily addressed through observation.

While Exhibit B.1.1 provides a general description of respondents, identifying the correct respondent is critical. The evaluation team will identify key contacts and the most appropriate respondents during initial telephone calls with pilot retailers and FNS. After identifying respondents, the team will contact them by email to request their participation and will follow up by telephone to further explain the evaluation and set times for interview (either by telephone or in-person depending on size of the pilot retailer, respondent role, etc.). FNS staff will be invited to accompany the evaluation team for in-person interviews.

For retailers operating the pilot in more than one State, the evaluation will conduct two rounds of interviews with each pilot retailer to allow for staggered starts across States (Appendix D). For retailers operating the pilot in only one State, the evaluation will conduct one round of interviews with the pilot retailer (Appendix D). The team will conduct one round of interviews with each State SNAP Agency EBT Coordinator (Appendix E). While conducting these interviews early in the pilot would allow the team to gather information while start-up is fresh in the minds of key informants, waiting until the pilot is operating in a “steady state” would allow the team to capture more of the pilot experience. Therefore, the team prefers later interviews as they should allow the team to answer the full array of research questions.

The evaluation team will conduct three rounds of interviews with the EBT host processors (Appendix B) and the third-party processor (Appendix C) across the overall pilot implementation period. Given the staggered start of the pilots across States noted below, this will allow us to capture information about each pilot while start-up is fresh in the minds of these key informants.

There will be a staggered start to the retailer pilots, with each retailer having its own schedule for implementation. Each retailer will operate its pilot in a specified area of a single State for at least one month prior to expanding to the full pilot area of that State and to additional States. The first interviews scheduled are likely to be those related to pilots that start earlier. The evaluation team will review documentation or reference materials produced by the pilots or other key informants prior to receiving OMB clearance.

The evaluation team will conduct a half-day training specifically for interview team members two weeks prior to the start of data collection. The interview team leads will develop data collector training materials which will include: an overview of the evaluation including its purpose, goals, and research questions; an overview of EBT functionality and stakeholders’ relationship to the pilot; and a review of data collection instruments, interview questions, and privacy protections. Practice interviews will be part of the training. FNS staff will be invited to attend the site visit training.

Each interview will be conducted by an experienced trained interviewer familiar with EBT and SNAP operations. Where feasible, a junior team member will take notes. Otherwise, interviews will be recorded and transcribed. The recording of the interview is voluntary and respondents will be told they can decline recording their responses with no penalities. After each round of interviews, teams will conduct telephone follow-up, as necessary, to clarify responses. The teams will use standardized semi-structured discussion guides for key informant interviews that will be tailored to focus on relevant questions, with appropriate follow-up probing and elaboration.

Operational Data

The evaluation team will collect four types of operational data (see Exhibit B.2.1). These data include pilot retailer detailed transaction files (when available; Appendix F), pilot retailer summary transaction files (Appendix G), and SNAP case records (Appendix I). Data will be provided by retailers and State SNAP agencies. In addition, the team will collect lists of stores with location data and store number from each retailer (Appendix H).

The evaluation team will collect monthly data from all retailers and participating States for twelve months from the start of online redemption. Collectively, our data collection period will run from the time the evaluation team receives OMB clearance through 12 months after each retailer is operational (estimated to be late summer 2020) (see Part A for project timeline). For pilot months occurring before OMB clearance, collection of operational data will be retrospective. During the planning phase of the evaluation, the evaluation team will share draft data specifications with retailers and State SNAP agencies and obtain their feedback on the feasibility of providing the specified data.

Exhibit B.2.1. Summary of Operational Data

File Name	File Description	Data Source	Frequency
Retailer Detailed Transaction Data Files	Detailed individual-level transaction data on SNAP online purchases	Pilot Retailers/ Web Service Providers	Monthly
Retailer Summary Transaction Data Files	Aggregated metrics on SNAP online purchases	Pilot Retailers/ Web Service Providers	Monthly
Retailer Store Files	Lists of stores with location data and FNS store authorization number	Pilot Retailers/ Web Service Providers	Monthly
State SNAP Case Record Files	Administrative data on SNAP participants	States	Monthly

The evaluation team will obtain summary transaction files directly from pilot retailers for the first twelve months of the pilot (Appendix G). The optional retailer detailed transaction file (Appendix F), if provided, will include individual-level, identifiable transaction data, that is not available through the SNAP Online Purchasing Pilot (OPP) EBT transaction data that will be provided by FNS (i.e., a dataset that includes some but not all of the online transaction data and SNAP participant data needed for the evaluation). The retailer detailed transaction data will include EBT and non-EBT transactions related to online orders made by SNAP customers, and the data elements will include payment method and amount, billing address, timing of orders and pickup/delivery, and reason for refunds. It may be necessary to obtain separate data on purchases and shipments, if the retailer fulfills purchases made with the same payment method in multiple shipments. The evaluation team will also request a monthly summary transaction file (i.e., summary data) that includes specific overall metrics needed to answer research questions. Finally, the evaluation team will obtain monthly lists of pilot stores with location data and FNS authorization number (Appendix H) .

Pilot retailers will transfer detailed and summary transaction data files directly to the evaluation team via a secure data transfer application. The evaluation team will establish Data Use Agreements (DUAs) with retailers on the timing and specific data elements the team expects to receive, and the geographic area to be covered. The data will be requested for the first twelve months of each retailer’s pilot. The files will include all transaction-level information detailed in the data requests for retailer detailed and summary transaction data files (Appendices F and G).

In addition to the transaction data, the evaluation team will request SNAP case record files from States (Appendix I). For each State participating in the pilot, the team will request State SNAP case record files for all active SNAP households. Data for all active SNAP participants are required to calculate the likelihood that SNAP household made online purchases and to estimate models of the household characteristics associated with online SNAP purchasing. The evaluation team will request State SNAP case record files for each month of the FNS online transaction data collection period to ensure that the team has demographic data for all SNAP households (including size of household, number of children, presence of elderly or disabled participants, benefit size, and benefit issuance date) during the pilot period. Requests will be limited to the time period in which a pilot is operating in a particular State. The evaluation team will request both household and individual level characteristics. The FNS OPP transaction data file will include address information for SNAP households from State SNAP case records. States generally provide timely updates of household addresses to the EBT processors. However, in case the household addresses in the FNS OPP transaction data file may not be current, the evaluation team will also request that the States provide household addresses in the case record files.

State SNAP Agencies will transfer SNAP case record files to the evaluation team. The evaluation team will collect monthly data from all States for twelve months from the start of online redemption in their States. For pilot months occurring before OMB clearance, collection of State SNAP case record files will be retrospective. The evaluation team will limit collection of SNAP case record files to the States in the pilot since only these States’ participants will be permitted to make online purchases.

Data Analysis

The evaluation team will conduct implementation and integrity analyses for each of the seven pilots and cross-site analyses. The team will use qualitative analysis software to code and analyze key informant interviews. The team will construct monthly household-level analysis files for all SNAP households who ever redeemed their benefits through online ordering and will use statistical software packages such as SAS or Stata to analyze quantitative data through descriptive statistics, geographic information systems (GIS) methods, and regression (see Exhibit B.2.2).

Exhibit B.2.2. Overview of Analysis Objectives, Topics, Data Sources and Methods

Objective	Data Sources	Topic	Analysis Methods
Implementation	Key informant interviews	Implementation process	Qualitative coding and synthesis
Implementation	Key informant interviews	Level of effort for implementation and expansion	Qualitative coding and synthesis Cost projections
Implementation and Integrity	Key informant interviews	Security protections	Qualitative coding and synthesis
Implementation	Online transaction data, SNAP case record data	Take-up/characteristics of SNAP customers making online purchases	Descriptive and regression analysis
Integrity	Online transaction data, SNAP retailer location data	Relationships of locations of online customers, pickup/delivery sites, and SNAP retailers	GIS methods
Integrity	Online transaction data, summary reports of customer profiles and activity	Patterns of customer use and problems (delays, refunds, orders not picked up)	Descriptive statistics

For analyses of data for the individual pilots, the principal comparisons will be between the characteristics of participating SNAP households (those that take up the online purchasing option) and non-participating SNAP households. For these comparisons, the team will uses test of differences in means or proportions, depending on the relevant characteristics, taking into account the number of observations in each group. In addition, the team will report t-test results for coefficients in the regression model of SNAP households participating in the online purchasing pilots. For the summary of results, the team will report tests of differences in key household-level averages among the pilots. The team will select key household-level average measures to compare and test for differences so as to avoid producing an excessive number of statistical tests that would likely result in numerous false positives (i.e., test results that are significant by chance). Due to the small number of pilots, it will not be appropriate to test the significance of differences in pilot-level summary measures across pilots.

B.3 Methods to Maximize the Response Rates and to Deal with Nonresponse

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield "reliable" data that can be generalized to the universe studied.

It is important to note that the retailers agreed to participate in the evaluation as part of their agreement with FNS. Retailers who cannot provide the data will not be allowed to continue in the pilot. EBT host processors are also required to participate in the evaluation. In addition, the evaluation team believes that the specialized nature of the respondent groups – retailer personnel, personnel from web service providers, third-party processor, EBT host processor, and State SNAP agency – who are deeply involved in pilot implementation and are interested in supporting FNS’ efforts to design and implement EBT online purchasing, will increase their propensity to respond.

B.4 Test of Procedures or Methods to be Undertaken

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The key informant interview discussion guides (Appendices B-E) were developed by evaluation team members and reviewed by Federal staff. The instruments were reviewed by two individuals with expertise in EBT and e-commerce operations. Their feedback was used to modify a limited number of questions and probes to improve the flow and clarity of questions.

The lists of data elements for the retailer detailed transaction (Appendix F) and summary data (Appendix G) files were originally based on specifications in the Request for Volunteers for the pilot. The evaluation team revised this list to incorporate all data elements needed from the retailers. The revised data elements were reviewed by Federal staff and representatives from four retailers to ensure that the data would be readily available. Their feedback was used to modify a number of data elements.

B.5 Individuals Consulted on Statistical Aspects & Individuals Collecting and/or Analyzing Data

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

The information will be collected and analyzed by Abt Associates, Altarum Institute, and Novo Dia. The sampling procedures were developed by Christopher Logan (telephone: 617-349-2821) and Ayesha Enver (telephone: 301-3470-5139) of Abt Associates. The sampling plans were reviewed internally by Jacob Klerman (telephone: 617-520-2613), Senior Fellow at Abt Associates. Eric Williams (telephone: 703-305-2640) from the Food and Nutrition Service also reviewed this supporting statement and provided comments that have been incorporated. Sarah Goodale (e-mail: [email protected]) from the National Agricultural Statistics Service also reviewed this supporting statement.

1 Originally nine retailers were selected, but one opted not to participate in the pilot and one withdrew.