Part B (2-15-11)

Part B: Collection of Information Employing Statistical Methods Mathematica Policy Research

Part B: Collection of Information Employing Statistical Methods

B1. Respondent Universe and Sampling Methods

Describe (including a numerical estimate) the potential respondent universe and any sampling or other respondent selection method to be used. Data on the number of entities (e.g., establishments, State and local government units, households, or persons) in the universe covered by the collection and in the corresponding sample are to be provided in tabular form for the universe as a whole and for each of the strata in the proposed sample. Indicate expected response rates for the collection as a whole. If the collection had been conducted previously, include the actual response rate achieved during the last collection.

The universe for this data collection is the six states that have been selected and have agreed to participate in the comprehensive case studies.

The initial state selection process employed a modernization index designed to identify states with the most advanced modernization initiatives. Florida, Georgia, Massachusetts, Utah, Washington, and Wisconsin were selected and have agreed to participate in this study. Because the states have already agreed to participate, a 100 percent response rate is expected for the in-person interviews in each of the states (Table B.1.1).

Table B.1.1 Sampling and Response Rate Among SNAP Staff and Partners

Respondent Type	Number of Offices (Universe)	Sampling Method	Respondents Contacted	Respondents Participating
State SNAP Office staff	6	Convenience sampling	60	60
County and Local SNAP Office Staff	Universe unknown	Convenience sampling	259a	259
SNAP Partners and Vendors	Universe unknown	Convenience sampling	74	74
Total			399	399
Expected response rate				100%

^aThis number consists of district/county SNAP staff (84), call center/centralized operations staff (21), and local office SNAP staff (154).

In order to gain meaningful, accurate insights into both the process of implementing modernization changes (documenting the new functions, roles, and responsibilities of staff and stakeholders) and the potential impacts of those changes, it is crucial to conduct interviews and observe program operations in at least four locations in each state. Visiting four sites in each state will produce a rich evidence base from which to draw conclusions about consistency of statewide implementation of modernization changes.

The four locations should reflect a mix of urban and rural environments and should capture key variations in modernization procedures and policies. To ensure the case studies yield accurate, comprehensive depictions of each state, it will be critical to thoughtfully select the local sites to visit and the respondents to interview.

Site Selection Procedures. Because state officials are to be interviewed, the state capital of each state will be one of the four sites selected. The remaining three sites will be selected based on three objectives:

1. Geographic Diversity. At least one of the four locations will represent one of the state’s largest urban areas and another will represent a rural area. The locations will also reflect the geographic diversity in the state—ensuring that more than one region of the state is visited.

2. Diversity in Modernization. Any diversity in modernization activities will be captured. The variation in modernization activities is likely to be greatest in states in which SNAP is county-administered.

3. Educated Opinions. Knowledgeable state officials will be asked to suggest sites and provide a rationale for visiting them. However, relying solely on state officials’ suggestions increases the chance of visiting only locations that are working smoothly. To minimize this risk, opinions from other knowledgeable sources will be solicited, such as (where relevant) the state’s caseworker union, a state welfare policy advocate, research staff in the state’s legislature, or national organizations that track state SNAP policies, such as the Center on Budget and Policy Priorities.

Respondent Selection Procedures. Once the sites are identified, the study team will use a tiered approach to identifying respondents, asking a point of contact at each level to help identify staff at that level, as well as a point of contact at the next lower level. The team will first ask state officials to identify key staff (those most knowledgeable about modernization) and businesses that should be interviewed at the state level. The team will then contact the director of the district or county administrative office that oversees SNAP for that location to identify local offices and CBOs that should be interviewed. Finally, we will work with the director of each local office to identify local key staff to interview. The materials that will be sent to office directors and points of contacts, such as letters, a project fact sheet, and answers to frequently asked questions, are included in Appendices G, H, and I; the scripts for introductory and reminder telephone calls are included in Appendix J and K, respectively.

Selection Methods for Focus Group Members. Four focus groups (two from group 1 and two from group 2) will be conducted in each of the six states (24 total). Table B.1.2 presents how the research design will define and identify the participants for the focus groups to meet the study objectives. A description of each subgroup follows the table, along with a discussion of the recruitment plan.

Table B.1.2 Identifying and Defining Focus Group Members

	Target Population	Identifying Focus Group Members
Group 1	Recent participants	Individuals who have received SNAP benefits for fewer than three months
	Long-term participants	Individuals who have received benefits for more than two years
Group 2	Eligible, nonparticipants	Individuals who are using a local emergency food service provider but are not enrolled in SNAP

The Group 1 focus group members will consist of both recent SNAP participants and long-term participants. The Group 2 focus group members will consist of only eligible, nonparticipants. Twenty participants will be recruited for each focus group with the intent of seating 10 participants per group.

Recent Participants. Participants with recent experience applying for and receiving SNAP benefits will provide valuable insight into client experiences with the modernization changes. These individuals will have recently applied for benefits under the new procedures for their state. For this group, individuals who applied in the previous three months will be recruited (a three-month window is used to minimize recall problems). They will participate in the same focus group as long-term participants.

Long-term Participants. Individuals who have been receiving benefits for more than two years will have first-hand experience with the modernization changes. These participants will be able to compare their experiences with current SNAP procedures with experiences in prior years. They will participate in the same focus groups as recent participants.

SNAP participants, both recent and long-term, will be identified and recruited using administrative case record data submitted by each study state. Data obtained through encrypted electronic files will be provided in two batches, one preceding each site visit. From these data, we will select a total of 400 cases – a random sample of 200 recent participants and a random sample of 200 long-term participants – in each state, within the nearest ZIP codes of the focus group site. Each sample will be sorted into a random order. Interviewers from Mathematica’s Survey Operations Center (SOC) will then call sampled clients, explain the study and its purpose, and ask them to participate in the focus group. The interviewers will proceed through each list until the target number of clients is recruited. Clients will be offered $25 to participate in the focus group and $5 for transportation to and from the focus group location. In addition to the $30 the focus group participants will receive, the parents will be offered an additional $15 for child care. All the invited SNAP clients will be informed that these incentives will not affect the value of their SNAP benefits. To help encourage participation, FNS will provide a light meal and refreshments. Those who accept the invitation for the focus group will be mailed a letter with information about the study (Appendix L) and detailed information about the time and location of the focus group. SOC interviewers will also make reminder calls (Appendix M) to participants a few days prior to the scheduled focus group to maximize attendance. Previous experience has shown that, despite these efforts, only about half of the 20 recruited participants are likely to attend, and a similar outcome is expected in this study. Therefore, we expect 10 attendees at a participant group in each state.

Nonparticipants. The second group includes low-income nonparticipants: individuals who are probably eligible for SNAP but are not participating in the program and have not done so in the previous 12 months. These respondents can directly address the barriers to program access and can provide valuable insight into the effectiveness of some modernization initiatives by discussing how likely they are to apply for benefits under the new procedures and what could be done to make SNAP more appealing to potential clients such as themselves.

Identifying non-participants who are likely to be eligible for SNAP poses more of a challenge because there is no list of eligible non-participants. Previous research, however, has demonstrated that a large proportion of people who use emergency food centers are not enrolled in SNAP. The study team will identify low-income non-participants by working with one or more large emergency-food providers in each of the sites.¹ To recruit potential non-participants for the discussion groups, a field interviewer will visit emergency food providers 10 to 14 days before the discussion group convenes. With the food providers’ approval, the interviewer will screen the providers’ users on site. They will be asked whether they are enrolled in SNAP, and if they are not, the interviewer will explain the purpose of the discussion group and ask if they would be willing to participate. The interviewer will use a short screening tool (Appendix E) to identify whether the individuals are likely to be eligible for SNAP, and then will collect contact information for those who are willing to participate in the focus group. Those who accept the invitation for the focus group will also be given a flyer (Appendix N) with detailed information about the time and location of the focus group.

Table B.1.3 presents the recruitment plan for the focus groups. The team will begin recruiting about two weeks before each group meets, aiming for 10 people. Assuming that about half the individuals who agree to participate will actually attend, the team will recruit 20 participants per group, expecting 10 to attend.

Table B.1.3. Recruitment Plan for Focus Groups in Each State

Targeted Subgroup	Samplea b	Sample Recruitedc	Focus Group Attendance Rate	Number Attendingc
Recent SNAP Participants	200	20	50%	10
Long-Term SNAP Participants	200	20	50%	10
Eligible Nonparticipants	100	60	33%	20
Total	500	100	44%d	40

^aRecent and long-term SNAP participants will participate together in the same focus group and efforts will be made to have as even a mix as possible when recruiting focus group respondents. Eligible nonparticipants include users of emergency food assistance providers.

^bThese estimates assume that about 80 percent of the sample will result in non-contact (i.e. bad phone number, mechanical answering device), ineligibility, or refusal.

^cTotal recruitment/attendance for focus groups in one state.

^dAverage attendance rate across the three groups

B2. Procedures for Collection of Information

Describe the procedures for the collection of information including:

• Statistical methodology for stratification and sample selection

• Estimation procedure

• Degree of accuracy needed for the purpose described in the justification

• Unusual problems requiring specialized sampling procedures, and

• Any use of periodic (less frequent than annual) data collection cycles to reduce burden.

This study employs three primary data collection activities: (1) in-person interviews with state and local SNAP staff and community partner and vendor staff, (2) focus groups with SNAP participants and eligible non-participants, and (3) extant data collection.

In-Person Interviews. None of the data collection activities requires a statistical methodology for sample stratification. Two researchers will conduct each semi-structured, in-person interview, typically at the respondent’s workplace. The senior researcher will lead the discussion, using the guide in Appendix C; the second researcher will primarily take notes. Some interviews will be with just one respondent; others may be with several staff members at once—interactions among the staff may provide additional information and lead to more lines of discussion.

After the interviews, the research team will prepare a site visit summary of individuals’ responses to the questions in the discussion guide. It will be used later to analyze the results of the in-person interviews and compare them to those from other data sources.

Focus Groups. An experienced moderator will lead the focus groups, which will meet in “neutral” facilities—that is, not associated with SNAP—so that the respondents feel comfortable speaking frankly. Most groups will be convened in a small conference room in a local hotel, although other venues will be explored, such as meeting rooms in community centers and libraries. Respondents will complete a form (Appendix O) to collect basic demographic information.

The focus group moderator will follow the guide in Appendix C. With the approval of all respondents, the discussion will be tape-recorded and later transcribed. Transcripts and notes will be used to analyze the results and compare them with those from other data sources.

Extant Data Collection. We will work closely with data managers from each of the six study states to most effectively collect, process, and analyze the data files that will be provided. We will clearly articulate data requirements and will assign a senior programmer to work with each state’s data manager to identify and resolve any technical problems that arise.

Ten years of monthly state case record extract data will be collected in two batches: one containing case records from July 2000 to December 2010 and the other containing case records from January to June 2011.

For application statistics and performance data, data collected in OMB No. 0584-0557 expiration date 7/2013, Performance Standards and Reporting for SNAP Modernization Initiatives study will be used first, to the extent possible, in an effort to reduce the burden on participating states. When the team has assessed the information available through that study, each state will be asked to provide the remaining necessary data.

Other relevant extant data will be collected directly from the states during site visits as well as from other sources -- such as local agencies, community partners, vendors, and any relevant reports and documentation – that are identified over the course of the study.

All data gathered from this collection will be analyzed qualitatively and will not be used to calculate parameter estimates for the population.

B3. Methods to Maximize Response Rates and to Deal with Nonresponse

Describe methods to maximize response rates and to deal with issues of non-response. The accuracy and reliability of information collected must be shown to be adequate for intended uses. For collections based on sampling, a special justification must be provided for any collection that will not yield “reliable” data that can be generalized to the universe studied.

Difficulties are not expected in securing interviews with staff members at SNAP offices or community partners and vendors. However, ensuring high participation rates for the focus group members is critical. Indeed, non-participants may be the least inclined to participate since they have the least stake in the program. The team will use several techniques used in previous studies to ensure having 10 participants in each group.

Structured Opportunities to Build Rapport. Confirmation letters (Appendix L) reiterating the purpose of the study, clearly stating the date, time, location, and directions, and addressing issues such as confidentiality will be sent to individuals who agree to attend the focus groups. Additionally, interviewers in charge of prescreening and recruiting them will be trained to build rapport during the first minutes of the initial telephone contact. Some groups will be scheduled in the early evening to make it easier for individuals to attend.

Reminder Calls. One to two days before each group, reminder calls (Appendix K) will be made to those who agreed to attend.

Respondent Incentives. Respondents will be offered a $25 incentive for participating in a focus group and will be reassured that accepting it will not affect their benefits or eligibility for SNAP or other programs. Light refreshments will also be provided. Moreover, potential focus group members will be offered a $5 transportation stipend if they express difficulty getting to the designated location and a $15 child care stipend if they express a need for it.

Special Considerations. Mathematica’s survey operations division will take into account special considerations of the target population. Because a significant portion of SNAP participants in some states are fluent in Spanish but not in English, the prescreening call will identify sites where there are large numbers of monolingual Spanish speakers, and focus groups will be held in Spanish (Appendix C)when necessary. The discussion group moderator will be bilingual, fluent in both Spanish and English.

B4. Tests of Procedures

Describe any tests of procedures or methods to be undertaken. Testing is encouraged as an effective means of refining collections of information to minimize burden and improve utility. Tests must be approved if they call for answers to identical questions from 10 or more respondents. A proposed test or set of tests may be submitted for approval separately or in combination with the main collection of information.

The interview and focus group guides for this collection will be semi-structured and, therefore, will not be tested. To make the interviews and focus groups run more efficiently, the guides will be tailored for each state so that only questions that are relevant for that state will be included. As such, the specific questions asked during interviews and focus groups will vary from state to state given that the content will be heavily dependent on the modernization initiatives implemented in each state.

B5. Individuals Consulted

Provide the name and telephone number of individuals consulted on statistical aspects of the design and the name of the agency unit, contractor(s), grantee(s), or other person(s) who will actually collect and/or analyze the information for the agency.

Mathematica staff and the FNS project officer contributed to planning for the survey and other aspects of the collection (Table B.5.1). Comments from the public and from National Agricultural Statistics Service (NASS) were also consulted.

Table B.5.1 Individuals Consulted on Data Collection or Analysis

Mathematica Staff (Contractor)
Scott Cody, Associate Director of Research and Project Director	202-484-4523
Kevin Conway, Deputy Project Director	609-750-4083
Lara Hulsey, Researcher	609-936-2778
Rhoda Cohen, Senior Survey Researcher and Technical Reviewer	609-275-2324
Laura Castner, Senior Researcher	202- 484-3282
FNS Staff
Rosemarie Downer, FNS Project Officer	703-305-2129

1 Although eligible non-participants who are using emergency food services may be systematically different from other eligible non-participants, they are different in a way that makes them particularly relevant for efforts to improve access since they demonstrate a need for food assistance.

13