Rosemarie Downer
U.S. Department of Agriculture
Food and Nutrition Service
3101 Park Center Dr.
Alexandria, VA 22302
703-305-2129
703-305-2576
Mathematica Policy Research
600 Maryland Avenue, SW, Suite 550
Washington, DC 20024
CONTENTS
Part Page
B. Collections of information employing statistical methods 1
B.1 Respondent Universe and Sampling Methods 1
B.2 Procedures for the Collection of Information 3
B.3 Methods to Maximize Response Rates and to Deal with Non-Response 6
B.4 Test of Procedures or Methods to be Undertaken 7
B.5 Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data 7
APPENDIX a: Legal citation
APPENDIX B: Federal Register Notice
APPENDIX C: public comments
APPENDIX D: paperwork reduction act certification
APPENDIX e: respondent invitation letters
APPENDIX f: project description
APPENDIX g: frequently asked questions
APPENDIX h: Request for performance data
APPENDIX i: survey access instructions
APPENDIX j: electronic survey
APPENDIX k: survey reminder call script
APPENDIX l: Phone interview scheduling call script
APPENDIX m: performance data reminder call
APPENDIX n: snap staff phone interview protocol
APPENDIX o: snap partner phone interview protocol
APPENDIX p: site visit interview topics
B.1. Respondent Universe and Sampling Methods
The universe for this new data collection is all 50 states and the District of Columbia. Mathematica, a contractor to FNS, will conduct all data collection and analysis for this project, will solicit responses to the survey and telephone interviews from state and local SNAP staff, will contact SNAP partners for telephone interviews, and will conclude with site visits to 10 states. The anticipated response rate is 83 percent (Table B.1).
Table B.1 Sampling and Response Rate Among Respondents
Respondent Type |
Number of Offices (Universe) |
Sampling Method |
Respondents Contacted |
Respondents Participating |
State SNAP Offices |
51 |
Convenience sampling |
51 |
178 |
County and local SNAP offices |
Universe unknown, plan to contact 150 |
Convenience and random sampling |
400 |
332 |
SNAP partners |
Universe unknown, plan to contact 203 |
Convenience sampling |
335 |
244 |
Totals |
786 |
754 |
||
Expected response rate |
83% |
|||
FNS’s goal is to understand the breadth of SNAP modernization performance measurement approaches that are underway and specific details of how states measure their performance with respect to SNAP modernization initiatives. With a focus on comprehensively listing performance measures, rather than generalizing to the population from a sample, discussions with each state as well as a convenience sample of local offices will yield the most accurate results. The survey and telephone interviews will focus on local offices and partners that other respondents believe to be actively participating in SNAP modernization performance measurement. A random sample of some local offices will collect data from respondents whose performance measurement activities may not be well known to other respondents. Random sampling of partners is not possible because SNAP offices will not have a comprehensive sample frame (partner list) that can serve as a sampling frame. The National Agricultural Statistics Service (NASS) has reviewed and approved the proposed data collection approach.
Data collection will begin with surveys and telephone interviews in each state. “Leading states,” with the most modernization initiatives and performance measurement efforts, will be the focus of local office and partner telephone interviews, and 10 states will be asked to participate in site visits.
From among the identified states, Mathematica will select 150 county and local offices for the survey and phone interviews through purposive and random selection. The purposive selection (of 112 offices, or 75 percent) will be based on information received through phone interviews with the state about modernization leaders within the state. The random selection of 38 county and local SNAP offices ensures that FNS will obtain information from respondents that may be using innovative measures and standards without the full knowledge of the state administrators, and provides an approach for selecting offices if state-level staff have no specific suggestions. Mathematica will collect data from twice as many offices in county-administered states as in state-administered states, based on the assumption that county-administered programs have more within-state variation in SNAP administration.
Mathematica will also interview 203 SNAP partner organizations, selecting them in consultation with FNS and based on recommendations from the state and local SNAP offices, which will help ensure a high response rate. The final list of partner contacts will contain a primary set of partners and a secondary set to contact if primary partners do not participate.
Data collection will conclude with visits to 10 states, including 5 that are known to be implementing many modernization initiatives and likely have developed a wide range of performance measures, and 5 that will be identified after we have conducted at least half of the state interviews to gather more information about states’ performance measurement activities. Site visits will collect more detail on how states collect and use performance measures and standards to monitor their modernization initiatives. Visits will focus on one locality that the state believes will best demonstrate performance measurement for its modernization initiatives.
To ensure a high response rate, FNS and Mathematica have considered the design, administration, sample selection, and promotion of this collection. For example, the survey is designed to allow states and localities to select any number of staff to participate, permitting states to identify those with the best knowledge of performance measurement among their own staff. And, while the survey will be provided electronically, Mathematica will also accept responses by email, fax, or mail. To select a sample that maximizes responses, Mathematica will contact local SNAP offices and partners based on recommendations from SNAP staff. Therefore, rather than selecting among all partners and offices, the collection will focus on only those offices and partners with the most active role in performance measurement. Finally, FNS has promoted awareness of the survey among state SNAP officials by mentioning it at the State SNAP Director’s Conference, and Mathematica will further promote the collection by gaining buy-in and participation from regional and state FNS staff, who will convey the legitimacy and importance of the project to their respective states and localities. Advance materials (Appendices E through I) and reminder phone calls from trained interviewers will also encourage responses.
B.2. Procedures for the Collection of Information
For this project, Mathematica will focus efforts to collect survey, telephone, and in-person interview data on states, localities, and partners that are actively participating in SNAP modernization and performance measurement. As described in section B.1, purposive sampling will govern most of the selection of respondents. The sole exception is random selection of approximately 38 local offices from a list of all local offices in focal states. Data gathered from the sample of respondents in this collection will not be used to calculate parameter estimates for the population. To assure the data quality while completing numerous telephone and on-site interviews, Mathematica will train and provide oversight for interviewers. The training will cover: recruiting respondents; collecting responses to the survey before the interviews; telephone interviewing with a tailored, structured protocol; consistently recording data; and refusal conversion. A task leader and two quality assurance monitors will assist interviewers with collecting and recording data, providing additional guidance and re-training as necessary.
Mathematica will begin data collection by sending advance materials about the project along with the survey and requests for 12 months of aggregated performance data. Each SNAP office in the sample will receive an electronic, downloadable survey in an Access Runtime executable file that they may choose to access on a secure file exchange website or on a CD-Rom. The survey requires no special software, includes a user-friendly survey interface that uses skip patterns, and prevents access to underlying data to ensure that questions or formatting are not mistakenly altered. To return the survey, respondents may upload the completed survey to the file exchange site, submit it via email, or submit printed responses via fax or mail. Mathematica interviewers will call respondents (Appendix K) within one to two weeks of releasing the survey. On this call, interviewers will remind respondents of the survey data request, explain that their responses will help tailor telephone interview questions, answer any questions about the advance materials, and arrange a mutually-agreeable interview time within a two-month window. Trained Mathematica staff will then interview SNAP state and local staff by telephone to obtain greater detail about existing SNAP modernization and performance measurement efforts (clarifying responses to survey questions as necessary).
After contacting SNAP staff, Mathematica will solicit telephone interview responses from SNAP partner organizations recommended by state and local SNAP staff. Partner contacts will use a similar procedure (beginning slightly later) of sending advance materials and a request for 12 months of aggregated performance data on measures they already have in place. Participating partners may be engaged in a range of activities, from housing a computer where clients can apply or recertify for benefits to developing and hosting an information system that SNAP agencies use for eligibility determination or performance measurement. Therefore, questions during the telephone interview with partners will focus on the initiative(s) that they partner with the state, county, or local office to do (see Appendix O).
Finally, Mathematica will visit, beginning within one month of the start of the telephone interviews, one site in each of 10 states for additional interviews and observations of how staff collect and use performance data. States selected for site visits will be those whose performance measurement activities are numerous and mature enough that on-site observation and more time to question staff about procedures would benefit the project, and visits will occur after survey and telephone interview responses to provide information to target our on-site questions. Each visit, conducted by two Mathematica staff over two-and-a-half days, will include a series of one-hour executive style interviews with state, county, and local SNAP staff and relevant program partners. Though topics for the interviews will be refined based on background information and input from FNS staff, the key topics will seek to: describe the performance measures and standards in detail; discuss how they are implemented, the results, and how they are used; and assess the pros and cons of applying the measures and standards across states (see Appendix P). The exact number and roles of staff to be interviewed interview will vary by site. Mathematica will develop a schedule for each visit that best accommodates the various locations of site staff members and their schedules.
B.3. Methods to Maximize Response Rates and to Deal with Non-Response
The expected response rate is 84%, which was calculated by dividing the expected number of participating respondents by the total number of respondents contacted.
The first of three strategies to maximize responses and handle non-responses is to emphasize the importance of the study to respondents. FNS and Mathematica have promoted awareness of the survey by mentioning it at the State SNAP Director’s Conference and by gaining buy-in and participation from regional FNS staff, who will convey the legitimacy and importance of the project to the states in their respective regions. Advance materials (Appendices E through I) will impart the goals and importance of the collection and will provide a contact number for respondents to verify the legitimacy of the study and ask questions. Calls to remind respondents to submit their survey responses and aggregated data, and to schedule telephone interviews, will convey this same message of study goals, importance, and legitimacy (Appendices K-M).
Second, Mathematica will mostly use referrals to identify the local SNAP offices and partners best suited to respond to the collection, which will help locate respondents who are more inclined to participate. We’re contacting partners suggested to us by others, which maximizes response rate better than Urban. To build the list of partners from the interviews with the county and local agency staff, Mathematica interviewers will explicitly describe characteristics of target organizations and the type of information needed from this collection. Supporting SNAP staff as they consider measures that partners may collect will ensure that staff name partners that have a substantial enough role as to suggest that they may have some measures in place.
Finally, we have minimized the respondent burden by creating an electronic survey that we will offer in multiple formats to meet respondents’ needs, and all subsequent interviews will use the survey data to identify and ask only the most important, relevant questions that the survey did not acquire, or that require clarification. Each SNAP office in the sample will receive an electronic, downloadable survey in an Access Runtime executable file that will be available on a secure file exchange website or via a CD-Rom (according to their preference). The survey requires no special software, includes a user-friendly survey interface that uses skip patterns, and prevents access to underlying data to ensure that questions or formatting are not mistakenly altered. And, state and local offices can select who should respond to which parts of the survey, allowing them to identify staff with knowledge that is best suited to the task.
B.4. Test of Procedures or Methods to be Undertaken
Mathematica pretested the survey and telephone interview instruments with Pennsylvania in December 2009, collecting comments on the clarity and usability of these instruments. Mathematica will modify the instruments based on this feedback. The state has numerous modernization initiatives and therefore tested a wide range of the survey interview questions.
B.5. Individuals Consulted on Statistical Aspects and Individuals Collecting and/or Analyzing Data
Mathematica, FNS, and NASS staff contributed to planning for the survey, sample frame, statistical methodology and other aspects of the collection (Table B.5).
Table B.5 Individuals Consulted on data collection or analysis
Mathematica Staff (Contractor) |
Laura Castner: Senior Researcher and Project Director |
202-484-3282 |
William Borden :Senior Fellow and Technical Reviewer |
609-275-2321 |
|
Scott Cody :Associate Director of Research, Technical Reviewer |
202-484-4523 |
|
Rhoda Cohen: Senior Survey Researcher and Technical Reviewer |
609-275-2324 |
|
Other Staff |
Rosemarie Downer: FNS Project Officer |
703-305-2129 |
David Hancock, NASS |
202-690-2388 |
| File Type | application/msword |
| Author | Lisa Walls |
| Last Modified By | rgreene |
| File Modified | 2010-05-11 |
| File Created | 2010-05-11 |