APPENDIX L
SNAP PARTNER PHONE INTERVIEW PROTOCOL
INTRODUCTION
My name is [X], from Mathematica Policy Research.
As I mentioned when we set up the interview, Mathematica is conducting research to collect information from state and local SNAP offices, and their partners and contractors, to identify the methods being used to measure the implementation of SNAP modernization initiatives.
Our report will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you by name or title.
I expect our conversation will take approximately 60 minutes.
OPTIONAL IF INTERVIEWER CHOOSES TO RECORD: I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording.
IF YES: Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time.
IF NO: That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.
Before I begin, do you have any questions for me about the project or what we will be discussing today after reading the background materials I sent you?
On this call, I’ll begin by asking you some background information about the SNAP initiatives that characterize your partnership, but we will spend most of the call discussing performance measurement for those initiatives.
Public Burden Statement
Public reporting burden for this collection of information will vary by agency, but is estimated to be 60 minutes, on average, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Services, Office of Research and Analysis, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302, ATTN: PRA (0584-xxxx*).
To start, I want to gather some basic information about your organization’s activities with respect to SNAP so that I can target our conversation. I will ask for greater detail on the measures you use to track each initiative.
1.1 What functions or services do you provide as a (partner/contractor) related to the SNAP program?
1.2 In this partnership, is your main contact the state, county, or local office?
1.3 Is there a formal arrangement that governs the partnership? Is there an MOU or a contract?
PROBE IF YES
Are any of the terms of your agreement based on performance of specific functions? IF YES: Please describe which functions and what performance is expected.
PROBE IF NO
Does the SNAP agency monitor your partnership activities? IF YES: How?
1.4 Does the (state/ county) compensate you for your services? IF YES: How?
PROBE IF YES
Is the compensation dependent upon a specific service volume or an achievement of some kind? IF YES: Please explain.
FOR THE REMAINDER OF THIS SECTION, ASK ABOUT ONLY INITIATIVE(S) PARTNER REPORTS PARTICIPATING IN DURING PART I.1.
Let’s focus now on call centers.
2.1 Is there one or more than one call center in operation within the state?
2.2 What region(s) within the state does the call center(s) serve?
GO TO NEXT RELEVANT INITIATIVE OR TO SECTION II
I want to ask some questions about the process that applicants use to access and submit online applications.
3.1 Must applications be completed in a single online session or is there the option to create an application account and return to it at a later time?
3.2 IF PARTNER OPERATES ONLINE SYSTEM: What timeframe do you use to determine if an application has been abandoned?
3.3 IF PARTNER OPERATES ONLINE SYSTEM: How long does the online system store applications that have not been completed?
3.4 Do community partners have special access to the online system? IF YES: Please describe the access privileges given to community partners.
3.5 IF PARTNER OPERATES ONLINE SYSTEM: How does the information obtained through the online applications system transition to the eligibility determination system that the SNAP office uses to calculate benefits?
PROBE:
Are online applications manually entered into the determination system?
If NO: Does that mean that every piece of information on the application is automatically transferred into the eligibility determination system?
If YES: What pieces of information manually entered into the eligibility system? What information is not transferred at all?
3.6 Is your state online screening tool integrated with the online application?
PROBE IF YES:
Is it possible to use the screening tool and have the application pre-populated with some of this information?`
GO TO NEXT RELEVANT INITIATIVE OR TO SECTION II
I want to get a few details about the document imaging initiative.
4.1 Do you record the source of the documents to be imaged and, if so, how?
4.2 Do you record the types of documents received and, if so, how?
4.3 How are scanned documents stored and accessed?
PROBE:
Is the document imaging system integrated with an electronic case management system?
IF YES: Please describe the process from the point at which a document is received to the point in which it is integrated in the electronic case management system.
GO TO NEXT RELEVANT INITIATIVE OR TO SECTION II
I have a few questions about the kiosks that are available for SNAP functions.
5.1 What are they primary functions of the kiosks?
PROBE:
For example: applicant pre-screening, download online application, submit unsigned application, submit signed application, and/or submit changes to submitted application.
5.2 Where are kiosks located?
5.3 Are all applicants permitted access to the kiosks?
5.4 Do you record applications received by individual kiosk and, if so, how?
5.5 Do you record whether a case worker or community partner provides assistance at a kiosk and, if so, how?
GO TO SECTION II
ASK QUESTIONS IN THIS SECTION FOR EACH PARTNERSHIP INITIATIVE
Next, we’ll discuss each initiative in turn to get some basic information about that initiative, and then I will ask some more specific questions about measures for that initiative.
To focus our conversation, I’ll ask all of my questions about measures for one initiative before moving on to ask similar questions about the next initiative.
Let’s focus now on [INITIATIVE NAME].
Do you collect any performance measures or data for [INITIATIVE NAME]?
IF NO – Thank you. Instead, let’s focus on [NEXT INITIATIVE NAME]
IF YES, CONTINUE WITH SECTION III
What process did you use to design performance measures for [INITIATIVE NAME]?
PROBES
Did you receive guidance from the state legislature, FNS national and regional offices, other states operating similar initiatives, and/or community or business partners?
Did you use data you already collect to calculate the performance measures for [INITIATIVE NAME]?
Did you attempt to collect new data to calculate the new performance measures?
Did the quality of the data influence your design?
Did you design performance measures in order to influence SNAP Accuracy and Integrity, Program Access, Efficiency, and Customer Service?
Does the state or county require you to collect any performance data or measures?
PROBE IF YES:
What are you required to collect?
Who requires you to collect those data, the state or the county?
Aside from the data and measures you are required to collect for the (state/county), does your organization collect any internal performance data or measures about [INITIATIVE NAME]?
PROBE IF YES
What do you collect?
How do you collect and maintain this information?
How do you use the information obtained to administer your program?
Have you experienced challenges, such as access, cost and time, or accuracy with collecting and using performance measures and data for [INITIATIVE NAME]? IF YES: Please describe the challenges.
PROBES IF YES:
Tell me about accessing data.
Would you describe the cost and time associated with obtaining the necessary data?
How would you characterize the level of accuracy in your data?
How did you identify and remedy the challenge(s)?
Next, I will ask a series of questions about three measures for [INITIATIVE NAME].
IF PARTNER HAS MORE THAN THREE MEASURES FOR THE INITIATIVE, ASK WHICH ARE THEIR HIGHEST PRIORITY MEASURES AND FOCUS ON THOSE.
WHEN DISCUSSING THE FIRST INITITATIVE: I have some more detailed questions about these measures to help us better understand the performance measurement process.
FOR SUBSEQUENT INITATIVES: Now let’s turn to some of those more detailed questions about the measures for [INITIATIVE NAME].
ASK ALL OF THE QUESTIONS IN SECTION B FOR EACH PERFORMANCE MEASURE OF INTEREST FOR THIS INITIATIVE, CYCLING THROUGH ALL OF SECTION B FOR EACH MEASURE BEFORE MOVING ON TO THE NEXT MEASURE FROM THE BEGINNING OF SECTION B.
1.1 What is the purpose for calculating [MEASURE NAME]?
1.2 How and how often do you review and report the results of [MEASURE NAME]?
1.3 How do you use the information [MEASURE NAME] provides you?
PROBE: For example, what actions do you take based on the results of [MEASURE NAME]?
1.4 Have you made changes to [MEASURE NAME] within the past 12 months? This could be the way you collect and store data, or the way you calculate, report, analyze, or use the measure.
[IF NO, GO TO NEXT QUESTION.]
PROBE IF YES: Please explain the change you made.
When did that change occur?
Did the change result in more or fewer observations being included?
Were there any other changes to collection and storage? (IF YES, CYCLE THROUGH PROBES AGAIN)
1.5 What procedures, if any, did your office establish to ensure that all data need to calculate [MEASURE NAME] are recorded or entered correctly?
PROBES:
What procedures are in place to ensure the data are complete and accurate?
Are the data collected in the same way across the state?
IF YES: How is that ensured?
IF NO: Please describe any known variations.
2.1 Can you provide more detail on how you calculate [MEASURE NAME]?
PROBES:
IF A MEASURE OF TIME: How do you define when the activity starts and stops?
IF A MEASURE OF NUMBER OF PEOPLE: Who is included in this count? Who is not included?
IF DEPENDENT ON A DEFINITION: How do you define …. (call/application/abandoned application/completed application, etc.)
2.2 How often do you calculate [MEASURE NAME]: real-time, daily, weekly, monthly, quarterly, or annually? Is it possible to calculate it at the state region, county, local office, and/or applicant levels of detail?
PROBE IF YES:
Exactly which levels of detail are possible for calculating [MEASURE NAME]?
Which level of detail do you use to assess your performance and make administration changes, if necessary?
PROBE IF NO
What is the greatest level of detail at which you can calculate [MEASURE NAME]?
Could the level of detail be finer if the measures were constructed differently? IF YES: How would this construction need to change to increase the level of detail? Is this change possible with the resources available to you?
3.1 Does [MEASURE NAME] use data that are manually data entered, captured through an automated process, or both?
PROBE IF MANUAL OR BOTH:
Do the data for [MEASURE NAME] reside in paper format only so the only way to calculate is from hand tallies and not by computer?
PROBE IF AUTOMATED OR BOTH:
Are the data transferred from paper to an electronic system? If so, how?
Are the data transferred from one electronic system to another? If so, how?
How effective do you consider the automation process to be?
Do you record any errors that occur through the automated data entry process? IF YES: Please describe what you record and how the process functions.
3.2 Can you describe the frequency and timeliness of data entry or automation for [MEASURE NAME]?
PROBES:
How often are data entered?
How soon is data entry or automation after measurement or data collection occurs?
3.3 How would you improve the efficiency of data collection?
3.4 How would you improve the efficiency of data entry?
NOTES TO INTERVIEWER:
(1) Performance standards include both a standard and a benchmark. The standard is the desired outcome of the activity (e.g., answering calls within 3 minutes). The benchmark is the desired rate of success (e.g., answering 80% of calls within 30 minutes).
(2) Performance incentives are related to the performance standard and can be a bonus (e.g., 90% of calls are answered within 3 minutes for a year and the call center gets more funding) or a sanction(e.g., Less than 80% of calls are answered within 3 minutes over the year and the call center gets less funding).
4.1 Do you have performance standards or performance incentives for [MEASURE NAME]?
IF NO:
Why did you choose not to establish them?
Are you currently considering implementing performance standards or performance measures for [INITIATIVE NAME]? IF YES: Please describe them.
Given that you do not use performance standards for [MEASURE NAME], I presume that there are also no performance incentives associated with the measure. Is this accurate? IF YES GO TO NEXT MEASURE.
IF YES
How was this performance standard or benchmark established?
Whose input did you consider?
Was the performance standard and/or incentive a legislative requirement?
What is the rationale for the performance standards in use? Has the rationale changed over time? IF YES: How?
Do these performance standards change over time? IF YES: Why?
How and how frequently do you track progress toward achieving these standards?
4.2 IF STANDARDS DO EXIST: Are any performance incentives linked to [MEASURE NAME]?
IF YES: Please describe the incentive and how it is applied.
How did you decide to implement the incentive?
Who or what must meet the performance standard in order to receive the incentive? Who does it apply to?
IF NO: DO SECTION B AGAIN FOR THE NEXT MEASURE FOR THIS INITIATIVE OR GO TO SECTION II AND CHOOSE NEXT MODERNIZATION INITIATIVE FOR DISCUSSION, AS APPROPRIATE.
4.3 What was your motivation for using the incentive?
4.4 How effective has the incentive been in terms of improving performance?
PROBE:
At what value of that element or measure do you define success?
Is there something about the process of collecting, reporting, or analyzing this measure that was critical?
DO SECTION B AGAIN FOR THE NEXT MEASURE FOR THIS INITIATIVE OR GO TO SECTION II AND CHOOSE NEXT MODERNIZATION INITIATIVE FOR DISCUSSION, AS APPROPRIATE.
REPEAT QUESTIONS FROM SECTION II AS NEEDED FOR EACH MODERNIZATION INITIATIVE. GO TO SECTION III ONCE YOU HAVE COMPLETED THIS PROCESS FOR ALL MODERNIZATION INITIATIVES.
With respect to the modernization initiatives we just discussed, is there any performance measure that you are not yet calculating but you would like to?
[IF NO: GO TO NEXT QUESTION]
PROBE IF YES:
What would be the advantages of calculating that performance measure and what is preventing you from calculating it?
How would the implementation cost for this measure compare to implementation costs for current measures?
Are there additional measures you would like to use or see the SNAP office use to assess program performance? IF YES: What additional measures are necessary?
Is there anything you think is important for the Food and Nutrition Service to know about your performance measurement of SNAP that we did not ask about?
Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes that we can share with FNS.
Before we sign off, I wanted to remind you that we are collecting 12 months of performance measure data from each organization we talk to on the phone. You can submit this to us in any way that is convenient for you, electronic, paper, or otherwise. Complete instructions are on the advance materials we sent you with the introductory letter, but you’re welcome to contact me at any time at [PHONE NUMBER] if you have questions.
Are there any questions you have for me before we finish?
END OF INTERVIEW
| File Type | application/msword |
| Author | Lisa Walls |
| Last Modified By | rdowner |
| File Modified | 2010-03-16 |
| File Created | 2010-03-04 |