APPENDIX K
SNAP STAFF PHONE INTERVIEW PROTOCOL
My name is [X], from Mathematica Policy Research.
As you may know, Mathematica is collecting information from state and local SNAP offices and their partners and contractors to identify how they are measuring the implementation of SNAP modernization initiatives. We really appreciate the time you and your staff took to complete the survey we sent you about the SNAP modernization initiatives being implemented in your state and the measures used to assess performance. I used that information to tailor the questions for this interview. Specifically, I would like to spend most of this conversation talking about [IDENTIFY INITIATIVE(S) FOR THIS CALL].
Our report will describe the range of responses expressed by staff, and may list the names of agencies and partners who contributed information, but we will not quote you by name or title.
I expect our conversation will take approximately 90 minutes.
OPTIONAL IF INTERVIEWER CHOOSES TO RECORD: I want to be sure I am keeping track of everything you are saying. May I record our discussion so that I can listen to it later when I write up my notes? No one outside of our research team will have access to the recording.
IF YES: Thank you. It will be helpful if you speak up, speak clearly, and speak one at a time.
IF NO: That’s no problem. I’ll take notes as you talk, but I may sometimes need to ask you to slow down or repeat so that I can get all the information.
First, do you have any questions for me about the project or what we will be discussing today after reading the background materials I sent you?
Thank you. I will begin by clarifying some of your responses on the survey, and then, I will ask for more details on how you have implemented a few modernization initiatives. The remaining questions will focus on the details of how you collect, analyze, and use performance measures about each SNAP modernization initiative.
Public Burden Statement
Public reporting burden for this collection of information will vary by agency, but is estimated to be 90 minutes, on average, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Services, Office of Research and Analysis, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302, ATTN: PRA (0584-xxxx*).
First, when I reviewed your survey, I found a few items that I would like to clarify before I begin asking you questions about your performance measures.
USE THESE GENERAL QUESTIONS TO CLARIFY INCONSISTENCIES IN THE SURVEY, IF APPLICABLE. IF THIS SECTION DOES NOT APPLY, GO TO SECTION II.
We noted that [QUESTION NUMBER] does not have a response. This question reads [READ QUESTION]. How would you respond to this question?
The answer to [QUESTION NUMBER] does not appear to coincide with the answer to [QUESTION NUMBER]; can you describe each to clarify further?
From another source(s), we learned that your state was doing or measuring [ACTIVITY], but the survey response indicates otherwise.
Can you explain this difference?
Has your measurement activity changed over time?
INSTRUCTIONS TO INTERVIEWER:
If needed, explain that possible sources are information from the region or from documents located on the state website. note that individual interviewers don’t always know the source for each individual piece of preliminary information we collected. If they would like a follow-up call about this source, say someone involved in that portion of the project will get back to them and contact your regional lead. Be sure you have information on the best way to contact the respondent for follow-up.
IF ANSWER YES TO SURVEY QUESTION #3 ON DOCUMENTATION: In a question on the survey that asked about documentation available, you or someone at your (state/agency) answered yes. Is it possible for you to provide us with a copy of the documents that describe performance measure calculation?
IF ANSWER NONE OF THE ABOVE TO SURVEY QUESTION #5 ON SNAP GOALS: In a question on the survey that asked about SNAP goals you or someone at your (state/agency) answered “none of the above.” Could you explain further why this measure does not align to the SNAP goals?
IF the ANSWER to the survey question ABOUT USE OF ARRA FUNDS was “NOT SURE”, ASK THE FOLLOWING QUESTION:
The American Recovery and Reinvestment Act of 2009, which is sometimes called ARRA, the Recovery Act, or the Stimulus, had some funds to support states’ SNAP administration. Your survey answers “not sure” to the question about the use of ARRA funds to support the implementation of [NAME MODERNIZATION INITIATIVE(S)], Do you have any additional information indicating whether ARRA funds were used to support the implementation of [NAME MODERNIZATION INITIATIVE(S)]?
Additional information for interviewer: The American Recovery and Reinvestment Act of 2009 was passed in February 2009. It is sometimes referred to as ARRA, the Recovery Act, or the Stimulus. We will refer to it as ARRA. The ARRA provided SNAP with nearly $300 million to support the administration of the program by states—$145 million in fiscal year 2009 and $150 million in fiscal year 2010. It is possible that states may use some of the ARRA funds they receive to implement specific modernization initiatives so that they can more efficiently determine eligibility and recertify SNAP participants. We now want to clarify the use of ARRA funds for modernization activities in your state.
IF the STATE USED ARRA FUNDS TO SUPPORT SNAP, ASK:
You or someone at your (state/agency) indicated that ARRA funds were used to support the implementation of [NAME MODERNIZATION INITIATIVE(S)]. Could you describe how those funds were used?
PROBES/EXAMPLES IF NEEDED:
Was the initiative supported by ARRA funds new or existing?
Were the ARRA funds intended only to support a particular functionality of the initiative (such as, to allow e-signature capabilities online)?
Did ARRA funds wholly support full or partial implementation of the initiative?
Would the initiative have been implemented without the use of ARRA funds?
Were any performance measures or standards created specifically because ARRA funds were used to support the implementation of [NAME MODERNIZATION INITIATIVE(S)]? Please provide details for each relevant initiative.
I will spend most of the remainder of this call focusing the conversation about how you measure the modernization initiatives your [STATE or AGENCY] is engaged in.
First, though, I have some basic questions about how your state records the origin and status of your SNAP applications.
Do you record whether applications were delivered to an office, filed online, submitted by community organization, mailed, faxed, or submitted by another method?
PROBE IF NO:
So, do applications appear the same within your SNAP system regardless of their source?
IF NO: Can you describe how the system differentiates the sources?
IF YES: go to section B.
PROBE IF YES:
Please describe how the application source is recorded in your SNAP system. Is there a single field in your SNAP eligibility system that is designated for this purpose, multiple fields to identify the source, or some other method?
PROBE If there is an “other” category: Please provide some examples of the type of source that would be labeled as other.
Thank you. Now, I have a few more basic questions about your modernization initiative(s) so that I have a better understanding about how SNAP modernization operates in your (state/area). Later, I will ask for greater detail on the measures you use to track each initiative.
1. To start, who operates your call center? Is it the state, a contractor, or someone else?
As part of this study, we would like to talk with staff operating call centers. Can you help me locate contact information for the right people to talk with?
GO TO NEXT RELEVANT INITIATIVE OR TO SECTION III
I want to ask some questions about the process that applicants use to access and submit online applications.
2.1 Do community partners have special access to the online system? IF YES: Please describe the access privileges given to community partners.
2.2 How does the information obtained through the online applications system transition to the eligibility determination system that the SNAP office uses to calculate benefits?
PROBE:
Are online applications manually entered into the determination system?
If NO: Does that mean that every piece of information on the application is automatically transferred into the eligibility determination system?
If YES: What pieces of information manually entered into the eligibility system? What information is not transferred at all?
2.3 IF STATE HAS SCREENING AND APPLICATIONS ONLINE: Is it possible to use the screening tool and have the application pre-populated with some of this information?
2.4 Who operates your online system? Is it the state, a contractor, or someone else?
As part of this study, we would like to talk with staff operating call centers. Can you help me locate contact information for the right people to talk with?
GO TO NEXT RELEVANT INITIATIVE OR TO SECTION III
I have some questions about your oversight activities with respect to SNAP partners and contractors. You have indicated that you partner with other organizations to [LIST ACTIVITIES PROVIDED ON SURVEY].
3.1 How formal (is/are) the arrangement(s) that govern(s) the (partnership/contract)? Is there an MOU or a contract?
PROBE IF YES:
Are any of the terms of your agreement based on performance of specific functions? IF YES: Please describe which functions and what performance is expected.
3.2 Do you compensate the partner for services provided? IF YES: How?
PROBE:
Is the compensation dependent upon a specific service volume or an achievement of some kind? IF YES: Please explain.
3.3 As part of this project, we are calling some partners and contractors to ask about SNAP performance measurement. Can you suggest [NUMBER] organizations in your (county/state) that are actively involved in collecting and reporting SNAP performance data?
3.4 Is there someone in your office we can follow up with to get contact information for the right person to speak with at each organization? (RECORD CONTACT INFORMATION)
GO TO NEXT RELEVANT INITIATIVE OR TO SECTION III
I want to get a few details about the document imaging initiative.
4.1 Do you record the source of the documents to be imaged and, if so, how?
4.2 Do you record the types of documents received and, if so, how?
4.3 How and where are scanned documents stored and accessed, and by whom?
PROBE:
Is the document imaging system integrated with an electronic case management system?
IF YES: Please describe the process from the point at which a document is received to the point in which it is integrated in the electronic case management system.
GO TO NEXT RELEVANT INITIATIVE OR TO SECTION III
I have a few questions about the kiosks that are available for SNAP functions.
5.1 What are they primary functions of the kiosks?
PROBE:
For example: applicant pre-screening, download online application, submit unsigned application, submit signed application, and/or submit changes to submitted application.
5.2 Where are kiosks located?
5.3 Are all applicants permitted access to the kiosks?
5.4 Do you record whether an application originated at a kiosk and, if so, how?
5.5 Do you record whether a case worker or community partner provides assistance at a kiosk and, if so, how?
GO TO NEXT RELEVANT INITIATIVE OR TO SECTION III
6.1 Do you record the type and number of interviews provided to applicants? IF YES: How?
6.2 Do you record the application decision according to the type of interview given?
IF NEEDED: Interview type can mean in-person or by phone, or can mean regular versus shortened interviews.
PROBE IF YES:
Do you record case characteristics for different types of interviews?
IF YES: Please describe this process.
IF NO: Do the capabilities exist in your system to be able to do this?
6.3 Do you record whether there was a problem completing the interview? IF YES: Please explain this process.
GO TO NEXT RELEVANT INITIATIVE OR TO SECTION III
6.4 Do you record whether an application is expedited? IF YES: How?
PROBE: Do you retain information on whether an application was expedited in the case record? For example, would you be able to summarize the characteristics of cases that had expedited versus regular applications according to demographics or approval status?
GO TO SECTION III
ASK QUESTIONS IN THIS SECTION FOR THE APPLICATION TRACKING SECTION AND FOR EACH INITIATIVE
ALL CALLS: Next, we’ll discuss each initiative in turn to get some basic information about that initiative, and then I will ask some more specific questions about measures for that initiative.
CALLS WITH STATES THAT DO APPLICATION TRACKING: I will begin by asking you about performance measures related to application tracking, and then will move on to discussing measures about each of the initiatives you just described to me.
ALL CALLS: To focus our conversation, I’ll ask all of my questions about measures for one initiative before moving on to ask similar questions about the next initiative.
Let’s focus now on [INITIATIVE NAME].
IF RESPONDENT DID NOT MARK ANY MEASURES FOR [INITIATIVE NAME] ON THE SURVEY: Do you collect any performance measures or data for [INITIATIVE NAME]?
IF NO – Thank you. Instead, let’s focus on [NEXT INITIATIVE NAME] – BEGIN A AGAIN.
What process did you use to design performance measures for [INITIATIVE NAME]?
PROBE:
Did you receive guidance from the state legislature, FNS national and regional offices, other states operating similar initiatives, and/or community or business partners?
Did you use data you already collect to calculate the performance measures for [INITIATIVE NAME]?
Did you collect new data to calculate the performance measures?
Did the quality of the data influence your design?
Did you design performance measures in order to influence SNAP Accuracy and Integrity, Program Access, Efficiency, and Customer Service?
3. IF INITIATIVE IS PARTNERING:] Do you require partners to collect any data or measures of their performance? IF YES: What are they required to collect? How frequently do they report that information, and to whom?
Have you experienced challenges, such as access, cost and time, or accuracy, with collecting and using [INITIATIVE NAME] performance measures and data? IF YES: Please describe the challenges and how you identified and remedied them.
PROBES IF YES:
Tell me about accessing data.
Would you describe the cost and time associated with obtaining the necessary data?
How would you characterize the level of accuracy in your data?
How did you identify and remedy the challenge(s)?
Next, I will ask a series of questions about three measures for [INITIATIVE NAME]. Mostly, these measures are the ones that you or your staff identified as high priority on the survey we sent you. IF NO HIGH PRIORITY MEASURES ARE IDENTIFIED, ASK RESPONDENT FOR THEIR PRIORITIES BASED ON TIMING AND PROJECT NEEDS.
WHEN DISCUSSING APPLICATION TRACKING OR THE FIRST INITIATIVE: I have some more detailed questions about these measures to help us better understand the performance measurement process.
FOR SUBSEQUENT INITIATIVES: Now let’s turn to some of those more detailed questions about the measures for [INITIATIVE NAME].
ASK ALL OF THE QUESTIONS IN SECTION B FOR EACH PERFORMANCE MEASURE OF INTEREST FOR THIS INITIATIVE, CYCLING THROUGH ALL OF SECTION B FOR EACH MEASURE BEFORE MOVING ON TO THE NEXT MEASURE FROM THE BEGINNING OF SECTION B.
1.1 How, and how often, do you review and use the results of [MEASURE NAME]?
PROBE:
For example, what actions do you take based on the results of [MEASURE NAME]?
1.2 Have you changed anything about [MEASURE NAME] within the past 12 months? This could be the way you collect and store data or the way you calculate, report, analyze, or use the measure. [IF NO, GO TO NEXT QUESTION.]
PROBE IF YES: Please explain the change you made.
Why did you make that change?
When did that change occur?
Did the change result in more or fewer observations being included?
Were there any other changes to collection and storage? (IF YES, CYCLE THROUGH PROBES AGAIN)
1.3 What procedures, if any, did your office establish to ensure that all data need to calculate [MEASURE NAME] are recorded or entered correctly?
PROBES:
What procedures are in place to ensure the data are complete and accurate?
Are the data collected in the same way across the state?
IF YES: How is that ensured?
IF NO: Please describe any known variations.
2.1 In the survey you explained that you calculated [MEASURE NAME] by [INSERT DESCRIPTION]. Can you provide more detail on how you calculate this measure?
PROBES:
IF A MEASURE OF TIME: How do you define when the activity starts and stops?
IF A MEASURE OF NUMBER OF PEOPLE: Who is included in this count? Who is not included?
IF DEPENDENT ON A DEFINITION: How do you define …. (call/application/abandoned application/completed application, etc.)
2.2 Your survey response indicated that it was possible to calculate [MEASURE NAME] with [REAL-TIME/DAILY/WEEKLY/MONTHLY/QUARTERLY/and ANNUAL] frequency. Is it possible to calculate it at the state region, county, local office, and/or applicant levels of detail?
PROBE IF YES:
Exactly which levels of detail is it possible to use for calculating [MEASURE NAME]?
Which level of detail do you use to assess your performance and make administration changes, if necessary?
PROBE IF NO:
What is the greatest level of detail at which you can calculate [MEASURE NAME]?
Could the level of detail be finer if the measure were constructed differently? IF YES: How would this construction need to change to increase the level of detail? Is this change possible with the resources available to you?
3.1 Does [MEASURE NAME] use data that are manually entered, captured through an automated process, or both?
PROBE IF MANUAL OR BOTH:
Do the data for [MEASURE NAME] reside in paper format only so the only way to calculate is from hand tallies and not by computer?
PROBES IF AUTOMATED OR BOTH:
Are the data transferred from paper to an electronic system? If so, how?
Are the data transferred from one electronic system to another? If so, how?
How effective do you consider the automation process to be?
Do you record any errors that occur through the automated data entry process? IF YES: Please describe what you record and how the process functions.
3.2 Can you describe the frequency and timeliness of data entry or automation for [MEASURE NAME]?
PROBES:
How often are data entered?
How soon is data entry or automation after the measurement or data collection occurs?
3.3 How would you improve the efficiency of data collection?
3.4 How would you improve the efficiency of data entry?
NOTES TO INTERVIEWER:
(1) Performance standards include both a standard and a benchmark. The standard is the desired outcome of the activity (e.g., answering calls within 3 minutes). The benchmark is the desired rate of success (e.g., answering 80% of calls within 30 minutes).
(2) Performance incentives are related to the performance standard and can be a bonus (e.g., 90% of calls are answered within 3 minutes for a year and the call center gets more funding) or a sanction(e.g., Less than 80% of calls are answered within 3 minutes over the year and the call center gets less funding).
4.1 IF THE SURVEY INDICATED NO PERFORMANCE STANDARDS OR INCENTIVES: I reviewed your responses from the survey and it indicated that you do not have performance standards or performance incentives for [MEASURE NAME]. Why did the SNAP office choose not to establish them?
Is the SNAP office currently considering implementing performance standards or performance incentives for [INITIATIVE NAME]? IF YES: Please describe them.
4.2 IF RESPONDENT SAYS THAT STANDARDS EXIST: How was this performance standard or benchmark established?
EXAMPLES, IF NEEDED:
Was input considered from state preferences or legislation, FNS/Region, industry, community partner, private contractor, and/or other states?
Was the performance standard and/or incentive a legislative requirement?
What was the rationale for the performance standards in use?
Has the rationale changed over time? IF YES: How?
Did these performance standards change over time? IF YES: Why?
How and how frequently do you track the progress toward achieving these standards?
4.3 IF SURVEY INDICATES INCENTIVES ARE LINKED TO THE MEASURE: You said on the survey that you have performance incentives linked to [MEASURE NAME]. Please describe how the performance incentive is applied.
PROBE:
How did the SNAP agency decide to implement the incentive?
Who or what must meet or exceed the performance standard in order to receive the incentive? Who does it apply to?
4.4 What was the original motivation for using the incentive?
4.5 How effective has the incentive been in terms of improving performance?
PROBE:
At what value of that element or measure do you define success?
Is there something about the process of collecting, reporting, or analyzing this measure that was critical?
DO SECTION B AGAIN FOR THE NEXT MEASURE FOR THIS INITIATIVE OR BEGIN SECTION III AGAIN FOR THE NEXT MODERNIZATION INITIATIVE, AS APPROPRIATE.
REPEAT QUESTIONS FROM SECTION III FOR THE APPLICATION TRACKING SECTION AND FOR EACH MODERNIZATION INITIATIVE. GO TO SECTION IV ONCE YOU HAVE COMPLETED THIS PROCESS FOR ALL MODERNIZATION INITIATIVES.
With respect to the modernization initiatives we just discussed, are there any performance measure that you are not yet calculating but you would like to do so?
[IF NO: GO TO NEXT QUESTION]
PROBE IF YES:
What would be the advantages of calculating that performance measure and what is preventing you from calculating it?
How would the implementation cost for this measure compare to implementation costs for current measures?
Are there performance measures that you would like to see all states report on? IF YES: What additional measures do you think are necessary?
Is there anything you think is important for the Food and Nutrition Service to know about your performance measurement of SNAP that we did not ask about?
Thank you for your time and helpful feedback. The information you have shared will be valuable to our team as we look across states and localities for themes that we can share with FNS.
Before we sign off, I wanted to remind you that we are collecting 12 months of performance measure data from each organization we talk to on the phone. You can submit this to us in any way that is convenient for you, electronic, paper, or otherwise. Complete instructions are on the advance materials we sent you with the survey, but you’re welcome to contact me at any time at [PHONE NUMBER] if you have questions.
Are there any questions you have for me before we finish?
END OF INTERVIEW
| File Type | application/msword |
| Author | Lisa Walls |
| Last Modified By | rdowner |
| File Modified | 2010-03-16 |
| File Created | 2010-03-04 |