State CN Agencies (Directors and Key Staff), SFSP Sponsors (Directors and Key Staff), and SFSP Sites (Supervisors) - SLT

Appendix C-3 State Director Interview Guide

OMB Control Number: 0584-XXXX

Expiration Date: XX/XX/20XX

Introduction

Good morning/afternoon. Thank you for taking the time to talk with me today. My name is [INTERVIEWER’S NAME] and I work for Westat, a private research company based in Rockville, Maryland. With me today is [NOTE TAKER’S NAME], who will be taking notes.

PURPOSE: The federal Food and Nutrition Service (FNS) is interested in understanding more about the administration and oversight of the Summer Food Service Program (SFSP) from the perspective of the States and SFSP sponsors and sites. FNS hired Westat to conduct a study to describe how the different entities administer and oversee the SFSP, any program integrity challenges, and any resources or training that could better support the effective administration and monitoring of the SFSP. During our conversation today, we would like to ask you about how you oversee the SFSP, any challenges you’ve encountered, and useful resources or training.

HOW YOU WERE SELECTED: We first conducted an online survey of all State agency directors, and from among those we worked with FNS to select 18 State directors for telephone interviews. We selected the 18 directors for interviews based on the size of a State’s SFSP programs, geography, and other characteristics.

Can you tell me if you helped to fill out the online survey?

[If yes] Okay, great. Some of the questions we have will touch on similar topics.

[If no] That’s okay. For some questions I will summarize for you what your State’s survey reported, but you don’t have to have filled out the survey to be able to answer the questions.

RISKS AND PRIVACY: There is little risk to being part of this study. We use all data we collect only for the purposes we describe. FNS knows which States were recruited for this study, but we will report the results of these interviews in the aggregate. Your name will not be linked to your responses. In our reports we may include direct quotes, but these will be presented without the speaker’s name and in such a way that you could not be identified. FNS will receive a redacted copy of the transcript of this interview that has been stripped of identifying information.

VOLUNTARY PARTICIPATION: Your participation is entirely voluntary. Refusal to participate will not have any impact on your position, your State, or child nutrition programs. You can take a break, skip questions, say something off the record, or stop participating at any time.

QUESTIONS: If you have questions about your rights and welfare as a research participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, the name of the research study you are calling about, which is the SFSP Integrity Study , and a phone number beginning with the area code. Someone will return your call as soon as possible.

We have planned for this discussion to last 90 minutes, until [TIME]. Is that still okay?

With your permission I would like to record this discussion to help us fill any gaps in our written notes. The recordings, transcripts, and any notes we have will be stored on Westat’s secure server and will be With your permission I would like to record this discussion to help us fill any gaps in our written notes. The recordings, transcripts, and any notes we have will be stored on our secure server and will be destroyed after the project is complete. FNS will not receive any audio recordings, but will receive a redacted copy of the interview transcript that has been stripped of identifying information.

Do you have any questions? [ANSWER ALL QUESTIONS]

May I turn on the audio recorder now? [TURN ON AUDIO RECORDER IF GIVES CONSENT]

Now that the audio recorder is on, do you agree to participate? [PAUSE FOR RESPONSE]

And do you consent to be audio recorded? [PAUSE FOR RESPONSE]

Warm Up and Context

1. To start, please tell me how long you have worked at the agency, and what your specific responsibilities are for the SFSP.

Probe: Oversee the program, train sponsors/sites, review data

2. From the survey you completed I see that there are [number of FTEs per survey question A2] at the State agency working on the SFSP. Do you feel you have enough State-level staff to properly administer and monitor the SFSP? Why or why not?

3. Do you feel you have enough funding to properly administer and monitor the SFSP? Why or why not?

4. Have any State-level policies impacted SFSP staffing or funding levels in the last five years or so? Describe.

Determining the Eligibility of Sponsors and Sites

Now I’m going to ask questions about the processes you use to determine and document sponsor and site eligibility for the SFSP.

5. When an organization applies for the first time to be considered as a SFSP sponsor, how do you determine whether the organization is eligible?

1. What documentation do sponsors submit to be considered and approved?

Probe: proof of nonprofit status, SFA status, proposed budget

6. [if respond “yes” to survey question E10] We see from the survey that the State received FNS approval to place additional requirements on sponsors beyond what the regulations require. What are those additional requirements?

   1. Why was that important to require of sponsors in your State?

7. What would you change, if anything, about the State’s process to vet potential sponsors?

8. In your opinion, what is the most important factor to consider when determining whether an organization can operate successfully as a first-time sponsor? Why?

9. What documentation do existing sponsors have to submit each year in order to be approved to continue their program?

Interviewer note: sponsors have to provide annual documentation to verify their continued eligibility. For example, they have to document their list of sites and nonprofit status.

10. [if respond “yes” to survey question E11] We see from the survey that the State received FNS approval to place additional requirements on sites beyond what the regulations require. What are those additional requirements?

    1. Why was that important to require of sites in your State?

11. What is the process to determine whether a potential open site is eligible?

1. What documentation do sponsors have to submit on their sites?

Probe: school data on free/reduced price, whether site is urban/rural, whether site is self-prep site or vended, migrant

2. How does the State review and verify this information?

3. Do you feel the current process is effective? Why or why not?

   1. Is there anything that would improve it?

12. How, if at all, is the process different when determining whether a potential closed enrolled site is eligible?

[Interviewer Note: use same sub-questions as in Question 11]

13. And finally, how is the process different when determining whether a potential camp site is eligible?

[Interviewer Note: use same sub-questions as in Question 11]

14. How do you handle situations when two sites want to serve the same area or same group of children?

15. I want to ask now about pre-approval visits. What is the State looking for when conducting pre-approval visits to first-time sponsors?

1. Are there any “red flags” you look for that might alert you to a problem? If yes, describe.

2. How effective are pre-approval visits in helping you identify potential issues with a sponsor?

16. What would you say is the most common reason the State has not approved potential sponsors in the past?

17. Does the State conduct pre-approval visits of sites?

[If yes]

1. Under what circumstances does the State decide to conduct a pre-approval visit to a site?

2. Are there any “red flags” you look for that might alert you to a problem? If yes, describe.

3. What is the most common reason the State has not approved a site in the past?

4. How effective are pre-approval visits in helping you identify potential issues with a site?

18. Tell me about any challenges in conducting the pre-approval visits to sponsors or sites.

Probe: sufficient staff and time to conduct visits, distances to travel

Providing Resources and Training

Your responses have been very helpful so far. Now I’m going to ask about State agency training and resources that help sponsors operate the program.

19. [If respondent enters more than “1” for survey question C1]. You told us in the survey that the State holds multiple annual trainings for sponsors. Tell me a bit about the different trainings.

Interviewer note: Try to understand the rationale for holding multiple trainings.

Probes: Are they training in multiple geographic areas; or separate trainings for each of the different sponsor types (i.e., camps, SFAs, etc.); or separate trainings for first-time sponsors v. more experienced sponsors?

20. I also see from the survey that the State provides annual training to sponsors on a range of topics. Which topics would you say the State emphasizes the most in the annual training?

21. How, if at all, is the training tailored to different sponsors?

Probe: first-time v. existing sponsors; sponsor type (camp, SFA, local government, etc.); sponsors in different geographic areas; affiliated v. unaffiliated.

22. Which sponsor staff are required to participate in the annual training?

Probe: directors, program managers, etc.

1. Are sponsors permitted to bring additional staff if they choose?

23. [if select first or second response option in survey question C8] You indicated in the survey that the State verifies that sponsors understand the information in the training by administering a post-test or something similar. Please describe to me what that looks like.

1. Which sponsor staff are required to complete those assessments?

2. What specifically does the [post-test or other] assess?

3. How is it administered? (online, paper, other)

4. In your opinion, how effective is the post-test at measuring sponsors’ knowledge?

24. Outside of the annual training we discussed, are there any additional trainings offered to sponsors?

If yes, probe to understand mode, target audience, topics covered, timing and frequency.

25. Apart from the trainings, describe any technical assistance that the State offers to sponsors.

26. Does the State assist with any of the trainings or resources that sponsors provide to sites? If so, describe.

27. Thinking about your own staff at the State who administer and oversee the SFSP, what additional SFSP training would you like FNS to provide to your staff?

28. Finally, what additional SFSP resources would you like FNS to provide to your staff?

1. When would you need these materials to be published in order to incorporate them into your sponsor trainings?

State Monitoring of Sponsors and Sites

My next set of questions will help me to better understand how the State conducts the required reviews of sponsors and sites.

29. How do you select which sponsors you will review each year, to meet the regulatory requirement for reviews?

Probe to understand the selection criteria and process.

30. What challenges does the State face in conducting the required number of sponsor reviews?

Probes - staffing, timing, distance, length of review

31. Walk me through the process for conducting a typical on-site review of a sponsor.

Probe: what are they looking at - meal records, allowable costs, health and sanitation documentation, procurement documents?

1. Are the visits announced or unannounced?

2. Does the review process differ by the type of sponsor? If so, how?

32. Before the reviewer goes on site, what materials do they examine to prepare for the review?

Probe: prior review findings, claims, requests for technical assistance, complaints.

33. What is different about the process when you’re conducting an on-site review of a sponsor that had findings in prior years?

34. What happens when you find a compliance issue during an on-site sponsor or site review?

1. How does the State follow-up on these issues to ensure corrective action is taken?

35. To the best of your knowledge, what are the most common compliance issues among sponsors?

1. What factors help to promote sponsor compliance and prevent serious deficiency or termination?

Probe to understand what the State does, if anything, to assist sponsors so that they continue participating from one year to the next.

36. Similarly, what are the most common compliance issues among sites?

What factors help to promote site compliance and prevent serious deficiency or termination?
Probe to understand what the State does, if anything, to assist sites so that they continue participating from one year to the next.

37. What aspects of the State agency’s process for conducting sponsor and site reviews do you think are particularly effective? Why?

38. How, if at all, do your program review findings in one year influence the training you offer to sponsors the following year?

Now I want to talk a bit about meal disallowances.

39. When you’re conducting a sponsor review, what is the process you follow if you have to disallow meals?

1. How does the State agency track the meal disallowances to make sure the sponsor doesn’t later include them in their claim?

2. How is the process different when disallowing meals following an off-site review of a sponsor’s meal count records?

40. [Calculate based on response to survey question H2] From the survey we see that [breakfast/lunch/dinner/snacks] is the most commonly disallowed meal type. Why do you think that is?

    1. In your opinion, are self-prepared or vended meals more likely to be disallowed? Explain your thinking.

41. Are there particular types of sites that are more likely to have meals disallowed? If so, which ones?

1. Why do you think those types of sites are more likely to have meals disallowed?

Probes: large/small, new/experienced, open/closed/enrolled/camps, affiliated/unaffiliated, schools/nonprofits/camps/mobile sites.

Variation in Sponsor Administration of SFSP

These next few questions are trying to get at any differences between the types of sponsors and sites and how they administer the program.

42. What differences do you see in how affiliated sponsors work with their sites to implement the SFSP compared with unaffiliated sponsors?

1. Do you adjust the way you oversee your sponsors to account for these differences?

43. Which sponsors seem to have more difficulty with training their sites?

Probe: nonprofits, local government, camps, and SFAs; longstanding v. new SFSP sponsors; those in certain geographic areas; sponsors with a lot of sites v. those with only a few sites; sponsors who are affiliated with their sites vs. those who are not

1. What, specifically, could they improve upon?

2. What kind of support could help those sponsors to improve their training?

Probe: training, TA, other resources

44. Which sponsors appear to have more difficulty with monitoring their sites?

Probe: nonprofits, local government, camps, and SFAs; longstanding v. new SFSP sponsors; those in certain geographic areas; sponsors with a lot of sites v. those with only a few site; sponsors who are affiliated with their sites vs. those who are not.

[Interviewer note: by ‘monitoring’ we mean reviewing sites’ records or visiting the sites]

1. Why do you think this sponsor type has more difficulty?

2. What, specifically, could they improve upon?

3. What kind of support could help those sponsors improve their monitoring efforts?

Probe: training, TA, other resources

Collecting and Reviewing Program Data

This has been very helpful. Now I’m going to ask about meal claims and program advances and meal claims.

45. What is the process at the State to review monthly SFSP meal claims from sponsors?

1. Have there been instances where you look at a meal claim and the numbers don’t look right?

   1. What is the most common reason a claim might be off?

   2. What does the State do to address a suspected error?

   3. Do certain types of sponsors tend to have more challenges with submitting accurate meal claims?

46. How often do you receive late claims from sponsors?

1. What are the circumstances?

47. [If applicable, per survey question G2] I see from the survey that the State portal is programmed to automatically perform edit checks when it receives the meal claims data. Tell me about those edit checks and what they look for.

1. Which edit checks are most frequently triggered?

2. If the edit check flags something as a potential error, what happens at that point?

Probe to understand any follow up and who does that.

48. Describe the process to review and approve requests from sponsors for advance payments.

1. What documentation do sponsors have to submit to be considered for an advance payment?

2. What happens if you feel the amount requested by a sponsor is too high or too low?

3. What criteria does the State use to decide what is a reasonable amount for advance payments?

49. How often do sponsors overestimate the amount of advance payment they need and they later need to return funds?

1. Is this more common for certain types of sponsors? Which ones and why?

50. Are there situations in which the State may withhold advance payments from a sponsor? If so, describe those situations.

Integrity Challenges/Wrap-Up

Next, I want to ask a few general questions about program integrity. For our purposes, we define “program integrity” as efforts to reduce fraud, ensure accurate eligibility determinations, and reduce improper payments.

51. In your opinion, what is the biggest program integrity issue among first-time sponsors?

1. What strategies have you used to help to address those issues?

52. In your opinion, what is the most important thing the State agency does to ensure program integrity in the SFSP?

53. What is the most important thing that FNS can do to help States promote program integrity?

The Food and Nutrition Service (FNS) is conducting this study to obtain information about the administration and oversight of the Summer Food Service Program and to identify potential barriers to ensuring the integrity and effective management of the program. Participation in this study is voluntary and the information collected will be used to determine resources, develop training, and provide technical assistance. Under the Privacy Act of 1974 and the System of Record Notice FNS-8 USDA/FNS Studies and Reports, any personally identifying information obtained will be kept private to the extent provided by law. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-NEW. The time required to provide this information collection is estimated to average 90 minutes per response, including the time to review instructions, search existing data resources, gather and maintain the data needed, and complete and review the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302 ATTN: PRA (0584-xxxx).

1