State CN Agencies (Directors and Key Staff), SFSP Sponsors (Directors and Key Staff), and SFSP Sites (Supervisors) - SLT

APPENDIX C-4. SPONSOR INTERVIEW GUIDE

OMB Control Number: 0584-0656

Expiration Date: XX/XX/20XX

Introduction

Good morning/afternoon. Thank you for taking the time to talk with me today. My name is [INTERVIEWER’S NAME] and I work for Westat, a private research company based in Rockville, Maryland. With me today is [NOTE TAKER’S NAME], who will be taking notes.

PURPOSE: The federal Food and Nutrition Service (FNS) is the office within the U.S. Department of Agriculture that operates the Summer Food Service Program (SFSP), sometimes referred to as the Summer Meals Program. FNS is interested in understanding more about the administration and oversight of the SFSP from the perspective of the States and SFSP sponsors and sites. FNS hired Westat to conduct a study to describe how the different entities administer and oversee the SFSP, any program integrity challenges, and any resources or training that could better support the effective administration and monitoring of the SFSP. During our conversation today, we would like to ask you about how you oversee the SFSP, any challenges you’ve encountered, and useful resources or training.

HOW YOU WERE SELECTED: We are conducting telephone interviews with 48 SFSP sponsors around the country. Westat worked with FNS to select a diverse group of sponsors based on the type of sponsor organization, the size of each sponsor’s program, the types of sites that sponsors work with, and how long they have operated the SFSP.

RISKS AND PRIVACY: There is little risk to being part of this study. We use all data we collect only for the purposes we describe. We will report the results of these interviews in the aggregate. Neither your name nor the name of your organization will be linked to any of your responses. In our reports we may include quotes from our respondents, but these will be presented without the speaker’s name and in such a way that you could not be identified. FNS will receive a redacted transcript of this interview, stripped of information that could identify you or your organization.

STUDY COSTS AND COMPENSATION: There is no cost to you to participate apart from the time you spend with us for this interview, nor is there compensation.

The Food and Nutrition Service (FNS) is conducting this study to obtain information about the administration and oversight of the Summer Food Service Program and to identify potential barriers to ensuring the integrity and effective management of the program. Participation in this study is voluntary and the information collected will be used to determine resources, develop training, and provide technical assistance. Under the Privacy Act of 1974 and the System of Record Notice FNS-8 USDA/FNS Studies and Reports, any personally identifying information obtained will be kept private to the extent provided by law. According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-XXXX. The time required to provide this information collection is estimated to average 60 minutes per response, including the time to review instructions, search existing data resources, gather and maintain the data needed, and complete and review the collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing this burden, to: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, 3101 Park Center Drive, Room 1014, Alexandria, VA 22302 ATTN: PRA (0584-xxxx).

VOLUNTARY PARTICIPATION: Your participation is entirely voluntary. Refusal to participate will not have any impact on your position, your organization, or child nutrition programs. You can take a break, skip questions, say something off the record, or stop participating at any time.

QUESTIONS: If you have questions about your rights as a research participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, the name of the research study you are calling about, which is the SFSP Integrity Study, and a phone number beginning with the area code. Someone will return your call as soon as possible.

We have planned for this discussion to last 1 hour, until [TIME]. Is that still okay?

With your permission I would like to record this discussion to help us fill any gaps in our written notes. The recordings, transcripts, and any notes we have will be stored on our secure server and will be destroyed after the project is complete. FNS will not receive any audio recordings, but will receive a redacted copy of the interview transcript that has been stripped of identifying information.

Do you have any questions? [ANSWER ALL QUESTIONS]

May I turn on the audio recorder now? [TURN ON AUDIO RECORDER IF GIVES CONSENT]

Now that the audio recorder is on, do you agree to participate? [PAUSE FOR RESPONSE]

And do you consent to be audio recorded? [PAUSE FOR RESPONSE]

Warm Up

Before I get started, I would like to emphasize that the questions I ask are not meant to test your knowledge of the SFSP program rules, but to better understand how the program actually operates. With that said, I would like to begin with a few background questions.

1. How long have you been working at [organization/agency]?

2. Approximately how many years has your [organization/agency] been sponsoring summer meals? Your best estimate is fine, but it’s also fine to say you don’t know.

   1. [SFA sponsors only] How many of those years were spent operating the SFSP as opposed to the Seamless Summer Option (SSO)?

3. About how many sites are you sponsoring this summer?

a. Are any of these mobile sites?

4. Sponsors may operate different types of sites, such as open sites, restricted open sites, closed enrolled sites, camp sites or migrant sites. Tell me about all of the different types of sites that you sponsor.

Probe: if they sponsor multiple types of sites, what are the rough proportions (i.e., half camps, half open sites)?

5. Sometimes sponsors and sites are part of the same general organization, while in other cases they work together only on SFSP. Which of those describes the relationship you have with your sites?

[Interviewer note: we’re getting at affiliated v. unaffiliated sites, and that terminology may be unfamiliar to sponsors.]

6. Approximately how many of your sites self-prepare meals versus using a vended meal service?

[Interviewer note: The focus of the question is on how the meals are prepared. For SFA sponsors, it does not matter if the interviewee is an employee of the school district or a private food service company.]

Sponsor Application and Training

7. Were you serving in your current role when your [organization/agency] submitted its most recent application to be a sponsor?

If yes:

1. What information did the State agency ask you to provide to prove that the [organization/agency] was eligible to sponsor SFSP?

2. How did you collect that information?

8. Do you usually attend the annual training that the State agency hosts on the SFSP?

[If yes]

1. Is that annual training usually in-person, remote, a webinar, or some other format?

   1. If you were the decision-maker, would you keep the training [in person/remote/via webinar] or would you prefer to be trained another way?

2. What do you like most about the annual training?

3. What do you think the State could improve?

9. Who else on your team participates in the annual trainings?

[Interviewer note: we’re looking for the positions of the people who participate, not actual names.]

1. Does everyone receive the same training, or does the type of training a person receives depend on their position?

   1. [If the training varies based on a person’s position] Please tell me about the different types of training offered to people in different positions.

[If NO other staff attend]

2. When you return to your organization, what information from the training do you pass on to the rest of your team?

10. Thinking about the SFSP sponsors who are brand new to the program, what topics do you think are most important to focus on with them during training?

11. What training materials or toolkits does the State make available to you?

Probe: to train new staff or keep experienced staff up to date

1. How exactly are those materials shared with you?

Probe: via email, in hard copy, posted online

2. Which of the resources do you feel are most useful? Why?

3. Which of the resources do you feel are least useful? Why?

4. What other resources do you wish you had from the State?

5. Do you use any resources from other sources not from the State? If so, what are they?

12. Apart from the annual training, does the State offer any other trainings on SFSP at any point during the year? If yes, describe.

    1. When are those trainings held?

    2. What topics are covered?

    3. What is the format for the training?

Probe: in-person, webinar, online modules, conference call, other?

1. [If online training module] Are the online training modules self-paced?

13. Apart from training and resources, what other technical assistance does the State provide?

    1. Do you find that assistance useful? Why or why not?

Site Identification and Training

14. When looking for potential sites how do you select which ones to reach out to?

    1. Does the State provide any resources to help you identify sites? Describe.

    2. Does the State provide any tools or resources to help you recruit sites? Describe.

    3. Has the State ever passed along information about sites that contacted the State to express interest in participating in SFSP?

       1. [If yes] Tell me a bit more about what happened.

15. Do organizations ever approach you to sponsor them as an SFSP site?

16. Have you received any resources from advocacy groups to help identify or recruit sites?

    1. [If yes] What types of resources did they provide?

17. Once you have identified a potential site, what steps do you take to determine that the site is eligible to participate in SFSP?

Probe: What documents are collected? What observations are recorded?

1. What information do you send to the State to show that a site is eligible?

18. Tell me about the training you provide to your sites.

1. When do you usually offer training?

2. Is it in person, via webinar, or some other mode?

3. What topics are covered in the training?

4. Do participants complete any pre- or post-assessments? Describe.

5. [If oversees multiple site types or any mobile sites] Are there any differences in the way you train your different sites? If yes, describe.

19. Who from the sites participates in the trainings?

[Interviewer note: we’re asking about the positions of those who participate, not actual names.]

1. Is attendance required for particular staff? If so, which staff?

2. How do you track attendance?

20. Thinking about SFSP sites who are brand new to the program, what topics are most important to focus on with them during training? Why?

21. What resources – such as toolkits or training materials – do you provide to your sites?

    1. Did you develop these yourself or borrow the materials from elsewhere?

22. What aspects of the SFSP do sites have the most questions about?

    1. Is this all sites or particular sites? (PLEASE EXPLAIN)

       1. [If particular sites] Which kinds of sites typically have these questions?

Sponsors’ Monitoring and Verification Practices

In this next section, we are interested in learning more about how sponsors oversee their sites.

23. To start, tell me how you monitor sites’ compliance with health and sanitation standards.

1. Who conducts the health inspections and meal quality tests at your sites?

Probe: local health department, governmental agency, independent contractor

2. When and how often are the inspections conducted?

3. How do you document those inspections and any resulting reports from health inspections?

4. How do you track corrective actions taken after reported violations?

5. What documentation do you send to the State regarding health inspections?

6. [If sponsor oversees multiple types of sites]: Are there any differences in the way you monitor your different sites? If yes, describe.

[Interviewer note: we’re particularly interested in how the process might differ for mobile sites]

24. What kind of on-site monitoring of your sites do you conduct?

1. How often do you visit your sites?

2. How do you decide which sites to visit?

3. What are you looking for during those visits?

   1. What are the most common mistakes you see when you visit your sites?

25. [if work with mix of affiliated and unaffiliated sites] What differences are there in how you monitor the sites that are part of your general organization compared to the sites that are not part of your organization?

What are the challenges for you as the sponsor organization to conduct the on-site monitoring visits of your sites?

Now I want to move on to talking about meal count records.

26. How do sites submit their meal counts?

Probe: email files, submit hard copies, submit via online data system

1. How often do they submit the meal counts?

27. [if sites submit via online data system or similar] Does the system have any built-in edit checks?

[If Yes]

1. What do those edit checks look for?

2. What happens if the edit checks identify a potential error?

Probe: Will a warning message appear? Will a field turn red? Do sites have to address the error prior to submission?

28. Do you review your sites’ meal counts before submitting them to the State?

[If No:]

1. Tell me a bit about the reasons why not.

Probe: they trust an electronic system to do the math, no time or staff to review, other?

[If Yes:]

2. What are you looking for in that review?

3. How often do you review the meal counts?

4. What are the most common errors you have found in the meal counts?

   1. How are you able to spot those errors?

5. What strategies have you found effective in working with sites to reduce meal count errors?

29. I understand some sites have pretty consistent attendance from one day to the next, whereas other sites can have big fluctuations in attendance. How common is it for a site to submit the same meal counts every day for an extended period of time?

[Interviewer note: an “extended period of time” would be a couple of weeks or longer.]

1. What kinds of follow-up, if any, do you do when this happens?

[If necessary: In other words, do you ever follow up with a site to ask about their meal counts?]

30. What would you say is the biggest challenge to collecting accurate meal count data from sites?

Probe to understand whether their response applies to all sites or specific types of sites.

1. [if work with mix of affiliated and unaffiliated sites] More specifically, are there any difficulties in collecting information from the sites that are not a part of your general organization? If so, describe.

31. How do you and your sites work to [order/prepare] only as much food as needed, and not have too many meals?

32. In addition to reporting meal counts to the State, do you also report average daily attendance across all your sites?

    1. [IF YES]: How do you calculate average daily attendance? [Probe for accuracy of current method and suggestions for better methods that produce a more accurate figure (e.g., counting children instead of meals; accounting for more than one meal service; accounting for operating days differently). Also probe to determine burden of providing ADA on a monthly basis.]

33. How do you keep track of day-to-day operations such as number of meals served at each site each day it is open?

34. Next, I will ask about some specific issues that sponsors have run into with their sites. If you have experienced any of them, I would like to hear if you have found any effective strategies in working with sites to minimize the incidence of those issues.

• sites serving meals outside of the approved times

• meals being served at locations other than the approved location

• sites not operating under the site type for which they were approved (i.e., open, restricted open, closed enrolled, camp, migrant)

• sites allowing children to take their meals offsite

• sites serving incomplete meals

• sites counting incomplete meals

• sites that count meals served to adults

• sites that count second meals

• unauthorized use of reimbursement funds (e.g., educational activities or organized sports) (Interviewer note: This may not apply when the sponsor and site are affiliated, since the sponsor may not give the site reimbursement funds, they only provide them their meals)

35. What are the consequences if a site does not improve?

36. [Ask only of sponsors that operate more than one type of site]: Are there certain types of sites that appear to struggle more with keeping records? If yes, which ones?

Probe: open, restricted open, closed enrolled, camps, migrant; longstanding v. new sites; those in certain geographic areas.

1. Why do you think this site type has more difficulty?

2. What, specifically, could these sites improve upon?

3. What work has been done with these sites to address those challenges?

   1. Which of those strategies do you think is most effective in improving the quality of their records?

State Agency On-Site Review

37. As you know, State agencies must make sure all sponsors and sites follow SFSP regulations, in part by conducting sponsor and site reviews. When was the last time the State conducted a review of your [organization/agency]?

1. What occurs during those reviews?

2. Do those reviews pose any challenges for [organization/agency], either before, during or after the visit?

38. We are interested in learning how the State agency identifies and follows up on any findings from their reviews. Can you think of a time when the State identified one or more issues that required corrective action? Describe.

1. What corrective actions were taken?

2. How long did it take you to make those corrections?

3. How did you document corrective action to the State?

4. Describe any follow-up monitoring conducted by the State.

Integrity Challenges and Wrap Up

39. Thinking about all the data that you’ve told us you receive from sites, which data do you most closely review?

    1. How did you learn that was important to review?

40. [sponsors that operate across more than one State] What presents the greatest challenge for your [organization/agency] as you operate the SFSP across States?

Probe: submitting data to multiple agencies, navigating policy differences across States

1. What strategies have you found to overcome that?

41. Thinking about the various program requirements placed on SFSP sponsors, is there any requirement that seems excessive to you? Explain.

    1. How would you revise that requirement?

42. Similarly, is there any requirement that seems outdated based on how the SFSP is currently operated?

    1. How would you revise that requirement?

43. Reflecting on everything we’ve discussed, what are you most proud of with regard to your summer meals program?

44. Those are all the questions I have. Is there anything else you would like to tell me about the SFSP?

Discussion of Next Phase of Data Collection

Before we end, I would like to tell you a bit about the next phase of the study, because it involves reaching out to one of your sites. In the next and final phase of the study, we are seeking input from SFSP site supervisors. Because of their direct role in making sure eligible children have access to healthy meals, site supervisors can provide a unique perspective on program challenges and helpful strategies for addressing those challenges. We would like to interview one of your sites, and the conversation would last no more than 30 minutes

We received a list of your sites from the State, and I want to go over that list and confirm the names of your sites to make sure our records are accurate. From each sponsor that we speak with, we will select one site for a 30-minute interview.

[Interviewer refer to list of site names and make note of any changes]

Thank you so much for your participation. We will let you know when we have selected which site we would like to contact for a 30-minute interview and ask for your help in contacting them.