Appendix C-7 SFSP Integrity Pre-Test Memo

APPENDIX C-7. SFSP INTEGRITY PRE-TEST MEMO

	Summer Food Service Program (SFSP) Integrity Study Pre-Test Memorandum
	March 8, 2019

This page is intentionally blank

Table of Contents

1. Introduction and Methods 1

1.1 Recruitment Methods 1

1.2 Data Collection 2

1.3 Data Analysis 3

2. Summary of Recommendations 4

2.1 State Child Nutrition Director Survey 4

2.2 State Child Nutrition Director Interview Guide 7

2.3 Sponsor Interview Guide 11

2.4 Site Supervisor Interview Guide 16

Introduction and Methods

1

The U.S. Department of Agriculture’s Food and Nutrition Service (FNS) has contracted with Westat to conduct the Summer Food Service Program (SFSP) Integrity Study. The study will provide FNS with important information about how State agencies oversee the SFSP to ensure program integrity. The research objectives of the study are to:

1. Examine and describe how State agencies administer and provide SFSP oversight and why they believe it is effective. Probe for evidence demonstrating the effectiveness of the States’ oversight processes.

2. Identify SFSP integrity challenges common across States and types of sites in the administration and oversight of SFSP.

3. Identify existing or State-recommended resources, training, or technical assistance that would better support State agencies in their effective administration and monitoring of the SFSP.

In preparation for study launch, Westat tested the State SFSP web-survey and the guides for the State, sponsor, and site interviews. The versions of the instruments we tested reflected edits made based on FNS comments on the draft instruments. The goals of the testing were to ensure that (1) respondents interpret the questions as intended and can easily respond; and (2) interviewers can easily administer the instruments. There were nine respondents for this data collection. Each respondent interview lasted between 60-90 minutes over telephone and were audio-recorded for later review and analysis.

1.1 Recruitment Methods

Westat, with FNS approval, selected nine respondents for pretesting, as displayed in Table 1.1 below.

Because the State survey will be completed by all State Child Nutrition (CN) Directors, we did not want to test the State survey or interview guide with a State CN Director. Instead, two FNS Regional Office staff and one State-level CN Assistant Director were selected to test the State-level instruments.

Table 1.1. Pretest respondents

Study Instrument(s)	Pretest Respondents
State Survey and Interview Guide	2 FNS Regional Office staff 1 State Child Nutrition Assistant Director
Sponsor Interview Guide	1 SFA Sponsor 1 Government Sponsor 1 Nonprofit Sponsor
Site Interview Guide	1 School Kitchen Manager 1 Program Manager at a Local Organization 1 Residential Camp Staffer

For pretests of the sponsor interview guide, the research team contacted sponsors who had provided meaningful input on a previous summer meals study to ask if they would be willing to participate. To ensure the interview guide was appropriate for the various sponsor types we expect to recruit in the main study, we pretested the instrument with one school food authority, one government organization and one private nonprofit organization.

Finally, to recruit site supervisors for pretests of the site interview guide, we first asked the three sponsors participating in a pretest to recommend one of their site supervisors who they felt would be able to provide helpful feedback on the site instrument. Two of the three sponsors recommended supervisors from sites they sponsor, and provided contact information. Because the third sponsor could not refer us to a site supervisor, we used a list of site supervisor contacts from a previous summer meals study to identify and recruit a third respondent for the pretest.

1.2 Data Collection

Trained interviewers conducted the pretest interviews by telephone. The interviews included the following:

• The interviewer administered the study introduction, explaining the study purpose and the respondent’s rights as a research subject.

• Respondents were asked for their verbal consent after interviewers explained the voluntary nature of their participation and confidentiality, and requested that the interview be audio recorded.

• The interviewer asked the relevant respondents for feedback on the State survey, with specific probes for survey questions that were flagged by either the respondent or interviewer ahead of time.

• The interviewer administered the relevant interview guides and administered scripted probes in addition to asking for reactions to specific words and questions.

• At the conclusion of each interview section, the respondents were provided an opportunity to offer any additional feedback or reactions.

• After the end of the entire pretest interview, the respondent was thanked for participating.

We tested the qualitative data collection instruments by administering them as written and observing how respondents responded, noting any difficulties they encountered. All pretest interviews were conducted over the phone. Interviewers administered the interview guides, and observed and documented issues that arose for both respondents and interviewers.

The process to pretest the State survey instrument differed from the pretests of the qualitative interview guides. To accommodate respondents’ schedules, the pretest telephone interview with the three respondents who pretested the State-level instruments covered both instruments in one phone call. The respondents received the survey instrument in advance and were asked to complete it prior to the scheduled telephone interview. During the telephone interview, the interviewer administered the State interview guide following the process described above. Once that pretest was complete, the interviewer reviewed the respondents’ completed surveys and probed into the following: 1) the overall experience and time burden to complete the survey; 2) responses that did not match our expectations; 3) the exhaustiveness of the response options provided; and 4) suggestions to improve the wording of particular questions or response options.

1.3 Data Analysis

Interviewers also served as analysts. They reviewed the interview recordings and their own notes to identify themes and patterns within respondents’ feedback, particularly any challenges in administering or responding to the instruments. The interviewers discussed the results of the analysis to reach consensus on the edits that needed to be made. Themes and patterns were organized and summarized into report form.

Summary of Recommendations

2

The remainder of this report summarizes the issues found in each tested instrument and provides recommendations for addressing the issues.

2.1 State Child Nutrition Director Survey

State CN Director Survey: Pretest Findings	Changes Made
Section A: State Agency Administration	Section A: State Agency Administration
Items: 1. How many staff in the State agency work on the Child Nutrition Programs administered by the State agency, including SFSP? Please provide the number of full-time equivalents (FTEs) for CNPs. 2. How many staff in the State agency work on SFSP? Please provide the number of full-time equivalents (FTEs) for SFSP. A respondent suggested rephrasing the questions to ask about the number of FTEs included in each State’s SAE and SAF plans. We added a note to the question to reference SAE and SAF plans as a source of this information.	Revised items: 1. Please provide the total number of full-time equivalent (FTE) State-level staff who work on the Child Nutrition Programs, including the SFSP? Use decimals if needed (e.g., 1.5 FTEs). Note: If your State’s SAE plan is current, please provide the total FTEs listed in the approved SAE plan. A2. Please provide the total number of full-time equivalent (FTE) State-level staff who work on the SFSP. Use decimals if needed (e.g., 1.5 FTEs). Note: If your State’s SAF plan is current, please provide the total FTEs listed in the approved SAF plan.
Section C: Training and Technical Assistance	Section C: Training and Technical Assistance
Items: Did the State agency provide a “refresher” training for sponsors at any time during summer 2019? In what format did the State agency provide refresher training for sponsors? Respondents were not familiar with the term “refresher” training.	Revised items: Apart from the annual training, did the State agency provide any additional training(s) for SFSP sponsors during 2019? [If yes] In what format did the State agency provide this additional training for sponsors?
Item: During summer 2019, which 3 topics did your sponsors have the most questions about for SFSP? These are the topics for which you provide the most frequent technical assistance. One respondent suggested re-wording two of the response options—“completing reimbursement forms” and “estimating meal counts”—for clarity.	Revised item: (response options) -Completing claims for reimbursement -Estimating yearly projected reimbursement
Section D: Program Resources	Section D: Program Resources
Item: Explain why the following resources are neither used nor provided. -Unfamiliar with this Resource -Not Found to be Useful -Distribute Similar Guidance Produced by the State -Other A respondent indicated that the “other” category was not necessary, as the three other response options would cover all situations.	Revised item: Removed the “other” category
A respondent suggested a new item be added to capture how the State agency disseminates information to sponsors and sites, as that can differ from State-to-State.	Added item: How does the State agency disseminate these resources? - Posted on State agency’s public website - Posted on private State portal that sponsors and sites can access -Hard copies distributed at trainings -Shared via email -Other
Section E: Sponsor and Site Approval Process	Section E: Sponsor and Site Approval Process
Item: What is the most common reason for disapproving a sponsor’s application? Respondent shared they have not disapproved a sponsor in the last five years. Suggested we add not applicable to the response options.	Added item: Not applicable
Items: How does the State agency confirm each proposed site’s area eligibility? How does the State agency confirm each closed enrolled site’s eligibility? How does the state agency confirm each camp’s eligibility? How does the State agency confirm that a site is rural? A respondent suggested replacing “confirm” with “verify” for all of these questions. The respondent also shared that there is a more manual process of mailing in or sending data in addition to uploads and that States can check against a census map when verifying that a site is rural. Changed wording and added response options.	Revised items: How does the State agency verify each proposed site’s area eligibility? How does the State agency verify each closed enrolled site’s eligibility? How does the state agency verify each camp’s eligibility? How does the State agency verify that a site is rural? Added response options: - Sponsor submits hard copy (paper) area eligibility documentation and State staff review and confirm - Sponsor uploads eligibility information from local schools and State staff review and confirm - Sponsor mails in documentation to the State and State staff upload - State checks the site location against census map
Section F: Program Monitoring	Section F: Program Monitoring
Item: To what extent does the State agency conduct unannounced reviews of sites? The respondent wasn’t clear on what “unannounced” meant in this context. We added a definition.	Added item: Note: by unannounced we mean that neither a site nor its sponsor is aware of the review in advance.
Item: Which of the following expenditures of SFSP funds are most frequently denied by the State agency? The respondent suggested additional response options. We also revised the root question to be more specific to summer 2019.	Revised item: For summer 2019, which of the following expenditures of SFSP funds did the State agency most frequently deny? Added response options: Excessive salary Not applicable
Section H: Meal Disallowances	Section H: Meal Disallowances
One respondent suggested we add a question about the total number of meals claimed in the State, so that we can get to the proportion of meals disallowed compared to the total.	Added items: How many breakfasts were claimed by the State in summer 2019? How many lunches and suppers were claimed by the State in summer 2019? How many snacks were claimed by the State in summer 2019?
Section I: Program Terminations and Turnover	Section I: Program Terminations and Turnover
The respondent suggested we had a question to find out how many sponsors determined seriously deficient in the prior year did not return.	Added item: How many SFSP sponsors that were flagged as “seriously deficient” during summer 2018 did not return to operate the program in summer 2019?

2.2 State Child Nutrition Director Interview Guide

State CN Director Interview: Pretest Findings	Changes Made
Global Issues
Pretests revealed that the interview would likely take 90 minutes rather than the 60 minutes originally planned.	Consent language has been updated to reflect 90 minutes.
A respondent suggested having an introductory sentence at the beginning of each section to improve the flow of the interview.	We added an introductory statement at the beginning of each section describing what will be discussed.
Section: Outreach and Sponsor/Site Approval	Section: Determining the Eligibility of Sponsors and Sites
Item: When an organization applies to be considerd as an SFSP sponsor, how do you determine whether the organization is an eligible sponsor? A respondent suggested we ask separate questions about first-time sponsors vs. existing sponsors, since there are differences in the application process. Revised original question and added a subsequent question about returning sponsors.	Revised item: When an organization applies for the first time to be considered as a SFSP sponsor, how do you determine whether the organization is eligible? New item: What documentation do existing sponsors have to submit each year in order to be approved to continue their program?
Item: What would you consider to be the most important factor when determining whether an organization can operate successfully as an SFSP sponsor? Why? Item revised based on respondent feedback provided in the previous item.	Revised item: In your opinion, what is the most important factor to consider when determining whether an organization can operate successfully as a first-time sponsor? Why?
Item: What is the process to determine whether a potential closed enrolled site is eligible? A respondent suggested we focus on differences in the process for closed enrolled sites compared to the previously discussed process for open sites, for efficiency and ease of discussion.	Revised item: How, if at all, is the process different when determining whether a potential closed enrolled site is eligible?
Section: Providing Resources and Training	Section: Providing Resources and Training
Item: I see from the survey that the State provides annual training to sponsors on [list topics mentioned in the survey]. Which topics are emphasized the most in the annual training? How, if at all, is the training tailored to the different sponsors? One respondent shared that they “got lost” when the interviewer was reading the list of topics mentioned in the survey. In addition it was awkward for the interviewer to read the list of topics. We removed the list and separated questions to more smoothly elicit information on all aspects of training.	Revised item: [If respondent enters more than “1” for survey question C1]. You told us in the survey that the State holds multiple annual trainings for sponsors. Tell me a bit about the different trainings. Item “a” was revised to become a root question: I also see from the survey that the State provides annual training to sponsors on a range of topics. Which topics would you say the State emphasizes the most in the annual training? Item “b” was revised to become a root question: How, if at all, is the training tailored to different sponsors? Probe: first-time v. existing sponsors; sponsor type (camp, SFA, local government, etc.); sponsors in different geographic areas; affiliated v. unaffiliated.
Section: State Monitoring of Sponsors and Sites	Section: State Monitoring of Sponsors and Sites
Item: How do you select which sponsors you will review each year? What challenges does the State face in conducting the required number of on-site sponsor reviews? One respondent was confused about whether the question was asking about State reviews of sponsors or other, less formal activities the State may conduct (e.g., desk review of records).	Revised item: How do you select which sponsors you will review each year, to meet the regulatory requirement for reviews? Probe to understand the selection criteria and process. Subquestion (a) becomes a separate question.
Item: What is the process you follow if you have to disallow meals? How does the State agency track the meal disallowances to make sure the sponsor doesn’t include them in their claim? Same comment as above about State reviews vs. less formal activities. Revised question to ask if different for non-review meal disallowances.	Revised item: When you’re conducting an on-site review, what is the process you follow if you have to disallow meals? How does the State agency track the meal disallowances to make sure the sponsor doesn’t later include them in their claim? How is the process different when disallowing meals following an off-site review of a sponsor’s meal count records?
A respondent suggested we ask if the reviewers look at prior reviews before going on site, since there are several staff that conduct reviews and the reviewer may not have visited the sponsor previously. We added a general question about review preparation and included probes about the types of materials that could be examined.	New Item: Before the reviewer goes on site, what materials do they examine to prepare for the review. Probes: prior review findings, claims, requests for technical assistance, complaints.
Item: What happens if you find something amiss during a sponsor or site review? Respondents shared that the term “amiss” wasn’t clear. Revised to “compliance issue.”	Revised item: What happens when you find a compliance issue during an on-site sponsor or site review? How does the State follow-up on these issues to ensure corrective action is taken?
Section: Variation in Sponsor Administration of SFSP	Section: Variation in Sponsor Administration of SFSP
Item: Tell me about any differences you see with regard to how different types of sponsors work with their sites to implement the SFSP. Probe: SFAs, nonprofits, local government, and camps; longstanding v new SFSP sponsors; those in certain geographic areas, sponsors with a lot of sites v those with only a few sites; sponsors who are affiliated with their sites vs those who are not. Respondents thought this question was too broad. It was unclear to them what was being asked so they had difficulty responding. We revised the question to focus on sponsors with affiliated sites vs. unaffiliated, since that is a key factor that differentiates how sponsors interact with sites. Subsequent questions probe for distinctions between other types of sponsors, geographic areas, etc.	Revised item: What differences do you see in how affiliated sponsors work with their sites to implement the SFSP compared with unaffiliated sponsors? Do you adjust the way you oversee your sponsors to account for these differences?
Item: Which types of sponsors do you feel could do a better job of training their sites? Probe: SFAs, nonprofits, local government, and camps; longstanding v new SFSP sponsors; those in certain geographic areas, sponsors with a lot of sites v those with only a few sites; sponsors who are affiliated with their sites vs those who are not. Respondents did not like “could do a better job” and thought this question could be more direct.	Revised item: Which sponsors have more difficulty with training their sites? Probe: nonprofits, local government, camps, and SFAs; longstanding v. new SFSP sponsors; those in certain geographic areas; sponsors with a lot of sites v. those with only a few site; sponsors who are affiliated with their sites vs. those who are not What, specifically, could they improve upon? What kind of support could help those sponsors to improve their training? Probe: training, TA, other resources
Section: Collecting and Reviewing Program Data	Section: Collecting and Reviewing Program Data
Section: Integrity Challenges	Section: Integrity Challenges and Wrap-Up
Items: [if applicable, see survey question F6] You told us in the survey that the State most closely monitors [FILL] among new sponsors in their first year or two of SFSP. Why is that important to monitor among new sponsors? [if applicable, see survey question F7] Similarly, you mentioned that the State most closely monitors [FILL] among sponsors that have administered the SFSP for at least two years. Why is that important to monitor, specifically among the experienced sponsors? How would you describe the strategies that the State uses in working with non-compliant sponsors? You mentioned in the survey that [list the challenge(s) they provide in the open-text response to survey question K1] present a challenge to program integrity in the SFSP. Which of these do you feel is the most significant challenge to program integrity in your State? How has the State tried to address that challenge? Was that approach effective? Respondents found these questions tedious (and some had been touched upon earlier in the interview) and suggested we streamline and focus on first-time sponsors and their integrity issues.	Revised item: In your opinion, what is the biggest program integrity issue among first-time sponsors? Are there any strategies that have helped to address those issues?

2.3 Sponsor Interview Guide

Sponsor Interview Guide: Pretest Findings	Changes Made
Global Issues
Overall, respondents who operated feeding programs during other parts of the year (e.g., CACFP, NSLP) had difficulty distinguishing SFSP trainings and requirements from those of other programs solely from memory. Respondent commented that some of the questions (e.g., 12b, 27, 36a) read like USDA requirements that would be the same for everyone rather than asking respondents specifics about how they accomplish meeting those regulations.	Interviewer needs to be prepared to continuously remind respondent to clarify when they are unsure if a certain program element is SFSP-specific. Interviewers will be trained to rephrase these questions if they are interpreted as asking about what the rules are to elicit responses more related to specifically how sponsors meet the requirements of the program.
Warm Up
Item: Approximately how long has your (organization/agency) been a SFSP sponsor? As with the sites, some sponsors that are SFAs might have been sponsoring SFSP during some summers and SSO during other summers. We added a probe to make it clear we are asking about SFSP.	Revised to: Approximately how many years has your (organization/agency) been sponsoring summer meals? Your best estimate is fine, but it’s also fine to say you don’t know. a. [SFA sponsors only] How many of those years were SFSP as opposed to the Seamless Summer Option (SSO)?
Item: What types of sites do you sponsor (e.g., open, restricted open, closed enrolled, camps, migrant)? Respondent said that without the probe, she would have responded as affiliated or nonaffiliated. Suggested explicitly defining what we mean by type of sites.	Revised to: Sponsors may operate different types of sites, such as open sites, restricted open sites, closed enrolled, camp sites, and migrant sites. Which of these best describes the sites you sponsor?
Sponsor Application and Training
Item: Thinking about the last annual training you received, what topics did the State cover in that training? Two of the three respondents had a little bit of trouble remembering all of the topics because there were several.	Question changed because we ask about topics in the survey. Instead of asking the respondent to recall topics, question has been changed so it does not rely on respondent’s memory but instead gathers information about what topics should be covered: Thinking about the SFSP sponsors who are brand new to the program, what topics do you think are most important to focus on with them during training?
A respondent helping to pretest the State-level interview guide shared that a barrier to SFSP administration is that the information from the State-led annual trainings is not always disseminated from the director who attends the training to the other staff they work with.	Added item to the sponsor interview guide to ask about this: Who else on your team participates in the annual trainings? [If NO other staff attend] When you return to your organization, what information from the training do you pass on to the rest of your team?
Site Identification and Training
Item: How do you identify potential sites? Respondent commented that this question was only relevant to sponsors in a growth phase and suggested rewording. Respondent explained that they are not currently identifying new sites.	Revised to: When looking for potential sites, how do you select which ones to reach out to?
Item: In what ways, if any, does the State agency help to find SFSP sites? Respondent believes that it is illegal for States to help sponsors find SFSP sites. This question was re-worded to clarify the intent.	Added question: Has the State ever passed along information about sites that contacted the State to express interest in participating in SFSP? [If yes] Tell me a bit more about what happened.
Item: How do you determine that each site is eligible to participate in SFSP? (Probe: What documents are collected? What observations are recorded?) Which of these data on the sites do you submit to the State? Respondent was thrown off by the way the word “data” was used in this context. Suggested revising the wording to “What information do you send to the State to show that a site is eligible?”	Revised to: Once you have identified a potential site, what steps do you take to determine that the site is eligible to participate in SFSP? Probe: What documents are collected? What observations are recorded? a. What information do you send to the State to show that a site is eligible?
Item: Do organizations ever approach you to be considered as SFSP sites? Respondent interpreted this question as sponsors approaching them rather than sites. Respondent suggested re-wording to make it clear we are asking if potential sites ever approach the sponsor.	Revised to: Do organizations ever approach you to sponsor them as an SFSP site?
Item: Have you received any resources from advocacy groups regarding identifying or recruiting sites? Respondent answered yes/no to this question.	Added a subquestion: [If yes] What types of resources did they provide?
Item: Tell me about the training you provide to your sites. When do you usually offer training? Is it in person, via webinar, or some other mode? What topics are covered in the training? Who from the sites participates in the trainings? I don’t need names, I am just asking about the positions of those who participate. Is attendance required for particular staff? How do you track attendance? Do participants complete any pre- or post-assessments? Describe. Respondent identified slight differences in how they train and monitor mobile sites compared to non-mobile sites.	Added a subquestion: [If oversees multiple site types] Are there any differences in the way you train your different sites? If yes, describe.
Sponsors’ Monitoring and Verification Practices
Item: To start, tell me how you make sure sites meet all health and sanitation standards. Who conducts the health inspections and meal quality tests at your sites? Probe: local health department, governmental agency, independent contractor When and how often are the inspections conducted? How do you document those inspections and any resulting reports? How do you track corrective actions taken after reported violations? What do you send to the State? Respondent suggested making it clear that the subquestions all refer to health and sanitation inspections, and not the sponsor’s on-site monitoring “inspections.” We further revised this question to focus on health inspections and eliminate the double-barreled wording, and to ask about differences by site type.	Revised subquestions to: Who conducts the health inspections and meal quality tests at your sites? Probe: local health department, governmental agency, independent contractor When and how often are the inspections conducted? How do you document those inspections and any resulting reports from health inspections? How do you track corrective actions taken after reported violations? What documentation do you send to the State regarding health inspections? [If sponsor oversees multiple types of sites]: Are there any differences in the way you monitor your different sites? If yes, describe. [Interviewer note: we’re particularly interested in how the process might differ for mobile sites]
Items: How do you review your sites’ meal counts for accuracy? What are you looking for in that review? How often do you review the meal counts? What are the most common reasons that errors are found in the meal counts? What strategies have you found effective in working with sites to reduce the number of errors in their data? Respondent said the answers to these questions would vary depending on the mode in which sites submit meal counts – email, hard copy, online data system, and may only apply to sites that used paper forms. Based on this feedback, we also added a screening question to first understand from the respondent the mode of submission – email, hard copy, online data system – to help the interviewer frame the subsequent questions.	Added the question: How do sites submit their meal counts? Probe: email files, submit hard copies, submit via online data system Do you review your sites’ meal counts before submitting them to the State? If yes: What are you looking for in that review? How often do you review the meal counts? What are the most common errors you have found in the meal counts? What strategies have you found effective in working with sites to reduce meal count errors?
Item: How do you add up meal claims data from across sites and calculate average daily attendance? Two of the three respondents explained that this question was confusing because as far as they know this does not happen. They explained their understanding that averages are calculated for each site, not across sites.	Revised to: In addition to reporting meal counts to the State, do you also report average daily attendance across all of your sites? [IF YES]: How do you calculate average daily attendance?
Item: What happens if you or your staff identify something is amiss at one of the sites, either through their data or through site visits? Probes: Sites serving meals outside of the approved times Meals being served at unauthorized locations Sites not operating under site type for which they were approved (e.g., open, restricted open, closed enrolled, camp, migrant) Sites allow children to take their meals offsite Sites serving and claiming reimbursement for incomplete meals Sites claiming reimbursement for meals served to adults Unauthorized use of reimbursement funds (e.g., educational activities or organized sports) Respondent and interviewer found these probes confusing the way they are organized. Suggested breaking them out into a separate question where the probes are sub-questions and ask the respondent about each one. Respondent was confused about what we meant by “unauthorized use of reimbursement funds,” whether it pertained to sponsors reimbursing sites or States reimbursing sponsors.	This question has been re-worded and combined with the following item. See below for new item wording.
Item: [For each integrity problem mentioned from the list above, ask]: What strategies have you found effective in working with sites to minimize the incidence (insert specific integrity problem)?	Revised to: Next, I will ask about some specific issues that sponsors have run into with their sites. If you have experienced any of them, I would like to hear if you have found any effective strategies in working with sites to minimize the incidence of those issues. Probes: sites serving meals outside of the approved times meals being served at locations other than the approved location sites not operating under site type for which they were approved (i.e., open, restricted open, closed enrolled, camp, migrant) sites allowing children to take their meals offsite sites serving and claiming reimbursement for incomplete meals sites claiming reimbursement for meals served to adults unauthorized use of reimbursement funds (e.g., educational activities or organized sports) (Interviewer note: This may not apply when the sponsor and site are affiliated, since the sponsor may not give the site reimbursement funds, they only give them their meals).
Integrity Challenges
Item: In your opinion, what are the primary challenges to monitoring your sites? Respondent asked if the question was getting at onsite monitoring and suggested specifying “while conducting a visit” if this is that the question is looking for. For flow, we moved this question earlier to follow other monitoring questions. Respondent also suggested that there are different challenges for sponsors who operate across State lines, and we added a question about that.	Revised item, which was moved earlier in the guide: What are the challenges for you as the sponsor organization to conduct the on-site monitoring visits of your sites? [Sponsors that operate across more than one State] What presents the greatest challenge for your [organization/agency] as you operate the SFSP across States? Probe: submitting data to multiple agencies, navigating policy differences across States

2.4 Site Supervisor Interview Guide

Site Supervisor Interview: Pretest Findings	Changes Made
Global Issues
Two of the three site pretest respondents were not certain which organization was considered the “sponsor.”	Interviewer will identify the sponsor in advance of each site supervisor interview and enter the sponsor name into the backgound information summary. If any respondents are uncertain about who their sponsor is, the interviewer can refer to that sponsor by name.
Warm up
Item: Approximately how long has [SITE] been serving summer meals? Your best estimate is fine, but it’s also fine to say you don’t know. One participant had been doing SSO for several years, and SFSP for one summer only. They were not certain whether to just report only the SFSP years, or both SFSP and SSO years.	Added a note to the question: [Interviewer note: If the site’s sponsor is a School Food Authority (SFA), then it is possible they operated the Seamless Summer Option (SSO) for part of the time they have been serving summer meals. If the site mentions this, ask them how many years were specifically SFSP]
Start Up and Training
Item: Please tell me about the training that you received from [SPONSOR NAME]. All three respondents asked whether we were specifically asking about training for SFSP, or about training for any child nutrition program they operate. For example, the participant from a school food authority operated NSLP during the school year. Interviewer clarified that we were interested in SFSP specific training.	Revised to: Please tell me about the training that you received from [SPONSOR NAME] for the summer meals program.
Item: Have you ever received any written guidance or resources from your sponsor? [If Yes] Please describe those resources. Which of these have you found most useful? One respondent was not certain how to answer because written materials had been received from a community partner, but not from sponsor, which caused them to question which entity was their sponsor. See Global Issues. Another respondent struggled to come up with examples for this because they were not sure which guidance comes from the sponsor or the State or partner organizations. One participant found subquestion b confusing. Suggested we clarify “useful for what”? Boosting participation? Meeting compliance? Reducing resource strain?	Background information summary will include the name of the sponsor organization, in case any site supervisors are not certain who their sponsor is. We added placeholders for questions involving sponsors so the interviewer can refer to the sponsor by name, and added a lead-in to this question to better explain our intention. Revised to: Sometimes an organization like yours will receive written guidance or resources to help then run their summer meals program. For example, some sites receive a written guide that explains the program rules, or a form to help keep track of meals served. Have you ever received any written guidance or resources from [SPONSOR NAME]? [If Yes] Please describe those resources. Which of these have you found most useful? Probe: For boosting participation, meeting program requirements, and reducing any staffing strain that might occur?
Monitoring Site Operations
Item: Please tell me a little about the health inspections process. Two respondents asked the interview to clarify whether the question was specifically related to health inspections for SFSP. One other respondent was part of a mobile site, and was confused by the question. They thought the question should only be asked if the site had an SFSP-specific health inspection, which may not happen for mobile sites.	Revised to: Please tell me a little about the health inspections process for the summer meals program. (Note to interviewer: This may not apply to mobile sites)
Item: [If site uses a vendor] How do you adjust your orders with the vendor to make sure you don’t get too many or too few meals? One respondent said, based on their knowledge of the program, this is typically the sponsor’s responsibility, not the site’s.	Added screening question before: Does your site order the meals for your site, or does does [SPONSOR NAME] place the orders? Do you have any issues with the vendor delivering on time? [If Yes]: How frequently does this happen? [if site orders] How do you adjust your orders with the vendor to make sure you don’t get too many or too few meals? [if sponsor orders] How do you communicate with [SPONSOR NAME] when you need more meals or fewer meals?