State, Local and Tribal Agencies (Directors, SFA & Schools)

APPENDIX C-12. SCHOOL FOOD AUTHORITY GUIDE

OMB Number: 0584-NEW

Expiration Date: XX/XX/XXXX

Evaluation of the School Meal Data Collection Process

SFA

Interview Guide

Introduction

Good morning/afternoon. Thank you again for taking the time to talk with me today. My name is [INTERVIEWER’S NAME] and I work for Westat, a private research company in Rockville, Maryland. With me today is [NOTE TAKER’S NAME] who will be taking notes.

PURPOSE: The federal Food and Nutrition Service (FNS) is interested in understanding and improving the processes used to collect and report program data. FNS hired Westat to conduct a study to describe how schools, School Food Authorities (SFAs), and state agencies collect and report school meal program data via three forms:

• FNS-10, Report of School Program Operations,

• FNS-742, School Food Authority Verification Collection Report, and

• FNS-834, State Agency (NSLP/SNAP) Direct Certification Rate Data Element Report.

We will also identify where errors tend to occur during data collection and transmission. The final product will be a description of the data collection processes for these forms, and recommendations to improve the process and increase accuracy.

During our conversation today, we would like to ask you about the data you provide to the State agency for two FNS forms:

• FNS-10, Report of School Program Operations. This form includes monthly free, reduced price and paid meal counts , along with data from October on the number of schools participating in each of the meal programs and the number of children approved for free and reduced price meals. The data on this form supports the federal cash reimbursement you receive from the State.

• FNS-742, School Food Authority Verification Collection Report. This form includes the total number of schools and students in your SFA, the number of schools operating under a special provision, and detailed information on the results of the annual free and reduced price certification and verification process, It is submitted annually by the State to FNS.

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-NEW. The time required for the School Food Authority to provide this information collection is estimated to average 90 minutes per response, including the time to review instructions, search existing data resources, gather and maintain the data needed, and complete and review the collection of information.

HOW YOU WERE SELECTED: We are conducting four state-based case studies that include qualitative interviews with 4 State Child Nutrition and SNAP agencies, 10 SFAs in each State, and 3 schools per SFA in order to describe the data collection processes and gather information to better understand areas for improvement. Westat worked with FNS to select states based on factors such as state size, FNS region, use and type of management information system (MIS) and direct certification system processes.

INFORMATION TO BE COLLECTED: As a key staff in the Child Nutrition office, we will ask about your involvement with data aggregation, processing, and transmission for the FNS-10 and FNS-742. We will also ask about your collaboration with different departments and any suggestions you may have on how to improve these processes.

RISKS AND PRIVACY: There is little risk to being part of this study. We use all data we collect only for the purposes we describe. Your name will not be linked to any of your responses. In our reports we may include quotes from our respondents, but these will be presented without the speaker’s name and in such a way that you could not be identified. Participating in the survey may not help you individually, but it may help us better understand how to improve the completion of FNS-10, FNS-742 and FNS-834.

STUDY COSTS AND COMPENSATION: There is no cost to you to join this study. The interview takes about 90 minutes.

VOLUNTARY PARTICIPATION: Your participation is entirely voluntary. Refusal to participate will not have any impact on your position or child nutrition programs. You can take a break, skip questions or stop participating at any time.

[NOTE TAKER’S NAME] is here with me to take notes, but know that the information you share in this discussion will be kept private.

QUESTIONS: If you have questions about your rights and welfare as a research participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, the name of the research study that you are calling about, which is the Evaluation of School Meal Data Collection, and a phone number beginning with the area code. Someone will return your call as soon as possible.

We have planned for this discussion to last 90 minutes. Is that still okay?

With your permission I would like to record this discussion to help us fill any gaps in our written notes. The recordings, transcripts, and any notes we have will be stored on Westat’s secure server and will be destroyed after the project is complete.

Do you have any questions? [ANSWER ALL QUESTIONS]

If you agree to participate in this interview, please read the following statement and sign your name below.

I have read the above information about this project and my rights as a participant. I consent to participate in this research and to have this discussion audio-taped.

Signature Date

Printed Name

If you do not agree to be audio recorded, please cross out “and to have this discussion audio-taped” in the sentence above.

Do you have any questions? [ANSWER ALL QUESTIONS]

May I turn on the audio recorder now?

PROVIDE RESPONDENT WITH A COPY OF THE FNS-10 AND FNS-742 AS A REFERENCE. DURING INTERVIEW DETERMINE TERMINOLOGY SFA USES AND MODIFY QUESTION WORDING AS NEEDED.

Warm Up

1. To start, please tell me about your role at the SFA and how long you have worked here.

a. What are your specific responsibilities with regard to collecting information on school meal data?

Probes:

Monitor direct certification, manage data tracking system, monitor whether meals meet the requirements, oversee the claims process, compile school data, clean school data, oversee the data validation process.

Context

2. Based on the latest data we have, it appears that your SFA oversees [XX] schools and that [XX number] implement Provision 2 and [XX] implement Provision 3. Does that number sound accurate?

a. Do you find the Provisions useful? Why or why not?

b. Why is one provision more prevalent than others in your State?

3. Our understanding is that [number] of your schools implement the Community Eligibility Provision. Does that number sound accurate?

a. Do you find the CEP option useful? Why or why not?

Processes for the FNS-10

4. Thinking about how you track data to support your SFA’s claim for reimbursement, do you use a paper or electronic system?

Probe:

a. Are you familiar with the FNS 10 form?

b. [If electronic] What do you call this file?

[IF YES, PROCEED to question 5].

[IF NO] Do you know the form as something else or do you not see the form at all?

5a. Starting with the National School Lunch Program, please walk me through what you do to provide information to the State on the number of paid, free, and reduced price meals that are served.

[REPEAT QUESTION FOR ALL ITEMS ON THIS FORM. REVIEW CONTENT/PURPOSE OF DATA ELEMENTS PRIOR TO ASKING QUESTIONS, TO ENSURE RESPONDENT UNDERSTANDS QUESTIONS]

Probes:

a. How do you get the information you need from the schools?

b. (If applicable) In what format does the data arrive from schools and RCCIs (Excel, database report, etc.)?

i. Does each school/RCCI have its own system and format or are they standardized?

c. When do you typically receive the data? (daily, weekly, monthly)

d. What type of quality credit checks are conducted to ensure the accuracy of the data?

5b. Are the School Breakfast Program and any Special Milk Programs handled the same or differently?

Probes:

a. [If different] How are they different?

b. How about for summer camps or nonresidential institutions?

6. Tell me about the system used to track and store the data you’ve just described.

Probes:

a. What software do you use?

b. Who is involved in collecting the data?

c. Does the system flag or notify you if there are suspected errors in the data?

d. Does the data system automatically show totals by week and month? If not, do you have to do that manually?

e. Does the data system have built-in reporting functions to make it easier to review the data? Describe.

f. Please describe any difficulties with using the system.

7. Who reviews the data before it’s submitted to the State? Describe that review process.

Probes:

a. How do you handle missing data? Do you leave fields blank or follow-up with the school?

i. How do missing data impact your attempts to aggregate data across all of your sites?

b. Do you have any challenges with timeliness of data from sites? For submission to the State agency?

8. When you submit the data to the State are you filling in online forms, uploading reports, or something else? Please describe.

Probes:

a. How would you describe the user friendliness of the data submission process to the State?

9. How often do you have to make corrections to data that has already been submitted to the State?

Probes:

a. What kinds of corrections usually need to be made? (INTERVIEWER: this is related to the FNS-10 30 and 90-day reports)

b. Why are you typically making corrections?

c. Tell us how you review and revise the data.

i. Who is involved?

ii. How often do you have to go back to the schools and ask them for revised data?

iii. How long does it usually take to complete these revisions?

d. What kinds of revisions are the most difficult to make? The easiest to make?

10. When in the data collection process would you guess that mistakes are most likely to occur (e.g. initial counting, determining eligibility)? In the data aggregation and validation process (e.g. fields not summing automatically)? In the data submission process (from school>SFA; from SFA>State)?

Probe:

a. How, if at all, are those errors flagged?

11. In general, what improvements do you think could be made to make it easier to submit school meal claims data?

Probes:

a. What additional guidance or clarity from the State would help?

b. Would you make any changes to the timing of when the reports are due?

c. Are there any elements of the system you use to submit the data that could be improved?

12. Have you recently made or are you considering any future changes to how your SFA collects or submits the school meal data to improve things that you feel are not working well? If so, describe.

13. Does your SFA operate the Seamless Summer Option?

[IF NO, PROCEED TO NEXT QUESTION]

[IF YES] Do you go through the same process to collect and validate the meal data during the summer as you do during the school year? If no, what does the process look like?

Probe:

a. Do you have the ability to provide separate meal counts for meals served through the Seamless Summer Option and the “regular” NSLP/SBP meals served during the summer?

i. If no, what are the challenges to providing separate counts? What kind of support would you need to be able to provide separate counts?

Processes for the FNS-742

This has been very helpful, thank you. Now let’s take a look at the FNS-742, School Food Authority Verification Collection Report. This is the annual report that includes detailed information on the results of the annual free and reduced price certification and verification process, as well as the number of sites you have with special provisions.

14. Thinking about how you track the results of the certification and verification process, do you use a paper or electronic system?

Probes:

a. Are you familiar with the FNS-742 form?

b. [If electronic] What do you call this file?

15. Starting with Section 1, walk me through what you have to do to fill this section out.

[REPEAT QUESTION FOR EACH SECTION (1-5) ON THE FORM. REVIEW CONTENT/PURPOSE OF DATA ELEMENTS IN THE SECTION PRIOR TO ASKING QUESTIONS.]

Probes:

a. When is the data for this section typically available/ready?

b. What type of quality checks are conducted to ensure the accuracy of the data?

16. The FNS‑742 contains data on student enrollment in various categories (see Column B, sections 1-4 on the FNS-742). Where does the data on enrollment come from?

Probe:

a. Do you know if the enrollment data collected in these sections are consistent with enrollment data on any other forms submitted to FNS, such as the FNS-10?

17. Now let’s take a look more closely at Section 5.

Probes:

a. For Section 5-2, can you explain how and when households are contacted for verification? (INTERVIEWER: be clear on when first notification goes out, what methods are used to contact households, how many attempts are made and how many households usually fail to respond to the notifications)

b. For Section 5-4, can you explain what Error Prone means?

c. In your opinion, for Section 5-7 what does direct verification mean?

d. Section 5-8, can you explain what you consider “non-response” vs “response? How do you count a partial or incomplete response? INTERVIEWER: is partial or incomplete counted as a response or non-response?

e. VC-1 at the bottom of the form, what does ‘total questionable applications verified for cause’ mean to you?

18. How does matching for direct certification happen in your SFA?

Probes:

a. Do you receive data on the categorically eligible from a state agency (ies)? Which agency (ies) and for which programs (SNAP, TANF, other)?

b. Does the system distinguish between students approved as categorically free based on other documentation (e.g. SNAP, TANF) versus those identified as categorically eligible through the meal application?

19. Tell me about the system used to track and store the direct certification data you’ve just described.

Probes:

a. Is the database the same as the one used for the meal counting and claiming we discussed above? If no, describe.

b. Who is involved in collecting the data?

c. Does the system flag or notify you if there are suspected errors in the data?

d. Does the data system automatically show the totals within and across schools/RCCIs or do you calculate those totals manually?

e. Does the data system have built-in reporting functions to make it easier to review the data? Describe.

f. Can you describe any difficulties using the system?

20. What other quality checks are conducted to ensure the accuracy of the data?

21. Who reviews the data before it’s submitted to the State? Describe that review process.

Probe:

a. How do you handle missing data? Do you leave fields blank or follow-up with the school?

22. When you submit the data to the State are you filling in online forms, uploading reports, or something else? Please describe.

Probe:

a. How would you describe the user friendliness of the data submission process to the State?

23. When in the data collection process would you guess that mistakes are most likely to occur (e.g. initial counting, determining eligibility)? In the data submission process from SFA>State)?

Probe:

a. How, if at all, are those errors flagged?

24. In general, how do you think the process of reporting on certification and verification could be improved?

Probe:

a. What additional guidance or clarity would help?

b. Would you make any changes to the timing of when the reports are due?

c. Are there any elements of the system you use to submit the data that could be more user-friendly? [note to interviewer: the “system” could be an online database, a simple excel document that they email, or something else].

25. How, it at all, does the timing of when the FNS-742 is due impact the accuracy of the data?

26. Have you recently or are you considering future changes to how you collect or submit the school meal data for the FNS-742 to improve things that you feel are not working well? If so, describe.

Interaction between the SFA and the State

27. What kind of guidance or assistance does the State provide with collecting or submitting the data?

Probe: Webinars, printed materials, one-on-one TA calls

a. When you have questions about the data or the process, who do you turn to for help?

28. Are you ever asked by the State to revise the data you submitted?

Probes:

a. How often would you estimate that occurs?

b. What are the most common reasons you’re asked to revise the data? Which data do you typically have to make edits to?

c. How do you resolve those issues?

d. Are there circumstances where you would not be able to revise the data?

Wrap-up

29. [IF PERSON HAS WORKED AT ANY SFA MORE THAN 1 YEAR] Has anything changed over the past few years with regard to the process you go through to collect and submit the data? Describe.

Probe:

a. If yes, did those changes make the process easier or harder?

30. Does your system collect or report any other school meal data beyond what we’ve already discussed? If so, describe.

Probe: Frequency of collection and reporting, what the reporting is for, data elements, etc.

Thank you so much for taking the time to talk with us today.

Do you have any final questions or comments?