State CN Agencies and Local Education Agencies (Directors and Key Staff) - SLT

OMB Approval No. 0584-####

Expiration Date: XX/XX/20XX

APPENDIX E7: Evaluation of the Independent review of applications Process

LEA Interview Guide

Introduction

Good morning/afternoon. Thank you for taking the time to talk with me today. My name is [INTERVIEWER’S NAME] and I work for Westat, a private research company based in Rockville, Maryland. With me today is [NOTE TAKER’S NAME], who will be taking notes.

PURPOSE: The federal Food and Nutrition Service (FNS) is interested in understanding more about the Independent Review of Applications (IRA) requirement for school meal programs from the perspective of the States and local education agencies (LEAs). FNS hired Westat to conduct a study to describe the IRA process and reporting via the FNS-874 form, which captures the activity of the second review, and explore the effectiveness of the IRA process to reduce administrative errors. The final product will be a description of the IRA process and reporting procedures, and recommendations to improve the process and increase accuracy. During our conversation today, we would like to ask you about your process to conduct an IRA and the data you provide to the State agency via the FNS-874 form.

HOW YOU WERE SELECTED: We are conducting telephone interviews with 30 LEA Directors in order to describe the data collection processes and gather information to better understand areas for improvement. Westat worked with FNS to select LEAs based on size, State characteristics, and LEA characteristics.

INFORMATION TO BE COLLECTED: We will ask about how your LEA reviews household applications for school meal programs, and how you conduct the second, independent review of applications. We will also ask about the reporting process for independent review via the FNS-874, and for any suggestions you may have on how to improve these processes.

RISKS AND PRIVACY: There is little risk to being part of this study. We use all data we collect only for the purposes we describe. FNS knows which LEAs were recruited for this study, but we will report the results of these interviews in the aggregate. Neither your name nor the name of your LEA will be linked to any of your responses. In our reports we may include quotes from our respondents, but these will be presented without the speaker’s name and in such a way that you could not be identified. FNS will receive a redacted transcript of this interview, stripped of information that could identify you or your LEA. Participating in the interview may not help you individually, but it may help us better understand how to improve the independent review process.

STUDY COSTS AND COMPENSATION: There is no cost to you to participate apart from the time you spend with us for this interview, nor is there compensation. The interview takes 90 minutes.

VOLUNTARY PARTICIPATION: Your participation is entirely voluntary. Refusal to participate will not have any impact on your position, your LEA, or child nutrition programs. You can take a break, skip questions or stop participating at any time.

QUESTIONS: If you have questions about your rights and welfare as a research participant, please call the Westat Human Subjects Protections office at 1-888-920-7631. Please leave a message with your full name, the name of the research study you are calling about, which is the Evaluation of the Independent Review of Applications, and a phone number beginning with the area code. Someone will return your call as soon as possible.

We have planned for this discussion to last 90 minutes. Is that still okay?

With your permission I would like to record this discussion to help us fill any gaps in our written notes. The recordings, transcripts, and any notes we have will be stored on Westat’s secure server and will be destroyed after the project is complete.

Do you have any questions? [ANSWER ALL QUESTIONS]

May I turn on the audio recorder now?

[TURN ON AUDIO RECORDER IF GIVES CONSENT]

Now that the audio recorder is on, do you agree to participate? [PAUSE FOR RESPONSE]

And do you consent to be audio recorded? [PAUSE FOR RESPONSE]

Interviewer: review the LEA’s 874 data prior to discussion to understand the most common types of errors, size, etc..

Warm Up and Context

1. To start, please tell me about your role at the LEA and how long you have worked here.

1. What are your specific responsibilities with regard to household applications for school meals?

Probe: Oversee the certification process, make initial eligibility determinations, conduct IR, manage data tracking system.

2. Are you listed on your LEA’s application to the State as the recipient for information on school meals or child nutrition programs?

[IF NO]

1. Who is?

2. Are you part of a contract management organization that the school district brought in to help with school meal applications?

2. Do households have the option to submit an online application?

[IF YES]

1. Roughly what percentage of households submit paper applications and what percentage submit online?

2. Has this ratio of paper v. online applications changed over time? How so?

3. In which school years did your LEA conduct a second review of applications?

4. Did your LEA ever do a similar review of eligibility determinations before you were required to conduct a second review?

Initial Eligibility Determination Process

I would like to know more about the overall process you go through to review applications and certify households for free or reduced price meals.

5. Please walk me through the process at your LEA to review a household’s application for the first time in order to make the initial eligibility determination.

[Listen for these common steps in the process: receive apps and date stamp them, check apps for completeness, contact households to supply missing data, review and make eligibility determination.]

1. Who is involved at each step in the process?

   1. Do you hire any temporary employees?

   2. Are these solely food service staff or LEA employees with other responsibilities?

2. What parts of the review process are automated using technology?

Probe to understand if there is a computer system or other technology that receives the applications, stores the data, or makes eligibility determinations using the inputted data..

3. How is the process different when an application claims categorical eligibility?

4. [if applicable] How does the initial review process differ for online versus paper applications?

6. What happens when an application is missing information?

Probe to understand any follow-up to the household.

1. What kind of information is most frequently forgotten or left off by households?

7. Where and how do staff record their initial eligibility determination for each application?

Probe to understand if this is a notation on the application, a check mark on a roster, an Excel list, a notation in the POS, or some other record keeping that indicates the determination. (Note: It could be indicated in multiple places.)

1. [if applicable] How does this differ for online versus paper applications?

2. Is the determination the only thing that’s recorded, or do staff record other notes as well?

Probe to understand if they make notes of the rationale for the determination, or supporting/lacking evidence.

8. Roughly how much time does the initial eligibility determination take for each application?

1. Does this vary at all? If so, how?

Probe by online versus paper applications if applicable.

2. What part of the process takes the most time?

9. What are the biggest challenges to making this initial determination?

Possible probes:

• Understanding eligibility requirements (categorical eligibility, applicable income sources, “includable” household members)

• Incomplete applications – need to contact households

• Volume of applications to review

• Compressed timeline

• Staff capacity/experience

• Training of temporary workers

• Technology

Independent Review Process - Details

For this next group of questions I would like you to think about the last time you had to do a second, independent review.

10. When in the year were you first notified by the State that you would have to do a second review of applications?

[Note: if cannot recall specific time, see if they can recall whether it was before or after the start of the school year]

1. Did the State tell you why your LEA was selected?

[If YES]

1. What was the reason?

11. After you heard that your LEA would have to conduct the second review, were any changes made the following school year to the initial review process? Describe.

12. Who conducts the second review?

Probe to get the person’s position/title. If multiple people, get position/title for all.

1. Did he/she have any prior involvement with the eligibility determination process? Explain.

2. What qualifications or training does the LEA require of the second reviewer(s)?

13. How easy or difficult is it to identify someone who can do the second review? Explain.

14. Like you did for the initial review, please walk me through the steps to conduct the second review.

1. [if applicable] How does the second review process differ for paper versus online applications?

2. Does the second reviewer look at the work of the first reviewer, or do they start their own review of the application from scratch?

3. How soon after the first review does the second review occur?

4. Who is involved at each step in the process?

5. What are the procedural differences between the first and second review?

6. What happens when both reviewers make the same determination?

7. What happens when the determinations are different between the two reviews?

   1. Is the independent review determination taken as final or is there further arbitration?

15. Where and how does the second reviewer record his/her eligibility determination for each application?

Probe to understand if this is a notation on the application, a check mark on a roster, an Excel list, a notation in the electronic POS, or some other record keeping that indicates the determination.

1. Is the determination the only thing that’s recorded, or do staff record other notes as well?

Probe to understand if they make notes of the rationale for the determination, or supporting/lacking evidence.

2. How does the LEA track changes in eligibility determination between the first and second reviews?

   1. Does the LEA simply track that a change was made, or does also it track the type of change and the reason for the change?

3. [if applicable] How does this differ for online versus paper applications?

16. How much time does it take to conduct the second review for each application?

1. Does this vary at all? If so, how?

Probe by online versus paper applications if applicable.

2. What takes the most amount of time?

3. How easy or difficult is it to conduct the second review within the 10-day timeframe to notify households of the determination?

4. What happens if the LEA cannot meet the 10-day timeline?

Probe: if cannot meet the timeline, do they retroactively approve applications as of the date they were submitted?

17. What are the main challenges to conducting the second review?

Possible probes:

• Insufficient documentation

• Volume of applications to review

• Training of review official

• Identifying the review official

• Staff capacity/experience

• Technology

• Time constraint (must be completed within 10 days of application receipt)

• Reporting

18. [if LEA was flagged for IR more than once] Has the second review process at your LEA changed over time? If so, how?

Probe if needed: change in the review official, training provided, how they document the process.

1. Did those changes make the process easier or harder?

19. [if LEA only flagged for IR once] Do you foresee the LEA making any internal changes to the second review process going forward? If so, describe the change and the reason for it.

Independent Review Process - General

20. What kinds of written policies or procedures do you have at the LEA that pertain to the second review?

1. Did the LEA develop those materials or did the State provide them?

[Interviewer request a copy of any LEA-developed documentation that differs from what the State disseminates]

2. [If NONE] What kind of documentation would be helpful to have as your staff implement this process?

21. Did the State offer any technical assistance or training to help you implement the second review? If so, describe.

1. How could the State better support LEAs in this work?

22. Do you know how the State agency decides that an LEA can be removed from the second review requirement? If so, describe.

Reporting and Monitoring

Next I want to talk a bit about reporting and the FNS-874 form, where you report the results of the second review.

23. In what format do you complete the FNS-874 (paper hard copy, Excel, online State reporting system, other)?

24. There are fields on the FNS-874 for the number of applications that changed status – free, reduced price, and paid--and fields for the number of applications with different types of errors, such as gross income calculation errors, categorical eligibility errors, and incomplete application errors. Since you first started filling out this form, what questions have you had about what you need to enter in each field?

1. Is it difficult to calculate or enter data for any of those fields?

25. What information, if any, did the State provide you to help you complete the form?

26. What is the process to complete the FNS-874 and submit it to the State?

2. Who compiles the data and completes the form? Who submits it?

3. How long does it take to complete the form?

4. Has the State ever followed up about what was reported on the FNS-874? If so, explain.

27. How much time does it take to complete and submit the FNS-874?

1. What part of the process takes the most time?

28. In your opinion, how could the form be improved?

29. Does the FNS-874 accurately capture what happens during the second review?

2. [IF NO] What gets missed on the form?

30. [If LEA reported making changes to determinations after the most recent IRA, per FNS-874. SKIP if LEA only reported making 1-2 errors] The data we have for the [YEAR-YEAR] school year indicates that [TYPE OF ERROR] was the most common type of error that your LEA found upon conducting the second review. Why do you think that error was the most common?

31. [NOTE: This question applies to all LEAs, but interviewer should pay particular attention if LEA was flagged under Criterion 1 (10%+ error) and reported no errors after IRA; don’t accept vague responses.]

The past few years of national data show that some LEAs make no changes to the initial eligibility determination in the second review. What are your hypotheses about why we see that trend in the data?

1. [If LEA reported making no changes] Why do you think that was the case in your LEA?

2. In general, do you think LEAs might be doing a better job during the initial review after being told that they’ll have to conduct a second review? Why or why not?

3. Do you think the State administrative reviews overestimate the error rate? Why or why not?

   • [If not addressed] Did you feel that was true for your LEA during the last State administrative review?

4. [SKIP if LEA reported making no changes (too pointed)] Do you think that some LEAs record their second review results as their initial review results, and so no changes would appear for that second review? Why or why not?

32. Does the State monitor any other aspect of the LEA’s second review apart from what is submitted on the FNS-874?

[IF YES]

1. What else does the State monitor?

2. How does the State monitor this? (i.e., request for documentation from LEA, check in via phone/email)

Wrap-up

33. In your opinion, how effective is the second review at catching and reducing errors?

34. How, if at all, has the second review changed your LEA’s approach to reviewing applications?

35. What has been the impact of the second review on the accuracy of certification and benefit determination?

36. Are there any other types of support that you would like to receive that would help to minimize the time and effort of conducting a second review?

Do you have any final questions or comments?

Thank you so much for taking the time to talk with us today.

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-XXXX. The time required to complete this information collection is estimated to average 90 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.