Appendix B4. IRA Pretest Results

Appendix B4. IRA Pre-Test Results

3

IRA Pre-Test Results

Table of Contents

Chapter Page

1 Introduction and Methods 1

1.1 Recruitment Methods 1

1.2 Data Collection 2

1.3 Data Analysis 3

1.4 Findings and Recommendations 3

2 Summary of Recommendations 4

2.1 State Director Survey 4

2.2 LEA Interview Guide 15

Table

1-1 Completed interviews by staff person title 2

Introduction and Methods

1

The U.S. Department of Agriculture’s Food and Nutrition Service (FNS) has contracted with Westat to study the independent review of applications (IRA) process for school meal benefits (IRA Study). The purpose of the study is to provide a description of the IRA process at the State and LEA levels, its results, and its overall effectiveness in reducing administrative certification error. The key research objectives relate to assessing how State agencies and LEAs implement and report on IRA.

In preparation for study launch, Westat tested the State Director Survey and the LEA Interview Guide. The goals of the testing were to ensure that (1) respondents interpret the questions as intended and can easily respond and (2) interviewers can easily administer the instruments. All testing materials were reviewed and approved by the Westat Institutional Review Board (IRB) prior to recruitment and data collection. All cognitive interviews were conducted by telephone.

1.1 Recruitment Methods

Westat, with FNS approval, selected Colorado and Arkansas to be the States in which the survey was tested. To be eligible to complete the interview, the respondent had to be either the person who filled the targeted role (e.g. Child Nutrition Director or similar), or a sufficiently knowledgeable alternative within the agency. We also tested the survey with two FNS regional office staff, one of whom previously worked at the State level, who had valuable insight into the workings of all States in their region and could provide a higher-level perspective on the survey instrument. We tested the LEA interview guide with three LEA Directors, each in a different State. With FNS approval, we selected the LEAs to attain diversity in size, geographic location, school type (public, private, charter), the number and types of errors flagged in the FNS-874 data from School Year (SY) 2016-17, and whether the LEA conducted an IRA in one school year or more than one school year.

FNS national and regional staff provided contact information for the targeted respondents, and Westat emailed each respondent to ask for their participation. Westat attached the survey and/or interview guide to the emails so that the respondent would be able to review the instruments in advance. As shown in Table 1-1, two staff at the Mountain Plains Regional Office, two staff at the Department of Education in Arkansas, and one staff from the Department of Education in Colorado were interviewed for the State Director Survey. LEA directors from Utah, Michigan, and Wisconsin were interviewed for the LEA Interview Guide.

Table 1-1. Completed interviews by staff person title

Instrument	Staff person title
State Director Survey	Mountain Plains Regional Office Senior Program Specialist 1 Senior Program Specialist 2 Colorado Department of Education Nutrition Programs Senior Consultant Arkansas Department of Education Director Associate Director
LEA Interview Guide	LEA Director in Utah LEA Director in Michigan LEA Director in Wisconsin

1.2 Data Collection

One trained interviewer conducted each interview, assisted by a note taker. The interview sessions lasted up to 90 minutes and included the following:

• The interviewer administered the study introduction – explaining the study purpose and the respondent’s rights as a research subject.

• Respondents were asked for their verbal consent after interviewers explained the voluntary nature of their participation and confidentiality.

• The interviewer followed the interview guide and administered scripted probes.

• The respondents were provided opportunities to offer any additional feedback or reactions at various points in the interview.

• After the end of the session, the respondent was thanked for participating.

The approach for testing these types of qualitative data collection instruments was to administer them as written and observe how respondents responded, noting any difficulties they encountered. All interviews were conducted over the phone. Interviewers administered the full in-depth interview guide and survey, timed the process, and observed and documented issues that arose for both respondents and interviewers. Respondents were e-mailed the guide in advance and were asked to review the material before administration. After completion of the interview, interviewers followed-up on any areas of difficulty respondents encountered while answering the questions.

1.3 Data Analysis

The interviewer and one note taker served as analysts. They reviewed their notes from each interview and produced a list of themes and patterns within the interview data. In particular, staff focused on problems and issues with the instruments, including areas where the respondents demonstrated confusion, hesitation, uncertainty, and/or discomfort. Staff discussed the results of the analysis to validate the findings and confirm recommendations. Themes and patterns were organized, evaluated, synthesized, and summarized into report form.

1.4 Findings and Recommendations

The remainder of this report summarizes the issues found in each tested instrument and provides recommendations for addressing the issues.

Summary of Recommendations

2

2.1 State Director Survey

Question number	Findings	Recommendations
	Global Issues. None.
A1, A2, A3, A4	Items A1, A2, A3, and A4: A1. How many LEAs were required to conduct a second review of applications in School Year 2016-2017 based on Criterion 1 of the program regulations? Criterion 1 includes all LEAs with 10 percent or more of the certification/benefit issuances in error, as determined during an Administrative Review. |___| Number of LEAs under Criterion 1 in School Year 2016-2017 A2: How many LEAs were required to conduct a second review of applications in School Year 2016-2017 based on Criterion 2 of the program regulations? Criterion 2 includes LEAs not identified under Criterion 1 that the State agency determines are at risk for certification error. |___| Number of LEAs under Criterion 2 in School Year 2016-2017. A3. How many LEAs were required to conduct a second review of applications in School Year 2017-2018 based on Criterion 1 of the program regulations? Criterion 1 includes all LEAs with 10 percent or more of the certification/benefit issuances in error, as determined during an Administrative Review. |___| Number of LEAs under Criterion 1 in School Year 2017-2018 A4. How many LEAs were required to conduct a second review of applications in School Year 2017-2018 based on Criterion 2 of the program regulations? Criterion 2 includes LEAs not identified under Criterion 1 that the State agency determines are at risk for certification error. |___| Number of LEAs under Criterion 2 in School Year 2017-2018. Respondents indicated that they would be able to specify the number of LEAs required to conduct the second review in a given year. One said that they do not naturally distinguish between those flagged under each criterion, but that they would be able to parse the data to provide the specific numbers. Respondents suggested that we first ask for the total number of LEAs required to conduct the second review, and then ask for that number broken out by criterion.	Combine items A1 and A2 and revise to: A1. How many LEAs were required to conduct a second review of applications in School Year 2016-2017? |___| Total Number of LEAs that Conducted a Second Review in School Year 2016-2017 A1a. How many of those were required to conduct a second review based on Criterion 1 of the program regulations? Criterion 1 includes all LEAs with 10 percent or more of the certification/benefit issuances in error, as determined during an Administrative Review. |___| Number of LEAs under Criterion 1 in School Year 2016-2017 A1b. How many LEAs were required to conduct a second review of applications based on Criterion 2 of the program regulations? Criterion 2 includes LEAs not identified under Criterion 1 that the State agency determines are at risk for certification error. |___| Number of LEAs under Criterion 2 in School Year 2016-2017 Combine items A3 and A4 and revise to: A2. How many LEAs were required to conduct a second review of applications in School Year 2017-2018? |___| Total Number of LEAs that Conducted a Second Review in School Year 2016-2017 A2a. How many of those were required to conduct a second review based on Criterion 1 of the program regulations? Criterion 1 includes all LEAs with 10 percent or more of the certification/benefit issuances in error, as determined during an Administrative Review. |___| Number of LEAs under Criterion 1 in School Year 2016-2017 A2b. How many LEAs were required to conduct a second review of applications based on Criterion 2 of the program regulations? Criterion 2 includes LEAs not identified under Criterion 1 that the State agency determines are at risk for certification error. |___| Number of LEAs under Criterion 2 in School Year 2017-2018.
A6	Item: When does the State agency first notify LEAs that they must conduct a second review of applications? SELECT ONLY ONE. In June of the school year before the LEA has to conduct the second review Upon identification that the LEA has to conduct the second review Some other time (PLEASE SPECIFY_____________) Each response option was considered to be appropriate, but respondents suggested including a drop-down menu that allows the respondent to specify the exact month by which the LEAs were notified. Respondents also indicated that LEAs may be notified of the second review requirement following the Administrative Review. We amended the second response option to include a reference to the Administrative Review.	Renumber and revise to: A4. When does the State agency first notify LEAs that they must conduct a second review of applications? SELECT ONLY ONE. Once a year, all at the same time We typically notify the States in [DROPDOWN MENU OF MONTHS] At different times; it depends when we identify the LEA (e.g., after an administrative review) Some other time (PLEASE SPECIFY________)
A7	Item: How does the State agency notify LEAs that they must conduct a second review of applications? SELECT ALL THAT APPLY. In-person during Administrative Review Telephone call Email message Letter mailed to the LEA Other (PLEASE SPECIFY_____________) One respondent indicated that LEAs requiring a second review are not usually notified in-person during the Administrative Review, but afterward in the Administrative Review report that is shared with the LEA. We amended the response options to include the Administrative Review report.	Renumber and revise to: A5. How does the State agency notify LEAs that they must conduct a second review of applications? SELECT ALL THAT APPLY. In-person during Administrative Review In the Administrative Review report shared with the LEA Telephone call Email message Letter mailed to the LEA Other (PLEASE SPECIFY_____________)
B1- B6	Items B1-B6: B1. Does the State agency provide training to LEAs on the second review of applications requirement? SELECT ONE RESPONSE. Yes, for all LEAs in the State Yes, but only for LEAs that are required to conduct the second review No  GO TO QUESTION B6 B2. When does the State agency typically hold training for LEAs on the second review of applications? ENTER MONTH: _________________ It varies from year to year B3. Which of the following topics does the State agency cover in the training for LEAs on the second review of applications? SELECT ALL THAT APPLY. Requirements for free and reduced price certifications Intent and purpose of the second review of applications provision Criteria used to identify LEAs to conduct the second review of applications How to select the second review official How to train the second review official How to perform the second review of applications Documentation and reporting requirements (i.e., FNS-874) State criteria for discontinuation of second review requirement Other (PLEASE SPECIFY):____________ B4. How does the State agency typically deliver the LEA training on the second review of applications? SELECT ALL THAT APPLY. Group training in person Group online training (e.g., Skype or GoToMeeting) One-on-one training in person One-on-one training over the telephone Online training module(s) Other (PLEASE SPECIFY): B5. Does the State agency require that the second review official(s) from LEAs attend or complete State-provided training? Yes, all second review officials must attend Only some second review officials must attend Explain:______________________________ No, it is not required B6. What topics do LEAs have the most questions about with regard to the second review of applications requirement? SELECT THE TOP 3 MOST COMMON AREAS OF INQUIRY BY LEAs. Requirements for free and reduced price certifications How to select the second review official How to train the second review official How to perform the second review of applications How to document the second review of applications Reporting (i.e., FNS-874) Other (PLEASE SPECIFY): We do not typically receive questions on the second review requirement Item B1 asks whether the State agency provides training to LEAs on the second review but does not ask the respondent to specify the extent of the training being provided. Thus, respondents were not sure what they should consider to be a “training.” They asked if it counted for them to include a single PowerPoint slide on the process that they include in their annual training, or if we were asking about a separate training that specifically focused on the IRA process. We are more interested in the latter, and revised the questions in section B to delve into any trainings that focus specifically on the IRA process. Respondents also suggested adding “how to approve an application as free, reduced, or paid” as a topic that is typically covered in the training focused on the second review. We added this as a response option under item B1b (formerly item B3). According to the respondents, “how to be removed from the IRA requirement” was a common inquiry made by LEAs. We added that as an additional response option under item B2 (formerly item B6).	Renumber and revise to B1-B2: B1. To what extent does the State agency provide training to LEAs on the second review of applications requirement? SELECT ALL THAT APPLY. It is briefly touched upon in the annual training we hold for all LEAs The State conducts a training focused on the second review process for all LEAs The State conducts a training focused on the second review process for only those LEAs required to complete the second review We do not train LEAs on the second review of applications requirement [PROGRAMMER: IF RESPONDENT MARKS THE 2ND OR 3RD RESPONSE OPTIONS, PROCEED TO B1A, ELSE PROCEED TO B2] B1a. When does the State agency typically hold the LEA training focused on the second review of applications? ENTER MONTH: _________________ It varies from year to year B1b. Which of the following topics does the State agency cover in the LEA training focused on the second review of applications? SELECT ALL THAT APPLY. How to approve an application as free, reduced price, or paid Intent and purpose of the second review of applications provision Criteria used to identify LEAs to conduct the second review of applications How to select the second review official How to train the second review official How to perform the second review of applications Documentation and reporting requirements (i.e., FNS-874) State criteria for discontinuation of second review requirement Other (PLEASE SPECIFY):______________ B1c. How does the State agency typically deliver the LEA training focused on the second review of applications? SELECT ALL THE APPLY. Group training in person Group online training (e.g., Skype or GoToMeeting) One-on-one training in person One-on-one training over the telephone Online training module(s) Other (PLEASE SPECIFY):____________ B1d. Does the State agency require that the second review official(s) from LEAs attend or complete State-provided training on the second review of applications? Yes, all second review officials must attend/complete Only some second review officials must attend/complete Explain:______________________________ No, it is not required to complete State-provided training B2. What topics do LEAs have the most questions about with regard to the second review of applications requirement? SELECT THE TOP 3 MOST COMMON AREAS OF INQUIRY BY LEAs. How to approve an application as free, reduced price, or paid How to select the second review official How to train the second review official How to perform the second review of applications How to document the second review of applications Reporting (i.e., FNS-874) How to be removed from the IRA requirement Other (PLEASE SPECIFY):_________ We do not typically receive questions on the second review requirement
C2, C3	Items C2 and C3: C2. How does the State agency handle a situation in which an LEA does not conduct a required second review of applications? SELECT ALL THAT APPLY. Require the LEA to conduct a second review in future school years Go onsite to the LEA to examine their certification process Review during the next Administrative Review Require a corrective action plan No policies in this area Have not encountered this situation Other (PLEASE SPECIFY):____________ C3. What documentation does the State agency use to determine that an LEA improved certification accuracy and may discontinue the second review in the following school year? Per Federal regulations, to discontinue the second review, data must demonstrate that no more than 5 percent of the applications reviewed by the LEA required a change in eligibility determination. SELECT ALL THAT APPLY. Data submitted by the LEA for the FNS-874 Data obtained by the State agency during Administrative Review at the LEA Other (PLEASE SPECIFY)_____________ Item C2 asks about the process the State agency follows when an LEA does not conduct a required second review. Respondents could not say how they would know that a LEA did not conduct a required second review. They suggested including a question that asks the State agency to describe the process for validating the completion of the second reviews, including a response option that indicates that the State agency does not verify the completion of required second reviews. Respondents also suggested including an item that asks if there is any follow up with LEAs if they report making no changes to their initial eligibility determinations. State agencies that do follow up with LEAs will be asked to briefly describe their process. State agencies that do not follow up with these LEAs will be taken to the next item which asks about how the agency would handle a situation in which a LEA does not conduct a required second review. Here, one respondent suggested adding the response option “Withhold claims reimbursements until the IRA is completed.” State agencies that do not have policies in this area or have not encountered this situation are able to indicate as such.	Add two additional items, renumber, and revise to: C2. How does the State agency validate that the second review has been completed each year? The State confirms it was completed during an administrative review Other (PLEASE SPECIFY): Not applicable, we do not validate the completion of the second review C3. Is there any follow up with an LEA if it reports making no changes to the initial eligibility determinations? Yes No  GO TO QUESTION C4 C3a. Briefly describe how the State follows up with LEAs that report making no changes: C4. How does the State agency handle a situation in which an LEA does not conduct a required second review of applications? SELECT ALL THAT APPLY. Require the LEA to conduct a second review in future school years Go onsite to the LEA to examine their certification process Review during the next Administrative Review Require a corrective action plan Withhold claims for reimbursement until the IRA is completed No policies in this area Have not encountered this situation Other (PLEASE SPECIFY):____________ C5. What documentation does the State agency use to determine that an LEA improved certification accuracy and may discontinue the second review in the following school year? Per Federal regulations, to discontinue the second review, data must demonstrate that no more than 5 percent of the applications reviewed by the LEA required a change in eligibility determination. SELECT ALL THAT APPLY. Data submitted by the LEA for the FNS-874 Data obtained by the State agency during Administrative Review at the LEA Other (PLEASE SPECIFY)_____________
D2	Item: How do LEAs submit data for the FNS-874? SELECT ALL THAT APPLY. LEAs enter data elements directly into the online State reporting system LEAs upload a spreadsheet or similar data file to the online State reporting system LEAs email a spreadsheet or similar data file to the State agency LEAs send a hard copy spreadsheet or similar data file in the mail Other (PLEASE SPECIFY):_______________ One respondent suggested that we revise the response options to be more specific and say “spreadsheet or Word document”. We made this revision.	Revise to: How do LEAs submit data for the FNS-874? SELECT ALL THAT APPLY. LEAs enter data elements directly into the online State reporting system LEAs upload a spreadsheet or Word document (or similar) to the online State reporting system LEAs email a spreadsheet or Word document (or similar) data file to the State agency LEAs scan handwritten documents and email them to the State agency LEAs scan handwritten documents and upload them to the online State reporting system LEAs send a hard copy spreadsheet or Word document (or similar) data file in the mail Other (PLEASE SPECIFY):_________________
D3	Item: Does the State agency’s reporting system have built-in edit checks for the data submitted by LEAs for the FNS-874? Yes No One respondent was not sure how to answer this question since they did not use their State reporting system to review data submitted by LEAs. Another respondent explained that data submitted by their LEAs was manually reviewed by a staff member. Thus, the response options for this item were amended to include “Not applicable, enter data directly into FPRS”. Those who indicate that their reporting system does have built-in (i.e., automated) edit checks will be asked to briefly describe those edit-checks via an open-text response field. Including an open-text response field with this item will allow us to learn more about the types of edit-checks being made, which may help to explain the errors that are still getting through. We also defined the term “edit check” to provide further clarification.	Revise to: Does the State agency’s reporting system have automated edit checks for the data submitted by LEAs for the FNS-874? An edit check is a means of checking data entered for validity (i.e., prevent erroneous data from being entered in a cell, flag missing data elements). Yes No Not applicable. Explain: ___________ D3a. Briefly describe the built-in edit checks in the system that houses the FNS-874 data:
D6	Item: Does the State agency use the data results of the second review of applications reported by LEAs to inform State training or technical assistance for LEAs? SELECT ONE RESPONSE. To a large extent To a moderate extent Not at all This item may not collect valuable data. It allows us to ascertain whether a State agency uses the data results to inform State training or technical assistance for LEAs, but we are not learning how they use these data results. Furthermore, respondents felt that the use of the word ‘inform” was awkward; they were not sure if we were asking whether they used the results to inform a group in the State agency or if we were asking whether the results were used to determine the level or type of training needed. Respondents felt that it would make more sense to include a text field so that they are able to explain how they are using the data results, rather than assign response options that are subjective (e.g., “large” or “moderate” extent). An additional response option was included to further define the use of the word “inform” (i.e., develop training for LEAs) and to allow the respondent to indicate that they do not use the data results for that reason.	Revise to: [PROGRAMMER: SKIP IF ANSWERED “We do not train LEAs on the second review of applications requirement” IN RESPONSE TO B1] Briefly describe how, if at all, the State agency uses the data results of the second review of applications reported by LEAs to develop training or technical assistance for LEAs? We do not use the data results to inform or further develop training for LEAs
E4	Item: Which of the following topics do the written policies and/or procedures for State-level staff address? SELECT ALL THAT APPLY. Factors used to identify LEAs under Criterion 2 Requirements for LEAs that are subject to second review (e.g., training, documentation, etc.) Reviewing the data submitted by LEAs Monitoring the second review process at the LEAs FNS-874 reporting requirements and procedures Discontinuing the second review requirement Other (PLEASE SPECIFY):__________________ Respondents felt that the response options for this item were appropriate. They suggested we add a response option to capture any follow-up with LEAs regarding the IRA process or data. All respondents felt that it would be easy to upload their written policies and/or procedures if requested. One respondent suggested we allow for multiple uploads as they use several resources.	Revise to: Which of the following topics do the written policies and/or procedures for State-level staff address? SELECT ALL THAT APPLY. Factors used to identify LEAs under Criterion 2 Requirements for LEAs that are subject to second review (e.g., training, documentation, etc.) Reviewing the data submitted by LEAs Monitoring the second review process at the LEAs Following up with LEAs regarding the IRA process or data FNS-874 reporting requirements and procedures Discontinuing the second review requirement Other (PLEASE SPECIFY):__________________ [PROGRAMMER: REQUEST UPLOAD OF POLICIES AND PROCEDURES]
F1	See original survey for item F1. Item F1 includes a list of potential challenges that the State agency may face in implementing the second review requirement. The respondent is asked to indicate the extent to which each of these factors is a challenge (e.g., significant, moderate, not a challenge, etc.). Respondents answered this item based on the amount of time each activity required rather than whether the activity was a true challenge for the respondent. We split this item into two separate tables: the first table will allow the respondent to indicate the factors that are the most time consuming for the State agency, the second table will ask the respondent to indicate the factors that are true challenges. Additionally, factors such as staff availability at the State agency and correcting FNS-874 data were moved to the second table. Based on the responses received to this item, “Ensuring the IRA process is carried out correctly” was added to the second table as an additional potential challenge. “Following up with LEAs to correct FNS-874 data” was added to the first table as an additional factor that may be time consuming for State agencies, per the recommendation of respondents.	Split item F1 into items F1 and F2. See revised survey for both items.
F2	See original survey for item F2. Item F2 includes a list of potential challenges that LEAs may face in implementing the second review requirement. One respondent was not sure how to quantify the words “most” and “some” in the column headers (e.g., “Significant challenge for most”). Furthermore, the extent to which these factors are challenging may depend on the size of the district - respondents suggested revising the column headers to ask whether these factors are very or somewhat challenging for large v. medium/small LEAs. Doing so will allow us to determine whether certain factors are challenging due to the size of the district. Large LEAs are defined as having 10,000 students or more; medium LEAs have between 2,500-9,999 students; and small LEAs have no more than 2,499 students- these definitions were added to the item so that respondents are aware of what is considered a large vs. medium vs. small LEA. Also, we changed the programming to allow up to two responses per row, which allows respondents to select one level of difficulty for large LEAs and one level of difficulty for medium/small LEAs.	See revised survey for new item F3 (renumbered from F2), which adds additional breakdown by LEA size to match categories in annual direct certification reports.
G1	Item: Overall, does the second review process help to reduce certification error? Why or why not? Respondents felt that they would not know whether the second review process truly helps to reduce certification error. They suggested revising the wording to indicate that we are asking for their opinion rather than hard data.	Revise to: Overall, do you think that the second review process helps to reduce certification error? Why or why not?
G4	Item: Please note anything else you would like to tell us about the second review of applications provision. Respondents felt that the word “provision” was not necessary to include in this item and that “second review of applications” would suffice. We made this change.	Revise to: Please note anything else you would like to tell us about the second review of applications.
Section H	Item: I, [State CN Director Name], have reviewed the information in this survey and confirm that it is an accurate accounting of the second review process in my State. Thank you for participating in this USDA study. Per input from FNS, we added a link at the end of the survey to where the final report will be posted online.	Revise to: I, [State CN Director Name], have reviewed the information in this survey and confirm that it is an accurate accounting of the second review of applications process in my State. Thank you for participating in this USDA study. FNS anticipates the study results will be published on the FNS website in Spring 2019, located here: https://www.fns.usda.gov/ops/child-nutrition-programs.

2.2 LEA Interview Guide

Question number	Findings	Recommendations
	Global Issues. None.
1	Item: NEW One respondent was part of a contract management organization that helps with school meal applications. She suggested asking at the start whether or not the respondent is listed on the LEA application to the State as the recipient for information on school meals or child nutrition programs so that we know from the start how integrated or involved the respondent is with the school district, and whether the State considers that person to be the primary point of contact at the LEA.	Include as part of item 1 and revise to: b. Are you listed on your LEA’s application as the recipient for information on school meals or child nutrition programs? [IF NO] i. Who is? ii. Are you part of a contract management organization that the school district brought in to help with school meal applications?
13	Item: When in the year were you first notified by the State that you would have to do a second review of applications? One respondent had difficulty remembering when she was first notified. The interviewer broadened the question and probed to ask if she could recall if it was before or after the school year started. That probe made it easier for the respondent to answer, and we added that as a note to the interviewers to probe further.	Revise to: When in the year were you first notified by the State that you would have to do a second review of applications? [Note: if cannot recall specific time, see if they can recall whether it was before or after the start of the school year]
20	Item: What kinds of written policies or procedures do you have at the LEA that pertain to the second review? [Interviewer request a copy] Did the LEA develop those materials or did the State provide them? [If NONE] What kind of documentation would be helpful to have as your staff implement this process? Respondents indicated that they usually use written policies or procedures provided by the State; they do not usually develop separate materials on their own. Thus, we will request materials from the LEA only if they developed documentation that differs from what the State provided them.	Revise to: What kinds of written policies or procedures do you have at the LEA that pertain to the second review? a. Did the LEA develop those materials or did the State provide them? [Interviewer request a copy of any LEA-developed documentation that differs from what the State disseminates] b. [If NONE] What kind of documentation would be helpful to have as your staff implement this process?
26	Item: How easy or difficult is it to complete the FNS-874? d. If difficult, how so? e. How could the form be improved? This item was split into three separate items to better understand the challenges: the first (new item 24) assesses whether the respondent has ever found it difficult to understand the what was needed to complete the form or to calculate or enter data for specific fields on the FNS-874. The second item delves into the time burden. The third item asks for the respondent’s opinion on how the form could be improved (new item 28).	Renumber and include as item 24: There are fields on the FNS-874 for the number of applications that changed status – free, reduced price, and paid, and fields for the number of applications with different types of errors, such as gross income calculation errors, categorical eligibility errors, and incomplete application errors. Since you first started filling out this form, what questions have you had about what you need to enter in each field? Is it difficult to calculate or enter data for any of those fields? Renumber and include as item 27: How much time does it take to complete and submit the FNS-874? What part of the process takes the most time? Renumber and include as item 28: In your opinion, how could the form be improved?
28, 30	Items 28 and 30: 28. [If LEA reported making no changes to determinations, per FNS-874] The data we have for the [YEAR-YEAR] school year indicates that your LEA found no errors in the initial determinations after conducting the second review. Were you surprised to learn that the second review found no errors? Why or why not? Probe, if not addressed: Do you think your LEA might be doing a better job during the initial review after learning that you would have to conduct a second review? Why or why not? Do you think the State administrative reviews overestimate the error rate? Why or why not? Are there errors that are difficult to uncover, and so the second reviewer might also miss them? 30. [If LEA reported making changes to determinations, per FNS-874] Some LEAs are selected to conduct a second review because the State’s administrative review finds that 10% or more of the LEA’s determinations were made in error. And yet the data shows that a number of those LEAs report making no changes following the second review. What are your hypotheses about why we see that trend in the data? Probe, if not addressed: Do you think LEAs might be doing a better job during the initial review after being told that they’ll have to conduct a second review? Why or why not? Do you think the State administrative reviews overestimate the error rate? Why or why not? Do you think that some of those LEAs record their second review results as their initial review results, and so no changes would appear for that second review? Why or why not? Are there errors that are difficult to uncover, and so the second reviewer might also miss them? Items 28 and 30 were combined and revised to be more concise. Specific notes for the interviewer were included with each of the new sub-questions. The respondents generally appeared comfortable answering these questions. However, one respondent asked “you mean, are they lying?”, when we asked “Do you think that some of those LEAs record their second review results as their initial review results, and so no changes would appear for that second review?” That highlighted for us the sensitivity of that question, but the respondent did not suggest that we should change it or omit it. We will be sure to discuss the sensitivity of these questions during data collector training. None of the respondents mentioned errors that might be missed by the second reviewer because they were difficult to uncover, and we deleted that last probe from these items as a sub-question. Also, an interviewer note was added to the new item, asking the interviewer to pay particular attention if the LEA was flagged under Criterion 1 and reported no errors after IRA. The interviewer should not accept vague responses to these questions because these are some of the key items of interest to FNS.	Renumber to revise to: 31. [NOTE: This question applies to all LEAs, but interviewer should pay particular attention if the LEA was flagged under Criterion 1 (10%+ error) and reported no errors after IRA; don’t accept vague responses.] The past few years of national data show that some LEAs make no changes to the initial eligibility determination in the second review. What are your hypotheses about why we see that trend in the data? [If LEA reported making no changes] Why do you think that was the case in your LEA? In general, do you think LEAs might be doing a better job during the initial review after being told that they’ll have to conduct a second review? Why or why not? Do you think the State administrative reviews overestimate the error rate? Why or why not? [if not addressed] Did you feel that was true for your LEA during the last State administrative review? [SKIP if LEA reported making no changed [too pointed] Do you think that some LEAs record their second review results as their initial review results, and so no changes would appear for that second review? Why or why not?
29	Item: [If LEA reported making changes to determinations per FNS-874] The data we have for the [YEAR-YEAR] school year indicates that [TYPE OF ERROR] was the most common type of error that your LEA found upon conducting the second review. Why do you think that error was the most common? This question was awkward to administer when the LEA only reported a single error on the FNS-874, so there was no “common” error to inquire about. We added an interview note that interviewers should only ask about the most common type of error if the LEA reported making more than two errors.	Renumber to revise to: 30. [If LEA reported making changes to determinations after the most recent IRA, per FNS-874. SKIP IF LEA ONLY REPORTED MAKING 1-2 ERRORS] The data we have for the [YEAR-YEAR] school year indicates that [TYPE OF ERROR] was the most common type of error that your LEA found upon conducting the second review. Why do you think that error was the most common?