State CN Agencies and Local Education Agencies (Directors and Key Staff) - SLT

OMB Approval No. 0584-####

Expiration Date: XX/XX/20XX

Appendix D4: Evaluation of the Independent Review Process: State Director Survey

INTRODUCTION

[PROGRAMMER: SHOW THIS INTRODUCTION EVERY TIME A USER LOGS IN]

The U.S. Department of Agriculture’s Food and Nutrition Service (FNS) is interested in understanding more about the Independent Review of Applications (IRA) requirement for school meal programs from the perspective of the States and local education agencies (LEAs). Under IRA, LEAs identified by the State agency must conduct a second, independent review of the eligibility determinations on household applications prior to notifying households of their eligibility status. FNS hired Westat to conduct a study to describe the IRA process and reporting via the FNS-874 form, which captures the activity of the IRA, also called the second review of applications. The study will also explore the effectiveness of the IRA process in reducing administrative errors.

As part of the IRA Study, Westat is conducting a survey of all State-level Child Nutrition agencies. The survey will answer questions like:

• How do States implement the IRA process?

• What is the burden on States to implement the IRA process?

• What opportunities are there to improve the process and minimize burden?

Your answers are important, and will help FNS understand implementation of the IRA requirement for school meal programs. There are no right or wrong answers. If you are unsure of how to answer a question, please give the best answer you can.

The survey link may be shared, and multiple staff in your agency may login to complete the survey. However, the State Child Nutrition Director must approve and submit the completed survey. This survey should take no more than 60 minutes to complete.

Your participation is entirely voluntary. Refusal to participate will not have any impact on your position, your State agency, or child nutrition programs. You may also skip questions that you do not wish to answer.

We will use all data we collect only for the purposes we describe. In the final report we will present the aggregated survey data, and will not link individual States to their responses. However, the raw survey data will be submitted to FNS at the end of the study.

You may login as many times as you wish in order to complete the survey between now and [DATE]. Simply click “Save and Continue Later” at the bottom of the screen before logging out to save your work. The survey will close on [DATE].

Please answer the questions in the survey based on current policies or procedures.

If you need additional information, please call [study phone number] or email us at [study email].

Thank you.

SECTION A. IDENTIFICATION AND NOTIFICATION OF LOCAL EDUCATIONAL AGENCIES

A1. How many LEAs were required to conduct a second review of applications in School Year 2016-2017?

|___| Total Number of LEAs that Conducted a Second Review in School Year 2016-2017

A1a. How many of those were required to conduct a second review based on Criterion 1 of the program regulations?

Criterion 1 includes all LEAs with 10 percent or more of the certification/benefit issuances in error, as determined during an Administrative Review.

|___| Number of LEAs under Criterion 1 in School Year 2016-2017

A1b. How many LEAs were required to conduct a second review based on Criterion 2 of the program regulations?

Criterion 2 includes LEAs not identified under Criterion 1 that the State agency determines are at risk for certification error.

|___| Number of LEAs under Criterion 2 in School Year 2016-2017.

[Programmer will auto-calculate this number by taking the total number of LEAs that conducted a second review and subtracting the number of LEAs flagged under Criterion 1]

[PROGRAMMER: IF NUMBER IN A1B IS GREATER THAN ZERO, REQUEST UPLOAD OF ALL CRITERIA 2 LEAS IN SY 16/17]

Please upload a Microsoft Excel file that lists all LEAs that were required to conduct a second review in SY 2016-2017 based on Criterion 2. Include only the LEA name and ID number.

A2. How many LEAs were required to conduct a second review of applications in School Year 2017-2018?

|___| Total Number of LEAs that Conducted a Second Review in School Year 2017-2018

A2a. How many of those were required to conduct a second review based on Criterion 1 of the program regulations?

Criterion 1 includes all LEAs with 10 percent or more of the certification/benefit issuances in error, as determined during an Administrative Review.

|___| Number of LEAs under Criterion 1 in School Year 2017-2018

A2b. How many LEAs were required to conduct a second review of applications based on Criterion 2 of the program regulations?

Criterion 2 includes LEAs not identified under Criterion 1 that the State agency determines are at risk for certification error.

|___| Number of LEAs under Criterion 2 in School Year 2017-2018.

[Programmer will auto-calculate this number by taking the total number of LEAs that conducted a second review and subtracting the number of LEAs flagged under Criterion 1]

[PROGRAMMER: IF NUMBER IN A2B IS GREATER THAN ZERO, REQUEST UPLOAD OF ALL CRITERIA 2 LEAS IN SY 17/18]

Please upload a Microsoft Excel file that lists all LEAs that were required to conduct a second review in SY 2017-2018 based on Criterion 2. Include only the LEA name and ID number.

Reminder: please answer the remaining questions in the survey based on current policies or procedures.

A3. Which of the following factors does the State agency use to identify LEAs for a second review of applications under Criterion 2 of the program regulations?

Criterion 2 includes LEAs not identified under Criterion 1 that the State agency determines are at risk for certification error.

SELECT ALL THAT APPLY.

Certification/benefit issuance error between 5 and 10 percent on Administrative Review

LEAs new to the National School Lunch Program (NSLP)

LEAs new to the School Breakfast Program (SBP)

LEAs with recently hired administrative staff

LEAs implementing a new electronic system

Other (PLEASE SPECIFY_____________)

State does not use Criterion 2 to identify LEAs

A4. When does the State agency first notify LEAs that they must conduct a second review of applications?

SELECT ONLY ONE.

Once a year, all at the same time

We typically notify the LEAs in [DROPDOWN MENU OF MONTHS]

At different times; it depends when we identify which LEAs have to conduct a second review (e.g., after an administrative review)

Some other time (PLEASE SPECIFY_____________)

A5. How does the State agency notify LEAs that they must conduct a second review of applications?

SELECT ALL THAT APPLY.

In-person during Administrative Review

In the Administrative Review report shared with the LEA

Telephone call

Email message

Letter mailed to the LEA

Other (PLEASE SPECIFY_____________)

SECTION B: TRAINING AND TECHNICAL ASSISTANCE

B1. To what extent does the State agency provide training to LEAs on the second review of applications requirement?

SELECT ALL THAT APPLY.

It is briefly touched upon in the annual training we hold for all LEAs

The State conducts a training focused on the second review process for all LEAs

The State conducts a training focused on the second review process for only those LEAs required to complete the second review

We do not train LEAs on the second review of applications requirement

[IF RESPONDENT MARKS THE 2^ND OR 3^RD RESPONSE OPTIONS, PROCEED TO B1A, ELSE PROCEED TO B2]

B1a. When does the State agency typically hold the LEA training focused on the second review of applications?

SELECT MONTH: _______(DROPDOWN MENU)__________

It varies from year to year

B1b. Which of the following topics does the State agency cover in the LEA training focused on the second review of applications?

SELECT ALL THAT APPLY.

How to approve an application as free, reduced price, or paid

Intent and purpose of the second review of applications provision

Criteria used to identify LEAs to conduct the second review of applications

How to select the second review official

How to train the second review official

How to perform the second review of applications

Documentation and reporting requirements (i.e., FNS-874)

State criteria for discontinuation of second review requirement

Other (PLEASE SPECIFY):

B1c. How does the State agency typically deliver the LEA training focused on the second review of applications?

SELECT ALL THAT APPLY.

Group training in person

Group online training (e.g., Skype or GoToMeeting)

One-on-one training in person

One-on-one training over the telephone

Online training module(s)

Other (PLEASE SPECIFY):

B1d. Does the State agency require that the second review official(s) from LEAs attend or complete State-provided training on the second review of applications?

Yes, all second review officials must attend/complete

Only some second review officials must attend/complete

Explain:______________________________

No, it is not required to complete State-provided training

B2. What topics do LEAs have the most questions about with regard to the second review of applications requirement?

SELECT THE TOP 3 MOST COMMON AREAS OF INQUIRY BY LEAs.

How to approve an application as free, reduced price, or paid

How to select the second review official

How to train the second review official

How to perform the second review of applications

How to document the second review of applications

Reporting (i.e., FNS-874)

How to be removed from the IRA requirement

Other (PLEASE SPECIFY):

We do not typically receive questions on the second review requirement

SECTION C: MONITORING THE SECOND REVIEW PROCESS

C1. How does the State agency monitor the accuracy of the second review of applications conducted by identified LEAs?

SELECT ALL THAT APPLY.

Review applications and second review process during each LEA’s Administrative Review

Review applications and second review process during technical assistance visits to LEAs

Review the FNS-874 data from each LEA and follow up on questionable data entries

Review LEA policies and procedures for second review of applications

Other (PLEASE SPECIFY):

None of the above

C2. How does the State agency validate that the second review has been completed each year?

The State confirms it was completed during an administrative review

Other (PLEASE SPECIFY):

Not applicable, we do not validate the completion of the second review

C3. Is there any follow up with an LEA if it reports making no changes to the initial eligibility determinations?

Yes

No  GO TO QUESTION C4

C3a. Briefly describe how the State follows up with LEAs that report making no changes:

C4. How does the State agency handle a situation in which an LEA does not conduct a required second review of applications?

SELECT ALL THAT APPLY.

Require the LEA to conduct a second review in future school years

Go onsite to the LEA to examine their certification process

Review during the next Administrative Review

Require a corrective action plan

Withold claims for reimbursement until the IRA is completed

No policies in this area

Have not encountered this situation

Other (PLEASE SPECIFY):

C5. What documentation does the State agency use to determine that an LEA improved certification accuracy and may discontinue the second review in the following school year?

Per Federal regulations, to discontinue the second review, data must demonstrate that no more than 5 percent of the applications reviewed by the LEA required a change in eligibility determination.

SELECT ALL THAT APPLY.

Data submitted by the LEA for the FNS-874

Data obtained by the State agency during Administrative Review at the LEA

Other (PLEASE SPECIFY)___________________________________________________

SECTION D: REPORTING

The questions in this section ask about annual reporting on the LEAs’ second review of applications. State agencies annually report the results to FNS on the FNS-874, Local Educational Agency Second Review of Applications. The report is due from the State agency to FNS by March 15.

D1. By what date does the State agency require LEAs to submit data for the FNS-874?

|___|___| / |___|___| MONTH/DAY

D2. How do LEAs submit data for the FNS-874?

SELECT ALL THAT APPLY.

LEAs enter data elements directly into the online State reporting system

LEAs upload a spreadsheet or Word document (or similar) to the online State reporting system

LEAs email a spreadsheet or Word document (or similar) data file to the State agency

LEAs scan handwritten documents and email them to the State agency

LEAs scan handwritten documents and upload them to the online State reporting system

LEAs send a hard copy spreadsheet or Word document (or similar) data file in the mail

Other (PLEASE SPECIFY):

D3. Does the State data system that houses LEA-level data for the FNS-874 have built-in edit checks to ensure the data submitted by LEAs are complete and accurate?

An edit check is a means of checking data entered for validity (i.e., prevent erroneous data from being entered in a cell, flag missing data elements).

Yes

No

Not applicable. Explain:

D3a. Briefly describe the types of built-in edit checks in the system that houses the FNS-874 data:

D4. Does the State agency typically need to request corrections to the data submitted by LEAs for the FNS-874?

SELECT ONE RESPONSE. YOUR BEST ESTIMATE IS ACCEPTABLE.

Yes, for more than 75% of the LEAs (almost all or all)

Yes, for 25-75% of LEAs (some)

Yes, for less than 25% of LEAs (a few)

Never  GO TO QUESTION D6

D5. What are the most common data corrections that the State agency requests from LEAs for data submitted for the FNS-874?

SELECT ALL THAT APPLY.

Supply missing data

The total number of reviewed applications does not equal the sum of the subtotals

The total number of changed applications does not equal the sum of the subtotals

The number of schools in the LEA does not match other records at the State agency

The number of enrolled students reported does not match other records at the State agency

Other (PLEASE SPECIFY):

D6. [PROGRAMMER: SKIP IF ANSWERED “We do not train LEAs on the second review of applications requirement” IN RESPONSE TO B1] Briefly describe how, if at all, the State agency uses the data results of the second review of applications reported by LEAs to develop training or technical assistance (TA) for LEAs?

If the State agency does not use the IRA data results to develop training or TA, simply check the box below.

We do not use the data results to inform or further develop training for LEAs

SECTION E: STATE POLICIES AND PROCEDURES

E1. Does the State agency have written policies or procedures pertaining to the second review of applications?

SELECT ALL THAT APPLY.

Yes, for State-level staff  [IF THIS IS THE ONLY RESPONSE OPTION SELECTED, GO TO QUESTION E4; OTHERWISE GO TO QUESTION E2]

Yes, for LEA staff

No  GO TO SECTION F

E2. Which of the following topics do the written policies and/or procedures for LEAs address?

SELECT ALL THAT APPLY.

Factors used to identify LEAs under Criterion 2

Training requirements for LEAs that are subject to second review

Second review documentation requirements

Qualifications for second review official

FNS-874 reporting requirements and procedures

Discontinuing the second review requirement

Other (PLEASE SPECIFY):

E3. How does the State agency disseminate these written policies and/or procedures to LEAs?

SELECT ALL THAT APPLY.

Through the State agency’s public website

Through a secure State portal that LEAs can access

At the annual State training for LEAs

Via email

Other (PLEASE SPECIFY):__________________________________________________

[PROGRAMMER: IF E1 RESPONSE IS “LEA ONLY,” GO TO SECTION F]

E4. Which of the following topics do the written policies and/or procedures for State-level staff address?

SELECT ALL THAT APPLY.

Factors used to identify LEAs under Criterion 2

Requirements for LEAs that are subject to second review (e.g., training, documentation, etc.)

Reviewing the data submitted by LEAs

Monitoring the second review process at the LEAs

Following up with LEAs regarding the IRA process or data

FNS-874 reporting requirements and procedures

Discontinuing the second review requirement

Other (PLEASE SPECIFY):_________________________

[PROGRAMMER: REQUEST UPLOAD OF POLICIES AND PROCEDURES]

SECTION F: CHALLENGES

F1. Please indicate the extent to which each of the following factors is time consuming for the State agency.

[PROGRAMMER: ALLOW RESPONDENTS TO CHECK ONLY ONE BOX PER ROW.]

	Very time consuming	Moderately time consuming	Not time consuming	Not Applicable	Not sure
Identifying LEAs to conduct a second review
Determining Criterion 2 selection factors
Training LEAs to conduct a second review
Providing technical assistance to LEAs on the second review process
Ensuring the IRA process is carried out correctly
Reviewing FNS-874 data from LEAs
Following up with LEAs to correct FNS-874 data
Submitting the FNS-874 report to FNS
Calculating LEA errors in order to determine whether they may discontinue the second review
Other (PLEASE SPECIFY):
Other (PLEASE SPECIFY):

F2. Below is a list of potential challenges that the State agency may face in implementing the second review of applications requirement. Please indicate the extent to which each of the following factors is a challenge for the State agency.

DO NOT consider the time required to complete each task.

[PROGRAMMER: ALLOW RESPONDENTS TO CHECK ONLY ONE BOX PER ROW.]

	Very challenging	Somewhat challenging	Not challenging	Not Applicable	Not sure
Staff availability at State agency for the IRA process (e.g., training LEAs on IRA process, reviewing IRA data, etc.)
Identifying LEAs to conduct a second review
Determining Criterion 2 selection factors
Training LEAs to conduct a second review
Providing technical assistance to LEAs on the second review process
Ensuring the IRA process is carried out correctly
Reviewing FNS-874 data from LEAs
Correcting FNS-874 data from LEAs
Submitting the FNS-874 report to FNS
Calculating LEA errors in order to determine whether they may discontinue the second review
Other (PLEASE SPECIFY):
Other (PLEASE SPECIFY):

The tables that follow ask about potential challenges that LEAs may face in implementing the second review of applications requirement. Based on your observations, indicate the extent to which of the following factors is a challenge for LEAs of different sizes.

[PROGRAMMER: RESPONDENT MAY ONLY SELECT ONE RESPONSE IN EACH ROW]

F3. Identifying staff who are qualified to conduct the first and second reviews of applications.

LEA Size	Very Challenging	Somewhat Challenging	Not Challenging	Not Applicable	Not Sure
Very Small (<500 students)
Small (500-999 students)
Medium (1,000-4,999 students)
Large (5,000-9,999 students)
Very Large (10,000+ students)

F4. Identifying staff who are available to conduct the first and second reviews of applications.

LEA Size	Very Challenging	Somewhat Challenging	Not Challenging	Not Applicable	Not Sure
Very Small (<500 students)
Small (500-999 students)
Medium (1,000-4,999 students)
Large (5,000-9,999 students)
Very Large (10,000+ students)

F5. Training staff to perform the second review.

LEA Size	Very Challenging	Somewhat Challenging	Not Challenging	Not Applicable	Not Sure
Very Small (<500 students)
Small (500-999 students)
Medium (1,000-4,999 students)
Large (5,000-9,999 students)
Very Large (10,000+ students)

F6. The volume of applications that staff need to review.

LEA Size	Very Challenging	Somewhat Challenging	Not Challenging	Not Applicable	Not Sure
Very Small (<500 students)
Small (500-999 students)
Medium (1,000-4,999 students)
Large (5,000-9,999 students)
Very Large (10,000+ students)

F7. Completing the second review within the 10-day timeframe.

LEA Size	Very Challenging	Somewhat Challenging	Not Challenging	Not Applicable	Not Sure
Very Small (<500 students)
Small (500-999 students)
Medium (1,000-4,999 students)
Large (5,000-9,999 students)
Very Large (10,000+ students)

F8. Inadequate technology hinders the second review of applications.

LEA Size	Very Challenging	Somewhat Challenging	Not Challenging	Not Applicable	Not Sure
Very Small (<500 students)
Small (500-999 students)
Medium (1,000-4,999 students)
Large (5,000-9,999 students)
Very Large (10,000+ students)

F9. Reporting the results of the second review process (FNS-874 form).

LEA Size	Very Challenging	Somewhat Challenging	Not Challenging	Not Applicable	Not Sure
Very Small (<500 students)
Small (500-999 students)
Medium (1,000-4,999 students)
Large (5,000-9,999 students)
Very Large (10,000+ students)

F10. Based on your observations, briefly describe any additional challenges that LEAs face in completing the second review of applications. ________________________________________________________________________________________________________________________________________________________________________

SECTION G: FINAL REFLECTIONS

G1. Overall, do you think that the second review process helps to reduce certification error? Why or why not?

G2. Briefly describe any changes that could minimize the burden on the State of the second review process.

G3. Briefly describe how the second review process could be improved.

G4. Please note anything else you would like to tell us about the second review of applications.

G5. Please list the names and titles of those who helped to complete this survey.

SECTION H: SUBMIT SURVEY (State Child Nutrition Director only)

I, [State CN Director Name], have reviewed the information in this survey and confirm that it is an accurate accounting of the second review of applications process in my State.

Thank you for participating in this USDA study.

FNS anticipates the study results will be published on the FNS website in Spring 2019, located here: https://www.fns.usda.gov/ops/child-nutrition-programs.

According to the Paperwork Reduction Act of 1995, an agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a valid OMB control number. The valid OMB control number for this information collection is 0584-XXXX. The time required to complete this information collection is estimated to average 60 minutes per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the data needed, and completing and reviewing the collection of information.