0584-0524 OMB memo - FNS-742 Data Study

Memorandum

Date: 06/29/2017

To: Stephanie Tatham, OMB Desk Officer

Through: Ruth Brown, United States Department of Agriculture, Office of the Chief

Information Officer

From: Lynnette Thomas

Food and Nutrition Service, Branch Chief, Planning & Regulatory Affairs

Re: Under Approved Generic OMB Clearance No. 0584-0524 Request for Approval for the FNS-742 Data Study

The Food and Nutrition Service (FNS) of the United States Department of Agriculture (USDA) is requesting approval for formative research under Approved Generic OMB Clearance No. 0584-0524.

The FNS-742 form (0584-0026, Expiration Date 3/31/2020) is FNS’ main source of administrative data for the National School Lunch and Breakfast Programs and one of the primary data sets that FNS uses in its work to identify and examine potential process improvements in the Programs. While a considerable amount of effort has been invested by FNS, State Agencies, and School Food Authorities (SFAs) in ensuring the reporting of accurate data, the dataset as a whole still contains many discrepancies that result in diminished confidence in subsequent analyses. In order to improve the quality of the data and support informed policy making, FNS is seeking to conduct a formative study to assess the feasibility of incorporating a comprehensive set of logic-based edit checks at the point in which SFAs are reporting 742 data to the State Agency.

1. Title of Project: FNS-742 Data Study

2. Control Number: 0584-0524, expiration date: 09/30/2019

3. Entities affected by this project

• School Food Authority (SFA) staff

• State Agency staff

• Vendors of software for the Child Nutrition Programs

5. Time Needed per Response

(a) Participants	(b) Description of Collection Activity	Time
		Minutes	Hours
State Agency	Follow-up discussions with research team in conference call and/or individually	180	3
	Review of instructions and implementation of study plan	60	1
	1. Familiarization with edit checks and correction instructions for the Verification Collection Report (FNS-742)	60	1
	2. Technical assistance provided to SFAs in the process of correcting errors in the data	600	10
SFAs	Follow-up discussions with research team in conference call and/or individually	60	1
	Correct each error in Verification Collection Report data submission	10	0.17
Software vendor	Write code to incorporate additional edit checks (figures represent time to modify program code to add one edit check)	5	0.08
	Determine best method for collecting data on the number and types of error messages encountered by SFAs	240	4
State Agencies: Declined participation	Communication regarding the project	15	0.25

6. Total Burden Hours on Public

(a) Participants	(b) Description of Collection Activity	(c) Instrument Type	(d) No. Respondents	(e) No. Responses Per Respondent	(f) Total Annual Responses (cxd)	(g) Hours Per Response	(h) Total Burden (exf)
State Agency	Follow-up discussions with research team in conference call and/or individually	Appendix C	1	3	3	3	9
	Review of instructions and implementation of study plan	Appendix A	1	1	1	1	1
	1. Familiarization with edit checks and correction instructions for the Verification Collection Report (FNS-742)	Appendix B	1	1	1	1	1
	2. Technical assistance provided to SFAs in the process of correcting errors in the data	Appendix B	1	341	34	10	344
SFAs	Follow-up discussions with research team in conference call and/or individually	Appendix C	6	1	6	1	6
	Correct errors in Verification Collection Report data submission	Appendix B	692	1.23	81	0.17	14
Software vendor	Write code to incorporate additional edit checks	Appendix B	1	784	78	0.08	6.5
	Determine best method for collecting data on the number and types of error messages encountered by SFAs	n/a	1	1	1	4	4
State Agencies: Declined participation	Communication regarding the project	Appendix A	4	1	4	0.25	1
				Total Burden Hours on Public			386.5

7. Formative study purpose, study design and methodology

Formative study purpose:

The FNS-742 Verification Summary Report (or 742) is an OMB-approved form (Control number 0584-0026 Expiration date 3/31/2020; Appendix B) that collects information from all School Food Authorities (SFAs) that participate in the National School Lunch and School Breakfast programs on certification and verification outcomes. These data are reported annually by SFAs to their State Agency, which are then submitted to FNS. The 742 is FNS’ main source of administrative data for the National School Lunch and Breakfast Programs and it is one of the primary data sets that the Child Nutrition Office of Program Integrity (OPI) uses in its work to identify and examine potential process improvements in the Programs. While there are a number of simple, logic-based edit checks already contained in the system through which FNS accepts 742 data submissions (called FPRS), the current array of edit checks do not capture all potentially identifiable errors. Therefore, the dataset as a whole still contains a large number of discrepancies that undermine the quality of any subsequent analyses. To help remedy this situation, OPI has developed a more robust series of edit checks that identify additional errors in the data. In order to promote informed policy making and sound technical assistance to Congress in the legislative process, FNS is working to improve the quality of the 742 data set. Toward that end, FNS is seeking to conduct a formative study to assess the feasibility of incorporating the additional edit checks at the point where SFAs submit the data to the State Agency.

Research questions

1. Are SFAs able to improve the accuracy of their 742 data by correcting the errors identified by OPI’s additional edit checks in a State data submission system?

2. What is the relative cost/burden of implementing these edit checks on SFAs, States, and software vendors, compared to current costs?

Study design and methodology: a pre-experimental (single group) pretest-posttest design

OPI recently discovered that some State Agencies around the country use State-level child nutrition software to collect 742 data from SFAs, but to OPI’s knowledge, the existing software does not incorporate edit checks that FNS has long identified as necessary to help ensure data integrity. OPI staff will identify and work with one State (and therefore one developer of child nutrition software) to incorporate edit checks into its system. These checks will not prevent submission of the data by SFAs; rather they will be faced with a message describing the error with a prompt to correct it prior to submitting their data. OPI will gather feedback from the software developer about the process of incorporating the additional checks into the system. Following the annual reporting process, OPI will also conduct follow-up phone calls with the State Agency staff to learn about the experience and process of providing support and technical assistance to the SFAs to correct errors, as well as with the SFA staff to hear about the process of actually fixing the errors in their records.

State selection:

The research team will identify and work with one State to implement this study. The State will be selected based on the presence of existing edit checks in their verification system/software (such as field validation for blank or missing values), that do not duplicate the edit checks developed by FNS. This will ensure that the software has the existing infrastructure to support an edit check process, and yet still has capacity for additional edit check functions. States possessing information about the use and type of local software systems used by SFAs in that State will be prioritized. As these systems may impact the accuracy of the data due to the automating or semi-automating the process of reporting, ideally, this would be controlled for in the analysis.

The research team anticipates that the selected State agency will make the modified version of the verification software available to every SFA rather than maintain two systems – one with the FNS changes and one without. Consequently, the research team is not recruiting individual SFAs for participation in this project. However, the research team will ask the State agency to identify up to 6 SFAs for qualitative interviews with FNS and the State, and to arrange those interviews.

Data collection:

• Qualitative interviews with State Agency staff, SFA staff, and the software developer (see Appendix C: Interview guides).

• List of the different software vendors/programs used by the SFAs in the State. This data will only be collected if it is available from the State Agency. OPI will not survey SFAs for this information.

• Quantitative analysis will be done with the FNS-742 data that is collected annually from SFAs. To the extent possible, OPI will work with the software developer to gather data on the number and type of errors that were flagged at the point of submission. This will allow a clear metric for which errors were corrected as a result of the edit check warning.

Data Collection Component	Candidate List	Agreed to Participate	Type of Interview
Interview with SA staff	5	1	Phone
Interviews with SFA staff	6	6	Phone
Interview with software developer	1	1	Phone
FNS-742	-	-	-
Total	12	8

Analysis:

• Qualitative data collected during the interviews will be subject to a thematic analysis post-study.

• FNS-742 data from the year prior to the study/intervention will be compared with data from the year of the intervention for the participating State. A State-level pre-post comparison of the total and average number of errors in the records will be conducted, as well as an SFA-level pre-post analysis of the number and type of errors present, controlling for use of verification software.

8. Federal Costs

We anticipate that incorporating the additional edit checks will be a relatively non-burdensome process, and moreover, we think that it should be feasible at no cost beyond the associated staff hours to make modifications since we will be supplying the logic/code necessary to incorporate the edit checks. Additionally, as stated above, one criterion for the State selection is that they have a system that is already edit check ready/capable. However, in the case that the State or its designated software vendor requests additional resources as a condition for making the changes, we would estimate $10,000 to be the upper limit of what would be reasonable and feasible for FNS to pay for the programming.

9. Confidentiality

This study does not involve the collection of sensitive or private information that would raise concern about confidentiality. As stated above, the quantitative data is routinely collected administrative data from the FNS-742. The qualitative data, collected during follow-up calls with the State and SFAs, is merely regarding the process of gathering and reporting that data to FNS and the State, respectively. See Appendix C for the interview guides that will be used during the qualitative data collection.

10. List of Appendices

Appendix A: State recruitment letter

Appendix B: Verification Collection Report (FNS-742)

Appendix C: Interview guides

1 Assuming 50% of SFAs with errors will reach out to the State for help correcting their data.

2 Average number of SFAs per State times the average percentage of SFAs that have one or more errors (from the 2016-17 FNS-742).

3 Among SFAs with errors, they have on average 1.2 errors (from the 2016-17 FNS-742).

4 There are 78 edit checks to incorporate, including those already included in the Food Program Reporting System (FPRS) that is used by States to submit data annually.