0584-0524 Clearance Memo Improving School Meal Program Accuracy Final

Memorandum

Date: May 9, 2019

To: James Crowe, OMB Desk Officer, Food and Nutrition Service

Through: Christina Sandberg, Food and Nutrition Service, Information Collection Clearance Officer, Planning & Regulatory Affairs; C.S.

Ruth Brown, United States Department of Agriculture, Office of Chief Information Office, Department Clearance Officer

From: Edward Harper, Director, Office of Program Integrity, Child Nutrition Programs, USDA- Food and Nutrition Service

Re: Request Approval to Perform Research Under Approved Generic OMB Clearance No. 0584-0524 for Improving School Meal Program Accuracy Through Pre-ordering and Grab and Go Systems

The Food and Nutrition Service (FNS) of the United States Department of Agriculture (USDA) is requesting approval for a formative research project under generic clearance number 0584-0524 Generic Clearance to Conduct Formative Research.

The proposed research will focus on the National School Lunch and School Breakfast Programs (NSLP and SBP) and specifically, on examining whether grab and go points of service and/or school meal pre-ordering applications can be structured to increase the percentage of reimbursable student meals and therefore reduce meal claiming errors made in schools. FNS realizes that in order to encourage schools and SFAs to voluntarily adopt strategies that increase program integrity, the strategies must be relatively simple and beneficial to the school or its students. Therefore, FNS will select schools that are willing to establish or have previously established expedited (i.e., “grab and go”) points of service and/or some form of pre-ordering system.

There are two potential interventions for this study and FNS may use one or both interventions, depending on the SFAs and schools that agree to participate. In the simplest case, FNS will work with schools to combine different food offerings at grab and go points of service to test options that are the most likely to result in students purchasing reimbursable meals. In the other scenario, FNS will work with SFAs and schools to combine pre-ordering with an expedited line in the cafeteria. If we are not able to find schools willing or able to test one of these new processes, we will select one or two schools and do an in-depth, descriptive study that details their use of either their grab and go points of sale or their pre-ordering applications in NSLP and SBP.

The following information is provided for your review:

1. Title of Project: Improving School Meal Program Accuracy through Pre-ordering and Grab and Go Systems

2. Control Number: 0584-0524, expiration date: 9/30/2019

3. Entities affected by this project

• State Agencies

State agencies will be involved in the SFA selection phase. FNS will identify up to three State agencies for initial phone calls to describe the research study, with the goal of identifying SFAs that are willing to participate in the study.

• School Food Authorities (SFAs)
  The research team will work with SFAs in the State or States that have agreed to participate in the study. SFAs will be asked to identify schools that are willing to participate in the study.
  
  

• Schools in participating SFAs
  Participating schools will agree to test one or more grab and go and/or pre-ordering configuration. FNS will select schools that are willing to establish, are in the process of establishing, or have already established some form of pre-ordering app and/or grab and go system. We anticipate working with between three and nine schools, and the schools (or students) will be randomly assigned into control and intervention groups. The study will include on-site data collection for some schools. Participating schools will also be asked to provide baseline data on the number of reimbursable, and the number of total meals served per lunch period for a one month period prior to the beginning of the study.¹ Participating schools will also provide limited school lunch transaction data, describing the number of reimbursable and non-reimbursable meals served. All data will be de-identified to exclude personally identifiable information. In a majority of point of sale systems, school meal transaction data are already collected by the schools’ software systems, so for those schools, the data collection will not require new record keeping. Record keeping for other systems should be minimal and is captured in the burden estimate.

• Students
  Students in this study will be given one or more grab and go and/or pre-order and pick up options. These grab and go options may include a variety of configurations, from dedicated lines in the cafeteria and kiosks that have cashiers, to vending machines. The study team will collect students’ meal choices selected from the grab and go line, kiosk, or vending machine and/or as recorded in the pre-ordering app, in addition to observational data on what is on the students’ trays as they go through the cafeteria line.

4. Number of Respondents:

Table 4.1 – Recruitment
Audience	# of Participants
State Agencies	3
School Food Authorities (SFAs)	9
Schools	9
Students	36,000
Total	36,021

Table 4.2 – Follow-up

Respondent Type	# Of participants
State Agencies	3
School Food Authorities (SFAs)	9
Schools	9
Total	21

Table 4.3 – Research Activities by Audience

Respondent Type	Research Activity	# of Participants
State Agencies	Communication regarding the project and identification of SFAs for the study	3
School Food Authorities (SFAs)	Recruitment and communication of research protocol	9
	Follow-up discussions with research team in conference call and/or individually.
	Introduce or modify pre-ordering or grab and go systems2
Schools	Recruitment and communication of research protocol	9
	Follow-up discussions with research team in conference call and/or individually
	Collect baseline data on the percentage of reimbursable meals presented to the cashier pre-intervention
	Establish or modify grab and go and/or pre-ordering systems
	Collect baseline data on the percentage of reimbursable meals presented to the cashier post-intervention
Students	Pre-order meal	18,000
	Pick up meal in an expedited cafeteria line	18,000
Total		36,021*

*This represents the total number of participants who are engaged in these activities. However, many of these participants are engaged in more than one activity. The total of unique respondents for this collection is 18,021.

5. Time Needed per Response

Table 5.1

Respondent Type	Type of Response	Time (Hours)
State Agencies	Communication regarding the project and identification of SFAs for the study	1
	Follow-up communication	1
School Food Authorities	Recruitment and communication of research protocol	1
	Follow-up discussions with research team in conference call and/or individually	1
	Establish or modify pre-ordering or grab and go systems	3
Schools	Recruitment and communication of research protocol	1
	Follow-up discussions with research team in conference call and/or individually	2
	Collect baseline data on the percentage of reimbursable meals presented to the cashier pre-intervention.	1
	Establish or modify pre-ordering or grab and go systems	1
	Collect baseline data on the percentage of reimbursable meals presented to the cashier post-intervention.	1
Students	Pre-order meal2	1 minute (0.02 hours)
	Pick up meal in expedited cafeteria line3	0

6. Total Burden Hours on Public: 615 hours. There are also 36,192 responses for this study.

Table 6.1

(a) Affected Public	(b) Type of Response	(c) Appendix	(d) Number of Respondents	(e) Responses per Respondent	(f) Number of Responses (d ˟ e)	(g) Hours per Response	(h) Total burden Hours (f ˟ g)
State Agency	Communication regarding the project and identification of SFAs for the study	Appendix A	3	2	6	1	6
State Agency	Follow-up discussions with research team in conference call and/or individually	Appendix D.2	3	2	6	1	6
SFAs	Recruitment and communication of research protocol	Appendix B and F	9	2	18	1	18
	Follow-up discussions with research team in conference call and/or individually	Appendix D.1	9	4	36	1	36
	Establish or modify the pre-ordering and/or grab and go system	n/a	9	1	9	3	27
Schools	Recruitment and communication of research protocol	Appendix C and F	9	2	18	1	18
	Follow-up discussions with research team in conference call and/or individually	Appendix D	9	5	45	2	90
	Collect baseline data on the percentage of reimbursable meals presented to the cashier pre-intervention	n/a	9	2	18	1	18
	Establish or modify grand and go and/or pre-ordering systems4	n/a	9	2	18	1	18
	Collect baseline data on the percentage of reimbursable meals presented to the cashier post-intervention	Appendix G	9	2	18	1	18
Students	Pre-order meal	Appendix E.1 and E.2	18,000	1	18,000	0.02	360
	Pick meal up in an expedited cafeteria line5	n/a	18,000	1	18,000	0	0
TOTAL			18,021	2	36,192	.02	615

7. Project Purpose, Methodology, and Formative Research Design:

Background:

The school nutrition programs operate in about 99,000 public and non‐profit private schools and residential child care institutions across the country. The program is available to all students who attend participating institutions. Students may qualify to receive school meals for free or at a reduced price on the basis of either household income or through participation in Federal, means-tested assistance programs. In order for those meals to qualify for Federal reimbursement, they must be complete, i.e., they must contain all of the components required by school meal patterns as specified at 7 CFR 210 and 220. If a meal is missing even one of its required components or if the meal pattern standards are otherwise not met, that meal does not qualify as reimbursable.

In 2015, an FNS study, Program Error in the NSLP and SBP: Findings from the Second Access, Participation, Eligibility and Certification Study (APEC II) (National School Lunch and School Breakfast Program Access, Participation, Eligibility and Certification Study, OMB Approval No.: 0584-0530, Discontinued date: 08/31/2015) determined that one of the most common errors in school meal programs results from mistakes in identifying meals as reimbursable at the point of sale, or “meal claiming” error. The vast majority of meal claiming errors (about 90 percent) are due to cashiers identifying non-reimbursable meals as reimbursable, which results in an overclaim. The FNS Child Nutrition Office of Program Integrity (OPI) is looking for ways to reduce program error and is particularly interested in developing low cost, low burden strategies towards that end. One possible avenue for reducing meal claiming error is to help schools increase the percent of meals that are reimbursable when they reach the cashier, and in this way, eliminate the opportunity for an overclaim.

Purpose:

USDA research also finds that too many students report having insufficient time to finish their meals. A 2011-2012 study (Special Nutrition Program Operations Study (SNPOS), OMB Control No.: 0584-0562, Expiration date: 09/30/2014) concluded that the average lunch period was 30 minutes or less in 95 percent of schools. The problem of short lunch periods is exacerbated because the time spent standing in line to get food often results in students having as few as 10 minutes to eat their lunches. Research studies on the effects of short lunch periods have shown that given limited time, students are more likely to choose less healthful, but faster options. The research also shows that having less time to eat contributes to food waste. In an effort to address the problems associated with short lunch periods, schools have begun adopting strategies that are part of an incentive structure to encourage students to return to the cafeteria.

Methodology/Research and Design:

FNS is working to design an experiment to increase the likelihood of students selecting reimbursable meals through two methods: providing different configurations of meal options, including pre-assembled, reimbursable meals in a “grab and go” setting and/or offering students an option to pre-order meals with emphasis on the selection of a reimbursable meal. In many schools with pre-ordering systems, staff assemble students’ meals prior to the meal service and students pick up their orders without standing in the usual cafeteria line. The purpose of the FNS study is to test whether schools can trade on that convenience to increase the percentage of reimbursable meals served and in that way, reduce program error.

Many schools offer grab and go or pre-ordering options to their students to satisfy student demand for convenience. And in many or most of those schools, students have the option to pick reimbursable or non-reimbursable options through those methods. Our study will test whether more restrictive grab and go or pre-ordering options – where students’ ability to select non-reimbursable meals is limited or eliminated entirely – would reduce their use of those options. Simply, the study will test whether behavioral science nudges will encourage the selection of reimbursable meals. Study interventions will range from options that allow students considerable flexibility in selecting their meals to the most restrictive case in which only reimbursable meals are available.

The proposed study is an optimization problem; we assume that more restrictive grab and go or pre-ordering options will reduce the student take-up of those options. The ultimate goal is to identify the most restrictive options that students will still use.⁷ Depending on the results of that test, FNS hopes to promote menu-restricted pre-ordering and grab and go options as integrity-promoting tools that are also popular with students. FNS understands that the most effective integrity promoting measures are ones that schools will want to adopt for reasons other than reducing program error.

The FNS study team will work with USDA’s State agency and school food authority (SFA) partners to identify schools to participate in the study. The team will seek school partners that have the capacity to support a grab and go option and/or currently have, or are seeking to adopt, a pre-ordering system. The study team will address the following research questions:

1. Can grab and go and/or pre-ordering systems increase the percentage of meals served that meet program requirements for federal reimbursement? and

2. How might schools optimize the setup of their grab and go and/or pre-ordering systems to maximize students’ selection of reimbursable meals?

The study team can address both of these research questions in participating schools that are in the process of establishing new grab and go or pre-ordering systems or would be willing to adopt paper-based pre-ordering systems. In participating schools with existing systems, the study team can address the second research question alone.

The study team will gather baseline data in the participating schools, independent of whether they are establishing new, or using preexisting, grab and go and/or pre-ordering systems. Previous work observing school lunch periods suggests that it will be relatively easy for schools to provide a count of reimbursable meals (something they must already collect and report monthly) and the total number of meals served. Following baseline data collection, new grab and go lines and/or pre-ordering systems will be made available to students in the participating schools.

FNS plans to test pre-ordering options using both paper and electronic apps and both can support testing student responses to a variety of configurations.⁶ These include giving students the freedom to select any combination of items, presenting students with options that encourage the selection of reimbursable combinations, and offering students only the option of ordering reimbursable meals. Grab and go systems can also be configured in ways that test student responses to different degrees of choice. Randomization into different treatment conditions will likely be at the school level, although it is possible students may be randomly assigned to intervention and control groups depending on the nature of the intervention.

The study team also hopes to test a new methodology of data collection for studies that rely on recording the content of students’ trays. Rather than record the names of individual food items on students’ trays (the traditional methods used in USDA studies), the study team will test the feasibility of taking pictures of food trays as they reach the cashier (the pictures will focus only on the trays; there will be no pictures with people). The study team will make it clear to participating States, SFAs, and schools that the pictures are only of the trays, and staff will be briefed on this aspect of the study prior to the data collection. OPI staff will review the photos, along with the menu/offerings for the day, to record the trays’ contents outside of the rushed cafeteria environment. This data collection method can be tested in the first year of the study (school year 2019-2020) at a small number of schools, likely in the DC metro region, since the sample does not have to be large or representative. If we find that the methodology is viable, we would then use it later in the study in order to establish baseline counts of reimbursable meals. It is worth noting that while we are not measuring meal claiming error in this study, testing this methodology will also help inform future USDA research where we do measure meal claiming error. If successful, this methodology could bring greater precision, potentially at a reduced cost, for future studies.

Objectives:

The primary goal of this study is to determine whether schools can appeal to students’ desire for convenience (i.e., through use of pre-ordering systems and/or expedited pick-up) to increase the selection of reimbursable school meals and reduce improper program payments. The project will begin with baseline data collection in the study schools. Baseline data collection will determine the percentage of meals that get to the cashier in reimbursable form. Once the baseline is established, the interventions may include the following:

• setting up or modifying grab and go systems that offer pre-packaged meals or meal components,

• establishing and/or modifying paper-based systems where students pre-order their school meals, or

• establishing or modifying a technology-based approach to pre-order meals.

The study team plans to work with participating schools (and their software vendors as necessary) to test a range of pre-ordering options that may include:

• offering students the same menu of food choices available in the cafeteria line,

• default sort ordering with more healthful choices (reimbursable meal components) prominently displayed,

• requiring complete meal selection, and

• some type of no-cost award system.

Grab and go systems may be configurable in some schools to test similar ideas.

The project will take place over two school years (SY), beginning as early as SY 2019-20.

The study team will work with States and SFAs to identify three to nine schools that are willing to participate in the study. Once the participating schools are selected, the research team will determine whether treatment conditions will be applied at the school or individual level. That decision will be based on the number of schools chosen and school characteristics (e.g., size of student population, ability to provide more than one intervention, etc.).

In phase one, researchers will test a new method to collect baseline data during a school meal service in up to four schools, likely in the DC metro area. Members of the research team will take photos of school lunch trays as they are presented to the school cashier (or cashiers). Those photos will then be analyzed by OPI staff to determine tray content and the percentage of trays that contained reimbursable meals. This portion of the project will not require any additional data or work on the part of participating schools.

Phase two will consist of instituting or modifying grab and go systems and/or pre-ordering systems in the selected schools. Researchers will work with schools and their vendors (as necessary) to develop the project interventions. FNS anticipates that one or more of the pre-ordering systems will use some type of technology (e.g., dedicated computer, ordering kiosk, or a pre-ordering app), and the study team will work with the schools and vendors to determine how best to incorporate the interventions into that technology.

Phase two of the project will have an on-site data collection component for some schools to again observe school lunch trays as they are presented to the school cashier (or cashiers).⁷ Researchers will determine whether those trays contained a reimbursable or non-reimbursable meal. Schools will also provide de-identified transaction data to researchers to provide information on the specific interventions in the electronic pre-ordering systems. These data are collected by the systems and therefore will not require significant effort from school staff.

Design/Sampling Procedures:

The research team expects to work with between three and nine schools over the course of the study. In general, schools selection will be based on: 1) their capability to implement the project requirements (e.g., providing separate cashier lines and/or some type of pre-ordering system which may include a range of possible interventions); 2) their willingness to participate; and 3) their interest in setting up or modifying a pre-ordering or grab and go system. Depending on the number and types of schools that agree to participate, the research team may consider randomizing treatment conditions at the student level.

Recruitment and Consent:

The research team will provide a written protocol to all SFAs and will hold individual phone calls with any SFA that requires assistance in implementing the research protocol.

Schools selected for participation will implement the following steps as part of the research protocol:

1. Work with research team to collect baseline data.

2. Implement grab and go and/or pre-ordering systems or agree to test minor modifications to those systems.

3. Work with the research team to incorporate different levels of the interventions.

4. If appropriate, work with the research team to randomly assign students to intervention and control groups.

5. Provide de-identified transaction data. (School will strip out the student’s name, eligibility status, and any other personally identifiable information.)

At the end of the project, FNS will provide participating schools, SFAs, and State agencies with site-specific results.

Compensation

No States, schools, or students will receive compensation for participation in this study.

Data Analysis:

Participating schools will not be representative of the overall population of schools, so the report will include only descriptive statistics. The research team will broadly assess whether or how pre-ordering and grab and go systems can be designed or modified to increase student selection of reimbursable meals. The analysis may also include a pairwise comparison of interventions as appropriate, depending on the interventions that are ultimately included in the study.

8. Federal Costs

The total estimated cost to the Federal Government is $49,021.20 over the study period. The largest cost to the Federal Government is for Federal government salaries of $35,845.20 to conduct the study, analyze the data, and draft the study report. The salary amounts include the number of hours required for staff to travel to the schools and collect the data, data analyses, and writing the report. The breakdown of Federal salary costs is as follows:

• 296 hours for a GS-13, Step 1 in the Washington, DC area at $47.52 per hour with an additional 33 percent ($15.68) to arrive at a fully loaded hourly rate of $63.20 ($18,707.20),

• 296 hours for a GS-12, Step 1 in the Washington, DC area at $39.96 per hour with an additional 33 percent ($13.19) to arrive at a fully loaded hourly rate of $53.15 ($15,732.40), and

• 16 hours of a Division Director’s time at a GS 15, Step 1 over the study period of $66.05 with an additional 33 percent ($21.80) to arrive at a fully loaded hourly rate of $87.85 ($1,405.60).

Federal employee pay rates are based on the Office of Personnel Management (OPM) salary table for 2019 for the Washington, DC, metro area locality (for the locality pay area of Washington-Baltimore-Arlington, DC-MD-VA-WV-PA) (at https://www.opm.gov/policy-data-oversight/pay-leave/salaries-wages/2019/general-schedule/)

The estimate also includes separate costs travel for two staff, including travel and per diem for $13,176.

Total Federal Cost: $49,021.20

9. Confidentiality

Data collected about student meal orders/trays will be treated with utmost care to ensure privacy. Data that comes from the pre-ordering software system will be de-identified by the school prior to sending it to FNS. We will provide instructions for how to do so once we know what information is collected by the system. We expect the student’s name, as well as their eligibility status, will need to be removed from the data set and replaced with a randomly assigned ID number. All other personally identifiable information will also be removed from the data set by the school before being sent to the study team.

During on-site observations, the study team will test the feasibility of taking pictures of student trays as they move through the lunch line. Data collectors will be instructed and trained to only take pictures of the trays and to ensure that no students are included in the photos, even in the background. OPI staff will review the photos, and if by chance, any identifiable images of students were captured, the photo will be deleted immediately and permanently from the data storage device.

10. List of Appendices

Appendix A – State Recruitment Letter

Appendix B – SFA Recruitment Letter

Appendix C – School Recruitment Letters

Appendix D.1 – SFA/School Follow-up Script

Appendix D.2 - State Follow-up Script

Appendix E.1 – Pre-ordering App Examples – Paper

Appendix E.2 – Pre-ordering App Examples - Electronic

Appendix F – Research Protocol

Appendix G – Anticipated Meal-App Record

1 Schools report the number of reimbursable meals to their SFAs through the normal course of business. Therefore these are data that are already collected. It should take little additional effort to report the total number of meals served per serving day in these schools.

2 The research team in conjunction with SFAs will identify schools that are willing to use or establish some type of grab and go system, are willing to adopt a paper pre-ordering app, are in the process of acquiring or already use some form of electronic pre-ordering app, or will be willing to use some combination of grab and go and pre-ordering.

2 In some schools, students may have the option of using either a paper or electronic pre-ordering app. These students will need to make their meal selections just as though they were in a cafeteria line, so any potential burden from this study is only the act of recording the meal in the app. FNS anticipates that not all students will use the app every day and that the actual ordering process will take 1 minute or less, so the burden here of 18,000 students × 1 minute is conservative.

3 A 2011-2012 study (Special Nutrition Program Operations Study (SNPOS), OMB Control No.: 0584-0562, Expiration date: 09/30/2014) concluded that the average lunch period was 30 minutes or less in 95 percent of schools. Researchers note that many students spend half or more of that time in line waiting for their meals, so an expedited line may result in time savings of 10 or 15 minutes per student.

4 Note that the project will be limited to schools that have or are willing to set up grab and go systems, and/or are willing to use paper-based pre-ordering forms or are in the process of setting up or have already set up pre-ordering systems. The time investments to set up these systems is not a cost of the project and the only cost not already incurred would be for printing pre-ordering forms if a paper-based option was adopted.

5 As noted in footnote 4, the average lunch period is 30 minutes or less in 95 percent of schools. With many students spending half or more of that time in line waiting for their meals, an expedited line may result in time savings of 10 or 15 minutes per student. Rather than try and estimate time savings, we conservatively assume there is no benefit to the students. There is no burden associated with a student using the expedited line.

7 An important note is that although student choice will be limited in some or all of the proposed interventions, the full range of options will still be available to all students by simply going to the cafeteria.

6 Electronic pre-ordering options will only be used in schools that already have that option, or that have already determined that they are going to adopt that technology and have made strides towards purchasing the app. No schools will be asked to adopt or spend money on an electronic app for this study.

7 The study team may not collect any on site data in schools that operated pre-ordering or grab and go systems prior to the study. In those schools, the test may be limited to comparing student selection from pre-ordering or grab and go systems as modified for the study to student selection from those systems prior to the start of the study.

4