M Pretest Memo

APPENDIX M. PRETEST MEMO

Memo

To: Ashley Chaifetz

From: 2024-2025 National School Foods Study Instrument Development Team

Date: 2/28/2024

Subject: Pretest Memorandum (Deliverable 1.3.3)

This memorandum describes the pretest procedures for the 2024-2025 National School Foods Study, summarizes the pretest findings, and lists the instrument changes we implemented based on the findings. The 2024-2025 National School Foods Study includes three components that were previously implemented as stand-alone Food and Nutrition (FNS) studies. These include the second School Nutrition and Meal Cost Study (SNMCS-II), the fourth School Food Purchase Study (SFPS-IV), and the evaluation of the Fresh Fruit and Vegetable Program (FFVP). Because instruments for SNMCS-II were recently pretested in 2018, the pretest focused on instruments and recruitment materials for the SFPS-IV and FFVP study components. All references to specific item numbers in this memorandum refer to the numbering in the instruments revised to reflect FNS feedback, delivered to Mathematica on January 9 or 12, 2024.

Pretest Recruitment and Procedures

Recruitment

In January 2024, Mathematica pretested instruments with two State Distributing Agencies (SDAs), three school food authorities (SFAs) in two states, two School Nutrition Managers (SNMs), and two students (See Table 1).

FNS identified two SDA representatives (from Iowa and New York) to participate in the pretest. Mathematica sent an email to both representatives describing the study and data request

The study team contacted four State Child Nutrition (CN) directors from Indiana (IN), Maine (ME), Michigan (MI), and Minnesota (MN) requesting that they each identify five SFAs to be considered for the pretest. Using the list of SFAs provided by the States and a few recommended by ProTeam, DIR sent an email invitation to each SFA which included an overview of the study and a request to participate in the pretest. We sent invitations to eight SFAs in ME and five SFAs in MI on January 8, and five SFAs in MN and one additional SFA in MI on January 9. We then sent follow-up emails to nonresponding SFAs on January 11. Four SFAs agreed to participate in the pretest (ME-1, MI-2, MN-1), of which three were selected to participate.¹ One additional SFA in ME agreed to allow the study team to pretest data collection instruments with two school nutrition managers and two students in their SFA, but did not agree to participate in the SFA pretest.

Once the SFAs were recruited, the study team worked with two SFA directors in ME to identify SNMs that would be willing to participate in the FFVP pretest. One school in each SFA was recruited. The SNM from one of the schools participated in a remote pretest and the school cafeteria staff and two students from the other school participated in an on-site pretest. One student was in first grade and the other student was in third grade.²

Table 1. Summary of pretest activities and participants

Pretest Participant	Activities and Instruments	Number of Participants	Thank-You Payment
School Food Purchase Study-IV (SFPS-IV) Component
SDAs	Complete the electronic process of submitting Quarterly USDA Foods data Debrief on USDA Foods data collection process	2	$0
SFA directors	Orientation call, Food Purchase Planning Interview, and debrief Complete the process of providing Quarterly Food Purchase Data Debrief on Quarterly Food Purchase Data process and SFA Director Survey (review of 7 new questions)	3	$100
Fresh Fruit and Vegetable Program (FFVP) Component: on-site pretest
SNM	Complete FFVP SNM Survey Pretest on-site data collection procedures for the FFVP Menu Survey Participate in debrief for FFVP SNM Survey, FFVP Menu Survey, and respond to questions related to pretest of FFVP modules of the Observation Guide	1	$50
Students	Complete the Student Interview (on site) and 20-minute debrief interview	2	$20
FFVP Component: remote pretest
SNM	Complete FFVP SNM Survey Review FFVP Menu Survey and FFVP modules of the Observation Guide Participate in 45-minute debrief on the FFVP SNM Survey, Procedures for administering the FFVP Menu Survey, and FFVP modules of the Observation Guide	1	$50

SFPS-IV Pretest Procedures

The pretest with the SDAs was designed to test the process for collecting quarterly data on USDA Foods. SDAs were asked to (1) review instructions for the USDA Foods Data request (F01.05 Request to SDAs to Submit USDA Foods Data and F01.11 SFPS Overview of USDA Foods Data), (2) submit data for five selected SFAs, and (3) provide feedback on the data collection process and instructions.

The SFPS-IV pretest with the SFAs was designed to test the process for collecting quarterly food purchase data and solicit feedback on new SFA Director Survey questions. SFAs were asked to (1) participate in a short Food Purchase Planning Interview (C14 SFPS Food Purchase Planning Interview) and provide feedback on the call and interview during a 45-minute orientation call, (2) complete the process of providing easily retrievable food purchase data (F01.01 SFPS Quarterly Program Data Form and Food Purchase Data Request Email and F01.02 SFPS Food Purchase Data Checklist), and (3) provide feedback on the food purchase data collection process and new survey questions (F03.01 SFA Director Survey) during a one-hour debrief call. The orientation and debrief calls were conducted virtually by a team of two or three.

FFVP Pretest Procedures

One SNM pretest was conducted remotely, and one was conducted in person. Both SNMs provided feedback on the FFVP SNM Survey (F04.02 FFVP SNM Survey (G2b)), the administration procedures for the FFVP Menu Survey (F02.03 FFVP Menu Survey (G2b)), and the FFVP modules of the Observation guide (F07 Obs Guide (G2ab, G3)). The SNM who participated in the remote pretest completed a hard copy of the FFVP SNM Survey and reviewed the FFVP Menu Survey and the FFVP content in the Observation Guide. During the debrief, the SNM provided feedback on the FFVP SNM Survey, the FFVP modules in the Observation Guide and the procedures for administering the FFVP Menu Survey.

At the school participating in the on-site pretest, the SNM was responsible for leading SBP and NSLP operations, and another member of the cafeteria staff was responsible for administering FFVP. Therefore, the FFVP cafeteria staff member took the lead in completing the FFVP SNM Survey in advance of the visit. During the pretest visit, the FFVP cafeteria staff participated in the survey debrief. While on site, the Mathematica interviewer administered all sections of the FFVP Menu Survey except the section on FFVP Snacks to the SNM and obtained feedback on the procedures for administering the FFVP Menu Survey. The Mathematica interviewer then pretested the FFVP snack section of the FFVP Menu Survey with the FFVP cafeteria staff member, by administering that section of the FFVP Menu Survey and obtaining feedback. The Mathematica interviewer also completed the FFVP modules of the Observation Guide while on site and made notes on things to revise. The interviewer reviewed their responses to the Observation Guide questions with the FFVP cafeteria staff and asked questions as needed to ensure accurate understanding of how FFVP operates at the school.

Two students participated in pretesting the FFVP content in the Student Interview (F08.01 StudInt-Eng_Sp(G2ab)). The interviewer administered the Student Interview to the first grader and obtained a burden estimate, then debriefed with the student after the interview was complete. The interviewer administered the Student Interview to the third grader using cognitive interviewing, stopping to probe on question comprehension and functioning of the instrument throughout the interview.

Pretest Findings

We discuss the findings and resulting changes to the instruments below, starting with the SFPS-IV component. We will also summarize the pre-test findings in the Office of Management and Budget package for the study. The final versions of the instruments will incorporate the changes based on the pretest.

SFPS-IV

SDA representatives and SFA directors were able to provide comprehensive data on USDA Foods direct deliveries and quarterly food purchases. SDA representatives reported that the instructions were easy to follow, and it took them 30 minutes or less to compile the USDA Foods direct deliveries data. Similarly, SFA directors said they were generally able to follow the instructions for the data request and reported that the survey questions were clear. Respondents did have some suggestions for improvements, which are detailed below.

Based on the findings from the pretest with the SDA representatives, we revised the following:

• One SDA representative reported that many respondents may know USDA Foods direct deliveries as “Brown Box.” We added “(formerly known as Brown Box)” after “USDA Foods Direct Deliveries” in the Overview of USDA Foods Data document (F01.11).

• Both SDA representatives provided some data on processed end products containing USDA Foods. One SDA representative was able to provide all necessary data, while the other representative reported that this information was not as readily accessible. As previously decided with FNS, because SDAs are not consistently able to provide information on processed products containing USDA Foods ingredients, we will collect it directly from SFAs. We added language to the Overview of USDA Foods document (F01.11) instructing SDA respondents that this data will be provided by the selected SFAs, and removed “and State Monthly Performance Reports for all further processed USDA foods” from the Request to SDAs to Submit USDA Foods Data email (F01.05).

• The two pretest SDA representatives were able to easily compile data on USDA Foods direct deliveries. As discussed with FNS on February 6, 2024, because SDAs use different systems and tools to manage data on USDA Foods direct deliveries, we removed the text, “and for States that use the Web-Based Supply Chain Management (WBSCM) system, the specific report(s) that may be submitted,” from the Request to SDAs to Submit USDA Foods Data email (F01.05).

Based on the findings from the pretest with the SFAs, we revised the following in the Food Purchase Planning Interview (C14):

• Q4. One SFA director was unsure whether they participate in a food-buying cooperative. We added the text, “These groups are often formed through agreements with other SFAs, or between Child Nutrition Program State agencies and SFAs, to increase purchasing power to competitively procure goods and services.”

• Q6. One SFA director later noted that she forgot to mention one vendor. We restructured the format of the question and added instructions to the interviewer to read all provided examples for each food type. This format will also allow for a more streamlined interview, with less unnecessary repetition.

• Q10. One SFA director was unsure what was meant by “editable format.” We amended the provided definition to read, “An editable format is an electronic data file, like a spreadsheet, or a report that can be edited on a computer, unlike a paper copy or scanned document.”

• Q11. One SFA director misunderstood this question and incorrectly reported that their district receives Commodity Letters of Credit or Cash in Lieu of Commodities. We changed the question to clarify, “Certain SFAs can elect to receive cash payments, or money, through Commodity Letters of Credit (CLOC) or Cash in Lieu instead of USDA-purchased foods or commodities.” We changed the response options to “Receive USDA Foods,” “Receive CLOC or Cash in Lieu,” “Other (specify),” and “Don’t know.”

• Based on observations during the pretest, we also made small edits to questions throughout to improve the flow of the interview.

We also revised the following in the SFA Director Survey (F03.01):

• F11. One pretest participant reported she did not know what foods or beverages contain artificial sweeteners because her district does not track that. We added a “don’t know” response option.

• F13. When asked about steps the SFA has taken to incorporate culturally relevant foods, one pretest participant mentioned that she posts the weekly menu on Facebook and sometimes receives feedback from parents and grandparents commenting on the culturally relevant meals offered. We have expanded the response option “Gather input from parents or the community on recipes” to include “…recipes and menus”.

• F14, F15. One pretest participant reported that one of the main cultural groups in her SFA, Franco-Americans, was not listed in the response options in the survey question. We have revised the response option, “American/European” to “Eastern European”, “Northern European”, and “Western European.” We also made further changes to expand the list of ethnic and cultural groups to be more comprehensive.

• F15. When asked about cultural communities represented in the student population and whether the SFA incorporates specific foods to reflect those cultures, all respondents agreed that adding “kosher diet” in the “Jewish” response option and “Halal or Islamic diet” in the “Muslim” option would be helpful. One noted that people think of kosher foods more than Jewish foods. We have added these descriptors in the selected response options.

• H6, P15, Q3, Q4. One pretest participant noted that he was unsure what should be included when asked about USDA DoD Fresh. We added additional details to multiple questions to clarify what we mean by USDA DoD Fresh.

• P16. We have made the following changes:

  • One pretest participant noted that the definition of “local” provided in this question allows SFAs to decide for themselves how to define it. Without providing more specific guidance, this respondent was concerned that there could be so much variation in how SFAs define local that the research team may not get the data they are looking for. We added the instruction, “Please use your SFA’s definition of “locally grown or produced” to answer this question.”

  • One respondent noted that she could only answer this question for foods she purchased from the farmer’s market and is unsure whether the other foods are local. We added “Don’t Know” as a response option for all items in the list. Respondents also made recommendations for fruits and vegetables to include or exclude from this list. We added corn, cabbage, green beans, spinach, green beans or wax beans, and zucchini.

Finally, we revised the following processes as part of the Quarterly Food Purchase Data request (F01.01 and F01.02):

• One SFA director reported that she oversees food service operations for multiple school districts. We added the request to exclude or flag food purchases for other school districts in the Food Purchase Data Checklist.

• One SFA director suggested the term “USDA Foods direct deliveries” was unclear and suggested referring to “brown box” as well. We added ”formerly known as brown box foods” in parentheses after “direct delivered USDA Foods” in both the Quarterly Food Purchase Data Request email and the Food Purchase Data Checklist.

• Respondents noted that they briefly looked at the Food Purchase Data Checklist, but primarily relied on the email text to compile their data. Two respondents forgot to include some invoices. We added the text, “Use this checklist to ensure you provide all food purchase data for the quarter” to both the Quarterly Food Purchase Data Request email and Food Purchase Data Checklist.

• One SFA director noted that respondents will interpret the term “local” differently. We clarified in both the Quarterly Food Purchase Data Request email and Food Purchase Data Checklist that they should only flag foods produced within their State.

• One SFA director noted that he spent a significant amount of time typing invoices into Excel, deleting columns, and separating text into separate columns in a velocity report to align with the data elements listed in the request. We changed the instructions on the format of the file in the Food Purchase Data Checklist to read, “Submit food purchase data in Excel or other editable format if available. Submit PDF files (e.g., invoices, etc.) if Excel files are not available. You may submit the data in the format in which it is available to you. You do not need to reorganize or reformat the data.”

• All respondents reported that it would be easier to email their food purchase data than to submit it to a secure transfer site. We have added an option submit the files via email.

• Two SFA directors were unable to identify processed end products containing USDA Foods as ingredients. One respondent was able to provide this information from a velocity report. We added text to the Food Purchase Data Checklist that this should be “available through velocity reports, USDA commodity unprocessed usage reports, or similar from your distributor.”

• All three SFA directors were able to indicate local foods purchased from local vendors. Two respondents were not able to identify locally grown or produced foods they received from their broadline distributor. We will keep the instructions as they currently read.

FFVP

Below we describe the changes to the FFVP instruments, content, and administration procedures based on the pretest. Note that in the final version of the instruments we have also responded to FNS comments from the revised instrument drafts. These revisions are not discussed in this memo. In general, the respondents were able to understand the questions and provide responses.

Based on feedback from the SNMs and other school cafeteria staff, we made the revisions listed below to the FFVP SNM Survey (F02.03).

• 8.2. We added “Too expensive” as a response option based on feedback from one respondent who said this is why they do not serve some fruits, such as watermelon.

• 9.2. We added “Too expensive” and “Requires cooking” as response options based on feedback from one respondent who said these are reasons why they do not serve some vegetables.

• 12.2. We revised response option b to read, “Low-fat or non-fat dip or salad dressing, not yogurt-based, including hummus” based on feedback from one respondent.

• 18. One respondent emphasized that a reason their school participates in FFVP is to provide additional food to their students, some of whom experience food insecurity. They also said that the opportunity to introduce students to fruits and vegetables was a reason why their school participates. To address this feedback, in Question 18 we added two response options: “We wanted to provide a free snack to students who experience food insecurity” and “FFVP provides an opportunity to introduce students to fruits and vegetables.”

• 20. One respondent described how inclement weather and other complications with deliveries can pose a challenge for their school when it comes to operating FFVP. To address this feedback, we added “Delivery logistics” as a response option in Question 20.

• Introduction. We heard from respondents that they sometimes needed to work with other staff, such as the SFA director, to complete the survey. Therefore we added introductory text explaining how the respondent can share their web survey link with others.

The process for administering the FFVP Menu Survey (F02.03) worked well in the field and all pretest respondents confirmed that they would readily be able to access the information needed to complete this instrument. To provide complete responses for the FFVP Menu Survey, one respondent said they would reference production records, menus, and receipts as needed. Another respondent said they would refer to their mainline vendor’s online database, where detailed product information is stored. We confirmed that the burden estimate of 30 minutes is accurate. We did not make any changes to the FFVP Menu Survey or administration procedures based on the pretest.

Based on the findings from the pretest with the SNMs and other school cafeteria staff, we revised the following in the Observation Guide (F07):

• We added a question to the end of the guide for field staff to indicate whether students were consuming non-FFVP snacks during the FFVP snack time.

Based on the findings from the pretest with the students, we revised the following in the Student Interview (F08.01):

• 2a, 2b, and 20. During pretesting we found that these questions were unnecessarily lengthy, which made them more difficult for respondents to understand. To address this finding and improve comprehension we simplified the wording of these questions.

• 74 and 75. We removed questions about race and ethnicity because the respondents did not understand them. We will request race and ethnicity data from schools during the process of student selection, and from parents when active consent is required.

1 The one SFA that was not selected was a single charter school in Minnesota.

2 The two students were selected by the school principal.