Appendix E Web Survey Pre-test Memo

School Meal Operations Study
Y5 Pre-test Findings

To:

Mathematica

Darcy Güngör and Susie Barr

From: Myah Scott, Joshua Holbrook, and Veronica Severn
Date: 3/12/24
Subject: School Meals Operations Study: Year 5 Pre-Test Findings (51903)

A. Introduction

This memorandum describes key findings from the School Meals Operations (SMO) Study Year 5 pretest of the State Agency Child Nutrition (CN) director survey and the revisions we recommend for the
instrument based on pre-test findings.
Mathematica conducted the pre-test in February 2024 with four respondents, which included State CN
directors in Michigan, Nebraska, New York, and Oklahoma Department of Human Services. We selected
respondents based on their interest and availability to participate within the pre-test time frame.
We emailed hard-copy versions of the survey to confirmed respondents and instructed them to note any
instructions or questions that were unclear or questions that were difficult to answer. We asked
respondents to track how long it took them to complete the survey. We conducted 30-minute debriefing
interviews with each pre-test respondent to identify questions that were confusing or difficult to answer,
and to gather their recommendations for changes. Appendix A is the version of the State Agency CN
Director Survey used in the pre-test, and Appendix B is the debriefing protocol for State CN directors.
The final version of the State Agency CN Director Survey (3.5) will be delivered to FNS in April 2024.

B. State Agency CN Director Survey
1. Burden
We asked respondents to record how long it took them to complete each section of the survey, including
the time needed to gather relevant data or information. Three of four respondents provided the time they
started and finished each survey section. One respondent provided the time they estimate it will take to
gather data needed to complete the survey instead of documenting their actual start and finish times for
each survey section. Respondents spent an average of 20 minutes completing the survey (Table 1).
However, two respondents estimated that the survey will take longer than 30 minutes to complete when
including time to gather the requested data. One respondent reported that it will take them an additional
30 to 40 minutes to gather site and SFA counts for sections B and D. Another respondent estimated it
would take over two hours to complete the survey. The Y5 State Agency CN Director Survey will be selfadministered online, so completion times could be slightly less because of programmed automatic skips in
the survey.
Although the average completion time, including both the reported and estimated time, is 54 minutes, we
estimate this survey could take some respondents less than 30 minutes to complete, but other respondents
may take up to two hours. Survey sections B and D are estimated to have the longest completion times,
followed by section E. We recommend reviewing these sections to identify low priority questions to drop
to ensure respondents can complete the survey within the 30 minute burden estimate.

Mathematica® Inc.

1

School Meal Operations Study
Y5 Pre-test Findings

Mathematica

Table 1. State CN Director Survey pre-test completion times by section and overall (in minutes)

Respondent Overall

Section A:
Meal Pattern
Requirements

Section B:
Summer
NonCongregate
Section C:
Meal Service Buy American

Section D:
Paid Lunch
Equity

Section E: Child
Nutrition Data
Systems

1a

14

1

3

4

1

5

2

20

1

7

1

2

9

3b

27

2

8

9

2

6

4c

156*

1

60*

5

60*

30*

Average

54

1

20

5

16

13

SY = school year.
a This

respondent did not have any local program operators operating SSO or SFSP.

b Respondent

D.

estimates that it would take an additional 30 to 40 minutes to collect accurate data for sections B and

c This

respondent recorded how much time they estimate it would take to gather data needed to complete sections B,
D, and E, instead of recording their actual start and stop times for those sections. The times listed for sections A
and C are reported times.

* These times are estimates of how long it would take to complete the section, not reported times.

2.

Modifications to the State Agency CN Director Survey

Most respondents reported that the survey was generally easy to understand and clearly organized, and
one respondent recommended reorganizing the survey to make it easier for SAs to delegate sections or
certain questions to appropriate staff members. All respondents noted that the relevant data from the
2023–2024 school year will be accessible when this survey is fielded in the fall, but some respondents
expect it will take more than 30 minutes to gather the requested data and complete the survey. Overall, the
respondents’ feedback yielded several recommendations for changes to the survey.
Table 2 provides additional details about pre-test respondents’ feedback on the survey and the changes we
recommend to address their comments. In response to their feedback, we recommend reorganizing the
survey questions, removing or revising certain questions to clarify intent, adding or revising terms, and
adding some additional response options.

Table 2. State Agency CN Director Survey respondent feedback and recommended changes
Questions
Overall organization
and flow of survey
sections

Mathematica® Inc.

Respondent feedback

Survey changes

One respondent shared that it is easier for
them to collect responses from staff when the
survey is organized by CN Program. They
recommended reordering the survey by moving
sections C (Buy American) and D (Paid Lunch
Equity) to immediately follow section A (Meal
Pattern Requirements).

We recommend reordering the survey so
that sections that apply to specific CN
Programs are grouped together. We
suggest the following order:
A. Meal Pattern Requirements
B. Paid Lunch Equity
C. Buy American
D. Summer Non-Congregate Meal Service
E. Child Nutrition Data Systems

2

School Meal Operations Study
Y5 Pre-test Findings
Timing

Mathematica
Two respondents estimated that the survey
would take more than 30 minutes to complete
when including time to gather the requested
data. One respondent estimated an additional
15 to 20 minutes for each section that asks for
number of LPOs. Another respondent
estimated needing about two hours to gather
information to complete the survey, citing that
more time is needed to gather information
about non-congregate sites, bulk meals, and
paid lunch equity.

We suggest revising the email that
Reginal Offices send to SAs and the
advance email to include a list of the data
requested to help respondents identify the
necessary data before they begin the
survey.
We also suggest making significant cuts to
the survey sections and questions. FNS
should consider dropping lower priority
and higher burden questions to ensure
respondents can gather the requested
information and complete the survey
within the 30 minute burden estimate.
Below are the questions respondents
expect to take the most time to answer.
B1 a/b
B2 a/b
B3 a/b
B4
B5 a/b
B6 a/b
B7 a/b
B9 a/b
B11 a/b
D1
D2
E21

Section B organization One respondent shared that it is easier for
and flow of survey
them to collect responses from staff when the
survey is organized by CN Program. They
questions
recommended reordering the questions by
displaying questions related to SSO together
and questions related to SFSP together.

We recommend reordering the survey so
questions about specific CN Programs are
grouped together. We suggest the
following order:
SSO: B1a-B3a, B5a-11a
SFSP: B1b-B3b, B4, B5b-11b

B5a, B5b

We recommend revising the glossary and
items B5 and B6 to include the definition
listed below for “migrant site.” We
recommend displaying the definition at the
first mention of the term, and then
including the definition as hover text on
subsequent items that reference the term.

All respondents referenced that federal
guidance provides a definition for the term
“migrant site” and indicated that these sites
serve children of migrant workers. One
respondent recommended defining this term in
the survey glossary section.

“A migrant site is a site that predominantly
serves the children of migrant workers.”
B6a, B6b

Mathematica® Inc.

One respondent reported that they cannot
easily access the data on non-congregate sites
right now but likely will be able to access this
data more easily later this year, after a few
systems updates. They noted that it can take
months to get this data from a vender for SAs
that use a vender for their data systems.
Another respondent reported that they must
access more than one system to obtain data on
non-congregate sites.

We recommend revising the introduction
section to include a list of the data
requested to help respondents prepare
before they begin the survey.

3

School Meal Operations Study
Y5 Pre-test Findings

Mathematica

B8a, B8b

One respondent reported that the term “typical” We recommend revising the question by
was confusing and it was unclear whether they replacing the word “typical” with “most
should report the average number of days’
common.”
worth of meals that were distributed or the
mode.

B10a, B10b

One respondent reported that this question
was confusing. They interpreted the question to
ask, “How much of the meal distributed was
made up of bulk components?” This
respondent thought SAs could interpret this
question differently.
One respondent recommended defining “Bulk
food” in the survey.

For clarity, we recommend revising the
question to read. “How much of the meal
distributed was made up of bulk
components?”
We also recommend revising the glossary,
B9, and B10 to include the following
definition for bulk food packages: “Food
packages that contain one or more items
that could be used for multiple meals or
portion sizes. For example, a quart of milk
provides four 1-cup servings.”

C1

One respondent chose “No” for C1 because
the SA does not have a State-specific policy.
This State implements the federal policy but
the language of options one (Yes, our State
implements the Federal policy) and two (Yes,
our State has a State-specific policy) did not
resonate with the respondent.

We recommend replacing option 0 (No)
with, “No, our State does not have a Statespecific policy or implement the Federal
policy."

D overall

One respondent noted it would be helpful to
receive a list of data points needed to complete
the survey ahead of receiving the survey, as
this would help SA staff collect the appropriate
data.

As noted above, we recommend revising
the email that Reginal Offices send to SAs
and the advance email to include a list of
the data requested to help respondents
identify the necessary data before they
begin the survey.

E1/E3

All respondents reported handling NSLP and
SBP processes in the same way, and noted it
is not necessary to ask about the use of paper
forms for these programs separately.

We recommend combining E1 and E3,
and E2 and E4, to ask about NSLP and
SBP processes in the same questions, for
example, “For NSLP and SBP, did your
State agency rely on paper forms (as
opposed to digital forms) for any of the
listed processes in SY 2023-2024?”

E1/E3/E5/7/E9 ”digital Two respondents were unsure of the definition
of digital forms and reported going “back and
forms”
forth” about the meaning of this term before
deciding on a response. For example, these
SAs reported using digitally saved Excel
spreadsheets and PDF forms for some
processes. One respondent reported that the
data from digitally saved Excel spreadsheets
and PDFs are not shared with other systems,
so this respondent considered them paper
forms. This respondent also recommended that
the study team define “digital forms” in the
survey. The other respondent decided to count
digitally saved Excel spreadsheets and PDFs
as digital forms.

Mathematica® Inc.

We recommend revising the glossary,
section E introduction, and relevant
section E items to include a revised
definition for “digital forms.” We
recommend displaying the definition at the
first mention of the term, and including the
definition as hover text on subsequent
items that reference the term.
When defining digital forms, we
recommend that FNS clarify whether
digitally saved files (for example, Excel
spreadsheets, fillable PDF forms, Word
documents, or scanned PDF forms) that
do not share information with other
systems are considered digital or paper
forms for the purpose of the survey.

4

School Meal Operations Study
Y5 Pre-test Findings

Mathematica

E1/E3/E5/7/E9
“procurement”

One respondent asked if they should think
We recommend that FNS clarify whether
about State level procurement when answering SAs should think about State or local
program operator procurement when
items E1/E3/E5/7/E9.
answering this question.

E1/E3/E5/7/E9
response options

Some SAs oversee SSO but did not have any
LPOs operating SSO.

We recommend adding the response
option below to items E5 and E7:
E5: “No SFAs operate SSO”
E7: “No sponsors operate SFSP”

E2/E4/E6/E8/E10

E11/E13

E15/E16

E20

One respondent reported that sometimes staff
choose to use paper forms, for example,
because staff have not been trained to use
systems or electronic forms.

We recommend adding the response
option below to items E2/E4/E6/E8/E10:

One respondent reported modifying their
existing system to accommodate the new data
for non-congregate meal service and
recommended adding a response option for
this situation.

We recommend adding the response
option below to items E11/E13:

One respondent described a data system as
something that was comprehensive and able to
integrate information. The respondent did not
consider storage of electronic documents (for
example, Excel spreadsheets and PDF forms)
with static information to be a data system. The
respondent recommended clarifying the study
definition of “CN data system” and placing the
definition somewhere that would “jump out” to
respondents.

We recommend revising the glossary,
section E introduction, and relevant
section E items to include a revised
definition for “CN data system.” We
recommend displaying the definition at the
first mention of the term, and then
including the definition as hover text on
subsequent items that reference the term.
When defining CN data system, we
recommend that FNS clarify whether an
integration feature is required to meet the
definition of a data system. For example, a
definition could be, “Digital systems used
to share, store, and/or manage CN
Program data, including digital forms.”

This respondent described having a digital
place to store claiming data used for federal
reporting, but they do not consider the agency
to have a data system overall. The respondent
reported that the system is old and the
respondent doesn’t have a way of knowing
where it is hosted.

We recommend leaving the question text
as is. Three of four respondents were able
to answer this question. Respondents who
do not have a way of knowing where their
data system is hosted can select “Don’t
know.”

One respondent reported that the TIG could
have been used to develop a new system,
implying a new system could be for something
other than for Summer EBT.

We recommend revising the response
option “Replaced an old system” to read,
“Built a new system or replaced an old
system.”

One respondent noted that TIG funds
potentially could have been used to pay a staff
member to support how funds are used, but the
respondent was unsure if paying staff was an
approved use of TIG funds.

We reviewed the Non-competitive
Technology Innovation Grant (nTIG)
webpage to identify potential uses for
these funds. We recommend adding the
response options below:

“State agency staff chose to use paper
forms”

“Updated existing data systems”

•
•
•

Mathematica® Inc.

Purchased technology equipment
and/or software or hardware
Hired new staff or contractors
Trained new staff or contractors

5

School Meal Operations Study
Y5 Pre-test Findings
E21

Mathematica
Respondents thought about different elements
of maintenance and operations costs when
answering E21. Two respondents noted that it
would take significant time to gather accurate
cost information to answer this question.

We recommend dropping this item.

Examples of various cost elements that SAs
considered when answering E21 are described
below.
•

Costs for “up keep,” server or cloud
based storage, and login and annual
licensing costs.

•

Costs for IT programming and CN
program staff time, and resources to
keep the system maintained.

•

Funds paid to the department of
management, technology, and budget
on a quarterly basis for the data
systems.

•

Costs associated with an annual
maintenance agreement with a
vendor.

CN = Child Nutrition; LPO = Local Program Operator; EBT = Electronic Benefit Transfer; SSO = Seamless Summer
Option; TIG = Technology Innovation Grant.

Mathematica® Inc.

6