L Sampling Plans

L Sampling Plans.docx

2024–2025 National School Foods Study

L Sampling Plans

OMB: 0584-0698

Document [docx]

Download: docx | pdf

APPENDIX L. SAMPLING PLANS

Sampling Plan

Here we present the plan for selecting the study samples and developing weights for the analysis. A key goal of the 2024-2025 National School Food Study is minimizing data collection costs and respondent burden while facilitating comparisons across the three study components (SNMCS-II, SFPS-IV, and FFVP). The sampling plan will provide nationally representative estimates of SFAs, schools, students (and their parents), and meals in SY 2024-2025.¹ In addition, the sample is designed to lead to estimates that are as comparable as possible with the estimates from SNMCS-I, SFPS-III, and, to a certain extent, FFVP-I, with the required levels of statistical precision. It is also designed to incorporate the Outlying Areas component (i.e., Alaska, Guam, Hawaii, Puerto Rico, and USVI) as part of the SNMCS-II study component. Two expanding policy-relevant subgroups are also incorporated into the design: SFAs and schools that are in Healthy School Meals for All States (HSMFA), formerly referred to as Universal School Meals or USM) and those that participate in the Community Eligibility Provision (CEP) option.

2.1. Overview of the Sample Design

The selection of the SFAs is the starting point of the sample design for the study. The sample is designed to meet the needs of the study objectives, as defined in Supporting Statement Part A Table A.1. The universe for SNMCS-II (Objectives 1 through 4) includes public school and charter-only SFAs (SFAs that only serve charter schools) in the contiguous 48 States and the District of Columbia (DC) plus the five States and territories in the Outlying Areas (Objective 3-.² SFPS‑IV (Objective 5) includes only public school SFAs, both in the contiguous 48 States and the District of Columbia. The FFVP evaluation (Objective 6) starts with a sample of SFAs (in the 48 contiguous States plus DC) that have at least one elementary school participating in that program. To address the SNMCS-II evaluation (Objectives 1 through 4), the sample for the study also includes schools, students, parents, and meals. To address the FFVP evaluation (Objective 6), the sample includes schools, students, and parents. The sample design for the study includes the following key features:

Creation of three random subframes, or groups, for selecting most of the SFAs in the sample. The goal of the three groups is to manage the SFA burden and reduce the cost of field operations. The SFPS-IV sample will be selected from one of the three subframes (Group 1), while the SNMCS-II sample will be drawn from all three subframes. The FFVP evaluation sample of SFAs will be drawn from a separate but overlapping frame of those SFAs with at least one elementary school participating in that program (a subset of the SFAs in the three subframes for the SFPS-IV and SNMCS-II study components).
A three-stage sample selection for a portion of the SNMCS-II sample, where SFAs are selected from each of the three subframes at the first stage, schools within SFAs are selected in the second stage, and (for Groups 2a and 3) either students (and their parents) or meals within schools are selected in the last stage. (Group 1 has only one stage of selection.)
Stratification by subgroups of interest and probability-proportionate-to-size (PPS) sampling for selecting the SFAs in the first stage. For the SNMCS-II and SFPS-IV study components, this approach ensures a sufficient sample size for subgroups of interest at the SFA, school, student/parent, and meal levels , and it achieves the targeted precision requirements without reducing comparability with previous studies. For the FFVP evaluation component, stratification gives some control over the composition of the SFA sample.
Adequate representation of HSMFA and CEP SFAs and schools. The HSMFA and/or CEP SFAs and schools are key subgroups of interest for the study. SFAs and schools that offer free meals to all students under Provision 2 or Provision 3 will be included in the CEP subgroup.
The sample will ensure adequate representation of SFAs and schools operating under CEP and/or in a HSMFA State and adequate precision of estimates for them by HSMFA reexamining the distribution of SFAs participating in CEP and/or HSMFA once we have the updated list of SFAs to be used for sampling, along with HSMFA and CEP indicators.³ Because both programs are continually expanding, it is likely that we may only need to stratify and oversample based on one of these indicators, using implicit stratification within strata for the other.
To account for the overlap between SFAs participating in CEP and those in HSMFA States, we will consider creating the following categories for sampling purposes:

HSMFA: SFAs in HSMFA States;
CEP: SFAs in non-HSMFA States with all schools participating in CEP or Provision 2/3;
Neither: SFAs in non-HSMFA States with one or more schools not participating in CEP or other Provision.

Exhibit 2.1 provides a summary of the sample design for the study. The sample design of SNMCSII will be implemented as planned in in SY 2019-2020 (with the exception of the possible HSMFA and CEP stratification), Group 1 will be expanded to accommodate the addition of SFPS-IV, and the incorporation of the FFVP. Therefore, the SNMCS (Mainland) study sample is designed to yield data from 522 SFAs, 1,061 schools, and 3,302 students, and 1,800 parents. In addition, the study will collect plate waste data to yield a sample of 4,140 reimbursable lunch trays and 2,120 reimbursable breakfast trays. The Outlying Areas sample will include about 34 SFAs (via selected schools) and 138 schools. The SFPS study sample will yield data from 364 SFAs, which includes 88 SFAs already sampled for the SNMCS (Mainland) study sample. The FFPV study sample will yield data form 100 SFAs, 100 schools, and 800 students.

The sampled SFAs will be divided into groups. Groups 1a, 1c, 2a, and 3 (n= 522 SFAs) will provide data to address Objectives 1-4. Groups 1a and 1b (n=364 SFAs) will provide data to address Objective 5. Group 2b (n=100 SFAs) will provide data for the FFVP evaluation study component (Objective 6). The very large SFAs (referred to as the “certainty” SFAs) are included in Groups 1a, 2a, and 3 (and in 2b if they include at least one FFVP elementary school but are only counted once to get a unique number of SFAs equal to 522 (excluding the Outlying Areas component and FFVP evaluation).

Sample sizes described here are stated in terms of the numbers of participating SFAs, schools, and students and their parents (that is, the target completed sample sizes). The sizes of the samples initially selected will be expanded to allow for nonparticipation due to ineligibility or noncooperation, based on evidence from recent studies. Unless otherwise noted, sample sizes listed here are based on target completed sample sizes.

2.1.1. Key Considerations and Methods

The national precision requirement for the SNMCS-II SFA Director Survey (Appendix F03.01) is ±0.05 for a population proportional outcome of 0.50 with a 95 percent confidence level. The subgroup precision requirement is ±0.10 for domains of 25 percent of the population with a population proportional outcome of 0.30 and a 95 percent confidence level.³ For the SFPS-IV component, the national precision requirement is ±0.05 for a population proportional outcome of 0.50 with a 90 percent confidence level. This survey sample will provide subgroup precision of ±0.10 for domains of 25 percent of the population with a population proportional outcome of 0.50 with a 90 percent confidence level. For the FFVP evaluation component, the national precision for school estimates in each group is ±0.10 for a population proportional outcome of 0.30 with a 95 percent confidence level; and is ±0.05 for student estimates in each group.

Prior to selecting SFAs, the study team will randomly divide the sampling frame into three separate subframes of the same size (i.e., Groups 1, 2, and 3) after separating the charter-only SFAs (those SFAs serving only charter schools). Because these groups are stratified random samples of the SFA frame, each group is expected to have the same distribution of SFAs in the frame in terms of SFA size (number of schools and students in the SFA), HSMFA and/or CEP status (explicit strata), and other SFA characteristics used to implicitly stratify the SFA sample HSMFA. In addition, the creation of the random groups ensures the representation of the SFA universe since SFAs will then be sampled from each nationally representative subframe.

SFAs eligible for the SNMCS (Mainland) or the SFPS serve public non-charter or charter schools in the contiguous 48 States and the District of Columbia. Ineligible SFAs include those serving private schools, only institutionalized populations, or those operated by States or the Federal government. These ineligible SFAs will be removed from the frame before the creation of the random groups. The SNMCS-II Outlying Areas component also includes those in Alaska, Hawaii, and the U.S. territories of Puerto Rico, Guam, and the Virgin Islands. For the FFVP evaluation study component, we will form a separate but overlapping sampling frame of all SFAs that have at least one elementary school participating in the FFVP, regardless of their sampling status for the SNMCS-II or SFPS-IV study components. These SFAs will also have a chance of selection into those other study components.

Based on our calculations, it is not feasible to sample sufficient numbers of FFVP participating schools under the planned SNMCS-II component without greatly increasing burden on respondents and negatively impacting the precision of the SNMCS-II component. As a result, to meet FNS’s goals for the FFVP study objectives, we included another SFA sampling stratum (Group 2b), independent of the SNMCS-II sampling stratum (Group 2a) to identify a nationally representative sample of FFVP participating elementary schools and students. Although this FFVP evaluation sample will be distinct from the sample of SFAs and schools included in the SNMCS-II components, we plan to administer similar data collection instruments across these two samples to allow for cross-component comparisons when possible.

Exhibit 2.1. Summary of the sample design

Notes: Sample sizes show target number of completes.

Each of the certainty SFAs will be included in Groups 1a, 2a, 2b (if they have at least one FFVP school), and 3. They will participate in all data collection activities for these groups.

FFVP = Fresh Fruit and Vegetable Program;; SFA = school food authority; SFPS = School Food Purchase Study; SNMCS= School Nutrition and Meal Cost Study.

Two types of SFAs have special treatment before the creation of the groups. First, the “certainty” SFAs based on their probability of selection, defined as those eligible SFAs serving a very large number of schools and students , will be invited to participate and set aside from the remainder of the frame. Each will be assigned to Groups 1a, 2a, 2b (if they have at least one FFVP elementary), and 3.⁴ This feature of the sampling design ensures the representation of the largest SFAs in all three groups, including Group 1, which contains the data collection to address the research questions in SFPS (Objective 5). Because these certainty SFAs represent only themselves, if they refuse to participate despite our best efforts, they will not be represented in the sample. However, we can release other large backup SFAs to compensate for the loss in sample size and associated precision.

The second type of SFAs with special treatment is charter-only SFAs (SFAs that serve only charter schools). These SFAs will also be identified and set aside into their own sampling frame (for the Group 1c sample) before creating the three random groups.

2.2. Sampling Frames

Selecting the samples requires high-quality sampling frames at each stage. Following the approach used in SNMCS-I, the SFA sampling frame for the study will be constructed by combining the most recent data from the SFA Verification Collection Report (FNS-742), the U.S. Department of Education’s (ED) Common Core of Data (CCD) “Local Education Agency (LEA or school district) Universe Survey” collected annually by the ED’s National Center for Education Statistics, and a Census file from the Small Area Income and Poverty Estimates Program with school district-level estimates of school-age children in poverty.⁵ The SFAs are those LEAs or districts that are found on the FNS-742. We will use linkage methods to combine these two files, as there is no unique identifier that exists on both files. To select the FFVP evaluation SFAs, we will obtain from FNS a list of elementary schools participating in the FFVP in 2023-2024 and deduplicate it to generate an SFA level file for sampling.

For SNMCS, the sampling frame for selecting schools within SFAs will be the most recent available CCD school-level file. The CCD file contains more detailed information than the FNS-742 and has information that allows the elimination of some types of ineligible schools (such as those serving institutional populations). The CCD school-level file also contains enrollment figures, grades served, demographic information, and charter school status that could be useful for explicit and implicit stratification and/or weighting adjustments. For the FFVP evaluation school sample, we will use the FNS-provided list of schools offering the FFVP to sample one participating school per sampled SFA.

The sampling frames for enrolled students will be lists obtained from sampled schools.

2.3. Selecting the Sample

Two sampling techniques are used to control the sample in the study: stratification and PPS selection. Explicit stratification controls the sample sizes on select domains of interest when the population and/or sample sizes are large enough to support stratification (that is, allowing at least two units to be sampled from each stratum). When the population and/or sample sizes become smaller (for example, a selection of three schools within an SFA), the combined use of stratification and PPS sampling within the strata can control the sample to a more granular degree to increase the yield of certain types of SFAs or schools, which otherwise would not be sufficiently represented in the sample. This process gives the SFAs or schools within subgroups of interest a higher measure of size (MOS), so they are selected with a higher probability than those in other domains. In particular, in Group 2a, SFA sampling is based on an MOS that includes the percentage of students approved for free or reduced-price meals in the SFA to increase the rate of such SFAs, schools, and students. PPS sample selection also accounts for differences in the number of schools or students within an SFA and school, respectively, to make the design more efficient for estimates at those levels in a multi-stage sample such as SNMCS and FFVP.

Chromy’s sequential PPS selection procedure will be used in all groups using explicit and implicit stratification (Chromy, 1979). Chromy’s sampling method implicitly stratifies (sorts) the units within each stratum by various characteristics and then selects the sample in a random sequential fashion. In this manner, the sample produced will have the same proportion of cases of a given type (for example, urban versus rural) as observed in the sampling frame (or very near to it). Therefore, the sampling process is subject to a smaller level of variation and produces a sample that matches the profile of the population.

The details of the stratification for controlling sample sizes of schools and students (including possible oversampling) are described in the following sections. Note that the sample and precision numbers that follow are based on simulations from the first iteration of SNMCS-II, using the 2018-2019 FNS-742 file and the 2017-2018 CCD file, with some data coming from the 2016-2017 CCD file. By the time we are ready to select the samples for this study, there will have been important changes in the numbers of SFAs and schools participating in the HSMFA and CEP programs. In addition, there have been additional changes to the study priorities since then that warrant re-examining the impact and competing demands of various design features. The numbers in these tables will be updated during sampling using distributions from the most recent data available and the design modified as necessary.

2.3.1. Sampling SFAs

After creating the SFA sampling frame and excluding ineligible SFAs, the study team will select SFAs in four steps after excluding SFAs ineligible for the study. First, the certainty SFAs⁶ will be identified and set aside. These certainty SFAs will each be assigned to Groups 1a, 2a, and 3.⁷ Second, SFAs that serve only charter schools will be identified and assigned to the charter-only strata in Group 1c. Third, the overall frame of the remaining SFAs will be stratified, and random selection methods will be used to assign them to the three groups in equal numbers. Fourth, the samples of SFAs will be selected within the groups using a stratified random PPS sample selection. Exhibit 2.2 summarizes the key parameters of the study design, which are also discussed below.

Exhibit 2.2. Key parameters of the sample design: SNMCS-II (Mainland), SFPS-IV, and FFVP evaluation

	Group 1			Group 2a (n=133)	Group 3 (n=265)	Group 2b (n=100)
	Group 1a (n=88)	Group 1b (n=276)	Group 1c (n=48)
SFA type	Public SFAs		Charter-only SFAs	Public SFAs		Public SFAs with one or more FFVP schools
Stratification	Up to 27 strata		3 strata	Up to 27 strata		3 strata
Stratifiers	HSMFA/CEP status Number of schools (3 categories) Number of students (3 categories)		HSMFA/CEP status	HSMFA/CEP status Number of schools (3 categories) Number of students (3 categories)		HSMFA/CEP status
Sample allocation	Semi-proportional allocation					Proportional allocation
Oversampled groups	HSMFA and CEP SFAs^a, larger SFAs by number of schools or enrollment					None
MOS for PPS selection	Number of students (weighted by FRP%)			Square root of the number of schools (weighted by poverty area)		Number of FFVP elementary schools

Notes: SFAs are in the 48 contiguous States and the District of Columbia. SFA sample sizes are target completes.

^aIn Group 1, the statistics of interest such as purchasing practices are at the SFA level; sampling using the total SFA enrollment (total number of students in the SFA) ensures that SFAs with large number of students are represented. In Groups 2a and 3, the statistics of interest are related to the schools within SFAs. Sampling based using the number of schools in the SFA ensures that SFAs with a large number of schools are represented in the sample, but smaller SFAs also are represented. In SY 2022-2023, slightly under one-half of schools (40,200) and SFAs (6,300) participated in CEP (https://frac.org/community-eligibility). CEP SFAs = SFAs serving only schools offering free meals to all students under CEP, Provision 2, Provision 3, or State-funded provisions and not in a HSMFA State. Non-HSMFA/CEP SFA = SFAs with one or more schools not offering free meals to all students. At this level, proportional representation of HSMFA and CEP SFAs will likely be sufficient without oversampling, making the sample more efficient. Inclusion of other SFAs and schools offering free meals to all students will further increase the precision of estimates for this subgroup.

CEP = Community Eligibility Provision; FFVP = Fresh Fruit and Vegetable Program; FRP = free or reduced-price; MOS = measure of size; PPS = probability-proportionate-to-size; SFA = school food authority; HSMFA = Healthy School Meals for All.

Large SFAs. In addition to the certainty SFAs assigned to Groups 1a, 2a, 2b (if they have at least one FFVP school), and 3, other SFAs may be selected with certainty within each group due to their size. The target completed sample sizes presented in the following sections include large SFAs selected with certainty and non-certainty SFAs in each group.

Group 1 SFAs (n = 412 total across Groups 1a, 1b, and 1c). Because there are disproportionally few SFAs with many schools (serving 20 or more) and/or students (serving 12,000 or more) and a relatively large number of charter-only SFAs (which are only included in Group 1), too few large SFAs or too many charter SFAs could be surveyed if simple random sampling is used. In addition, based on specifications in the SNMCS-II study plan, the study also needs to target, to a small degree, SFAs with students in poverty.

Stratified sampling, coupled with a PPS selection process, will be used in the study to address these concerns. This stratification will enhance the ability to examine SFA-level results at a more granular level without reducing the ability to compare data from the study components to their earlier counterparts.

The sample will be allocated across these strata in a semi-proportional manner, relative to the number of SFAs in each stratum, so that in the smaller strata, a higher sampling rate is applied to ensure enough SFAs are selected to meet subgroup-level analyses on the three dimensions. A higher sampling rate for HSMFA SFAs and other SFAs with all CEP schools may be employed to ensure a large enough pool of these schools for analyses.^⁸ Within each stratum, SFAs will be sampled using PPS methods so that SFAs with higher proportions of students approved for free or reduced-price meals have a slightly higher chance of selection than other types of SFAs. This approach will create a larger sample of these SFA types for analyses.⁹ Implicit stratification within each stratum will be used to help ensure the sample is balanced by FNS region and urbanicity. Because certainty selections will be identified first and removed from the frame, the sample allocation will be adjusted (these two steps are conducted interactively so that all certainty cases are accounted for), and the final sample will be pulled.

Because Groups 1a, 1b, and 1c are designed to support SFA-level analysis, their MOS is designed around the SFA’s characteristics, including the poverty status of students.

Up to 27 strata for Groups 1a and 1b will initially be created based on all combinations of (1) whether or not an SFA is in a HSMFA State or all schools in the SFA are CEP schools or neither, (2) three ranges in the number of schools, and (3) three ranges in the number of students associated with the SFA.^¹⁰ When crossing the three categories of number of schools with the three categories of number of students, there will likely be a few sparse categories, such as SFAs with a large number of students and a small number of schools, or vice versa. As part of the sample implementation process, we will examine the number of SFAs in each of the strata and combine sparse strata as needed. Similarly, depending on the distribution of CEP and/or HSMFA status of SFAs, we may choose to explicitly stratify by one and not both indicators. Three or four strata for Group 1c will be created using the HSMFA/CEP status.¹¹ After the certainty SFAs are assigned to Group 1a, an initial sample of non-charter public SFAs will be drawn from the 27 (or fewer) strata and randomly assigned to either Group 1a or Group 1b (within those same strata). The initial samples for these groups will be sufficient to yield a completed sample of 88 SFAs in Group 1a (when combined with the certainty SFAs) and 276 SFAs in Group 1b. The SFAs sampled for Groups 1a and 1b (SFPS) will be randomly assigned to each of four data collection quarters using the original sampling strata as well as Farm Production Regions. The certainty SFAs will be randomly assigned across the quarters. If there are 6 of them, we will randomly assign 2 to each of quarters 1 and 2 and 1 to each of quarters 3 and 4. Finally, a sample of charter-only SFAs sufficient to yield 48 completed surveys will be selected in Group 1c. Details for the stratification and PPS selection are described in the following paragraphs. The starting samples and response rate assumptions for each Group are presented in Section 3.3.

Group 2a SFAs (n = 133). The Group 2a sample is designed to complete the SNMCS-II SFA Director Survey (Appendix F03.01) and a sample of schools and students to participate in the SNM Survey, Principal Survey, Menu Survey (Appendices F04.01, F03.07, and F02.01), and other school-level data collection activities, as well as student- and parent-level data collection. As in Groups 1a and 1b, public school SFAs will likely be stratified by three HSMFA/CEP status strata (whether the SFA is in an HSMFA State, whether all schools in the SFA are CEP, or other SFAs) and up to nine size strata based on the ranges in (1) the number of schools, and (2) the number of students associated with the SFA, for a total of 27 strata. An equivalent allocation plan to Groups 1a and 1b will be used, and PPS sampling within strata will be applied.^¹²Because Group 2a is also used to produce school-, principal-, and student/parent-level estimates, the MOS values will be adjusted for PPS selection using a MOS based on the square root of the number of schools in the SFA¹³ (to account for the multi-stage sample) in conjunction with higher MOS values for SFAs in high poverty areas. Before selecting the SFA sample, any additional SFAs to be selected with certainty based on their size will be identified and sampled. The study will then explicitly stratify SFAs not sampled with certainty into the strata for Group 2a and select a sufficient sample using PPS methods within each stratum to yield 133 total SFAs that will consist of the certainty SFAs assigned to Group 2a, the newly identified certainty SFAs, and the sample of non-certainty SFAs.

Group 2b SFAs (n = 100). As described above, the Group 2b will include only SFAs that have at least one FFVP school. We plan to stratify the sample by HSMFA/CEP status but will not disproportionately sample across strata. We will implicitly stratify by SFA enrollment. We will sample SFAs with PPS sampling, using the number of FFVP elementary schools as the measure of size, to minimize the weighting design effect and enhance precision at the school level. Although proportional sampling across strata improves efficiency for overall estimates, subgroup estimation will likely not be possible at the desired levels of precision. With a minimal design effect (estimated to be 1.2), we plan to recruit 100 SFAs for Group 2b.

Group 3 SFAs (n = 265). The Group 3 sample is designed to provide everything that the Group 2a sample provides, other than student- and parent-level data, as well as data for estimation of meal costs, school food service revenues, and plate waste. Plate waste observations (Appendix F09) will be conducted in schools in a random subset of 69 Group 3 SFAs. The Group 3 sampling will follow the same design as Group 2a. A sufficient sample will be selected to yield 265 SFAs in Group 3, including the certainty SFAs assigned to Group 3, any additional certainty SFAs for this group, and randomly selected non-certainty SFAs.

Exhibit 2.3 presents a summary of the SFA sample for SNMCS (Objectives 1-4). Estimates of the number of SFAs by the dimensions of stratum membership are provided along with the proposed sample allocation across the completed set of 522 SFAs in Groups 1a, 1c, 2a, and 3 combined.¹⁴ Population and sample estimates are also presented for various subgroups. These estimates are based on simulations reported in the SNMCS-II study plan using the 2018-2019 FNS-742 file and the 2017-2018 CCD file, with some data coming from the 2016-2017 CCD file or other sources. The precision levels associated with these sample sizes are presented later.

Exhibit 2.3. Target completed SFA sample sizes for Objectives 1-4

Subgroups	SFAs
	Population				Sample
	Count		Percentage		Count		Percentage
Number of Schools^a
1–2	4,512		36		168		34
3–4	3,911		31		148		30
5 or more	4,212		33		174		36
Number of Students^a
1–350	2,545		20		121		24
351–1,200	3,879		31		142		29
More than 1,200	6,211		49		231		47
Urbanicity
Urban	6,063		41		205		38
Rural	8,612		59		329		62
Percentage of Students in Poverty^b
0 to 17 percent	6,871		56		170		33
More than 17 percent to 35 percent	4,736		39		300		57
More than 35 percent	551		5		57		10
Percentage of Students Approved for F/RP Meals
0 to 45 percent	9,801		67		325		61
46 to 63 percent	3,031		21		123		23
64 percent or more	1,843		13		86		16
Charter SFA
Yes	2,040		14		48		10
No	12,635		86		486		90
HSMFA/CEP Status^c
SFAs in HSMFA States	1,482		10		112		21
Other SFAs where all schools are CEP schools	5,033		34		161		30
All other SFAs	8,159		56		261		49
FNS Region
Mid-Atlantic	1,488	10		50		9
Midwest	3,708	25		112		21
Mountain Plains	2,290	16		75		14
Northeast	1,651	11		59		11
Southeast	1,290	9		69		13
Southwest	2,219	15		93		17
West	2,029	14		75		14
Total	14,675	100		534^d		100

Source: Simulations from the first iteration of SNMCS-II, using the 2018-2019 FNS-742 file and the 2017-2018 CCD file, with some data coming from the 2016-2017 CCD file. During sampling, this table will be updated and a comparable table for Objective 5 SFAs will be produced.

Notes: The total number of cases across subgroups may not sum to the total due to rounding.

^aExcluding charter-only SFAs.

^bThe poverty variable has a considerable amount of missing data; the totals here reflect the non‑missing cases.

^cThe simulations in the SNMCS-II study plan did not include SFAs with other universal free meal provisions in the CEP SFA group.

^dTotal equals 534 because it includes the certainty SFAs in all three groups (Group 1a, 2a, 3).

CCD = Common Core of Data; HSMFA = Healthy School Meals for All; CEP = Community Eligibility Provision; FNS = Food and Nutrition Service; F/RP = free or reduced-price; LEA = local education agency; SFA = school food authority.

2.3.2. Sampling Schools in Groups 2a and 3

Public, non-charter schools will be sampled from both Group 2a and Group 3 SFAs, targeting an average of two participating schools per SFA in Group 2a and three participating schools per SFA in Group 3.^¹⁵ This will yield expected sample sizes of completed school-level data for 265 schools in Group 2a and 796 schools in Group 3.¹⁶

Before sampling, the following will be removed from the school-level frames: charter schools; schools that serve only prekindergarten or kindergarten students; schools that do not participate in the NSLP; and schools that are Residential Child Care Institutions (RCCIs). For Group 2a, the study team will select the schools within each SFA using PPS sampling with the square root of enrollment as the MOS, implicitly stratifying by school level. In many cases, the number of schools in the SFA is small.

Following SNMCS-I, the schools in Group 3 SFAs will be explicitly and/or implicitly stratified by school level (elementary, middle, and high schools) and, as feasible, a sufficient sample in each stratum will be selected to yield one completed school of each type for the cost study data collection activities.¹⁷ Because some SFAs will not have all three school types, the final sample sizes for each type of school will not be equal. However, the selection process adopted for the study increases the percentage of middle schools in the sample from 16.9 percent (based on the rate present in the school population) to approximately 23.0 percent to improve subgroup-level analyses for middle schools. When a school has grades that span typical elementary and middle school grades, the CCD codes it as an elementary school. Similarly, when a school has grades that span typical middle and high school grades, the CCD codes it as a high school. This may result in some sampled SFAs having no middle schools unless we reclassify them. To get enough middle schools in the sample for Group 3, we will discuss with FNS the possibility of reclassifying some of these situations as middle schools for sampling purposes. . A summary of the expected school sample is presented in Exhibit 2.4.

Exhibit 2.4. Target completed sample sizes: Schools in Groups 2a and 3

Subgroups	Population		Sample
Subgroups	Count or mean	Percentage	Group 2	Group 3	Groups 2 and 3	Percentage Groups 2 and 3
School Type^a
Elementary	55,300	60	129	358	487	46
Middle	15,668	17	53	188	241	23
High	21,746	24	83	249	333	31
Urbanicity
Urban	60,232	65	173	479	652	61
Rural	32,482	35	92	317	409	39
Racial/Ethnic Distribution of Students (mean %)
Non-Hispanic Black	14.84	n.a.	10.48	10.43	10.44	n.a.
Non-Hispanic White	51.38	n.a.	63.29	62.19	62.46	n.a.
Hispanic	24.22	n.a.	20.12	17.08	17.84	n.a.
Other	9.55	n.a.	6.12	10.30	9.26	n.a.
Students Approved for F/RP Meals
0 to 45 percent	40,624	44	101	283	384	36
46 to 63 percent	19,109	21	62	232	294	28
64 percent or more	32,981	36	102	281	383	36
HSMFA/CEP Status^b
Schools in HSMFA States	12,424	13	57	170	227	21
Non-HSMFA CEP schools	39,774	43	79	239	319	30
Other schools	40,516	44	129	387	516	49
FNS Region
Mid-Atlantic	9,828	11	27	60	87	8
Midwest	16,885	18	47	179	226	21
Mountain Plains	10,931	12	40	108	148	14
Northeast	9,080	10	27	85	112	11
Southeast	15,520	17	39	128	167	16
Southwest	13,544	15	47	142	189	18
West	16,926	18	38	94	132	12
Total	92,714	100	265	796	1,061	100

Source: Simulations from the first iteration of SNMCS-II, using the 2018-2019 FNS-742 file and the 2017-2018 CCD file, with some data coming from the 2016-2017 CCD file. The numbers in the table will be updated during sampling using the most recent data available.

^aSchool types are mutually exclusive. Schools classified as elementary include schools with middle and high school grades as well as elementary grades. Schools classified as high schools include schools with middle school grades as well as high school grades.

^bCEP schools are drawn from both the all-CEP SFA stratum and the not-all CEP SFA stratum. The simulations in the SNMCS-II study plan did not include SFAs with other universal free meal provisions in the CEP SFA group.

CCD = Common Core of Data; HSMFA = Healthy School Meals for All; CEP = Community Eligibility Provision; FNS = Food and Nutrition Service; F/RP = free or reduced-price; n.a. = not applicable; SFA = school food authority.

2.3.3. Sampling Students in Group 2a

As shown in Exhibit 2.5, a random systematic sample of students will be selected to yield eight completes in each Group 2a participating school (265 schools, 3,302 total students, 1,800 parents) to participate in the student- and parent-level data collection for various subgroups.

Exhibit 2.5. Target completed sample sizes: Students in Group 2a and student trays in Group 3

Subgroups	Group 2a		Sample Group 3
Subgroups	Students	Parents	Student lunch trays	Student breakfast tray
SchoolType^a
Elementary	1,549	844	1,456	745
Middle	754	411	1,365	699
High	1,000	545	1,318	675
Urbanicity
Urban	2,154	1,174	2,451	1,256
Rural	1,148	626	1,689	864
Race/Ethnicity
Non-Hispanic Black	391	213	544	279
Non-Hispanic White	2,166	1,181	2,808	1,440
Hispanic	557	303	634	325
Approved for F/RP Meals
Yes	1,966	1,072	2,300	1,177
No	1,336	728	1,840	943
FNS Region
Mid-Atlantic	337	184	332	170
Midwest	585	319	891	456
Mountain Plains	491	268	257	132
Northeast	343	187	421	216
Southeast	501	273	990	506
Southwest	587	320	692	355
West	458	250	557	285
HSMFA/CEP Status of School^b
Schools in HSMFA States	706	385	884	453
Non-HSMFA CEP schools	991	540	1,242	636
All other schools	1,605	875	2,013	1,031
Total	3,302	1,800	4,140	2,120

CCD = Common Core of Data; HSMFA = Healthy School Meals for All; CEP = Community Eligibility Provision; FNS = Food and Nutrition Service; F/RP = free or reduced-price; SFA = school food authority.

2.3.4. Sampling for Plate Waste Observations in Group 3

In a subset of 138 of the 796 Group 3 schools, student trays will be selected sequentially (every “nth” student throughout the meal periods) at random in the cafeteria line for lunch and for breakfast. A sufficient number of trays will be selected to yield 30 completed lunch tray observations per school and about 15 breakfast trays per school. This approach is expected to yield 4,140 student trays (plate waste observations) for lunch and 2,120 for breakfast . If multiple serving lines are present, the observations will be allocated evenly across the serving lines.

This subset of schools will be selected using a two-stage stratified sample design. First, we will identify which of the 265 participating Group 3 SFAs have at least 2 plate waste-eligible schools (defined below). Among these, we will randomly select approximately 85 SFAs with the goal of recruiting 69 SFAs to participate in the plate waste data collection. Next, approximately two schools will be selected from the plate-waste-eligible schools sampled in these SFAs to yield 138 schools (with an approximately equal number of elementary, middle, and high schools).

To maintain consistency with SNMCS-I, schools sampled for the plate waste observations must meet the following criteria: (1) lunch and breakfast (if the school offers breakfast) must be served and consumed in the cafeteria, and (2) schools must meet a minimum threshold for the number of reimbursable lunches served per day to help ensure that the target number of observations can be completed in each sampled school during a 1-day site visit. A minimum of 172 lunches must be served. Based on SNMCS-I, we expect approximately 87 percent of the tray observations to be included in the analysis, so an average of 34 to 35 trays must be observed per school. Field interviewers (FIs) can tag and observe one in every five trays as students exit the lunch line. Therefore, a school must serve a minimum of 172 lunches to reach the goal of 30 observations.

In SNMCS-I, fewer schools than expected met the initial eligibility criteria, making identifying schools for the plate waste study sample challenging. Moreover, the plate waste sample was not fully identified before the start of data collection, and information on the eligibility of schools for the plate waste study was unavailable from the sampled SFAs. Therefore, the plate waste sample was not entirely a probability sample, and traditional selection weights could not be constructed. These issues can be addressed by prioritizing screening for plate waste eligibility during recruitment. Questions were added to the SFA Director Planning Interview (Appendix C16) to ask Group 3 SFAs about the location of meal service in sampled schools and the number of daily reimbursable lunches. In addition, the total number of breakfasts and lunches claimed in all elementary, middle, and high schools for the school year will be collected in all Group 3 SFAs. These questions were added to the SFA Follow-Up Web Survey (Appendix F05.06). These counts will be used in the post-stratification of the plate waste weights and to decide on the number of SFAs to subsample for the plate waste study.

Student trays will be sampled for plate waste observation (Appendix F09) using a systematic sampling method. For example, every 10th student with a reimbursable meal might be selected to build the sample (the actual interval will be determined based on the number of reimbursable meals served in the school and the desired number of plate waste observations). To allow sampling weights to account for the probability of selection of student trays, the study will attempt to give all trays an approximately equal probability of selection across all meal periods and meal service lines.

2.3.5. Sampling for the Outlying Area Component

The Outlying Areas respondent universe includes public SFAs and schools in Alaska, Guam, Hawaii, Puerto Rico, and USVI. The universe will be based on the combined FNS-742 and the CCD. The estimated size of the respondent universe, along with the target completed sample sizes, are presented in Exhibit 2.6.

Exhibit 2.6 Respondent universe and sampling plan for each outlying area

State/ territory	Population		Target completed sample	Expected Precision (in dollars)
State/ territory	SFAs	Schools	SFAs	Schools	Confidence half interval	MDD to mainland (50% power)	MDD to mainland (80% power)
Alaska	43	431	29^a	51	0.31	0.45	0.65
Guam	1	41	1	24	0.32	0.47	0.67
Hawaii	1	255	1	63	0.33	0.47	0.67
Puerto Rico	1	1,108	1	NA	NA	NA	NA
USVI	2	27	2	NA	NA	NA	NA

Source: Simulations from the 2018-2019 FNS-742 file and 2017-2018 CCD file, with some data coming from the 2016-2017 CCD file except for sample sizes for Puerto Rico and USVI. The numbers in the table will be updated during sampling using the most recent data available.

^aThis number reflects Mathematica’s internal simulation. The number of SFAs actually included in Alaska will be determined by the school selection.

^bEach area is compared to the mainland separately; the outlying areas as a whole will not be compared to the mainland.

NA = not applicable; SFA = school food authority; USVI = U.S. Virgin Islands.

For Alaska, Guam, and Hawaii, a stratified sampling plan will be used to select a representative sample of schools in each outlying area. In contrast to the sample design for the mainland study, the first stage of selection of SFAs was eliminated to improve the precision in the cost estimates for each of the outlying areas and respond to the fact that Guam and Hawaii each have only one public SFA. A random sample of schools stratified by school type (elementary, middle, and high school) will be selected. Schools in Alaska and Guam will be stratified by additional characteristics. In Alaska, most of the school meals are served in a small number of large SFAs; the majority of SFAs in the State are small and rural. Schools will therefore be stratified by SFA size to ensure representation of small SFAs; the school selection will determine which SFAs are included in the study. Schools in Guam will be stratified by whether they are managed by a food service management company (FSMC) so that both FSMC-managed schools and non-FSMC-managed schools are included. For Puerto Rico and USVI, we plan to do a more limited data collection at the SFA level only, including all three SFAs in these two territories. The sampling design is structured to obtain an equal level of statistical precision in each outlying area and, similarly, to produce an equal level of minimum detectible differences for comparing the meal costs in each area to the cost of meals in the 48 contiguous States and DC. There could be a situation in which we get schools to participate in Guam or Hawaii but not the SFA. The same thing could happen on a larger scale for Alaska. And it is always possible that any of the 3 SFAs in Puerto Rico and the USVI refuse to participate.

2.3.6. Sampling schools for FFVP (Group 2b)

Among the 100 SFAs with at least one FFVP-participating school, we will randomly sample one participating FFVP school per SFA in this group, for a total of 100 schools.

2.3.7. Sampling students for FFVP

We will select samples of students in grades 1 through 6 within the FFVP schools (or only up to grade 5 if that is the highest grade in the school). We will sample these students randomly across all classrooms that include these grades using the procedures used in the SNMCS-II component (Group 2a). Although the FFVP-I evaluation incorporated classroom sampling, by eliminating classroom sampling we will reduce the design effect (due to clustering) and maximize the precision of student-level estimates. We will aim to complete data collection with 8 students per school. This will yield 800 students in the schools participating in the FFVP evaluation study component.

2.3.8. Sampling to Account for Nonresponse

At each stage of the selection process, a large enough sample will be selected to increase our likelihood of achieving the target completed sample sizes, considering nonresponse and ineligibility, based on SNMCS-I and accounting for general downward trends in response rates since then.We will select augmented samples of SFAs (larger sample size than we expect to need) and then randomly release a subsample assuming an expected sample yield based on prior experience, keeping the remaining sample available for release in random replicates as needed within stratum. Schools will be randomly ordered within SFAs so that, in response to refusals or ineligibles, recruiters may recruit the next unit on the list until the desired number of cooperating SFAs is obtained. To account for SFAs where the number of schools available is not sufficient to obtain the desired number of two participating schools per SFA in Group 2a and three in Group 3, we will increase the number of schools sampled for other SFAs based on the overall target of 1,061 participating schools in Groups 2a and 3 combined.¹⁸

2.4. Sample Sizes and Statistical Precision

This section presents the design effects and expected precision levels for SFA-, school-, and student-level estimates based on the target completed sample sizes for SNMCS-II (Mainland), FFVP, SFPS-IV, and outlying areas component. Exhibit 2.7 lists the expected design effects for the SFAs for SNMCS-II (Mainland), FFVP, and SFPS-IV

Exhibit 2.7. Average SFA and school-level design effects and completed sample sizes

Study	Sampling unit	Average design effect (deff)	Sample size	Source
SFPS-IV	SFA	1.26	364	Groups 1a and 1b
SNMCS-II (Mainland)	SFA	1.38	522^a	Groups 1a, 1c, 2a, and 3
	School	2.23	796	Group 3
	School	2.83	1,061	Groups 2a and 3
	Student	5.20	3,302	Group 2a
	Parent	5.20	1,800	Group 2a
	Lunch / Breakfast	5.00	4,140 / 2,120	Group 3
FFVP	School	1.20	100	Group 2b
	Student	2.50	800	Group 2b

^a The certainty SFAs are included in Groups 1a, 2a, and 3 but are only counted once to get a unique number of SFAs equal to 522.

The sample size of 100 elementary schools participating in FFVP for Objective 6 (FFVP) was designed to meet the precision of national estimates for population proportions of 0.30 and a difference of ±0.10 with a 95 percent confidence level for schools. The 8 participating students per school will yield precision of ±0.05. These precision estimates assume a design effect of 1.2 at the school level and 2.5 at the student level.

The expected SFA-level precision levels for SNMCS (Objectives 1-4) are presented in Exhibit 2.8a with the 95 percent confidence interval for 522 SFAs for a 30 percent population characteristic for SFA-level estimates for each subgroup, which is typical of outcomes observed in SNMCS-I. Precision calculations assuming a more conservative 50 percent population characteristic (not shown) are a bit larger but still within the desired precision for subgroup estimates of at least 25 percent of the population for nearly all subgroups. As shown, the sample design results in an expected precision level of ±0.048 points for the overall sample of 522 SFAs and achieves precision levels of ±0.10 points (or better) for any subgroups that make up 25 percent or more of the population. Based on results from SNMCS-I, the SNMCS-II study plan estimated (not shown) the average design effect of 1.38 from the PPS selection and the expected nonresponse adjustments for SFA-level estimates.

The expected SFA-level precision levels for SFPS (Objective 5) are presented in Exhibit 2.8b. The sample size of 364 SFAs was designed to meet the precision of national estimates for population proportions of 0.50 and a difference of ±0.05 with a 90 percent confidence level.¹⁹ This sample reflects the losses from using a complex design (e.g., unequal weighting and stratification of SFAs in Groups 1a and 1b). This sample size meets the precision target of a subpopulation of at least 25 percent of the total population with a proportion of 0.5 and a difference of ±0.10 with a 90 percent confidence level.

Exhibit 2.8a. Expected precision levels for SFA-level estimates for SNMCS (Mainland) (Objectives 1-4)

Subgroups	Target completed sample sizes	95% CI Half interval (percentage points)
Number of Schools
1–2^a	168	8.8
3–4^a	148	9.4
5 or more^a	174	8.7
Number of Students
1–350	121	10.0
351–1,200^a	142	9.3
More than 1,200^a	231	7.3
Urbanicity
Urban^a	205	7.7
Rural^a	329	6.1
Percentage of Students in Poverty^b
0 to 17 percent^a	170	8.5
More than 17 percent to 35 percent^a	300	6.4
More than 35 percent	57	14.6
FNS Region
Mid-Atlantic	50	15.6
Midwest^a	112	10.4
Mountain Plains	75	12.8
Northeast	59	14.3
Southeast	69	13.3
Southwest	93	11.4
West	75	12.8
Charter SFA
Yes	48	15.9
Percentage of Students Approved for F/RP Meals
0 to 45 percent	325	6.1
46 to 63 percent	123	10.0
64 percent or more	80	11.9
HSMFA/CEP Status^c
SFAs in HSMFA States	112	10.0
Non-HSMFA SFAs with all CEP schools	161	8.3
All other SFAs^a	261	6.5
Total	534^d	4.6

Source: Simulations from the first iteration of SNMCS-II, using the 2018-2019 FNS-742 file and 2017-2018 CCD file, with some data coming from the 2016-2017 CCD file.

Note: Confidence intervals for subgroups are based on a 30 percent outcome while confidence intervals for the total row is based on a 50 percent outcome.

^aSubgroup represents 25 percent or more of the population.

^bThe percentage of students living in poverty variable has a considerable amount of missing data; the totals here reflect the non-missing cases.

^cThe simulations in the SNMCS-II study plan did not include SFAs with other universal free meal provisions in the CEP SFA group.

^dTotal equals 534 because it includes the certainty SFAs in Groups 1a, 2a, and 3.

CCD = Common Core of Data; CEP = Community Eligibility Provision; CI = confidence interval; FNS = Food and Nutrition Service; F/RP = free or reduced-price; HSMFA = Healthy School Meals for All; LEA = local education agency; SFA = school food authority.

Exhibit 2.8b. Expected precision levels for SFA-level estimates for SFPS (Objective 5)

Subgroups	Target completed sample sizes	90% CI Half interval (percentage points)
Number of Schools
1–2^a	125	7.6
3–4^a	110	8.1
5 or more^a	129	7.4
Number of Students
1–350	89	9.0
351–1,200^a	105	8.3
More than 1,200^a	170	6.5
Urbanicity
Urban^a	140	7.2
Rural^a	224	5.6
Percentage of Students in Poverty^b
0 to 17 percent^a	117	7.8
More than 17 percent to 35 percent^a	207	5.9
More than 35 percent	39	13.5
FNS Region
Mid-Atlantic	34	14.5
Midwest^a	76	9.7
Mountain Plains	51	11.8
Northeast	40	13.3
Southeast	47	12.3
Southwest	64	10.6
West	51	11.8
Percentage of Students Approved for F/RP Meals
0 to 45 percent	221	5.7
46 to 63 percent	84	9.2
64 percent or more	59	11.1
HSMFA/CEP Status^c
SFAs in HSMFA States	76	9.7
Non-HSMFA SFAs with all CEP schools	110	8.1
All other SFAs^a	178	6.3
Total	364	4.4

Source: Simulations from the first iteration of SNMCS-II, using the 2018-2019 FNS-742 file and 2017-2018 CCD file, with some data coming from the 2016-2017 CCD file.

Notes: Confidence intervals for subgroups are based on a 30 percent outcome while confidence interval for the total is based on a 50 percent outcome. Charter SFAs are excluded from Objective 5.

^aSubgroup represents 25 percent or more of the population.

The expected precision levels for school-level estimates for the Groups 2a and 3 in SNMCS are presented in Exhibit 2.9. For the sample of 1,061 schools (Groups 2a and 3 combined) that will complete the Menu Survey, SNM Survey, and Principal Survey, the expected precision level for a 30 percent population characteristic is ±4.6 percentage points for the overall sample and ±10 percentage points (or better) for any subgroups that make up 25 percent or more of the population. We also show in Exhibit 2.9 an expected precision level of ± 4.8 percentage points for the overall sample of 796 Group 3 schools that will be included in the study of meal costs and school food service revenues and precision levels of ±10 percentage points (or better) for any subgroups that make up 25 percent or more of the population.

Exhibit 2.9. Expected precision levels for school-level estimates (SNMCS-II Mainland)

		Groups 2a and 3 combined		Group 3 only
Subgroups		Target completed sample sizes	CI half interval (percentage points)	Target completed sample sizes		CI half interval (percentage points)
School Type
Elementary^a		487	6.8	358		8.0
Middle		241	9.7	188		11.0
High		332	8.3	249		9.6
Urbanicity
Urban^a		409	7.5	317		8.5
Rural^a		653	5.9	479		6.9
Racial/Ethnic Distribution of Students (mean %)
Non-Hispanic Black		111	14.3	83		16.6
Non-Hispanic White		663	5.9	495		6.8
Hispanic		189	11.0	136		13.0
Other		98	15.2	82		16.7
Students Approved for F/RP Meals
0 to 45 percent^a	384		7.7	283	9.0
46 to 63 percent	294		8.8	232	9.9
64 percent or more^a	383		7.7	281	9.0
FNS Region
Mid-Atlantic	87		16.2	60	19.5
Midwest	226		10.0	179	11.3
Mountain Plains	148		12.4	108	14.5
Northeast	112		14.3	85	16.4
Southeast	167		11.7	128	13.4
Southwest	189		11.0	142	12.7
West	132		13.1	94	15.6
HSMFA/CEP Status of School^b
Schools in HSMFA States	227		10.0	170	10.3
Non-HSMFA CEP schools^a	319		8.5	239	8.7
All other schools^a	516		6.7	387	6.8
Total	1,061		5.1	796	5.2

Source: Simulations from the first iteration of SNMCS-II, using the 2018-2019 FNS-742 file and 2017‑2018 CCD file, with some data coming from the 2016-2017 CCD file. The numbers in the table will be updated during sampling using the most recent data available.

Notes: Confidence intervals for subgroups are based on a 30 percent outcome while confidence intervals for the total is based on a 50 percent outcome. The level of precision for school estimates for the combined Group 2a and Group 3 samples is only slightly better than that for the Group 3 sample alone. This is because combining the two samples introduces an additional design effect at a final value of 2.48 relative to the design effect of 1.95 for the Group 3 sample alone (which incorporates the SFA design effect of 1.38). This phenomenon is a necessary consequence of meeting the sometimes-competing precision requirements for each survey objective and the associated study components.

^aSubgroup represents 25 percent or more of the population.

CI = confidence interval; CCD = Common Core of Data; CEP = Community Eligibility Provision; FNS = Food and Nutrition Service; F/RP = free or reduced-price; HSMFA = Healthy School Meals for All; SFA = school food authority.

Using the SNMCS-I meal cost data, the original SNMCS-II study plan estimated that, for a school-based cost estimate of the national average meal cost of $2.36 (averaged over schools and accounting for the selection of SFAs) with a standard deviation of $0.98, the precision would be ± $0.105. The average design effect is estimated to be 2.23 in Group 3 and 2.83 in Groups 2a and 3 combined with the weighting adjustments described later.

Exhibit 2.10 presents the expected precision levels for the student- and parent-level estimates in Group 2a and the tray-level estimates in Group 3. As shown, the sample design results in an expected precision level of ±3.9 percentage points for the overall sample of 3,302 complete student interviews in Group 2a (for a 50 percent population characteristic) and expected precision levels of ±10 percentage points (or better) for any subgroups that make up 25 percent or more of the population (for a 30 percent population characteristic). For the parent interviews, the sample design results in an expected precision level of ±5.3 percentage points for the overall sample of 1,800 complete interviews in Group 2a (for a 50 percent population characteristic) and expected precision levels of ±10 percentage points (or better) for any subgroups that make up 25 percent or more of the population (for a 30 percent population characteristic).

For the plate waste observations (Appendix F09) in Group 3, the expected precision levels are ±3.4 and ±4.8 percentage points, respectively, for the overall samples of 4,140 lunch trays and 2,120 breakfast trays. Expected precision levels are ±10 percentage points (or better) for any subgroup representing 25 percent or more of the population.

Exhibit 2.10. Expected precision levels for student-, parent-, and tray-level estimates for Groups 2a and 3 (SNMCS Mainland)

Subgroups	Group 2a students		Group 2a parents		Group 3 plate waste observations
	Group 2a students		Group 2a parents		Lunch		Breakfast
	Target completed sample sizes	CI	Target completed sample sizes	CI	Target completed sample sizes	CI	Target completed sample sizes	CI
	Target completed sample sizes	half interval (percentage points)	Target completed sample sizes	half interval (percentage points)	Target completed sample sizes	half Interval (percentage points)	Target completed sample sizes	half interval (percentage points)
School type
Elementary^a	1,549	5.2	844	7.0	1,456	5.8	745	8.4
Middle	754	7.5	411	10.1	1,365	6.0	700	8.6
High^a	1,000	6.5	545	8.8	1,318	6.1	675	8.8
Urbanicity
Urbana	2,154	4.4	1,174	6.0	2,451	4.4	1,256	6.7
Rural^a	1,148	6.0	626	8.2	1,689	5.4	864	7.3
Race/ethnicity
Non-Hispanic Black	391	10.4	213	14.0	544	9.0	279	11.0
Non-Hispanic White^a	2,166	4.4	1,181	6.0	2,808	4.0	1,440	4.9
Hispanic	557	8.7	303	11.8	634	8.3	325	10.2
Approved for free/reduced-price meals
Yesa	1,966	4.6	1072	6.3	2,300	4.6	943	6.7
Noa	1,336	5.6	728	7.6	1,840	5.1	1,177	7.3
FNS region
Mid-Atlantic	337	11.2	184	15.1	332	12.1	170	17.0
Midwest	585	8.5	319	11.5	891	7.3	456	10.4
Mountain  Plains	491	9.2	268	12.5	257	13.7	132	19.3
Northeast	343	11.1	187	15.0	421	10.7	216	15.1
Southeast	501	9.2	273	12.4	990	7.0	506	9.8
Southwest	587	8.5	320	11.4	692	8.4	355	11.7
West	458	9.6	250	13.0	557	9.3	285	13.1
HSMFA/CEP status of schoolb
Schools in HSMFA States	706	7.7	385	10.4	884	6.9	453	9.6
Non-HSMFA CEP schools^a	991	6.5	540	8.8	1,242	5.8	636	8.1
All other schools^a	1,605	5.1	875	6.9	2,013	4.6	1,031	6.4
Total	3,302	3.9	1,800	5.3	4,140	3.4	2,120	4.8

Note: Confidence intervals are based on a 30 percent outcome.

^aSubgroup represents 25 percent or more of the population.

^bCEP schools are drawn from both the all-CEP SFA stratum and the not-all CEP SFA stratum. The simulations from the SNMCS-II study plan did not include SFAs with other universal free meal provisions in the CEP SFA group.

CEP = Community Eligibility Provision; CCD = Common Core of Data; CI = confidence interval; FNS = Food and Nutrition Service; F/RP = free or reduced-price; HSMFA = Healthy School Meals for All.

The minimum detectable differences (MDDs) for school- and student- level comparisons between Group 2a and 2b for Objective 6 are presented in Exhibit 2.11 for a population outcome of 0.30. Because the sample design does not oversample non-FFVP schools and students in Group 2a, the MDDs presented are based on an estimate that about half the elementary schools in the Group 2a sample will not be participating in FFVP. Once data are collected, the study contractor will confirm the MDD that could be detected when comparing Group 2b FFVP schools and students to Group 2a non-FFVP schools and students. If the MDD is acceptable to FNS, the study contractor will implement the analyses as described in the NSFS study plan. Otherwise, the contractor will produce estimates among the Group 2b FFVP schools and students only and describe these findings in context of all Group 2a schools and students (both FFVP and non-FFVP).

Exhibit 2.11. Estimated minimum detectable differences for school- and student- level comparisons for FFVP for a population outcome of 0.30.

	FFVP (Group 2b) (percentage points)	Non-FFVP (Group 2a) (percentage points)
Elementary Schools
Sample Size	100	65
Design Effect	1.2	2.8
MDD (Group 2b to 2a)	0.302
Elementary Students
Sample Size	800	511
Design Effect	2.5	5.2
MDD (Group 2b to 2a)	0.148

Note: This assumes 80 percent power and a type I error rate of 0.05.

Exhibit 2.12 provides precision estimates for the outlying area component. In this table, precision estimates are shown in dollars instead of percentages.

Exhibit 2.12 Respondent universe, target sample sizes, and expected precision for each outlying area

State/ territory	Population		Target completed sample	Expected Precision (in dollars)
State/ territory	SFAs	Schools	SFAs	Schools	Confidence half interval	MDD to mainland (50% power)	MDD to mainland (80% power)
Alaska	43	431	29^a	51	0.31	0.45	0.65
Guam	1	41	1	24	0.32	0.47	0.67
Hawaii	1	255	1	63	0.33	0.47	0.67
Puerto Rico	1	1,108	1	NA	NA	NA	NA
USVI	2	27	2	NA	NA	NA	NA

^aThis number reflects Mathematica’s internal simulation. The number of SFAs actually included in Alaska will be determined by the school selection.

^bEach area is compared to the mainland separately; the outlying areas as a whole will not be compared to the mainland.

NA = not applicable; SFA = school food authority; USVI = U.S. Virgin Islands.

2.5. Weighting

Analysis weights will be constructed for each type of data collected at the SFA, school, student, parent, and tray levels. The weights will account for the probabilities of selection and differential response rates across the subgroups. In the last step of weighting, the study team will post-stratify each component set of weights so that they total to benchmarks obtained from the most recent CCD and FNS-742 data by SFA-, school-, and student-level characteristics, considering the ineligibility of the units identified during the study. At a minimum, the study team will prepare a final set of nonresponse and post-stratified weights for the following study components:

SFA weights for the SNMCS-II (Mainland) SFAs sampled in Groups 1a, 1c, 2a, and 3.

SFA weights for the SFPS-IV SFAs sampled in Groups 1a and 1b.
School weights for the SNMCS-II (Mainland) schools sampled in Groups 2a and 3.
Cost study weights for the SNMCS-II SFAs and schools sampled in Group 3 and outlying areas.
Student and parent weights for the SNMCS-II for Group 2a schools.
Plate waste observation weights for the SNMCS-II subset of sampled schools in Group 3 (lunch and breakfast).
SFA-, school-, and student-level weights for the FFVP evaluation study component (Group 2b).
SFA and school weights for the Outlying Areas study component.

For each of the above categories, these weights may include multiple weights across study instruments or combinations of instruments. The study team will create the base weights for each component and adjust for nonresponse within important sample subgroups using a procedure that models the probability of responding to the component based on the available data collected in the sampling frame. In this procedure, the sampling weights of the responding cases are adjusted by the inverse of the weighted response rate within weighting classes.

As a final process in preparing weights, the study team will adjust weights using calibration or post-stratification methods (Deville & Särndal, 1992) to ensure weighted totals or proportions match those for which we have comparable data from the sampling frame or other published sources. Details around the specific weights by instrument are provided below.

2.5.1. SFA Weights for the SNMCS-II SFA Director Survey

First, an SFA weight for the SNMCS-II SFA Director Survey (Appendix F03.01) within each group will be created. Then, a composite weight will be used to combine the SNMCS-II SFA Director Survey observations across Groups 1a, 1c, 2a, and 3. The first weighting factor will be each SFA’s sampling weight (the inverse of its probability of selection within its group). The next factor will be a nonresponse adjustment (or adjustments) at the SFA level. To combine SNMCS-II SFA Director Survey data across the four groups, a composite factor will be used so that each Group contributes proportionately to the overall estimate.²⁰

2.5.2. SFA Weights for the SFPS-IV SFA Director Survey

First, an SFA base weight will be constructed for the SFPS-IV SFA Director Survey (Appendix F03.01) for the SFAs within Groups 1a and 1b. When putting data together from the four quarters, we will determine whether an additional weighting factor will be necessary. One weighting component is an adjustment for being selected for Group 1. To combine data between Group 1a and 1b, a composite factor will be used. The next factor will be a nonresponse adjustment (or adjustments) at the SFA level.

2.5.3. School Weights

School-level weights will be constructed for each school-level instrument (or combinations of instruments) based on the school’s probability of selection within the SFA for all but the meal cost estimates at the SFA level, discussed in the next section. (Because the Outlying Areas schools are selected at the first stage, we calculate that sampling rate directly, not through their SFA.) One weighting component is an adjustment for being selected for the Group 2 or Group 3 subframe. The basic school weight will then be adjusted for nonresponse for a given survey component and multiplied by the SFA weight to compute school-level estimates. The final version of each school weight will be ratio adjusted within groups (Group 2a, Group 2b, Group 3, and Groups 2a and 3 combined) to reflect the best estimate of the number of eligible schools and then trimmed. The school weights will differ for each group. In addition, the school weights will be constructed so that subgroup analyses can be made within HSMFA or non-HSMFA CEP schools and between HSMFA, non-HSMFA CEP, and non-CEP/HSMFA schools. Finally, a separate set of school-level weights will be constructed that reflect school enrollment. These weights will allow FNS to examine, for example, the percentage of students that attend schools with specific characteristics.

2.5.4. Cost Study Weights

Following the approach used in SNMCS-I, three sets of weights will be created for use in the estimation of meal costs. These weights—for the Group 3 sample—include (1) a set of school-level weights to be used to aggregate school-level cost component estimates up to the SFA level, (2) SFA-level weights needed to estimate national average costs and to support exploratory and confirmatory analyses of the relationships between SFA characteristics and SFA-level average meal costs, and (3) school-level weights needed to estimate national average costs at the school level and support exploratory analyses of the relationships between school characteristics and school-level average meal costs. The approach to developing each of these weights is described below. One weighting component is an adjustment for being selected for the Group 3 subframe.

a. Within-SFA Cost Study School Weights and Meal Cost Estimates

The first set of cost analyses requires a weight for each responding school within a Group 3 SFA. These weights are used to aggregate school-level estimates of meal costs up to the SFA level. However, participation in the school meal programs and, therefore, meal cost varies by school type (elementary, middle, and high schools). To account for this variation, a weight for each school type will be calculated following the approach developed in SNMCS-I. This approach estimates a proxy for meal costs by imputing the amount of Federal reimbursement each school would have received if all meals (lunches and breakfasts separately) and afterschool snacks served during the reference period were reimbursed at the free meal rate.²¹

Next, using CCD estimates of enrollment in each Group 3 SFA by school type, the share of each Group 3 SFA’s per capita reimbursement will be estimated for each school type. Then, for each school type in each SFA, this estimated share will be divided by the number of schools of that type with adequate cost data and multiply the result by a ratio that will adjust for the number of types of schools with adequate cost data in that SFA. If one qualifying school of each school type is present in an SFA, their weights will simply be the shares of per capita reimbursements obtained in the second step. These school weights are multiplied by the school meal costs and then rolled up to produce an SFA-level estimate.

b. Aggregate Cost Study SFA Weights

The second set of cost analyses requires a weight for Group 3 SFAs that provided selected data and for which their sampled schools reported certain additional data. Specifically, SFAs will be included in these analyses and assigned weights if (1) at least two schools within the SFA had completed the SNM Cost Interview and the Menu Survey (Appendices F05.10 and F02.01),²² (2) the SFA completed the On-Site and Follow-Up Cost Interviews (Appendices F05.02 and F05.08), (3) there was a Principal Cost Interview (Appendix F05.12) from at least one school, and (4) the SFA director completed the SFA Follow-Up Web Survey (Appendix F05.06). To construct the weights for this sample of SFAs, the study will begin with the intermediate Group 3 SFA weight calculated for the SFA Director Survey (Appendix F03.01) before compositing with the Group 1 and 2 weights. The study will then model the propensity to satisfy the aggregate SFA cost study requirements using SFA frame variables as predictors. Estimated propensities will then be used to form four propensity strata and calculate the average response propensity within each propensity stratum. The intermediate weight for each qualifying SFA will be divided by the response propensity for its stratum to create the adjusted weight. If necessary, these SFA weights will be ratio adjusted and trimmed, as described earlier.

c. National Cost Study School Weights

The third set of cost analyses requires a weight for the Group 3 schools that responded to the SNM and Menu Surveys and Cost Interviews (Appendices F04.01, F02.01, F05.02, F05.10, and F05.12) and for which cost data were supplied by their SFAs. To construct these weights, the study will start with the Group 3 school weight prior to compositing with Group 2a. This intermediate weight will be assigned to schools that completed at least one SNM, Menu, or Principal Survey (Appendices F04.01, F02.01, and F03.07) but not all three.²³ Using propensity modeling, the intermediate weights will be adjusted so that they are nonzero only for schools that meet the eligibility requirements for their inclusion in the national cost study and, overall, the weights of such schools sum to national school totals (for non-charter schools). Specifically, the study will estimate the propensity for schools with intermediate Group 3 weights to qualify for the cost study using school-level frame variables. Strata will then be formed from the estimated propensities and intermediate weights for qualifying schools by their respective stratum response propensity. If necessary, the weights will be ratio adjusted and trimmed.

2.5.5. Student/Parent Weights for Group 2a

Student-level weights for the Student and Parent Interviews (collected from students and parents in Group 2a schools; Appendices F08.01 and F08.04) will be constructed. One weighting component is an adjustment for being selected for the Group 2 subframe. The basic weight will be the combination of the SFA and school-level weight, each adjusted for nonresponse. The study will then adjust for probabilities of selection of the students within schools for nonresponse. We will then adjust for parental consent or assent, followed by adjustments for nonresponse to an instrument or combinations of instruments among consented students. To avoid response bias, nonresponse adjustments at the individual (parent, student) level will be based on a nonresponse analysis using covariates available for both responding and nonrespondent students and weighting methodology that tries to balance weight stability, bias, and variance issues.

2.5.6. Plate Waste Weights

Traditional selection weights will be constructed for the plate waste observations (collected in a subset of Group 3 SFAs and schools; Appendix F09). In constructing these weights, the study will start with the Group 3 SFA weight (which also accounts for being selected for the Group 3 subframe) and account for the probability of selecting an SFA into the sample for the plate waste subsample among SFAs that meet the eligibility criteria. The study will then account for any SFAs that choose not to participate in the plate waste data collection and for the probability of selection of each school, among schools that meet the eligibility criteria within sampled and participating plate waste-selected SFAs, along with any schools that choose not to participate in study activity. The study will then adjust for probabilities of selection of the student trays for observation within schools, making use of data on the total number of reimbursable breakfasts and lunches served on the day of observation.

2.5.7. FFVP evaluation Weights (SFA, school, and student)

We will construct weights that account for the selection of SFAs with one or more schools participating in the FFVP and adjust those sampling weights for SFA nonparticipation. If an SFA we thought had one or more FFVP schools turns out to have none, the SFA will not be recruited and will be coded as ineligible for this study component. We will later calibrate the weights to the updated (SY 2024-2025) list of SFAs with at least one school participating in the program.

Although these FFVP evaluation SFAs also have a chance of selection in the SNMCS-II and SFPS-IV study components, we do not have to account for this multiplicity in the weights as there are no plans to combine estimates (for example, FFVP-participating schools in group 2a) due to the vastly different sampling rates, which would cause combined estimates to be less efficient than using the FFVP evaluation sample on its own.

Within each SFA, we will account for the probability of selection of the FFVP-participating school (if more than one exists in the SFA) and for any refusal to participate in this study component, followed by an adjustment for nonresponse to the school-level (for the FFVP SNM Survey, FFVP Menu Survey, and Observation Guide; Appendices F04.02, F02.04, and F07).

Within each school, we will account for the probability of selection of the students in the study component’s eligible grades, followed by adjustments for lack of parental consent or assent, and for nonresponse to child-level instruments (Student Interview, dietary recall (for in-school intake), reimbursable meal sales data; Appendices F08.01, F08.02, and F06).

2.5.8. Outlying Area component weights

The sampling weights for outlying areas Alaska, Guam, and Hawaii will be constructed within strata at the school level, which is their primary sampling unit. The SFA sampling weights for Guam, Hawaii, Puerto Rico, and USVI will be set to 1, as they will be included in the study with certainty. The sampling weight for Alaska SFAs will be constructed to reflect the fact that the first stage of sampling is the school and not the SFA. The associated SFA sampling weights will reflect their probability of being selected into the sample via their sampled school(s).

1 While it is expected that most States will be represented in the SFA, school, student, and meal samples, States are not sampled directly. Therefore, this study cannot be used to produce State-level estimates.

2 The charter-only SFAs are included in the universe used in addressing Objective 1 (Group 1c), but not included in the universe used in addressing Objectives 2-5.

3 This criterion was selected based on an internal analysis of key characteristics in SNMCS-I, which found many to be around 30 percent. LOA is SFA- and reported costs-only. If USVI's two public SFAs are purchasing food for the private schools from the foodservice account, the costs will be on the financial statement. A bigger threat to generating territory-specific meal cost estimates is the limited information on one of the two public SFA's financial statements, and FNS agreed that we would attempt data collection with it.

4 The number of certainty SFAs will be determined during SFA sampling.

5 The data presented in this sampling plan are based on the first iteration of SNMCS-II, using the 2018-2019 FNS-742 file and 2017-2018 CCD file, with some data coming from the 2016-2017 CCD file. In drawing the actual sample, the study team will have access to more current files. For purposes of describing the sample design, the FNS-742 file was linked to the CCD file by district name, state, phone, and ZIP code associated with the SFA, to append the LEA IDs associated with each SFA and link to the corresponding schools. Some SFAs or schools were missing information for one or more of the characteristics used to define subgroups (percentage of students in poverty, percentage of students approved for free or reduced-price meals, enrollment, and urbanicity). In these instances, missing values were imputed.

6 These very large SFAs would likely be selected with certainty if included in any sampling group, and therefore must be represented in each group.

7 Separately, when constructing the sampling frame for FFVP in Group 2b, these same SFAs would be included if they have an FFVP elementary school),

8 Oversampling of HSMFA and non-HSMFA CEP-only SFAs may not be necessary due to the increasing prevalence of this option.

9 Oversampling of one group through the use of a higher MOS value may in turn reduce the number of units sampled on another group. We have worked to balance these conflicts to achieve samples that will support the aforementioned subgroup analyses.

10 CCD and FNS-742 database data will be used to divide the number of schools in the SFAs into three categories (small, medium, large) corresponding to the 30th and 60th percentiles to ensure that enough SFAs with a large number of schools are included in the sample. Similarly, for student SFA size, three categories (small, medium, large) will be used. All possible combinations of these categories create nine strata.

11 Public SFAs will also be stratified by whether or not the SFA is in an HSMFA State, or all the schools in the SFA are CEP schools or neither; when HSMFA and CEP status is crossed with the size and charter strata, we obtain up to 27 strata. In contrast, charter-only SFA will be stratified into three or four strata defined by HSMFA and CEP status.

12 As with Group 1, some collapsing of strata will likely be required for SFAs with a large number of schools and small number of students, or vice versa.

13 The prior SNMCS-II study plan found a modest level of variability in the number of schools within each of the strata to both warrant including this factor in the MOS and for smoothing out these differences with the square root function to create a compromise between the use of this sample for both SFA-level and school-level estimates. Specifically, for SFA-level analyses the preferred sample is one for which each SFA is selected using the same probability of selection. In contrast, if the SFA is selected as a first step in selecting and preparing school-level estimates, SFAs with a larger number of schools should be selected at a higher rate than SFAs with a small number of schools. The use of an SFA MOS based on the square root of the number of schools strikes a compromise between the two approaches to provide an SFA sample that will serve both analytical objectives.

14 The six certainty SFAs are included in Groups 1a, 2a, and 3 but are only counted once to get a unique number of SFAs equal to 522.

15 In some SFAs, the study will need to obtain four or five school participants to reach the overall sample goal given some SFAs may not have a school in each of the elementary, middle, and high school strata.

16 The sampled schools in the certainty SFAs differ between Groups 2a and 3. Therefore, the total number of sampled schools (1,061) includes no double-counting of schools in these SFAs.

17 Schools in Group 3 will provide data to estimate meal costs. Because meal costs may vary in schools that serve students of different ages, the design calls for stratification of schools by school type (elementary, middle, and high). Because schools in Group 2a will not contribute data for the estimation of meal costs, explicit stratification by school type is not necessary.

18 In Group 3, schools will be selected in pairs within the elementary, middle, and high school strata as in SNMCS-I to ensure a completed component is obtained from a school from each school level to the extent possible.

19 The effective sample of 289 SFAs was computed using the Binomial exact test of hypothesis (2 sides) for p=0.5 with a difference of ±0.05 for α=0.1. This sample is inflated to 364 using a design effect =1.26. This design effect accounts for losses from the use of a complex design to select the SFAs in Groups 1a and 1b and for the weighting adjustments.

20 The composite is adjusted to the proportion of SFAs responding from each group. It also can reflect the differences in design effects among the groups. Level and support exploratory analyses of the relationships between school characteristics and school-level average meal costs. The approach to developing each of these weights is described below.

21 This approach implicitly assumes that the free meal reimbursement rate is a reasonable proxy for the average cost per-meal.

22 For SFAs with only one or two schools, the SFA will be considered to have sufficient school-level meal production cost data if one school completed the Menu Survey and the SNM Cost Interview.

23 Although the Principal Cost Interview is also important for measuring full school-level costs, the study team will impute these data when they are missing. That is, a response to this interview is not a requirement for receiving a weight for the national cost study.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Title	8675.02.03: Report Template
Author	Mathematica
File Modified	0000-00-00
File Created	2024-07-31