O_Sampling Plan_4.20.21

O. Sampling Plan

This page has been left blank for double-sided copying.

In this appendix, we present the plan for selecting the SNACS-II sample and constructing weights for the analysis. The appendix also includes details of the sampling overview presented in the main text. The overall objective of the sampling plan is to provide nationally representative samples of programs of Child and Adult Care Food Program (CACFP); the children, teens, and infants served by CACFP programs; and CACFP meals and snacks for program year (PY) 2022–2023. The SNACS-II sample design builds on that for SNACS-I to ensure comparability of estimates across the two studies and provide required levels of statistical precision, while minimizing data collection costs and respondent burden. In addition, the SNACS-II design addresses specific challenges faced in SNACS-I.

We will use a multistage stratified cluster sampling design. Because sampling frames for CACFP programs can be provided only at the State level, the first stage of selection—also known as the primary sampling unit (PSU)—will be the State (including the District of Columbia, but excluding Alaska and Hawaii). We will select a nationally representative probability sample of 25 States. In the second stage, we will select a sample of Core-Based Statistical Areas (CBSAs)¹ and clusters of non-CBSA counties from the selected States as secondary sampling units (SSUs). In the third stage, we will sample CACFP programs² within sampled SSUs. In the fourth and final stage, we will sample children, teens, and infants within the sampled programs.

We will select a sample of CACFP programs to complete the Provider Survey and Menu Survey under Objectives 1 and 2. (We will also ask programs that serve infants to complete the Infant Menu Survey under Objective 5.) We will select a subsample of these programs to participate in onsite data collection activities to address specific research questions within each of the study objectives. We will select the samples of children, teens, and infants within this subsample of programs with onsite data collection. We describe the plan for each stage of selection in the sections below.

1. Selection of States

In the first stage of selection, we will select a national probability sample of 25 States. In comparison with the 20 States sampled in SNACS-I, we expect this larger number of PSUs to improve the precision of estimates by reducing the design effect due to intra-State correlation. The sampling frame will comprise the 48 contiguous States and the District of Columbia. In order to minimize the design effect, we will design this probability sample of States to support an overall self-weighting sample of programs within domains of interest. We will oversample States with a higher proportion of children served in rural counties/areas to ensure adequate sample sizes of rural programs. In addition, we will sample at least one State in each of the seven FNS regions.

To achieve these objectives, the first step is to identify the set of States that should be included with certainty, and then use a stratified systematic probability proportional to size (PPS) design to select the remaining noncertainty States, with the strata defined primarily by FNS regional representation. Ideally, the measure of size (MOS) for the PPS selection would be aggregated CACFP average daily attendance (ADA) for the State. However, because these data will not be available for all States prior to sampling, we will use the total number of children ages 5 to 18 living in low-income households³ as a proxy for ADA.

Certainty States are those with the largest MOS, described more fully below. We will stratify non-certainty States by the seven FNS regions: Mid-Atlantic, Midwest, Mountain Plains, Northeast, Southeast, Southwest, and Western. In some of the States in the Mid-Atlantic and Northeast regions, nearly all of the children in low-income households live in metropolitan areas. To address this issue, we will use the approach used in SNACS-I. Specifically, we will combine the Mid-Atlantic and Northeast regions and then subdivide the combined region into two strata: (1) a highly urban stratum consisting of the States where nearly all of the children in low-income households live in metropolitan areas and (2) a highly rural stratum consisting of the States where a larger proportion of children in low-income households live in rural areas. This will permit a more efficient allocation of rural versus urban SSUs, relative to strata defined strictly by FNS region.

We will determine the number of sampled States allocated to each stratum after we have identified the certainty States. Within each stratum, we will compute the MOS of States using a weighted combination of the separate estimates of proportion of children in low-income households by metropolitan status. This relative weighting increases the probabilities of selection for States with larger shares of rural CACFP programs, allowing for an oversampling of these programs. We define the MOS for State k in the first stage of selection thus:

0.75 * ( ) + 0.25 *( ).⁴

We will use this MOS to determine certainty States within each stratum.⁵

Oversampling of States with relatively higher rural percentages will result in some minor weighting design effects, but this will ensure that at least 25 percent of the sample of programs is from rural areas.

We will contact the State Child Nutrition Directors from the 25 selected States to provide a list of CACFP programs in their State.⁶ Each State’s list should contain the contact information of eligible CACFP programs, contact information for their providers, contact information for their sponsors (if applicable), and other auxiliary information that we will use for sampling at later stages, including the program’s ADA, program type, and contact and location information. We will select programs within the SSUs selected within each State.

2. Selection of Secondary Sampling Units

In the second stage, we will select a stratified PPS systematic sample of geographically defined SSUs within the sampled States. Sampling SSUs increases the cost efficiency of in-person data collection by restricting the geographic range that interviewers must travel, although it does introduce a design effect due to clustering. We will use the listings of CACFP programs obtained from the sampled States to define geographically contiguous SSUs. The first set of SSUs to define will be CBSAs, (i.e.metropolitan and micropolitan areas).⁷ For non-CBSA counties, we will require a minimum of 12 CACFP programs for each SSU; we will combine contiguous counties as needed until we reach this minimum. In SNACS-I, SSUs were defined to have approximately 30 listed providers.⁸ We may adjust the minimum number depending on the distribution of programs across the SSUs after we collect the lists of programs from the sampled States.

In sampling the SSUs, we will adopt the approach used in SNACS-I; that is, we will use non-CBSA counties to represent rural areas and use CBSA counties to represent urban areas. As in SNACS-I, we will sample a total of 80 SSUs—60 urban (CBSA) SSUs and 20 rural (non-CBSA) SSUs. This will result in a limited degree of oversampling for the rural SSUs to help ensure that 25 percent of sampled programs are located in rural SSUs.

The MOS used in sampling SSUs will be the aggregated ADA within the SSU. Using an MOS based directly on program ADA rather than census information about children in poverty (as was done in SNACS-I) is expected to yield more precise estimates. We will first determine whether there are any certainty SSUs with very large MOS. Next, within each of the 25 sampled States, we will stratify the remaining non-certainty SSUs by CBSA status (metropolitan; micropolitan; non-CBSA) and we will sort them geographically within strata for systematic PPS sampling. In most of the non-certainty States, we will select two CBSA SSUs and one non-CBSA SSU. However, in the States where there are very few low-income children living in non-CBSA areas, we will not select any non-CBSA areas. In certain States, we may also consider increasing the sample size of non-CBSA SSUs to ensure that we will select one non-CBSA SSU in each sampled State.

3. Selection of programs

In sampling programs of providers, we must address two important challenges. First, multiple providers may be affiliated with a single sponsor. Sponsors often play an important role in the operations of their constituent providers. Consequently, including multiple providers from the same sponsor may incur nontrivial intra-class correlation. Moreover, this approach would increase sponsors’ response burden, which may result in a low rate of cooperation among sampled providers. Second, individual providers may operate more than one program and even more than one type of program. For example, a child-care center may also operate an at-risk after-school (AR) center or an outside-school-hours care center (OSHCC). Asking a single provider to participate in data collection activities for more than one program or program type would impose a substantial response burden and could lead to lower rates of cooperation.

To avoid these challenges, our approach to sampling programs of providers will use systematic sampling, where the sample is sorted by sponsor, provider, and program type (by using their individual ADAs, described below). This sorting, which will group programs within the same provider together and providers within the same sponsor together, will minimize the chances of sampling more than one provider per sponsor and, among providers who operate more than one program (multiple programs of the same type or different types of programs), the chances of sampling a provider more than once. To avoid confusion, the following description of our approach to sampling providers focuses on programs because this—programs associated with providers—is the actual sampling unit. Each sampled program will be associated with a specific provider.

Within each of the 80 selected SSUs, we will sample CACFP programs through an equal selection probability design, with some oversampling of particular types of programs. We will reduce the State program lists to those within the 80 sampled SSUs and use this to construct the program sampling frame. The sample of programs selected for recruitment will be large enough to complete data collection with 1,340 programs across the five primary types of programs identified in the Performance Work Statement (PWS):

• 310 child care centers,

• 310 Head Start centers,

• 320 family day care homes (FDCHs),⁹

• 200 AR centers, and

• 200 OSHCCs.

In each sampled SSU, we will stratify the list of CACFP programs into seven mutually exclusive groups:

1. Sponsored child care centers,

2. Independent child care centers,

3. Head Start centers,

4. FDCHs,

5. AR centers,

6. OSHCCs, and

7. Programs associated with providers that operate more than one type of program.¹⁰

Within each of the seven strata, providers may appear multiple times in the list, once for each program they operate. With the exception of AR centers, all programs in each stratum will be eligible for selection. We will exclude AR centers that are “drop-in-only” centers from the sample. We expect that approximately 10 percent of AR centers serve only drop-in children and, thus, we would consider them ineligible.

Within each of the seven strata, we will sort programs by the ADA of their sponsor (where applicable) and, within that, by the ADA of their provider, and within that, by their own ADA. We will then draw a systematic sample of programs with equal selection probabilities within each stratum. We will sample most programs from the first six strata because most programs are within CACFP providers that operate only one type of program.¹¹

This sampling approach will help ensure representation of programs in different geographical areas and of different sizes, as well as minimize the possibility of sampling multiple programs of the same type from the same sponsor. The sorting by the sponsor’s ADA results in providers of a given type associated with the same sponsor being grouped together in the sorted list, and subsequently selecting a systematic sample of programs of providers from that sorted list minimizes the likelihood of sampling multiple programs of the same type from the same sponsor.

We will link programs sampled in the seventh stratum to the providers under which they operate, and the specific program that was sampled will be communicated to the provider when they are recruited into the study. For example, providers who operate both a child care center and an AR center will know that we are asking them to provide data (and, where applicable, cooperate with onsite data collection) for the child care center or the AR center—whichever program was sampled—not both. If, during recruitment or data collection, a program sampled from strata one through six is found to operate under a provider that operates more than one type of program, we will select one of the programs randomly (we will apply a weighting adjustment for this subsampling).

We assume an 80 percent combined retention rate (between recruitment and the start of data collection) and response rate among programs recruited to complete the Provider Survey and Menu Survey (for Objectives 1 and 2). Table O.1 shows the recruited sample sizes and the expected number of completed surveys for CACFP programs overall and by subgroup.

AR centers and OSHCCs might stop operations between school years. Although we will recruit and collect data from programs within the same school year, we can expect some churn between the sample selection of programs and the start of recruiting. We will address this by selecting a 50 percent reserve sample (divided into release groups of equal sizes) in addition to the initial sample. The reserve sample will include all program types, not just AR centers and OSHCCs. We will first release the initial sample and prioritize program recruitment according to whether the programs in the initial sample are operating when recruiting begins. To achieve the target number of recruited programs, we may release the reserve sample in batches while we are monitoring the recruitment response rates.

Table O.1. Recruited and completed program sample sizes, by subgroup: Objectives 1 and 2

CACFP program type	Recruited programs	Completed programs
Total	1,704	1,340
Key subgroups
Child care centers	388	310
Independent centers	203	162
Sponsored centers	185	148
Head Start centers	388	310
FDCHs	400	320
AR centers	278	200
OSHCCs	250	200
Other subgroups
Urbanicity of child care centers, Head Start centers and FDCHs
Rural	294	235
Urban	882	705
Sponsorship of sponsored centers
Sponsored, affiliated	115	92
Sponsored, unaffiliated	70	56
Corporate/chain	80	64
Other sponsored	105	84
Size of center, for child care centers and Head Start centers
Small centers	259	207
Medium centers	259	207
Large centers	259	207
Tier of FDCH
FDCH Tier I	200	160
FDCH Tier II	200	160

Note: The recruited sample sizes reflect an 80 percent combined retention and response rate among recruited programs. For AR centers, we anticipate a 10 percent ineligibility rate.

We will select the sample of programs within each key subgroup to achieve the targeted number of recruited programs. The recruited and completed program sample sizes in the “Other subgroups” are estimates and subject to variation. Details may not sum to totals due to rounding.

AR = at-risk; CACFP = Child and Adult Care Food Program; FDCH = family day care home; OSHCC = outside-school-hours care center.

4. Selection of children

To address Objectives 3a and 3b, we will collect child-level data on site in a subsample of 420 of the 1,340 programs contributing data for Objectives 1 and 2. For the onsite subsample, we will design the sample to select 90 child care centers, 90 Head Start centers, 120 FDCHs, 60 AR centers, and 60 OSHCCs (Table O.2). Specifically, when we select the sample of programs, we will designate random subsamples of programs of each type as part of the onsite subsample. In child care centers, Head Start centers, and FDCHs, we will focus the child sample on the primary age groups served by these programs—namely, 1-to-5-year-olds. Similarly, in AR centers and OSHCCs, we will focus the child sample on the primary age group served by these programs—6-to-12-year-olds. This focused selection of children will avoid the problems encountered in SNACS-I, where children outside these age ranges were allowed into the sample but, because of small sample sizes, ultimately contributed little to analyses of child-level outcomes. For example, in SNACS-I, the collection of dietary intake data included 51 children 6 years and older among providers who operated early child care (ECC) programs and 64 children ages 3 to5 among providers who operated before- and after-school programs. Where separate tabulations were prepared for these age groups, virtually all of the point estimates were statistically unreliable.

Table O.2. Recruited and completed program and child sample sizes, by program subgroup: Objectives 3a and 3b

CACFP program type	Recruited programs	Completed programs	Programs per SSU	Responding children per program	Completed children
Total	534	420	5.3	5.1	2,160
Key subgroups
Child care centers	113	90	1.1	6.0	540
Independent centers	59	47	0.6	6.0	282
Sponsored centers	54	43	0.5	6.0	258
Head Start centers	113	90	1.1	6.0	540
FDCHs	150	120	1.5	3.0	360
AR centers	83	60	0.8	6.0	360
OSHCCs	75	60	0.8	6.0	360
Other subgroups
Urbanicity of child care centers, Head Start centers and FDCHs
Rural	94	75	0.9	4.8	360
Urban	282	225	2.8	4.8	1,080
Sponsorship of sponsored centers
Sponsored, affiliated	34	27	0.3	6.0	162
Sponsored, unaffiliated	20	16	0.2	6.0	96
Corporate/ chain	24	19	0.2	6.0	114
Other sponsored	30	24	0.3	6.0	144
Size of center, for child care centers and Head Start centers
Small centers	75	60	0.8	6.0	360
Medium centers	75	60	0.8	6.0	360
Large centers	75	60	0.8	6.0	360
Tier of FDCH
FDCH Tier I	75	60	0.8	3.0	180
FDCH Tier II	75	60	0.8	3.0	180

Note: The recruited program sample size reflects an 80 percent combined retention and response rate among recruited programs. For AR centers, we anticipate a 10 percent ineligibility rate.

We will select the sample of programs within each key subgroup to achieve the targeted number of recruited programs. The recruited and completed program and child sample sizes in “Other subgroups” are estimates and subject to variation. Details may not sum to totals due to rounding.

AR = at-risk; CACFP = Child and Adult Care Food Program; FDCH = family day care home; OSHCC = outside-school-hours care center; SSU = secondary sampling unit.

1. Selecting classrooms

We will collect the information needed to sample a classroom during the Pre-Visit Planning Interview (see Appendix P). Specifically, we will obtain a roster of all children and the number of children included in each classroom. As in SNACS-I, in center-based programs (child care centers, Head Start centers, AR centers, and OSHCCs), the sampling procedure will be as follows:

We will treat a program with no defined classrooms as a single “classroom”.

1. If a program has a single classroom of 30 or fewer children, we will collect data from that classroom.

2. If a program has a single “classroom” of more than 30 ungrouped children, we will divide the classroom into sub-classrooms by age group. Within an age group with at least 14 children, we will select one age group to provide a sub-classroom. Specifically:

1. For early child care programs with 31-90 ungrouped children, the age groups for sub-classrooms are 1-2 years and 3-5 years;

2. For before and after school programs with 31-90 ungrouped children, the age groups for sub-classrooms are 6-9 years and 10-12 years;

3. For programs with over 90 ungrouped children, we will treat each single year of age as a sub-classroom.

3. If a program does not have any classrooms or sub-classrooms with at least 14 children, we will pair the classrooms/sub-classrooms with others and treat the pair as a sampling unit, so that (to the extent possible) the sampling units have at least 14 children.

4. Finally, we will select one classroom/sub-classroom sampling unit via simple random sampling.

For FDCHs, we will obtain a roster of all age-eligible children.

2. Selecting children

Within the sampled classroom¹² or FDCH, we will use the rosters to customize the study invitation packages that will be given to each parent. The invitation package will include a consent form that asks the parents about their children’s usual attendance patterns. The sampling frame for children will comprise all children in a sampled classroom or FDCH whose parents consent to participate in the study and who are expected to attend on the day(s) of observation at the program. During the Pre-Visit Planning Interview, we will ask programs to identify any children with medical or special dietary needs that require meal accommodations. Unless the program has a policy that restricts serving certain foods to all children (for example, peanut butter) or to all children in the subject child’s classroom, we will exclude children with medical or special dietary needs from the sample frame.

In centers, we will observe up to nine randomly selected children in each randomly selected classroom, with the assumption that this will yield complete child-level data collection for an average of six children per program. In FDCHs, we will observe up to four randomly selected children per home, with the assumption that we will collect complete data for an average of three children.

As in SNACS-I, we do not plan to target equal numbers of children in different age subgroups (for example, 1-to-2-year-olds and 3-to-5-year-olds). Disproportional allocation could increase the design effect and therefore the total sample needed. We designed the sample to provide sufficient precision for subgroups comprising at least 20 percent of the sample. As in SNACS-I, 3-to-5-year-olds will be the most heavily sampled group.

Overall, we expect to complete data collection for 2,160 children for Objectives 3a and 3b. Table O.2 shows target child sample sizes, overall, and by CACFP program subgroup. Table O.3 summarizes the eligibility criteria and sample sizes for the selection of children from each program.

Table O.3. Age eligibility criteria, sample sizes, and anticipated completed child sample sizes, by program type: Objectives 3a and 3b

CACFP program type	Age-eligible children	Observed children per program	Completed children per program
Child care centers and Head Start centers	1 to 5 years	1 classroom; up to 9 children per classroom	6 children per center
FDCHs	1 to 5 years	Up to 4 children per FDCH	3 children per FDCH
AR centers and OSHCCs	6 to 12 years	1 classroom; up to 9 children per classroom	6 children per center

AR = at-risk; CACFP = Child and Adult Care Food Program; FDCH = family day care home; OSHCC = outside-school-hours care center.

For each sampled child, we will collect two days of dietary intake data—one in-care day (ICD) and one out-of-care day (OCD). In addition, for a subsample of children in child care centers, Head Start centers, and FDCHs, we will collect a third day of dietary intake data (either an additional ICD or an additional OCD) to support estimation of usual dietary intakes. Within the programs sampled for onsite data collection, we will select two mutually exclusive subsamples of programs to contribute a third day of dietary recall data (one subsample for the additional ICD and one subsample for the additional OCD). We will designate all children in these subsamples for the third day of dietary recall, with the goal of obtaining the complete usual intake data for 218 children in each subsample. For the ICDs in the usual intake subsample, we will conduct an additional day of meal observations for sampled children and collect a second ICD dietary recall from parents¹³ of the sampled children. For the OCDs in the usual intake subsample, we will conduct an additional OCD dietary recall with parents of the sampled children.

In SNACS-I, the “usual intake subsample” was spread across all types of providers and age groups. In the end, however, usual intake estimates were generated only for 4-to-8-year-olds. Presumably, this was due to small sample sizes for some age groups. With the anticipated completed sample sizes shown in Table O.2, a 10 percent subsample of children in AR centers and OSHCCs (total target completes = 720) would yield 72 additional ICDs and 72 additional OCDs. This number of second recalls is unlikely to support estimation of usual intakes for the two Dietary Reference Intakes age groups participating in the CACFP in these settings(6-to-8-year-olds and 9-to-12-year-olds) participating in the CACFP in AR centers and OSHCCs. For this reason, we will allocate the full usual intake subsample to 3-5 year-old children participating in the CACFP through ECC programs. This should provide (approximately) 218 second ICDs and (approximately) 218 second OCDs (rather than 144 of each) for use in estimating usual intakes among these children.

3. Plate waste observations

We will also use the meal observations completed for children sampled under Objectives 3a and 3b to assess child-level plate waste under Objective 4. As in SNACS-I, we will use all of the meal observation data collected for ICDs, including children whose parents may not complete the ICD dietary recall. The data will come from four categories of observations: (1) children with a completed parent ICD dietary recall; (2) children without a completed parent ICD dietary recall; (3) children sampled for a second ICD that also have a completed parent ICD dietary recall; and (4) children sampled for a second ICD without a completed parent ICD dietary recall.

We assume we will have 2,160 observations from children with completed parent ICD dietary recalls and, assuming a 75 percent response rate, another 720 observations for children whose parents do not complete the ICD recall. Using the meal observations conducted for children in the usual intake subsample for ICDs will result in an additional 218 observations for children with completed second ICDs and, assuming an 83 percent response rate, an additional 44 observations for children whose parents do not complete the second ICD recall. Thus, child-level data to support Objective 4 analyses will come from an estimated 3,142 meal observations from 2,880 children (2,160 plus 720).

5. Selection of teens

To address Objective 3c (the teen study), we will collect data from teens—defined as children ages 10 to 18 years—and their parents in AR centers and OSHCCs, with the goal of collecting complete data for 720 teen-parent dyads. A complete dyad will include a teen who completes the teen survey and a parent who completes the teen parent interview. We will sample enough teens to complete 960 teen surveys and, assuming a 75 percent response rate among parents of these teens, 720 complete teen-parent dyads. All children ages 10 to 12 years sampled for Objectives 3a and 3b will be included in the teen study. To reach the 720 teen-parent dyad completes, we will sample one additional classroom in each of the 60 AR centers and 60 OSHCCs participating in the data collection (see Table O.2). We will attempt to obtain consent for all teens in these classrooms and, among the consented teens, will sample additional teens from each classroom. The number of additional teens needed to reach the targeted number of completes will depend on the number of 10-to-12-year-olds sampled for Objectives 3a and 3b. Based on the actual observed distribution of teens across AR centers and OSHCCs, we will revise the sampling approach as needed. We will increase or reduce the number of teens sampled exclusively for the teen study, factoring in the expected response rate, to ensure we achieve the targeted number of completes.

6. Selection of infants

To address Objective 5, we will sample infants within the subsample of ECC programs (child care centers, Head Start centers, and FDCHs) selected for onsite child-level data collection. Based on data from the CACFP Sponsor and Provider Characteristics Study,¹⁴ there is variation in the proportion of programs that serve infants—an estimated 78 percent of FDCHs, 72 percent of independent centers, 62 percent of sponsored centers, and 29 percent of Head Start centers (those that have an Early Head Start component).^15,16 Thus, we expect the following number of completed programs in the onsite subsample (see Table O.2) will have at least one enrolled infant: 34 of the 47 independent center completes, 27 of the 43 sponsored center completes, and 26 of the 90 Head Start center completes (for a total of 87 completed child care center programs with at least one infant); and 94 of the 120 FDCHs. In theory, we could achieve a target sample of 300 infants if we successfully recruited and completed data collection for at least one infant in 94 FDCHs and two to three infants in 87 child care centers. However, the actual number of FDCHs and centers with one or more enrolled infants may be more or less than anticipated. We will attempt to obtain consent for all infants in each center and FDCH. To account for an anticipated response rate of 50 percent and the potential that the distribution of programs with infants may differ from the above estimates, we will sample up to five consented infants per center and up to three consented infants per FDCH to reach our target of 300 completes. For centers and FDCHs with only a small number of infants, we may include in the sample all infants who are consented. As the data collection proceeds, we may adjust the number of infants sampled per program to ensure we achieve our target of completed data collection for 300 infants.

7. Estimates of meal costs

In addressing Objective 6, we will estimate average costs of producing CACFP meals and snacks. As specified in the PWS, FDCHs will not be included in these analyses. Without the FDCHs, we need to select supplementary samples of center-based programs to achieve desired levels of precision for the meal cost estimates. We will select the supplementary samples at the same time as the main sample. As shown in Table O.4, we will select a supplementary sample designed to yield 144 center-based programs to contribute data for Objective 6 (60 child care centers, 60 Head Start centers, 12 AR centers and 12 OSHCCs). We will do this using the same approach used to select programs for the main sample but restricting the sampling frame to programs not selected into the main sample. The weights for analyses of meal costs will take into account the probability of selection at each phase: the main sample selection and the supplementary sample selection. Combined with the sample of 300 center-based programs sampled for onsite data collection under Objectives 3a and 3b (the “main sample” in Table O.4), this will yield a total sample of 444 center-based programs for the meal cost data collection (150 child care centers, 150 Head Start centers, 72 AR centers, and 72 OSHCCs).

Table O.4. Recruited and completed program sample sizes by program subgroup: Objective 6

CACFP program type	Recruited programs	Completed programs					Programs per SSU
		Main sample	Supplemental sample		Total
Total	550	300		144		444
Key subgroups
Child care centers	181	90		60		150	1.9
Independent centers	94	47		30		77	1.0
Sponsored centers	87	43		30		73	0.9
Head Start centers	181	90		60		150	1.9
AR centers	99	60		12		72	0.9
OSHCCs	89	60		12		72	0.9
Other subgroups
Urbanicity of child care centers and Head Start centers
Rural	91	45		30		75	0.9
Urban	272	135		90		225	2.8
Sponsorship of sponsored centers
Sponsored-affiliated	54	26		19		45	0.6
Sponsored-unaffiliated	33	16		11		27	0.3
Corporate/chain	37	18		13		31	0.4
Other sponsored	50	24		17		41	0.5
Size of center, for child care centers and Head Start centers
Small centers	121	60		40		100	1.3
Medium centers	121	60		40		100	1.3
Large centers	121	60		40		100	1.3

Note: Details may not sum to totals due to rounding.

We assume that all programs that complete the on-site data collection will also complete the cost interview. For programs selected in the supplementary sample, we assume an 87 percent response rate to the cost interview. For AR centers, it also reflects an additional 10 percent ineligibility rate.

We will select the sample of programs within each key subgroup to achieve the targeted number of recruited programs. The recruited and completed program sample sizes in the “Other subgroups” are estimates and subject to variation.

AR = at-risk; CACFP = Child and Adult Care Food Program; OSHCC = outside-school-hours care center; SSU = secondary sampling unit.

8. Design effects and precision of estimates

For the full program sample, we will target a total of 1,340 participating programs. Assuming a conservative intra-PSU + SSU correlation of 10 percent and a design effect from weighting (DW) of 1.5,¹⁷ this will result in an overall design effect of about 1.7 to 1.9 for each key program subgroup.¹⁸ The half-width of a 95 percent confidence interval (CI) is 7.7 to 9.1 percentage points for each key program subgroup, resulting in a half-width of 3.7 percentage points for all program types combined (Table O.5). All of the key subgroups and most of the other subgroups have half-widths of 10 percentage points (rounded) or less. Subgroups within the sponsored center subgroup have larger half-widths because some of these center types are rare (that is, they represent small percentages of the universe of CACFP programs).

Table O.5. Precision levels for program subgroups: Objectives 1 and 2

CACFP program type	Program completes	Programs per SSU	Overall design effect	95 percent half-width CI (percentage points)
Total	1,340			3.7
Key subgroups
FDCHs	320	4.0	1.9	7.7
Head Start centers	310	3.9	1.9	7.7
Child care centers	310	3.9	1.9	7.7
Independent centers	162	2.0	1.7	9.9
Sponsored centers	148	1.9	1.6	10.3
AR centers	200	2.5	1.7	9.1
OSHCCs	200	2.5	1.7	9.1
Other subgroups
Urbanicity of child care centers, Head Start centers and FDCHs
Rural	235	2.9	1.8	8.6
Urban	705	8.8	2.7	6.0
Sponsorship of sponsored centers
Sponsored, affiliated	92	1.1	1.5	12.6
Sponsored, unaffiliated	56	0.7	1.5	16.0
Corporate/chain	64	0.8	1.5	15.0
Other sponsored	84	1.1	1.5	13.2
Size of center, for child care centers and Head Start centers
Small centers	207	2.6	1.7	9.0
Medium centers	207	2.6	1.7	9.0
Large centers	207	2.6	1.7	9.0
Tier of FDCH
FDCH Tier I	160	2.0	1.7	10.0
FDCH Tier II	160	2.0	1.7	10.0

Note: Assumes 25 PSUs and 80 SSUs; weighting design effect =1.5; intra-PSU + SSU correlation = .10; and variable population percent = 50. Details may not sum to totals due to rounding.

AR = at-risk; CACFP = Child and Adult Care Food Program; CI = confidence internal; FDCH = family day care home; OSHCC = outside-school-hours care center; SSU = secondary sampling unit.

At the child level (for Objectives 3a and 3b), we assume an intra-program correlation of 30 percent^19,20. This is a relatively conservative assumption that should cover most of the survey variables. For the full sample of children, the half-width is 4.1 percentage points, and for children within the key program subgroups, all half-widths are 11.8 percentage points or less (Table O.6). Half-widths for children in some of the other program subgroups are larger; however, half-widths for children in the non-key subgroups—urban and rural center programs and small, medium, and large centers—are 10 percentage points or less.

Table O.6. Precision levels for child-level estimates, by program subgroup: Objectives 3a and 3b

CACFP program type	Total responding parents/caregivers	Overall design effect	95 percent half-width CI (percentage points)
Total	2,160		4.1
Key subgroups
FDCHs	360	2.8	8.6
Head Start centers	540	4.0	8.5
Child care centers	540	4.0	8.5
Independent centers	282	3.8	11.3
Sponsored centers	258	3.8	11.8
AR centers	360	3.8	10.0
OSHCCs	360	3.8	10.0
Other subgroups
Urbanicity of child care centers, Head Start centers and FDCHs
Rural	360	3.2	9.3
Urban	1,080	6.0	7.3
Sponsorship of sponsored centers
Sponsored, affiliated	162	3.8	14.9
Sponsored, unaffiliated	96	3.8	19.4
Corporate/chain	114	3.8	17.9
Other sponsored	144	3.8	15.7
Size of center, for child care centers and Head Start centers
Small centers	360	3.8	10.0
Medium centers	360	3.8	10.0
Large centers	360	3.8	10.0
Tier of FDCH
FDCH Tier I	180	2.4	11.3
FDCH Tier II	180	2.4	11.3

Note: Assumes 25 PSUs and 80 SSUs; weighting design effect = 1.5; intra-PSU + SSU correlation = .10; intra-program correlation = .30; and variable population percent = 50. Details may not sum to totals due to rounding.

AR = at-risk; CACFP = Child and Adult Care Food Program; CI = confidence internal; FDCH = family day care home; OSHCC = outside-school-hours care center.

The precision of the estimates generated for the teen-parent dyad subgroups (for Objective 3c) will be the same as those for the younger children in the AR centers and OSHCCs (10 percentage points). The precision of the estimates generated teen-only subgroups will be less than 10 percentage points given the larger number of total teen survey completes relative to the number of teen-parent dyad completes. For plate waste estimates for Objective 4, precision estimates will be higher than those for children in Objectives 3a and 3b, as the analysis will use additional meal observations that are collected from children but not used for Objective 3a. For the sample of infants (Objective 5), we compute a half-width of a 95 percent CI of 8.0.

For estimating meal costs, we assume an intra-PSU + SSU correlation of 10 percent and a DW of 1.5; this will result in a design effect of 1.6 for child care and Head Start centers and 1.5 for AR centers and OSHCCs.²¹ We assume a population standard deviation of 35 percent of the mean. This is a very conservative benchmark—the range of costs across programs should be tighter in general. The half-width of a 95 percent CI is 4.1 percentage points for the full sample of programs, 7.2 percentage points for child care centers and Head Start centers, and 9.9 percentage points for AR centers and OSHCCs (Table O.7).

Table O.7. Precision levels for estimates of meal costs, by program subgroup: Objective 6

CACFP program type	Meal cost program subsample size	Programs per SSU	Overall design effect	95 percent half-width CI (percentage points)
Total	444			4.1
Key subgroups
Child care centers	150	1.9	1.6	7.2
Independent centers	78	1.0	1.5	9.5
Sponsored centers	72	0.9	1.5	9.9
Head Start centers	150	1.9	1.6	7.2
AR centers	72	0.9	1.5	9.9
OSHCCs	72	0.9	1.5	9.9
Other subgroups
Urbanicity of child care centers and Head Start centers
Rural	75	0.9	1.5	9.7
Urban	225	2.8	1.8	6.1
Sponsorship of sponsored centers
Sponsored, affiliated	44	0.6	1.5	12.6
Sponsored, unaffiliated	27	0.3	1.5	16.1
Corporate/chain	31	0.4	1.5	15.1
Other sponsored	40	0.5	1.5	13.2
Size of center, for child care centers and Head Start centers
Small centers	100	1.3	1.5	8.5
Medium centers	100	1.3	1.5	8.5
Large centers	100	1.3	1.5	8.5

Note: Assumes 25 PSUs and 80 SSUs; weighting design effect =1.5; intra-PSU+SSU correlation = .10; and population standard deviation = 35 percent of the mean. Details may not sum to totals due to rounding.

AR = at-risk; CACFP = Child and Adult Care Food Program; CI = confidence internal; OSHCC = outside-school-hours care center; SSU = secondary sampling unit.

9. Weighting

We will compute analysis weights at the program and child levels for each instrument or combinations of instruments, consistent with proposed analysis plans and completion rates. We will design the weights to bring the weighted distribution of the sample back in line with the population distribution and greatly reduce the potential for bias resulting from nonresponse. The various analysis weights comprise base weights that account for selection probabilities and adjustments to those weights for nonresponse.

The base weight for each stage of selection also accounts for the sampling probabilities of prior selection stages and any nonparticipation in those prior stages. For example, the base weight for a program is the inverse of the probability of selection for the program and will be the product of the PSU adjusted sampling weight, SSU adjustment weight, and the program sampling weight. We will then adjust these cumulative base weights for program nonresponse. We will compute the nonresponse adjustment factors within subsets of programs referred to as “weighting cells.” Determinants of weighting cells may include variables such as geography, level of urbanization, type and size of program, and other program characteristics. For child-level weights, we will examine whether child-level factors such as gender and age (if the programs give us that information) are correlated with child-level response propensity or child-level outcomes to inform whether those factors should be included in in child-level nonresponse cell creation. To compute the child-level weight, we will start with the program weight, make a child-level adjustment for inability to obtain consent and then compute and apply the factor for sampling children among the consented children in the program. When the response rate for a particular program or provider is high, we may consider the use of a within-program adjustment to take advantage of the correlations among those children without introducing large weighting effects.

In addition to these full-sample analysis weights, we will attach a series of jackknife replicate weights to each data record for variance estimation. In addition to the replicate weights, we will also provide stratum and unit codes in the data files to permit calculation of standard errors using the full sample weights with Taylor series approximations.

For the noncertainty States, the variance strata will be the same as the sampling strata and the variance units will be the selected States. We will pair the non-CBSA SSUs across certainty States according to their geographic locations to form variance strata. We will arrange the CBSA SSUs within the certainty States into pairs according to their geographic locations to form variance strata. We will provide information that permits analysts to account for the finite population correction (due to the high sampling rates of States in the first stage of selection) when computing variance estimates. We expect the resulting variance estimates to be an overestimation, because of the cross-State pairing of the non-CBSA SSUs in certainty States, given that we will select only a small number of non-CBSA SSUs. Also, the systematic selection of CBSA SSUs in the certainty States reduces the true variance in a way that we cannot capture, also resulting in a conservative variance estimate.

Finally, we will conduct a nonresponse bias analysis for any data collection component with a unit response rate below 80 percent or, where applicable, a cumulative response rate below 80 percent. The goals of nonresponse bias analysis are to assess the extent to which nonresponse has introduced an appreciable risk of bias, and the weighting process has corrected for any such risk. Procedures to achieve these goals include (1) identifying the factors associated with nonresponse that are available for both respondents and nonrespondents, (2) determining which of these factors are also associated with key study variables, and (3) checking to see if the weighting process corrected any imbalance on characteristics associated with both nonresponse and key study variables.

1 Metropolitan and Micropolitan Statistical Areas are collectively referred to as Core-Based Statistical Areas. Metropolitan statistical areas have at least one urbanized area of 50,000 or more people. Micropolitan statistical areas are a new set of statistical areas that have at least one urban cluster of at least 10,000 but less than 50,000 population.

2 The Performance Work Statement (PWS) mentions selecting a sample of sponsors in one place, but sampling sponsors is not mentioned in the section on study design. Because the SNACS-I Office of Management and Budget clearance package indicates that sponsors were not included as a sampling stage, we have assumed that the text in the PWS was an error. Nonetheless, although we will not be formally sampling sponsors, we will coordinate with sponsors during recruitment (before contacting any sponsored providers, we will contact sponsors to gain their cooperation). We will also collect cost-related data from sponsors, as needed.

3 Defined as households with annual incomes less than 100 percent of the Federal poverty level, based on the most recently available year of American Community Survey (ACS) one-year State-level estimates. The latest ACS one-year State-level estimates were published in October 2020.

4 For simplicity, we use the term “poor children” in the equation rather than “children in low-income households.”

5 States with much larger MOS than the others (i.e., exceeding the total MOS divided by sample size) are determined as certainty states.

6 We will contact whoever oversees the CACFP program in each State. The office may differ from the “Child Nutrition Director” depending on the State.

7 CBSAs that split across States will include only the part of the CBSA within the sampled State.

8 In SNACS-I, the number of providers in each area was not available prior to SSU selection and therefore was estimated using data on low-income children from the ACS and Childcare Aware of America (2012) with further adjustments for whether the SSU is urban or rural.

9 The FDCH sample is 320 to allow for Tier I and Tier II sample sizes of 160 each, which is necessary to achieve the 10 percentage point half-width target.

10 Groups 1 through 6 exclude programs associated with providers that operate more than one type of program.

11 In SNACS-I, which did not take the issue of multiple programs into account in sampling providers, only 51 of more than 3,000 sampled providers operated more than one program. Although this is a small overall proportion, it is important to plan for this situation. Based on SNACS-I recruitment information, only 20 of these 51 providers completed any data collection activities, and none provided data for both of the programs for which they were sampled.

12 From here forward, the term “classroom” implicitly refers to sub-classroom, where appropriate.

13 We use the term parent for simplicity; a parent or guardian must provide consent to participate in the study, but the data collection respondent can be a parent, guardian, caregiver, grandparent, or other adult most knowledgeable about the child’s food intake.

14 Glantz, F.B., A.A. Germuth, T.F. Macaluso, and K. Della Torre. “Findings of the CACFP Sponsor and Provider Study.” Prepared by Kokopelli Associates, LLC, under Contract No. AG-3198-C-13-0012. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service, 2018.

15 See Exhibit 2.11 in Volume 2 of the CACFP Sponsor and Provider Characteristics Study. Overall (all types of centers combined), 57 percent of CACFP centers serve infants.

16 Glantz, Frederic B., Germuth, Amy A., Macaluso, Theodore F., and Della Torre, Karen. (2018). CACFP Sponsor and Provider Characteristics Study. Prepared by Kokopelli Associates, LLC., Contract No. AG-3198-C13-0012. Alexandria, VA: U.S. Department of Agriculture, Food and Nutrition Service, Office of Policy Support, Project Officer: Ashley Chaifetz. Available online at: www.fns.usda.gov/research-and-analysis.

17 This weighting design effect used throughout (1.5) is meant to be very conservative, accounting for both differential sampling rates and nonresponse adjustments.

18 The overall design effect is , where is 1.50, = 4 or 3.875 or 2.5 (average cluster size), and = 10 percent.

19 The overall design effect is , where is 1.50, = 1.50 or 1.13 (average number of programs per SSU), , with the approximating formula from Skinner and colleagues (1989).

20 Skinner, C.J., D. Holt, and T.M.F. Smith. Analysis of Complex Surveys. New York: John Wiley, 1989, p. 39.

21 The overall design effect is , where is 1.50, = 1.88 (average number of programs per SSU), = 10 percent for the centers and, for AR centers and OSHCCs, the design effect is simply DW = 1.5, as is less than 1.