Memo_Sampling Households 2.5.18

Date:	February 05, 2018
To:	Alice Ann Gola and Grant Lovellette
From:	Hyunshik Lee, Tracy Vericker, and Melissa Rothstein
Subject:	Summer Meals Study: Sampling Households

This memo describes how the sample sizes for the Caregiver Survey for the Summer Meals Study (SUMS) are derived.

The precision requirements of the study are set at the summer meals site level. For estimates of Summer Food Service Program (SFSP) sites, the precision requirement is 5 percentage points of the half length of the 95 percent confidence interval to estimate a population proportion of 50 percent. The precision requirement is set at 10 percentage points for important subgroups – including Seamless Summer Option (SSO) sites; open versus closed sites; urban versus rural sites; and sites with School Food Authority (SFA) sponsors versus those with other sponsor types. We determined that we need 600 sites to meet the overall precision requirement for SFSP sites and 150 for subgroups. We will select the site sample from compiled list of sites from sampled States. A longer discussion of the site sampling strategy is included in Supporting Statement Part B of the OMB package.

For the Caregiver Survey, FNS does not pre-specify precision requirements. This is because there is no list of summer meals program participants from which we can sample and determine population sizes of subgroups of interest to FNS. Due to this limitation, we set sample size targets to achieve precision of 5 percentage points for summer meals participants versus non-participants and 10 percentage points for potential subgroups of interest, such as participants living in households with incomes ≤ 185 percent of poverty versus those living in households with incomes > 185 percent of poverty and age subgroups (e.g., <5, 5 – 9, 10 – 13, 14 – 18).

As noted, there is no frame for us to select a sample of summer meals program participants and non-participants. Thus, we need to build a frame. We plan to use multiple frames to select households living in the catchment area¹ of each respondent site as given below:

• Onsite participants (i.e., children attending the sampled site);

• SNAP households living in the site catchment area; and

• Postal addresses within the site catchment area.

These frames will be prepared for each respondent site. In order to accommodate the data collection field schedule, we will request lists of approved sponsors and sites for sampling from States at two points in time. In early 2018, we will request the 2017 list of sponsors and sites; we will use this list to select the majority of the 2018 site sample – “continuing sites”.² Then, in June 2018, we will request the lists of new sites approved by the State as of June 1^st – “new sites”. The SNAP and Postal frames will provide both participant and nonparticipant households.

Some factors and assumptions about these frames that are used for sample size calculation are given as follows:

• For the Postal frame, we need to screen for households with children – it is estimated based on the American Community Survey data that 40 percent of households have a child and are eligible for the survey—and summer meals participation status;

• From the SNAP frame, households with children can be identified; screening is only needed to determine summer meals participation status;

• Based on Gordon et al. (2003),³ it is estimated that about 15 percent of eligible households participate in the summer meals programs and 85 percent of households do not participate, which implies that too many nonparticipants will be screened in and subsampling will be needed to control the sample size;

• We assume that 25 percent of SNAP households will respond to the screener, and that 20 percent of Postal households will respond to the screener;

• For continuing sites, deduplication between the SNAP and Postal frames will be done before sampling. For the new sites, there is no time for deduplication before sampling. This requires a larger screener sample from the SNAP frame; therefore, the SNAP screener sample size is increased by 25 percent to make up for the estimated loss due to deduplication after sampling;

• The response rates to the main survey for screened (and subsampled) households are assumed to be 70 percent for the SNAP and Postal frames, 30 percent for onsite participants from open sites, and 50 percent for onsite participants from closed sites;⁴ and

• We allocate more sample to the SNAP frame because it is more cost efficient (since we know ahead of time that they have children).

Based on these considerations, we allocate the number of completes to three frame sources for each open site as follows:

• Onsite: 1 participant;

• SNAP frame: 2 participants and 3 nonparticipants; and

• Postal frame: 1 participant and 1 nonparticipant.

For each closed site, the allocation is as follows:

• Closed/Enrolled: 4 participants;

• SNAP frame: 6 nonparticipants; and

• Postal frame: 4 participants.

We expect to lose one-half of the nonparticipant completes (4) due to site-specific eligibility criteria for closed sites (e.g., age restrictions for enrollment), resulting in 4 eligible nonparticipants from each closed site, on average.

This sampling strategy will yield a sample of 3,000 summer meals participants and 3,000 non-participants. Assuming a design effect of 3.5, this will give a precision of 2.4 percentage points, which meets the 5 percent limit (see Table 1). The frames are set up to assess subgroups by site types. For instance, for SFSP sites, each open site will yield 4 complete participants and 4 nonparticipants. With 400 open sites there will be altogether 1,600 participants and 1,600 nonparticipants. Assuming a design effect of 3.5, subgroup analysis with a sample size of 400 will give a precision of 9.4 percentage points, which meets the 10 percent limit. For closed sites where there will be 800 participants and 800 (eligible) nonparticipants, the precision for the participant and nonparticipant subgroups will be 6.6 percentage points. For SSO sites, there will be 600 participants and 600 non-participants. Within each group, the precision will be 7.6 percentage points. Until Caregiver Survey data are collected, we will not know for sure whether we can achieve precision at the 10 percentage point threshold for all demographic subgroups of interest; however, the sample size targets we have planned for allow for subgroups as small as 400 to meet the subgroup precision requirement.

Table 1. Precision and Power Analysis for Various Overall and Subgroup Sample Sizes

Survey type	Overall/subgroup	Sample size	DEFF	Precision (%)	MDD (%)
Caregiver and Child	Overall	6,000	3.5	±2.4	NA
Caregiver and Child	Participants/Non-participants	3,000	3.5	±3.4	7.8
Caregiver and Child	Subgroup	2,000	3.5	±4.2	9.5
Caregiver and Child	Subgroup	1,000	3.5	±5.9	13.3
Caregiver and Child	Subgroup	800	3.5	±6.6	14.8
Caregiver and Child	Subgroup	600	3.5	±7.6	17.0
Caregiver and Child	Subgroup	400	3.5	±9.4	20.5
Site	Overall	750	1.6	±4.6	NA
Site	Subgroup within SFSP	600	1.5	±5.0	NA
Site	Subgroup within SFSP	300	1.5	±7.1	15.8
Site	Subgroup within SFSP	150	1.5	±10.0	21.8
Site	SSO	150	1.4	±9.7	NA
Sponsor	Overall with SSO	385	3	±8.8	NA
Sponsor	SFSP Only	308	3	±9.9	NA

Starting with these allocated completes and applying various rates described above, the screener sample size and the onsite (enrolled) sample size are determined. As the result of this sample size determination procedure, we obtain two sample size summary tables shown below (See Tables 2 and 3), one for continuing sites and another for new sites. Note that the total number of completes is 6,800 households (3,000 participants and 3,800 nonparticipants), but we expect to lose 800 nonparticipants from closed sites due to ineligibility based on the survey data.

Table 2. Sample Size Calculations for the Participant and Non-participant Household Surveys from Continuing Sites

Sample size calculation for a single site									Total all sites
Group	Frame	Initial sample size	Screen RR	% of HHs with eligible children	Summer meal participation or non-participation rate	Sub-sampling rate	Survey RR	Target number of completes	Completes	Screens3
SFSP Open site (n=280)
Participant	Onsite	4	NA	NA	1	1	0.3	1	280	0
Participant	SNAP	77	0.25	NA1	0.15	1	0.7	2	560	21,560
Nonparticipant	SNAP		0.2	NA1	0.85	0.328	0.7	3	840
Participant	Postal	96	0.25	0.4	0.15	1	0.7	1	280	26,880
Nonparticipant	Postal		0.2	0.4	0.85	0.219	0.7	1	280
SFSP Closed site (n=140)
Participant	Enrolled	8	NA2	NA	1	1	0.5	4	560	NA2
Nonparticipant	SNAP	35	0.25	NA1	1	1	0.7	6	840	4,900
Nonparticipant	Postal	36	0.2	0.4	1	1	0.7	2	280	5,040
SSO site (n=105)
Participant	Onsite	4	NA	NA1	1	1	0.3	1	105	0
Participant	SNAP	77	0.25	NA1	0.15	1	0.7	2	210	8,085
Nonparticipant	SNAP		0.2	NA1	0.85	0.328	0.7	3	315
Participant	Postal	96	0.25	0.4	0.15	1	0.7	1	105	10,080
Nonparticipant	Postal		0.2	0.4	0.85	0.219	0.7	1	105
								Total	4,760	76,545

¹ Not applicable because the SNAP households have already been screened for children.

² No need to screen as eligibility and participation are known from the administrative data.

³ The screener sample size is obtained by multiplying the initial sample size and the site sample size (e.g., 21,560 = 77*280).

Table 3. Sample Size Calculations for the Participant and Non-participant Household Surveys from New Sites

Sample size calculation for a single site									Total all sites
Group	Frame	Initial sample size	Screen RR3	% of HHs with eligible children	Summer meal participation or non-participation rate	Sub-sampling rate	Survey RR3	Target number of completes	Completes	Screens4
SFSP Open site (n=120)
Participant	Onsite	4	NA	NA	1	1	0.3	1	120	0
Participant	SNAP	96	0.2	NA1	0.15	1	0.7	2	240	11,520
Nonparticipant	SNAP		0.16	NA1	0.85	0.329	0.7	3	360
Participant	Postal	96	0.25	0.4	0.15	1	0.7	1	120	11,520
Nonparticipant	Postal		0.2	0.4	0.85	0.219	0.7	1	120
SFSP Closed site (n=60)
Participant	Enrolled	8	NA2	NA	1	1	0.5	4	240	NA2
Nonparticipant	SNAP	43	0.2	NA1	1	1	0.7	6	360	2,580
Nonparticipant	Postal	36	0.2	0.4	1	1	0.7	2	120	2,160
SSO site (n=45)
Participant	Onsite	3	NA	NA1	1	1	0.3	1	45	0
Participant	SNAP	96	0.2	NA1	0.15	1	0.7	2	90	4,320
Nonparticipant	SNAP		0.16	NA1	0.85	0.329	0.7	3	135
Participant	Postal	96	0.25	0.4	0.15	1	0.7	1	45	4,320
Nonparticipant	Postal		0.2	0.4	0.85	0.219	0.7	1	45
								Total	2,040	36,420

¹ Not applicable because the SNAP households have already been screened for children.

² No need to screen as eligibility and participation are known from the administrative data.

³ For the SNAP frame, the screener response rate is reduced by 20%.

⁴ The screener sample size is obtained by multiplying the initial sample size and the site sample size (e.g., 11,520 = 96*120).

1 We plan to define the catchment area for each site by drawing a circle around the site with a fixed radius (1 mile for urban sites and 5 miles for rural sites).

2 Based on 2015 and 2016 SFSP site lists from Iowa, Michigan, North Carolina, Pennsylvania, and Texas, we estimate that 64 percent of the total 2018 site population will consist of continuing sites (those that participated in SFSP/SSO in 2017) and the remaining 36 percent will be new sites.

3 Gordon, A., Briefel, R., and Allhouse, J. (2003). Feeding low-income children when school is out – the Summer Food Service program: Executive Summary Food Assistance and Nutrition Research Report No. (FANRR-30).

4 The response rates are goals, which we believe can be achieved within the survey constraints. We assume a higher rate for SNAP and Postal samples because they are screener respondents and would be more inclined to respond to the main survey. We also assume a higher rate for caregivers of enrolled children in closed sites believing that they will be more cooperative in survey participation than those caregivers of open site children.

CC: Melissa Rothstein