Appendix Q. APEC III School Sample Selection Memo

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

1. Introduction
This is the second sampling selection memo prepared under the APEC III contract. The
first sampling memo, titled Deliverable 25: SFA Sample Selection Memo, was submitted to FNS
on May 5, 2016. That report provided details of the procedures used to select the first-stage sample
of SFAs. This memo describes the second-stage sampling procedures used to select schools within
the sampled SFAs. Section 2 provides an overview of the APEC III sample design and a summary
of the SFA study notification and school verification process. Section 3 provides details of the
second-stage selection of schools.

2. Overview of Sample Design
The sampling plan for APEC III is designed to develop a sample of School Food
Authorities (SFAs) and schools for data collection and analyses to produce nationally representative
estimates of error rates, for National School Lunch Program (NSLP) and School Breakfast Program
(SBP) separately, that are fully compliant with the Improper Payments Elimination and Recovery
Act (IPERA) requirements. These NSLP and SBP estimates will be based on data collected in
School Year 2017-2018. The primary statistical goal is to provide precise estimates of error rates for
the program by source. To achieve this goal, a multistage stratified probability sampling design will
be utilized to select households for the study (i.e., the household survey) where (a) the first-stage
sampling units (FSUs) are composed of a nationally representative sample of SFAs; (b) the secondstage sampling units (SSUs) are composed of stratified samples of schools within SFAs; and (c) the
third-stage sampling units (TSUs) are composed of samples of students (households) within schools.
The sampling plan is similar to the general multistage sampling approach used in APEC I and II, but
with an increased focus on Community Eligibility Provision (CEP) SFAs and schools to reflect the
recent growth in the use of the CEP by SFAs.

Page | 3

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

2.1

SFA Sample
Because the process by which certification occurs differs between CEP and non-CEP

schools, schools that have elected to participate in CEP will be sampled separately from non-CEP
schools. For sampling purposes, we divided SFAs into those with no CEP schools and those with at
least one CEP school based on data from two data sets provided by FNS (FY15 FNS742_02222016.XLSX and CEP FY15-16 National Election Data ‒ September 2015 3-25-16.XLSX).
However, during SFA study notification (see Section 2.2) some SFAs that had been classified as
“non-CEP” SFAs for sampling purposes were later determined to contain schools that elected the
CEP option after 2015. Thus, both CEP schools and non-CEP schools will be selected from SFAs
designated as CEP and non-CEP SFAs. There were also a few instances where a CEP SFA was
found to contain no CEP schools based on data reported by the SFA.
In this memo, the terms “non-CEP SFA” and “CEP SFA” refer to status at the time of
sampling, and not to current status. Note that for analysis purposes, SFAs will be classified
according to the current CEP status. It is also possible that some of the sampled schools’ CEP status
may change for SY 2017-2018. If this occurs, the school status will be updated according to their
current status. The appropriate data collection approach for CEP versus non-CEP school will be
conducted. The sample sizes for each type of school are sufficient to accommodate such changes.
For both types of SFAs, the largest SFAs in terms of sampling measure of size1 were
included in the sample with certainty. There were six certainty SFAs among the non-CEP SFAs and
23 certainty SFAs among the CEP SFAs. For each of the two types of SFAs, the remaining (noncertainty) SFAs were selected with probabilities proportionate to the sampling measure of size (i.e.,
either (a) the number of students certified for free or reduced-price lunch in the non-CEP SFAs; or
(b) a weighted sum of the number of students certified for free or reduced-price lunch in non-CEP
schools and enrollment in CEP schools in the CEP SFAs). Note that while the sample of non-

1

The sampling measure of size (MOS) is either (a) the number of students certified for free or reduced-price meals as
reported in the FNS-742 frame in the case of the non-CEP SFAs; or (b) a composite measure of size developed from
the number of students certified for free or reduced-priced meals in CEP schools (i.e., enrollment of the school
multiplied by the ISP multiplied by 1.6) and the number of students certified for free or reduced-price meals in nonCEP schools in the case of the CEP SFAs.

Page | 4

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

certainty SFAs included a reserve sample to be used in case some SFAs were ineligible or refused to
confirm participation, it was not necessary to utilize the reserve sample for data collection. Table 2-1
summarizes the distribution of the SFA sample by type of SFA and FNS region. Additional details
of the SFA sample selection procedures are provided in Deliverable 25: SFA Sample Selection
Memo.

Table 2-1. Distribution of SFA sample by type of SFA and region
Non-CEP SFAs

FNS Region
1. Northeast
2. Mid Atlantic
3. Southeast
4. Midwest
5. Southwest
6. Mountain Plains
7. Western

Total

2.2

CEP SFAs

ALL SFAs

Certainty

NonCertainty

Certainty

NonCertainty

Certainty

NonCertainty

0
0
5
0
0
0
1

13
18
27
29
34
17
48

2
2
9
3
4
0
3

10
14
39
21
21
6
10

2
2
14
3
4
0
4

23
32
66
50
55
23
58

25
34
80
53
59
23
62

6

186

23

121

29

307

336

Total

SFA Study Notification and School Verification Results
Of the 336 sampled SFAs, 323 (including all 29 certainty SFAs and 294 non-certainty SFAs)

were successfully confirmed for participation in the study. This represents a response rate of 96
percent. Of the 192 SFAs that were originally selected as non-CEP SFAs, 10 were later found to
have CEP schools. Similarly, of the 144 SFAs that were originally selected as CEP SFAs, three were
later found to have no eligible CEP schools. The distribution of the sampled SFAs by response
status and region are summarized in Table 2-2.

Page | 5

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

Table 2-2. Distribution of sample SFAs by type of SFA, response status, and region
Non-CEP SFAs*

FNS Region
1. Northeast
2. Mid Atlantic
3. Southeast
4. Midwest
5. Southwest
6. Mountain Plains
7. Western

Total

Resp.
SFA

NonResp.
SFA

CEP SFAs**

Total

Resp.
SFA

NonResp.
SFA

ALL SFAs

Total

Resp.
SFA

NonResp.
SFA

Total

13
17
31
28
34
17
43

0
1
1
1
0
0
6

13
18
32
29
34
17
49

12
16
46
22
25
6
13

0
0
2
2
0
0
0

12
16
48
24
25
6
13

25
33
77
50
59
23
56

0
1
3
3
0
0
6

25
34
80
53
59
23
62

183

9

192

140

4

144

323

13

336

* Ten SFAs (including one certainty SFA) that had been classified as non-CEP for sampling purposes were later found
to have CEP schools.
** Three SFAs (including one certainty SFA) that had been classified as CEP for sampling purposes were later found to
have no eligible CEP schools.

2.3

Selection of Subsample of SFAs for School Sampling
As indicated above, 323 of the 336 sampled SFAs were confirmed for participation in the

study, far exceeding the number expected under an assumed SFA response rate of 80 percent (the
assumption used for design purposes). Because the number of responding SFAs was much greater
than the minimum requirement, for data collection, a random sample of 302 of the 323 SFAs was
selected for subsequent school sampling.2 The remainder of this section provides a summary of
how the 302 SFAs were selected.
First, all 29 certainty SFAs were retained for subsequent school sampling. Second, the
remaining 294 responding SFAs were then sorted by CEP status and measure of size. Finally, from
the sorted list, 273 SFAs were selected systematically with equal probability. Table 2-3 shows the
distribution of the retained SFAs by SFA type and FNS region, and the corresponding numbers of
eligible CEP and non-CEP schools that were reported by the SFAs.

2

The remaining 21 SFAs could potentially be used to provide a reserve sample of schools. However, we do not
anticipate that this will be necessary.

Page | 6

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

Table 2-3. Distribution of retained SFAs by type of SFA and number of CEP and non-CEP schools,
by FNS region
Non-CEP SFAs

FNS Region
1. Northeast

No. of
SFAs
11

CEP SFAs

No. of
non-CEP
schools
78

No. of
CEP
schools
3

No. of
SFAs
11

No. of
non-CEP
schools
1,526

All SFAs
No. of
CEP
schools
784

No. of
SFAs
22

No. of
non-CEP
schools
1,604

No. of
CEP
schools
787

2. Mid Atlantic

14

403

1

15

294

726

29

697

727

3. Southeast

29

1,908

6

44

837

1,766

73

2,745

1,772

4. Midwest

25

185

-

20

147

1,381

45

332

1,381

5. Southwest

33

642

120

24

623

1,021

57

1,265

1,141

6. Mountain Plains

17

508

-

6

68

88

23

576

88

7. Western

41

1,155

93

12

1,158

635

53

2,313

728

170

4,879

223

132

4,653

6,401

302

9,532

6,624

Total

2.4

Preparation of School Sampling Frame
From the responding SFAs, we collected school data that were necessary for school

sampling. The data were first obtained from public records and later verified by the SFAs. These
data included school name, address, lowest and highest grades, CEP status, CEP group where
applicable, total enrollment, number of students eligible for free or reduced-price meals, and other
information. Quality control checks were conducted on the data reported by SFAs, and the data
were updated when needed. For example, duplicate schools were removed from the list after
verifying they were indeed duplicates. Missing data were retrieved from SFAs through follow-up,
and inconsistent data (e.g., total enrollment was less than the number of students eligible for free or
reduced-price meals) were investigated and corrected. Finally, schools that were pre-kindergarten
only were removed from the school lists since they are not eligible for APEC III. The updated
school list (i.e., school sampling frame) contained a total of 16,156 schools, of which 9,532 are nonCEP and 6,624 are CEP (Table 2-3).

3. School Sample
As described in the following subsections, the school sample was selected in the following
steps. First, the sample size goals (targets) were revised to reflect changes in the CEP status of SFAs
and the higher than anticipated SFA response rates. Next, the samples were allocated to SFAs in a
Page | 7

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

manner designed to select either one or three schools per non-certainty SFA as per the study design
(discussed briefly below). The required numbers of schools were then selected with probabilities
proportionate to measure of size within the SFA. To select the CEP school sample, an additional
stage of sampling was implemented in some SFAs to maximize the number of schools to be selected
from the same CEP group whenever possible.
As described in the APEC III Final Study Design, the school sample selection incorporates a
requirement to sample one or three schools from each SFA. While the APEC III sampling plan
preserves the APEC II sample design, it includes an increase in the number of SFAs sampled to
facilitate a thorough exploration of the relationship between SFAs implementation strategies and
error rates. These additional analyses are an enhancement for APEC III. To capture the relationship
between SFA policies and error rates we need to sample at least one school per additional SFA.
Sampling more than one school for the additional SFAs is unnecessary as the sample of schools
already meets precision objectives. Sampling only one school from the additional sample of SFAs
will (a) provide a school that is linked to the additional SFA; (b) add additional schools to the
sample; and (c) avoid an exponential increase in costs by unnecessarily doubling the number of
schools in the sample. The latter not only reduces the overall burden on SFAs and schools but also
judiciously manages data collection operations and costs while augmenting the sample size and
maintaining precision. Note that this restriction does not apply to the 29 certainty SFAs.
As an additional note, the sampling of schools follows the plan outlined in the APEC III Final Study
Design. The study design provided estimates on number of schools, which have been finalized (and
differ slightly) based on final school data verified by the SFAs.

3.1

Sample Size Goals
Table 3-1 summarizes the target sample size of 626 schools for APEC III (438 non-CEP

schools and 188 CEP schools). The numbers shown in the table refer to the desired number of
schools (participating schools) for subsequent household/application sampling. In order to ensure
that these sample sizes can be achieved, a larger sample will initially be selected. For sampling
purposes, we assume that the response rate among the selected schools will be at least 80 percent.
Thus, a total of approximately 782 schools would have to be sampled to yield 626 participating
Page | 8

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

schools, of which approximately 548 will be non-CEP schools and 234 will be CEP schools. The
numbers of schools to be selected assuming an 80 percent response rate are shown in Table 3-2.
Note that the actual numbers to be selected will differ slightly from those shown in this table due to
implementation of the one- or three- schools per SFA requirement described at the beginning of this
section. Figure 3-1 provides a summary of the number of SFAs and schools sampled.

Table 3-1. Target number of schools agreeing to participate in APEC III

SFA Type
Non-CEP

CEP

All Types

Certainty
status of SFA

Number of
participating
SFAs
retained for
school
sampling

Certainty
Non-certainty

6
164

18
331

1
12

19
343

Subtotal

170

349

13*

362

Certainty
Non-certainty

23
109

33
56

47
128

80
184

Subtotal

132

89

175

264

Certainty
Non-certainty

29
273

51
387

48
140

99
527

Grand total

302

438

188

626

Non-CEP
Schools

CEP
Schools

Total
Schools

* At the time the SFA sample was selected, the non-CEP SFAs by definition were not
expected to have CEP schools. It was later determined during SFA recruitment that a small
number of non-CEP schools had become CEP.

Page | 9

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

Table 3-2. Expected number of schools to be selected to achieve
target sample sizes based on 80 percent response rate

SFA Type
Non-CEP

CEP

All Types

Certainty
status of SFA

Number of
participating
SFAs
retained for
school
sampling

Certainty
Non-certainty

6
164

23
414

1
15

24
429

Subtotal

170

437

16

453

Certainty
Non-certainty

23
109

41
70

58
160

99
230

Subtotal

132

111

218

329

Certainty
Non-certainty

29
273

64
484

59
175

124
658

Grand total

302

548

234

782

Non-CEP
Schools

CEP
Schools

Total
Schools

Note: Table 3-2 shows the “target” numbers used to establish sampling rates. The final sample sizes differ slightly from these numbers due to the
implementation of the one- or three- schools per SFA requirement described earlier. Figure 3-1 and Table 3-3 show the actual sample sizes.

Page | 10

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

Figure 3-1. Summary of SFAs and schools sampled

302 SFAS

170 non CEP SFAs

132 CEP SFAs

1
16 CEP Schools

437 non CEP Schools

214 CEP Schools

110 non CEP Schools

547 non CEP Schools

230 CEP Schools

777 Total Schools

1.

3.2

Note: These numbers reflect the original designation of the SFA type (non-CEP or CEP) during sampling. During the SFA Confirmation
and Verification, some SFAs were found to fall into a different category. The original sampling status was maintained, but we sampled
according to their true status. As a result, some CEP schools are sampled from SFAs originally designated as non-CEP.

Allocation of School Sample to SFAs
In general, the approximately optimum allocation of the school sample to the 302 SFAs

retained for school sampling (see Section 2.3) depends on the number of eligible schools in the SFA,
the measure of size of the eligible schools in the SFA, and the probability of selecting the SFA in
which the schools are located. Specifically, for each of the categories of schools defined by (a) SFA
type (non-CEP vs. CEP) and (b) school type (non-CEP vs. CEP) within SFA, let
m

=

the number of sample SFAs in a given category;
Page | 11

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

𝑃ℎ𝑆𝐹𝐴 =

the probability of selecting SFA h;

𝑁ℎ

=

the total number of eligible schools reported by SFA h during recruitment;

𝑀ℎ𝑖

=

the measure of size (MOS) of school i in SFA h based on information collected during
SFA recruitment;

𝑀ℎ

=

̂
𝑀

=

𝑛𝑠

=

ℎ
∑𝑁
𝑖=1 𝑀ℎ𝑖 = the total MOS of all 𝑁ℎ schools in SFA h;

𝑆𝐹𝐴
∑𝑚
= the estimated total MOS based on the 𝑚 sample SFAs;
ℎ=1 𝑀ℎ /𝑃ℎ

the number of schools of a particular type (i.e., either non-CEP school or CEP school)
to be sampled from the m sampled SFAs.

Note that the ns ’s correspond to the sample sizes given in Table 3-2. Because we want to
select the schools with overall probabilities proportionate to size (PPS), where the size measure for
school i in SFA h is defined by 𝑀ℎ𝑖 , the optimum allocation of the 𝑛𝑠 schools to SFA h is given by
𝑆𝐹𝐴
𝑆𝐹𝐴
̂
𝑛ℎ = 𝑛𝑠 (𝑀ℎ /𝑃ℎ𝑆𝐹𝐴 ) /( ∑𝑚
ℎ=1 𝑀ℎ /𝑃ℎ ) = 𝑛𝑠 (𝑀ℎ /𝑃ℎ ) / 𝑀,

(1)

and the corresponding within-SFA selection probability of selecting school i in SFA h is
𝑤𝑖𝑡ℎ𝑖𝑛
̂ ).
𝑃ℎ𝑖
= 𝑛ℎ (𝑀ℎ𝑖 /𝑀ℎ ) = (1/𝑃ℎ𝑆𝐹𝐴 ) (𝑛𝑠 𝑀ℎ𝑖 /𝑀

(2)

The within-SFA sampling rates given by equation (2) are approximately optimal because the
resulting overall probability of selecting school i in SFA h is
𝑜𝑣𝑒𝑟𝑎𝑙𝑙
𝑤𝑖𝑡ℎ𝑖𝑛
̂,
𝑃ℎ𝑖
= 𝑃ℎ𝑆𝐹𝐴 𝑃ℎ𝑖
= 𝑛𝑠 𝑀ℎ𝑖 /𝑀

(3)

which is the desired probability of selection under a PPS sample design. The number of schools to
be selected will vary from PSU to PSU but will average to be around 𝑛𝑠 /𝑚 .
However, the study design specifies the selection of either one or three schools from an SFA
whenever possible. This applies only to the non-certainty SFAs. Because of the relative importance
of the certainty SFAs, there are no restrictions on the number of schools to be selected from them.
Thus, where feasible, the non-certainty SFAs will be randomly assigned to one of two groups, A or
B. If assigned to A, three schools will be selected. If assigned to B, one school will be selected. Note
that if an SFA contains two or fewer eligible schools, then only one school will be sampled.

Page | 12

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

While this method of sample allocation is unbiased, it will increase sampling variances and
thus reduce statistical precision. The reason for the reduced precision is that there will be an
increased clustering effect due to the selection of three schools per SFA in a proportion of the
SFAs. The reduction in precision will depend on the magnitude of the intraclass correlation between
schools within SFAs of the characteristics being measured in the survey (e.g., error rates or improper
dollar amounts). To illustrate, consider the simple case where the overall goal is to sample two
schools per SFA. Let 𝑦̅ denote an estimated mean based on the sample of 𝑛 = 𝑚 𝑛̅ = 2𝑚 schools,
where 𝑚 = the number of sample SFAs and 𝑛̅ = 2 schools per SFA. Then the variance of 𝑦̅ is
approximately V1(𝑦̅) = 𝜎 2 [1 + (𝑛̅ -1) ] / 𝑚 𝑛̅ = 𝜎 2 [1 +  ] / 2𝑚 , where 𝜎 2 is the population
variance of the y-characteristic and is the intraclass correlation between schools within SFAs.
However, under the proposed sample design, the goal will be to select either 1 or 3 schools per SFA.
Suppose, for the sake of simplicity, that in one-half of the sampled SFAs we will select exactly 1
school per SFA, while in the remaining half we will select exactly 3 schools per SFA. Let A denote
the set of SFAs where three schools per SFA will be selected, and let B denote the set where one
school will be selected. Let 𝑦̅Adenote the estimated mean based on the 3𝑚/2 schools in the SFAs in
A, and let 𝑦̅B denote the estimated mean based on the 𝑚/2 schools in the SFAs in B.
The overall estimate obtained by combining the results of A and B is the average of the two
sample estimates, i.e., 𝑦̅AB = (𝑦̅A+ 𝑦̅B)/2, and the variance of the combined estimate is given
approximately by V2(𝑦̅AB) = 𝜎 2 [1 + (1+2 )/3] / 2𝑚 . The ratio V1(𝑦̅) / V2(𝑦̅AB) = 3(1+ )/(4+2)
is always less than 1 and provides a measure of how much smaller the variance of 𝑦̅ is (under the
traditional two-stage sampling approach) compared with the variance of 𝑦̅AB from the proposed
“three-or-one” sampling scheme. If  = 0, V1(𝑦̅) = 0.75 V2(𝑦̅AB) (i.e., in this case, the variance based
on the proposed sampling scheme is about 33% greater than that from the traditional approach). If 
= 0.20, V1(𝑦̅) = 0.82 V2(𝑦̅AB). The two methods have equal variance when  = 1. Despite the
potentially larger sampling variances, the sampling method is unbiased and will yield statistically valid
estimates from the survey.

Page | 13

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

3.3

Clustering the Sample of CEP Schools by CEP Group
The way in which the identified student percentages (ISPs) are determined for the CEP

schools varies from SFA to SFA. In some SFAs, the ISP is determined on a school-by-school basis.
In others, the ISP is determined on a districtwide basis. Yet in other SFAs, the ISP is determined for
specific groups (i.e., collections of schools) within the SFA such that all schools within a group
receive the same ISP. Because the determination of error rates depends on how the schools are
grouped, it is desirable to cluster the sampled schools by the ISP groups within the SFA. This can be
accomplished by identifying the various groups of schools within the SFA, and then randomly
selecting one group with probability proportionate to size (where the size of the cluster is the sum of
the size measures of the CEP schools in the cluster). Within the selected group (cluster), up to three
CEP schools will be sampled. In other words, an additional stage of sampling will be introduced in
some SFAs where the CEP schools are grouped for ISP determination. The proposed approach will
help ensure that multiple sampled schools are from the same ISP group to the extent feasible, but
does not guarantee it because all schools in the SFA must be given appropriate chances of selection
to avoid potential sampling biases. For example, if in a particular SFA there are ISP groups
consisting of three or more schools as well as groups consisting of individual schools, it is possible
that schools selected for the sample are from different ISP groups. However, when ISP groups do
not contain the desired target numbers, schools will be selected from different ISP groups. Note that
the clustering does not apply to SFAs in which each CEP school is its own group, nor those SFAs in
which all CEP schools belong to the same group. Additional details about the selection of ISP
groups are provided in Section 3.5.

3.4

Selection of the Non-CEP School Sample
A total of 547 non-CEP schools were selected for the study, including 437 in non-CEP

SFAs and 110 in CEP SFAs. The samples were selected following the general guidelines outlined in
Section 3.2. Additional details are provided below, separately for the non-CEP and CEP SFAs.
Figure 3-2 provides a summary of the non-CEP schools.

Page | 14

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

Figure 3-2. Summary of non-CEP school sample

547 non CEP Schools

437 from non CEP SFA

110 from CEP SFA

Selected From 168 non CEP
SFAs

Selected from 55 CEP SFAs

37 SFAs with one school
0 SFAs with two schools
127 SFAs with three schools
4 SFAs with more than three schools

38 SFAs with one school
1 SFAs with two schools
13 SFAs with three schools
3 SFAs with more than three schools

Page | 15

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

3.4.1 Non-CEP Schools in Non-CEP SFAs
A total of 437 non-CEP schools in non-CEP SFAs was selected for the study sample. Of
these, 23 were in certainty SFAs, and 414 were in non-certainty SFAs. The 23 schools in the six
certainty SFAs were selected as follows: First, all of the eligible schools in the certainty SFAs were
sorted by SFA, three school levels (elementary, middle, secondary/combined)3 within SFA, and
measure of size (MOS) within school level. From the sorted list, a systematic sample of 23 schools
was selected with probabilities proportionate to the MOS using standard algorithms (e.g., see Kish,4
1965, page 230). The number of schools sampled per certainty SFA varied from 3 to 7.
Ordinarily, the sample of schools in the non-certainty SFAs would be selected with the
optimal probabilities given by equation (2) (see Section 3.2). However, because of the requirement to
restrict the number of sample schools to either 1 or 3 schools per SFA to the extent feasible, the
optimal probabilities of selection had to be modified to meet these conditions. The modifications
were made in a manner designed to minimize the adverse impact on design effects. Thus, the 414
schools in the non-certainty SFAs were selected using the procedures outlined below. The set
number given below is simply a label to group SFAs that meet the criteria for that set.


Set 11: If the non-CEP SFA contained two or fewer non-CEP schools, or the
expected sample size for the non-CEP SFA was less than 1.5 non-CEP schools
and there were no CEP schools in the SFA, exactly one non-CEP school was
sampled.



Set 12: If the non-CEP SFA contained both CEP and non-CEP schools, and the
expected number of non-CEP schools to be sampled was less than 1.5, then
exactly one non-CEP school was sampled. In addition, up to three CEP schools
were sampled and set aside for further processing as described in Section 3.5.



Set 21: If the non-CEP SFA contained three or more non-CEP schools and no
CEP schools, and the expected number of non-CEP schools to be sampled was
between 1.5 and 2.5, the SFA was randomly assigned with appropriate
probabilities to one of two sampling groups, A or B. If assigned to B, exactly one
school was sampled. If assigned to group A, three schools were sampled.

3

School level was as an implicit rather than an explicit stratifier to help ensure that schools of the three grade levels were
appropriately represented in across the entire sample of SFAs.

4

Kish, L. (1965). Survey Sampling. John Wiley & Sons: New York.

Page | 16

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO



Set 22: If the non-CEP SFA contained both CEP and non-CEP schools, and the
expected number of CEP schools to be sampled was 1 or 2, while the expected
number of non-CEP schools to be sampled was more than 1.5, then (a) 1 or 2
CEP schools were sampled according to the expected number to be sampled;
and (b) the SFA was randomly assigned to sampling group A or B, and if
assigned to B, one non-CEP school was sampled, but if assigned to A, three nonCEP schools were sampled.



Set 32: If the non-CEP SFA contained only non-CEP schools and the expected
number of non-CEP schools to be sampled was greater than 2.5, the SFA was
randomly assigned with appropriate probabilities to one of two sampling groups,
A or B. If assigned to B, exactly one school was sampled. If assigned to group A,
three schools were sampled.

The distribution of the sample of non-CEP schools in the non-CEP SFAs is summarized in
Table 3-3 (on page 24). A list of the selected schools is provided in an Excel file accompanying this
memo.

3.4.2 Non-CEP Schools in CEP SFAs
A total of 110 non-CEP schools in CEP SFAs were selected for the study sample. Of these,
41 were in certainty CEP SFAs, and 69 were in non-certainty CEP SFAs. The 41 schools in the 23
certainty CEP SFAs were selected. First, all of the eligible schools in the certainty SFAs were sorted
by SFA, three school levels (elementary, middle, and secondary/combined) within SFA, and MOS
within school level. Next, from the sorted list, a systematic sample of 41 schools was selected with
probabilities proportionate to the MOS using standard algorithms (e.g., see Kish, 1965, page 230).
The number of non-CEP schools selected per certainty SFA varied from 0 to 19.
Similar to the selection of schools in the non-CEP SFAs, the sample of schools in the noncertainty CEP SFAs ordinarily would be selected with the optimal probabilities given by equation
(2). However, because of the requirement to restrict the number of sampled schools to either 1 or 3
CEP schools per SFA to the extent feasible, the optimal probabilities of selection were adjusted to
meet these conditions in a manner designed to minimize the adverse impact on design effects. Thus,
the 69 non-CEP schools in the non-certainty CEP SFAs were selected using the procedures outlined
below. Again, the set number given below is simply a label to group SFAs that meet the criteria for
that set.
Page | 17

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO



Set 11: If there were fewer than three CEP schools and no non-CEP schools in
the SFA, then exactly one CEP school was selected from the SFA.



Set 12: If there were fewer than three CEP schools and fewer than three nonCEP schools in the SFA, then one CEP school and one non-CEP school was
selected from the SFA.



Set 13: If there were fewer than three CEP schools in the SFA and the expected
number of non-CEP schools to be sampled in the SFA was between 1.5 and 2.5,
then one CEP school and one or three non-CEP schools were selected from the
SFA depending on the expected sample size.



Set 14: If there were fewer than three CEP schools in the SFA and the expected
number of non-CEP schools to be sample was greater than 2.5, then one CEP
school and three non-CEP schools were selected from the SFA.



Set 21: If there were only CEP schools in the SFA and the expected number of
CEP schools to be sampled in the SFA was between 1.5 and 2.5, then the SFA
was randomly assigned with appropriate probabilities to one of two sampling
groups, A or B. If assigned to B, exactly one CEP school was sampled from the
SFA. If assigned to group A, three CEP schools were sampled from the SFA.



Set 22: If the SFA contained both CEP and non-CEP schools, and the expected
number of CEP schools to sample was less than 1.0 but the expected number of
non-CEP schools to sample was 2.0 or greater, then one CEP school was
selected with probability equal to the expected number of CEP schools, and
three non-CEP schools were selected from the SFA.



Set 23.1: If the SFA contained both CEP and non-CEP schools, and the
expected number of CEP schools to sample was between 1.5 and 2.5, but the
expected number of non-CEP schools to sample was less than 0.4, then (a) the
SFA was randomly assigned with appropriate probabilities to one of two
sampling groups, A or B, and if assigned to B, exactly one CEP school was
sampled from the SFA, or if assigned to group A, three CEP schools were
sampled from the SFA; and (b) one non-CEP school was selected from the SFA
with probability 0.10 or 0.50, depending on the expected number of schools to
be sampled.



Set 23.2: If the SFA contained both CEP and non-CEP schools, and the
expected number of CEP schools to sample was greater than 1.5 and the
expected number of non-CEP schools to sample was less than 2.2, then the SFA
was randomly assigned with appropriate probabilities to one of two sampling
groups, A or B, and if assigned to B, exactly one CEP school and one non-CEP
school were sampled from the SFA, or if assigned to group A, three CEP
Page | 18

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

schools and one non-CEP schools were sampled from the SFA.


Set 24: If the SFA contained both CEP and non-CEP schools, and the expected
number of CEP schools to sample was less than 1.5 and the expected number of
non-CEP schools to sample was less than 2.2, then one CEP school and one
non-CEP school were selected from the SFA.

The distribution of the sample of non-CEP schools in the CEP SFAs is summarized in
Table 3-3 (on page 24). A list of the selected schools is provided in an Excel file accompanying this
memo.

3.5

Selection of the CEP School Sample
As noted in Table 3-3 (see page 24), a total of 230 CEP schools were selected for the study,

including 16 in non-CEP SFAs and 214 in CEP SFAs. The samples were selected following the
general guidelines outlined in Section 3.2. Additional details are provided below, separately for the
non-CEP and CEP SFAs. Figure 3-3 provides a summary of the CEP school sample.
Unlike the non-CEP school sample, the way in which CEP schools were selected depended
on whether one or two or more schools were to be selected from the SFA. If a single school was to
be selected from the SFA, the school was selected randomly from the eligible schools in the SFA. If
two or more schools were to be selected from the SFA, then an additional stage of selection was
introduced for some SFAs. In this case, an ISP group (i.e., a “sample cluster”) was selected in the
first stage with probability proportionate to size, and then schools within the selected clusters were
randomly selected in the second stage. Some ISP groups contained fewer schools than the target
sample size, and those groups were combined into clusters so that each cluster contained at least the
target number of schools. When those clusters are selected, the sample schools within them may
come from different ISP groups. There are a few exceptions where the combining of small ISP
groups was not conducted and schools were selected in a single stage without the initial selection of
clusters: (1) SFAs in which each school uniquely represented an ISP group; (2) SFAs in which the
vast majority of ISP groups contained a single school, but a few groups had three or more schools;
(3) SFAs in which most schools were in one group, but there were several smaller groups with fewer
Page | 19

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

CEP schools than the target number; or (4) SFAs in which all schools belonged to only one ISP
group. For the first three scenarios, the sampled schools may come from different ISP groups,
although the chances for this to happen in the third scenario are very small. Additional details about
the selection of the CEP schools are described in Sections 3.5.1 and 3.5.2
Figure 3-3. Summary of CEP school sample

230 CEP Schools
1
16 from non CEP SFA

214 from CEP SFA

Sampled from 132 SFAs

83 SFAs with 1 CEP
School Sampled

Random Selection of
Schools

49 SFAs with more than
1 CEP School Sampled

37 SFAs
One ISP Group
(district wide)

1 SFA
Single School ISP
Groups

Random Selection of Schools

11 SFAs Multi School
ISP Groups

1 or 2 groups by PPS, then
Random Selection of Schools

Page | 20

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

3.5.1 CEP Schools in Non-CEP SFAs
As indicated previously, there were a small number of non-CEP SFAs containing both CEP
and non-CEP schools. In these SFAs, one to three CEP schools were sampled as described below.
First, all of the CEP schools in the non-CEP SFAs were sorted by SFA, three school levels
(elementary, middle, secondary/combined) within SFA, and MOS within school level. Next, from
the sorted list, a target number (see Section 3.4.1 for the assignment of target number) of schools
were selected with probabilities proportionate to the MOS. It was not necessary to select an ISP
group first, since all schools belonged to only one ISP group in the SFAs where more than one CEP
schools were to be selected.
The distribution of the sample of CEP schools in the non-CEP SFAs is summarized in
Table 3-3 (see page 24). A list of the selected schools is provided in Attachment A.

3.5.2 CEP Schools in CEP SFAs
Selecting CEP Schools in Certainty CEP SFAs
A total of 214 CEP schools in the CEP SFAs was selected for the study sample. Of
these, 58 were in certainty CEP SFAs, and 156 were in non-certainty CEP SFAs. The 58
schools in the 23 certainty CEP SFAs were selected as described below.
First, we determined the number of CEP schools to be selected from each SFA by
allocating the 58 sample schools in proportion to the measure of size of the SFA. For those
SFAs with an allocation of one sample school, exactly one school was selected randomly
from the eligible schools in the SFA.
For those SFAs with an allocation of two or more schools, we identified three types
of SFAs as follows:


Type 1: SFAs with only one ISP group across the entire SFA;
Page | 21

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO



Type 2: SFAs in which all ISP groups consisted of only one school; and



Type 3: SFAs with multiple ISP groups, most of which contained 2 or more schools.

From the Type 1 and 2 SFAs, the specified number of CEP schools were selected randomly
and systematically from a list of the eligible schools in the SFA that been ordered by grade level
(elementary, middle, secondary/combined). From the Type 3 SFAs, one or two ISP groups (i.e.,
clusters) were initially selected with probability proportionate to size, and the specified number of
schools were randomly selected from the selected clusters. In general, the aim was to sample no
more than three CEP schools per cluster. However, there was one SFA (City of Chicago) where all
of the CEP schools were in the same ISP group and the target sample size was 9 CEP schools.
Therefore, 9 CEP schools were selected from the same ISP group for that SFA. The number of
CEP schools selected per certainty SFA varied from 0 to 9. Because the schools were selected across
the entire set of certainty SFAs using a single random starting point, it was possible by chance to
skip over (i.e., not select) the schools in some SFAs.

Selecting CEP Schools in Non-Certainty CEP SFAs
Similar to the selection of schools in the non-CEP SFAs, the sample of CEP schools in the
non-certainty CEP SFAs ordinarily would be selected with the optimal probabilities given by
equation (2) in Section 3.2. However, because of the requirement to restrict the number of sampled
schools to either 1 or 3 CEP schools per SFA to the extent feasible, the optimal probabilities of
selection were adjusted to meet these conditions in a manner designed to minimize the adverse
impact on design effects. Thus, the 156 CEP schools in the non-certainty CEP SFAs were selected
using the procedures outlined below. Again, the set number given below is simply a label to group
SFAs that meet the criteria for that set.



Set 11: If there were fewer than three CEP schools and no non-CEP schools in
the SFA, then one CEP school was selected from the SFA.



Set 12: If there were fewer than three CEP schools and fewer than three nonPage | 22

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

CEP schools in the SFA, then one CEP school was selected from the SFA.


Set 13: If there were fewer than three CEP schools in the SFA and the expected
number of non-CEP schools to be sampled in the SFA was between 1.5 and 2.5,
then one CEP school was selected from the SFA.



Set 14: If there were fewer than three CEP schools in the SFA and the expected
number of non-CEP schools to be sampled was greater than 2.5, then one CEP
school was selected from the SFA.



Set 21: If there were only CEP schools in the SFA and the expected number of
CEP schools to be sampled in the SFA was between 1.5 and 2.5, then the SFA
was randomly assigned with appropriate probabilities to one of two sampling
groups, A or B. If assigned to group B, exactly one CEP school would be
sampled from the SFA. If assigned to group A, three CEP schools would be
sampled from the SFA according to whether the SFA was Type 1, 2, or 3 as
indicated above.



Set 22: If the SFA contained both CEP and non-CEP schools, and the expected
number of CEP schools to sample was less than 1.0 but the expected number of
non-CEP schools to sample was 2.0 or greater, then one CEP school was
selected from the SFA with probability equal to the expected number of CEP
schools.



Set 23: If the SFA contained both CEP and non-CEP schools, and the expected
number of CEP schools to sample was between 1.5 and 2.5, the SFA was
randomly assigned with appropriate probabilities to one of two sampling groups,
A or B. If assigned to group B, one CEP school would be sampled from the
SFA. If assigned to group A, three CEP schools would sampled from the SFA
according to whether the SFA was Type 1, 2, or 3 as indicated above.



Set 24: If the SFA contained both CEP and non-CEP schools, and the expected
number of CEP schools to sample was less than 1.5 and the expected number of
non-CEP schools to sample was less than 2.2, then one CEP school and one
non-CEP school were selected from the SFA.

The distribution of the sample of CEP schools in the CEP SFAs is summarized in
Table 3-3 (on page 24). A list of the selected schools is provided in a separate Excel file
accompanying this memo (Attachment A).

Page | 23

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

Table 3-3. Distribution of the school sample by type of SFA, FNS region, and SFA certainty
status
Non-CEP SFAs
CEP
Schools

CEP SFAs

FNS Region / SFA
certainty status

Non-CEP
Schools

FNS Region:
1. Northeast
2. Mid Atlantic
3. Southeast
4. Midwest
5. Southwest
6. Mountain Plains
7. Western

27
35
84
63
80
45
103

1
1
2
0
7
0
5

28
36
86
63
87
45
108

21
10
26
4
19
6
24

22
25
72
34
35
9
17

43
35
98
38
54
15
41

48
45
110
67
99
51
127

23
26
74
34
42
9
6

71
71
184
101
141
60
133

SFA Certainty Status:
Certainty
Non-certainty

23
414

1
15

24
429

41
69

58
156

99
225

64
483

59
171

123
654

Total

437

16

453

110

214

324

547

230

777

Total

Non-CEP
Schools

CEP
Schools

ALL SFAs

Total

Non-CEP
Schools

CEP
Schools

Total

Page | 24

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

Attachment A
List of Sampled Schools
TO BE PROVIDED IN AN EXCEL FILE

Page | 25

APPENDIX Q. APEC III SCHOOL SAMPLE SELECTION MEMO

This page is intentionally blank

Page | 26