Download:
pdf |
pdfNATIONAL CENTER FOR EDUCATION STATISTICS
NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
National Assessment of Educational Progress (NAEP)
2019 and 2020
Long-Term Trend (LTT) 2020 Update 2
Appendix G1
NAEP 2013 Sample Design
OMB# 1850-0928 v.16
September 2018
No changes since v.10
NAEP Technical Documentation Website
NAEP Technical Documentation NAEP
2013 Sample Design
The sample design for NAEP 2013 included
samples for various operational, special
study, and pilot test assessments.
Representative samples were drawn for the
following operational assessments:
2013 State Assessment Sample
Design
2013 National Assessment Sample
Design
national assessments in mathematics
and reading in public and private schools at grades 4, 8, and 12;
state-by-state and Trial Urban District Assessments (TUDA) assessments in
mathematics and reading in public schools at grades 4 and 8; and
state-by-state assessments in mathematics and reading in public schools at
grade 12 in 13 states.
Representative samples were drawn for the following special studies and pilot test
assessments:
pilot test of the computer-based assessment of Technical and Engineering
Literacy (TEL) in public schools at grade 8;
a special mathematics assessment in public and private schools in Puerto Rico
at grades 4 and 8;
Accessible Booklet Study in reading in public and private schools at grades 4
and 8;
study to examine the link between Lexile and NAEP reading in public and
private schools at grade 8;
study to obtain NAEP grade 12 mathematics scores for students in the National
High School Longitudinal Study (HSLS) in public schools;
study to examine the relationship between NAEP grade 8 and grade 12
mathematics scales, conducted in public schools at grades 9, 10, and 11 in two
states that conducted PISA assessments in 2012; and
pilot tests in reading and mathematics in public and private schools at grades 4
and 8.
The samples for the operational assessments were organized into four distinct
components and sampled separately. The samples for the special studies and pilot
tests were integrated into these various components:
mathematics and reading assessments in public schools at grades 4 and 8;
mathematics and reading assessments in public schools at grade 12;
mathematics and reading assessments in private schools at grades 4, 8, and
12; and
computer-based TEL pilot assessment in public schools at grade 8.
The national assessments were designed to achieve nationally representative samples
of public and private school students in the fourth, eighth, and twelfth grades. Their
Appendix G NAEP 2019-2020
2
target populations included all students in public, private, Bureau of Indian Education
(BIE), and Department of Defense Education Activity (DoDEA) schools, who were
enrolled in grades 4, 8, and 12 at the time of assessment.
For the fourth- and eighth-grade mathematics and reading assessments in public
schools, the NAEP state student samples and assessments constituted the NAEP
national student samples and assessments. Nationally representative samples were
drawn for the remaining populations of private school students, DoDEA students, and
BIE students in the fourth and eighth grades.
The TUDA samples formed part of the corresponding state public school samples,
and the state samples formed the public school grades 4 and 8 part of the national
sample.
At grade 12, the national samples for mathematics and reading consisted of 13 state
samples of public schools and additional samples of public, private, BIE, and DoDEA
schools to represent the balance of the nation.
All samples except the TEL pilot sample were based on a two-stage sample design:
selection of schools within strata; and
selection of students within schools.
The computer-based TEL pilot sample was based on a three-stage sample design:
selection of primary sampling units (PSUs);
selection of schools within strata; and
selection of students within schools.
In the three-stage design for the TEL pilot sample, schools were stratified and
selected within the sampled PSUs. The sample of schools was selected with
probability proportional to a measure of size based on the estimated grade 8 student
enrollment.
The state assessments were designed to achieve representative samples of students
in the respective grade. At grades 4 and 8, the target populations included all
students in each participating jurisdiction, which included states, District of
Columbia, DoDEA, and school districts chosen for the TUDA assessments. At grade
12, the target population consisted of all students in each of the 13 participating
states. Each sample was designed to produce aggregate estimates with reliable
precision for all the participating jurisdictions, as well as estimates for various
student subpopulations of interest.
In the PISA linking study, samples of students in grades 9 through 11 were selected
from the schools selected for the grade 12 public school samples in Florida and
Massachusetts.
The figure below illustrates the various sample types and subjects.
Components of the NAEP samples, by assessment subject, grade, and
school type: 2013
Appendix G NAEP 2019-2020
3
SOURCE: U.S. Department of Education, Institute of Education Sciences, National
Center for Education Statistics, National Assessment of Educational Progress (NAEP),
2013 Assessments.
Appendix G NAEP 2019-2020
4
NAEP Technical Documentation Website
NAEP Technical Documentation Sample
Design for the 2013 State Assessment
Each assessment cycle, a sample of students in
designated grades within both public and private
schools throughout the United States is selected for
assessment. In state assessment years, of which 2013
is an example, the samples of public schools and their
students in each state are large enough to support
state-level estimates.
Target Population
Sampling Frame
Stratification of Schools
School Sample Selection
The NAEP 2013 state assessments covered fourth- and Substitute Schools
eighth-grade students in public schools for operational
mathematics and reading. A representative sample of
Ineligible Schools
students was drawn in each participating jurisdiction,
including the 50 states, the District of Columbia, Puerto
Student Sample Selection
Rico, Bureau of Indian Education (BIE) schools,
Department of Defense Education Activity (DoDEA)
School and Student
schools, and in school districts chosen for the Trial
Participation
Urban District Assessment (TUDA) study. The state
operational mathematics and reading assessments also
covered twelfth-grade students in public schools in 13 states for each subject. A
representative sample of public school students was drawn in each participating
jurisdiction.
All jurisdictions, including the TUDA districts, were included in the mathematics and
reading assessments at grades 4 and 8, with the exception of Puerto Rico, where a special
mathematics assessment was conducted instead of the operational mathematics and
reading assessments. Also, BIE was not designed as a reportable jurisdiction for the 2013
state assessments, but a nationally representative sample of students in BIE schools was
selected.
Generally for the state assessments, each non-TUDA jurisdiction sample is designed to
produce aggregate estimates with approximately equal precision for all the participating
jurisdictions, as well as estimates for various subpopulations of interest. The target
sample size for these jurisdictions is 3,150 for each operational subject. In 2013, the
samples for operational mathematics and reading at grades 4 and 8 were designed in this
fashion. At grades 4 and 8, the overall target student sample size for the operational
samples in each non-TUDA jurisdiction was 6,600—3,150 each for mathematics and
reading and 300 for pilot tests. For the mathematics assessment in Puerto Rico, the target
sample size was 5,800 at grades 4 and 8. At grade 12, the target sample sizes varied by
jurisdiction. Details can be found in the school sample selection.
The target population for the NAEP 2013 state assessment included students in public
schools who were enrolled in grades 4, 8, and 12 at the time of assessment. The
sampling frame included public schools having the relevant grade in each jurisdiction. The
samples were selected based on a two-stage sample design:
selection of schools within participating jurisdictions; and
Appendix G NAEP 2019-2020
5
selection of students within schools.
From the stratified frame of public schools for each grade within each jurisdiction, a
systematic random sample of grade-eligible schools was drawn with probability
proportional to a measure of size based on the estimated grade-specific enrollment of the
school.
For the TUDA study, schools were sampled from the 21 participating TUDA districts at the
same time schools were selected for the jurisdiction samples. The TUDA districts are listed
below:
Albuquerque Public Schools, New Mexico;
Atlanta Public Schools, Georgia;
Austin Independent School District, Texas;
Baltimore City Public Schools, Maryland;
Boston Public Schools, Massachusetts;
Charlotte-Mecklenburg Schools, North Carolina;
Chicago Public Schools, Illinois;
Cleveland Metropolitan School District, Ohio;
Dallas Independent School District, Texas;
Detroit Public Schools, Michigan;
District of Columbia Public Schools, District of Columbia;
Fresno Unified School District, California;
Hillsborough County Public Schools, Florida;
Houston Independent School District, Texas;
Jefferson County Public Schools (Louisville), Kentucky;
Los Angeles Unified School District, California;
Miami-Dade County Public Schools, Florida;
Milwaukee Public Schools, Wisconsin;
New York City Department of Education, New York;
School District of Philadelphia, Pennsylvania; and
San Diego Unified School District, California.
These subsamples affected the design of the state samples in those states where TUDA
districts were oversampled. In each of these states, there were distinct sampling rates for
each TUDA district and for the balance of the state (i.e., the rest of the state not in a
TUDA district).
Each selected school provided a list of eligible enrolled students from which a systematic
sample of students was drawn. In fourth- and eighth-grade schools, 63 students, if
possible, were selected from each school: 30 for mathematics, 30 for reading, and 3 for
the pilot tests. In twelfth-grade schools, 60 students, if possible, were selected from each
school: 30 for mathematics and 30 for reading. Details can be found in the student
sample selection.
Appendix G NAEP 2019-2020
6
NAEP Technical Documentation Website
NAEP Technical Documentation Target
Population for the 2013 State
Assessment
The target population for the 2013 state assessment included all students in public schools
in the United States who were enrolled in fourth or eighth grade and, for 13 states,
students enrolled in twelfth grade. In addition, students enrolled in fourth and eighth grades
attending Bureau of Indian Education (BIE) schools, Department of Defense Education
Activity (DoDEA) schools, and public schools in Puerto Rico were included. BIE was not
designed as a reportable jurisdiction for the 2013 state assessments, but a nationally
representative sample of students in BIE schools was selected.
Appendix G NAEP 2019-2020
7
NAEP Technical Documentation Website
NAEP Technical
Documentation Sampling Frame for the
2013 State Assessment
Drawing the school samples for the 2013 assessments
Fourth- and Eighth-Grade
required a comprehensive list of public schools in each
Schools and Enrollment in
jurisdiction containing information for stratification
Public School Sampling Frame
purposes. As in previous NAEP assessments,
the Common Core of Data (CCD) file developed by NCES
Twelfth-Grade Schools and
was used to construct the sampling frame. The CCD file
Enrollment Public School
corresponding to the 2009-2010 school year provided
Sampling Frame in 13 States
the frame for all regular and state-operated public,
Bureau of Indian Education (BIE), Department of
New-School Sampling Frame
Defense Education Activity (DoDEA) schools, and
schools in Puerto Rico.
The sampling frame excluded ungraded schools, vocational schools with no enrollment,
special education-only schools, prison and hospital schools, virtual or online schools, homeschool entities, and juvenile correctional institutions.
For quality control purposes, school and student counts from the NAEP 2013 sampling
frame were compared to school and student counts from the previous frame (2012). No
revisions to the frame were needed as a result of this check.
Appendix G NAEP 2019-2020
8
NAEP Technical Documentation Website
NAEP Technical Documentation Fourth- and Eighth-Grade
Schools and Enrollment in the 2013 Public School
Sampling Frame
The following table displays, by jurisdiction, the number of fourth- and eighth-grade public schools and their estimated
enrollment, as contained in the Common Core of Data (CCD) sampling frame. Grade-specific enrollment was estimated for each
school as the average grade enrollment for grades 1 through 8.
Number of schools and enrollment in public school sampling frame, state assessment, by grade and
jurisdiction: 2013
Jurisdiction
Total
Alabama
Alaska
Arizona
Arkansas
California–Fresno
California–Los Angeles
California–San Diego
California–Balance
Colorado
Connecticut
Delaware
Florida–Hillsborough County
Florida–Miami
Florida–Balance
Georgia–Atlanta
Georgia–Balance
Hawaii
Idaho
Illinois–Chicago
Illinois–Balance
Indiana
Iowa
Kansas
Kentucky–Jefferson County
Kentucky–Balance
Louisiana
Maine
Maryland–Baltimore
Maryland–Balance
Massachusetts–Boston
Massachusetts–Balance
Michigan–Detroit
Appendix G NAEP 2019-2020
Grade 4
Schools
Enrollment
52,652
3,755,038
748
59,269
365
9,827
1,179
83,555
496
37,017
70
5,823
516
51,516
138
10,040
5,122
396,594
1,013
63,266
604
41,489
115
9,647
164
15,161
274
26,903
1,659
160,638
61
4,264
1,181
125,981
202
14,155
368
21,351
488
29,942
1,855
124,553
1,101
80,245
674
35,421
729
35,907
96
7,617
635
44,068
789
55,300
341
13,945
124
6,292
763
55,578
79
4,106
894
67,192
121
7,584
Grade 8
Schools
Enrollment
28,515
3,664,355
484
57,283
285
9,701
769
81,283
315
35,831
26
5,457
162
46,818
63
9,691
2,567
394,159
520
59,357
306
42,667
60
9,398
80
15,020
160
25,573
910
156,199
26
3,369
522
120,938
80
12,843
200
20,466
470
29,469
1,133
124,839
485
79,856
387
34,991
412
34,645
42
7,030
349
42,373
531
50,584
211
14,205
93
5,501
263
55,486
38
3,808
444
68,154
74
5,001
9
Michigan–Balance
1,735
Minnesota
952
Mississippi
436
Missouri
1,170
Montana
400
Nebraska
568
Nevada
379
265
New Hampshire
New Jersey
1,366
New Mexico–Albuquerque
99
New Mexico–Balance
332
New York–New York City
709
New York–Balance
1,659
North Carolina–Charlotte
105
North Carolina–Balance
1,309
North Dakota
260
Ohio–Cleveland
83
Ohio–Balance
1,794
Oklahoma
895
Oregon
767
Pennsylvania–Philadelphia
177
Pennsylvania–Balance
1,565
Rhode Island
170
South Carolina
618
South Dakota
328
Tennessee
998
Texas–Austin
80
Texas–Dallas
147
Texas–Houston
179
Texas–Balance
3,871
Utah
582
Vermont
224
Virginia
1,137
Washington
1,214
West Virginia
425
Wisconsin–Milwaukee
115
Wisconsin–Balance
996
Wyoming
188
Other jurisdictions
Bureau of Indian Education (BIE)
135
Department of Defense Education Activity (DoDEA)
108
District of Columbia (TUDA)
87
District of Columbia–Balance
44
Puerto Rico
1,017
SOURCE: U.S. Department of Education, Institute of Education Sciences, National
National Assessment of Educational Progress (NAEP), 2013 State Assessment.
110,658
61,086
38,958
68,189
10,863
22,085
33,851
14,495
100,453
7,594
18,137
63,731
128,564
11,245
106,884
6,995
3,573
129,785
49,300
42,827
12,098
117,452
10,437
55,228
9,380
75,934
6,862
12,932
16,525
337,085
46,508
6,419
93,610
77,826
20,875
5,807
55,190
6,849
968
695
290
727
289
339
158
137
741
42
161
458
877
38
656
188
81
1,009
593
414
142
772
57
298
254
565
24
38
64
2,039
232
122
388
606
202
89
544
93
115,376
61,246
36,999
68,007
10,912
21,503
33,028
15,191
99,535
6,903
17,280
61,278
131,957
9,852
101,491
7,330
3,550
129,136
46,433
42,949
10,970
121,829
10,842
52,433
9,306
71,570
5,360
10,113
12,738
320,696
42,593
6,364
92,179
77,099
20,637
5,371
55,692
6,568
3,246
109
7,507
63
3,369
37
1,538
42
38,842
407
Center for Education Statistics,
Appendix G NAEP 2019-2020
10
2,785
5,589
2,357
1,950
37,363
NAEP Technical Documentation Website
NAEP Technical Documentation Twelfth-Grade Schools
and Enrollment in the 2013 Public School Sampling
Frame in 13 States
The following table presents the number of schools and estimated enrollment for the twelfth-grade CCD frame for the 13 state
assessments.
Number of schools and enrollment in public school sampling frame, grade 12
state assessment, by jurisdiction: 2013
Jurisdiction
1
Total
Arkansas
Connecticut
Florida
Idaho
Illinois
Iowa
Massachusetts
Michigan
New Hampshire
New Jersey
South Dakota
Tennessee
West Virginia
Schools
5,710
Enrollment
859,758
297
245
965
210
954
408
371
1,032
89
432
191
369
147
32,035
41,607
176,821
19,057
149,998
37,793
67,923
126,382
15,749
97,690
8,796
67,111
18,796
1 The
aggregate of the 13 states participating in the state assessments at grade 12.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center
for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 State
Assessment.
Appendix G NAEP 2019-2020
11
NAEP Technical Documentation Website
NAEP Technical Documentation New-School
Sampling Frame for the 2013 State
Assessment
The Common Core of Data (CCD) file used for the frame corresponds to the 2009-2010 school year,
whereas the assessment year is the 2012-2013 school year. During this 3-year period, some
schools closed, some changed structure (one school becoming two schools, for example), and others
came into existence.
As was done in previous years, to achieve as close to full coverage as possible, the school frame was
supplemented by a sample of new schools obtained from a sample of districts. Each sampled district
was sent a list of the CCD schools and asked to add in any new schools or old schools that had
become newly eligible for grades 4, 8, and 12.
Since asking every school district to list new and newly-eligible schools would have generated too
much of a burden, a sample of districts was contacted to obtain a list of new schools. To represent
the unsampled districts in the full sample of schools, weights for schools included in the new-school
sample were adjusted to reflect the district selection probability.
The goal was to allow every new school a chance of selection, thereby fully covering the target
population of schools in operation during the 2012-2013 school year. The first step in this process
was the development of a new-school frame through the construction of a district-level file from the
CCD school-level file. To develop the frame, the district-level file was divided into two files: one
for small districts and a second for medium and large districts.
Small districts contained no more than three schools on the frame in total, with no more than one
school at each targeted grade (4, 8, and 12). New schools in small districts were identified during
school recruitment and added to the sample if the old school was sampled. From a sampling
perspective, the new school was viewed as an “annex” to the sampled school that had a well-defined
probability of selection equal to that of the old school. The “frame” in this case was, in fact, the
original frame; when the old school was sampled in a small district, the new school was automatically
sampled as well.
The remaining districts were defined as medium and large districts. In these districts, a frame of new
schools was developed based on information provided by the district. To limit the required effort, the
new-school frame was created through developing information on a sample of medium and large
public school districts in each jurisdiction.
Prior to district sampling, specific districts were in sample with certainty. They included the following
districts:
districts in jurisdictions where all schools were selected for sample at any of grades 4, 8 or 12;
state-operated districts;
districts in states with fewer than 10 districts;
charter-only districts (that is, districts containing no schools other than charter schools); and
Appendix G NAEP 2019-2020
12
TUDA districts.
The remaining districts in each jurisdiction (except the certainty jurisdictions) were separated into two
strata of large- and medium-size districts. These strata were defined by computing an aggregate
percentage of enrollment for each district within the state (removing districts in the certainty strata
defined above) and sorting in descending order by percentage of jurisdiction enrollment represented
by the district. All districts up to and including the first district at or above the 80th cumulative
percentage were defined as large districts. The remaining districts were defined as medium districts.
An example is given below. A state's districts are ordered by percentage enrollment. The first six
become large districts and the last six become medium districts.
Large and medium districts example, state assessment, by enrollment, stratum, and
district: 2013
District
Percentage enrollment
Cumulative percentage enrollment
Stratum
1
20
20
L
2
20
40
L
3
15
55
L
4
10
65
L
5
10
75
L
6
10
85
L
7
5
90
M
8
2
92
M
9
2
94
M
10
2
96
M
11
2
98
M
12
2
100
M
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for
Education Statistics, National Assessment of Educational Progress (NAEP), 2013 State
Assessment.
The target sample size for each jurisdiction was 10 districts. Where possible, eight large and two
medium districts were selected. However, in the example above, since there are only six large
districts, all of the large districts and four of the medium districts were selected for the new-school
inquiry.
If sampling was needed in the medium stratum, the medium districts were selected with equal
probability. If sampling was needed in the large stratum, the large districts were sampled with
probability proportional to enrollment. These probabilities were retained and used in later stages of
sampling and weighting, as the district probability then represented the number of other districts that
were not sampled to be surveyed for new schools.
The selected districts in each jurisdiction were then sent a listing of all their schools that appeared on
the 2009-2010 CCD file and were asked to provide information about the new schools not included in
the file and grade span changes of existing schools. These listings provided by the selected districts
were used as sampling frames for selection of new public schools and updates of existing schools.
This process was conducted through the NAEP State Coordinator in each jurisdiction. The
coordinators were sent the information for all sampled districts in their respective states and were
responsible for returning the completed updates.
Appendix G NAEP 2019-2020
13
The eligibility of a school was determined based on the grade span. A school also was classified as
“newly-eligible” if a change of grade span had occurred such that the school status changed from
ineligible to eligible in a particular grade.
Appendix G NAEP 2019-2020
14
NAEP Technical Documentation Website
NAEP Technical
Documentation Stratification of Schools for
the 2013 State Assessment
The purpose of school stratification is to increase the efficiency and
Stratification
ensure the representativeness of the school samples in terms of
Variables
important school-level characteristics, such as geography (e.g., states
and TUDA districts), urbanicity, and race/ethnicity classification. NAEP
school sampling utilizes two types of stratification: explicit and implicit.
Explicit stratification partitions the sampling frame into mutually exclusive groupings called strata.
The systematic samples selected from these strata are independent, meaning that each is
selected with its own unique random start. The explicit school strata for the 2013 NAEP state
assessments were usually states. If a state contained Trial Urban District Assessment (TUDA)
districts, the explicit strata were each individual TUDA district and the balance of the state. In
2013, there were 21 participating TUDA districts in the NAEP state assessment program. They are
listed below:
Albuquerque Public Schools, New Mexico;
Atlanta Public Schools, Georgia;
Austin Independent School District, Texas;
Baltimore City Public Schools, Maryland;
Boston Public Schools, Massachusetts;
Charlotte-Mecklenburg Schools, North Carolina;
Chicago Public Schools, Illinois;
Cleveland Metropolitan School District, Ohio;
Dallas Independent School District, Texas;
Detroit Public Schools, Michigan;
District of Columbia Public Schools, District of Columbia;
Fresno Unified School District, California;
Hillsborough County Public Schools, Florida;
Houston Independent School District, Texas;
Jefferson County Public Schools (Louisville), Kentucky;
Los Angeles Unified School District, California;
Miami-Dade County Public Schools, Florida;
Milwaukee Public Schools, Wisconsin;
New York City Department of Education, New York;
School District of Philadelphia, Pennsylvania; and
San Diego Unified School District, California.
Implicit stratification involves sorting the sampling frame, as opposed to grouping the frame. For
NAEP, schools are sorted by key school characteristics within explicit strata and sampled
systematically using this ordering. This type of stratification ensures the representativeness of the
school samples with respect to the key school characteristics. The implicit school stratification
variables for the 2013 state assessments included urbanicity, race/ethnicity classification, and
achievement score/median income. Further details about these variables can be found here.
Appendix G NAEP 2019-2020
15
NAEP Technical Documentation Website
NAEP Technical
Documentation Stratification Variables for
the 2013 State Assessment
The implicit stratification of public schools for the NAEP
2013 state assessments involved three dimensions:
urbanicity classification (urban-centric locale);
race/ethnicity classification; and
achievement level or median income.
The urbanicity stratum is the top-level implicit
stratification variable and is assigned within each explicit
stratum. It is derived from the NCES urban-centric locale
variable and classifies schools based on location (city,
suburb, town, rural) and proximity to urbanized areas.
It has 12 possible values.
Stratification by Urbanicity
Classification
Stratification by Race/ethnicity
Classification
Stratification by Achievement
Data and Median Income
Missing Stratification Variables
The race/ethnicity stratum classifies schools by the relative magnitude of enrollment of nonHispanic White, non-Hispanic Black, Hispanic, Asian, American Indian/Alaska Native,
Hawaiian/Pacific Islander, and students classified as two or more races represented in schools.
The source of the race/ethnicity data is the Common Core of Data (CCD). The race/ethnicity
stratum is the second-level variable in the stratification hierarchy and is nested within the
urbanicity stratum.
The last stratification dimension is a classification of schools based on either achievement data
or median household income. For most states, it is based on achievement data. However, not all
states provide achievement data. In these cases, median household income is used instead.
Median income comes from the 2000 Census and it corresponds to the zip code area where the
school is located.
Missing values for stratification variables were imputed.
The implicit stratification in this three-fold hierarchical procedure was achieved via a "serpentine
sort" within a given explicit stratum. This sort was accomplished by alternating between
ascending and descending sort order on each variable successively through the sort hierarchy.
Within this sorted list the schools were arranged in serpentine order by achievement data (or
median household income) within each cell determined by the two higher stratification variables
(urbanicity and race/ethnicity classifications), with ascending order for achievement data/median
household income used in every other cell, and descending order for achievement data/median
household income used in the remaining cells, giving an ascending-descending-ascendingdescending pattern. Schools in these urbanicity and race/ethnicity classification cells were also
sorted in serpentine order. Within each urbanicity and race/ethnicity classification cells, schools
were sorted in ascending order within one urbanicity stratum, by descending order within the
next urbanicity stratum, and so on. The following table shows an oversimplified example to
illustrate the ascending-descending-ascending-descending pattern of the serpentine sort.
Appendix G NAEP 2019-2020
16
Stratification variables sorted by serpentine sort: 2013
TUDA
Yes
Urbanicity
Large City
Race/ethnicity level
High minority
Achievement score
20
22
27
30
Low minority
29
26
20
18
Mid-size City
Low minority
15
25
27
31
High minority
35
32
30
28
No
Mid-size City
High minority
20
22
27
30
Low minority
29
26
20
18
Large City
Low minority
15
25
27
31
High minority
35
32
30
28
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center
for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 State
Assessment.
Appendix G NAEP 2019-2020
17
NAEP Technical Documentation Website
NAEP Technical Documentation Stratification by
Urbanization Classification for the 2013 State
Assessment
The creation of the urbanicity classification variable was based on the NCES urban-centric locale and was defined
within each explicit stratum. The NCES urban-centric locale contains the following categories:
1. Large City: Territory inside an urbanized area and inside a principal city with population of 250,000 or more;
2. Mid-size City: Territory inside an urbanized area and inside a principal city with population less than
250,000 and greater than or equal to 100,000;
3. Small City: Territory inside an urbanized area and inside a principal city with population less than 100,000;
4. Large Suburb: Territory outside a principal city and inside an urbanized area with population of 250,000 or
more;
5. Mid-size Suburb: Territory outside a principal city and inside an urbanized area with population less
than 250,000 and greater than or equal to 100,000;
6. Small Suburb: Territory outside a principal city and inside an urbanized area with population less than
100,000;
7. Fringe Town: Territory inside an urban cluster that is less than or equal to 10 miles from an urbanized area;
8. Distant Town: Territory inside an urban cluster that is more than 10 miles and less than or equal to
35 miles from an urbanized area.
9. Remote Town: Territory inside an urban cluster that is more than 35 miles of an urbanized area;
10. Fringe Rural: Census-defined rural territory that is less than or equal to 5 miles from an urbanized area, as
well as rural territory that is less than or equal to 2.5 miles from an urban cluster;
11. Distant Rural: Census-defined rural territory that is more than 5 miles but less than or equal to 25
miles from an urbanized area, as well as rural territory that is more than 2.5 miles but less than or equal to
10 miles from an urban cluster;
12. Remote Rural: Census-defined rural territory that is more than 25 miles from an urbanized area and is also
more than 10 miles from an urban cluster; and
13. Outside of the United States: Department of Defense Education Activity (DoDEA) overseas schools or Puerto
Rico.
For the definitions of the geographic terms used in these descriptions, please refer to the Census Bureau’s
website (for example, www.census.gov/population/www/metroareas/aboutmetro.html)
The urbanicity classification cells were created by starting with the original NCES urban-centric locale
categories. Urbanicity strata were collapsed with neighboring strata until a minimum cell size criterion, in terms
of the percentage of students, was met. The minimum cell size criterion varied by type of explicit stratum. The
criterion for explicit strata comprising the largest TUDA districts (Los Angeles, New York City, Chicago, Miami, and
Houston) was 12 percent; for the other TUDA districts, it was 18 percent; and for all other explicit strata, it was
9 percent.
The urbanicity classification variable was equal to the original NCES urban-centric locale if no collapsing was
necessary. If collapsing was necessary, the collapsing scheme first collapsed within the four major strata (city,
suburbs, town, rural). For example, urbanicity categories 1, 2, and 3 within city were collapsed (1 with 2, 2 with
3) if cells 1 or 3 were deficient. If the middle cell (e.g., 2) was deficient, then it was collapsed with the smaller of
the two end cells. If a collapsed pair was still deficient, it was collapsed with the remaining unit within the major
stratum. That is, a single city cell would be created by collapsing the large city, mid-size city, and small city cells.
If a cell was still deficient after collapsing within major stratum, further collapsing across major strata occurred
as needed until the deficiency was resolved. The values of the urbanicity classification variable were set equal to
Appendix G NAEP 2019-2020
18
the cell value of the final level of collapsing.
Prior experience with this type of stratification has shown that the greatest efficiency of stratification results when
cities and suburb fringe areas are always kept separate from towns and rural areas, even if the enrollment
criterion is violated.
Appendix G NAEP 2019-2020
19
NAEP Technical Documentation Website
NAEP Technical Documentation Stratification by
Race/Ethnicity Classification for the 2013 State
Assessment
Race/ethnicity classification was based on the second and third largest race/ethnicity percentages (among nonHispanic White, non-Hispanic Black, Hispanic, Asian, American Indian/Alaska Native, Hawaiian/Pacific Islander, and
students classified as two or more races) within each urbanicity classification stratum. The race/ethnicity strata
were formed using one of three classification schemes as follows:
Case 1: Urbanicity cells where both the second and third largest race/ethnicity groups contained less than 7
percent of students in the urbanicity cell were not stratified by race/ethnicity enrollment (race/ethnicity
stratification value was set to 0). There were no race/ethnicity strata formed within these urbanicity cells.
Case 2: Urbanicity cells where the second largest race/ethnicity group contained at least 7 percent but no more
than 15 percent of students in the urbanicity cell were stratified into three race/ethnicity cells. Schools were
ordered by the sum of the percentage of race/ethnicity enrollment for the second and third largest groups within
the urbanicity cell and then divided into three approximately equal size groups in terms of students.
Case 3: Urbanicity cells where both the second and third largest race/ethnicity groups contained more than 15
percent of students in the urbanicity cell were stratified into four race/ethnicity cells. The second largest group
provided the primary stratification variable; the third largest group provided the secondary stratification variable.
Within an urbanicity cell, schools were first sorted based on the primary stratification variable. Then they were
divided into two strata of schools containing approximately equal numbers of students. Within each of these two
strata, the schools were sorted by the secondary stratification variable and subdivided into two substrata of
schools containing approximately equal numbers of students. The four race/ethnicity classifications consisted of the
following values; low primary variable/low secondary variable, low primary variable/high secondary variable, high
primary variable/low secondary variable, and high primary variable/high secondary variable.
Appendix G NAEP 2019-2020
20
NAEP Technical Documentation Website
NAEP Technical Documentation Stratification by
Achievement Data and Median Income for the 2013
State Assessment
Jurisdictions Using Achievement
The achievement data obtained from each jurisdiction are derived from the results
Data or Median Household
of state assessment programs. The contents of the achievement data files varied by Income in Stratification
jurisdiction and included achievement measures for a variety of subjects, grades,
and multiple assessment programs. One achievement measure was selected for
each responding jurisdiction to be used in the stratification process. Where available, the achievement data were used
for implicit stratification by grade. Since the achievement data are more current than the median household income data, as
well as more likely to be well-correlated to NAEP assessment scores, they were judged to be a more effective stratification
variable. The achievement measures were selected according to the following criteria:
Achievement measures from state assessments conducted in mathematics and reading (in that order of priority) were
utilized, if available. For grade 4, data from fourth-grade assessments were used, if available; otherwise, data from
third-grade assessments. For grade 8, data from eighth-grade assessments were used, if available; otherwise, data
from seventh-grade assessments. For both grades, data from 2009 assessments (the latest available) were used. For
grade 12, achievement measures were not available.
Achievement measures should match to at least 70 percent of the schools on the sampling frames.
Achievement measures should differentiate schools from one another. For example, district-level measures, those with
high missing rates or pass/fail indicators, were judged not to be useful for differentiating schools. In addition,
achievement measures that did not have good dispersion were not used for stratification.
All other things being equal, the possibilities for score types were average scale score, median scale score, percentile
rank, median percentile rank, normal curve equivalent, raw score, index score, and percentage above a particular cut
score or quartile. In general, the availability varied for any given state/grade/subject/year.
Achievement data useful for implicit stratification were obtained from 50 of 52 jurisdictions for both fourth- and eighthgrade assessments. Where achievement data were not used, median household income was used based on the zip code
area in which the school is located. The source of median household income is the 2000 Census.
Appendix G NAEP 2019-2020
21
NAEP Technical Documentation Website
NAEP Technical Documentation Jurisdictions Using Achievement
Data or Median Household Income in Stratification for the 2013
State Assessment
This table shows whether achievement data or median household income was used as a stratification variable for participating
jurisdictions. Neither achievement nor median income data was available for stratification of Bureau of Indian Education (BIE) and Department of
Defense Education Activity (DoDEA) schools. The estimated grade enrollment was used in these two jurisdictions.
Type of data, achievement or median household income, used for stratification, state assessment, by grade and
jurisdiction: 2013
Jurisdiction
Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Appendix G NAEP 2019-2020
Grade 4
Achievement
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
NO
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
Income
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
YES
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
Grade 8
Achievement
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
NO
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
YES
22
Income
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
YES
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
NO
Texas
YES
NO
YES
NO
Utah
YES
NO
YES
NO
Vermont
YES
NO
YES
NO
Virginia
YES
NO
YES
NO
Washington
YES
NO
YES
NO
West Virginia
YES
NO
YES
NO
Wisconsin
YES
NO
YES
NO
Wyoming
YES
NO
YES
NO
Other jurisdictions
Bureau of Indian Education (BIE)
—
—
—
—
Department of Defense Education Activity (DoDEA)
—
—
—
—
District of Columbia
YES
NO
YES
NO
Puerto Rico
NO
YES
NO
YES
— Not available.
NOTE: With the exception of the state of Nebraska, and the jurisdiction of Puerto Rico, in all other states and the District of Columbia
achievement data was used as a stratification variable for the 2013 state assessment.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of
Educational Progress (NAEP), 2013 State Assessment.
Appendix G NAEP 2019-2020
23
NAEP Technical Documentation Website
NAEP Technical Documentation Missing
Stratification Variables for the 2013 State
Assessment
Schools with missing stratification variables had their data imputed as follows:
Schools with missing estimated grade enrollment had their estimated grade enrollment set to 20.
Schools missing the urbanicity (urban-centric locale) variable were assigned the modal value of
urbanicity for schools in the same five-digit zip code or the same city. The mean ethnicity percentage
was imputed at the five-digit zip code level only if all schools were missing ethnicity at the district level,
and only at the three-digit zip code prefix if the five-digit zip code mean was missing as well.
Schools with missing or questionable values in race/ethnicity enrollment data—those in which the
summation of the ethnicity percentages did not fall in the range 97 through 103, indicating a gross
error—were assigned the average race/ethnicity enrollment within their school district, five-digit zip
code, or three-digit zip code prefix.
Schools with missing achievement data in jurisdictions and grades for which achievement data were
used in stratification were assigned the mean achievement data value within their urbanization and
race/ethnicity classification. The achievement data were imputed only for those schools in jurisdictions
and grades in which achievement data were used for stratification.
Schools missing median household income were assigned the mean value of median household income
for the three-digit zip code prefix in which they were located. In some cases, imputation was not
possible at the three-digit zip code level, and needed to be done at the city and state level.
Appendix G NAEP 2019-2020
24
NAEP Technical Documentation Website
NAEP Technical Documentation School
Sample Selection for the 2013 State
Assessment
Computation of Measures of
For the grades 4, 8, and 12 public school state assessment
Size
samples, schools were sampled independently from each
jurisdiction with probability proportional-to-size (PPS) using
School Sample Sizes: Frame
systematic sampling. Prior to sampling, schools in each
and New School
jurisdiction were sorted by the appropriate implicit
stratification variables (urbanicity status, race/ethnicity status,
Evaluation of the Samples
and achievement score or zip code-based median household
Using State Achievement Data
income) in a serpentine order. A school's measure of size was
a complex function of the school's estimated grade
enrollment. Schools whose measure of size was larger than
the sampling interval could be selected or “hit” multiple times. Schools with multiple hits were
selected with certainty and had larger student sample sizes.
The sampled schools for the public school state assessment samples came from two frames: the
public school sample frame (as constructed from the Common Core of Data (CCD)) and the newschool sampling frame.
Schools from the CCD-based frame were sampled at a rate that would yield specific target student
sample sizes for each jurisdiction. At grades 4 and 8, jurisdictions had a target sample size of
6,600 students - 3,150 students each for the reading and mathematics operational
assessments and 300 students for pilot tests. For the special mathematics assessment in Puerto
Rico, the target sample size was 5,800 students. By design, Bureau of Indian Education
(BIE) schools were not part of the state assessments this year. However, separate BIE school
samples were selected based on target student sample sizes that were large enough to ensure
that BIE schools were sufficiently represented in the national samples.
At grade 12, the target sample sizes of students differed by jurisdiction and are shown in the
following table. These numbers reflect the desired number of assessed students for the reading
and mathematics operational assessments (2,300 students per subject) and an upward
adjustment to offset expected rates of school and student attrition due to nonresponse and
ineligibility.
Target sample sizes of assessed students, grade 12 state assessment, by
jurisdiction: 2013
Jurisdiction
Arkansas
Connecticut
Florida
Idaho
Appendix G NAEP 2019-2020
Target student sample size
6,200
6,750
6,600
6,250
25
Illinois
7,250
Iowa
6,850
Massachusetts
6,250
Michigan
7,400
New Hampshire
8,350
New Jersey
6,500
South Dakota
6,500
Tennessee
7,400
West Virginia
6,650
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center
for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 State
Assessment.
Prior to selection, schools were deeply stratified in each jurisdiction to ensure that the school
sample distribution reflected the school population distribution as closely as possible, with regard
to the stratification variables, to miminimize sampling error. The success of this approach was
shown by comparing the proportion of minorities enrolled in schools (based on CCD values for
each school), median income, and urban-centric locale (viewed as an interval variable) reported
in the original frame against the school sample.
In addition, the distribution of state assessment achievement scores for the original frame can be
compared with that of the school sample for those jurisdictions for which state assessment
achievement data are available, as was done in the evaluation of the samples using state
achievement data. The number of significant differences found in this analysis was smaller than
what would be expected to occur by chance, given the large number of comparisons that were
made. The number of significant differences remained small even with the use of a finite
population correction factor in the calculation of the sampling variances. The close adherence of
sample values to frame values suggested there is little evidence that the school sample for NAEP
2013 is not representative of the frame from which it was selected. The achievement/median
income variable is used as the third-level sort order variable in the school systematic selection
procedure. While it may be a rather low-level sort variable, it still helps control how representative
the sampled schools are in terms of achievement. The close agreement between frame and
sample values of these achievement/median income variables provided assurance that the
selected sample is representative of the frame with respect to achievement status.
Appendix G NAEP 2019-2020
26
NAEP Technical Documentation Website
NAEP Technical Documentation Computation of
Measures of Size for the 2013 State
Assessment
In designing each school sample, five objectives underlie the process of determining the probability of
selection for each school and how many students are to be sampled from each selected school containing
the respective grade:
to meet the target student sample size for each grade;
to select an equal-probability sample of students;
to limit the number of students selected from any one school;
to ensure that the sample within a school does not include a very high percentage of the students
in the school, unless all students are included; and
to reduce the rate of sampling of small schools, in recognition of the greater cost and burden per
student of conducting assessments in such schools.
The goal in determining the school's measure of size is to optimize across the last four objectives in terms
of maintaining the accuracy of estimates and the cost effectiveness of the sample design. In certain
jurisdictions, a census of students was taken so as to meet, as nearly as possible, the target student
sample size. Elsewhere, to meet the target student sample and achieve a reasonable compromise among
the other four objectives above, the following algorithm was used to assign a measure of size to each
school based on its enrollment per grade as indicated on the sampling frame.
The preliminary measures of size ( MOSjs ) were set as follows:
where x js is the estimated grade enrollment for school s in jurisdiction j, y j the target within-school student
sample size for jurisdiction j, and z js the within-school take-all student cutoff for jurisdiction j to which
school s belongs.
For grades 4 and 8, the target sample size and take-all cutoff were 63 students and 70 students for all
jurisdictions, respectively, with the exception of Puerto Rico, where the target sample size was 50
students, and the take-all cutoff was 55 students. For grade 12, the target sample size and take-all cutoff
were 60 students and 66 students, respectively.
The preliminary measure of size reflects the need to lower the expected number of very small schools in
Appendix G NAEP 2019-2020
27
the sample, as the marginal cost for each assessed student in these schools is higher. These very small
schools are sampled at half the rate of the larger schools, and their weights are doubled to account for
the half sampling.
The next task in this development is to describe bj, the constant of proportionality for a specified
jurisdiction. It is a sampling parameter that, when multiplied by a school’s preliminary measure of size
( MOSjs ), yields the school’s final measure of size. It is computed in such a way that, when used with the
systematic sampling procedure, the target student sample size is achieved.
The final measure of size, Ejs, is defined as:
The quantity uj (the maximum number of “hits” allowed) in this formula is designed to put an upper
bound on the burden for the sampled schools. In most jurisdictions, uj was set to 3. In Alaska, uj was set
to 8, and in Puerto Rico, uj was set to 1.
In addition, new and newly-eligible schools were sampled from the new-school frame. The assigned
measures of size for these schools,
,
used the bj and uj values from the CCD-based school frame for the jurisdiction (i.e., the same sampling
rate as for the CCD-based school sample within each jurisdiction). The variable π djs is the probability of
selection of the district into the new-school district (d) sample.
Appendix G NAEP 2019-2020
28
NAEP Technical Documentation Website
NAEP Technical Documentation School Sample Sizes:
List Frame-Based and New School for the 2013 State
Assessment
The following table lists the number of sampled schools taken from the public school sampling frame (as constructed from
the Common Core of Data) and the new-school sampling frame, for both fourth and eighth grades, by participating
jurisdiction. The school counts shown are at the time of sampling. After school sampling, it was determined that in some
Trial Urban District Assessments (TUDAs) a few schools did not contribute to the TUDA's Adequate Yearly Progress (AYP).
These schools were then classified as out of scope for the TUDA but in scope for the state.
NAEP state frame-based and new public school samples, state assessment, by grade and jurisdiction: 2013
Jurisdiction
Total
Alabama
Alaska
Arizona
Arkansas
California–Fresno
California–Los Angeles
California–San Diego
California–Balance
Colorado
Connecticut
Delaware
Florida–Hillsborourgh County
Florida–Miami
Florida–Balance
Georgia–Atlanta
Georgia–Balance
Hawaii
Idaho
Illinois–Chicago
Illinois–Balance
Indiana
Iowa
Kansas
Kentucky–Jefferson County
Kentucky–Balance
Louisiana
Maine
Maryland–Baltimore
Appendix G NAEP 2019-2020
Total
school
sample
8,350
120
200
120
120
50
90
60
100
120
120
100
60
90
90
60
100
120
130
100
100
120
140
150
50
100
130
160
70
Grade 4
Frame
school
sample
8,170
120
200
120
120
50
80
60
100
120
120
100
60
80
90
60
100
120
130
100
100
120
140
140
50
100
120
160
70
New
school
sample
190
0
#
#
10
0
#
0
#
0
#
10
#
#
#
0
#
#
#
#
0
#
#
10
0
#
10
#
#
Total
school
sample
6,970
110
150
120
110
30
80
40
110
120
110
70
50
80
90
30
100
60
100
100
100
110
120
130
40
100
150
120
70
Grade 8
Frame
school
sample
6,760
110
140
120
110
30
80
40
100
120
110
60
50
80
90
30
100
60
100
100
100
110
120
130
40
100
120
120
60
New
school
sample
210
0
#
10
#
0
10
#
10
#
#
10
#
10
#
0
#
#
#
#
0
#
#
10
0
#
30
0
#
29
Maryland–Balance
100
Massachusetts–Boston
80
Massachusetts–Balance
110
Michigan–Detroit
80
Michigan–Balance
110
Minnesota
130
Mississippi
120
Missouri
130
Montana
200
Nebraska
170
Nevada
120
130
New Hampshire
New Jersey
120
New Mexico–Albuquerque
60
New Mexico–Balance
100
New York–New York City
80
New York–Balance
80
North Carolina–Charlotte
50
North Carolina–Balance
110
North Dakota
270
Ohio–Cleveland
90
Ohio–Balance
120
Oklahoma
140
Oregon
130
Pennsylvania–Philadelphia
60
Pennsylvania–Balance
110
Rhode Island
120
South Carolina
120
South Dakota
190
Tennessee
120
Texas–Austin
60
Texas–Dallas
60
Texas–Houston
80
Texas–Balance
110
Utah
120
Vermont
220
Virginia
110
Washington
120
West Virginia
150
Wisconsin–Milwaukee
70
Wisconsin–Balance
120
Wyoming
200
Other jurisdictions
Bureau of Indian Education
20
(BIE)
Department of Defense
120
Education Activity (DoDEA)
District of Columbia (TUDA)
90
District of Columbia–Balance
50
Puerto Rico
170
# Rounds to zero.
NOTE: Numbers of schools are rounded to nearest ten.
Appendix G NAEP 2019-2020
100
80
110
80
110
130
110
130
200
170
110
130
120
60
90
80
80
50
100
260
80
110
140
130
60
110
120
110
190
120
50
50
80
100
110
220
110
120
150
70
120
190
#
0
0
0
0
#
10
#
0
0
#
0
#
0
#
#
0
0
10
10
#
#
10
#
0
0
#
#
0
#
#
#
#
10
10
0
0
#
0
#
#
10
100
40
100
70
110
130
110
130
150
130
90
90
110
40
80
90
70
40
100
190
90
110
130
130
60
100
60
110
150
110
30
40
50
110
120
120
110
120
110
60
110
100
100
40
100
70
110
130
110
120
150
130
90
90
110
40
80
80
70
40
100
190
80
110
130
120
60
100
60
110
150
110
20
40
50
110
110
120
110
120
110
60
110
90
0
0
#
0
0
10
0
#
0
0
#
0
#
#
#
#
0
#
#
#
#
#
10
10
#
0
10
#
0
#
#
#
#
10
10
#
#
0
#
#
#
10
20
0
10
10
#
110
10
70
60
10
90
40
160
#
10
10
40
50
130
40
40
120
#
10
10
Detail may not sum to totals due to rounding.
30
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2013 State Assessment.
The following table lists the number of sampled schools taken from the public school sampling frame (as constructed from
the Common Core of Data) and the new-school sampling frame, for twelfth grade, by participating jurisdiction.
NAEP state frame-based and new public school samples, grade 12 state assessment, by jurisdiction: 2013
Jurisdiction
1
Total
Arkansas
Connecticut
Florida
Idaho
Illinois
Iowa
Massachusetts
Michigan
New Hampshire
New Jersey
South Dakota
Tennessee
West Virginia
Total school sample
1,460
Frame school sample
1,460
New school sample
10
100
110
120
100
130
120
110
140
80
110
140
130
90
100
110
120
100
130
120
110
140
80
110
140
130
90
0
0
#
0
0
0
#
#
0
#
0
0
0
1 The
aggregate of the 13 states participating in the state assessments at grade 12.
# Rounds to zero.
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2013 State Assessment.
Appendix G NAEP 2019-2020
31
NAEP Technical Documentation Website
NAEP Technical Documentation Evaluation of the
Samples for the 2013 State Assessment Using State
Achievement Data
The purpose of this analysis was to determine whether public schools selected for the 2013 samples were representative of
the schools on the NAEP sampling frames in terms of student achievement. Percentiles of the achievement distributions were
compared between the frame and sample schools for each public school jurisdiction in grades 4, 8, and 12.
Achievement Data
For grades 4 and 8, the achievement variable used in the analysis was the same variable used in the NAEP sample design to
stratify the public school frame. For most jurisdictions, the variable was an achievement score provided by the jurisdiction.
However, for some jurisdictions where achievement data were not available, median household income from the 2000
Census was used. (In 2000, the Census determined median household income based on the five-digit zip code area in which
the school was located.) The achievement data consisted of various types of school-specific achievement measures from
state assessment programs. The type of achievement data available varied by jurisdiction. For instance, in some states, the
measure was the average score for a given state assessment. In other states, the measure was a percentile rank or
percentage of students above a specific score. For grade 12, since achievement data was not available, median household
income based on zip code area from the 2000 Census was used.
During frame development, not every record on the Common Core of Data (CCD) file matched to the achievement data files
created for the National Center for Education Statistics (NCES), even in jurisdictions where those data were generally
available. For schools that did not match, their achievement score was imputed by a mean matching imputation approach
using the mean achievement score for schools with complete achievement data within the same jurisdiction-urbanicityrace/ethnicity stratum combination.
Methodology
To determine whether the distributions between the frame and sample schools were different, comparisons of percentile
estimates were made for the 10th, 25th, 50th, 75th, and 90th percentile levels as well as the mean for each public school
jurisdiction by grade. Frame and sample school estimates were considered statistically different if the frame value fell
outside the 95 percent confidence interval of the corresponding sample estimate. The percentile values for the frame schools
were calculated by weighting each school by the estimated number of students in the given grade. The percentile estimates
for the sample schools were calculated using school weights and weighted by the school measure of size (estimated number
of students in the given grade). The 95 percent confidence intervals for the school sample estimates were calculated in
WesVar—software for computing estimates of sampling variance from complex sample survey (Westat, 2000b —using the
Woodruff method (Sarndal, Swensson, and Wretman 1992) with the use of a finite population correction factor.
Results
As mentioned above, sample and frame achievement distributions were determined to be different if at least one of the
percentile estimates or the mean differed significantly at the 95 percent confidence level. Out of all the jurisdiction and grade
comparisons (excluding jurisdictions where all schools in the frame were selected), only 14 of the 810 distributions compared
were found to be significantly different. They are shown in the table below
Summary of significant differences in achievement measures between the sample and the frame, state
assessment, by jurisdiction and grade: 2013
Grade
4
Jurisdiction
Delaware
Fresno TUDA
Achievement
data / median
income
Achievement
data
Achievement
Appendix G NAEP 2019-2020
Estimate
75th
percentile
25th
Frame
87.23
Sample
86.47
Confidence interval
(86.38, 87.20)
46.52
46.42
(46.36, 46.51)
32
8
San Diego
TUDA
Maine
New Mexico
New Mexico
South Dakota
South Dakota
Detroit TUDA
Detroit TUDA
Hillsborough
TUDA
Houston TUDA
12
Illinois
data
Achievement
data
Achievement
data
Achievement
data
Achievement
data
Achievement
data
Achievement
data
Achievement
data
Achievement
data
Achievement
data
Achievement
data
Median income
percentile
mean
25th
percentile
50th
percentile
mean
25th
percentile
90th
percentile
75th
percentile
90th
percentile
75th
percentile
mean
66.19
66.97
(66.26, 67.69)
45.94
46.28
(46.16, 46.38)
39.98
37.58
(37.19, 39.85)
42.15
41.21
(40.47, 41.95)
68.86
69.00
(68.87, 70.00)
90.93
89.53
(89.21, 90.54)
62.68
61.45
(59.36, 62.48)
75.26
74.67
(74.15, 75.21)
76.35
76.46
(76.36, 76.58)
78.83
79.11
(78.91, 79.31)
10th
31,564.65
30,157.62
(28,203.36,
percentile
31,475.48)
Tennessee
Median income
90th
55,748.34
52,008.12
(51,304.59,
percentile
55,454.43)
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2013 State Assessment.
The number of significant differences found in this analysis was smaller than what would be expected to occur by chance,
given the large number of comparisons that were made. Also, the number of significant differences remained small even
with the added use of a finite population correction factor in the calculation of the sampling variances. Even in the
statistically significant cases, the close adherence of sample values to frame values suggests there is little evidence that the
school sample for NAEP 2013 is not representative of the frame from which it was selected. The achievement/median income
variable is used as the fourth-level sort order variable in the school systematic selection procedure. While it may be a rather
low level sort variable, it still helps control how representative the sampled schools are in terms of achievement. The close
agreement between frame and sample values of these achievement/median income variables provided assurance that the
selected sample is representative of the frame with respect to achievement or income status.
Appendix G NAEP 2019-2020
33
NAEP Technical Documentation Website
NAEP Technical
Documentation Substitute Schools for the
2013 State Assessment
As participation is effectively mandatory by law at fourth and eighth grades, substitute
schools for nonresponding schools were not provided. However, participation was not
mandatory at twelfth grade, and substitute schools were used. Substitutes were preselected
for the twelfth-grade public school sample by sorting the school frame file according to the
actual order used in the sampling process (the implicit stratification). Each sampled school
had each of its nearest neighbors on the school frame file selected as a potential substitute.
The last sort ordering was by grade enrollment. The result was that the nearest neighbors
had grade enrollment values very close to that of the sampled school. To be eligible as a
potential substitute, the neighbor needed to be a nonsampled school (for any grade). The
school also needed to be in the same implicit stratum as the sampled school. If both nearest
neighbors were eligible to be substitutes, the one with a closer grade enrollment was
chosen.
Five substitutes participated in the twelfth-grade public school sample in the 13 states.
Appendix G NAEP 2019-2020
34
NAEP Technical Documentation Website
NAEP Technical Documentation Ineligible
Schools for the 2013 State Assessment
Eligible Schools Sampled
The Common Core of Data (CCD) public school file
from which most of the sampled schools were
Ineligible Sampled Schools by
drawn corresponds to the 2009-2010 school year,
Ineligibility Type
some 3 years prior to the assessment school year.
During the intervening period, some of these
schools either closed, no longer offered the grade of interest, or were ineligible for other
reasons. In such cases, the sampled school was coded as ineligible.
Appendix G NAEP 2019-2020
35
NAEP Technical Documentation Website
NAEP Technical Documentation Eligible
Schools Sampled for the 2013 State
Assessment
The following table shows the number of eligible fourth- and eighth-grade schools sampled for
each NAEP 2013 state assessment jurisdiction.
Eligible sampled schools, state assessment, by grade and jurisdiction: 2013
Jurisdiction
Total
Alabama
Alaska
Arizona
Arkansas
California–Fresno
California–Los Angeles
California–San Diego
California–Balance
Colorado
Connecticut
Delaware
Florida–Hillsborourgh County
Florida–Miami
Florida–Balance
Georgia–Atlanta
Georgia–Balance
Hawaii
Idaho
Illinois–Chicago
Illinois–Balance
Indiana
Iowa
Kansas
Kentucky–Jefferson County
Kentucky–Balance
Louisiana
Appendix G NAEP 2019-2020
Grade 4
Total
school
sample
8,350
120
200
120
120
50
90
60
100
120
120
100
60
90
90
60
100
120
130
100
100
120
140
150
50
100
130
Eligible
school
sample
7,860
110
180
120
120
50
80
60
100
120
110
90
60
80
90
50
100
120
120
90
90
110
130
140
50
100
110
Grade 8
Total
school
sample
6,970
110
150
120
110
30
80
40
110
120
110
70
50
80
90
30
100
60
100
100
100
110
120
130
40
100
150
Eligible
school
sample
6,440
100
120
110
110
20
80
40
100
110
110
50
50
70
80
20
100
60
100
90
90
100
110
120
30
100
120
36
Maine
Maryland–Baltimore
Maryland–Balance
Massachusetts–Boston
Massachusetts–Balance
Michigan–Detroit
Michigan–Balance
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico–Albuquerque
New Mexico–Balance
New York–New York City
New York–Balance
North Carolina–Charlotte
North Carolina–Balance
North Dakota
Ohio–Cleveland
Ohio–Balance
Oklahoma
Oregon
Pennsylvania–Philadelphia
Pennsylvania–Balance
Rhode Island
South Carolina
South Dakota
Tennessee
Texas–Austin
Texas–Dallas
Texas–Houston
Texas–Balance
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin–Milwaukee
Wisconsin–Balance
Wyoming
Other jurisdictions
Bureau of Indian Education
Appendix G NAEP 2019-2020
160
70
100
80
110
80
110
130
120
130
200
170
120
130
120
60
100
80
80
50
110
270
90
120
140
130
60
110
120
120
190
120
60
60
80
110
120
220
110
120
150
70
120
200
150
70
100
70
100
60
100
120
110
130
190
160
120
130
120
60
100
80
70
50
100
250
70
110
140
130
60
100
120
110
180
110
50
50
80
110
110
220
110
120
140
60
120
180
120
70
100
40
100
70
110
130
110
130
150
130
90
90
110
40
80
90
70
40
100
190
90
110
130
130
60
100
60
110
150
110
30
40
50
110
120
120
110
120
110
60
110
100
110
60
100
40
100
40
100
110
110
120
140
120
90
90
110
40
80
80
70
30
100
180
70
110
130
120
50
90
60
110
140
110
20
40
50
110
110
120
110
110
100
50
100
90
20
20
10
10
37
(BIE)
Department of Defense
120
100
70
60
Education Activity (DoDEA)
District of Columbia (TUDA)
90
80
40
30
District of Columbia–Balance
50
40
50
40
Puerto Rico
170
150
130
120
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to
rounding. "Balance" refers to the part of the state outside of the TUDA district(s).
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center
for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 State
Assessment.
The following table shows the number of eligible twelfth-grade schools sampled for each NAEP
2013 state assessment jurisdiction.
Eligible sampled schools, grade 12 state assessment, by jurisdiction: 2013
Jurisdiction
Total 1
Arkansas
Connecticut
Florida
Idaho
Illinois
Iowa
Massachusetts
Michigan
New Hampshire
New Jersey
South Dakota
Tennessee
West Virginia
Total school sample
1,460
Eligible school sample
1,390
100
110
120
100
130
120
110
140
80
110
140
130
90
100
100
110
90
120
120
110
130
80
110
130
120
90
1 The
aggregate of the 13 states participating in the state assessments at grade 12.
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to
rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center
for Education Statistics, National Assessment of Educational Progress (NAEP), 2013 State
Assessment.
Appendix G NAEP 2019-2020
38
NAEP Technical Documentation Website
NAEP Technical Documentation Ineligible Sampled
Schools by Ineligibility Type for the 2013 State
Assessment
The following table shows the unweighted counts and percentages of NAEP 2013 state assessment fourth- and
eighth-grade schools that were eligible and ineligible, by reason for ineligibility.
School eligibility status, state assessment, by grade and eligibility status: 2013
Grade 4
Grade 8
Unweighted
Unweighted
Unweighted
Unweighted
count of schools
percentage
count of schools
percentage
Eligibility status
All sampled public schools
8,350
100.00
6,970
100.00
Eligible
7,860
94.13
6,400
92.40
No eligible students in grade
56
0.67
48
0.69
Does not have sampled grade
109
1.31
144
2.07
School closed
259
3.10
186
2.67
Not a regular school
55
0.66
116
1.66
Other ineligible school
16
0.19
33
0.47
Duplicate on sampling frame
1
0.01
2
0.03
NOTE: Numbers of schools are rounded to nearest ten, except those pertaining to ineligible schools. Detail
may not sum to totals because of rounding. Percentages are based on rounded counts.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education
Statistics, National Assessment of Educational Progress (NAEP), 2013 State Assessment.
The following table shows the unweighted counts and percentages of NAEP 2013 state assessment twelfth-grade
schools that were eligible and ineligible, by reason for ineligibility.
School eligibility status, grade 12 state assessment, by eligibility status: 2013
Eligibility status
Total1
Eligible
No eligible students in grade
Does not have sampled grade
School closed
Not a regular school
Other ineligible school
Duplicate on sampling frame
Unweighted count of schools
1,460
Unweighted percentage
100.00
1,390
7
9
19
22
13
0
95.21
0.48
0.62
1.30
1.51
0.89
0.00
1 The
aggregate of the 13 states participating in the state assessments at grade 12.
NOTE: Numbers of schools are rounded to nearest ten, except those pertaining to ineligible schools. Detail
may not sum to totals because of rounding. Percentages are based on rounded counts.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education
Statistics, National Assessment of Educational Progress (NAEP), 2013 State Assessment.
Appendix G NAEP 2019-2020
39
NAEP Technical Documentation Website
NAEP Technical Documentation Student
Sample Selection for the 2013 State
Assessment
Within each sampled school, a sample of students was selected from a listing of the students in
the sampled grade such that every student had an equal chance of selection. The student lists
were submitted either electronically using a system known as E-filing or on paper. In Efiling, student lists are submitted in Excel files by either school coordinators or NAEP State
Coordinators. The files can be submitted for one school at a time (known as single school E-file
submission) or for an entire jurisdiction at once (known as multiple school E-file submission). Efiling allows schools to easily submit student demographic data electronically with the student lists,
easing the burden on NAEP field supervisors and school coordinators. Schools that are unable to
submit their student lists using the E-filing system provide hardcopy lists via the student listing
form to NAEP field supervisors. In 2013, there were 16,898 schools that E-filed their student lists,
while 548 lists were submitted using the student listing form.
In year-round, multi-track schools, students who were not scheduled to be in school on the
assessment day were removed from the student lists prior to sampling. Student base weights were
adjusted to account for these students.
The sampling process was the same, regardless of list submission type. The sampling process was
systematic(e.g., if the sampling rate was one-half, a random starting point of one or two was
chosen, and every other student on the list was selected). For E-filed schools only, where
demographic data was submitted for every student on the frame, students were sorted by gender
and race/ethnicity before the sample was selected to implicitly stratify the sample.
In the certainty jurisdictions, all students were sampled in all schools. Otherwise, the sample
size for grades 4 and 8 was 63 students (with the exception of 50 students in Puerto Rico), and
the sample size for grade 12 was 60 students. Larger schools may have been selected with
certainty in the sampling process and thus may have a larger sample size. In addition, most
fourth-grade schools chose the option of taking all students when enrollment was less than 90
students. This increased the fourth-grade sample size in many states beyond the designated
target.
Some students enrolled in the school after the sample was selected. In such cases, new enrollees
were sampled at the same rate as the students on the original list.
In fourth- and eighth-grade schools, sampled students were randomly assigned to mathematics,
reading, and pilot as follows: 30 students for mathematics, 30 students for reading, and 3
students for pilot. However, for schools in Puerto Rico, only the special mathematics assessment
was conducted. In twelfth-grade schools, sampled students were randomly assigned to
mathematics and reading as follows: 30 students for mathematics and 30 students for reading.
This was implemented by spiraling: the booklets assigned to sampled students were provided from
booklet packets that had, on average, the correct ratio of each of the relevant assessments in a
randomized order.
Some of the students who were English language learners (ELL) or students with disabilities
Appendix G NAEP 2019-2020
40
(SD) were excluded from the assessment because they could not be assessed with the
accommodations NAEP provides.
Appendix G NAEP 2019-2020
41
NAEP Technical Documentation Website
NAEP Technical Documentation School and
Student Participation in the 2013 State
Assessment
In all cases in the 2013 state assessment for grades 4
Weighted Response Rates of Fourth-Grade
and 8, the weighted response rates for schools in each
School Sample by Participating Jurisdiction
jurisdiction exceeded the 85 percent standard
established by the National Center for Education
Weighted Response Rates of Eighth-Grade
Statistics (NCES). As participation is effectively
School Sample by Participating Jurisdiction
mandatory, substitute schools for nonresponding
schools were not provided. Participation was not
Weighted Response Rates of Twelfth-Grade
School Sample by Participating State
mandatory at grade 12, and substitute schools were
used.
Weighted Student Response and Exclusion
In every NAEP survey, some of the sampled students
Rates, Mathematics Assessment
are not assessed for the following reasons:
Weighted Student Response and Exclusion
withdrawn students;
Rates, Reading Assessment
excluded students with disabilities (SD);
excluded English language learner (ELL)
students; or
students absent from both the original session and the make-up session (not excluded but not
assessed).
Withdrawn students are those who have left the school before the original assessment. Excluded
students were determined by their school to be unable to meaningfully take the NAEP assessment in
their assigned subject, even with an accommodation. Excluded students must also be classified as SD
and/or ELL. Other students who were absent for the initial session are assessed in the makeup session.
The last category includes students who were not excluded (i.e., “were to be assessed”) but were not
assessed either due to absence from both sessions or because of a refusal to participate. Assessed
students are also classified as assessed without an accommodation or assessed with an accommodation.
The latter group can be divided into SD students assessed with an accommodation, ELL students
assessed with an accommodation, or students who are both SD and ELL and accommodated. Note that
some SD and ELL students are assessed without accommodations, and students who are neither SD nor
ELL can only be assessed without an accommodation.
The weighted response rates utilize the student base weights and indicate the weighted percentage of
assessed students among all students to be assessed. The exclusion rates, in contrast, provide the
weighted percentage of excluded SD or ELL students among all absent, assessed, and excluded
students.
Appendix G NAEP 2019-2020
42
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted Response Rates of
Fourth-Grade School Sample by Participating Jurisdiction for the
2013 State Assessment
The following table presents unweighted counts and weighted response rates at grade 4 for sampled eligible and participating schools. States
with Trial Urban District Assessment (TUDA) districts are shown in multiple rows: for the TUDA district(s) and for the state as a whole (the TUDA
district[s] plus the rest of the state). The weighted school response rates estimate the proportion of the student population that is represented by
the participating school sample prior to substitution.
Participation is effectively mandatory for all states and districts, but not for Bureau of Indian Education (BIE) or Department of Defense Education
Activity (DoDEA) schools.
School counts and response rates of sampled eligible schools, grade 4 state assessment, by jurisdiction: 2013
Jurisdiction
Total
Alabama
Alaska
Arizona
Arkansas
California–Fresno
California–Los Angeles
California–San Diego
California
Colorado
Connecticut
Delaware
Florida–Hillsborough County
Florida–Miami
Florida
Georgia–Atlanta
Georgia
Hawaii
Idaho
Illinois–Chicago
Illinois
Indiana
Iowa
Kansas
Kentucky–Jefferson County
Kentucky
Louisiana
Maine
Maryland–Baltimore
Maryland
Massachusetts–Boston
Massachusetts
Michigan–Detroit
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Appendix G NAEP 2019-2020
Number of sampled eligible
schools
7,860
110
180
120
120
50
80
50
290
120
110
90
60
80
220
50
150
120
120
90
180
110
130
140
50
150
110
150
70
170
70
170
50
150
120
110
130
190
160
Number of participating
schools
7,830
110
170
120
120
50
80
50
290
120
110
90
60
80
220
50
150
120
120
90
180
110
130
140
50
150
110
150
70
170
70
170
50
150
120
110
130
190
160
Weighted school response rates
(percent)
99.67
100.00
99.48
100.00
100.00
100.00
100.00
100.00
99.17
100.00
97.22
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
97.98
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
99.85
100.00
43
Nevada
120
120
100.00
New Hampshire
130
130
100.00
New Jersey
120
120
100.00
New Mexico–Albuquerque
50
50
100.00
New Mexico
150
150
99.69
New York–New York City
80
80
100.00
New York
160
150
98.84
North Carolina–Charlotte
50
50
100.00
North Carolina
160
160
100.00
North Dakota
250
250
99.86
Ohio–Cleveland
70
70
100.00
Ohio
180
180
100.00
Oklahoma
140
140
100.00
Oregon
130
130
100.00
Pennsylvania–Philadelphia
60
60
100.00
Pennsylvania
160
160
100.00
Rhode Island
120
120
100.00
South Carolina
110
110
100.00
South Dakota
180
180
100.00
Tennessee
110
110
100.00
Texas–Austin
50
50
100.00
Texas–Dallas
50
50
100.00
Texas–Houston
80
80
100.00
Texas
290
290
100.00
Utah
110
110
99.08
Vermont
220
220
100.00
Virginia
110
110
100.00
Washington
120
120
99.09
West Virginia
140
140
100.00
Wisconsin–Milwaukee
60
60
100.00
Wisconsin
180
180
100.00
Wyoming
180
180
100.00
Other jurisdictions
Bureau of Indian Education (BIE)
20
10
80.19
Department of Defense Education
100
100
99.23
Activity (DoDEA)
District of Columbia (TUDA)
80
80
100.00
District of Columbia
120
120
100.00
Puerto Rico
150
150
100.00
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment
of Educational Progress (NAEP), 2013 State Assessment.
Appendix G NAEP 2019-2020
44
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted Response Rates of
Eighth-Grade School Sample by Participating Jurisdiction for the
2013 State Assessment
The following table presents unweighted counts and weighted response rates at grade 8 for sampled eligible and participating schools. States
with Trial Urban District Assessment (TUDA) districts are shown in multiple rows: for the TUDA district(s) and for the state as a whole (the TUDA
district[s] plus the rest of the state). The weighted school response rates estimate the proportion of the student population that is represented by
the participating school sample prior to substitution.
Participation is effectively mandatory for all states and districts, but not for Bureau of Indian Education (BIE) or Department of Defense Education
Activity (DoDEA) schools.
School counts and response rates of sampled eligible schools, grade 8 state assessment, by jurisdiction: 2013
Jurisdiction
Total
Alabama
Alaska
Arizona
Arkansas
California–Fresno
California–Los Angeles
California–San Diego
California
Colorado
Connecticut
Delaware
Florida–Hillsborourgh County
Florida–Miami
Florida
Georgia–Atlanta
Georgia
Hawaii
Idaho
Illinois–Chicago
Illinois
Indiana
Iowa
Kansas
Kentucky–Jefferson County
Kentucky
Louisiana
Maine
Maryland–Baltimore
Maryland
Massachusetts–Boston
Massachusetts
Michigan–Detroit
Michigan
Minnesota
Mississippi
Missouri
Montana
Appendix G NAEP 2019-2020
Number of sampled eligible
schools
6,440
100
120
110
110
20
70
30
230
110
110
50
50
70
200
20
120
60
100
90
180
100
110
120
30
130
120
110
50
160
40
140
40
150
110
110
120
140
Number of participating
schools
6,420
100
120
110
110
20
70
30
230
110
110
50
50
70
200
20
120
60
100
90
180
100
110
120
30
130
120
110
50
160
40
140
40
150
110
110
120
140
Weighted school response rates
(percent)
99.47
100.00
99.91
99.03
100.00
100.00
100.00
100.00
100.00
100.00
98.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
97.06
100.00
100.00
100.00
99.04
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
98.99
100.00
100.00
99.80
45
Nebraska
120
120
100.00
Nevada
90
90
100.00
New Hampshire
90
90
100.00
New Jersey
110
110
100.00
New Mexico–Albuquerque
30
30
100.00
New Mexico
120
120
99.68
New York–New York City
80
80
99.00
New York
160
150
93.08
North Carolina–Charlotte
30
30
100.00
North Carolina
130
130
100.00
North Dakota
180
180
99.92
Ohio–Cleveland
70
70
100.00
Ohio
170
170
100.00
Oklahoma
130
130
100.00
Oregon
120
120
100.00
Pennsylvania–Philadelphia
50
50
100.00
Pennsylvania
150
150
100.00
Rhode Island
60
60
100.00
South Carolina
110
110
100.00
South Dakota
140
140
100.00
Tennessee
110
110
100.00
Texas–Austin
20
20
100.00
Texas–Dallas
40
40
100.00
Texas–Houston
50
50
100.00
Texas
210
210
100.00
Utah
110
110
100.00
Vermont
120
120
100.00
Virginia
110
110
100.00
Washington
110
110
100.00
West Virginia
100
100
100.00
Wisconsin–Milwaukee
50
50
100.00
Wisconsin
150
150
100.00
Wyoming
90
90
100.00
Other jurisdictions
Bureau of Indian Education (BIE)
10
10
69.29
Department of Defense Education
60
60
99.40
Activity (DoDEA)
District of Columbia (TUDA)
30
30
100.00
District of Columbia
70
70
100.00
Puerto Rico
120
120
100.00
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment
of Educational Progress (NAEP), 2013 State Assessment.
Appendix G NAEP 2019-2020
46
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted Response Rates of
Twelfth-Grade School Sample by Participating State for the
2013 State Assessment
The following table presents unweighted counts and weighted response rates at grade 12 for sampled eligible and participating schools. The
weighted school response rates estimate the proportion of the student population that is represented by the participating school sample prior to
substitution.
School counts and response rates of sampled eligible schools, grade 12 state
assessment, by jurisdiction: 2013
Jurisdiction
Total 1
Arkansas
Connecticut
Florida
Idaho
Illinois
Iowa
Massachusetts
Michigan
New Hampshire
New Jersey
South Dakota
Tennessee
West Virginia
Number of sampled
eligible schools
1,390
Number of
participating schools
1,380
Weighted school response
rates (percent)
98.78
100
100
110
90
120
120
110
130
80
110
130
120
90
100
100
100
90
110
120
110
130
80
110
130
120
90
100.00
98.93
99.05
100.00
90.38
100.00
99.04
100.00
100.00
98.14
99.74
100.00
100.00
1 The
aggregate of the 13 states participating in the state assessments at grade 12.
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to
rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for
Education Statistics, National Assessment of Educational Progress (NAEP), 2013 State Assessment.
Appendix G NAEP 2019-2020
47
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted Student
Response and Exclusion Rates for the 2013 State
Mathematics Assessment
The following table presents the weighted student response and exclusion rates for the mathematics assessment. States with
Trial Urban District Assessment (TUDA) districts are shown in multiple rows: for the TUDA district(s) and for the state as a
whole (the TUDA district[s] plus the rest of the state). The weighted response rates utilize the student base weights and
indicate the weighted percentage of assessed students as a percentage of all students to be assessed. The exclusion rates give
the weighted percentage of excluded students, those with disabilities (SD) or students who were English language learners
(ELL), among all absent, assessed, and excluded students.
Weighted student response and exclusion rates, state mathematics assessment, by
grade and jurisdiction: 2013
Jurisdiction
Total
Alabama
Alaska
Arizona
Arkansas
California–Fresno
California–Los
Angeles
California–San
Diego
California
Colorado
Connecticut
Delaware
Florida–
Hillsborough
County
Florida–Miami
Florida
Georgia–Atlanta
Georgia
Hawaii
Idaho
Illinois–Chicago
Illinois
Indiana
Iowa
Kansas
Weighted
student
response
rates
(percent)
94.49
94.82
93.18
95.07
94.66
93.58
95.80
Grade 4
Weighted
percentage
of all
students
who were
SD and
excluded
1.25
1.03
0.98
0.88
1.16
0.90
1.57
Weighted
percentage
of all
students
who were
ELL and
excluded
0.46
0.10
0.22
0.34
0.10
0.22
1.07
95.18
1.11
94.79
92.34
93.85
94.36
95.74
95.07
94.11
95.42
94.18
94.70
95.24
94.85
94.40
95.18
95.16
94.79
Appendix G NAEP 2019-2020
Weighted
student
response
rates
(percent)
93.02
94.23
91.72
93.42
95.00
92.52
94.39
Grade 8
Weighted
percentage
of all
students
who were
SD and
excluded
1.28
0.91
1.01
0.98
1.80
1.65
1.14
Weighted
percentage
of all
students
who were
ELL and
excluded
0.40
0.13
0.23
0.32
0.24
0.79
0.82
0.74
92.60
1.89
1.00
1.54
1.04
1.19
1.98
1.11
1.20
0.35
0.22
0.22
0.10
93.59
93.47
92.44
90.65
93.78
1.20
1.05
1.81
1.03
1.35
0.70
0.23
0.34
0.32
0.13
0.93
1.25
0.80
1.34
0.81
1.00
0.71
0.72
1.31
0.53
1.43
1.66
0.76
0.19
0.15
0.53
0.44
0.58
0.39
0.21
0.20
0.31
92.63
91.06
91.57
93.38
90.26
94.15
94.80
94.48
92.49
93.74
93.94
0.96
1.14
0.72
1.30
0.97
1.03
0.77
0.79
1.58
0.73
1.57
1.29
0.64
0.00
0.25
0.88
0.11
0.65
0.25
0.05
0.04
0.10
48
Kentucky–
Jefferson County
Kentucky
Louisiana
Maine
Maryland–
Baltimore
Maryland
Massachusetts–
Boston
Massachusetts
Michigan–Detroit
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico–
Albuquerque
New Mexico
New York–New
York City
New York
North Carolina–
Charlotte
North Carolina
North Dakota
Ohio–Cleveland
Ohio
Oklahoma
Oregon
Pennsylvania–
Philadelphia
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas–Austin
Texas–Dallas
Texas–Houston
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin–
Milwaukee
Wisconsin
Wyoming
Appendix G NAEP 2019-2020
94.66
1.03
0.71
93.37
1.60
0.22
94.67
94.49
93.95
94.32
1.26
0.97
1.94
1.24
0.19
0.12
0.29
0.34
94.54
94.14
92.79
89.54
1.98
1.03
1.21
1.50
0.18
0.03
0.15
0.20
94.22
93.72
0.76
2.83
0.24
1.46
92.08
91.61
1.21
2.25
0.52
0.88
93.74
90.92
94.14
94.85
95.44
95.42
93.92
95.37
95.75
93.74
94.85
94.71
1.75
4.33
1.58
1.27
0.67
1.32
1.64
1.50
1.14
1.14
0.94
0.93
0.46
0.83
0.44
0.18
0.10
0.09
0.18
0.25
0.40
0.08
0.26
0.51
91.98
91.58
92.93
91.58
93.80
94.25
92.28
93.41
92.80
91.60
92.26
90.76
1.40
4.29
1.96
1.50
0.77
1.24
1.44
1.59
0.75
0.99
1.20
1.45
0.77
0.00
0.58
0.27
0.03
0.03
0.03
0.26
0.30
0.07
0.47
0.31
95.06
91.74
1.07
0.44
0.42
1.12
93.07
91.78
1.52
0.99
0.32
1.05
92.27
94.18
0.88
0.91
0.51
0.56
91.15
90.94
1.59
1.00
0.43
0.39
94.19
95.57
93.62
94.29
94.35
94.18
94.71
1.12
2.39
3.70
1.20
1.77
1.95
2.84
0.31
0.22
0.73
0.13
0.17
0.51
0.95
92.95
94.98
91.57
93.07
92.97
92.91
92.67
1.02
2.71
2.15
1.47
1.41
1.38
2.79
0.28
0.33
0.54
0.05
0.30
0.12
1.02
94.30
94.98
96.08
95.36
94.21
93.69
95.79
96.62
95.36
94.79
95.04
94.35
93.50
94.77
94.68
1.43
0.95
1.02
1.42
1.08
1.57
1.93
1.22
1.34
1.08
1.26
1.23
2.00
1.65
3.07
0.28
0.21
0.10
0.09
0.37
0.78
0.84
1.03
0.62
0.43
0.14
0.38
0.36
0.09
0.60
92.17
93.93
94.19
94.44
92.81
90.97
93.81
92.37
93.82
92.07
93.91
93.39
90.87
92.62
91.60
1.40
0.72
1.17
1.17
1.62
1.60
1.97
1.74
1.32
1.32
0.70
0.76
1.70
1.69
3.56
0.30
0.39
0.18
0.25
0.22
0.51
1.11
0.77
0.80
0.30
0.19
0.29
0.42
0.00
0.93
95.42
94.65
1.64
0.89
0.31
0.18
94.25
93.66
1.43
1.43
0.19
0.07
49
Other jurisdictions
Bureau of
Indian
Education (BIE)
Department of
Defense
Education
Activity
(DoDEA)
District of
Columbia
(TUDA)
District of
Columbia
Puerto Rico 1
93.34
1.42
0.00
92.02
3.28
0.00
95.05
1.17
0.74
94.47
0.82
0.49
95.52
1.27
0.85
90.15
0.75
1.23
95.09
0.86
0.62
91.26
0.41
0.71
94.47
0.00
0.24
92.75
0.01
0.02
1 In Puerto Rico, a special mathematics assessment was conducted instead of the operational
mathematics assessment.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for
Education Statistics, National Assessment of Educational Progress (NAEP), 2013 State Mathematics
Assessment.
Similarly, the following table presents the weighted student response and exclusion rates for the twelfth-grade mathematics
assessment.
Weighted student response and exclusion rates, grade 12 state mathematics assessment, by jurisdiction:
2013
Weighted student
response rates
(percent)
92.09
81.22
77.25
89.17
85.16
83.05
81.71
86.94
76.64
Weighted percentage of all students
Weighted percentage of all students
who were SD and excluded
who were ELL and excluded
Jurisdiction
Arkansas
2.78
0.30
Connecticut
1.62
0.18
Florida
3.01
0.29
Idaho
1.61
0.04
Illinois
1.82
0.14
Iowa
1.13
0.00
Massachusetts
2.12
0.46
Michigan
1.84
0.10
New
1.58
0.02
Hampshire
New Jersey
84.10
1.56
0.33
South Dakota
87.48
1.45
0.06
Tennessee
88.15
2.45
0.15
West Virginia
83.68
2.00
0.00
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2013 State Mathematics Assessment.
Appendix G NAEP 2019-2020
50
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted Student
Response and Exclusion Rates for the 2013 State
Reading Assessment
The following table presents the weighted student response and exclusion rates for the reading assessment. States with
Trial Urban District Assessment (TUDA) districts are shown in multiple rows: for the TUDA district(s) and for the state as a
whole (the TUDA district[s] plus the rest of the state). The weighted response rates utilize the student base weights and
indicate the weighted percentage of assessed students as a percentage of all students to be assessed. The exclusion rates
give the weighted percentage of excluded students, those with disabilities (SD) or students who were English language
learners (ELL), among all absent, assessed, and excluded students.
Weighted student response and exclusion rates, state reading assessment, by grade and
jurisdiction: 2013
Grade 4
Jurisdiction
Total
Alabama
Alaska
Arizona
Arkansas
California–Fresno
California–Los
Angeles
California–San
Diego
California
Colorado
Connecticut
Delaware
Florida–
Hillsborough
County
Florida–Miami
Florida
Georgia–Atlanta
Georgia
Hawaii
Idaho
Illinois–Chicago
Illinois
Indiana
Weighted
student
response
rates
(percent)
92.93
94.26
91.91
93.67
93.21
93.27
94.30
Grade 8
Weighted
percentage
of all
students
who were
SD and
excluded
1.94
0.99
1.03
1.21
1.82
3.10
2.19
Weighted
percentage
of all
students
who were
ELL and
excluded
0.57
0.18
0.48
0.26
0.27
1.15
1.38
1.02
93.78
2.19
1.08
2.26
1.22
1.09
4.17
0.64
1.29
0.45
0.63
0.99
0.43
93.42
93.46
91.38
91.59
91.85
2.09
0.97
1.76
2.92
1.34
1.17
0.31
0.52
0.71
1.02
1.57
2.08
0.99
4.02
1.33
1.35
1.01
0.96
1.97
3.42
1.10
0.13
0.98
0.79
0.21
0.95
0.41
0.71
94.21
91.72
92.20
93.67
90.58
93.64
94.72
93.76
93.12
0.73
1.28
1.02
3.66
1.22
1.41
0.87
1.16
1.75
2.15
0.68
0.00
0.21
0.90
0.27
0.98
0.39
0.15
Weighted
student
response
rates
(percent)
94.70
95.49
93.65
95.46
95.16
94.94
94.63
Weighted
percentage
of all
students
who were SD
and excluded
2.14
1.02
1.22
0.77
0.96
2.19
1.78
Weighted
percentage
of all
students
who were
ELL and
excluded
0.90
0.11
0.45
0.31
0.15
1.26
1.08
94.74
2.02
94.88
93.66
94.29
94.34
94.92
95.37
93.98
95.96
95.34
93.97
94.99
94.58
95.13
94.40
Appendix G NAEP 2019-2020
51
Iowa
Kansas
Kentucky–
Jefferson County
Kentucky
Louisiana
Maine
Maryland–
Baltimore
Maryland
Massachusetts–
Boston
Massachusetts
Michigan–Detroit
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico–
Albuquerque
New Mexico
New York–New
York City
New York
North Carolina–
Charlotte
North Carolina
North Dakota
Ohio–Cleveland
Ohio
Oklahoma
Oregon
Pennsylvania–
Philadelphia
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas–Austin
Texas–Dallas
Texas–Houston
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Appendix G NAEP 2019-2020
95.11
95.07
95.03
0.85
1.60
3.97
0.23
0.31
1.71
93.44
93.42
94.71
1.09
1.53
3.84
0.22
0.19
0.75
94.97
94.73
93.65
93.62
2.53
1.04
1.60
13.70
0.66
0.12
0.13
2.37
93.93
93.78
92.34
89.73
3.06
1.05
1.30
15.33
0.29
0.19
0.28
1.14
94.40
94.03
8.97
3.08
5.04
1.83
93.77
93.05
7.85
2.09
1.86
1.95
93.77
92.09
94.64
94.93
94.99
95.26
94.40
95.83
95.10
93.45
94.87
93.43
2.05
5.17
3.31
2.26
0.50
1.18
2.63
3.19
1.38
2.43
1.43
0.68
0.85
0.93
0.77
0.59
0.02
0.05
0.38
0.48
0.48
0.19
0.32
0.11
91.82
91.37
93.66
91.30
93.72
92.55
91.61
92.32
92.19
91.46
92.01
93.46
1.35
5.56
2.99
2.10
0.57
0.86
2.26
2.52
0.89
2.55
2.22
1.20
1.04
0.27
0.64
0.27
0.13
0.16
0.15
0.50
0.14
0.55
0.42
1.19
94.55
92.44
0.89
0.65
0.32
1.13
93.39
91.17
1.22
0.58
0.89
1.17
93.06
94.49
0.88
0.58
0.55
0.38
90.46
92.20
0.48
1.06
0.58
0.99
94.88
96.28
94.08
94.58
94.58
93.98
94.61
1.56
3.81
4.10
2.43
1.56
2.32
2.92
0.46
0.46
0.73
0.21
0.34
0.95
1.10
92.51
94.07
91.90
93.08
93.43
92.62
91.35
1.43
4.00
2.74
2.08
1.26
1.36
2.87
0.40
0.37
1.01
0.14
0.16
0.24
1.00
94.42
94.78
94.64
95.69
95.34
94.12
96.08
96.63
95.50
93.71
95.05
94.93
93.71
93.62
2.13
0.68
1.57
2.04
2.61
2.77
3.48
2.82
3.21
2.72
1.07
1.10
2.43
1.67
0.25
0.70
0.22
0.30
0.66
2.33
15.48
4.69
2.46
0.92
0.10
0.58
0.73
0.11
91.94
92.96
94.03
95.01
93.54
88.54
93.98
93.58
93.78
93.00
92.93
92.97
91.22
93.10
1.34
0.74
1.75
2.41
3.03
3.07
2.38
2.89
2.85
2.69
0.59
1.04
2.09
1.79
0.47
0.62
0.16
0.57
0.20
1.21
1.75
1.60
0.90
0.69
0.37
0.36
0.57
0.03
52
Wisconsin–
93.65
3.73
0.55
93.15
3.62
Milwaukee
Wisconsin
94.97
1.43
0.23
94.11
1.44
Wyoming
94.38
1.11
0.29
93.15
1.05
Other jurisdictions
95.63
0.69
0.00
92.93
2.84
Bureau of
Indian
Education
(BIE)
95.48
5.03
1.41
94.13
3.32
Department of
Defense
Education
Activity
(DoDEA)
94.50
1.56
0.99
90.18
0.94
District of
Columbia
(TUDA)
District of
94.46
1.18
0.70
91.33
0.97
Columbia
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for
Education Statistics, National Assessment of Educational Progress (NAEP), 2013
State Reading Assessment.
0.72
0.26
0.12
0.95
0.87
1.59
0.86
Similarly, the following table presents the weighted student response and exclusion rates for the twelfth-grade reading
assessment.
Weighted student response and exclusion rates, grade 12 state reading assessment, by jurisdiction:
2013
Weighted student
response rates
(percent)
90.21
79.77
77.34
88.68
83.72
84.26
79.84
87.21
76.91
Weighted percentage of all
students who were SD and
excluded
2.49
2.28
2.97
1.55
2.23
1.41
1.65
3.96
2.15
Weighted percentage of all
students who were ELL and
excluded
0.20
0.27
0.70
0.17
0.20
0.13
0.43
0.14
0.44
Jurisdiction
Arkansas
Connecticut
Florida
Idaho
Illinois
Iowa
Massachusetts
Michigan
New
Hampshire
New Jersey
84.67
1.61
0.18
South Dakota
86.17
1.60
0.06
Tennessee
88.82
2.86
0.08
West Virginia
84.28
2.37
0.00
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2013 State Reading Assessment.
Appendix G NAEP 2019-2020
53
NAEP Technical Documentation Website
NAEP Technical Documentation Sample
Design for the 2013 National Assessment
The 2013 national assessment included
mathematics and reading assessments in public and
private schools at grades 4, 8, and 12.
The sample designs aimed to achieve nationally
representative samples of students in the defined
populations who were enrolled at the time of
assessment.
Fourth- and Eighth-Grade Public
School National Assessments
Twelfth-Grade Public School
National Assessment
Private School National Assessment
The samples were based on a two-stage sample design:
selection of schools within strata; and
selection of students within schools.
The samples of schools were selected with probability proportional to a measure of size
based on the estimated grade-specific enrollment in the schools.
For fourth- and eighth-grade public schools, the NAEP state student samples and
assessments constitute the NAEP national student samples and assessments.
For the twelfth-grade public schools, the national sample consisted of 13 state samples and
an additional sample that represented the remaining 37 states and the District of Columbia.
Nationally representative samples were also drawn for the private school students in grades
4, 8, and 12.
Appendix G NAEP 2019-2020
54
NAEP Technical Documentation Website
NAEP Technical Documentation 2013 Fourthand Eighth-Grade Public School National
Assessment
For the mathematics and reading assessments in fourth- and eighth-grade public schools, the national
samples were the state assessment samples for each jurisdiction. All jurisdictions participated in the
mathematics and reading assessments, with the exception of Puerto Rico, where a special mathematics
assessment was conducted instead of the operational mathematics and reading assessments. Also,
Bureau of Indian Education (BIE) was not designed as a reportable jurisdiction for the 2013 state
assessments, but a nationally representative sample of students in BIE schools was selected.
Appendix G NAEP 2019-2020
55
NAEP Technical Documentation Website
NAEP Technical Documentation 2013
Twelfth-Grade Public School National
Assessment
Target Population
The twelfth-grade public school sample for the NAEP 2013
study was designed to achieve a nationally representative
Sampling Frame
sample of twelfth-grade students enrolled in public schools
Stratification of Schools
in the United States. The sample was also designed to
achieve state-level representative samples in 13 specific
School Sample Selection
states. These states were Arkansas, Connecticut, Florida,
Idaho, Illinois, Iowa, Massachusetts, Michigan, New
Hampshire, New Jersey, South Dakota, Tennessee, and West Substitute Schools
Virginia.
Ineligible Schools
The target sample size of assessed students for the twelfthStudent Sample Selection
grade public school sample was 80,000 assessed
students: 4,600 students in each of the 13 state-assessment
School and Student
states (approximately 60,000 students combined) and 20,000
Participation
students from the the remaining 37 states, the District of
Columbia, the Bureau of Indian Education (BIE) schools,
and Department of Defense Education Activity
(DoDEA) schools located within the 50 states and the District of Columbia. Prior to sampling, the
target sample sizes were adjusted upward to offset expected school and student attrition due to
nonresponse and ineligibility.
The twelfth-grade public school sample was based on a two-stage design that involved selection
of schools within strata and selection of students within schools. The first-stage sample of
schools was selected with probability proportional to a measure of size based on estimated
grade 12 student enrollment in the schools.
The students in the twelfth-grade public school sample were assessed in mathematics and
reading.
Appendix G NAEP 2019-2020
56
NAEP Technical Documentation Website
NAEP Technical Documentation Target Population
for the 2013 Twelfth-Grade Public School National
Assessment
The target population for the 2013 twelfth-grade public school national assessment included all students who were
enrolled in twelfth-grade public schools in the 50 states and the District of Columbia. The sample included Bureau of
Indian Education (BIE) schools and Department of Defense Education Activity (DoDEA) schools located within the 50
states and the District of Columbia.
Appendix G NAEP 2019-2020
57
NAEP Technical Documentation Website
NAEP Technical Documentation Sampling Frame for
the 2013 Twelfth-Grade Public School National
Assessment
As with the NAEP state sample, the sampling frame for public schools was derived
Twelfth-Grade Schools and
from the Common Core of Data (CCD) file corresponding to the 2009-2010 school
Enrollment in Public School
Sampling Frame
year. The CCD files provided the frame for all regular public, state-operated
public, Bureau of Indian Education (BIE), and Department of Defense Education
New-School Sampling Frame
Activity (DoDEA) schools open during the 2009-2010 school year.
for the National Assessment
The sampling frame excluded ungraded schools, vocational schools with no
enrollment, special education-only schools, prison and hospital schools, home
school entities, virtual or online schools, adult and evening schools, and juvenile correctional institutions.
Appendix G NAEP 2019-2020
58
NAEP Technical Documentation Website
NAEP Technical Documentation Twelfth-Grade
Schools and Enrollment in the 2013 Public School
Sampling Frame
The following table presents the number of schools and estimated enrollment for the twelfth-grade Common Core of
Data (CCD) frame by sampling stratum.
NAEP twelfth-grade Common Core of Data (CCD) frame public school enrollment and
counts, national assessment, by sampling stratum: 2013
Sampling Stratum
Schools
Percent
Estimated enrollment
Percent
Total
23,433
100.00
3,476,820
100.00
Arkansas
297
1.27
32,035
0.92
Connecticut
245
1.05
41,607
1.20
Florida
965
4.12
176,821
5.09
Idaho
210
0.90
19,057
0.55
Illinois
954
4.07
149,998
4.31
Iowa
408
1.74
37,793
1.09
Massachusetts
371
1.58
67,923
1.95
Michigan
1,032
4.40
126,382
3.63
New Hampshire
89
0.38
15,749
0.45
New Jersey
432
1.84
97,690
2.81
South Dakota
191
0.82
8,796
0.25
Tennessee
369
1.57
67,111
1.93
West Virginia
147
0.63
18,796
0.54
Remainder
17,723
75.63
2,617,062
75.27
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for
Education Statistics, National Assessment of Educational Progress (NAEP), 2013 Assessment.
Appendix G NAEP 2019-2020
59
NAEP Technical Documentation Website
NAEP Technical Documentation New-School Sampling
Frame for the 2013 Twelfth-Grade Public School
National Assessment
The Common Core of Data (CCD) file used for the CCD-based sampling frame corresponds to the 2009-2010 school year,
whereas the assessment year is the 2012-2013 school year. During this 3-year period, some schools closed, some changed
structure (one school becoming two schools, for example), and others came into existence.
To achieve as close to full coverage as possible, the CCD-based school sampling frame was supplemented by a sample of new
schools obtained from a sample of districts. Each sampled district was sent a list of the CCD schools and asked to add in any
new schools or old schools that had become newly eligible for grades 4, 8, or 12.
Since asking every school district to list new and newly-eligible schools would have generated too much of a burden, a sample
of districts was contacted to obtain a list of new schools. To represent the unsampled districts in the full sample of schools,
weights for schools included in the new-school sample were adjusted to reflect the district selection probability. This was done
for fourth-, eighth-, and twelfth-grade schools in one step, and this step is described in the new-school frame.
The following table presents the number of schools and estimated enrollment for the twelfth-grade new school frame by
sampling stratum.
NAEP twelfth-grade new school frame for the public school national assessment: school counts and
estimated enrollment by sampling stratum: 2013
Sampling Stratum
Schools
Percent
Estimated enrollment
Percent
Total
398
100.00
21,953
100.00
Arkansas
2
0.50
64
0.29
Connecticut
0
0.00
0
0.00
Florida
62
15.58
2,440
11.11
Idaho
1
0.25
10
0.05
Illinois
4
1.01
177
0.81
Iowa
0
0.00
0
0.00
Massachusetts
2
0.50
187
0.85
Michigan
2
0.50
53
0.24
New Hampshire
0
0.00
0
0.00
New Jersey
4
1.01
135
0.61
South Dakota
0
0.00
0
0.00
Tennessee
3
0.75
199
0.91
West Virginia
0
0.00
0
0.00
Remainder
318
79.90
18,688
85.13
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2013 Assessment.
Appendix G NAEP 2019-2020
60
NAEP Technical Documentation Website
NAEP Technical Documentation Stratification of
Schools for the 2013 Twelfth-Grade Public School
National Assessment
Prior to sampling, the twelfth-grade public school frame was stratified to increase the efficiency and ensure the
representativeness of the school sample in terms of important school-level characteristics, such as geography (e.g., states),
urbanicity, and race/ethnicity composition. The school frame was stratified using two types of stratification, explicit and
implicit.
Explicit stratification partitions the sampling frame into mutually exclusive groupings called sampling strata. The systematic
samples selected from these strata are independent, meaning that each is selected with its own unique random start. The
twelfth-grade public school sample had 14 sampling strata: one for each of the 13 states with state assessments and an
additional stratum representing the remainder of the frame (schools in the remaining 37 states, District of Columbia, and
all BIE and DoDEA schools).
Implicit stratification involves sorting the sampling frame, as opposed to grouping the frame. For NAEP, schools are sorted
in serpentine fashion by key school characteristics within sampling strata and sampled systematically using this ordering.
This type of stratification ensures the representativeness of the school samples with respect to the key school
characteristics.
Schools in each state stratum were implicitly stratified by urbanicity classification, race/ethnicity classification, and median
income, similarly to the grades 4 and 8 public school samples. (See stratification of schools of grades 4 and 8 public school
samples for details.)
Schools in the remainder stratum were implicitly stratified by:
census division;
urbanicity classification;
race/ethnicity classification;
school type (public, BIE, DoDEA); and
median income.
The New England and Mid-Atlantic census divisions were collapsed into a single implicit stratum comprising the census
region Northeast, as Connecticut, Massachusetts, New Hampshire, and New Jersey were all in the twelfth-grade public
school state assessment. The remaining census divisions were not collapsed.
The urbanicity classification strata were derived from the NCES urban-centric locale variable from the Common Core of Data
(CCD), which classifies schools based on location (city, suburb, town, rural) and proximity to urbanized areas. Urban-centric
locale has 12 possible values.
The urbanicity classification cells were created by starting with the original 12 NCES urban-centric locale categories within
each census division stratum. Any cell with an expected school sample size less than four was combined with a neighboring
cell within the same census division stratum. Collapsing was first done among the subcategories within a location class.
(For example, the subcategories for location class city are 1:large, 2:mid-size, and 3:small. If one of these subcategories
was deficient then either 1:large was collapsed with 2:mid-size; 3:small collapsed with 2:mid-size; or 2:mid-size collapsed
with the smaller of 1:large or 3:small.) If the collapsed cell was still too small, all three subcategories within a location class
were combined.
If a collapsed location class still had an expected school sample size less than four, then it was collapsed with a
neighboring collapsed location class. That is, 1:city would be collapsed with 2:suburb or 3:town would be collapsed with
4:rural. If additional collapsing was necessary all location classes were combined. No collapsing across census division
strata was allowed or necessary.
The final result of this was a set of census division-urbanicity strata with all strata having expected school sample sizes of
at least four schools.
Appendix G NAEP 2019-2020
61
Schools within the urbanicity classification strata were further stratified into race/ethnicity classification strata. The first
division was a dichotomization of each urbanicity stratum into a low and a high Black/Hispanic stratum (the cutoff was 15
percent Black and Hispanic students). If the expected school sample size of resultant strata was less than or equal to 8.0,
then this was the final urbanicity-race/ethnicity stratum. If the expected school sample size exceeded 8.0, a further division
was made.
For the low Black/Hispanic stratum, there were only five urbanicity strata that had a large enough expected school sample
size, and these were dichotomized by state. The table below describes the dichotomization.
Strata for low race/ethnicity strata with expected school sample sizes greater than 8: 2013
Urbanicity
stratum
Census division stratum
Group 1 states
Group 2 states
East North Central division
Rural Fringe
Indiana, Wisconsin
Ohio
West North Central
Rural Distant
Iowa, Kansas, Missouri, North Dakota, South Dakota
division
Nebraska
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2013 Assessment.
Within the high Black/Hispanic stratum, the number of substrata was based on the expected school sample size. If the
expected sample size was between 8.0 and 12.0, there were two substrata; if the expected sample size was between 12.0
and 16.0, there were three substrata; and if the expected sample size was over 16.0, there were four substrata.
The substrata were defined by percent Black and Hispanic students, with the cutoffs for substrata defined by weighted
percentiles (with the weight equal to expected hits for each school). For two substrata, the cutoff was the weighted
median; for three substrata, the weighted 33rd and 67th percentiles; for four substrata, the weighted median and
quartiles.
The implicit stratification within these census division-urbanicity-race/ethnicity status strata was based on school type
(public, BIE, DoDEA) and median income of the ZIP code area containing the school.
Appendix G NAEP 2019-2020
62
NAEP Technical Documentation Website
NAEP Technical Documentation Sampling of Schools
for the 2013 Twelfth-Grade Public School National
Assessment
Computation of Measures of
For the twelfth-grade public school assessment sample, schools were sampled
Size
independently from each sampling stratum with probability proportional to size using
systematic sampling. Prior to sampling, schools in each sampling stratum
School Sample Sizes: Frame
were sorted by the appropriate implicit stratification variables in a serpentine
and New School
order. A school's measure of size was a complex function of the school's estimated
grade enrollment. As with the grades 4 and 8 public school state assessment
samples, multiple hits were allowed for each school in the state-based
sampling strata, but not in the remainder stratum containing the schools in the remaining states and District of
Columbia.
The sampled schools for the twelfth-grade public school assessment came from two frames: the public school sample
frame (as constructed from the Common Core of Data (CCD) and the new-school sampling frame.
For the CCD-based school frame, schools in the state-based sampling strata were sampled at a rate that would
yield 4,600 assessed students per stratum. Schools in the remainder stratum were sampled at a rate that would yield a
national sample of 26,100 assessed students.
The schools in the new school frame were sampled at the same rate as the CCD-based school frame.
Appendix G NAEP 2019-2020
63
NAEP Technical Documentation Website
NAEP Technical Documentation Computation of Measures
of Size for the 2013 Twelfth-Grade Public School National
Assessment
In designing the twelfth-grade public school sample, six objectives underlie the process of determining the probability of selection
for each school and the number of students to be sampled from each selected school:
to meet the target student sample size for each explicit sampling stratum;
to select an equal-probability sample of students from each explicit sampling stratum;
to limit the number of students selected from any one school;
to ensure that the sample within a school does not include a very high percentage of the students in the school, unless all
students are included;
to reduce the rate of sampling of small schools, in recognition of the greater cost and burden per student of conducting
assessments in such schools; and
to increase the number of Black and Hispanic students in the sample.
The goal in determining the school's measure of size is to optimize across the middle four objectives in terms of maintaining the
accuracy of estimates and the cost effectiveness of the sample design.
To increase the number of Black and Hispanic students in the sample, the measure of size for schools with relatively high
proportions of Black and Hispanic students (15 percent or more) were doubled. This oversampling was limited to only the
remainder stratum, (that is, the stratum comprising schools that are not in states with state assessments). The target student
sample sizes for the state-based strata are large (4,600 assessed) and should yield a sufficient number of Black and Hispanic
students.
For schools with high proportions of Black and Hispanic students in the remainder stratum, the preliminary measures of size
(MOS) were calculated as follows:
where x js is the estimated grade 12 student enrollment for school s in stratum j.
For all other schools (those in the state-based strata or with a low proportion of Black and Hispanic students in the remainder
stratum), the preliminary measures of size (MOS) were calculated set as follows:.
where x js is the estimated grade 12 student enrollment for school s in stratum j.
Appendix G NAEP 2019-2020
64
The preliminary school measure of size was rescaled to create an expected number of hits by applying a multiplicative constant bj,
which varies by stratum j. The design for the twelfth grade school sample allowed multiple hits. For example, a school with two hits
will have twice as many students sampled as a single-hit school. To limit respondent burden, constraints were placed on the
number of hits allowed per school. For schools in the state-based sampling strata, the limit was three hits. For schools in the
remainder stratum, it was one hit.
It follows that the final measure of size, Ejs, was defined as:
where uj is the maximum number of hits allowed.
In addition, new and newly-eligible schools were sampled from the new-school frame. The assigned measures of size for these
schools,
,
used the bj and uj values from the CCD-based school frame for stratum j (i.e., the same sampling rate as for the CCD-based
school sample within each stratum). The variable π djs is the probability of selection of the district into the new-school district (d)
sample.
In addition, an adjustment was made to the initial measures of size in an attempt to reduce school burden by minimizing the
number of schools selected for both the High School Longitudinal Study (HSLS) and the grade 12 public school NAEP assessments.
The NAEP sampling procedures used an adaptation of the Keyfitz process to compute conditional measures of size that, by design,
minimized the overlap of schools selected for both the NAEP and HSLS assessments.
Appendix G NAEP 2019-2020
65
NAEP Technical Documentation Website
NAEP Technical Documentation School Sample Sizes: List FrameBased and New School for the 2013 Twelfth-Grade Public School
National Assessment
The following table presents the number of schools selected for the twelfth-grade public school sample by sampling frame (Common Core of Data
(CCD) and new school ) and sampling stratum.
NAEP public school sample counts for grade 12 national assessment, by sampling stratum and sampling frame (CCD, new
school): 2013
State
Total school sample
CCD-based school frame sample
New-school frame sample
Total
2,030
2,020
10
Arkansas
100
100
0
Connecticut
110
110
0
Florida
120
120
#
Idaho
100
100
0
Illinois
130
130
0
Iowa
120
120
0
Massachusetts
110
110
#
Michigan
140
140
#
New Hampshire
80
80
0
New Jersey
110
110
#
South Dakota
140
140
0
Tennessee
130
130
0
West Virginia
90
90
0
Remainder
570
560
10
# Rounds to zero.
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of
Educational Progress (NAEP), 2013 State Assessment.
Appendix G NAEP 2019-2020
66
NAEP Technical Documentation Website
NAEP Technical Documentation Substitute Schools
for the 2013 Twelfth-Grade Public School National
Assessment
Though efforts were made to secure the participation of all schools selected, it was anticipated that not all schools
would choose to participate. NAEP uses school substitution to mitigate the effect of bias due to nonresponse. A
nonparticipating sampled school is replaced by its substitute when the original school is considered a final refusal.
For the twelfth-grade public school sample, substitute schools were preselected for all sampled schools by sorting the
school frame file according to a sort order very close to that used in sample selection (the implicit stratification). The
two exceptions to this were as follows: (1) estimated grade enrollment replaces median income as the last sort
variable, and (2) school type in the stratification hierarchy was crossed with state (rather than used alone) in the
stratum comprising the remaining states and the District of Columbia. The first change guaranteed that the selected
substitute would have a grade enrollment very close to that of the originally selected school. The second change
guaranteed that any selected substitutes would be within the same state as the originally sampled nonresponding
school.
The two candidates for substitutes were then the two nearest neighbors of the originally sampled school on this
revised sort order. To be eligible as a potential substitute, the neighbor needed to be a nonsampled school (for any
grade) and within the same sampling stratum. If both nearest neighbors were eligible to be substitutes, the one with
a closer grade enrollment was chosen.
Nationally, 11 substitutes ultimately participated in the twelfth-grade public school sample.
Appendix G NAEP 2019-2020
67
NAEP Technical Documentation Website
NAEP Technical Documentation Eligibility Status
of Schools for the 2013 Twelfth-Grade Public School
National Assessment
The Common Core of Data (CCD) public school frame from which most of the sampled schools were drawn
corresponds to the 2009-2010 school year, some three years prior to the assessment school year. During the
intervening period, some of these schools either closed, no longer offered grade 12, or were ineligible for other
reasons. In such cases, the sampled schools were considered to be ineligible.
The table below presents unweighted counts of ineligible schools and their eligibility, by status, for the twelfth-grade
public school sample.
NAEP twelfth-grade sample public schools, national assessment, by eligibility status: 2013
Eligibility status
Unweighted count of schools
Unweighted percent
Total
2,030
100.00
Eligible
1,940
95.57
Has sampled grade, but no eligible students
7
0.34
Does not have sampled grade
15
0.74
Closed
22
1.08
Not a regular school
37
1.82
Duplicate on sampling frame
0
0.00
Other ineligible
13
0.64
NOTE: Numbers of schools are rounded to nearest ten, except those pertaining to ineligible schools. Detail
may not sum to totals due to rounding. Percentages are based on rounded counts.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education
Statistics, National Assessment of Educational Progress (NAEP), 2013 National Assessment.
Appendix G NAEP 2019-2020
68
NAEP Technical Documentation Website
NAEP Technical Documentation Student Sample
Selection for the 2013 Twelfth-Grade Public School
National Assessment
The target student sample size within the sampled schools for the twelfth-grade public sample was 60 students. However,
schools with 66 or fewer students automatically had all students sampled.
There was only one spiral type for the twelfth-grade public school sample. In the spiral, 51.5 percent of the booklets were
reading and 48.5 percent were mathematics.
The process of list submission, sampling students from year-round schools, sampling new enrollees, and determining
student eligibility and exclusion status was the same as the process used for the NAEP state student sample.
Appendix G NAEP 2019-2020
69
NAEP Technical Documentation Website
NAEP Technical Documentation School and Student
Participation in the 2013 Twelfth-Grade Public School
National Assessment
Twelfth-grade public school participation in NAEP is not mandatory. Although a small portion
of the participating school sample consisted of substitute schools, it is preferable when
calculating school response rates to do so on the basis of school participation before
substitution.
In every NAEP administration, some of the sampled students are not assessed for the
following reasons:
Weighted School Response
Rates
Weighted Student Response
and Exclusion Rates for
Mathematics
Weighted Student Response
and Exclusion Rates for
Reading
withdrawn students;
excluded students with disabilities (SD);
excluded English language learners (ELL); or
students absent from both the original session and the make-up session (not excluded but not assessed).
Withdrawn students are those who have left the school before the original assessment.
Excluded students were determined by their school to be unable to meaningfully take the NAEP assessment in their assigned
subject, even with an accommodation. Excluded students must also be classified as SD and/or ELL. Other students who were
absent for the initial session are assessed in the makeup session. The last category includes students who were not excluded
(i.e., “were to be assessed”) but were not assessed either due to absence from both sessions or because of a refusal to
participate. Assessed students are also classified as assessed without an accommodation or assessed with an accommodation.
The latter group can be divided into SD students assessed with an accommodation, ELL students assessed with an
accommodation, or students who are both SD and ELL and accommodated. Note that some SD and ELL students are assessed
without accommodations, and students who are neither SD nor ELL can only be assessed without an accommodation.
Appendix G NAEP 2019-2020
70
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted School
Response Rates for the 2013 Twelfth-Grade Public School
National Assessment
The following table presents unweighted counts of eligible sampled schools and participating schools, as well as weighted school
response rates, for the twelfth-grade public school samples in which the 2013 mathematics and reading assessments were
conducted. The weighted school response rates estimate the proportion of the student population that is represented by the
participating school sample prior to substitution.
School response counts and rates before substitution, twelfth-grade public schools, national assessment, by
region: 2013
Number of sample eligible
Number of participating
Weighted school response rate prior to
Region
schools
schools
substitution (percent)
National
1,940
1,880
92.80
Northeast
460
450
94.59
Midwest
580
560
90.22
South
620
600
91.65
West
270
270
96.17
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2013 National Assessment.
Appendix G NAEP 2019-2020
71
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted Student Response and
Exclusion Rates for the 2013 Twelfth-Grade Public School
National Mathematics Assessment
The following table presents the weighted student response and exclusion rates for the 2013 mathematics assessment for twelfth-grade public
schools. The exclusion rates give the percentage excluded, among all eligible students. Excluded students must necessarily be either students with
disabilities (SD) or English language learners (ELL). The response rates indicate the percentage of students assessed among those who it was
intended would take the assessment within the participating schools. Thus, students who were excluded are not included in the denominators of
the response rates.
Weighted student response and exclusion rates for twelfth-grade public schools, national mathematics assessment,
by region: 2013
Weighted student response
Weighted percentage of all students
Weighted percentage of all students
Region
rates (percent)
who were SD and excluded
who were ELL and excluded
National
84.17
2.22
0.23
Northeast
81.09
2.15
0.26
Midwest
84.02
1.74
0.11
South
86.34
2.45
0.10
West
83.34
2.35
0.51
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of
Educational Progress (NAEP), 2013 National Mathematics Assessment.
Appendix G NAEP 2019-2020
72
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted Student Response and
Exclusion Rates for the 2013 Twelfth-Grade Public School
National Reading Assessment
The following table presents the weighted student response and exclusion rates for the 2013 reading assessment for twelfth-grade public schools.
The exclusion rates give the percentage excluded, among all eligible students. Excluded students must necessarily be either students with
disabilities (SD) or English language learners (ELL). The response rates indicate the percentage of students assessed among those who it was
intended would take the assessment within the participating schools. Thus, students who were excluded are not included in the denominators of
the response rates.
Weighted student response and exclusion rates for twelfth-grade public schools, national reading assessment, by region:
2013
Weighted percentage of all students who
Weighted student response
Weighted percentage of all students
were ELL and excluded
Region
rates (percent)
who were SD and excluded
National
83.77
2.38
0.33
Northeast
80.11
1.84
0.31
Midwest
84.17
2.14
0.17
South
85.50
2.91
0.27
West
83.58
2.17
0.60
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of
Educational Progress (NAEP), 2013 National Reading Assessment.
Appendix G NAEP 2019-2020
73
NAEP Technical Documentation Website
NAEP Technical Documentation2013
Private School National Assessment
The private school samples were designed
to produce nationally representative samples of
students enrolled in private schools in the United
States. Fourth-, eighth-, and twelfth-grade students
were assessed in mathematics and reading.
Target Population
Sampling Frame
Stratification of Schools
School Sample Selection
Mathematics and reading pilots, a Knowledge and
Skills Appropriate (KaSA) Study, and a Reading
accessible booklet study were also conducted in the Substitute Schools
private school samples for fourth- and eighth-grade.
Ineligible Schools
The target sample sizes of assessed students for
Student Sample Selection
each grade and subject are shown in the table
below. Prior to sampling, these target sample sizes
School and Student
were adjusted upward to offset expected rates of
Participation
school and student attrition due to nonresponse and
ineligibility.
Samples were based on a two-stage design that involved selection of schools
within strata and selection of students within schools. The first-stage samples of
schools were selected with probability proportional to a measure of size based on the
estimated grade-specific enrollment in the schools.
Target sample sizes of assessed students, private school national
assessment, by subject and grade: 2013
Grade
Total
Mathematics
Reading
Pilot/Special Studies
Total
15,730
7,400
7,500
830
4
6,335
3,000
3,000
335
8
6,495
3,000
3,000
495
12
2,900
1,400
1,500
†
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences,
National Center for Education Statistics, National Assessment of Educational
Progress (NAEP), 2013 National Assessment.
Appendix G NAEP 2019-2020
74
NAEP Technical Documentation Website
NAEP Technical Documentation Target
Population for the 2013 Private School National
Assessment
The target population for the 2013 Private School National Assessment included all students enrolled in
private schools in grades 4, 8, and 12 within the 50 states and the District of Columbia.
Appendix G NAEP 2019-2020
75
NAEP Technical Documentation Website
NAEP Technical Documentation Sampling
Frame for the 2013 Private School National
Assessment
Fourth-, Eighth-, and Twelfth-Grade
The frame of the private schools in all three grades was
Schools and Enrollment in the Private
developed from the 2009-2010 Private School Universe
School Sampling Frame
Survey (PSS), a survey conducted by the U.S. Census Bureau
for the National Center for Education Statistics (NCES). The
New-School Sampling Frame for the
PSS is a biennial mail survey of all private schools in the 50
Private School Assessment
states and the District of Columbia. The PSS frame of
schools comprises both a list frame and an area frame. The
2009-2010 list frame is an assembly of the 2007-2008 PSS
frame and more up-to-date lists from state education agencies, private school associations, and other
easily accessible sources. To improve the coverage of the PSS list frame, the Census Bureau also
conducted a survey to locate private schools in a random sample of geographic areas throughout the
United States. The areas were single counties or groups of counties sampled from an area frame
constructed from all counties in the nation. Within each selected area a complete list of private schools
was gathered using information from the Yellow Pages, religious institutions, local education agencies,
chambers of commerce, and local government offices. Schools not already on the list frame were identified
and added to the frame of private schools. A weighting component was computed by the Census Bureau
so that the additional area-frame schools would represent all schools absent from the list frame, not just
those in the selected areas.
The sampling frame excluded schools that were ungraded, provided only special education, were part of
hospital or treatment center programs, were juvenile correctional institutions, were home-school entities,
or were for adult education.
Private school affiliation is unknown for nonrespondents to the PSS. Because NAEP response rates differ
vastly by affiliation, to better estimate the target sample size of schools for each affiliation, additional
work was done to obtain affiliation for nonrespondents to the PSS. If a nonresponding school responded
to a previous PSS (either two or four years prior), affiliation was obtained from the previous response. For
those schools that were nonrespondents for the last three cycles of the PSS, in some cases Internet
research was used to establish affiliation. There were still schools with unknown affiliation remaining after
this process.
For quality control purposes, school and student counts from the 2013 sampling frame were compared to
school and student counts from previous NAEP frames (2011 and 2009). No major issues were found.
Appendix G NAEP 2019-2020
76
NAEP Technical Documentation Website
NAEP Technical Documentation Fourth-, Eighth-, and
Twelfth-Grade Schools and Enrollment in the 2013 Private
School Sampling Frame
The following table displays, by grade and affiliation, the number of private schools in the sampling frame and their estimated
enrollment. For grades 4 and 8, enrollment was estimated for each school as the Private School Universe Survey (PSS)-reported
enrollment averaged across grades 1 through 8. For grade 12, the average was computed over grades 9 through 12.
The counts presented below are of schools with known affiliation. Schools with unknown affiliation do not appear in the table because
their grade span, affiliation, and enrollment were unknown. Although PSS is a school universe survey, participation is voluntary and not
all private schools respond. Since the NAEP sample must represent all private schools, not just PSS respondents, a small sample of PSS
nonrespondents with unknown affiliation was selected for each of the targeted grades to improve NAEP coverage.
Number of schools and enrollment in private school sampling frame, national assessment, by affiliation and
grade: 2013
Grade
4
Affiliation
Number of schools
Estimated enrollment
Total
19,553
354,543
Catholic
5,669
156,505
Non-Catholic private
13,884
198,038
8
Total
17,607
342,303
Catholic
5,214
156,583
Non-Catholic private
12,393
185,720
12
Total
9,138
317,449
Catholic
1,295
150,454
Non-Catholic private
7,843
166,995
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2013 National Assessment.
Appendix G NAEP 2019-2020
77
NAEP Technical Documentation Website
NAEP Technical Documentation New-School
Sampling Frame for the 2013 Private School
National Assessment
Whereas the Private School Universe Survey (PSS) file used for the frame corresponds to the 2009-2010 school year,
the NAEP assessment year was the 2012-2013 school year. During this 3-year period, some schools closed, some
changed their grade span, and still others came into existence.
To achieve as close to full coverage as possible, the private school frame was supplemented by a sample of new
Catholic schools. The goal was to allow every such school a chance of selection, thereby fully covering the target
population of Catholic schools in operation during the 2012-2013 school year. The first step in this process was the
development of a new-school frame through the construction of a diocesan-level file from the PSS school-level file.
To develop the frame, the diocesan-level file was divided into two files: one for small dioceses and the other for
medium and large dioceses.
Small dioceses contained no more than three schools on the frame in total, with no more than one school at each
grade (fourth, eighth, and twelfth). New schools in small dioceses were identified during school recruitment and
added to the sample if the old school in the same diocese was sampled at the relevant grade. From a sampling
perspective, the new school was viewed as an “annex” to the sampled school that had a well-defined probability of
selection equal to that of the old school. The “frame” in this case was, in fact, the original frame; when the old
school was sampled in a small diocese, the new school was automatically sampled as well.
To limit respondent burden and keep the level of effort within reasonable bounds, the new-school frame was
created using information obtained from a sample of the remaining dioceses. The remaining dioceses were separated
into two strata of large- and medium-size dioceses. These strata were defined by computing the percentage of the
nation’s total Catholic school enrollment each diocese represents, sorting the dioceses in descending order by that
percentage, and cumulating the percentages across the sorted file. All dioceses up to and including the first diocese
at or above the 80th cumulative percentage were defined as large dioceses. The remaining dioceses were defined as
medium dioceses.
A simplified example is given below. Dioceses are ordered by percentage enrollment. The first six become large
dioceses and the last six become medium dioceses.
Example showing assignment of Catholic dioceses to the large and medium strata, private school
national assessment: 2013
Diocese
Diocese
Diocese
Diocese
Diocese
Diocese
Diocese
Diocese
Diocese
Diocese
Diocese
Diocese
1
2
3
4
5
6
7
8
9
10
11
Percent enrollment
20
20
15
10
10
10
5
2
2
2
2
Appendix G NAEP 2019-2020
Cumulative percentage enrollment
20
40
55
65
75
85
90
92
94
96
98
78
Stratum
L
L
L
L
L
L
M
M
M
M
M
Diocese 12
2
100
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education
Statistics, National Assessment of Educational Progress (NAEP), 2013 National Assessment.
M
In actuality, there were 72 large and 102 medium dioceses in the sampling frame.
The target sample size was 10 dioceses total: 8 large and 2 medium. In the medium stratum, the dioceses were
selected with equal probability. In the large stratum, dioceses were sampled with probability proportional to
enrollment. These probabilities were retained and used in all later stages of sampling and weighting in order to
represent all dioceses, whether or not they had been selected as new school samples for the assessment.
Each selected diocese was sent a listing of its schools extracted from the 2009-2010 PSS file and was asked to
provide information about new schools and any changes to grade span in existing schools. This information provided
by the selected dioceses was used to create sampling frames for the selection of new Catholic schools. The process
of obtaining the information was conducted with the help of the National Catholic Educational Association (NCEA).
NCEA was sent the school lists for the 10 sampled dioceses and was responsible for returning the completed
updates.
The eligibility of a new school at a particular grade was determined by its grade span. A school already on PSS also
was classified as “new” if a change of grade span had occurred such that the school status changed from ineligible
to eligible at a particular grade.
Appendix G NAEP 2019-2020
79
NAEP Technical Documentation Website
NAEP Technical Documentation Stratification of
Schools for the 2013 Private School National
Assessment
Explicit stratification for the NAEP 2013 private school samples was by private school type: Catholic, nonCatholic, and unknown affiliation. Private school affiliation was unknown for nonrespondents to the NCES Private
School Universe Survey (PSS) for the past three cycles.
The implicit stratification of the schools involved four dimensions. Within each explicit stratum, the private
schools were hierarchically sorted by census region, urbanicity status, race/ethnicity status, and estimated grade
enrollment. The implicit stratification in this four-fold hierarchical stratification was achieved via a "serpentine
sort".
Census region was used as the first level of implicit stratification for the NAEP 2013 private school sample. All
four census regions were used as strata.
The next level of stratification was an urbanicity classification based on urban-centric locale, as specified on the
PSS. Within a census region-based stratum, urban-centric locale cells that were too small were collapsed. The
criterion for adequacy was that the cell had to have an expected school sample size of at least six.
The urbanicity variable was equal to the original urban-centric locale if no collapsing was necessary to cover an
inadequate original cell. If collapsing was necessary, the scheme was to first collapse within the four major
strata (city, suburbs, town, and rural). For example, if the expected number of large city schools sampled was
less than six, large city was collapsed with midsize city. If the collapsed cell was still inadequate, they were
further collapsed with small city. If a major stratum cell (all three cells collapsed together) was still deficient, it
was collapsed with a neighboring major stratum cell. For example, city would be collapsed with suburbs.
The last stage of stratification was a division of the geographic/urbanicity strata into race/ethnicity strata if the
expected number of schools sampled was large enough (i.e., at least equal to 12). This was done by deciding
first on the number of race/ethnicity strata and then dividing the geography/urbanicity stratum into that many
pieces. The school frame was sorted by the percentage of students in each school who were Black, Hispanic, or
American Indian. The three racial/ethnic groups defining the race/ethnicity strata were those that have
historically performed substantially lower on NAEP assessments than White students. The sorted list was then
divided into pieces, with roughly an equal expected number of sampled schools in each piece.
Finally, schools were sorted within stratification cells by estimated grade enrollment.
Appendix G NAEP 2019-2020
80
NAEP Technical Documentation Website
NAEP Technical Documentation Sampling of
Schools for the 2013 Private School National
Assessment
Computation of Measures
The private school samples were selected with probability proportional to size
of Size
using systematic sampling from a sorted list. A school's measure of size was a
complex function of the school's estimated grade enrollment. For all
three grades, only one "hit" was allowed per school.
School Sample Sizes:
Frame and New School
Schools were ordered within each school type using a serpentine sort involving
the following variables:
census region;
urbanicity classification (based on urban-centric locale);
race/ethnicity status; and
estimated grade enrollment.
A systematic sample was then drawn with probability proportional to size using this serpentine sorted list and
the measures of size.
Schools with unknown affiliation were treated separately. A sample of about 30 schools with unknown
affiliation was selected at each of the three grades.
Appendix G NAEP 2019-2020
81
NAEP Technical Documentation Website
NAEP Technical Documentation School Sample Sizes: List
Frame-Based and New School for the 2013 Private School
National Assessment
The following table presents the number of schools selected from the private school sampling frame (constructed from the Private School
Universe Survey file) and the new-school sampling frame, for grades 4, 8, and 12, by school type.
NAEP private school national assessment frame-based and new school samples, by grade and school type: 2013
New school sample
Grade and private school type
Total school sample
Frame school sample
Grade 4
All private
410
410
#
Catholic
130
130
#
Non-Catholic private
250
250
0
Unknown affiliation
30
30
0
Grade 8
All private
400
390
10
Catholic
130
130
10
Non-Catholic private
240
240
0
Unknown affiliation
30
30
0
Grade 12
All private
160
160
#
Catholic
40
40
#
Non-Catholic private
100
100
0
Unknown affiliation
25
30
0
# Rounds to zero.
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2013 National Assessment.
Appendix G NAEP 2019-2020
82
NAEP Technical Documentation Website
NAEP Technical Documentation Substitute
Schools for the 2013 Private School National
Assessment
Substitutes were preselected for the private school samples by sorting the school frame file according to the
actual order used in the sampling process (the implicit stratification). Each sampled school had its two
nearest neighbors on the school frame file identified as potential substitutes. As the last sort ordering was
by grade enrollment, the nearest neighbors had grade enrollment values very close to that of the sampled
school.
Schools were disqualified as potential substitutes if they were already selected in the private school sample
or assigned as a substitute for another private school (earlier in the sort ordering). Schools assigned as
substitutes for twelfth-grade schools were disqualified as potential substitutes for fourth- and eighth-grade
schools, and schools assigned as substitutes for eighth-grade schools were disqualified as potential
substitutes for fourth-grade schools.
If both nearest neighbors were still eligible to be substitutes, the one with the closer grade enrollment was
chosen. If both nearest neighbors had the same grade enrollment (an uncommon occurrence), one of the
two was randomly selected.
In the process described above, only schools with the same affiliation were selected as substitutes.
Appendix G NAEP 2019-2020
83
NAEP Technical Documentation Website
NAEP Technical Documentation Ineligible
Schools for the 2013 Private School National
Assessment
Eligibility Status of Sampled
The Private School Universe Survey (PSS) school file, from which most of
Schools by Grade and Private
the sampled schools were drawn, corresponds to the 2009-2010 school
School Type
year, 3 years prior to the assessment school year. During the intervening
period, some of these schools either closed, no longer offered the grade
Ineligible Sampled Schools by
of interest, or were ineligible for other reasons. In such cases, the
Ineligibility Type
sampled schools were coded as ineligible.
Appendix G NAEP 2019-2020
84
NAEP Technical Documentation Website
NAEP Technical Documentation Eligibility Status of
Sampled Schools for the 2013 Private School National
Assessment
The following table presents a breakdown by private school type of ineligible and eligible schools in the fourth-, eighth-,
and twelfth-grade private school samples. There are considerable differences across private school types at grades 4, 8,
and 12. Schools whose private school type was unknown at the time of sampling subsequently had their affiliation
determined during data collection. Therefore, such schools are not broken out separately.
Eligibility status of sampled private schools, national assessment, by grade and private school type:
2013
Fourth grade
Eighth grade
Twelfth grade
Private school type
Eligibility status
Count
Percentage
Count
Percentage
Count
Percentage
All private
Total
410
100.00
400
100.00
160
100.00
Ineligible
60
14.63
70
17.50
40
25.00
Eligible
350
85.37
330
82.50
120
75.00
Roman Catholic
Total
130
100.00
130
100.00
40
100.00
Ineligible
10
7.69
10
7.69
0
0.00
Eligible
130
100.00
120
92.31
40
100.00
Other private
Total
280
100.00
270
100.00
120
100.00
Ineligible
60
21.43
70
25.93
40
33.33
Eligible
220
78.57
200
74.07
80
66.67
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to rounding. Percentages
are based on rounded counts.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics,
National Assessment of Educational Progress (NAEP), 2013 National Assessment.
Appendix G NAEP 2019-2020
85
NAEP Technical Documentation Website
NAEP Technical Documentation Sampled Schools
for the 2013 Private School National Assessment
The table below presents unweighted counts of sampled schools, by grade and eligibility status, for the private
school samples.
NAEP sample private schools, national assessment, by grade and eligibility status: 2013
Grade and eligibility status
Unweighted count of schools Unweighted percentage
All fourth-grade sampled private schools
410
100.00
Eligible
350
85.37
Has sampled grade, but no eligible students
15
3.66
Does not have sampled grade
11
2.68
Closed
22
5.37
Not a regular school
12
2.93
Duplicate on sampling frame
0
0.00
Other ineligible
2
0.49
All eighth-grade sampled private schools
400
100.00
Eligible
330
82.50
Has sampled grade, but no eligible students
11
2.75
Does not have sampled grade
19
4.75
Closed
28
7.00
Not a regular school
13
3.25
Duplicate on sampling frame
0
0.00
Other ineligible
2
0.50
All twelfth-grade sampled private schools
160
100.00
Eligible
120
75.00
Has sampled grade, but no eligible students
4
2.50
Does not have sampled grade
14
8.75
Closed
7
4.38
Not a regular school
12
7.50
Duplicate on sampling frame
0
0.00
Other ineligible
2
1.25
NOTE: Numbers of schools are rounded to nearest ten, except those pertaining to ineligible schools. Detail
may not sum to totals due to rounding. Percentages are based on rounded counts.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education
Statistics, National Assessment of Educational Progress (NAEP), 2013 National Assessment.
Appendix G NAEP 2019-2020
86
NAEP Technical Documentation Website
NAEP Technical Documentation Student Sample
Selection for the 2013 Private School National
Assessment
The target student sample size within sampled schools varied by grade. For fourth grade, the target was 64
students; and for eighth grade, the target was 65 students. However, schools with 72 or fewer students
automatically had all students sampled. In addition, at grade 4 only, a school that had more than 72 students but
fewer than 90 students could choose to have all students sampled. For schools sampled for the twelfth grade the
target was 60 students. However, schools with 66 or fewer students had all students sampled.
There was only one spiral type for each grade. The percentage of booklets by subject within the spiral for each
grade is given below.
Percentage of booklets, private school national assessment, by subject within the
spiral and grade: 2013
Grade
Mathematics
Reading
Pilot/Special
4
45.49
46.19
8.33
8
44.94
45.00
10.06
12
48.51
51.49
†
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for
Education Statistics, National Assessment of Educational Progress (NAEP), 2013 National
Assessment.
The process of student list submission, sampling students from year-round schools, sampling new enrollees, and
determining student eligibility and exclusion status was the same as for the state NAEP student sample.
Appendix G NAEP 2019-2020
87
NAEP Technical Documentation Website
NAEP Technical Documentation School and Student
Participation in the 2013 Private School National
Assessment
Private school participation in NAEP is not mandatory. The 2013 assessment holds true
to the historic pattern of having higher rates of participation in Catholic schools than
among non-Catholic schools. Although a portion of the participating school sample
consisted of substitute schools, it is preferable to calculate school response rates on
the basis of school participation before substitution.
In every NAEP survey, some of the sampled students are not assessed for the
following reasons:
withdrawn students;
excluded students with disabilities (SD);
excluded English language learners (ELL) students; or
students absent from both the original session and the makeup session (not
excluded but not assessed).
Weighted School Response
Rates
Weighted Student
Response and Exclusion
Rates for Mathematics
Weighted Student
Response and Exclusion
Rates for Reading
Withdrawn students are those who have left the school before the original assessment. Excluded students were
determined by their school to be unable to meaningfully take the NAEP assessment in their assigned subject, even with
an accommodation. Excluded students must also be classified as SD and/or ELL. Other students who were absent for
the initial session are assessed in the makeup session. The last category includes students who were not excluded (i.e.,
“were to be assessed”) but were not assessed either due to absence from both sessions or because of a refusal to
participate. Assessed students are also classified as assessed without an accommodation or assessed with an
accommodation. The latter group can be divided into SD students assessed with an accommodation, ELL students
assessed with an accommodation, or students who are both SD and ELL and accommodated. Note that some SD and
ELL students are assessed without accommodations, and students who are neither SD nor ELL can only be assessed
without an accommodation.
The weighted response rates utilize the student base weights and indicate the weighted percentage of assessed
students among all students to be assessed. The exclusion rates, in contrast, provide the weighted percentage of
excluded SD or ELL students among all absent, assessed, and excluded students.
Appendix G NAEP 2019-2020
88
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted School
Response Rates for the 2013 Private School National
Assessment
The following table presents counts of eligible sampled schools and participating schools, as well as weighted school
response rates, for the private school samples in which the mathematics and reading operational assessments were
conducted. The weighted school response rates estimate the proportion of the student population that is represented
by the participating school sample prior to substitution.
Private school response rates, national assessment, by school type and grade: 2013
Eligible
sampled
schools
350
130
220
330
120
200
Participating schools, Weighted school response rate prior
including substitutes
to substitution (percent)
Private school type
All private
280
71.19
Catholic
120
88.65
Non-Catholic private
160
56.94
8
All private
260
69.63
Catholic
120
87.18
Non-Catholic
140
53.51
private
12
All private
120
90
53.34
Catholic
40
30
68.06
Non-Catholic private
80
50
38.52
NOTE: Numbers of schools are rounded to nearest ten. Detail may not sum to totals due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education
Statistics, National Assessment of Educational Progress (NAEP), 2013 National Assessment.
Grade
4
Appendix G NAEP 2019-2020
89
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted Student Response
and Exclusion Rates for the 2013 Private School National
Mathematics Assessment
The following table presents the weighted student response and exclusion rates for the mathematics assessment. The exclusion rates give the
percentage of students excluded among all eligible students. Excluded students must be either students with disabilities (SD) or English
language learners (ELL). The response rates indicate the percentage of students assessed among those who were intended to take the
assessment within the participating schools. Thus, students who were excluded are not included in the denominators of the response rates.
Weighted student response and exclusion rates for private schools, national mathematics assessment, by school type
and grade: 2013
Weighted percentage of all students
Weighted student
Weighted percentage of all students
who were ELL and excluded
Private school type
response rate
who were SD and excluded
All private
95.61
0.06
0.03
Catholic
95.60
0.00
0.06
Non-Catholic private
95.62
0.11
0.00
8
All private
94.74
0.19
0.08
Catholic
95.73
0.10
0.16
Non-Catholic private
93.50
0.26
0.00
12
All private
86.51
0.63
0.00
Catholic
85.53
0.83
0.00
Non-Catholic private
87.96
0.42
0.00
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment
of Educational Progress (NAEP), 2013 Mathematics Assessment.
Grade
4
Appendix G NAEP 2019-2020
90
NAEP Technical Documentation Website
NAEP Technical Documentation Weighted Student Response
and Exclusion Rates for the 2013 Private School National
Reading Assessment
The following table presents the weighted student response and exclusion rates for the reading assessment. The exclusion rates give the
percentage of students excluded among all eligible students. Excluded students must be either students with disabilities (SD) or English
language learners (ELL). The response rates indicate the percentage of students assessed among those who were intended to take the
assessment within the participating schools. Thus, students who were excluded are not included in the denominators of the response rates.
Weighted student response and exclusion rates for private schools, national reading assessment, by school type and
grade: 2013
Weighted student
Weighted percentage of all students
Weighted percentage of all students
Private school type
response rate
who were SD and excluded
who were ELL and excluded
All private
95.85
0.46
0.07
Catholic
95.75
0.17
0.06
Non-Catholic private
95.96
0.71
0.08
8
All private
95.45
0.18
0.12
Catholic
96.07
0.21
0.00
Non-Catholic private
94.67
0.16
0.23
12
All private
85.52
0.78
0.05
Catholic
84.67
0.81
0.10
Non-Catholic private
86.75
0.75
0.00
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2013 Reading Assessment.
Grade
4
Appendix G NAEP 2019-2020
91
| File Type | application/pdf |
| Author | Molin, Ed C |
| File Modified | 2019-07-17 |
| File Created | 2018-02-06 |