NAEP 2024 Appendix G Sample Design
NAEP 2024 Appendix G v30.docx
National Assessment of Educational Progress (NAEP) 2024 Amendment #3
NAEP 2024 Appendix G Sample Design
OMB: 1850-0928
NATIONAL CENTER FOR EDUCATION STATISTICS
NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
National Assessment of Educational Progress (NAEP) 2024
Appendix G
NAEP 2018 Sample Design
OMB# 1850-0928 v.30
June 2023
The 2018 Weighting Procedures documentation is the most current version available to the public. At this time, there is not a timeline for when the details for later assessment years will be publicly available.
NAEP Technical Documentation Website
NAEP Technical Documentation NAEP 2018 Sample Design
The NAEP 2018 sample design consisted of a nationally representative sample of students for the following operational and pilot assessments:
social
sciences assessments
in civics,
geography, and
U.S. history
at grade
8; technology
and engineering
literacy (TEL)
assessment at
grade 8;
a
science pilot
test at
grades 4,
8, and
12; a
reading pilot
test at
grade 12;
and
a
mathematics pilot
test at
grade 12.
In
addition to
the operational
and pilot
assessments, special
studies were
conducted, including:
reading scenario-based
tasks (SBT)
at grades
4, 8,
and 12;
and
oral
reading fluency
(ORF) at
grade 4.
Selection of Primary Sampling Units
2018 Public School Social Sciences Assessment 2018 Private School Social Sciences Assessment 2018 Public School TEL Assessment
2018 Private School TEL Assessment School and Student Participation Results
This was accomplished by designing separate sample components for public and private schools. The selected samples were based on a three-stage sample design:
selection
of primary
sampling units
(PSUs) selection
of schools
within strata,
and selection
of students
within schools
The samples of schools were selected with probability proportional to a measure of size based on the estimated enrollment in the schools at grades 4, 8, and 12.
The target population included all students in public and private schools, including Bureau of Indian Education (BIE) and Department of Defense Education Activity (DoDEA) schools, who were enrolled in grades 4, 8, and 12, respectively, at the time of assessment.
The figure below illustrates the various sample types and subjects. Assessments were either paper-based (PBA) or digitally based (DBA).
Components of the NAEP samples, by assessment subject, grade, and school type: 2018
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessments.
The sample design for the operational assessments is described in more detail in subsequent pages.
http://nces.ed.gov/nationsreportcard/tdw/sample_design/2018/naep_2018_sample_design.aspx
NAEP Technical Documentation 2018 Private School Social Sciences Assessment
The NAEP 2018 sample design yielded nationally representative samples of private school students in grade 8 for social sciences through a three-stage approach:
selection
of primary
sampling units
(PSUs), selection
of schools
within strata,
and selection
of students
within schools.
The sample of schools was selected with probability proportional to a measure of size based on the estimated grade enrollment in the schools.
The 2018 sampling plan was designed to assess 5,200 eighth-graders in private schools for social sciences. These students were allocated among tests in civics, geography, and U.S. history. Target sample sizes were adjusted to reflect expected private school and student response and eligibility.
Schools on the sampling frame were explicitly stratified prior to sampling by private school affiliation (Catholic, non-Catholic, and unaffiliated). Within affiliation type, schools were implicitly stratified by PSU type (certainty/noncertainty). In certainty PSUs,
Target Population Sampling Frame Stratification of Schools Sampling of Schools Substitute Schools Ineligible Schools Student Sample Selection
further stratification was by census region, urbanization classification, and estimated grade enrollment. In noncertainty PSUs, additional stratification was by PSU stratum, urbanization classification, and estimated grade enrollment.
From the stratified frame of private schools, systematic random samples of eighth-grade schools were drawn with probability proportional to a measure of size based on the estimated grade enrollment of the school in the relevant grade.
Each selected school in the private school sample provided a list of eligible enrolled students from which a systematic, equal probability sample of students was drawn.
NAEP Technical Documentation Ineligible Private Schools for the 2018 Social Sciences Assessment
The Private School Universe Survey (PSS) school file from which most of the sampled schools were drawn corresponds to the 2015–2016 school year, two years prior to the assessment school year. During the intervening period, some of these schools either closed, no longer offered the grade of interest, or were ineligible for other reasons. In such cases, the sampled schools were coded as ineligible.
The table below presents unweighted counts of sampled private schools by eligibility status, including the reason for ineligibility.
Number of sampled private schools, social sciences assessment, grade 8, by eligibility status: 2018
-
Eligibility status
Unweighted count of schools
Unweighted percentage
All eighth-grade sampled private schools
330
100.00
Eligible schools
270
81.82
No eligible students in grade
13
3.94
Does not have sampled grade
14
4.24
School closed
16
4.85
Not a regular school
16
4.85
Other ineligible school
1
0.30
NOTE: Total and eligible school counts are rounded to nearest ten. Percentages are based on rounded counts. Detail may not sum to total due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
-
Eligibility status
Unweighted count of schools
Unweighted percentage
Duplicate
on sampling
frame 0 0.00
The table below presents unweighted counts of sampled private schools by private school type and eligibility status.
Number of sampled private schools, social sciences assessment, grade 8, by private school type and eligibility status: 2018
-
Private school type
Eligibility status
Unweighted count of schools
Unweighted percentage
All Private
Total
330
100.00
Eligible
270
81.82
Ineligible
60
18.18
-
Catholic
Total
80
100.00
Eligible
80
100.00
Ineligible
4
5.00
Other Private
Total
250
100.00
Eligible
190
76.00
Ineligible
56
22.40
NOTE: Total and eligible school counts are rounded to nearest ten. Percentages are based on rounded counts. Detail may not sum to total due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
NAEP Technical Documentation Sampling Frame for the 2018 Private School Social Sciences Assessment
The primary sampling frames for private schools were developed from the Private School Universe Survey (PSS) corresponding to the 2015-2016 school year. The PSS file is the Department of Education’s primary database of elementary and secondary private schools in the 50 states and the District of Columbia, and it is based on a survey conducted by the U.S. Census Bureau during the 2015-2016 school year. These sampling frames are referred to as the PSS-based sampling frames.
The sampling frame was restricted to schools located in the primary sampling units (PSUs) selected for the NAEP 2018 social sciences assessment. In addition, the sampling frame excluded ungraded schools, vocational schools with no enrollment, special-education-only schools, homeschool entities, prison and hospital schools, and juvenile correctional institutions. Vocational schools with no enrollment serve students who split their time between the vocational school and their home school.
The following table presents the number of schools and estimated enrollment for the private school frame for the social sciences assessment at grade 8. The unweighted estimated enrollment is restricted to the selected PSUs. The weighted estimated enrollment incorporates the PSU weight (inverse of the probability of selecting the PSU), and thus is a national estimate of the number of private school students in eighth grade.
Number of schools and enrollment in private school sampling frame, social sciences assessment, grade 8, by affiliation: 2018
-
Affiliation
Number of schools
Estimated enrollment (unweighted)
Estimated enrollment (weighted)
Total
10,816
210,856
305,373
Catholic
2,839
86,509
121,713
Non-Catholic
5,820
113,562
169,226
Unknown affiliation
2,157
10,785
14,435
NOTE: Detail may not sum to totals due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
For quality control purposes, school and student counts from the sampling frame were compared to school and student counts from previous private school frames for eighth grade. No major discrepancies were found.
NAEP Technical Documentation Sampling of Private Schools for the 2018 Social Sciences Assessment
In the design of each school sample, five objectives underlie the process of determining the probability of selection for each school and how many students are to be sampled from each selected school containing grade-eligible students. The five objectives are
to
meet the
target student
sample size;
to
select an
equal-probability sample
of students;
to
limit the
number of
students who
are selected
from a
school;
to
ensure that
the sample
within a
school does
not include
a very
high percentage
of the
students in
the school,
unless all
students are
included; and
to reduce
the rate
of sampling
of small
schools, in
recognition of
the greater
cost and
burden per
student of
conducting assessments
in such
schools.
The goal in determining the school's measure of size is to optimize across the last four objectives in terms of maintaining the accuracy of estimates and the cost- effectiveness of the sample design.
Therefore, to meet the target student sample size objective and achieve a reasonable compromise among the other four objectives, the following algorithm was used to assign a measure of size to each school based on its estimated grade enrollment as indicated on the sampling frame.
The measures of size vary by enrollment size. The initial measures of size (MOS) were set as follows:
For eighth grade:
where Xjs is the estimated grade enrollment for grade j in school s, PSCHWTs = the Private School Universe Survey area frame weight for school s, computed by the U.S. Census Bureau, and PSU_WTs = the PSU weight for school s.
The measures of size for schools in the Honolulu primary sampling unit (PSU) are doubled to increase their chances of selection:
Schools in the Honolulu PSU have their measures of size doubled to ensure at least one sampled school from the PSU. The Honolulu PSU is a certainty not due to its size, but because it is unique.
The next task in this development is to describe bj, the constant of proportionality for each grade. It is a sampling parameter that, when multiplied with a school’s preliminary measure of size (Mjs), yields the school’s final measure of size. It is computed in such a way that, when used with the systematic sampling procedure, the target student sample size is achieved. For private schools, this parameter varied by private school affiliation (Catholic, non-Catholic, and unknown affiliation).
The final measure of size, Ejs, is defined as:
The quantity uj (the maximum number of “hits” allowed) in this formula is designed to put an upper bound on the burden for the sampled schools. For private schools, uj is 1 because by design a school could not be selected, or "hit," in the sampling process more than once within a grade.
Schools were ordered within each jurisdiction using the serpentine sort described under the stratification of private schools. A systematic sample was then drawn using this serpentine sorted list and the measures of size. The number of private schools selected for eighth-grade social sciences was approximately 330.
NAEP Technical Documentation Stratification of Private Schools for the 2018 Social Sciences Assessment
For the private school sampling frame file, schools were explicitly stratified by private school affiliation (Catholic, non-Catholic, and unknown affiliation). Private school affiliation was unknown for nonrespondents to the NCES Private School Universe Survey (PSS). Within private school type, separate implicit stratification schemes were used to sort schools in certainty primary sampling units (PSUs) and noncertainty PSUs. In all cases, the implicit stratification was achieved via a "serpentine sort".
Within
each certainty
PSU, the
schools were
hierarchically sorted
by census
region,
urbanization
classification (four
categories based
on urban-centric
locale), and
estimated
grade enrollment.
Schools
in noncertainty
PSUs were
hierarchically sorted
by PSU
stratum,
urbanization
classification (four
categories based
on urban-centric
locale), and
estimated grade
enrollment.
NAEP Technical Documentation Student Sample Selection for the 2018 Private School Social Sciences Assessment
Students in private schools were selected in the same way as students in the public schools.
NAEP Technical Documentation Substitute Private Schools for the 2018 Social Sciences Assessment
Substitutes were preselected for the private school samples by sorting the school frame file according to the actual order used in the sampling process (the implicit stratification). For operational reasons, the original selection order was embedded within the sampled primary sampling unit (PSU) and state. Each sampled school had each of its nearest neighbors within the same sampling stratum on the school frame file identified as a potential substitute. Since grade enrollment was used as the last sort ordering variable, the nearest neighbors had grade enrollment values very close to that of the sampled school. This was done to facilitate the selection of about the same number of students within the substitute as would have been selected from the original sampled school.
Schools were disqualified as potential substitutes if they were already selected in any of the original private school samples or assigned as a substitute for another private school (earlier in the sort ordering).
If both nearest neighbors were still eligible to be substitutes, the one with a closer grade enrollment was chosen. If both nearest neighbors were equally distant from the sampled school in their grade enrollment (an uncommon occurrence), one of the two was randomly selected.
Of the approximately 330 originally sampled private schools for the eighth-grade social sciences assessment, about 100 schools had substitutes activated when the original eligible schools did not participate. Ultimately, about 30 of the activated substitute private schools participated in a social sciences assessment.
NAEP Technical Documentation Target Population of the 2018 Private School Social Sciences Assessment
The target populations for the 2018 civics, geography, and U.S. history private school assessments included all students who were enrolled in eighth grade in private schools located in the 50 states and the District of Columbia.
NAEP Technical Documentation 2018 Private School Technology and Engineering Literacy (TEL) Assessment
The NAEP 2018 sample design yielded nationally representative samples of private school students in grade 8 for TEL through a three-stage approach:
selection
of primary
sampling units
(PSUs), selection
of schools
within strata,
and selection
of students
within schools.
Target Population Sampling Frame Stratification of Schools
The sample of schools was selected with probability proportional to a measure of size based on the estimated grade enrollment in the schools.
The 2018 sampling plan was designed to assess 1,600 eighth-graders in private schools for TEL. Target sample sizes were adjusted to reflect expected private school and student response and eligibility.
Schools on the sampling frame were explicitly stratified prior to sampling by private school affiliation (Catholic, non-Catholic, and unaffiliated). Within affiliation type, schools were implicitly stratified by PSU type (certainty/noncertainty). In certainty PSUs,
Sampling of Schools Substitute Schools Ineligible Schools Student Sample Selection
further stratification was by census region, urbanization classification, and estimated grade enrollment. In noncertainty PSUs, additional stratification was by PSU stratum, urbanization classification, and estimated grade enrollment.
From the stratified frame of private schools, systematic random samples of eighth-grade schools were drawn with probability proportional to a measure of size based on the estimated grade enrollment of the school in the relevant grade.
Each selected school in the private school sample provided a list of eligible enrolled students from which a systematic, equal probability sample of students was drawn.
NAEP Technical Documentation Ineligible Private Schools for the 2018 Technology and Engineering Literacy (TEL) Assessment
The Private School Universe Survey (PSS) school file from which most of the sampled schools were drawn corresponds to the 2015–2016 school year, two years prior to the assessment school year. During the intervening period, some of these schools either closed, no longer offered the grade of interest, or were ineligible for other reasons. In such cases, the sampled schools were coded as ineligible.
The table below presents unweighted counts of sampled private schools by eligibility status, including the reason for ineligibility.
Number of sampled private schools, technology and engineering literacy (TEL) assessment, grade 8, by eligibility status: 2018
-
Eligibility status
Unweighted count of schools
Unweighted percentage
All
eighth-grade
sampled
private
schools 140 100.00
-
Eligibility status
Unweighted count of schools
Unweighted percentage
-
Eligible schools
120
85.71
No eligible students in grade
5
3.57
Does not have sampled grade
7
5.00
School closed
5
3.57
Not a regular school
5
3.57
Other ineligible school
0
0.00
Duplicate on sampling frame
0
0.00
NOTE: Total and eligible school counts are rounded to nearest ten. Percentages are based on rounded counts. Detail may not sum to total due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Technology and Engineering Literacy (TEL) Assessment.
The table below presents unweighted counts of sampled private schools by private school type and eligibility status.
Number of sampled private schools, technology and engineering literacy (TEL) assessment, grade 8, by private school type and eligibility status: 2018
-
Private school type
Eligibility status
Unweighted count of schools
Unweighted percentage
All Private
Total
140
100.00
Eligible
120
85.71
Ineligible
22
15.71
-
Catholic
Total
30
100.00
Eligible
30
100.00
Ineligible
1
3.33
Other Private
Total
110
100.00
Eligible
90
81.82
Ineligible
21
19.09
NOTE: Total and eligible school counts are rounded to nearest ten. Percentages are based on rounded counts. Detail may not sum to total due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Technology and Engineering Literacy (TEL) Assessment.
NAEP Technical Documentation Sampling Frame for the 2018 Private School Technology and Engineering Literacy (TEL) Assessment
The primary sampling frames for private schools were developed from the Private School Universe Survey (PSS) corresponding to the 2015-2016 school year. The PSS file is the Department of Education’s primary database of elementary and secondary private schools in the 50 states and the District of Columbia, and it is based on a survey conducted by the U.S. Census Bureau during the 2015-2016 school year. These sampling frames are referred to as the PSS-based sampling frames.
The sampling frame was restricted to schools located in the primary sampling units (PSUs) selected for the NAEP 2018 TEL assessment. In addition, the sampling frame excluded ungraded schools, vocational schools with no enrollment, special-education-only schools, homeschool entities, prison and hospital schools, and juvenile correctional institutions. Vocational schools with no enrollment serve students who split their time between the vocational school and their home school.
The following table presents the number of schools and estimated enrollment for the private school frame for grade 8 for TEL. The unweighted estimated enrollment is restricted to the selected PSUs. The weighted estimated enrollment incorporates the PSU weight (inverse of the probability of selecting the PSU), and thus is a national estimate of the number of private school students in eighth grade.
Number of schools and enrollment in private school sampling frame, technology and engineering literacy (TEL) assessment, grade 8, by affiliation: 2018
-
Affiliation
Number of
schools
Estimated enrollment (unweighted)
Estimated enrollment (weighted)
Total
9,939
192,560
334,324
Catholic
2,531
78,262
134,634
Non-Catholic
5,391
104,213
185,404
Unknown affiliation
2,017
10,085
14,286
NOTE: Detail may not sum to totals due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Technology and Engineering Literacy (TEL) Assessment.
For quality control purposes, school and student counts from the sampling frame were compared to school and student counts from previous private school frames for eighth grade. No major discrepancies were found.
NAEP Technical Documentation Sampling of Private Schools for the 2018 Technology and Engineering Literacy (TEL) Assessment
In the design of each school sample, five objectives underlie the process of determining the probability of selection for each school and how many students are to be sampled from each selected school containing grade-eligible students. The five objectives are
to
meet the
target student
sample size
to
select an
equal-probability sample
of students
to
limit the
number of
students who
are selected
from a
school
to
ensure that
the sample
within a
school does
not include
a very
high percentage
of the
students in
the school,
unless all
students are
included; and
to reduce
the rate
of sampling
of small
schools, in
recognition of
the greater
cost and
burden per
student of
conducting assessments
in such
schools
The goal in determining the school's measure of size is to optimize across the last four objectives in terms of maintaining the accuracy of estimates and the cost- effectiveness of the sample design.
Therefore, to meet the target student sample size objective and achieve a reasonable compromise among the other four objectives, the following algorithm was used to assign a measure of size to each school based on its estimated grade enrollment as indicated on the sampling frame.
The measures of size vary by enrollment size. The initial measures of size (MOS) were set as follows:
For eighth grade:
where Xjs is the estimated grade enrollment for grade j in school s, PSCHWTs = the Private School Universe Survey area frame weight for school s, computed by the U.S. Census Bureau, and PSU_WTs = the PSU weight for school s.
The measures of size for schools in the Honolulu primary sampling unit (PSU) are doubled to increase their chances of selection:
Schools in the Honolulu PSU have their measures of size doubled to ensure at least one sampled school from the PSU. The Honolulu PSU is a certainty not due to its size, but because it is unique.
The next task in this development is to describe bj, the constant of proportionality for each grade. It is a sampling parameter that, when multiplied with a school’s preliminary measure of size (Mjs), yields the school’s final measure of size. It is computed in such a way that, when used with the systematic sampling procedure, the target student sample size is achieved. For private schools, this parameter varied by private school affiliation (Catholic, non-Catholic, and unknown affiliation).
The final measure of size, Ejs, is defined as:
The quantity uj (the maximum number of “hits” allowed) in this formula is designed to put an upper bound on the burden for the sampled schools. For private schools, uj is 1 because by design a school could not be selected, or "hit," in the sampling process more than once within a grade.
In addition, an adjustment was made to the measures of size in the TEL sample to attempt to reduce school burden by minimizing the number of schools selected for 1) both TEL and social sciences and 2) both TEL and the International Computer and Information Literacy Study (ICILS). For TEL, an adaptation of the Keyfitz process was used to compute conditional measures of size that, by their design, minimized the overlap of schools selected for both TEL and either of the other two samples.
Schools were ordered within each jurisdiction using the serpentine sort described under the stratification of private schools. A systematic sample was then drawn using this serpentine sorted list and the measures of size. The number of private schools selected for eighth-grade TEL was approximately 140.
NAEP Technical Documentation Stratification of Private Schools for the 2018 Technology and Engineering Literacy (TEL) Assessment
For the private school sampling frame file, schools were explicitly stratified by private school affiliation (Catholic, non-Catholic, and unknown affiliation). Private school affiliation was unknown for nonrespondents to the NCES Private School Universe Survey (PSS). Within private school type, separate implicit stratification
schemes were used to sort schools in certainty primary sampling units (PSUs) and noncertainty PSUs. In all cases, the implicit stratification was achieved via a "serpentine sort".
Within
each certainty
PSU, the
schools were
hierarchically sorted
by census
region
urbanization
classification (four
categories based
on urban-centric
locale)
estimated
grade enrollment
Schools
in noncertainty
PSUs were
hierarchically sorted
by PSU
stratum,
urbanization
classification (four
categories based
on urban-centric
locale), and
estimated grade
enrollment.
NAEP Technical Documentation Student Sample Selection for the 2018 Private School Technology and Engineering Literacy (TEL) Assessment
Students in private schools were selected in the same way as students in the public schools.
NAEP Technical Documentation Substitute Private Schools for the 2018 Technology and Engineering Literacy (TEL) Assessment
Substitutes were preselected for the private school samples by sorting the school frame file according to the actual order used in the sampling process (the implicit stratification). For operational reasons, the original selection order was embedded within the sampled primary sampling unit (PSU) and state. Each sampled school
had each of its nearest neighbors within the same sampling stratum on the school frame file identified as a potential substitute. Since grade enrollment was used as the last sort ordering variable, the nearest neighbors had grade enrollment values very close to that of the sampled school. This was done to facilitate the selection of about the same number of students within the substitute as would have been selected from the original sampled school.
Schools were disqualified as potential substitutes if they were already selected in any of the original private school samples or assigned as a substitute for another private school (earlier in the sort ordering). Schools assigned as substitutes for eighth-grade social sciences were disqualified as potential substitutes for eighth- grade TEL schools.
If both nearest neighbors were still eligible to be substitutes, the one with a closer grade enrollment was chosen. If both nearest neighbors were equally distant from the sampled school in their grade enrollment (an uncommon occurrence), one of the two was randomly selected.
Of the approximately 140 originally sampled private schools for the eighth-grade TEL assessment, about 50 schools had substitutes activated when the original eligible schools did not participate. Ultimately, about 10 of the activated substitute private schools participated in the TEL assessment.
NAEP Technical Documentation Target Population of the 2018 Private School Technology and Engineering Literacy (TEL) Assessment
The target population for the 2018 TEL private school assessment included all students who were enrolled in eighth grade in private schools located in the 50 states and the District of Columbia.
NAEP Technical Documentation 2018 Public School Social Sciences Assessment
The NAEP 2018 sample design yielded nationally representative samples of public school students in grade 8 for social sciences through a three-stage approach:
Target Population
selection
of primary
sampling units
(PSUs), selection
of schools
within strata,
and selection
of students
within schools.
The sample of schools was selected with probability proportional to a measure of size based on the estimated grade enrollment in the schools.
The 2018 sampling plan was designed to assess 46,800 eighth-graders in public schools for social sciences. These students were allocated among tests in civics, geography, and U.S. history. Target sample sizes were adjusted to reflect expected public school and student response and eligibility.
Schools on the sampling frame were explicitly stratified prior to sampling by PSU type (certainty/noncertainty). Within certainty
Sampling Frame Stratification of Schools Sampling of Schools Substitute Schools Ineligible Schools Student Sample Selection
PSUs, schools were implicitly stratified by census region, urbanization classification, race/ethnicity stratum, and estimated grade enrollment. Within noncertainty PSUs, schools were implicitly stratified by PSU stratum, urbanization classification, and race/ethnicity percentage.
From the stratified frame of public schools, systematic random samples of eighth-grade schools were drawn with probability proportional to a measure of size based on the estimated grade enrollment of the school in the relevant grade.
Additionally, American Indian, Alaska Native, Black, and Hispanic students were oversampled at moderate rates as follows. First, schools in a high American Indian/Alaska Native stratum (i.e., schools with more than five percent American Indian and Alaska Native students and at least five American Indian or Alaska Native students in the sample grade) were sampled at four times the rate (by quadrupling their measure of size) as schools not in a high American Indian/Alaska Native stratum to implement oversampling of American Indian and Alaska Native students. Second, schools not in a high American Indian/Alaska Native stratum but in a high Black/Hispanic stratum (i.e., schools that were not oversampled for American Indian and Alaska Native students and with more than 15 percent Black and Hispanic students and at least 10 Black or Hispanic students in the sample grade) were sampled at twice the rate (by doubling their measure of size) as schools not in a high Black/Hispanic stratum to implement oversampling of Black and Hispanic students.
Finally, schools in the Honolulu PSU were oversampled at twice the rate (by doubling their measure of size) as schools not in the Honolulu PSU. This was done to ensure at least one school was sampled from this PSU. The PSU was selected with certainty not due to its size, but because it is unique.
Each selected school in the public school sample provided a list of eligible enrolled students from which a systematic sample of students was drawn. Within each school, students were selected with equal probability.
NAEP Technical Documentation Ineligible Public Schools for the 2018 Social Sciences Assessment
The Common Core of Data (CCD) public school file from which most of the sampled schools were drawn corresponds to the 2015-2016 school year, two years prior to the assessment school year. During the intervening period, some of these schools either closed, no longer offered the grade of interest, or became ineligible for other reasons. In such cases, the sampled schools were considered to be ineligible.
The table below presents unweighted counts of sampled public schools by eligibility status, including the reason for ineligibility.
Number of sampled public schools, social sciences assessment, grade 8, by eligibility status: 2018
-
Eligibility status
Unweighted count of schools
Unweighted percentage
All eighth-grade sampled public schools
800
100.00
Eligible schools
760
95.00
No eligible students in grade
4
0.50
Does not have sampled grade
14
1.75
School closed
5
0.63
Not a regular school
10
1.25
Other ineligible school
0
0.00
Duplicate on sampling frame
0
0.00
NOTE: Total and eligible school counts are rounded to nearest ten. Percentages are based on rounded counts. Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
NAEP Technical Documentation Sampling Frame for the 2018 Public School Social Sciences Assessment
Drawing the school samples for the 2018 assessment required a comprehensive list of public schools in each jurisdiction containing information for stratification purposes. As in previous NAEP assessments, the Common Core of Data (CCD) file developed by NCES was used to construct the sampling frame. The CCD file corresponding to the 2015-2016 school year provided the frame for all regular public, state-operated public, Bureau of Indian Education (BIE), and Department of Defense Education Activity (DoDEA) schools in the 50 states and the District of Columbia.
New-School Sampling Frame
The sampling frame was restricted to schools located in the primary sampling units (PSUs) selected for the NAEP 2018 social sciences assessment. In addition, the sampling frame excluded ungraded schools, vocational schools with no enrollment, special-education-only schools, homeschool entities, prison or hospital schools, and juvenile correctional institutions. Vocational schools with no enrollment serve students who split their time between the vocational school and their home school.
The public school frame for the social sciences assessment contained approximately 13,200 schools. The estimated eighth-grade enrollment (unweighted) for these schools was 2.15 million and the estimated eighth-grade enrollment (weighted) was 3.71 million. The unweighted estimated enrollment is restricted to the selected PSUs for social sciences. The weighted estimated enrollment incorporates the PSU weight (inverse of the probability of selecting the PSU), and thus is a national estimate of the number of public school students in eighth grade.
For quality control purposes, school and student counts from the sampling frame were compared to school and student counts from previous public school frames for eighth grade. No major discrepancies were found.
NAEP Technical Documentation New-School Sampling Frame for the 2018 Public School Social Sciences Assessment
The most current Common Core of Data (CCD) file available was used to construct the public school frame for NAEP 2018. However, the information on that file was two years out of date by the time of the NAEP assessment. During that two-year period, some schools closed, others changed grade span, and still others came into existence.
One can improve coverage by asking districts to provide information on currently open schools that were not listed in the 2015–2016 CCD file used to create the NAEP public school frame, and also to report grade span changes that may have caused a CCD-listed school to become newly eligible for eighth grade. Asking all districts to do this would have imposed an undue burden, so instead, a random sample of districts was contacted to obtain lists of new and newly eligible schools. The goal was to allow every new or newly eligible school a chance of selection, thereby fully covering the target population of schools in operation during the 2017–2018 school year.
The first step in this process was the development of a new-school frame through the construction of a district-level file from the CCD school-level file. The new- school frames for both social sciences and technical engineering literacy (TEL) were constructed at the same time. Since the social sciences and TEL assessments were to be conducted within a total of 138 primary sampling units (PSUs), only districts that fell within the boundaries of those PSUs were eligible for sampling. Once the district-level file was subset to just the targeted PSUs, it was divided into three files: the first containing state-operated and charter school districts, the second containing small districts, and the third containing large districts.
State-operated districts and districts containing no schools other than charter schools require special handling. In survey years when state-level assessments are conducted, NAEP State Coordinators are asked to provide the names of all new charter-only and state-run schools. However, these types of school districts tend not to be geographically compact, and it is not feasible to link such a district to a single PSU, except at the individual school level. The smaller the proportion of a
state’s population falling within sampled PSUs, the less likely that a specific new school of this type will be added to the frame and the more likely that state personnel will have expended unnecessary effort in providing updated information that will not be used. For this reason, for the NAEP 2018 assessment, the charter-only and state-run district component of the new school procedure was implemented only in states where more than 60 percent of youth fell within sampled PSUs. This meant that this component of the new-school sampling frame procedure was implemented in 21 states plus the District of Columbia, which taken together contain about 67 percent of the nation's youth.
The remaining districts were classified as small or large. A small district usually contains no more than three schools on the frame in total, with no more than one school at each targeted grade (fourth, eighth, and twelfth). However, for NAEP 2018 new schools were only selected for grade 8 assessments. Therefore, for NAEP 2018, a small district contains no more than one school with grade 8 on the frame in total. New schools in small districts were identified during school recruitment and added to the sample if the frame school in the same district was sampled for eighth grade. From a sampling perspective, the new school was viewed as an “annex” to the sampled school that had a well-defined probability of selection equal to that of the frame school. Thus when the frame school was sampled in a small district for eighth grade, any new school was automatically sampled for eighth grade as well.
Large districts were divided into 77 strata based on the NAEP 2018 PSU sampling strata, with districts in certainty PSUs grouped together in a single stratum. The district sample was allocated to each of the 77 strata proportional to the percent of the U.S. population of eighth-graders contained in that stratum, with the caveat that each stratum had to be allocated at least one district. This allocation was then adjusted because it resulted in too many districts in the certainty strata and not enough in the noncertainty strata. Once the allocation to each stratum had been fixed, districts were sampled from a sorted list using systematic sampling with probability proportional to size and a random start, with the measure of size being the number of eighth-graders enrolled in the district. Within the certainty PSU stratum, districts were sorted in a serpentine manner by state and measure of size prior to sampling. In all other strata the districts were sorted by measure of size alone. District selection probabilities were retained and used in all subsequent stages of sampling and weighting.
The selected districts were then sent a listing of all their schools that appeared on the 2015–2016 CCD file and were asked to provide information about any schools missing from CCD, and grade span changes of existing schools. This information provided by the sampled districts was used to construct sampling frames for the selection of new or newly eligible public schools and also for updating the status of existing schools (e.g., school closings). This process was conducted through the NAEP State Coordinator in each jurisdiction. The coordinators were sent the information for all sampled districts in their respective states and were responsible for returning the completed updates.
The eligibility of a school was determined based on the grade span and whether it was located in a sampled PSU. A school was also classified as "newly eligible" if a change of grade span had occurred such that the school status changed from ineligible to eligible at eighth grade.
This process yielded 307 schools on the eighth-grade new school sampling frame for social sciences. These schools contained an estimated 19,143 eighth-grade students.
NAEP Technical Documentation Sampling of Public Schools for the 2018 Social Sciences Assessment
In the design of each school sample, six objectives underlie the process of determining the probability of selection for each school and the number of students to be sampled from each selected school containing grade-eligible students. The six objectives are
to
meet the
overall target
student sample
size
to
select an
equal-probability sample
of students
to
limit the
number of
students selected
from any
one school
to
ensure that
the sample
within a
school does
not include
a very
high percentage
of the
students in
the school,
unless all
students are
included
to
reduce the
rate of
sampling of
small schools,
in recognition
of the
greater cost
and burden
per student
of conducting
assessments in
such schools;
and to
increase the
number of
American Indian/Alaska
Native (AIAN),
Black, and
Hispanic students
in the
sample
The goal in determining the school's measure of size is to optimize across the middle four objectives in terms of maintaining the accuracy of estimates and the cost- effectiveness of the sample design.
Therefore, to meet the target student sample size objective and achieve a reasonable compromise among the next four objectives, the following algorithm was used to assign a measure of size to each school based on its estimated grade enrollment as indicated on the sampling frame.
The measures of size vary by enrollment size. The initial measures of size (MOS) were set as follows:
For eighth grade
where Xjs is the estimated grade enrollment for grade j in school s, and PSU_WTs is the PSU weight for school s.
A school with more than 5 percent AIAN students and at least 5 AIAN students in the sample grade is in the high AIAN stratum for NAEP. The measures of size for schools in the high AIAN stratum are quadrupled to increase their chances of selection. A school that is not in the high AIAN stratum and with more than 15 percent Black and Hispanic students and at least 10 Black or Hispanic students in the sample grade is in the high Black/Hispanic stratum for NAEP. The measures of size for schools in the high Black/Hispanic stratum or in the Honolulu primary sampling unit (PSU) are doubled to increase their chances of selection:
Schools in the Honolulu PSU have their measures of size doubled to ensure at least one sampled school from the PSU. The Honolulu PSU is a certainty not due to its size, but because it is unique.
The next task in this development is to describe bj, the constant of proportionality for each grade. It is a sampling parameter that, when multiplied with a school’s preliminary measure of size (Mjs), yields the school’s final measure of size. It is computed in such a way that, when used with the systematic sampling procedure, the target student sample size is achieved. For social sciences public schools, bj is 0.000112622 for eighth grade.
The final measure of size is defined as:
The quantity uj (the maximum number of “hits” allowed) in this formula is designed to put an upper bound on the burden for the sampled schools. For public schools, uj is 1 because by design a school could not be selected, or "hit," in the sampling process more than once within a grade.
In addition, new and newly-eligible schools were sampled from the new school frame. The final measure of size for these schools is defined as:
The variable πdjs is the probability of selection of the district d into the new-school district sample.
Schools were ordered within each jurisdiction using the serpentine sort described under the stratification of public schools. A systematic sample was then drawn using this serpentine-sorted list and the measures of size. The number of public schools selected for social sciences was approximately 800, including approximately 10 new or newly-eligible schools.
NAEP Technical Documentation Stratification of Public Schools for the 2018 Social Sciences Assessment
For the public school sampling frame file, separate implicit stratification schemes were used to sort schools in certainty primary sampling units (PSUs) and noncertainty PSUs. The implicit stratification was achieved via a "serpentine sort."
For certainty PSUs, the schools were hierarchically sorted by
census
region
urbanization
classification (four
categories based
on urban-centric
locale), race/ethnicity
stratum
estimated
grade enrollment
If there were less than six expected sampled schools for a particular urbanization classification cell (nested within the census region), the cell was collapsed with a neighboring urbanization classification cell. If the expected sampled schools exceeded 12, then the race/ethnicity strata were defined based on the total percentage of Black, Hispanic, and American Indian/Alaska Native students. The strata were defined so that there were at least six expected sampled schools for each race/ethnicity stratum. If the urbanization classification stratum had an expected sample size less than 12, no race/ethnicity strata were generated, and the final sort variable was total percentage of Black, Hispanic, and American Indian/Alaska Native students rather than estimated grade enrollment.
Schools
in noncertainty
PSUs were
hierarchically sorted
by PSU
stratum
urbanization
classification (four
categories based
on urban-centric
locale)
percentage
of Black,
Hispanic, and
American Indian/Alaska
Native students
The collapsing of cells within the noncertainty PSUs was implemented in a fashion similar to that described for certainty PSUs.
NAEP Technical Documentation Student Sample Selection for the 2018 Public School Social Sciences Assessment
All eighth-grade students in the school were sampled if the school had 80 or fewer students in that grade. Otherwise, a sample of 75 students was selected without replacement.
The assessments were conducted in three session types: digitally-based (DBA) civics/geography/U.S. history, paper-based (PBA) geography/U.S. history, and PBA civics. No more than two session types were assigned to any one school. In schools with fewer than 24 eighth-graders, only one session type, assigned randomly, was conducted. In schools with 24 or more eighth-graders, the DBA session type plus one of the PBA session types, assigned randomly, were conducted.
Assignment to subject within a given session type was done through spiraling of booklets (for PBA) or test forms (for DBA). Session type and subject assignment were carried out in a coordinated fashion, with approximately 4 in 13 selected students assigned to geography, 5 in 13 selected students assigned to U.S. history, and 4 in 13 selected students assigned to civics.
The process of list submission, sampling students from year-round schools, sampling newly identified students (including new enrollees), and determining student eligibility and exclusion status was the same as the process used for the NAEP 2017 state student samples.
NAEP Technical Documentation Substitute Public Schools for the 2018 Social Sciences Assessment
Substitutes were preselected for the public school samples by sorting the school frame file according to the actual order used in the sampling process (the implicit stratification). For operational reasons, the original selection order was embedded within the sampled primary sampling unit (PSU) and state. Each sampled school had each of its nearest neighbors within the same sampling stratum on the school frame file identified as a potential substitute. When grade enrollment was used as the last sort ordering variable, the nearest neighbors had grade enrollment values very close to that of the sampled school. This was done to facilitate the selection of about the same number of students within the substitute as would have been selected from the original sampled school.
Schools were disqualified as potential substitutes if they were already selected in any of the original public school samples or assigned as a substitute for another public school (earlier in the sort ordering).
If both nearest neighbors were still eligible to be substitutes, the one with a closer grade enrollment was chosen. If both nearest neighbors were equally distant from the sampled school in their grade enrollment (an uncommon occurrence), one of the two was randomly selected.
Of the approximately 800 originally sampled public schools for the eighth-grade social sciences assessments, about 80 schools had a substitute activated because the original eligible school did not participate. Ultimately, less than 10 of the activated substitute public schools participated in a social sciences assessment.
NAEP Technical Documentation Target Population of the 2018 Public School Social Sciences Assessment
The target populations for the 2018 civics, geography, and U.S. history public school assessments included all students who were enrolled in eighth grade in public schools, Bureau of Indian Education (BIE) schools, and Department of Defense Education Activity (DoDEA) schools located in the 50 states and the District of Columbia.
NAEP Technical Documentation 2018 Public School Technology and Engineering Literacy (TEL) Assessment
The NAEP 2018 sample design yielded nationally representative samples of public school students in grade 8 for TEL through a three-stage approach:
selection
of primary
sampling units
(PSUs), selection
of schools
within strata,
and selection
of students
within schools.
The sample of schools was selected with probability proportional to a measure of size based on the estimated grade enrollment in the schools.
The 2018 sampling plan was designed to assess 14,400 eighth-graders in public schools for TEL. Target sample sizes were adjusted to reflect expected public school and student response and eligibility.
Schools on the sampling frame were explicitly stratified prior to sampling by PSU type (certainty/noncertainty). Within certainty PSUs, schools were implicitly stratified by census region, urbanization classification, race/ethnicity stratum, and estimated grade enrollment. Within noncertainty PSUs, schools were implicitly stratified by PSU stratum, urbanization classification, and race/ethnicity percentage.
Target Population Sampling Frame Stratification of Schools Sampling of Schools Substitute Schools Ineligible Schools Student Sample Selection
From the stratified frame of public schools, systematic random samples of eighth-grade schools were drawn with probability proportional to a measure of size based on the estimated grade enrollment of the school in the relevant grade.
Additionally, American Indian, Alaska Native, Black, and Hispanic students were oversampled at moderate rates as follows. First, schools in a high American Indian/Alaska Native stratum (i.e., schools with more than five percent American Indian and Alaska Native students and at least five American Indian or Alaska Native students in the sample grade) were sampled at four times the rate (by quadrupling their measure of size) as schools not in a high American Indian/Alaska Native stratum to implement oversampling of American Indian and Alaska Native students. Second, schools not in a high American Indian/Alaska Native stratum but in a high Black/Hispanic stratum (i.e., schools that were not oversampled for American Indian and Alaska Native students and with more than 15 percent Black and Hispanic students and at least 10 Black or Hispanic students in the sample grade) were sampled at twice the rate (by doubling their measure of size) as schools not in a high Black/Hispanic stratum to implement oversampling of Black and Hispanic students.
Finally, schools in the Honolulu PSU were oversampled at twice the rate (by doubling their measure of size) as schools not in the Honolulu PSU. This was done to ensure at least one school was sampled from this PSU. The PSU was selected with certainty not due to its size, but because it is unique.
Each selected school in the public school sample provided a list of eligible enrolled students from which a systematic sample of students was drawn. Within each school, students were selected with equal probability.
NAEP Technical Documentation Ineligible Public Schools for the 2018 Technology and Engineering Literacy (TEL) Assessment
The Common Core of Data (CCD) public school file from which most of the sampled schools were drawn corresponds to the 2015–2016 school year, two years prior to the assessment school year. During the intervening period, some of these schools either closed, no longer offered the grade of interest, or became ineligible for other reasons. In such cases, the sampled schools were considered to be ineligible.
The table below presents unweighted counts of sampled public schools by eligibility status, including the reason for ineligibility.
Number of sampled public schools, technology and engineering literacy (TEL) assessment, grade 8, by eligibility status: 2018
-
Eligibility status
Unweighted count of schools
Unweighted percentage
All eighth-grade sampled public schools
620
100.00
Eligible schools
590
95.16
No eligible students in grade
2
0.32
Does not have sampled grade
10
1.61
School closed
4
0.65
Not a regular school
8
1.29
Other ineligible school
1
0.16
Duplicate on sampling frame
0
0.00
NOTE: Total and eligible school counts are rounded to nearest ten. Percentages are based on rounded counts. Detail may not sum to total due to rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Technology and Engineering Literacy (TEL) Assessment.
NAEP Technical Documentation Sampling Frame for the 2018 Public School Technology and Engineering Literacy (TEL) Assessment
Drawing the school samples for the 2018 TEL assessment required a comprehensive list of public schools in each jurisdiction containing information for stratification purposes. As in previous NAEP assessments, the Common Core of Data (CCD) file developed by NCES was used to construct the sampling frame. The CCD file corresponding to the 2015–2016 school year provided the frame for all regular public, state-operated public, Bureau of Indian Education (BIE), and Department of Defense Education Activity (DoDEA) schools in the 50 states and the District of Columbia.
New-School Sampling Frame
The sampling frame was restricted to schools located in the primary sampling units (PSUs) selected for the NAEP 2018 TEL assessment. In addition, the sampling frame excluded ungraded schools, vocational schools with no enrollment, special-education-only schools, homeschool entities, prison or hospital schools, and juvenile correctional institutions. Vocational schools with no enrollment serve students who split their time between the vocational school and their home school.
The public school frame for TEL contained approximately 11,400 schools. The estimated eighth-grade enrollment (unweighted) for these schools was 1.88 million and the estimated eighth-grade enrollment (weighted) was 3.70 million. The unweighted estimated enrollment is restricted to the selected PSUs for TEL. The weighted estimated enrollment incorporates the PSU weight (inverse of the probability of selecting the PSU), and thus is a national estimate of the number of public school students in eighth grade.
For quality control purposes, school and student counts from the sampling frame were compared to school and student counts from previous public school frames for eighth grade. No major discrepancies were found.
NAEP Technical Documentation New School Sampling Frame for the 2018 Public School Technology and Engineering Literacy (TEL) Assessment
The new school sampling frame for NAEP 2018 was developed for both the social sciences and TEL assessments at the same time. Thus the details of the new school sampling frame for TEL are described in the new-school sampling frame for social sciences.
This process yielded 271 schools on the eighth-grade new school sampling frame for TEL. These schools contained an estimated 16,826 eighth-grade students.
NAEP Technical Documentation Sampling of Public Schools for the 2018 Technology and Engineering Literacy (TEL) Assessment
In the design of each school sample, five objectives underlie the process of determining the probability of selection for each school and the number of students to be sampled from each selected school containing grade-eligible students. The six objectives are
to
meet the
target student
sample size;
to
select an
equal-probability sample
of students;
to
limit the
number of
students that
are selected
from a
school;
to
ensure that
the sample
within a
school does
not include
a very
high percentage
of the
students in
the school,
unless all
students are
included;
to
reduce the
rate of
sampling of
small schools,
in recognition
of the
greater cost
and burden
per student
of conducting
assessments in
such schools;
and
to
increase the
number of
American Indian/Alaska
Native (AIAN),
Black, and
Hispanic students
in the
sample.
The goal in determining the school's measure of size is to optimize across the middle four objectives in terms of maintaining the accuracy of estimates and the cost- effectiveness of the sample design.
Therefore, to meet the target student sample size objective and achieve a reasonable compromise among the next four objectives, the following algorithm was used to assign a measure of size to each school based on its estimated grade enrollment as indicated on the sampling frame.
The measures of size vary by enrollment size. The initial measures of size (MOS) were set as follows: For eighth grade
where Xjs is the estimated grade enrollment for grade j in school s, and PSU_WTs is the PSU weight for school s.
A school with more than 5 percent AIAN students and at least 5 AIAN students in the sample grade is in the high AIAN stratum for NAEP. The measures of size for schools in the high AIAN stratum are quadrupled to increase their chances of selection. A school that is not in the high AIAN stratum and with more than 15 percent Black and Hispanic students and at least 10 Black or Hispanic students in the sample grade is in the high Black/Hispanic stratum for NAEP. The measures of size for schools in the high Black/Hispanic stratum or in the Honolulu primary sampling unit (PSU) are doubled to increase their chances of selection:
Schools in the Honolulu PSU have their measures of size doubled to ensure at least one sampled school from the PSU. The Honolulu PSU is a certainty not due to its size, but because it is unique.
The next task in this development is to describe bj, the constant of proportionality for each grade. It is a sampling parameter that, when multiplied with a school’s preliminary measure of size (Mjs), yields the school’s final measure of size. It is computed in such a way that, when used with the systematic sampling procedure, the target student sample size is achieved. For TEL public schools, bj is 0.000099771 for eighth grade.
The final measure of size, Ejs, is defined as:
The quantity uj (the maximum number of “hits” allowed) in this formula is designed to put an upper bound on the burden for the sampled schools. For public schools, uj is 1 because by design a school could not be selected, or "hit," in the sampling process more than once within a grade.
In addition, new and newly-eligible schools were sampled from the new school frame. The final measure of size for these schools is defined as:
The variable πdjs is the probability of selection of the district d into the new-school district sample.
In addition, an adjustment was made to the measures of size in the TEL sample to attempt to reduce school burden by minimizing the number of schools selected for 1) both TEL and social sciences and 2) both TEL and the International Computer and Information Literacy Study (ICILS). For TEL, an adaptation of the Keyfitz process was used to compute conditional measures of size that, by their design, minimized the overlap of schools selected for both TEL and either of the other two samples.
Schools were ordered within each jurisdiction using the serpentine sort described under the stratification of public schools. A systematic sample was then drawn using this serpentine-sorted list and the measures of size. The number of public schools selected for TEL was approximately 620, including approximately 10 new or newly-eligible schools.
NAEP Technical Documentation Stratification of Public Schools for the 2018 Technology and Engineering Literacy (TEL) Assessment
For the public school sampling frame file, separate implicit stratification schemes were used to sort schools into certainty primary sampling units (PSUs) and noncertainty PSUs. The implicit stratification was achieved via a "serpentine sort."
For
certainty PSUs,
the schools
were hierarchically
sorted by
census region,
urbanization
classification (four
categories based
on urban-centric
locale),
race/ethnicity
stratum, and
estimated grade
enrollment.
If there were less than six expected sampled schools for a particular urbanization classification cell (nested within the census region), the cell was collapsed with a neighboring urbanization classification cell. If the expected number of sampled schools exceeded 12, then the race/ethnicity strata were defined based on the percentages of Black, Hispanic, and American Indian/Alaska Native students. The strata were defined so that there were at least six expected sampled schools for each race/ethnicity stratum. If the urbanization classification stratum had an expected sample size less than 12, no race/ethnicity strata were generated, and the final sort variable was the total percentage of Black, Hispanic, and American Indian/Alaska Native students rather than estimated grade enrollment.
Schools in noncertainty PSUs were hierarchically sorted by
PSU
stratum,
urbanization
classification (four
categories based
on urban-centric
locale), and
percentage of
Black, Hispanic,
and American
Indian/Alaska Native
students.
The collapsing of cells within the noncertainty PSUs was implemented in a fashion similar to that described for certainty PSUs.
NAEP Technical Documentation Student Sample Selection for the 2018 Public School Technology and Engineering Literacy (TEL) Assessment
All eighth-grade students in the school were sampled if the school had 30 or fewer students in that grade. Otherwise, a sample of 30 students was selected without replacement.
The process of list submission, sampling students from year-round schools, sampling newly identified students (including new enrollees), and determining student eligibility and exclusion status was the same as the process used for the NAEP 2017 state student samples.
NAEP Technical Documentation Substitute Public Schools for the 2018 Technology and Engineering Literacy (TEL) Assessment
Substitutes were preselected for the public school samples by sorting the school frame file according to the actual order used in the sampling process (the implicit stratification). For operational reasons, the original selection order was embedded within the sampled primary sampling unit (PSU) and state. Each sampled school had each of its nearest neighbors within the same sampling stratum on the school frame file identified as a potential substitute. When grade enrollment was used as the last sort ordering variable, the nearest neighbors had grade enrollment values very close to that of the sampled school. This was done to facilitate the selection of about the same number of students within the substitute as would have been selected from the original sampled school.
Schools were disqualified as potential substitutes if they were already selected in any of the original public school samples or assigned as a substitute for another public school (earlier in the sort ordering). Schools assigned as substitutes for eighth-grade social sciences were disqualified as potential substitutes for eighth- grade TEL schools.
If both nearest neighbors were still eligible to be substitutes, the one with a closer grade enrollment was chosen. If both nearest neighbors were equally distant from the sampled school in their grade enrollment (an uncommon occurrence), one of the two was randomly selected.
Of the approximately 620 originally sampled public schools for the eighth-grade TEL assessment, about 50 schools had a substitute activated because the original eligible school did not participate. Ultimately, less than 10 of the activated substitute public schools participated in the TEL assessment.
NAEP Technical Documentation Target Population of the 2018 Public School Technology and Engineering Literacy (TEL) Assessment
The target population for the 2018 TEL public school assessment included all students who were enrolled in eighth grade in public schools, Bureau of Indian Education (BIE) schools, and Department of Defense Education Activity (DoDEA) schools located in the 50 states and the District of Columbia.
NAEP Technical Documentation School and Student Participation Results for the 2018 Assessment
Participation in NAEP is not mandatory. Although a portion of the participating school sample consisted of substitute schools, it is preferable to calculate school response rates on the basis of school participation before substitution.
In
every NAEP
survey, some
of the
sampled students
are not
assessed. Examples
of such
students are
as follows:
withdrawn students,
School response rates for the Civics assessment
School response rates for the Geography assessment
excluded
students with
disabilities (SD),
excluded English
learner (EL)
students, or
students
absent from
both the
original session
and the
makeup session
(not excluded
but not
assessed).
Withdrawn students are those who have left the school before the original assessment. Excluded students were determined by their school to be unable to meaningfully take the NAEP assessment in their assigned subject, even with an accommodation. Excluded students must also be classified as SD and/or EL. Other students who were absent for the initial session are assessed in the makeup session. The last category includes students who were not excluded (i.e., were to be assessed) but were not assessed, either due to absence from both sessions or because of a refusal to participate. Assessed students are also classified as assessed without an accommodation or assessed with an accommodation. The latter group can be divided into SD students assessed with an accommodation, EL students assessed with an accommodation, or students who are both SD and EL and accommodated. Note that some SD and EL students are assessed without accommodations, and students who are neither SD nor EL can only be assessed without an accommodation.
The weighted student response rates utilize the student base weights and indicate the weighted percentage of assessed students among all students to be assessed. The exclusion rates, in contrast, provide the weighted percentage of excluded SD or EL students among all eligible students, i.e., absent, assessed, and excluded students.
School response rates for the U.S. History assessment
School response rates for the Technology and Engineering Literacy (TEL) assessment
Student response and exclusion rates for the Civics assessment
Student response and exclusion rates for the Geography assessment
Student response and exclusion rates for the U.S. History assessment
Student response and exclusion rates for the Technology and Engineering Literacy (TEL) assessment
NAEP Technical Documentation School Response Rates for the 2018 Civics Assessment
The following table presents counts of eligible sampled schools and participating schools, as well as weighted school response rates, for the 2018 civics assessment. The weighted school response rates estimate the proportion of the student population that is represented by the participating school sample prior to substitution.
School response counts and rates for public and private schools before substitution, civics assessment at grade 8, by school type, geographic region, and affiliation: 2018
-
School type and
geographic region/affiliation
Number of eligible sampled schools
Number of participating schools
Weighted school response rate prior to substitution (percent)
-
School type and
geographic region/affiliation
Number of eligible sampled schools
Number of participating schools
Weighted school response rate prior to substitution (percent)
National all1 1,030 800 81.82
-
National public
760
640
84.44
Northeast public
110
100
94.60
Midwest public
150
110
71.19
South public
310
260
86.13
West public
200
170
86.37
National private
270
150
48.66
Catholic
80
70
85.85
Non-Catholic
190
80
27.28
1Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States.
NOTE: National public includes students from public schools only. It includes charter schools, but excludes Bureau of Indian Education schools and Department of Defense Education Activity schools. It is used when comparing national data to those of states, urban districts, or regions. School counts are rounded to nearest ten. Detail may not sum to totals because of rounding. Percentages are based on unrounded counts.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
NAEP Technical Documentation School Response Rates for the 2018 Geography Assessment
The following table presents counts of eligible sampled schools and participating schools, as well as weighted school response rates, for the 2018 geography assessment. The weighted school response rates estimate the proportion of the student population that is represented by the participating school sample prior to substitution.
School response counts and rates for public and private schools before substitution, geography assessment at grade 8, by school type, geographic region, and affiliation: 2018
-
School type and
geographic region/affiliation
Number of eligible sampled schools
Number of participating schools
Weighted school response rate prior to substitution (percent)
National all1 1,030 800 81.82
-
National public
760
640
84.44
Northeast public
110
100
94.60
Midwest public
150
110
71.19
South public
310
260
86.13
West public
200
170
86.37
National private
270
150
48.66
Catholic
80
70
85.85
Non-Catholic
190
80
27.28
1Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States.
NOTE: National public includes students from public schools only. It includes charter schools, but excludes Bureau of Indian Education schools and Department of Defense Education Activity schools. It is used when comparing national data to those of states, urban districts, or regions. School counts are rounded to nearest ten. Detail may not sum to totals because of rounding. Percentages are based on unrounded counts.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
NAEP Technical Documentation School Response Rates for the 2018 Technology and Engineering Literacy (TEL) Assessment
The following table presents counts of eligible sampled schools and participating schools, as well as weighted school response rates, for the 2018 TEL assessment. The weighted school response rates estimate the proportion of the student population that is represented by the participating school sample prior to substitution.
School response counts and rates for public and private schools before substitution, technology and engineering literacy (TEL) assessment at grade 8, by school type, geographic region, and affiliation: 2018
-
School type and
geographic region/affiliation
Number of eligible sampled schools
Number of participating schools
Weighted school response rate prior to substitution (percent)
-
School type and
geographic region/affiliation
Number of eligible sampled schools
Number of participating schools
Weighted school response rate prior to substitution (percent)
National all1 710 580 86.42
-
National public
590
520
88.79
Northeast public
90
80
96.29
Midwest public
110
90
80.96
South public
240
220
91.01
West public
150
130
86.80
National private
120
60
58.38
Catholic
30
30
89.82
Non-Catholic
90
30
36.19
1Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States.
NOTE: National public includes students from public schools only. It includes charter schools, but excludes Bureau of Indian Education schools and Department of Defense Education Activity schools. It is used when comparing national data to those of states, urban districts, or regions. School counts are rounded to nearest ten. Detail may not sum to totals because of rounding. Percentages are based on unrounded counts.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Technology and Engineering Literacy (TEL) Assessment.
NAEP Technical Documentation School Response Rates for the 2018 U.S. History Assessment
The following table presents counts of eligible sampled schools and participating schools, as well as weighted school response rates, for the 2018 U.S. history assessment. The weighted school response rates estimate the proportion of the student population that is represented by the participating school sample prior to substitution.
School response counts and rates for public and private schools before substitution, U.S. history assessment at grade 8, by school type, geographic region, and affiliation: 2018
-
School type and
geographic region/affiliation
Number of eligible sampled schools
Number of participating schools
Weighted school response rate prior to substitution (percent)
National all1 1,030 800 81.82
-
National public
760
640
84.44
Northeast public
110
100
94.60
Midwest public
150
110
71.19
South public
310
260
86.13
West public
200
170
86.37
National private
270
150
48.66
Catholic
80
70
85.85
Non-Catholic
190
80
27.28
1Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States.
NOTE: National public includes students from public schools only. It includes charter schools, but excludes Bureau of Indian Education schools and Department of Defense Education Activity schools. It is used when comparing national data to those of states, urban districts, or regions. School counts are rounded to nearest ten. Detail may not sum to totals because of rounding. Percentages are based on unrounded counts.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
NAEP Technical Documentation Student Response and Exclusion Rates for the 2018 Civics Assessment
The following table presents the weighted student response and exclusion rates for the 2018 civics assessment. The exclusion rates give the percentage excluded, among all eligible (i.e., assessed, absent, or excluded) students. Excluded students must necessarily be either students with disabilities (SD) or English learners (EL). The response rates indicate the percentage of students assessed among those who it was intended would take the assessment from within the participating schools. Thus, students who were excluded are not included in the denominators of the response rates.
Weighted student response and exclusion rates for public and private schools, civics assessment at grade 8, by school type, geographic region, and affiliation: 2018
-
School type and
geographic region/affiliation
Weighted student response rates (percent)
Weighted percent of all eligible students
who are SD and excluded
Weighted percent of all eligible students
who are EL and excluded
National all1 92.01 1.42 0.56
-
National public
91.96
1.54
0.60
Northeast public
89.34
1.30
0.95
Midwest public
92.44
1.30
0.41
South public
92.61
2.08
0.49
West public
92.50
1.07
0.70
National private
92.93
0.03
0.03
Catholic
94.64
0.08
0.08
Non-Catholic
90.60
0.00
0.00
1Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States.
NOTE: National public includes students from public schools only. It includes charter schools, but excludes Bureau of Indian Education schools and Department of Defense Education Activity schools. It is used when comparing national data to those of states, urban districts, or regions. SD = students with disabilities; EL = English learners.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
NAEP Technical Documentation Student Response and Exclusion Rates for the 2018 Geography Assessment
The following table presents the weighted student response and exclusion rates for the 2018 geography assessment. The exclusion rates give the percentage excluded, among all eligible (i.e., assessed, absent, or excluded) students. Excluded students must necessarily be either students with disabilities (SD) or English learners (EL). The response rates indicate the percentage of students assessed among those who it was intended would take the assessment from within the participating schools. Thus, students who were excluded are not included in the denominators of the response rates.
Weighted student response and exclusion rates for public and private schools, geography assessment at grade 8, by school type, geographic region, and affiliation: 2018
-
School type and
geographic region/affiliation
Weighted student response rates (percent)
Weighted percent of all eligible students
who are SD and excluded
Weighted percent of all eligible students
who are EL and excluded
National all1 92.03 1.30 0.68
-
National public
91.95
1.39
0.72
Northeast public
90.04
1.20
1.53
Midwest public
92.84
1.50
0.36
South public
92.49
1.61
0.67
West public
91.83
1.07
0.59
National private
93.41
0.21
0.11
Catholic
94.22
0.00
0.00
Non-Catholic
92.08
0.35
0.17
1Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States.
NOTE: National public includes students from public schools only. It includes charter schools, but excludes Bureau of Indian Education schools and Department of Defense Education Activity schools. It is used when comparing national data to those of states, urban districts, or regions. SD = students with disabilities; EL = English learners.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
NAEP Technical Documentation Student Response and Exclusion Rates for the 2018 Technology and Engineering Literacy (TEL) Assessment
The following table presents the weighted student response and exclusion rates for the 2018 TEL assessment. The exclusion rates give the percentage excluded, among all eligible (i.e., assessed, absent, or excluded) students. Excluded students must necessarily be either students with disabilities (SD) or English learners (EL). The response rates indicate the percentage of students assessed among those who it was intended would take the assessment from within the participating schools. Thus, students who were excluded are not included in the denominators of the response rates.
Weighted student response and exclusion rates for public and private schools, technology and engineering literacy (TEL) assessment at grade 8, by school type, geographic region, and affiliation: 2018
-
School type and
geographic region/affiliation
Weighted student response rates (percent)
Weighted percent of all eligible students
who are SD and excluded
Weighted percent of all eligible students
who are EL and excluded
National all1 93.16 1.05 0.59
-
National public
93.15
1.14
0.64
Northeast public
91.37
1.42
1.30
Midwest public
93.36
0.81
0.49
South public
93.50
1.11
0.51
West public
93.69
1.24
0.52
National private
93.38
0.00
0.00
Catholic
94.54
0.00
0.00
Non-Catholic
91.70
0.00
0.00
1Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States.
NOTE: National public includes students from public schools only. It includes charter schools, but excludes Bureau of Indian Education schools and Department of Defense Education Activity schools. It is used when comparing national data to those of states, urban districts, or regions. SD = students with disabilities; EL = English learners.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Technology and Engineering Literacy (TEL) Assessment.
NAEP Technical Documentation Student Response and Exclusion Rates for the 2018 U.S. History Assessment
The following table presents the weighted student response and exclusion rates for the 2018 U.S. history assessment. The exclusion rates give the percentage excluded, among all eligible (i.e., assessed, absent, or excluded) students. Excluded students must necessarily be either students with disabilities (SD) or
English learners (EL). The response rates indicate the percentage of students assessed among those who it was intended would take the assessment from within the participating schools. Thus, students who were excluded are not included in the denominators of the response rates.
Weighted student response and exclusion rates for public and private schools, U.S. history assessment at grade 8, by school type, geographic region, and affiliation: 2018
-
School type and
geographic region/affiliation
Weighted student response rates (percent)
Weighted percent of all eligible students
who are SD and excluded
Weighted percent of all eligible students
who are EL and excluded
National all1 92.20 1.29 0.64
-
National public
92.21
1.39
0.68
Northeast public
89.66
1.28
1.04
Midwest public
93.05
1.05
0.18
South public
92.70
1.55
0.55
West public
92.67
1.48
1.07
National private
92.02
0.08
0.11
Catholic
92.60
0.00
0.09
Non-Catholic
91.09
0.13
0.13
1Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity schools located in the United States.
NOTE: National public includes students from public schools only. It includes charter schools, but excludes Bureau of Indian Education schools and Department of Defense Education Activity schools. It is used when comparing national data to those of states, urban districts, or regions. SD = students with disabilities; EL = English learners.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Social Sciences Assessment.
NAEP Technical Documentation Selection of Primary Sampling Units (PSUs) for the 2018 Assessment
The first stage of sampling for the 2018 assessment was the selection of primary sampling units (PSUs). A PSU is a geographic area comprising an individual county or a group of contiguous counties. Three sets of sample PSUs were selected for the 2018 assessments: one for the social sciences assessments, one for the technology and engineering literacy (TEL) assessment, and one for the pilot and special studies assessments. For social sciences, 105 PSUs were selected. For TEL and pilot/special studies, two sets of 67 PSUs were selected.
The PSU samples were drawn using a stratified sample design with one PSU selected per stratum or stratum pair with probability proportional to population size. The size measure used for PSU sampling was persons 17 years of age and younger from 2015 U.S. Census Bureau population estimates.
PSU Generation: Metropolitan Statistical Areas
PSU Generation: Certainty PSUs
PSU Generation: Non- Metropolitan Statistical Areas
The PSU sampling frame was constructed by partitioning all counties in the entire United States (the 50 states and the District of Columbia) into 1,001 non-overlapping PSUs as follows:
Each
metropolitan statistical
area (metro
area) was
considered a
separate PSU,
unless it
crossed census
region boundaries.
When this
happened, the
part within
each region
was made
a separate
PSU; and
PSU Frame Stratification Final PSU Samples
Non-metro
area PSUs
were constructed
from contiguous
non-metro area
counties within
the same
state that
had minimum
populations of
15,000 youths
in the
Northeast and
South census
regions and
10,000 youths
in the
Midwest and
West census
regions.
Measures of size for constructing the PSUs were based on youth population data obtained from the 2010 Decennial Census summary files.
For all three PSU samples, 29 PSUs on the PSU sampling frame were included in the sample with certainty (selected with a probability of 1). The inclusion of these PSUs in the sample with certainty provided the approximate optimum, cost-efficient sample of schools and students when samples were drawn within them at the required national sampling rate.
The remaining PSUs were grouped into noncertainty PSU sampling strata within eight primary strata, which were defined by census region and metropolitan status. The stratification of PSUs within the eight primary strata was based on characteristics shown to be highly correlated with student performance such as minority status, income, education, renter status, and percentage of female-headed households. These data were obtained at the county level from the 2006–10 American Community Survey (ACS) and then aggregated to the PSU level. Seventy-six noncertainty PSU strata were formed. These PSU strata were then paired to form 38 stratum pairs.
NAEP Technical Documentation Final Primary Sampling Unit (PSU) Samples for the 2018 Assessment
There were three sets of primary sampling unit (PSU) samples for the 2018 assessment: one for the social sciences assessments, one for the technology and engineering literacy (TEL) assessment, and one for the pilot and special studies assessments. The first set (for social sciences) had 105 sample PSUs of which 29 were certainty and 76 were noncertainty. The other two sets (for TEL and pilot/special studies, respectively) each had 67 sample PSUs of which 29 were certainty and 38 were noncertainty. All three sets had the exact same 29 certainty PSUs. The noncertainty PSUs were distinct from each other to the extent possible; however, ten noncertainty PSUs were in more than one PSU sample. In particular, five noncertainty PSUs were selected for both the social sciences and TEL samples, and a different five noncertainty PSUs were selected for both the social sciences and pilot/special studies samples.
To select the noncertainty PSUs for the social sciences assessments, one PSU was selected from each of the 76 noncertainty strata defined in Final Primary Sampling Unit Strata. Each PSU was selected with probability proportionate to size, where the size measure was the number of persons 17 years of age and younger from the 2015 Census Bureau population estimates.
To select the noncertainty PSUs for the TEL assessment, the 76 noncertainty strata were paired and one noncertainty stratum was randomly sampled from each of the 38 pairs. Then one PSU was selected from each of the 38 sampled strata with probability proportionate to size, where the size measure was the number of persons 17 years of age and younger from the 2015 Census Bureau population estimates.
The noncertainty PSUs for the pilot and special studies assessments were selected using a procedure similar to that used for TEL, but the noncertainty PSUs were drawn from the 38 noncertainty strata that were not sampled for the TEL assessment.
In addition, to reduce the burden of any particular school when selecting the 2018 sample PSUs, efforts were made to minimize overlap with the 2013, 2014, 2015, and 2016 PSU samples. There was a small PSU sample that included 32 noncertainty PSUs in 2017, with which overlap control was not attempted.
The table below shows the distribution of the 2018 sample PSUs for each assessment by metropolitan status (metropolitan/non-metropolitan), census region, and PSU type (certainty/noncertainty) by metropolitan status.
Distribution of sampled primary sampling units (PSUs) for the social sciences, TEL, and pilot/special studies assessments, by PSU type: 2018
-
PSU type
Number of sampled PSUs social sciences
Number of sampled
PSUs TEL
Number of sampled PSUs pilot/special studies
Total
105
67
67
Metropolitan status
Metropolitan
85
57
57
Non-metropolitan
20
10
10
Census region
Northeast
13
8
8
Midwest
23
14
14
South
41
25
25
West
28
20
20
Certainty/metropolitan status
Certainty
29
29
29
Non-certainty metropolitan
56
28
28
Non-certainty non-metropolitan
20
10
10
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
NAEP Technical Documentation Primary Sampling Unit (PSU) Frame Stratification for the 2018 Assessment
The primary sampling unit (PSU) strata were determined by census region and metropolitan status (metropolitan or non-metropolitan) for a total of eight "primary" strata. Measures of size were defined for each of these strata, determined by the relative share of the eventual PSU sample (the sample size is designed to be proportional to the number of youths). The PSU stratum measure of size then is the total number of youths (persons 17 years of age and younger) in the stratum. The table below presents these counts for each of the eight primary strata. The relative share of the PSU sample size for each stratum is the
Stepwise Regression Analysis Results for PSU Stratification
Final PSU Strata
number of youths in the stratum divided by the total number of youths, multiplied by 76 (the total number of noncertainty PSU strata). This is shown in the fifth column of the table below. The resulting number is then rounded to the nearest even integer (the integer needs to be even to facilitate variance estimation). Some manual tweaking to the rounding is needed such that the total number of final PSU strata sums to 76. The results of these calculations are given in the table below.
Noncertainty primary sampling unit frame size statistics, by primary stratum: 2018
-
Primary stratum
PSUs
Counties
Youths
Target number of final
PSU strata
Set number of final PSU
strata
Youths per final PSU
stratum
Total noncertainty PSUs
972
2,902
41,202,551
76
76
542,139
Northeast region metropolitan
43
84
4,422,552
8.2
8
552,819
Northeast region non- metropolitan
48
94
1,046,020
1.9
2
523,010
Midwest region metropolitan
91
229
7,009,814
12.9
12
584,151
Midwest region non- metropolitan
228
762
3,423,867
6.3
6
570,645
South region metropolitan
141
454
13,076,698
24.1
24
544,862
South region non-metropolitan
250
871
5,056,398
9.3
8
632,050
West region metropolitan
68
92
5,508,264
10.2
12
459,022
West region non-metropolitan
103
316
1,658,938
3.1
4
414,735
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
The division of the primary strata into the final strata was done on a stratum-by-stratum basis. The criteria for good PSU strata were 1) the strata should have as nearly equal measures of size as possible (to reduce sampling variance), and 2) the strata should be as heterogeneous in measured achievement as possible (i.e., there should be strata with low mean achievement, strata with mid-level mean achievement, and strata with high mean achievement). This second criterion will also ultimately reduce the variance of the assessment estimates since the final PSU sample will be balanced in terms of assessment means.
PSU assessment means from the current year cannot be used as assessments are only conducted after sampling is completed. Information is available about PSU sociodemographic characteristics in advance, however. An analysis was done within each primary stratum to find sociodemographic variables that were good predictors of performance on the eighth-grade reading assessments conducted in five previous NAEP cycles (2002, 2003, 2005, 2007, and 2009). Using these sociodemographic variables to define final strata should increase the chance of having efficient stratum definitions. Stepwise Regression Analysis Results for PSU Stratification describes this analysis for each primary stratum.
The final step in stratification was to define the desired number of final strata using the selected stratifiers, while constructing final strata that were as close to equal size as possible (with size defined by number of youth). The objective was to establish final strata that had a high between-stratum variance for the stratifiers (i.e., which "spread out" the stratifiers as much as possible). This was accomplished through the use of proprietary software developed for this purpose. Adjustments were then done manually. These strata are given in Final PSU Strata.
NAEP Technical Documentation Final Primary Sampling Unit (PSU) Strata for the 2018 Assessment
The strata were defined using the selected stratifiers from the stepwise regression analysis (see Stepwise Regression Analysis Results for PSU Stratification). The cutoffs were selected so that roughly equal measures of size were represented by each stratum.
The number of stratifiers used to define the noncertainty PSU strata within each primary stratum ranged from 1 to 5 stratifiers depending on the size of the primary stratum. For instance, the Northeast non-metropolitan primary stratum, which had about 1 million youths in noncertainty PSUs, used only one stratifier; whereas the South metropolitan primary stratum had about 13 million youths in noncertainty PSUs and used five stratifiers.
The final noncertainty PSU strata are presented in summary tables for each primary PSU stratum. The tables show the definition, number of PSUs, and size of each stratum.
Stratification for Northeast metropolitan noncertainty primary sampling units
Stratification for Northeast non-metropolitan noncertainty primary sampling units
Stratification for Midwest metropolitan noncertainty primary sampling units
Stratification for Midwest non-metropolitan noncertainty primary sampling units
Stratification for South metropolitan noncertainty primary sampling units
Stratification for South non-metropolitan noncertainty primary sampling units
Stratification for West metropolitan noncertainty primary sampling units
Stratification for West non-metropolitan noncertainty primary sampling units
NAEP Technical Documentation Stratification for Midwest Metropolitan Noncertainty Primary Sampling Units
The following table provides the definition, number of PSUs, and size of each noncertainty PSU stratum in the Midwest metropolitan primary stratum. Columns 2 through 5 show the characteristics used to define the strata along with their respective cutoffs. The size of each stratum is given in the last column and is in terms of the number of youths (persons 17 years of age and younger).
Stratification for Midwest metropolitan noncertainty primary sampling units (PSUs), by stratum: 2018
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
Quaternary stratifier
PSUs
Measure of size
Total
†
†
†
†
91
7,009,814
1
Percentage of female-headed households <= 9.6
Percentage of female-headed households <= 8.4
†
†
14
605,685
2
Percentage of female-headed households <= 9.6
Percentage of female-headed households > 8.4
†
†
15
598,683
3
Percentage of female-headed households > 9.6
Percentage of renters <= 30.6
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <= 14.2
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <= 10.1
14
554,591
4
Percentage of female-headed households > 9.6
Percentage of renters <= 30.6
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <= 14.2
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific
Islander youth > 10.1
10
567,774
5
Percentage of female-headed
Percentage of renters <= 30.6
Percentage of Black, Hispanic, American Indian/Alaska Native, or
Percentage of Black, Hispanic, American Indian/Alaska Native, or
8
553,670
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
Quaternary stratifier
PSUs
Measure of size
households > 9.6
Native Hawaiian/Other Pacific
Islander youth > 14.2
Native Hawaiian/Other Pacific
Islander youth <= 17.6
6
Percentage of female-headed households > 9.6
Percentage of renters <= 30.6
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific
Islander youth > 14.2
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific
Islander youth > 17.6
11
561,298
7
Percentage of female-headed households > 9.6
Percentage of renters (30.6-
32.2]
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <= 16.6
Percentage of female-headed
households <= 12.1
7
579,377
8
Percentage of female-headed households > 9.6
Percentage of renters (30.6-
32.2]
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <= 16.6
Percentage of female-headed
households > 12.1
2
560,007
9
Percentage of female-headed households > 9.6
Percentage of renters (30.6-
32.2]
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific
Islander youth > 16.6
Percentage of renters <= 31.8
2
637,089
10
Percentage of female-headed households > 9.6
Percentage of renters (30.6-
32.2]
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific
Islander youth > 16.6
Percentage of renters > 31.8
2
593,316
11
Percentage of female-headed households > 9.6
Percentage of renters > 32.2
Percentage of female-headed
households <= 12.2
†
4
641,186
12
Percentage of female-headed households > 9.6
Percentage of renters > 32.2
Percentage of female-headed
households > 12.2
†
2
557,138
Mean
†
†
†
†
†
584,151
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
NAEP Technical Documentation Stratification for Midwest Non- Metropolitan Noncertainty Primary Sampling Units
The following table provides the definition, number of PSUs, and size of each noncertainty PSU stratum in the Midwest non-metropolitan primary stratum. Columns 2 and 3 show the primary and secondary characteristics used to define the strata along with their respective cutoffs. The size of each stratum is given in the last column and is in terms of the number of youths (persons 17 years of age and younger).
Stratification for Midwest non-metropolitan noncertainty primary sampling units (PSUs), by stratum: 2018
-
Stratum
Primary stratifier
Secondary stratifier
PSUs
Measure of
size
Total
†
†
228
3,423,867
1
Percentage of children below the
poverty line <= 16.1
Percentage of children below the poverty line <= 13.7
41
578,068
2
Percentage of children below the
poverty line <= 16.1
Percentage of children below the poverty line > 13.7
36
584,857
3
Percentage of children below the poverty line (16.1-20.7]
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native
Hawaiian/Other Pacific Islander youth <= 5.4
38
562,740
4
Percentage of children below the poverty line (16.1-20.7]
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native
Hawaiian/Other Pacific Islander youth > 5.4
37
566,454
5
Percentage of children below the
poverty line > 20.7
Percentage of children below the poverty line <= 24
38
561,298
6
Percentage of children below the
poverty line > 20.7
Percentage of children below the poverty line > 24
38
570,450
Mean
†
†
†
570,645
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
NAEP Technical Documentation Stratification for Northeast Metropolitan Noncertainty Primary Sampling Units
The following table provides the definition, number of PSUs, and size of each noncertainty PSU stratum in the Northeast metropolitan primary stratum. Columns 2 and 3 show the primary and secondary characteristics, respectively, used to define the strata along with their respective cutoffs. The size of each stratum is given in the last column and is in terms of the number of youths (persons 17 years of age and younger).
Stratification for Northeast metropolitan noncertainty primary sampling units (PSUs), by stratum: 2018
-
Stratum
Primary stratifier
Secondary stratifier
PSUs
Measure of size
Total
†
†
43
4,422,552
1
Percentage of female-headed households <= 11
Percentage of female-headed households <= 10.3
10
530,198
2
Percentage of female-headed households <= 11
Percentage of female-headed households > 10.3
7
607,551
3
Percentage of female-headed households (11-
11.6]
Percentage of persons aged 25+ who completed high school
<= 89.7
7
554,849
4
Percentage of female-headed households (11-
11.6]
Percentage of persons aged 25+ who completed high school
> 89.7
3
529,360
5
Percentage of female-headed households (11.6-
12.7]
Percentage of female-headed households <= 12.5
5
588,464
6
Percentage of female-headed households (11.6-
12.7]
Percentage of female-headed households > 12.5
3
533,891
7
Percentage of female-headed households >
12.7
Percentage of female-headed households <= 13.5
2
560,747
8
Percentage of female-headed households >
12.7
Percentage of female-headed households > 13.5
6
517,492
Mean
†
†
†
552,819
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
NAEP Technical Documentation Stratification for Northeast Non- Metropolitan Noncertainty Primary Sampling Units
The following table provides the definition, number of PSUs, and size of each noncertainty PSU stratum in the Northeast non-metropolitan primary stratum. Column 2 shows the primary characteristic used to define the strata along with the cutoffs. The size of each stratum is given in the last column and is in terms of the number of youths (persons 17 years of age and younger).
Stratification for Northeast non-metropolitan noncertainty primary sampling units (PSUs), by stratum: 2018
-
Stratum
Primary stratifier
PSUs
Measure of size
Total
†
48
1,046,020
1
Percentage of persons aged 25+ with a college degree <= 19.1
23
517,103
2
Percentage of persons aged 25+ with a college degree > 19.1
25
528,917
Mean
†
†
523,010
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
NAEP Technical Documentation Stratification for South Metropolitan Noncertainty Primary Sampling Units
The following table provides the definition, number of PSUs, and size of each noncertainty PSU stratum in the South metropolitan primary stratum. Columns 2 through 6 show the characteristics used to define the strata along with their respective cutoffs. The size of each stratum is given in the last column and is in terms of the number of youths (persons 17 years of age and younger).
Stratification for South metropolitan noncertainty primary sampling units (PSUs), by stratum: 2018
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
Quaternary stratifier
Quinary stratifier
PSUs
Measure of
size
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
Quaternary stratifier
Quinary stratifier
PSUs
Measure of
size
-
Total
†
†
†
†
†
141
13,076,698
1
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
28.2
Per capita household income <= $23,025
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
<= 14.9
13
509,921
2
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
28.2
Per capita household income <= $23,025
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth >
14.9
13
537,563
3
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
28.2
Per capita household income ($23,025-$25,326]
Percentage of female- headed households <=
12.1
6
535,903
4
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
28.2
Per capita household income ($23,025-$25,326]
Percentage of female- headed households >
12.1
6
535,677
5
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
28.2
Per capita household income ($25,326-$27,540]
Percentage of female- headed households <=
11.8
6
453,396
6
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
28.2
Per capita household income ($25,326-$27,540]
Percentage of female- headed households >
11.8
3
649,016
7
Percentage of
female- headed
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or
Per capita household income ($27,540-$28,621]
†
3
560,145
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
Quaternary stratifier
Quinary stratifier
PSUs
Measure of
size
households
<= 16.9
Native Hawaiian/Other
Pacific Islander youth <=
28.2
8
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
28.2
Per capita household income > $28,621
†
5
530,358
9
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(28.2-30.6]
Percentage of female- headed households <= 13
†
4
555,172
10
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(28.2-30.6]
Percentage of female- headed households > 13
†
6
534,928
11
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(30.6-33.2]
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
32.2
†
4
531,798
12
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(30.6-33.2]
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth >
32.2
†
3
570,040
13
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth >
33.2
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
36.9
†
6
528,166
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
Quaternary stratifier
Quinary stratifier
PSUs
Measure of
size
-
14
Percentage of
female- headed households
<= 16.9
Percentage of renters <=
33.5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth >
33.2
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth >
36.9
†
6
531,294
15
Percentage of
female- headed households
<= 16.9
Percentage of renters >
33.5
Per capita household income <= $23,655
Percentage of renters <=
36.9
†
10
559,491
16
Percentage of
female- headed households
<= 16.9
Percentage of renters >
33.5
Per capita household income <= $23,655
Percentage of renters >
36.9
†
11
549,737
17
Percentage of
female- headed households
<= 16.9
Percentage of renters >
33.5
Per capita household income ($23,655-$26,682]
Percentage of female- headed households <=
14.2
†
6
574,570
18
Percentage of
female- headed households
<= 16.9
Percentage of renters >
33.5
Per capita household income ($23,655-$26,682]
Percentage of female- headed households > 14.2
†
4
559,664
19
Percentage of
female- headed households
<= 16.9
Percentage of renters >
33.5
Per capita household income > $26,682
Percentage of renters <=
38.7
†
3
542,958
20
Percentage of
female- headed households
<= 16.9
Percentage of renters >
33.5
Per capita household income > $26,682
Percentage of renters >
38.7
†
2
590,223
21
Percentage of
female- headed
Per capita household income <=
$21,548
Percentage of renters <=
31.7
†
†
4
561,295
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
Quaternary stratifier
Quinary stratifier
PSUs
Measure of
size
households >
16.9
22
Percentage of
female- headed households >
16.9
Per capita household income <=
$21,548
Percentage of renters >
31.7
†
†
9
556,241
23
Percentage of
female- headed households >
16.9
Per capita household income >
$21,548
Percentage of female- headed households <= 18.7
†
†
5
507,084
24
Percentage of
female- headed households >
16.9
Per capita household income >
$21,548
Percentage of female- headed households > 18.7
†
†
3
512,058
Mean
†
†
†
†
†
†
544,862
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
NAEP Technical Documentation Stratification for South Non-Metropolitan Noncertainty Primary Sampling Units
The following table provides the definition, number of PSUs, and size of each noncertainty PSU stratum in the South non-metropolitan primary stratum. Columns 2 through 4 show the characteristics used to define the strata along with their respective cutoffs. The size of each stratum is given in the last column and is in terms of the number of youths (persons 17 years of age and younger).
Stratification for South non-metropolitan noncertainty primary sampling units (PSUs), by stratum: 2018
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
PSUs
Measure of
size
Total
†
†
†
250
5,056,398
1
Percentage of female- headed households <= 12.6
Per capita household income <= $20,111
Percentage of female- headed households <=
11.3
33
637,694
2
Percentage of female- headed households <= 12.6
Per capita household income <= $20,111
Percentage of female- headed households > 11.3
33
632,174
3
Percentage of female- headed households <= 12.6
Per capita household income ($20,111-$22,659)
†
32
646,216
4
Percentage of female- headed households <= 12.6
Per capita household income > $22,659
†
28
639,839
5
Percentage of female- headed households (12.6-
16.2)
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
<= 29.2
†
32
629,483
6
Percentage of female- headed households (12.6-
16.2)
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth >
29.2
†
32
637,564
7
Percentage of female- headed households > 16.2
Per capita household income <= $17,691
†
31
614,601
8
Percentage of female- headed households > 16.2
Per capita household income > $17,691
†
29
618,827
Mean
†
†
†
†
632,050
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
NAEP Technical Documentation Stratification for West Metropolitan Noncertainty Primary Sampling Units
The following table provides the definition, number of PSUs, and size of each noncertainty PSU stratum in the West metropolitan primary stratum. Columns 2 through 4 show the characteristics used to define the strata along with their respective cutoffs. The size of each stratum is given in the last column and is in terms of the number of youths (persons 17 years of age and younger).
Stratification for West metropolitan noncertainty primary sampling units (PSUs), by stratum: 2018
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
PSUs
Measure of
size
Total
†
†
†
68
5,508,264
1
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
18.4
Percentage of renters
<= 29.3
†
8
440,377
2
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
18.4
Percentage of renters
(29.3-31]
†
6
488,282
3
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
18.4
Percentage of renters >
31
Percentage of persons aged 25+ with a
college degree <= 28.1
10
440,238
4
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
18.4
Percentage of renters >
31
Percentage of persons aged 25+ with a
college degree > 28.1
9
448,976
5
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(18.4-44.3]
Percentage of renters
<= 33.6
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
21
2
514,753
6
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(18.4-44.3]
Percentage of renters
<= 33.6
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth > 21
5
426,090
7
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(18.4-44.3]
Percentage of renters >
33.6
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth <=
32.4
7
465,236
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
-
Stratum
Primary stratifier
Secondary stratifier
Tertiary stratifier
PSUs
Measure of
size
-
8
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(18.4-44.3]
Percentage of renters >
33.6
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth >
32.4
5
477,876
9
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(44.3-54.4]
Percentage of renters
<= 37.9
†
4
457,572
10
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth
(44.3-54.4]
Percentage of renters >
37.9
†
2
457,621
11
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth >
54.4
Percentage of persons aged 25+ with a college
degree <= 15.1
†
7
443,193
12
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native Hawaiian/Other Pacific Islander youth >
54.4
Percentage of persons aged 25+ with a college
degree > 15.1
†
3
448,050
Mean
†
†
†
†
459,022
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
NAEP Technical Documentation Stratification for West Non-Metropolitan Noncertainty Primary Sampling Units
The following table provides the definition, number of PSUs, and size of each noncertainty PSU stratum in the West non-metropolitan primary stratum. Columns 2 and 3 show the primary and secondary characteristics, respectively, used to define the strata along with their respective cutoffs. The size of each stratum is given in
the last column and is in terms of the number of youths (person 17 years of age and younger).
Stratification for West non-metropolitan noncertainty primary sampling units (PSUs), by stratum: 2018
-
Stratum
Primary stratifier
Secondary stratifier
PSUs
Measure of
size
Total
†
†
103
1,658,938
1
Percentage of female-headed
households <= 9.7
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native
Hawaiian/Other Pacific Islander youth <= 12
27
417,593
2
Percentage of female-headed
households <= 9.7
Percentage of Black, Hispanic, American Indian/Alaska Native, or Native
Hawaiian/Other Pacific Islander youth > 12
28
404,250
3
Percentage of female-headed
households > 9.7
Percentage of female-headed households <= 11.9
26
412,860
4
Percentage of female-headed
households > 9.7
Percentage of female-headed households > 11.9
22
424,235
Mean
†
†
†
414,735
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
NAEP Technical Documentation Stepwise Regression Analysis Results for Primary Sampling Unit (PSU) Stratification for the 2018 Assessment
The objective was to find the optimum set of primary sampling unit (PSU)-level sociodemographic characteristics in terms of strength of relationship to achievement. The PSU-level values of these characteristics were derived from the 2010 Decennial Census summary files and the 2006–10 American Community Survey (ACS) estimates, computed by combining the county-level data (using county youth estimates as the relative weighting factor for each county within the PSU). The characteristics used and their abbreviations as used in the tables, were as follows:
aggregate
minority group
percentages (percentage
of Black,
Hispanic, American
Indian/Alaska Native,
or Native
Hawaiian/Other Pacific
Islander students);
income levels
(per capita
household income,
percentage of
children below
the poverty
line);
education
levels in
the population
(i.e., percentage
of persons
aged 25+
who completed
high school,
percentage of
persons aged
25+ with
a college
degree); percentage
of renters
(i.e., percentage
of householders
who rent
rather than
own their
place of
residence); and
percentage
of female-headed
households.
These PSU-level census characteristics were analyzed with the eighth-grade reading assessment scores from five previous NAEP cycles (2002, 2003, 2005, 2007, and 2009). The criterion was that good strata should be heterogeneous for each of the five characteristics (i.e., within-stratum variance for each assessment value should be low and between-stratum variance high).
The analysis was done separately within each of the eight primary strata (census region by metro status), using a forward stepwise regression approach, with a p- value of 20 percent. The results of the regression model were used to generate the final PSU strata.
NAEP Technical Documentation Primary Sampling Unit (PSU) Generation: Certainty PSUs for the 2018 Assessment
Any primary sampling unit (PSU) was defined as a certainty PSU if it had 500,000 or more youths or if it represented more than 80 percent of its assigned stratum. The estimated number of youths used to designate certainty PSUs was the number of persons aged 17 or under from the 2010 Decennial Census. These PSUs were so large that a sample of schools was taken from all of them (rather than from only a subsample of them, as with noncertainty PSUs). The Honolulu, Hawaii PSU was included as a certainty by design in order to reduce the variances of estimates for Asian and Native Hawaiian/Other Pacific Islander students. A total of 29 PSUs were classified as certainties in the 2018 frame. The table below provides a listing of the certainty PSUs by census region. Note that the names of the metropolitan statistical areas do not represent the cities proper. Rather they can and do cross jurisdiction and county boundaries (for example, the Boston- Cambridge-Quincy metropolitan statistical area includes Massachusetts and New Hampshire). The "Number of youths" column in the table reflects updated 2015
U.S. Census Bureau population estimates.
Metropolitan statistical area definitions for certainty PSUs, by census region: 2018
-
Census region/Metropolitan statistical area
Jurisdiction
Number of counties
Number of youths
Total † 241 32,442,560
-
Census region/Metropolitan statistical area
Jurisdiction
Number of counties
Number of youths
-
Northeast
†
39
6,433,831
Boston-Cambridge-Quincy
MA-NH
7
974,107
New York-Northern New Jersey-Long Island
NY-NJ-PA
23
4,261,793
Philadelphia-Camden-Wilmington (Northeast part)
PA-NJ
9
1,197,931
Midwest
†
64
5,214,658
Chicago-Joliet-Naperville
IL-IN-WI
14
2,243,956
Detroit-Warren-Livonia
MI
6
977,193
Kansas City
MO-KS
15
522,649
Minneapolis-St. Paul-Bloomington
MN-WI
13
829,194
St. Louis
MO-IL
16
641,666
South
†
98
10,111,925
Atlanta-Sandy Springs-Marietta
GA
28
1,436,804
Baltimore-Towson
MD
7
618,770
Dallas-Fort Worth-Arlington
TX
12
1,875,641
Houston-Sugar Land-Baytown
TX
10
1,794,208
Miami-Fort Lauderdale-Pompano Beach
FL
3
1,236,681
Orlando-Kissimmee-Sanford
FL
4
531,519
San Antonio-New Braunfels
TX
8
612,614
Tampa-St. Petersburg-Clearwater
FL
4
605,816
Washington-Arlington-Alexandria
DC-VA-MD-WV
22
1,399,872
West
†
40
10,682,146
Denver-Aurora-Broomfield
CO
10
661,775
Honolulu
HI
1
214,852
Las Vegas-Paradise
NV
1
498,564
Los Angeles-Long Beach-Santa Ana
CA
2
2,995,992
Phoenix-Mesa-Glendale
AZ
2
1,127,596
Portland-Vancouver-Hillsboro
OR-WA
7
531,629
Riverside-San Bernardino-Ontario
CA
2
1,185,021
Sacramento--Arden-Arcade--Roseville
CA
4
530,234
San Diego-Carlsbad-San Marcos
CA
1
728,037
San Francisco-Oakland-Fremont
CA
5
939,388
San Jose-Sunnyvale-Santa Clara
CA
2
452,028
† Not applicable.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
-
Census region/Metropolitan statistical area
Jurisdiction
Number of counties
Number of youths
Seattle-Tacoma-Bellevue WA 3 817,030
NAEP Technical Documentation Primary Sampling Unit (PSU) Generation: Metropolitan Statistical Areas for the 2018 Assessment
Primary Sampling Units (PSUs) for NAEP are classified as either metropolitan statistical areas1 (metro areas) or non-metro areas. Metro area PSUs are those that are made up of counties in metro areas.
Each metro area constitutes a separate PSU, except when it crosses census region boundaries. Such metro areas are split along regional boundaries with each regional part considered its own distinct PSU. For example, the Louisville-Jefferson County, KY-IN metro area was partitioned into two PSUs, one for the counties in Kentucky which are part of the South region and the other for counties in Indiana which are part of the Midwest region.
In total, there were 372 metro area PSUs, 29 of which were defined as certainty PSUs. The remaining 343 metro area PSUs, covering a total of 859 counties, constituted the noncertainty portion of the metro area PSU sampling frame. The table below presents the number of PSUs, the number of counties represented, and the estimated number of youths (total and mean per PSU) in noncertainty metro area PSUs by census region. These estimates come from the county-level estimates of numbers of persons aged 0 to 17 from the 2015 U.S. Census Bureau population estimates.
Noncertainty metropolitan primary sampling unit (PSU) frame, by census region: 2018
-
Census region
PSUs
Counties
Youths
Mean number of youths per PSU
Total
343
859
30,017,328
87,514
Northeast
43
84
4,422,552
102,850
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
1Based on the 2009 metro area definitions, the most recent available metro area definitions at the time of PSU construction, from the U.S. Office of Management and Budget (OMB Bulletin No. 10-02).
-
Census region
PSUs
Counties
Youths
Mean number of youths per PSU
-
Midwest
91
229
7,009,814
77,031
South
141
454
13,076,698
92,743
West
68
92
5,508,264
81,004
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
1Based on the 2009 metro area definitions, the most recent available metro area definitions at the time of PSU construction, from the U.S. Office of Management and Budget (OMB Bulletin No. 10-02).
NAEP Technical Documentation Primary Sampling Unit (PSU) Generation: Non-Metropolitan Statistical Areas for the 2018 Assessment
Primary sampling units (PSUs) for NAEP are classified as either metro area or non-metro area. Non-metro area PSUs are PSUs that are made up of counties that are not part of any metropolitan statistical areas1.
An algorithm was used to define a preliminary set of non-metro area PSUs satisfying specific design constraints. The algorithm attempted to form PSUs that were geographically compact, of a minimum population size (15,000 youths in the Northeast and South census regions, and 10,000 youths in the Midwest and West census regions) and that also did not cross state boundaries. The input set consisted of all non-metro area counties. The county which had the largest maximum point-to-point distance was addressed first. It was grouped with adjacent non-metro area counties until the minimum PSU size was met. The algorithm was then run on the remaining non-metro area counties not yet assigned to a PSU to combine the county with the largest maximum point-to-point distance among the remaining counties with its adjacent non-metro area counties until the minimum PSU size was met. This process was repeated until all counties were grouped into PSUs.
When the algorithm was unable to create PSUs that conformed to the specific design constraints, manual adjustments were made. The end result of this procedure was that all non-metro area PSUs were contained within state boundaries, but in some cases the PSU size fell slightly below the pre-specified minimum.
In total, there were 629 non-metro area PSUs covering a total of 2,043 counties, all of which constitute the non-metro area PSU sampling frame. The table below presents the number of PSUs, the number of counties represented, and the estimated number of youths (total and mean per PSU) in the non-metro area PSU sampling frame by census region. The estimated number of youths (persons aged 0 to 17) for each county comes from the 2015 U.S. Census Bureau population estimates.
Non-metropolitan statistical area primary sampling unit (PSU) frame, by census region: 2018
-
Census region
PSUs
Counties
Youths
Mean number of youths per PSU
Total
629
2,043
11,185,223
17,783
Northeast
48
94
1,046,020
21,792
Midwest
228
762
3,423,867
15,017
South
250
871
5,056,398
20,226
West
103
316
1,658,938
16,106
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National Assessment of Educational Progress (NAEP), 2018 Assessment.
1 Based on the 2009 metro area definitions, the most recent available metro area definitions at the time of PSU construction, from the U.S. Office of Management and Budget (OMB Bulletin No. 10-02).
| File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
| Author | Molin, Ed C |
| File Modified | 0000-00-00 |
| File Created | 2023-09-22 |
© 2025 OMB.report | Privacy Policy