Appendix H Weighting Procedures

NATIONAL CENTER FOR EDUCATION STATISTICS
NATIONAL ASSESSMENT OF EDUCATIONAL PROGRESS
Appendix H
NAEP 2011 Weighting Procedures

Request for Clearance for
NAEP Assessments for 2017-2019
OMB# 1850-NEW v.1
(previous OMB# 1850-0790 v.43)

July 29, 2016

1 of 60

NAEP Technical Documentation Website
Weighting Procedures for the 2011 Assessment
NAEP assessments use complex sample designs to create student samples that generate
population and subpopulation estimates with reasonably high precision. Student sampling
weights ensure valid inferences from the student samples to their respective populations.
In 2011, weights were developed for students sampled at grades 4, 8, and 12 for
assessments in mathematics, reading, science, and a writing computer-based
assessment (WCBA). Each student was assigned a weight to be used for making
inferences about students in the target population. This weight is known as the final
full-sample student weight, which contains the following major components:
the student base weight;
school nonresponse adjustments;
student nonresponse adjustments;
school weight trimming adjustments;
student weight trimming adjustments; and
student raking adjustment.

Computation of Full-Sample Weights
Computation of Base Weights
School and Student Nonresponse
Adjustments
School and Student Weight
Trimming Adjustments
Student Raking Adjustment
Computation of Replicate Weights for
Variance Estimation
Quality Control on Weighting
Procedures

The student base weight is the inverse of the overall probability of selecting a student and assigning that student to a particular
assessment. The sample design that determines the base weights is discussed in the NAEP 2011 sample design section.
The student base weight is adjusted for two sources of nonparticipation: school level and student level. These weighting adjustments
seek to reduce the potential for bias from such nonparticipation by
increasing the weights of students from participating schools similar to those schools not participating; and
increasing the weights of participating students similar to those students from within participating schools who did not attend
the assessment session (or makeup session) as scheduled.
Furthermore, the final weights reflect the trimming of extremely large weights at both the school and student level. These weighting
adjustments seek to reduce variances of survey estimates.
Starting in 2009, an additional weighting adjustment was implemented in the state samples so that estimates for key student-level
characteristics were in agreement across assessments in reading, mathematics, and science. This procedure was implemented using a
raking procedure.
In addition to the final full-sample weight, a set of replicate weights was provided for each student. These replicate weights are used
to calculate the variances of survey estimates using the jackknife repeated replication method. The methods used to derive these
weights were aimed at reflecting the features of the sample design, so that when the jackknife variance estimation procedure is
implemented, approximately unbiased estimates of sampling variance are obtained. In addition, the various weighting procedures
were repeated on each set of replicate weights to appropriately reflect the impact of the weighting adjustments on the sampling
variance of a survey estimate. In 2011, a finite population correction (fpc) factor was used in computing variance estimates for the
reading, mathematics, and science assessments. See Computation of Replicate Weights for Variance Estimation for details.
Quality control checks were carried out throughout the weighting process to ensure the accuracy of the full-sample
and replicate weights. See Quality Control for Weighting Procedures for the various checks implemented and main findings of
interest.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011.aspx

7/22/2016 2:46 PM

2 of 60

Computation of Full-Sample Weights for the 2011 Assessment
The full-sample or final student weight is the sampling weight used to derive NAEP
student estimates of population and subpopulation characteristics for a specified grade
(4, 8, or 12) and assessment subject (reading, mathematics, science, and
writing [WCBA]). The full-sample student weight reflects the number of students that
the sampled student represents in the population for purposes of estimation. The
summation of the final student weights over a particular student group provides an
estimate of the total number of students in that group within the population.

Computation of Base Weights
School and Student Nonresponse Weight
Adjustments
School and Student Weight Trimming
Adjustments

The full-sample weight, which is used to produce survey estimates, is distinct from a
Student Raking Adjustment
replicate weight that is used to estimate variances of survey estimates. The full-sample
weight is assigned to participating students and reflects the student base weight after the
application of the various weighting adjustments. The full-sample weight for student k from school s in stratum j (FSTUWGTjsk) can
be expressed as follows:

where
STU_BWTjsk is the student base weight;
SCH_NRAFjs is the school-level nonresponse adjustment factor;
STU_NRAFjsk is the student-level nonresponse adjustment factor;
SCH_TRIMjs is the school-level weight trimming adjustment factor;
STU_TRIMjsk is the student-level weight trimming adjustment factor; and
STU_RAKEjsk is the student-level raking adjustment factor.
School sampling strata for a given assessment varies by school type and grade. See the links below for descriptions of the school
strata for the various assessments.
Reading, mathematics, and science at grades 4 and 8 for public schools
Reading, mathematics, and science at grades 4 and 8 for private schools
WCBA at grades 8 and 12 for public schools
WCBA at grades 8 and 12 for private schools

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_comp_full_samp_weights.aspx

7/22/2016 2:46 PM

3 of 60

Computation of Base Weights for the 2011 Assessment
Every sampled school and student received a base weight equal to the reciprocal of its probability of
selection. Computation of a school base weight varies by

School Base Weights

Student Base Weights
type of sampled school (original or substitute);
sampling frame (new school frame or not); and
assessment subject (writing [WCBA] samples involved the selection of geographic units known as primary sampling units
[PSUs], but reading, mathematics, and science did not).
Computation of a student base weight reflects
the student's overall probability of selection accounting for school and student sampling;
assignment to session type at the school- and student-level; and
the student's assignment to the reading, mathematics, science, or writing (WCBA) assessment.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_base.aspx

7/22/2016 2:46 PM

4 of 60

School Base Weights for the 2011 Assessment
The school base weight for a sampled school is equal to the inverse of its overall probability of
selection. The overall selection probability of a sampled school differs by
type of sampled school (original or substitute);
sampling frame (new school frame or not); and
assessment subject (writing [WCBA] samples involved the selection of geographic units known
as primary sampling units [PSUs], but reading, mathematics, and science did not).
The overall selection probability of an originally selected school in a reading, mathematics, or science
sample is equal to its probability of selection from the NAEP public/private school frame.
The overall probability of selection of an originally selected school in the WCBA sample reflects two
components:

Substitute schools for the
2011 private school
national assessment
Substitute public schools
for the 2011
computer-based
assessment
Substitute private schools
for the 2011
computer-based
assessment

the probability of selection of the PSU; and
the probability of selection of the school within the selected PSU from the NAEP public/private
school frame.
The overall selection probability of a school from the new school frame in a reading, mathematics, science, or WCBA sample is the
product of two quantities:
the probability of selection of the school's district into the new-school district sample; and
the probability of selection of the school into the new school sample.
Substitute schools are preassigned to original schools and take the place of original schools if they refuse to participate. For weighting
purposes, they are treated as if they were the original schools that they replaced, so substitute schools are assigned the school base
weight of the original schools.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_base_school.aspx

7/22/2016 2:46 PM

5 of 60

Student Base Weights for the 2011 Assessment
Every sampled student received a student base weight, whether or not the student participated in the assessment. The student base
weight is the reciprocal of the probability that the student was sampled to participate in the assessment for a specified subject. The
student base weight for student k from school s in stratum j (STU_BWTjsk) is the product of seven weighting components and can be
expressed as follows:

where
SCH_BWTjs is the school base weight;
SCHSESWTjs is the school-level session assignment weight that reflects the conditional probability, given the school, that the
particular session type was assigned to the school;
WINSCHWTjs is the within-school student weight that reflects the conditional probability, given the school, that the student was
selected for the NAEP assessment;
STUSESWTjsk is the student-level session assignment weight that reflects the conditional probability, given that the particular
session type was assigned to the school, that the student was assigned to the session type;
SUBJFACjsk is the subject spiral adjustment factor that reflects the conditional probability, given that the student was assigned
to a particular session type, that the student was assigned the specified subject;
SUBADJjs is the substitution adjustment factor to account for the difference in enrollment size between the substitute and
original school; and
YRRND_AFjs is the year-round adjustment factor to account for students in year-round schools on scheduled break at the time
of the NAEP assessment and thus not available to be included in the sample.
The within-school student weight (WINSCHWTjs) is the inverse of the student sampling rate in the school.
The subject spiral adjustment factor (SUBJFACjsk) adjusts the student weight to account for the spiral pattern used in
distributing mathematics, reading, science, or writing (WCBA) booklets to the students. The subject factor varies by grade, subject,
school type (public/private), jurisdiction (states participating in the science assessment/states declining to participate in the science
assessment), and it is equal to the inverse of the booklet proportions (mathematics, reading, science, or WCBA) in the overall spiral
for a specific sample.
For cooperating substitutes of nonresponding original sampled schools, the substitution adjustment factor (SUBADJjs) is equal to the
ratio of the estimated grade enrollment for the original sampled school to the estimated grade enrollment for the substitute school. The
student sample from the substitute school then "represents" the set of grade-eligible students from the original sampled school.
The year-round adjustment factor (YRRND_AFjs) adjusts the student weight for students in year-round schools who do not attend
school during the time of the assessment. This situation typically arises in overcrowded schools. School administrators in year-round
schools randomly assign students to portions of the year in which they attend school and portions of the year in which they do not
attend. At the time of assessment, a certain percentage of students (designated as OFFjs) do not attend school and thus cannot be
assessed. The YRRND_AFjs for a school is calculated as 1/(1-OFFjs/100).

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_base_stud.aspx

7/22/2016 2:46 PM

6 of 60

School and Student Nonresponse Weight Adjustments for the 2011 Assessment
Nonresponse is unavoidable in any voluntary survey of a human population. Nonresponse leads to
School Nonresponse Weight
the loss of sample data that must be compensated for in the weights of the responding sample
Adjustment
members. This differs from ineligibility, for which no adjustments are necessary. The purpose of
the nonresponse adjustments is to reduce the mean square error of survey estimates. While the
Student Nonresponse Weight
nonresponse adjustment reduces the bias from the loss of sample, it also increases variability
Adjustment
among the survey weights leading to increased variances of the sample estimates. However, it is
presumed that the reduction in bias more than compensates for the increase in the variance, thereby
reducing the mean square error and thus improving the accuracy of survey estimates. Nonresponse adjustments are made in the NAEP
surveys at both the school and the student levels: the responding (original and substitute) schools receive a weighting adjustment
to compensate for nonresponding schools, and responding students receive a weighting adjustment to compensate for nonresponding
students.
The paradigm used for nonresponse adjustment in NAEP is the quasi-randomization approach (Oh and Scheuren 1983). In this
approach, school response cells are based on characteristics of schools known to be related to both response propensity and
achievement level, such as the locale type (e.g., large principal city of a metropolitan area) of the school. Likewise, student response
cells are based on characteristics of the schools containing the students and student characteristics, which are known to be related to
both response propensity and achievement level, such as student race/ethnicity, gender, and age.
Under this approach, sample members are assigned to mutually exclusive and exhaustive response cells based on predetermined
characteristics. A nonresponse adjustment factor is calculated for each cell as the ratio of the sum of adjusted base weights for all
eligible units to the sum of adjusted base weights for all responding units. The nonresponse adjustment factor is then applied to the
base weight of each responding unit. In this way, the weights of responding units in the cell are "weighted up" to represent the full set
of responding and nonresponding units in the response cell.
The quasi-randomization paradigm views nonresponse as another stage of sampling. Within each nonresponse cell, the paradigm
assumes that the responding sample units are a simple random sample from the total set of all sample units. If this model is valid, then
the use of the quasi-randomization weighting adjustment will eliminate any nonresponse bias. Even if this model is not valid, the
weighting adjustments will eliminate bias if the achievement scores are homogeneous within the response cells (i.e., bias is eliminated
if there is homogeneity either in response propensity or in achievement levels). See, for example, chapter 4 of Little and Rubin
(1987).

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp.aspx

7/22/2016 2:46 PM

7 of 60

School Nonresponse Weight Adjustment for the 2011 Assessment
The school nonresponse adjustment procedure inflates the weights of cooperating
schools to account for eligible noncooperating schools for which no substitute
schools participated. The adjustments are computed within nonresponse cells and
are based on the assumption that the cooperating and noncooperating schools
within the same cell are more similar to each other than to schools from different
cells. School nonresponse adjustments were carried out separately by sample; that
is, by
grade (4, 8, 12);
school type (public, private); and
assessment subject (reading, mathematics, science, writing [WCBA]).

Development of Initial School Nonresponse
Cells
Development of Final School Nonresponse
Cells
School Nonresponse Adjustment Factor
Calculation

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_schl.aspx

7/22/2016 2:46 PM

8 of 60

Development of Final School Nonresponse Cells for the 2011 Assessment
Limits were placed on the magnitude of cell sizes and adjustment factors to prevent unstable nonresponse adjustments
and unacceptably large nonresponse factors. All initial weighting cells with fewer than six cooperating schools or adjustment factors
greater than 3.0 for the full sample weight were collapsed with suitable adjacent cells. Simultaneously, all initial weighting cells for
any replicate with fewer than four cooperating schools or adjustment factors greater than the maximum of 3.0 or two times the full
sample nonresponse adjustment factor were collapsed with suitable adjacent cells. Initial weighting cells were generally collapsed in
reverse order of the cell structure; that is, starting at the bottom of the nesting structure and working up toward the top level of the
nesting structure.

Public School Samples for Reading, Mathematics, and Science at Grades 4 and 8
For the public school samples, cells with the most similar race/ethnicity classification within a given jurisdiction/Trial Urban District
Assessment (TUDA) district and urbanicity (urban-centric locale) stratum were collapsed first. If further collapsing was required after
all levels of race/ethnicity strata were collapsed, cells with the most similar urbanicity strata were combined next. Cells were never
permitted to be collapsed across jurisdiction or TUDA district.

Private School Samples for Reading, Mathematics, and Science at Grades 4 and 8
For the private school samples, cells with the most similar race/ethnicity classification within a given affiliation, census division, and
urbanicity stratum were collapsed first. If further collapsing was required after all levels of race/ethnicity strata were collapsed, cells
with the most similar urbanicity classification were combined. Any further collapsing occurred across census division strata but never
across affiliation.

Public School Samples for WCBA at Grades 8 and 12
For the public school samples, cells with similar high percentage Black/Hispanic status within a given census region and urbanicity
stratum were collapsed first. If further collapsing was required, cells with the most similar urbanicity strata within a given census
region were combined next. No further collapsing occurred after all levels of urbanicity strata were collapsed. That is,
collapsing never occurred across census region.

Private School Samples for WCBA at Grades 8 and 12
For the private school samples, if collapsing was necessary, all census region cells within a given affiliation were collapsed. However,
collapsing never occurred across affiliation.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_schl_final.aspx

7/22/2016 2:46 PM

9 of 60

Development of Initial School Nonresponse Cells for the 2011 Assessment
The cells for nonresponse adjustments are generally functions of the school sampling strata for the individual samples. School
sampling strata usually differ by assessment subject, grade, and school type (public or private). Assessment subjects that are
administered together by way of spiraling have the same school samples and stratification schemes. Subjects that are not spiraled with
any other subjects have their own separate school sample. In NAEP 2011, the reading, mathematics, and science assessments were
spiraled together, but writing (WCBA) was not spiraled with any other subject.
The initial nonresponse cells for the various NAEP 2011 samples are described below.

Public School Samples for Reading, Mathematics, and Science at Grades 4 and 8
For these samples, initial weighting cells were formed within each jurisdiction using the following nesting cell structure:
Trial Urban District Assessment (TUDA) district vs. the balance of the state for states with TUDA districts;
urbanicity (urban-centric locale) stratum; and
race/ethnicity classification, or achievement level, or median income, or grade enrollment.
In general, the nonresponse cell structure used minority stratum as the lowest level variable. However, where there was only
one race/ethnicity category within a particular urbanicity stratum, categorized achievement or median income data were used instead.

Private School Samples for Reading, Mathematics, and Science at Grades 4 and 8
The initial weighting cells for these samples were formed within each grade using the following nesting cell structure:
affiliation;
census division stratum;
urbanicity stratum; and
race/ethnicity classification.

Public School Samples for WCBA at Grades 8 and 12
The initial weighting cells for these samples were formed within each subject and grade using the following nesting cell structure:
census region;
urban-centric locale; and
indicator of high percentage of Black/Hispanic students within school.

Private School Samples for WCBA at Grades 8 and 12
The initial weighting cells for these samples were formed within each subject and grade using the following nesting cell structure:
affiliation; and
census region.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_schl_initial.aspx

7/22/2016 2:46 PM

10 of 60

School Nonresponse Adjustment Factor Calculation for the 2011 Assessment
In each final school nonresponse adjustment cell c, the school nonresponse adjustment factor SCH_NRAFc was computed as follows:

where
Sc is the set of all eligible sampled schools (cooperating original and substitute schools and refusing original schools with
noncooperating or no assigned substitute) in cell c,
Rc is the set of all cooperating schools within Sc,
SCH_BWTs is the school base weight,
SCH_TRIMs is the school-level weight trimming factor,
SCHSESWTs is the school-level session assignment weight, and
Xs is the estimated grade enrollment corresponding to the original sampled school.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_schl_factor.aspx

7/22/2016 2:46 PM

11 of 60

Student Nonresponse Weight Adjustment for the 2011 Assessment
The student nonresponse adjustment procedure inflates the weights of assessed
students to account for eligible sampled students who did not participate in the
assessment. These inflation factors offset the loss of data associated with absent
students. The adjustments are computed within nonresponse cells and are based
on the assumption that the assessed and absent students within the same cell are
more similar to one another than to students from different cells. Like its
counterpart at the school level, the student nonresponse adjustment is intended to
reduce the mean square error and thus improve the accuracy of NAEP
assessment estimates. Also, like its counterpart at the school level, student
nonresponse adjustments were carried out separately by sample; that is, by

Development of Initial Student Nonresponse
Cells
Development of Final Student Nonresponse
Cells
Student Nonresponse Adjustment Factor
Calculation

grade (4, 8, 12);
school type (public, private); and
assessment subject (reading, mathematics, science, writing [WCBA]).

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_stud.aspx

7/22/2016 2:46 PM

12 of 60

Development of Final Student Nonresponse Cells for the 2011 Assessment
Similar to the school nonresponse adjustment, cell and adjustment factor size constraints are in place to prevent unstable nonresponse
adjustments or unacceptably large adjustment factors. All initial weighting cells with either fewer than 20 participating students or
adjustment factors greater than 2.0 for the full sample weight were collapsed with suitable adjacent cells. Simultaneously, all initial
weighting cells for any replicate with either fewer than 15 participating students or an adjustment factor greater than the maximum of
2.0 or 1.5 times the full sample nonresponse adjustment factor were collapsed with suitable adjacent cells.
Initial weighting cells were generally collapsed in reverse order of the cell structure; that is, starting at the bottom of the nesting
structure and working up toward the top level of the nesting structure. Race/ethnicity cells within SD/ELL group, school nonresponse
cell, age, and gender classes were collapsed first. If further collapsing was required after collapsing all race/ethnicity classes, cells
were next combined across gender, then age, and finally school nonresponse cells. Cells are never collapsed across SD/ELL for any
sample.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_stud_final.aspx

7/22/2016 2:46 PM

13 of 60

Development of Initial Student Nonresponse Cells for the 2011 Assessment
Initial student nonresponse cells are generally created within each sample as defined by grade, school type (public, private), and
assessment subject. However, when subjects are administered together by way of spiraling, the initial student nonresponse cells are
created across the subjects in the same spiral. The rationale behind this decision is that spiraled subjects are in the same schools and
the likelihood of whether an eligible student participates in an assessment is more related to its school than the subject of the
assessment booklet. In NAEP 2011, the reading, mathematics, and science assessments were spiraled together, but writing (WCBA)
was not spiraled with any other subject. The initial student nonresponse cells for the various NAEP 2011 samples are described
below.
Nonresponse adjustment procedures are not applied to excluded students because they are not required to complete an assessment.

Public School Samples for Reading, Mathematics, and Science at Grades 4 and 8
The initial student nonresponse cells for these samples were defined within grade, jurisdiction, and Trial Urban District Assessment
(TUDA) district using the following nesting cell structure:
students with disabilities (SD)/English language learners (ELL) by subject;
school nonresponse cell;
age1 (classified into "older" student and "modal age or younger" student);
gender; and
race/ethnicity.
The highest level variable in the cell structure separates students who were classified either as having disabilities (SD) or as English
language learners (ELL) from those who are neither, since SD or ELL students tend to score lower on assessment tests than
non-SD/non-ELL students. In addition, the students in the SD or ELL groups are further broken down by subject, since rules for
excluding students from the assessment differ by subject. Non-SD and non-ELL students are not broken down by subject, since the
exclusion rules do not apply to them.

Private School Samples for Reading, Mathematics, and Science at Grades 4 and 8
The initial weighting cells for these private school samples were formed hierarchically within grade as follows:
SD/ELL;
school nonresponse cell;
age1 (classified into "older" student and "modal age or younger" student);
gender; and
race/ethnicity.
Although exclusion rules differ by subject, there were not enough SD or ELL private school students to break out by subject as was
done for the public schools.

Public School Samples for WCBA at Grades 8 and 12
The initial weighting cells for these samples were formed hierarchically within grade using the following cell structure:
SD/ELL;
school nonresponse cell;
age1 (classified into "older" student and "modal age or younger" student);
gender; and
race/ethnicity.

Private School Samples for WCBA at Grades 8 and 12
The initial weighting cells for these samples were formed hierarchically within grade using the following cell structure:
SD/ELL;

7/22/2016 2:46 PM

14 of 60

school nonresponse cell;
age1 (classified into "older" student and "modal age or younger" student);
gender; and
race/ethnicity.
1

Older students are those born before October 1, 2000, for grade 4; October 1, 1996, for grade 8; and October 1, 1992, for grade 12.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_stud_initial.aspx

7/22/2016 2:46 PM

15 of 60

Student Nonresponse Adjustment Factor Calculation for the 2011 Assessment
In each final student nonresponse adjustment cell c for a given sample, the student nonresponse adjustment factor STU_NRAFc was
computed as follows:

where
Sc is the set of all eligible sampled students in cell c for a given sample,
Rc is the set of all assessed students within Sc,
STU_BWTk is the student base weight for a given student k,
SCH_TRIMk is the school-level weight trimming factor for the school associated with student k,
SCH_NRAFk is the school-level nonresponse adjustment factor for the school associated with student k, and
SUBJFACk is the subject factor for a given student k.
The student weight used in the calculation above is the adjusted student base weight, without regard to subject, adjusted for school
weight trimming and school nonresponse.
Nonresponse adjustment procedures are not applied to excluded students because they are not required to complete an assessment. In
effect, excluded students were placed in a separate nonresponse cell by themselves and all received an adjustment factor of 1. While
excluded students are not included in the analysis of the NAEP scores, weights are provided for excluded students in order to estimate
the size of this group and its population characteristics.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_nonresp_stud_factor.aspx

7/22/2016 2:46 PM

16 of 60

School and Student Weight Trimming Adjustments for the 2011 Assessment
Trimming of School
Weight trimming is an adjustment procedure that involves detecting and reducing extremely large
Base Weights
weights. "Extremely large weights" generally refer to large sampling weights that were not anticipated in
the design of the sample. Unusually large weights are likely to produce large sampling variances for
Trimming of Student
statistics of interest, especially when the large weights are associated with sample cases reflective of rare
Weights
or atypical characteristics. To reduce the impact of these large weights on variances, weight reduction
methods are typically employed. The goal of employing weight reduction methods is to reduce the mean
square error of survey estimates. While the trimming of large weights reduces variances, it also introduces some bias. However, it is
presumed that the reduction in the variances more than compensates for the increase in the bias, thereby reducing the mean square
error and thus improving the accuracy of survey estimates (Potter 1988). NAEP employs weight trimming at both the school and
student levels.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_trimming_adj.aspx

7/22/2016 2:46 PM

17 of 60

Trimming of School Base Weights for the 2011 Assessment
Large school weights can occur for schools selected from the NAEP new-school sampling frame and for private schools. New schools
that are eligible for weight trimming are schools with a disproportionately large student enrollment in a particular grade from a school
district that was selected with a low probability of selection. The school base weights for such schools may be large relative to what
they would have been if they had been selected as part of the original sample.
To detect extremely large weights among new schools, a comparison was made between a new school's school base weight and its
ideal weight (i.e., the weight that would have resulted had the school been selected from the original school sampling frame). If the
school base weight was more than three times the ideal weight, a trimming factor was calculated for that school that scaled the base
weight back to three times the ideal weight. The calculation of the school-level trimming factor for a new school s is expressed in the
following formula:

where
EXP_WTs is the ideal base weight the school would have received if it had been on the NAEP public school sampling frame,
and
SCH_BWTs is the actual school base weight the school received as a sampled school from the new school frame.
Twenty-one (21) new schools had their weights trimmed: seven at grade 4, and fourteen at grade 8.
Private schools eligible for weight trimming were Private School Universe Survey (PSS) nonrespondents who were found
subsequently to have either larger enrollments than assumed at the time of sampling, or an atypical probability of selection given their
affiliation, the latter being unknown at the time of sampling. For private school s, the formula for computing the school-level weight
trimming factor SCH_TRIMs is identical to that used for new schools. For private schools,
EXP_WTs is the ideal base weight the school would have received if it had been on the NAEP private school sampling frame
with accurate enrollment and known affiliation, and
SCH_BWTs is the actual school base weight the school received as a sampled private school.
No private schools had their weights trimmed.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_base_schtrim.aspx

7/22/2016 2:46 PM

18 of 60

Trimming of Student Weights for the 2011 Assessment
Large student weights generally come from compounding nonresponse adjustments at the school and student levels with artificially
low school selection probabilities, which can result from inaccurate enrollment data on the school frame used to define the school size
measure. Even though measures are in place to limit the number and size of excessively large weights—such as the implementation of
adjustment factor size constraints in both the school and student nonresponse procedures and the use of the school trimming
procedure—large student weights can occur due to compounding effects of the various weighting components.
The student weight trimming procedure uses a multiple median rule to detect excessively large student weights. Any student weight
within a given trimming group greater than a specified multiple of the median weight value of the given trimming group has its
weight scaled back to that threshold. Student weight trimming was implemented separately by grade, school type (public or private),
and subject. The multiples used were 3.5 for public school trimming groups and 4.5 for private school trimming groups. Trimming
groups were defined by jurisdiction and Trial Urban District Assessment (TUDA) districts for the public school samples for reading
and mathematics at grades 4 and 8, and science at grade 8; by the nation for the public school and private school samples for writing
[WCBA] at grades 8 and 12; and by affiliation (Catholic, Conservative Christian, Lutheran, and Other private) for the private school
samples for reading, mathematics, and science at grades 4 and 8.
The procedure computes the median of the nonresponse-adjusted student weights in the trimming group g for a given grade and
subject sample. Any student k with a weight more than M times the median received a trimming factor calculated as follows:

where
M is the trimming multiple,
MEDIANg is the median of nonresponse-adjusted student weights in trimming group g, and
STUWGTgk is the weight after student nonresponse adjustment for student k in trimming group g.
In the 2011 assessment, relatively few students had weights considered excessively large. Out of the approximately 987,000 students
included in the combined 2011 assessment samples, 710 students had their weights trimmed.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_studtrim.aspx

7/22/2016 2:46 PM

19 of 60

Student Weight Raking Adjustment for the 2011 Assessment
Weighted estimates of population totals for student-level subgroups for a given
Development of Final Raking Dimensions
grade will vary across subjects even though the student samples for each subject
generally come from the same schools. These differences are the result of
Raking Adjustment Control Totals
sampling error associated with the random assignment of subjects to students
through a process known as spiraling. For state assessments in particular, any
Raking Adjustment Factor Calculation
difference in demographic estimates between subjects, no matter how small, may
raise concerns about data quality. To remove these random differences and
potential data quality concerns, a new step was added to the NAEP weighting procedure starting in 2009. The new step adjusts the
student weights in such a way that the weighted sums of population totals for specific subgroups are the same across all subjects. The
new weighting step was implemented using a raking procedure and applied only to state-level assessments.
Raking is a weighting procedure based on the iterative proportional fitting process developed by Deming and Stephan (1940) and
involves simultaneous ratio adjustments to two or more marginal distributions of population totals. Each set of marginal population
totals is known as a dimension, and each population total in a dimension is referred to as a control total. Raking is carried out in a
sequence of adjustments. Sampling weights are adjusted to one marginal distribution and then to the second marginal distribution, and
so on. One cycle of sequential adjustments to the marginal distributions is called an iteration. The procedure is repeated until
convergence is achieved. The criterion for convergence can be specified either as the maximum number of iterations or an absolute
difference (or relative absolute difference) from the marginal population totals. More discussion on raking can be found in Oh and
Scheuren (1987).
For NAEP 2011, the student raking adjustment was carried out separately in each state for the reading and mathematics public school
samples at grades 4 and 8, and science public school samples at grade 8. The dimensions used in the raking process were National
School Lunch Program (NSLP) eligibility, race/ethnicity, SD/ELL status, and gender. The control totals for these dimensions were
obtained from the NAEP student sample weights of the reading, mathematics, and science samples combined.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_stud_raking.aspx

7/22/2016 2:46 PM

20 of 60

Development of Final Raking Dimensions for the 2011 Assessment
The raking procedure involved four dimensions. The variables used to define the dimensions are listed below along with the
categories making up the initial raking cells for each dimension.
National School Lunch Program (NSLP) eligibility
1. Eligible for free or reduced-price lunch
2. Otherwise
Race/Ethnicity
1. White, not Hispanic
2. Black, not Hispanic
3. Hispanic
4. Asian
5. American Indian/Alaska Native
6. Native Hawaiian/Pacific Islander
7. Two or More Races
SD/ELL status
1. SD, but not ELL
2. ELL, but not SD
3. SD and ELL
4. Neither SD nor ELL
Gender
1. Male
2. Female
In states containing districts that participated in TUDA at grades 4 and 8, the initial cells were created separately for each TUDA
district and the balance of the state. Similar to the procedure used for school and student nonresponse adjustments, limits were placed
on the magnitude of the cell sizes and adjustment factors to prevent unstable raking adjustments that could have resulted
in unacceptably large or small adjustment factors. Levels of a dimension were combined whenever there were fewer than 30 assessed
or excluded students (20 for any of the replicates) in a category, if the smallest adjustment was less than 0.5, or if the largest
adjustment was greater than 2 for the full sample or for any replicate.
If collapsing was necessary for the race/ethnicity dimension, the following groups were combined first: American Indian/Alaska
Native with Two or More Races; Asian with the Pacific Islander; and Black, not Hispanic with Hispanic. If further collapsing was
necessary, the five categories American Indian/Alaska Native; Two or More Races; Asian; Native Hawaiian/Pacific Islander; and
White, not Hispanic were combined. In some instances, all seven categories had to be collapsed.
If collapsing was necessary for the SD/ELL dimension, the SD/not ELL and SD/ELL categories were combined first, followed by
ELL/not SD if further collapsing was necessary. In some instances, all four categories had to be collapsed.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_stud_raking_final_dim.aspx

7/22/2016 2:46 PM

21 of 60

Raking Adjustment Control Totals for the 2011 Assessment
The control totals used in the raking procedure for NAEP 2011 grades 4, 8, and 12 were estimates of the student population derived
from the set of assessed and excluded students pooled across subjects. The control totals for category c within dimension d were
computed as follows:

where
Rc(d) is the set of all assessed students in category c of dimension d,
Ec(d) is the set of all excluded students in category c of dimension d,
STU_BWTk is the student base weight for a given student k,
SCH_TRIMk is the school-level weight trimming factor for the school associated with student k,
SCH_NRAFk is the school-level nonresponse adjustment factor for the school associated with student k,
STU_NRAFk is the student-level nonresponse adjustment factor for student k, and
SUBJFACk is the subject factor for student k.
The student weight used in the calculation of the control totals above is the adjusted student base weight, without regard to subject,
adjusted for school weight trimming, school nonresponse, and student nonresponse. Control totals were computed for the full sample
and for each replicate independently.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_stud_raking_ctrl_tots.aspx

7/22/2016 2:46 PM

22 of 60

Raking Adjustment Factor Calculation for the 2011 Assessment
For assessed and excluded students in a given subject, the raking adjustment factor STU_RAKEk was computed as follows:
First, the weight for student k was intialized as follows:

where
STU_BWTk is the student base weight for a given student k,
SCH_TRIMk is the school-level weight trimming factor for the school associated with student k,
SCH_NRAFk is the school-level nonresponse adjustment factor for the school associated with student k,
STU_NRAFk is the student-level nonresponse adjustment factor for student k, and
SUBJFACk is the subject factor for student k.
Then, the sequence of weights for the first iteration was calculated as follows for student k in category c of dimension d:
For dimension 1:

For dimension 2:

For dimension 3:

For dimension 4:

where
Rc(d) is the set of all assessed students in category c of dimension d,
Ec(d) is the set of all excluded students in category c of dimension d, and
Totalc(d) is the control total for category c of dimension d.
The process is said to converge if the maximum difference between the sum of adjusted weights and the control totals is 1.0 for each

7/22/2016 2:46 PM

23 of 60

category in each dimension. If after the sequence of adjustments the maximum difference was greater than 1.0, the process continues
to the next iteration, cycling back to the first dimension with the initial weight for student k equalling STUSAWTkadj(4) from the
previous iteration. The process continued until convergence was reached.
Once the process converged, the adjustment factor was computed as follows:

where STUSAWTk is the weight for student k after convergence.
The process was done independently for the full sample and for each replicate.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_stud_raking_factor_cal.aspx

7/22/2016 2:46 PM

24 of 60

Computation of Replicate Weights for the 2011 Assessment
In addition to the full-sample weight, a set of 62 replicate weights was provided for
each student. These replicate weights are used in calculating the sampling variance
of estimates obtained from the data, using the jackknife repeated replication method.
The method of deriving these weights was aimed at reflecting the features of the
sample design appropriately for each sample, so that when the jackknife variance
estimation procedure is implemented, approximately unbiased estimates of sampling
variance are obtained. This section gives the specifics for generating the replicate
weights for the 2011 assessment samples. The theory that underlies the jackknife
variance estimators used in NAEP studies is discussed in the section Replicate
Variance Estimation.

Defining Variance Strata and Forming
Replicates
Computing School-Level Replicate Factors
Computing Student-Level Replicate
Factors
Replicate Variance Estimation

In general, the process of creating jackknife replicate weights takes place at both the
school and student level. The precise implementation differs between those samples that involve the selection of Primary Sampling
Units (PSUs) and those where the school is the first stage of sampling. The procedure for this second kind of sample also differed in
2011 from all previous NAEP assessments. The change that was implemented permitted the introduction of a finite population
correction factor at the school sampling stage, developed by Rizzo and Rust (2011). In past assessments this adjustment factor has
always been implicitly assumed equal to 1.0, resulting in some overestimation of the sampling variance.
For each sample, the calculation of replicate weighting factors at the school level was conducted in a series of steps. First, each school
was assigned to one of 62 variance estimation strata. Then, a random subset of schools in each variance estimation stratum was
assigned a replicate factor of between 0 and 1. Next, the remaining subset of schools in the same variance stratum was assigned a
complementary replicate factor greater than 1. All schools in the other variance estimation strata were assigned a replicate factor of
exactly 1. This process was repeated for each of the 62 variance estimation stratum so that 62 distinct replicate factors were assigned
to each school in the sample.
This process was then repeated at the student level. Here, each individual sampled student was assigned to one of 62 variance
estimation strata, and 62 replicate factors with values either between 0 and 1, greater than 1, or exactly equal to 1 were assigned to
each student.
For example, consider a single hypothetical student. For replicate 37, that student’s student replicate factor might be 0.8, while for the
school to which the student belongs, for replicate 37, the school replicate factor might be 1.6. Of course, for a given student, for most
replicates, either the student replicate factor, the school replicate factor, or (usually) both, is equal to 1.0.
In the case of PSU-based samples, the replication procedure was only carried out at the school level. Conceptually, one can include
this process under the framework of replication at both school and student levels but where the replicate factors at the student level are
equal to 1.0 for every replicate for every student.
A replicate weight was calculated for each student, for each of the 62 replicates, using weighting procedures similar to those used for
the full-sample weight. Each replicate weight contains the school and student replicate factors described above. By repeating the
various weighting procedures on each set of replicates, the impact of these procedures on the sampling variance of an estimate is
appropriately reflected in the variance estimate.
Each of the 62 replicate weights for student k in school s in stratum j can be expressed as follows:

where
STU_BWTjsk is the student base weight;
SCH_REPFACjs(r) is the school-level replicate factor for replicate r;
SCH_NRAFjs(r) is the school-level nonresponse adjustment factor for replicate r;
STU_REPFACjsk(r) is the student-level replicate factor for replicate r;
STU_NRAFjsk(r) is the student-level nonresponse adjustment factor for replicate r;
SCH_TRIMjs is the school-level weight trimming adjustment factor;

7/22/2016 2:46 PM

25 of 60

STU_TRIMjsk is the student-level weight trimming adjustment factor; and
STU_RAKEjsk(r) is the student-level raking adjustment factor for replicate r.
Specific school and student nonresponse and student-level raking adjustment factors were calculated separately for each replicate,
thus the use of the index (r), and applied to the replicate student base weights. Computing separate nonresponse and raking adjustment
factors for each replicate allows resulting variances from the use of the final student replicate weights to reflect components of
variance due to these various weight adjustments.
School and student weight trimming adjustments were not replicated, that is, not calculated separately for each replicate. Instead, each
replicate used the school and student trimming adjustment factors derived for the full sample. Statistical theory for replicating
trimming adjustments under the jackknife approach has not been developed in the literature. Due to the absence of a statistical
framework, and since relatively few school and student weights in NAEP require trimming, the weight trimming adjustments were not
replicated.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts.aspx

7/22/2016 2:46 PM

26 of 60

Computing School-Level Replicate Factors for the 2011 Assessment
The calculation of school-level replicate factors for an assessment depended on the variance replication procedure, with or without
finite population corrections for the first stage of sampling.
Mathematics, Reading, and Science Assessments
The replicate variance estimation approach for the mathematics, reading, and science assessments involved finite population
corrections at the school level. The calculation of school-level replicate factors for these assessments depended upon whether or not a
school was selected with certainty. For certainty schools, the school-level replicate factors for all replicates are set to unity – this is
true regardless of whether or not the variance replication method uses finite population corrections – since certainty schools are not
subject to sampling variability. Alternatively, one can view the finite population correction factor for such schools as being equal to
zero. Thus, for each certainty school in a given assessment, the school-level replicate factor for each of the 62 replicates (r = 1, ..., 62)
was assigned as follows:

where SCH_REPFACjs(r) is the school-level replicate factor for school s in primary stratum j for the r-th replicate.
For noncertainty schools, where variance strata were formed by grouping schools into pairs or triplets, school-level replicate factors
were calculated for each of the 62 replicates based on this grouping. For schools in variance strata comprising pairs of schools, the
school-level replicate factors, SCH_REPFACjs(r), r = 1,..., 62, were calculated as follows:

where
min(πj1, πj2) is the smallest school probability between the two schools comprising Rjr,
Rjr is the set of schools within the r-th variance stratum for primary stratum j, and
Ujs is the variance unit (1 or 2) for school s in primary stratum j.
For noncertainty schools in variance strata comprising three schools, the school-level replicate factors, SCH_REPFACjs(r), r = 1,...,
62, were calculated as follows:
For school s from primary stratum j, variance stratum r,

while for r' = r + 31 (mod 62):

7/22/2016 2:46 PM

27 of 60

and for all other r* other than r and r':

where
min(πj1, πj2,πj3) is the smallest school probability among the three schools comprising Rjr,
Rjr is the set of schools within the r-th variance stratum for primary stratum j, and
Ujs is the variance unit (1, 2, or 3) for school s in primary stratum j.
Computer-Based Writing Assessments
The replicate variance estimation approach for the computer-based writing assessments did not involve school-level finite population
corrections. As described in the section Defining Variance Strata and Forming Replicates, variance strata were defined by grouping
first-stage units (schools or geographic PSUs) into pairs or triplets. The calculation of the school-level replicate factors for each of the
62 replicates was based on this grouping.
For schools in variance strata comprising pairs of first-stage units, the school-level replicate factors, SCH_REPFACjs(r), r = 1,..., 62,
were calculated as follows:

where
Rjr is the set of schools within the r-th variance stratum for primary stratum j, and
Ujs is the variance unit (1 or 2) for school s in primary stratum j.
For schools in variance strata comprising three first-stage units, school-level replicate factors for school were calculated as follows:
For school s in primary stratum j, variance stratum r,

while for r' = r + 31 (mod 62):

7/22/2016 2:46 PM

28 of 60

and for all other r* other than r and r':

where
Rjr is the set of schools within the r-th variance stratum for primary stratum j, and
Ujsis the variance unit (1, 2, or 3) for school s in primary stratum j.
In primary strata with fewer than 62 variance strata, the replicate weights for the “unused” variance strata (the remaining ones up to
62) for these schools were set equal to the school base weight (so that those replicates contribute nothing to the variance estimate).

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts_schl.aspx

7/22/2016 2:46 PM

29 of 60

Computing Student-Level Replicate Factors for the 2011 Assessment
The calculation of student-level replicate factors for an assessment depended on whether the variance replication procedure
incorporated finite population corrections at the first stage of sampling.
Mathematics, Reading, and Science Assessments
For the mathematics, reading, and science assessments, which involved school-level finite population corrections, the student-level
replication factors were calculated the same way regardless of whether or not the student was in a certainty school.
For students in student-level variance strata comprising pairs of students, the student-level replicate factors, STU_REPFACjsk(r), r =
1,..., 62, were calculated as follows:

where
πs is the probability of selection for school s,
Rjsr is the set of students within the r-th variance stratum for school s in primary stratum j, and
Ujsk is the variance unit (1 or 2) for student k in school s in stratum j.
For students in variance strata comprising three students, the student-level replicate factors STU_REPFACjsk(r), r = 1,..., 62, were
calculated as follows:

while for r' = r + 31 (mod 62):

and for all other r* other than r and r':
STU_REPFACjsk(r*) = 1
where
πs is the probability of selection for school s,
Rjsr is the set of students within the r-th replicate stratum for school s in stratum j, and
Ujsk is the variance unit (1, 2, or 3) for student k in school s in stratum j.
Note, for students in certainty schools, where πs = 1, the student replicate factors are 2 and 0 in the case of pairs, and 1.5, 1.5, and 0 in
the case of triples.

7/22/2016 2:46 PM

30 of 60

Computer-Based Writing Assessments
For the computer-based writing assessments, which did not involve first-stage finite population corrections and there were no
certainty schools, replication at the student level was not relevant. As a consequence, the replicate factors for all replicates for every
student in the computer-based writing assessments were set to unity. That is, the student-level replicate factors, STU_REPFACjsk(r), r
= 1,..., 62, were calculated as follows:
STU_REPFACjsk(r) = 1.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts_stud.aspx

7/22/2016 2:46 PM

31 of 60

Defining Variance Strata and Forming Replicates for the 2011 Assessment
In the NAEP 2011 assessment, replicates were formed separately for each sample indicated by grade (4, 8, 12), school type (public,
private), and assessment subject (mathematics, reading, science, computer-based writing). To reflect the school-level finite population
corrections in the variance estimators for the two-stage samples used for mathematics, reading, and science assessments, variance
strata were formed at both the school and student levels for these samples. For the computer-based writing assessments, which did not
use school-level finite population corrections in the variance estimators, variance strata were formed only at the first-stage level.
The first step in forming replicates was to create variance strata in each primary stratum. In 2011, the mathematics, reading, and
science assessments required formation of variance strata at both the school and student levels. The computer-based writing
assessments required formation of variance strata only at the first-stage level.
The next step was to sort the appropriate sampling unit (school or student) in the order of its selection within the primary stratum and
then pair off into preliminary variance strata. Sorting sample units by their order of sample selection reflects the implicit stratification
and systematic sampling features of the sample design. Within each primary stratum with an even number of sampling units, all of the
preliminary variance strata consisted of pairs of sampling units. However, within primary strata with an odd number of sampling
units, all but one variance strata consisted of pairs of sampling units, while the last one consisted of three sampling units.
If there were more than 62 preliminary variance strata within a primary stratum, the preliminary variance strata were grouped to form
62 variance strata. This grouping effectively maximized the distance in the sort order between grouped preliminary variance strata.
The first 62 preliminary variance strata, for example, were assigned to 62 different final variance strata in order (1 through 62), with
the next 62 preliminary variance strata assigned to final variance strata 1 through 62, so that, for example, preliminary variance
stratum 1, preliminary variance stratum 63, preliminary variance stratum 125 (if in fact there were that many), etc., were all assigned
to the first final variance stratum.
If, on the other hand, there were fewer than 62 preliminary variance strata within a primary stratum, then the number of final variance
strata was set equal to the number of preliminary variance strata. For example, consider a primary stratum with 111 sampled units
sorted in their order of selection. The first two units were in the first preliminary variance stratum; the next two units were in the
second preliminary variance stratum, and so on, resulting in 54 preliminary variance strata with two sample units each (doublets). The
last three sample units were in the 55th preliminary variance stratum (triplet). Since there are no more than 62 preliminary variance
strata, these were also the final variance strata.
Within each preliminary variance stratum containing a pair of sampling units, one sampling unit was randomly assigned as the first
variance unit and the other as the second variance unit. Within each preliminary variance stratum containing three sampling units, the
three first-stage units were randomly assigned variance units 1 through 3.
Reading, Mathematics, and Science Assessments
As described above, the mathematics, reading, and science assessments required variance strata at both the school and student level.
At the school-level for these samples, formation of variance strata did not pertain to certainty schools, since they are not subject to
sampling variability, but only to noncertainty schools. The primary stratum for noncertainty schools was the highest school-level
sampling stratum variable listed below, and the order of selection was defined by sort order on the school sampling frame.
Trial Urban District Assessment (TUDA) districts, remainder of states (for states with TUDAs), or entire states for the public
school samples at grades 4 and 8; and
Private school affiliation (Catholic, Lutheran, Conservative Christian, and Other private) for the private school samples at
grades 4 and 8.
At the student-level, all students were assigned to variance strata. The primary stratum was school, and the order of selection was
defined by session number and position on the administration schedule.
Computer-Based Writing Assessments
As described above, variance strata for the computer-based writing assessments were formed at the first-stage sampling level and so
differed by certainty and noncertainty PSUs. For noncertainty PSUs, the first-stage sampling units were PSUs, and the primary
stratum was the combination of region and metropolitan status (MSA or non-MSA). For certainty PSUs, the first-stage sampling units
were schools, and the primary stratum was school type (public or private).

7/22/2016 2:46 PM

32 of 60

For noncertainty PSUs, where only one PSU was selected per PSU stratum, variance strata were formed by pairing sampled PSUs
with similar stratum characteristics within the same primary stratum (region by metropolitan status). This was accomplished by first
sorting the 38 sampled PSUs by PSU stratum number and then grouping adjacent PSUs into 19 pairs. The values for a PSU stratum
number reflect region and metropolitan status, as well as demographic and socioeconomic characteristics such as percentage of Black
youth and percentage of children whose family income is below the poverty threshold. The formation of these 19 variance strata in
this manner models a design of selecting two PSUs with probability proportional to size with replacement from each of 19 strata.
For certainty PSUs, the first stage of sampling is at the school level, and the formation of variance strata must reflect the sampling of
schools within the certainty PSUs. Variance strata were formed by sorting the sampled schools in the 29 certainty PSUs by their order
of selection within a primary stratum (school type) so that the sort order reflected the implicit stratification (region, locality type,
minority status, and student enrollment for public schools; and region, private school type, and student enrollment size for private
schools) and systematic sampling features of the sample design.
The first-stage units were then paired off into 43 preliminary variance strata. Within each primary stratum with an even number of
first-stage units, all of the preliminary variance strata were pairs, and within primary strata with an odd number of first-stage units,
one of the variance strata was a triplet (the last one), and all others were pairs.
If there were more than 43 preliminary variance strata within a primary stratum, the preliminary variance strata were grouped to form
43 variance strata. This grouping effectively maximized the distance in the sort order between grouped preliminary variance strata.
The first 43 preliminary variance strata, for example, were assigned to 43 different final variance strata in order (1 through 43), with
the next 43 preliminary variance strata assigned to final variance strata 1 through 43, so that, for example, preliminary variance
stratum 1, preliminary variance stratum 44, preliminary variance stratum 87 (if there were that many), etc., were all assigned to the
first final variance stratum. The final variance strata for the schools in the certainty PSUs were 1 through 43.
Within each pair of preliminary variance strata, one first-stage unit, designated at random, was assigned as the first variance unit and
the other first-stage unit as the second variance unit. Within each triplet preliminary variance stratum, the three schools were
randomly assigned variance units 1 through 3.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts_strata.aspx

7/22/2016 2:46 PM

33 of 60

Replicate Variance Estimation for the 2011 Assessment
Variances for NAEP assessment estimates are computed using the paired jackknife replicate variance procedure. This technique is
applicable for common statistics, such as means and ratios, and differences between these for different subgroups, as well as for more
complex statistics such as linear or logistic regression coefficients.
In general, the paired jackknife replicate variance procedure involves initially pairing clusters of first-stage sampling units to form H
variance strata (h = 1, 2, 3, ...,H) with two units per stratum. The first replicate is formed by assigning, to one unit at random from the
first variance stratum, a replicate weighting factor of less than 1.0, while assigning the remaining unit a complementary replicate
factor greater than 1.0, and assigning all other units from the other (H - 1) strata a replicate factor of 1.0. This procedure is carried out
for each variance stratum resulting in H replicates, each of which provides an estimate of the population total.
In general, this process is repeated for subsequent levels of sampling. In practice, this is not practicable for a design with three or
more stages of sampling, and the marginal improvement in precision of the variance estimates would be negligible in all such cases in
the NAEP setting. Thus in NAEP, when a two-stage design is used – sampling schools and then students – beginning in 2011
replication is carried out at both stages. (See Rizzo and Rust (2011) for a description of the methodology.) When a three-stage design
is used, involving the selection of geographic Primary Sampling Units (PSUs), then schools, and then students, the replication
procedure is only carried out at the first stage of sampling (the PSU stage for noncertainty PSUs, and the school stage within certainty
PSUs). In this situation, the school and student variance components are correctly estimated, and the overstatement of the
between-PSU variance component is relatively very small.
The jackknife estimate of the variance for any given statistic is given by the following formula:

where
represents the full sample estimate of the given statistic, and
represents the corresponding estimate for replicate h.
Each replicate undergoes the same weighting procedure as the full sample so that the jackknife variance estimator reflects the
contributions to or reductions in variance resulting from the various weighting adjustments.
The NAEP jackknife variance estimator is based on 62 variance strata resulting in a set of 62 replicate weights assigned to each
school and student.
The basic idea of the paired jackknife variance estimator is to create the replicate weights so that use of the jackknife procedure
results in an unbiased variance estimator for simple totals and means, which is also reasonably efficient (i.e., has a low variance as a
variance estimator). The jackknife variance estimator will then produce a consistent (but not fully unbiased) estimate of variance for
(sufficiently smooth) nonlinear functions of total and mean estimates such as ratios, regression coefficients, and so forth (Shao and
Tu, 1995).
The development below shows why the NAEP jackknife variance estimator returns an unbiased variance estimator for totals and
means, which is the cornerstone to the asymptotic results for nonlinear estimators. See for example Rust (1985). This paper also
discusses why this variance estimator is generally efficient (i.e., more reliable than alternative approaches requiring similar
computational resources).
The development is done for an estimate of a mean based on a simplified sample design that closely approximates the sample design
for first-stage units used in the NAEP studies. The sample design is a stratified random sample with H strata with population weights
Wh, stratum sample sizes nh, and stratum sample means
. The population estimator and standard unbiased variance estimator
are:

7/22/2016 2:46 PM

34 of 60

with

The paired jackknife replicate variance estimator assigns one replicate h=1,…, H to each stratum, so that the number of replicates
equals H. In NAEP, the replicates correspond generally to pairs and triplets (with the latter only being used if there are an odd number
of sample units within a particular primary stratum generating replicate strata). For pairs, the process of generating replicates can be
viewed as taking a simple random sample (J) of size nh/2 within the replicate stratum, and assigning an increased weight to the
sampled elements, and a decreased weight to the unsampled elements. In certain applications, the increased weight is double the full
sample weight, while the decreased weight is in fact equal to zero. In this simplified case, this assignment reduces to replacing
with
, the latter being the sample mean of the sampled nh/2 units. Then the replicate estimator corresponding to stratum r is

The r-th term in the sum of squares for

is thus:

In stratified random sampling, when a sample of size nr/2 is drawn without replacement from a population of size nr,, the sampling
variance is

See for example Cochran (1977), Theorem 5.3, using nr, as the “population size,” nr/2 as the “sample size,” and sr2 as
the “population variance” in the given formula. Thus,

Taking the expectation over all of these stratified samples of size nr/2, it is found that

In this sense, the jackknife variance estimator “gives back” the sample variance estimator for means and totals as desired under the
theory.
In cases where, rather than doubling the weight of one half of one variance stratum and assigning a zero weight to the other, the
weight of one unit is multiplied by a replicate factor of (1+δ), while the other is multiplied by (1- δ), the result is that

In this way, by setting δ equal to the square root of the finite population correction factor, the jackknife variance estimator is able to
incorporate a finite population correction factor into the variance estimator.
In practice, variance strata are also grouped to make sure that the number of replicates is not too large (the total number of variance
strata is usually 62 for NAEP). The randomization from the original sample distribution guarantees that the sum of squares

7/22/2016 2:46 PM

35 of 60

contributed by each replicate will be close to the target expected value.
For triples, the replicate factors are perturbed to something other than 1.0 for two different replicate factors, rather than just one as in
the case of pairs. Again in the simple case where replicate factors that are less than 1 are all set to 0, with the replicate weight factors
calculated as follows.
For unit i in variance stratum r

where weight wi is the full sample base weight.
Furthermore, for r' = r + 31 (mod 62):

And for all other values r*, other than r and r´,wi(r*) = 1.
with
for replicate r, where
is the
In the case of stratified random sampling, this formula reduces to replacing
sample mean from a “2/3” sample of 2nr/3 units from the nr sample units in the replicate stratum, and replacing with
for
is the sample mean from another overlapping “2/3” sample of 2nr/3 units from the nrsample units in the
replicate r', where
replicate stratum.
The r-th and r´-th replicates can be written as:

From these formulas, expressions for the r-th and r´-th components of the jackknife variance estimator are obtained (ignoring other
sums of squares from other grouped components attached to those replicates):

These sums of squares have expectations as follows, using the general formula for sampling variances:

7/22/2016 2:46 PM

36 of 60

Thus,

as desired again.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_repwts_appdx.aspx

7/22/2016 2:46 PM

37 of 60

Quality Control on Weighting Procedures for the 2011 Assessment
Given the complexity of the weighting procedures utilized in NAEP, a range of quality
control (QC) checks was conducted throughout the weighting process to identify potential
problems with collected student-level demographic data or with specific weighting
procedures. The QC processes included
checks performed within each step of the weighting process;

Main QC Findings of Interest
Participation, Exclusion, and
Accommodation Rates
Nonresponse Bias Analyses

checks performed across adjacent steps of the weighting process;
review of participation, exclusion, and accommodation rates;
checking demographic data of individual schools;
comparisons with 2009 demographic data; and
nonresponse bias analyses.
To validate the weighting process, extensive tabulations of various school and student characteristics at different stages of the
process were conducted. The school-level characteristics included in the tabulations were minority enrollment, median income (based
on the school ZIP code area), and urban-centric locale. At the student level, the tabulations included race/ethnicity, gender, relative
age, students with disability (SD) status, English language learners (ELL) status, and participation status in National School Lunch
Program (NSLP).

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_qa.aspx

7/22/2016 2:46 PM

38 of 60

Main Quality Control Findings of Interest for the 2011 Assessment
Final participation, exclusion, and accommodation rates are presented in quality control tables for
each grade and subject by geographic domain and school type. School-level
participation rates have been calculated according to National Center for Education Statistics
(NCES) standards as they have been for previous assessments.
School-level participation rates were below 85 percent for private schools at all three grades (4, 8,
and 12), for public schools of the Bureau of Indian Education (BIE) for grades 4 and 8, and for
Colorado for grade 8. Student-level participation rates were also below 85 percent for students in
Detroit for grade 8 public schools. As required by NCES standards, nonresponse bias analyses
were conducted on each reporting group falling below the 85 percent participation threshold.

Final Participation,
Exclusion, and
Accommodation Rates
Grade 4 Mathematics
Grade 4 Reading
Grade 8 Mathematics
Grade 8 Reading
Grade 8 Science
Grade 8 Writing (WCBA)
Grade 12 Writing (WCBA)

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_qa_findings.aspx

7/22/2016 2:46 PM

39 of 60

Participation, Exclusion, and Accommodation Rates for Grade 8 Mathematics for
the 2011 Assessment
The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 mathematics
assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column
headings of the table.
The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The
rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is
represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school
population that is represented by the responding schools in the sample. These rates differ because schools differ in size.
Participation, exclusion, and accommodation rates, grade 8 mathematics assessment, by school type and jurisdiction: 2011

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

8,700

97.57

88.42

209,000

2.45

92.72

9.74

National
Northeast all
Midwest all
South all
West all

8,600

97.54

88.31

204,000

2.46

92.71

9.67

1,400
2,400
2,700
2,100

95.45
98.76
97.65
97.80

79.60
91.49
89.96
89.74

33,200
45,900
73,300
50,200

2.25
2.62
3.22
1.30

91.82
93.33
93.13
92.19

14.40
9.84
8.35
8.11

National public
Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
District of Columbia
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana

7,500
125
167
136
126
257
130
119
68
102
233
132
81
113
223
113

99.79
100.00
99.90
99.02
100.00
100.00
99.87
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

99.76
100.00
97.93
99.08
100.00
100.00
97.31
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

192,000
3,200
2,900
3,200
3,200
8,400
3,200
3,100
3,200
3,000
7,100
4,800
3,300
3,400
4,700
3,200

2.64
1.21
3.15
1.14
1.36
1.08
0.85
1.31
3.08
4.37
1.83
2.73
1.87
1.28
2.32
2.56

92.53
93.72
89.48
92.97
92.53
91.81
92.71
92.90
93.15
89.99
92.56
92.94
91.67
94.32
93.35
92.93

10.14
3.59
14.39
8.87
11.63
7.47
9.96
12.26
10.86
15.28
16.13
7.41
10.69
7.23
11.56
12.15

School type and jurisdiction
All
all1

Weighted
percent of
students
accommodated

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools, but not schools in Puerto Rico.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

40 of 60

School type and jurisdiction
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

138
148
156
163
143
176
154
180
168
121
136
200
169
100
96
116
136
174
159
209
194
149
144
168
61
116
261
123
236
125
124
108
140
117
179

100.00
100.00
100.00
100.00
100.00
99.05
99.46
100.00
100.00
100.00
100.00
99.86
100.00
99.70
100.00
100.00
99.09
99.08
100.00
99.99
100.00
100.00
99.10
100.00
100.00
100.00
100.00
100.00
99.09
100.00
100.00
100.00
100.00
100.00
100.00

100.00
100.00
100.00
100.00
100.00
98.82
98.47
100.00
100.00
100.00
100.00
98.41
100.00
97.35
100.00
100.00
99.40
99.67
100.00
99.47
100.00
100.00
99.26
100.00
100.00
100.00
100.00
100.00
99.63
100.00
100.00
100.00
100.00
100.00
100.00

3,000
3,200
4,600
3,100
3,100
4,300
4,500
4,900
3,500
3,100
3,000
3,000
3,000
3,300
3,100
3,000
4,000
4,700
5,000
2,700
4,300
3,000
3,400
4,500
3,100
3,200
3,500
3,300
9,100
3,500
2,400
3,200
3,700
3,200
4,100

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

1.44
1.33
3.33
1.44
1.55
6.32
4.03
3.60
2.11
1.07
1.35
1.58
3.59
3.07
1.75
4.21
1.96
1.38
1.84
4.26
5.02
9.82
1.43
2.43
1.26
3.76
1.72
3.77
5.16
2.72
1.12
2.87
1.66
1.51
2.00

93.13
93.02
93.50
92.63
91.83
92.30
91.65
92.84
93.04
93.56
93.71
89.95
93.50
93.73
90.87
92.20
91.30
91.04
91.83
94.66
92.54
92.29
93.12
91.60
92.06
93.57
94.31
91.46
93.67
91.14
93.96
93.47
91.82
93.34
92.92

13.93
9.08
7.96
12.76
14.44
6.55
14.99
7.81
8.65
6.19
10.32
9.34
9.13
8.62
13.94
13.88
9.83
18.35
12.36
8.97
9.68
4.19
10.54
13.12
13.48
7.84
7.29
7.85
5.11
8.43
14.82
9.27
10.17
9.24
14.18

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools, but not schools in Puerto Rico.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

41 of 60

School type and jurisdiction
Wyoming
BIE

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

108
116

100.00
83.16

100.00
84.68

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

2,500
1,100

1.25
1.85

92.42
91.35

10.81
11.45

DoDEA 2
Trial Urban (TUDA) Districts
Albuquerque
Atlanta
Austin
Baltimore City
Boston
Charlotte
Chicago
Cleveland
Dallas
Detroit
Fresno
Hillsborough
Houston
Jefferson County, KY
Los Angeles
Miami
Milwaukee
New York City
Philadelphia
San Diego
District of Columbia (TUDA)

72

98.56

95.31

2,100

2.68

95.40

7.98

34
26
24
75
45
38
117
71
41
63
23
50
50
46
71
86
59
91
60
31
50

100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

1,400
1,600
1,800
1,400
1,400
1,700
2,200
1,300
1,700
1,900
1,500
1,600
2,500
1,700
2,300
2,900
1,400
2,500
1,500
1,300
1,700

3.42
2.50
4.64
12.50
5.79
1.32
3.33
5.59
4.89
8.07
1.19
1.84
5.61
3.16
1.30
1.87
4.90
1.04
6.71
2.83
6.58

89.29
92.80
91.43
87.44
92.02
92.39
95.57
91.35
93.88
84.39
91.56
93.23
92.79
91.83
92.39
92.99
91.92
91.35
90.86
94.84
88.34

12.40
8.38
9.07
7.50
18.94
11.29
15.84
24.17
6.24
8.23
6.64
20.81
5.47
8.08
9.21
17.82
25.48
24.15
17.82
7.94
18.13

National private
Catholic
Non-Catholic private
Lutheran
Conservative Christian
Other private

930
332
598
141
149
308

74.40
93.23
57.54
92.73
72.51
45.71

69.89
92.09
58.75
91.45
74.78
45.85

8,800
4,600
4,200
1,000
1,200
2,000

0.50
0.53
0.47
0.33
0.47
0.49

94.65
95.07
94.26
96.31
94.46
93.90

4.54
3.32
5.70
2.13
3.67
7.06

Puerto Rico

110

100.00

100.00

5,100

1.03

93.14

17.11

1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools, but not schools in Puerto Rico.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

42 of 60

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_gr8math_2011.aspx

7/22/2016 2:46 PM

43 of 60

Participation, Exclusion, and Accommodation Rates for Grade 8 Writing (WCBA)
for the 2011 Assessment
The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 writing
[WCBA] assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the
column headings of the table.
The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The
rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is
represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school
population that is represented by the responding schools in the sample. These rates differ because schools differ in size.

Participation, exclusion, and accommodation rates, grade 8 writing (WCBA) assessment, by school type and
jurisdiction: 2011

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
only)

Number
of
students
sampled,
rounded

1,000

97.27

87.29

1,000

97.27

178
204
400
265

National public
National private
Catholic
Non-Catholic private

School type and
jurisdiction
All
all1

National
Northeast all
Midwest all
South all
West all

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

27,400

1.72

93.99

8.05

87.29

27,400

1.72

93.99

8.05

95.36
98.83
97.15
97.43

80.39
93.61
86.74
86.34

4,400
5,300
10,700
7,000

2.08
1.78
1.71
1.38

93.20
94.19
94.62
93.43

11.42
10.14
6.48
5.80

890

99.73

99.86

25,200

1.84

93.99

8.49

157
50
107

71.21
95.53
52.06

68.66
95.23
56.42

2,200
1,100
1,100

0.28
0.53
0.08

93.97
94.72
93.35

3.01
3.12
2.92

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_g8writing_2011.aspx

7/22/2016 2:46 PM

44 of 60

Participation, Exclusion, and Accommodation Rates for Grade 12 Writing (WCBA)
for the 2011 Assessment
The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 12 writing
[WCBA] assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the
column headings of the table.
The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The
rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is
represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school
population that is represented by the responding schools in the sample. These rates differ because schools differ in size.

Participation, exclusion, and accommodation rates, grade 12 writing (WCBA) assessment, by school type and
jurisdiction: 2011

School type and
jurisdiction
All
National all1
Northeast all
Midwest all
South all
West all
National public
National private
Catholic
Non-Catholic private

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
only)

Number
of
students
sampled,
rounded

1,400

93.52

89.26

1,400

93.52

245
266
495
367

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

36,500

2.33

86.97

6.65

89.26

36,500

2.33

86.97

6.65

91.91
96.93
94.66
89.70

79.32
95.76
90.91
85.00

6,400
7,300
13,200
9,700

2.09
2.20
2.72
2.02

84.18
86.40
88.36
87.73

8.87
8.79
4.79
5.70

1,200

96.04

97.12

33,400

2.52

86.96

6.84

177
55
122

67.23
76.60
58.35

67.45
75.42
65.42

3,100
1,500
1,600

0.27
0.11
0.42

87.16
85.96
88.34

4.68
3.31
6.02

1

Includes national public, national private, Bureau of Indian Education, and Department of Defense Education Activity
schools located in the United States.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Writing Assessment.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_gr12writing_2011.aspx

7/22/2016 2:46 PM

45 of 60

Participation, Exclusion, and Accommodation Rates for Grade 4 Mathematics for
the 2011 Assessment
The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 4 mathematics
assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column
headings of the table.
The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The
rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is
represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school
population that is represented by the responding schools in the sample. These rates differ because schools differ in size.
Participation, exclusion, and accommodation rates, grade 4 mathematics assessment, by school type and jurisdiction: 2011

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

9,400

97.44

91.77

242,000

2.05

94.53

11.62

National
Northeast all
Midwest all
South all
West all

9,200

97.41

91.65

237,000

2.06

94.53

11.51

1,600
2,400
2,900
2,300

95.19
97.76
97.64
98.27

84.70
91.40
93.31
94.56

37,800
51,900
84,200
60,400

1.78
1.82
2.65
1.54

94.25
94.37
94.73
94.58

15.43
11.48
10.67
10.16

National public
Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
District of Columbia
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana

8,200
117
202
127
123
292
124
116
109
153
235
174
118
137
193
117

99.83
98.95
100.00
99.03
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

99.86
99.76
100.00
99.25
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

225,000
3,300
3,100
4,200
3,900
10,100
4,000
3,400
3,900
2,400
8,000
6,000
3,800
4,000
5,600
3,900

2.22
1.16
2.81
1.02
1.00
1.56
1.14
1.27
3.59
5.24
1.59
1.65
1.76
1.21
2.26
2.15

94.44
95.00
92.61
94.29
94.87
95.27
92.20
93.37
94.12
94.54
94.51
94.48
93.35
95.34
93.33
94.69

12.12
4.17
17.86
15.32
14.07
7.07
14.32
15.55
11.92
14.27
18.87
10.35
11.27
8.51
12.84
14.06

School type and jurisdiction
All
all1

Weighted
percent of
students
accommodated

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department
of Defense Education Activity schools, but not schools in Puerto Rico..
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

46 of 60

School type and jurisdiction
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

141
148
158
134
166
173
202
172
148
117
131
206
181
118
133
118
155
158
173
272
205
140
149
167
125
114
209
119
306
130
226
115
141
152
190

100.00
99.18
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
99.17
100.00
100.00
100.00
99.97
100.00
100.00
99.08
100.00
100.00
100.00
100.00
100.00
99.08
100.00
100.00
100.00
100.00
100.00
100.00

100.00
99.30
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
98.83
100.00
100.00
100.00
99.61
100.00
100.00
99.36
100.00
100.00
100.00
100.00
100.00
99.23
100.00
100.00
100.00
100.00
100.00
100.00

3,700
3,500
5,500
3,900
3,500
5,200
5,600
4,700
4,100
3,400
3,900
3,500
3,400
4,400
3,600
3,600
4,700
5,200
5,900
3,400
4,900
3,500
4,100
5,200
3,500
3,700
3,600
3,900
11,000
4,500
3,000
4,100
4,400
3,400
4,900

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

1.41
1.65
3.08
1.74
1.59
5.61
3.17
2.15
1.50
0.83
1.65
1.52
1.50
2.29
1.74
3.30
2.61
1.33
1.78
3.59
2.32
8.27
2.67
1.39
0.93
1.27
1.78
3.40
4.15
2.02
1.58
2.08
1.90
1.50
1.66

94.94
94.25
94.48
93.65
94.48
94.66
94.16
94.11
94.02
94.99
93.58
94.24
95.57
94.93
93.95
94.52
93.89
94.13
94.01
95.26
94.03
95.42
93.47
94.13
94.41
94.15
95.31
93.79
95.24
93.69
94.12
94.76
94.31
94.58
94.98

14.81
12.54
8.54
17.54
15.07
11.17
15.25
8.75
12.52
5.87
10.16
7.68
14.36
22.47
15.33
14.19
14.86
20.59
12.44
9.31
13.21
6.72
14.60
12.58
13.31
10.28
9.00
10.37
8.31
10.20
14.70
11.71
13.75
8.78
15.56

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department
of Defense Education Activity schools, but not schools in Puerto Rico..
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

47 of 60

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

Wyoming
BIE

202
135

100.00
83.26

100.00
84.85

DoDEA 2
Trial Urban (TUDA) Districts
Albuquerque
Atlanta
Austin
Baltimore City
Boston
Charlotte
Chicago
Cleveland
Dallas
Detroit
Fresno
Hillsborough
Houston
Jefferson County, KY
Los Angeles
Miami
Milwaukee
New York City
Philadelphia
San Diego
District of Columbia (TUDA)

120

98.91

52
66
55
70
86
57
95
86
55
58
53
56
86
56
76
87
68
82
57
54
106

National private
Catholic
Non-Catholic private
Lutheran
Conservative Christian
Other private
Puerto Rico

School type and jurisdiction

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

3,200
1,200

1.61
1.38

93.72
91.91

11.97
16.51

97.32

3,800

2.74

94.06

10.38

100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

2,000
2,100
2,000
1,700
1,900
1,900
2,700
1,600
2,000
1,500
2,200
1,800
3,100
2,200
2,600
2,900
1,500
2,700
1,700
1,900
1,600

2.74
0.99
3.96
11.23
4.64
1.15
2.35
5.59
2.94
5.71
1.27
1.68
4.12
4.91
1.76
2.85
2.75
1.63
3.88
2.61
6.48

93.15
96.13
94.12
93.05
93.74
94.44
94.44
94.44
96.63
88.79
94.15
95.05
95.17
95.22
94.87
96.27
94.32
94.24
94.52
94.87
93.98

19.19
8.38
16.52
8.11
16.87
11.73
20.21
20.97
7.70
6.41
6.77
25.94
14.46
9.50
9.35
22.60
27.73
27.11
15.93
8.20
15.15

748
264
484
107
123
254

73.51
96.27
55.34
94.87
73.13
42.23

68.47
95.93
56.27
92.38
70.86
44.57

6,300
3,400
2,900
722
925
1,300

0.30
0.25
0.35
0.36
0.30
0.37

95.60
95.86
95.37
96.62
94.16
95.74

4.59
3.75
5.33
3.31
2.79
6.73

139

100.00

100.00

4,800

0.47

94.52

22.94

1 Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department
of Defense Education Activity schools, but not schools in Puerto Rico..
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

48 of 60

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_g4math_2011.aspx

7/22/2016 2:46 PM

49 of 60

Participation, Exclusion, and Accommodation Rates for Grade 4 Reading for the
2011 Assessment
The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 4 reading
assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column
headings of the table.
The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The
rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is
represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school
population that is represented by the responding schools in the sample. These rates differ because schools differ in size.
Participation, exclusion, and accommodation rates, grade 4 reading assessment, by school type and jurisdiction: 2011

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

9,200

97.41

91.65

National
Northeast all
Midwest all
South all
West all

9,200

97.41

91.65

1,600
2,400
2,900
2,300

95.19
97.76
97.64
98.27

National public
Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
District of Columbia
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana

8,200
117
202
127
123
292
124
116
109
153
235
174
118
137
193
117

99.83
98.95
100.00
99.03
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

School type and jurisdiction
All
all1

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

245,000

3.65

94.57

9.64

245,000

3.65

94.57

9.64

84.70
91.40
93.31
94.56

39,100
53,700
87,100
62,600

3.75
2.45
5.19
2.19

94.18
94.61
94.61
94.76

13.51
10.44
8.07
8.75

99.86
99.76
100.00
99.25
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

233,000
3,400
3,200
4,400
4,100
10,500
4,100
3,500
4,000
2,400
8,300
6,300
3,900
4,100
5,800
4,000

3.92
2.27
2.03
1.42
1.26
2.21
1.43
2.20
6.98
3.26
2.17
6.31
2.27
1.81
1.63
1.20

94.55
95.31
92.56
94.43
94.70
95.22
92.67
93.99
95.06
94.66
94.55
94.42
93.39
95.46
93.82
95.23

10.12
3.52
19.65
12.84
13.45
6.10
13.44
14.80
7.62
16.24
18.02
5.64
11.39
7.40
13.30
14.49

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

50 of 60

School type and jurisdiction
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

141
148
158
134
166
173
202
172
148
117
131
206
181
118
133
118
155
158
173
272
205
140
149
167
125
114
209
119
306
130
226
115
141
152
190

100.00
99.18
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
99.17
100.00
100.00
100.00
99.97
100.00
100.00
99.08
100.00
100.00
100.00
100.00
100.00
99.08
100.00
100.00
100.00
100.00
100.00
100.00

100.00
99.30
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
98.83
100.00
100.00
100.00
99.61
100.00
100.00
99.36
100.00
100.00
100.00
100.00
100.00
99.23
100.00
100.00
100.00
100.00
100.00
100.00

3,800
3,600
5,700
4,000
3,700
5,400
5,800
4,900
4,300
3,500
4,100
3,600
3,600
4,500
3,700
3,800
4,900
5,400
6,100
3,600
5,000
3,600
4,300
5,300
3,600
3,900
3,700
4,000
11,300
4,700
3,100
4,200
4,500
3,500
5,100

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

1.01
2.21
8.73
1.33
1.56
10.33
5.69
3.52
1.56
1.06
1.65
4.25
4.33
1.12
2.78
9.09
5.71
2.56
2.21
6.49
5.77
4.97
2.62
2.91
2.06
2.74
3.18
7.05
9.93
4.14
2.38
2.79
2.84
1.70
1.86

95.68
95.16
94.41
93.88
93.86
94.44
94.48
94.40
94.46
93.78
94.56
93.94
95.31
95.59
93.93
94.76
93.43
93.75
93.81
96.01
94.22
95.14
94.58
94.28
95.01
94.30
95.71
94.71
94.83
94.15
93.54
94.73
95.44
95.16
94.53

14.99
11.63
3.78
17.26
14.74
6.72
12.31
7.10
10.47
5.60
9.29
5.13
10.97
17.76
13.85
8.85
10.06
19.39
12.12
6.34
9.43
9.30
12.70
11.17
12.39
7.26
7.61
6.82
3.14
7.66
13.64
9.77
11.67
7.94
15.95

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

51 of 60

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

Wyoming
BIE

202
135

100.00
83.26

100.00
84.85

DoDEA 2
Trial Urban (TUDA) Districts
Albuquerque
Atlanta
Austin
Baltimore City
Boston
Charlotte
Chicago
Cleveland
Dallas
Detroit
Fresno
Hillsborough
Houston
Jefferson County, KY
Los Angeles
Miami
Milwaukee
New York City
Philadelphia
San Diego
District of Columbia (TUDA)

120

98.91

52
66
55
70
86
57
95
86
55
58
53
56
86
56
76
87
68
82
57
54
106

National private
Catholic
Non-Catholic private
Lutheran
Conservative Christian
Other private

748
264
484
107
123
254

School type and jurisdiction

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

3,400
1,300

1.98
1.62

94.66
90.96

12.48
14.88

97.32

4,000

6.74

94.10

6.70

100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

2,100
2,200
2,100
1,700
2,000
2,000
2,700
1,600
2,000
1,500
2,200
1,900
3,200
2,200
2,700
3,000
1,600
2,800
1,800
2,000
1,700

5.11
3.81
16.49
16.89
8.06
1.63
2.10
5.41
18.48
7.01
2.30
2.55
14.45
9.61
1.89
3.84
2.67
2.49
3.41
3.61
3.93

92.87
96.23
94.26
92.70
94.48
94.57
95.27
93.03
95.49
88.99
93.71
94.61
95.27
94.80
95.11
95.68
94.75
93.01
94.40
95.08
94.99

12.48
5.57
4.50
3.44
14.36
10.24
19.13
20.79
2.80
5.39
6.01
24.39
3.57
5.11
8.93
21.80
28.77
26.24
16.23
7.29
18.04

73.51
96.27
55.34
94.87
73.13
42.23

68.47
95.93
56.27
92.38
70.86
44.57

6,500
3,500
3,000
753
950
1,300

0.51
0.45
0.56
1.17
0.00
0.74

94.91
95.49
94.41
96.23
95.40
93.71

4.23
3.52
4.84
2.69
1.96
6.48

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_gr4reading_2011.aspx

7/22/2016 2:46 PM

52 of 60

7/22/2016 2:46 PM

53 of 60

Participation, Exclusion, and Accommodation Rates for Grade 8 Reading for the
2011 Assessment
The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 reading
assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column
headings of the table.
The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The
rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is
represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school
population that is represented by the responding schools in the sample. These rates differ because schools differ in size.
Participation, exclusion, and accommodation rates, grade 8 reading assessment, by school type and jurisdiction: 2011

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

8,600

97.54

88.31

National
Northeast all
Midwest all
South all
West all

8,600

97.54

88.31

1,400
2,400
2,700
2,100

95.45
98.76
97.65
97.80

National public
Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
District of Columbia
Florida
Georgia
Hawaii
Idaho
Illinois
Indiana

7,500
125
167
136
126
257
130
119
68
102
233
132
81
113
223
113

99.79
100.00
99.90
99.02
100.00
100.00
99.87
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

School type and jurisdiction
All
all1

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

197,000

3.23

93.01

8.58

197,000

3.23

93.01

8.58

79.60
91.49
89.96
89.74

32,200
44,400
70,800
48,500

3.65
2.87
4.00
2.07

92.23
93.60
93.09
92.92

12.98
9.34
7.01
7.12

99.76
100.00
97.93
99.08
100.00
100.00
97.31
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

186,000
3,000
2,900
3,100
3,100
8,100
3,100
2,900
3,100
2,900
6,900
4,600
3,200
3,200
4,600
3,100

3.48
2.07
1.82
1.17
1.50
2.17
1.57
2.25
5.25
2.84
2.33
4.40
2.21
1.77
1.62
2.08

92.84
94.17
91.24
93.72
93.81
93.19
92.12
92.28
93.01
89.51
91.62
93.50
92.40
94.08
93.67
92.91

8.97
3.84
15.69
8.48
11.47
6.31
9.83
12.02
8.95
16.48
15.10
6.27
10.01
6.24
11.91
12.55

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department
of Defense Education Activity schools.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

54 of 60

School type and jurisdiction
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia
Wisconsin

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

138
148
156
163
143
176
154
180
168
121
136
200
169
100
96
116
136
174
159
209
194
149
144
168
61
116
261
123
236
125
124
108
140
117
179

100.00
100.00
100.00
100.00
100.00
99.05
99.46
100.00
100.00
100.00
100.00
99.86
100.00
99.70
100.00
100.00
99.09
99.08
100.00
99.99
100.00
100.00
99.10
100.00
100.00
100.00
100.00
100.00
99.09
100.00
100.00
100.00
100.00
100.00
100.00

100.00
100.00
100.00
100.00
100.00
98.82
98.47
100.00
100.00
100.00
100.00
98.41
100.00
97.35
100.00
100.00
99.40
99.67
100.00
99.47
100.00
100.00
99.26
100.00
100.00
100.00
100.00
100.00
99.63
100.00
100.00
100.00
100.00
100.00
100.00

2,900
3,100
4,400
3,000
3,000
4,200
4,400
4,700
3,400
3,000
2,800
2,900
2,900
3,200
3,000
2,900
3,900
4,600
4,900
2,600
4,200
2,900
3,300
4,400
3,000
3,100
3,300
3,200
8,800
3,400
2,300
3,100
3,600
3,100
4,000

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

0.75
1.90
7.22
1.00
1.73
8.43
6.32
4.83
2.84
0.96
1.39
4.03
4.73
1.93
4.15
7.08
5.74
3.10
2.06
7.90
5.75
4.34
2.13
3.10
1.16
5.30
3.22
6.31
6.04
3.70
2.76
3.62
2.11
1.45
2.20

92.53
93.46
94.27
92.69
92.31
91.82
92.18
93.15
92.58
92.33
94.09
92.04
93.78
92.86
92.22
92.32
91.29
91.33
92.09
93.50
93.25
92.52
92.34
91.91
92.66
93.72
94.70
91.99
93.74
92.06
93.05
93.66
92.07
92.44
93.81

13.68
8.49
4.21
13.36
14.05
4.03
12.42
6.58
7.43
5.63
10.63
6.47
7.42
8.87
11.57
10.71
6.30
16.61
11.66
5.62
8.72
9.37
8.93
12.73
13.70
4.14
5.66
4.63
2.91
6.05
12.94
7.73
9.42
7.19
13.64

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department
of Defense Education Activity schools.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

55 of 60

School type and jurisdiction
Wyoming
BIE

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted
by base
weight
only)

Number
of
students
sampled,
rounded

108
116

100.00
83.16

100.00
84.68

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

2,400
1,100

1.96
2.01

92.67
89.62

10.77
12.89

DoDEA 2
Trial Urban (TUDA) Districts
Albuquerque
Atlanta
Austin
Baltimore City
Boston
Charlotte
Chicago
Cleveland
Dallas
Detroit
Fresno
Hillsborough
Houston
Jefferson County, KY
Los Angeles
Miami
Milwaukee
New York City
Philadelphia
San Diego
District of Columbia (TUDA)

72

98.56

95.31

2,000

3.26

91.78

7.80

34
26
24
75
45
38
117
71
41
63
23
50
50
46
71
86
59
91
60
31
50

100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

1,400
1,600
1,700
1,300
1,400
1,600
2,100
1,200
1,600
1,900
1,500
1,500
2,500
1,600
2,300
2,800
1,400
2,500
1,500
1,300
1,600

7.29
3.55
9.01
16.95
9.66
2.04
2.28
5.21
5.95
7.96
1.90
1.77
6.40
6.83
2.04
3.77
3.33
2.63
4.69
1.49
3.86

88.93
92.38
93.15
88.94
89.97
92.97
94.92
91.23
92.60
85.41
92.21
94.45
94.12
91.58
91.67
92.91
90.89
91.54
91.11
95.63
87.69

9.06
6.29
4.96
3.31
14.19
9.66
15.81
24.77
4.86
7.95
6.37
20.97
3.87
6.06
8.84
15.63
27.81
22.49
19.54
9.03
19.31

National private
Catholic
Non-Catholic private
Lutheran
Conservative Christian
Other private

930
332
598
141
149
308

74.40
93.23
57.54
92.73
72.51
45.71

69.89
92.09
58.75
91.45
74.78
45.85

8,600
4,500
4,100
1,000
1,200
1,900

0.47
0.38
0.56
0.48
0.29
0.69

94.77
95.36
94.22
95.12
93.55
94.39

4.32
3.29
5.29
2.79
2.00
7.03

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all Department
of Defense Education Activity schools.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

56 of 60

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_gr8reading_2011.aspx

7/22/2016 2:46 PM

57 of 60

Participation, Exclusion, and Accommodation Rates for Grade 8 Science for the
2011 Assessment
The following table displays the school- and student-level response, exclusion, and accommodation rates for the grade 8 science
assessment by school type and jurisdiction. Various weights were used in the calculation of the rates, as indicated in the column
headings of the table.
The participation rates reflect the participation of the original sample schools only and do not reflect any effect of substitution. The
rates weighted by the base weight and enrollment show the approximate proportion of the student population in the jurisdiction that is
represented by the responding schools in the sample. The rates weighted by just the base weight show the proportion of the school
population that is represented by the responding schools in the sample. These rates differ because schools differ in size.

Participation, exclusion, and accommodation rates, grade 8 science assessment, by school type and jurisdiction: 2011

School type and
jurisdiction
All
all1

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
only)

Number
of
students
sampled,
rounded

8,600

97.32

88.19

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

141,000

1.57

92.87

10.58

National
Northeast all
Midwest all
South all
West all

8,600

97.32

88.19

141,000

1.57

92.87

10.58

1,400
2,400
2,700
2,100

95.45
98.76
97.65
96.86

79.60
91.49
89.96
89.18

24,000
32,400
47,700
35,100

1.36
1.60
1.70
1.49

92.02
93.40
93.10
92.68

15.71
10.41
9.71
8.38

National public
Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
District of Columbia
Florida
Georgia
Hawaii
Idaho
Illinois

7,500
125
167
136
126
257
130
119
68
102
233
132
81
113
223

99.54
100.00
99.90
99.02
100.00
100.00
84.41
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

99.57
100.00
97.93
99.08
100.00
100.00
87.04
100.00
100.00
100.00
100.00
100.00
100.00
100.00
100.00

138,000
2,600
2,400
2,700
2,700
2,800
2,200
2,500
2,600
3,000
2,700
2,800
2,700
2,800
3,900

1.69
1.08
1.08
0.88
0.95
1.76
0.90
1.32
1.69
1.49
1.24
1.55
1.99
1.46
1.14

92.81
93.22
89.90
93.19
94.08
92.96
92.56
91.41
92.10
87.56
93.10
92.81
92.59
93.06
94.03

11.10
4.13
16.37
9.33
11.63
7.76
10.32
12.55
12.20
17.52
16.28
8.43
10.80
6.66
12.40

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

58 of 60

School type and
jurisdiction

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
only)

Number
of
students
sampled,
rounded

Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Michigan
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Carolina
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
West Virginia

113
138
148
156
163
143
176
154
180
168
121
136
200
169
100
96
116
136
174
159
209
194
149
144
168
61
116
261
123
236
125
124
108
140
117

100.00
100.00
100.00
100.00
100.00
100.00
99.05
99.46
100.00
100.00
100.00
100.00
99.86
100.00
99.70
100.00
100.00
99.09
99.08
100.00
99.99
100.00
100.00
99.10
100.00
100.00
100.00
100.00
100.00
99.09
100.00
100.00
100.00
100.00
100.00

100.00
100.00
100.00
100.00
100.00
100.00
98.82
98.47
100.00
100.00
100.00
100.00
98.41
100.00
97.35
100.00
100.00
99.40
99.67
100.00
99.47
100.00
100.00
99.26
100.00
100.00
100.00
100.00
100.00
99.63
100.00
100.00
100.00
100.00
100.00

2,600
2,500
2,600
3,800
2,600
2,600
2,600
2,700
2,600
2,900
2,500
2,400
2,500
2,500
2,700
2,600
2,500
3,300
3,900
2,900
2,200
2,600
2,400
2,800
2,600
2,600
2,700
2,800
2,700
3,200
2,900
2,000
2,600
3,000
2,700

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

1.29
0.94
1.41
2.72
1.21
1.83
1.93
3.20
2.74
1.96
0.92
1.23
1.53
1.44
1.23
2.15
1.22
1.75
1.39
1.61
3.22
2.13
2.86
1.55
1.03
0.65
1.19
1.22
1.43
2.36
1.83
1.39
2.68
1.88
1.60

93.83
92.75
94.45
93.04
93.36
92.66
92.54
92.20
92.28
92.13
92.49
93.44
91.02
94.57
93.07
90.78
91.77
91.94
91.24
92.21
94.63
92.62
92.26
92.66
93.28
92.15
94.22
95.08
92.35
93.04
91.80
93.95
94.01
91.86
93.48

12.92
14.34
9.06
8.19
13.21
13.97
10.87
15.97
8.36
8.48
6.22
9.93
9.08
11.59
11.17
13.11
17.29
10.28
18.41
12.12
10.14
12.38
10.04
10.18
14.73
14.31
9.39
8.01
10.05
7.69
9.47
14.15
9.96
9.68
9.14

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

7/22/2016 2:46 PM

59 of 60

Number
of
schools
in
original
sample,
rounded

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
and
enrollment)

School
participation
rates
(percent)
before
substitution
(weighted by
base weight
only)

Number
of
students
sampled,
rounded

Wisconsin
Wyoming
BIE

179
108
116

100.00
100.00
83.16

100.00
100.00
84.68

DoDEA 2

72

98.56

930
332
598

74.40
93.23
57.54

School type and
jurisdiction

National private
Catholic
Non-Catholic private

Weighted
percent
of
students
excluded

Weighted
student
participation
rates
(percent)
after
makeups

Weighted
percent of
students
accommodated

2,400
2,000
112

1.91
1.31
0.00

93.21
92.27
88.14

13.62
11.27
6.07

95.31

1,700

1.33

94.33

9.55

69.89
92.09
58.75

903
476
427

0.18
0.35
0.00

93.65
94.27
93.03

4.74
4.65
4.84

1

Includes national public, national private, and Bureau of Indian Education schools located in the United States and all
Department of Defense Education Activity schools.
2 Department of Defense Education Activity.
NOTE: Detail may not sum to totals because of rounding.
SOURCE: U.S. Department of Education, Institute of Education Sciences, National Center for Education Statistics, National
Assessment of Educational Progress (NAEP), 2011 Assessment.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/resp_excl_accomm_rates_g8science_2011.aspx

7/22/2016 2:46 PM

60 of 60

Nonresponse Bias Analyses for the 2011 Assessment
NCES statistical standards call for a nonresponse bias analysis to be conducted for a sample with a response rate below 85 percent at
any stage of sampling. Weighted school response rates for the 2011 assessment indicated a need for school nonresponse bias analyses
for private school samples in grades 4 and 8 (operational subjects), for public school samples for the Bureau of Indian Education
(BIE) in grades 4 and 8, and for private school samples in grades 8 and 12 (computer-based writing). Student nonresponse bias
analyses were necessary for students in Detroit for grade 8 public schools (mathematics). Additionally, a student nonresponse bias
analysis was required to handle the special case of session nonresponse in the science sample in grade 8 Colorado public
schools. Thus, six separate school-level analyses and two separate student-level analyses were conducted.
The procedures and results from these analyses are summarized briefly below. The analyses conducted consider only certain
characteristics of schools and students. They do not directly consider the effects of the nonresponse on student achievement, the
primary focus of NAEP. Thus, these analyses cannot be conclusive of either the existence or absence of nonresponse bias for student
achievement. For more details, please see the NAEP 2011 NRBA report
(818KB).
Each school-level analysis was conducted in three parts. The first part of the analysis looked for potential nonresponse bias that was
introduced through school nonresponse. The second part of the analysis examined the remaining potential for nonresponse bias after
accounting for the mitigating effects of substitution. The third part of the analysis examined the remaining potential for nonresponse
bias after accounting for the mitigating effects of both school substitution and school-level nonresponse weight adjustments. The
characteristics examined were Census region, reporting subgroup (private school type), urban-centric locale, and size of school
(categorical).
Based on the school characteristics available, for the private school samples at grades 8 and 12, there does not appear to be evidence
of substantial potential bias resulting from school substitution or school nonresponse. However, the analyses suggest that a potential
for nonresponse bias remains for the grade 4 private school samples. This result is evidently related to the fact that, among
non-Catholic schools, larger schools were less likely to respond. Thus, when making adjustments to address the underrepresentation
of non-Catholic schools among the respondents, the result is to overrepresent smaller schools at the expense of larger ones. The
limited school sample sizes involved means that it is not possible to make adjustments that account fully for all school characteristics.
Please see the full report for more details.
Each student-level analysis was conducted in two parts. The first part of the analysis examined the potential for nonresponse bias that
was introduced through student nonresponse. The second part of the analysis examined the potential for bias after accounting for the
effects of nonresponse weight adjustments. The characteristics examined were gender, race/ethnicity, relative age, National School
Lunch Program eligibility, student disability (SD) status, and English language learner (ELL) status. For Colorado, additional school
characteristics were examined: Census region, urban-centric locale, size of school (categorical), and state-based achievement
(categorical).
Based on the student characteristics available, for the grade 8 Detroit student samples, there does not appear to be evidence of
substantial potential bias resulting from student nonresponse. The same result can be concluded for grade 8 Colorado student samples,
when considering student characteristics. However, analyses of the school characteristics suggest that a potential for nonresponse bias
remains. Please see the full report for more details.

http://nces.ed.gov/nationsreportcard/tdw/weighting/2011/weighting_2011_qa_nonresp_bias_analyses.aspx

7/22/2016 2:46 PM