Nonresponse Bias Analysis Overview

Nonresponse Bias Analysis

A comprehensive nonresponse bias analysis will be conducted for the 2015–16 NTPS. The analysis will evaluate the extent of potential bias introduced by nonresponse at both unit and item levels, and the extent to which noninterview weighting adjustments mitigated bias at the unit level.

Unit-level Nonresponse

Overview of Methodology

Because NCES Statistical Standard 4-4 requires analysis of unit nonresponse bias for any survey stage with a base-weighted response rate of less than 85 percent, it is anticipated that schools, principals, and teacher file will all need to be evaluated for potential bias. Comparisons between the frame and respondent populations will be made before and after the noninterview weighting adjustments are applied in order to evaluate the extent to which the adjustments reduced or eliminated nonresponse bias. The following section explains the methodology and describes how the conclusions will be summarized.

As outlined in appendix B of the NCES Statistical Standards (U.S. Department of Education 2003), the degree of nonresponse bias is a function of two factors: the nonresponse rate and how much the respondents and nonrespondents differ on survey variables of interest. The mathematical formulation to estimate bias for a sample mean of variable y is as follows:

where

= the estimated mean based on all eligible sample cases

= the estimated mean based only on respondent cases

= the estimated mean based only on nonrespondent cases

= the estimated number of cases (i.e., )

= the estimated number of nonrespondents

= the estimated number of respondents

A variable-free estimate of the bias, referred to as a relative bias, will be used to compare biases across all variables included in the analysis. The relative bias for an estimated mean using only the respondent data, , is calculated using the following formula:

Relative bias is estimated for variables known for respondents and nonrespondents. There are a number of variables available for each data file from the 2015–16 NTPS sampling frames. The variables to be used are presented in exhibit 1.

Exhibit 1. Variables to use in the NTPS unit nonresponse bias analysis: 2015–16

Regular public schools, principals, and teacher listing form Enrollment Percent of enrollment with race other than White Percent free or reduced price lunch eligible Locale Pupil-teacher ratio Grade level Region Number of teachers at the school Title 1 status State State by enrollment State by locale State by grade level Charter status School type Regular public school teachers Enrollment Percent of enrollment with race other than White Percent free or reduced price lunch eligible Locale Pupil-teacher ratio Grade level Region Number of teachers at the school Title 1 status Subject taught State State by subject State by locale State by grade level Charter status School type

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Teacher and Principal Survey (NTPS), 2015-16 – forthcoming survey documentation.

The following steps will be followed to compute the relative bias. First, the nonresponse bias will be estimated and tested to determine if the bias is significant at the 5 percent level. Second, noninterview adjustments are computed, and the variables listed above will be included in the nonresponse models. The noninterview adjustments, which are included in the weights, are designed to significantly reduce or eliminate unit nonresponse bias for variables included in the models. Third, after the weights are computed, any remaining bias is estimated for the variables listed above and statistical tests are performed to check for any remaining significant nonresponse bias. For this comparison, nonresponse bias is calculated as the difference between the base-weighted sample mean and the nonresponse-adjusted respondent mean, which evaluates the effectiveness of each noninterview adjustment in mitigating nonresponse bias. Sample units found to be ineligible for NTPS are excluded from the analysis.

The following tables provide an outline for how the data is to be summarized.

Table 1. Summary of NTPS component (school, principal, teacher listing form, teacher) unit nonresponse bias – 2015–16

Nonresponse bias statistics	Total
Before noninterview adjustment
Mean estimated percent relative bias (absolute value)	x
Median estimated percent relative bias (absolute value)	x
Percent of variable categories significantly biased	x
After noninterview adjustment
Mean estimated percent relative bias (absolute value)	x
Median estimated percent relative bias (absolute value)	x
Percent of variable categories significantly biased	x

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Teacher and Principal Survey (NTPS), “Public <component> Documentation Data File,” 2015–16.

Table 2. Effects of nonresponse adjustment on bias reduction – NTPS public <component> unit nonresponse bias: 2015–16

Before nonresponse adjustment	Change in absolute difference		After nonresponse adjustment	Number of categories
Not significant	-		Significant	X
Significant	>50% Reduction		Not significant	X
			Significant	X
	10%-50% Reduction		Not significant	X
			Significant	X
	<10% Reduction		Not significant	X
			Significant	X
	Increase in difference		Not significant	X
		Significant		X

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Teacher and Principal Survey (NTPS), “Public <component> Documentation Data File,” 2015–16.

Item-level Nonresponse

Overview of Methodology

The item bias analysis will examine the overall response rate for each item on all NTPS data files. The analysis will include examining the item response rates by the characteristics listed in exhibit 2 below, using the final weight for all in-scope sampled units. If the overall response rate for the item falls below 70 percent, the item will be footnoted in NCES publications with “Item response rate fell below 70 percent” as a method of cautioning the user that the low item response rate introduces some potential for bias in the imputation procedure. For any characteristic where the item response rate is less than 85 percent, a more detailed analysis will be done by the characteristics listed in exhibit 2. The results are highlighted if that particular cell had a significantly higher or lower response rate than the file as a whole and bolded if the difference is noteworthy. A noteworthy difference meets the following conditions:

• The difference relative to the overall response rate for the particular item was greater than 10 percent.

• The absolute difference was greater than one percentage point.

• The cell had at least 30 interviews.

Exhibit 2. Variables to be used in the NTPS item nonresponse bias analysis: 2015–16

Public schools and principals Enrollment Percent of enrollment with race other than White Percent free or reduced price lunch eligible Locale Pupil-teacher ratio Grade level Region Number of teachers Title 1 status State Public school teachers Enrollment Percent of enrollment with race other than White Percent free or reduced price lunch eligible Locale Pupil-teacher ratio Grade level Region Number of teachers Title 1 status Subject taught State

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Teacher and Principal Survey (NTPS), 2015-16 – forthcoming survey documentation.