Part B NTPS 2015-16 Full-Scale

The respondent universe for the 2015-16 full-scale data collection consists of 96,405 public schools in the 50 U.S. states and the District of Columbia (DC) that offer instruction in any of grades K-12. To be eligible for inclusion in the sample, schools must provide classroom instruction to students, have one or more teachers to provide instruction, serve students in at least one of grades 1-12 or the ungraded equivalent, must be located in one or more buildings, and must be located in the U.S. and not in the outlying areas or U.S. territories.

NCES’ 2013-14 Common Core of Data (CCD) will be used to construct the public school sampling frame. The respondent universe for charter schools will be identified as those public charter schools that meet the NTPS definition of an eligible school found on the CCD. Table 1.1 below presents the number of public schools on the 2012-13 CCD by urbanicity and school level. The universe has been adjusted to remove K-terminal schools, which are not eligible for NTPS.

Table 1.1. Respondent universe by urbanicity and school level for the proposed public school sample, based on the 2012-13 CCD

School level
Region	Primary	Middle	High	Combined	Total
City	15,233	3,584	5,124	2,182	26,123
Suburban	17,865	4,980	5,746	1,515	30,106
Town	6,223	2,381	3,430	924	12,958
Rural	13,608	3,989	7,266	2,355	27,218
Total	52,929	14,934	21,566	6,976	96,405

SOURCE: 2012-13 CCD.

Teachers

Teachers will be randomly sampled within the second design stage from roster information provided by each participating sampled school. Teachers within the sampled school are classified as ineligible for NTPS if they are a short-term substitute teacher, student teacher, a teacher’s aide, or do not teach any of grades K-12 or comparable ungraded levels. This information is obtained from the Teacher Questionnaire.

Procedures for Collecting Information
1. Sampling

The final 2015-16 NTPS samples will include no more than:

8,300 schools and school principals (7,000 traditional public and 1,300 charter schools);
40,000 teachers (34,300 traditional public, and 5,700 charter school teachers); plus
An additional 1,000 schools and principals will be selected for an internet response experiment following similar procedures as the main NTPS sample

Sampling – Public Schools

The level of precision achieved by the 2011-12 SASS was evaluated to inform the sample design decisions for the 2015-16 NTPS. The precision analysis was based on important analysis variables and on generic proportions to address other important SASS characteristics. The following variables and values were evaluated:

by state (public schools);
by school type (public charter, traditional public, overall public);
by urbanicity within school type;
by grade level (primary, middle, high, and combined);
by grade level and urbanicity within school type;
by poverty status (more than 75% of students eligible for free or reduced price lunch, 75% or less);
by poverty status, grade level, and urbanicity within school type; and
by teachers’ years of experience, subject, race/ethnicity.

The desired level of precision for NTPS estimates was defined in terms of a 95% confidence interval half-width (corresponding to 1.96 times the standard error). The desired goal was to achieve a 95% confidence interval half-width of 2.5% on a generic 20% characteristic.

Table 2.1.a presents a portion of the analysis for public schools by school type, grade level, urbanicity, and poverty status. Presented are the anticipated number of responding schools or principals for the NTPS design and the expected precision based on the analysis of SASS 11-12.

Table 2.1a. School-level precision analysis results for major characteristics of interest - NTPS 2015-16

Characteristic	Frame Schools	Expected Completed Interviews	Expected Standard Error	95% Confidence Interval Half-Width¹	Design Effect	Studies Needed to Achieve 2.5% Half-width Criterion
All	95,464	5,300²	0.68%	1.33%	1.51	1
Charter	6,254	707	1.75%	3.43%	1.35	2
Non-charter	89,210	4,593	0.72%	1.40%	1.46	1
Primary	52,868	2,418	0.95%	1.86%	1.35	1
Middle	14,912	940	1.52%	2.97%	1.35	2
High	21,199	1,340	1.47%	2.86%	1.78	2
Combined	6,485	602	2.15%	4.20%	1.73	3
City	25,818	1,599	1.24%	2.41%	1.51	1
Suburban	29,900	1,670	1.18%	2.30%	1.44	1
Town	12,785	771	1.72%	3.38%	1.44	2
Rural	26,961	1,260	1.40%	2.75%	1.55	2
High poverty	23,731	1,321	1.33%	2.59%	1.44	2
Low/medium poverty	71,733	3,979	0.79%	1.54%	1.54	1

¹ The bold values represent domains that do not meet the 2.5% criterion for the confidence interval half-width in one study.

² The small difference in this total (5,300) as compared to the estimated number of respondents in Part A’s burden table (5,312) is due to rounding.

Table 2.1.b provides the analogous precision analysis for public school teachers. The expected standard errors were calculated based on the SASS 11-12 and scaled for the expected NTPS 2015-16 number of respondents.

Table 2.1.b. Teacher-level precision analysis results for major characteristics of interest – NTPS 2015-16

Characteristic	Frame Full-Time Equivalent Teachers (In 1000s)	Expected Teacher Completed Interviews	Expected Standard Error	95% Confidence Interval Half-Width	Design Effect
All	3,088.3	26,500	0.41%	0.80%	2.82
Charter	132.7	3,022	1.18%	2.30%	2.60
Non-charter	2,955.5	23,478	0.43%	0.84%	2.68
Primary	1,490.6	10,951	0.64%	1.25%	2.75
Middle	543.2	4,869	0.94%	1.84%	2.68
High	908.4	7,924	0.74%	1.46%	2.70
Combined	146.0	2,755	1.24%	2.43%	2.63
City	904.8	8,163	0.76%	1.48%	2.94
Suburban	1,187.4	9,210	0.69%	1.36%	2.75
Town	364.2	3,626	1.09%	2.13%	2.66
Rural	631.8	5,501	0.90%	1.76%	2.78
High poverty	724.7	6,317	0.85%	1.68%	2.89
Low/medium poverty	3,088.3	20,183	0.47%	0.93%	2.79

Based on this analysis, the sampling frame will be partitioned into 50 sampling strata for public schools. Public schools are divided into two categories to create the sampling strata: charter schools (18 strata) and traditional public schools (32 strata).

Charter schools are identified from the CCD. The 18 charter school sampling strata are constructed by crossing the school grade level variable (primary, middle, high, and combined) with urbanicity (city, suburban, town, and rural) and poverty status (high, low/medium percent of students eligible for free or reduced price lunch).

The 32 sampling strata for the traditional public schools, excluding charters schools, are defined by crossing the four-category school grade level variable (primary, middle, high, and combined) with urbanicity (city, suburban, town, and rural) and poverty status (more than 75% of students eligible for free or reduced price lunch, 75% or less percent of students eligible for free or reduced price lunch).

Grade level (# Groups)

State

Grade level (# Groups)

State

Primary (6)

City high poverty

City low/medium poverty

Suburban high poverty

Suburban low/medium poverty

Town

Rural

High (4)

City high poverty

City low/medium poverty

Suburban

Town and rural

Middle (3)

City high poverty

City low/medium poverty

Suburban, town, and rural

Combined (5)

City high poverty

City low/medium poverty

Suburban

Town

Rural

Sampling – Teachers within All Schools

Teachers will be randomly sampled from roster information provided by each participating sampled school. No stratification will be used as the precision analysis indicated that no oversampling is needed. The average number of teachers sampled per school will be five for primary and combined schools, and 5.375 for middle and high schools. The larger cluster size for middle and high schools allows for greater precision of teachers by subject area. The maximum number of teachers per school will be set at 20 to avoid overburdening the schools.

Sampling – Principals within All Schools

For each sampled traditional public and public charter schools, the principal will be included in the survey as a result of the school being selected.

Survey Weights

Schools, principals, and teachers will be weighted by the inverse of the probability of selection. The final weight contains adjustments for nonresponse and any other sampling or field considerations that arise after the sample has been drawn.

Response Rates

We expect the response rates of the 2015-16 NTPS to approximate those of the 2011-12 SASS or to fall lower given the long-term trend in declining response rates for federal surveys. Table 2.3 provides the base-weighted response rates for the 2011-12 SASS.

Table 2.3. Base-weighted response rates for the 2011-12 SASS by respondent type and school type

School Type	Unit of Observation
School Type	Teacher	Principal	School
Traditional Public	77.92%	72.90%	72.68%
Charter	70.36%	69.67%	69.15%

Procedures for Collection of Information

The data collection methods for the 2015-16 NTPS will be based on those used in the 2014-15 NTPS Pilot Test and past cycles of SASS. The 2015-16 NTPS will seek school coordinators and will utilize clerical look-ups of school websites for teacher lists and email addresses along with commercial vendor sources. School coordinators were utilized in past SASS cycles but not in the 2014-15 NTPS Pilot Test.

Beginning in August 2015, all schools will receive an initial mailout package addressed to the principal at the school address. The package will contain a letter to the principal, a letter to the school coordinator (including instructions for completing online a brief screener interview using the NTPS Respondent Status Center), a Teacher Listing Form, a School Questionnaire, and a Principal Questionnaire. The completion of the questionnaires is to be facilitated by the school coordinator. The internet-based Respondent Status Center will allow schools to upload or manually enter the Teacher Listing Form, as well as check the status of all of the school’s questionnaires.

From September 2015 through October 2015, follow-up operations will be conducted for schools that have not returned the Teacher Listing Form. These operations will consist of field staff contacting the schools by telephone or personal visit. Schools will be assigned follow up priority at the time of sampling based on their likelihood to respond and impact on final estimates.

Additional reminder letters to the school and school coordinator will be sent in October and November 2015. Field staff will conduct telephone and personal visit follow-up of remaining nonresponding schools in December 2015 through May 2016.

3. Methods for Maximizing Response Rates

A variety of procedures will be employed to ensure high response rates at both the level of the responding unit (i.e., sample member) and at the level of the individual survey items in each survey questionnaire. The final NTPS design is based on results from the 2014-15 pilot test, which examined effectiveness of different contact strategies (OMB# 1850-0803 approved in July 2014).

The entire survey process, starting with securing research cooperation from key public school groups and individual sample members and continuing throughout the distribution and collection of individual questionnaires, is designed to increase survey response rates. In addition, we believe that the following elements of the data collection plan, in particular, will contribute to overall success of the survey and will enhance the survey response rates.

Visible support from top-level Federal, State, and local education officials. Without the support of high-level officials in the U.S. Department of Education, State Education Agencies, and the sampled local school districts, surveys of public school principals and teachers cannot be successfully implemented. Obtaining endorsements from these officials is a critical factor in the success of the data collection procedures. Top-level Education Department officials will need to fully support the data collection by endorsing the survey in writing and sending advance letters and notices to sampled districts' Superintendents, and individual survey participants (principals and teachers) to encourage participation.
Endorsements from key public school groups. The level of interest and cooperation demonstrated by key groups can often greatly influence the degree of participation of survey respondents. Endorsements are viewed as a critical factor in soliciting cooperation from state and local education officials. The NTPS is seeking endorsement by the following organizations or agencies:

American Association of School Administrators

Association of American Educators

American School Counselors Association

Association of Supervision and Curriculum Development

American Federation of Teachers

American Counseling Association

Association for Middle Level Education

Council of Chief State School Officers

Council of the Great City Schools

National Association of Elementary School Principals

National Association of Secondary School Principals

National Education Association

American Association of School Librarians

American Montessori Society

National Parent Teacher Association

As more endorsements are received, they will be added to questionnaires’ cover page (Attachment A).

Stressing the importance of the survey and the respondents' participation. Official letters will be used to motivate respondents to return surveys. For 2015 we plan to send an initial letter from the Director of the Census Bureau. Follow up letters will be sent from the Commissioner or Associate Commissioner of NCES. The additional personalization of survey materials (cover letters and survey packets with teachers' names) is also expected to have positive effects on the response rates.
Minimize the survey burden on school-level authorities. The procedures for the surveys are designed to minimize the survey burden on schools and sampled individuals (principals and teachers) and the survey instruments have been designed to be completed as quickly and easily as possible.

Good questionnaire design techniques have been employed to minimize item nonresponse. All completed questionnaires from the 2011-12 SASS have been carefully analyzed to determine which items had the highest levels of item nonresponse. This information guided NCES in reviewing the clarity of item wording, definitions, and instructions. Items that were not considered to be effective or useful were deleted so as to streamline the questionnaires and ease the response burden.

NTPS also plans to provide links to or incorporate data from other NCES collections such as EDfacts and the Civil Rights Data Collection (CRDC) into final datasets to allow researchers and policymakers to include additional data in their analyses. This will further reduce the need to collect data from schools that have already been collected at the state or district level.

Seeking the recruitment of a school coordinator. An important procedural measure for helping to maximize response rates is the plan to establish a school-based "survey coordinator" to serve as a primary point of contact for NTPS staff. The use of a school coordinator is expected to help keep response rates high, provide some minimal data quality checks, and simplify the follow-up process by having one point of contact.
Tailoring non response follow up strategies. After sample selection, cases will be assigned a “priority” flag based on the weighted response influence of the case. The weighted response influence takes into account both the response propensity and the base weight of a school to create a measure of a school’s potential effect on nonresponse weighting adjustments and final estimates. The weighted response influence can be calculated as:

where: is the final weighted response influence for a school,

is the baseweight for a school, and

is the estimated response propensity for a school

As the formula shows, a case with either an extremely high weight or an extremely low response propensity will have a large response influence, reflecting the fact that if they are a nonrespondent, they will disproportionately affect the nonresponse adjustment cell in which they are located. Missing that particular school’s information may result in biased estimates (if variables in the propensity model are related to outcomes of interest), and will certainly result in increased variance in the estimates (due to more variable final weights). In order to avoid having extreme weights drive the value of weighted response influence, we take the natural log of the base weight in the formula.

Using data from the 2011-2012 SASS and the 2014 NTPS Pilot Test, including frame variables and response statuses, we calculated a weighted response propensity model, keeping in mind domains of interest and estimates of interest for the 2015 NTPS. Specific categories of variables available for evaluation include geography, urbanicity, racial/ethnic makeup, enrollment, grades levels, free lunch recipiency, and type of school. These variables are available in the 2011-2012 SAS sample files, the NTPS Pilot Test sample files, and the 2015 NTPS sample file, allowing us to leverage past experience in creating the response propensity models.

The priority flag will be used throughout data collection to make non response follow up decisions. For example, cases with the highest weighted response influence will receive early follow up by field interviewers rather than progressing through multiple data collection operations that are likely to be unproductive.

Weighted response influence was also used, with great success, by the National Survey of College Graduates (NSCG) for identifying high influence sample persons. These high influence cases were targeted during data collection with an experimental incentive to increase sample balance and overall response rate. Response influence has been implemented by the NSCG in production as a way of identifying high influence sample persons for data collection interventions. More information about response influence can be found in: Särndal, C., Lundström, S. (2008). Assessing auxiliary vectors for control of nonresponse bias in the calibration estimator. Journal of Official Statistics 24, 167-191.

The 2015 NTPS adaptive design approach will not target principals, as individuals, or teachers. A priority flag, based on the response propensity scores developed by the Census Center for Adaptive Design (CAD), is assigned at the school level. During data collection, the priority flag will be used to move high priority schools to field operations earlier in the process along with schools that have not been assigned a coordinator. Schools in the high priority group generally do not respond until later in the data collection process and ultimately require field intervention. By moving those schools to field contact after the first two mailouts, we will reduce costs by eliminating the third and fourth mailings, which have low probabilities of impacting response, and raise the probability of response by providing the field staff more time to secure the completed questionnaires. Since the principal questionnaires are pursued at the school level and not at the individual level, intervening with the principals falls into this operation. Throughout data collection the cases assigned to field will be reviewed by NTPS staff on a daily basis. The priority flags will be used to direct the Field Representatives on the order to pursue cases. Regarding the references to NSCG -- The response propensity model developed for NTPS was based on the response propensity model developed for NSCG.

Efforts will be focused on obtaining cooperation and improving response rates at the school level for a number of reasons. Past administrations of the Schools and Staffing Survey (SASS) have shown that if cooperation is obtained at the school level, teachers and principals are highly likely to respond. Additionally, evaluation of the schools’ response propensities using the model developed by CAD showed that the nonresponse in past administrations of SASS was driven primarily at the school level. Results showed that schools in special districts are the primary driving force behind low response propensity. Special districts are those that require additional applications or documentation to collect data in their schools. Nearly 80% of the schools with high propensity for non-response reside in these special districts. For this reason, resources will be reallocated to focus heavily on obtaining approvals from these special district schools in order to boost response rates for this group.

Additionally, past administrations of SASS demonstrated that response rates tend to be very low for urban schools. The 2015 NTPS includes a flier that will be sent specifically to urban schools (as determined from frame data on the CCD). This flier will be roughly the size of one half of a standard sheet of paper and is customized to include interesting statistics about city schools and why their participation is important (see Attachment B).

Finally, the NTPS teacher-level response rates are calculated by multiplying response at the school level to the Teacher Listing Form (TLF) by response at the teacher level. In the past this has meant that if the school did not cooperate by not completing the TLF, teachers from that school could not be sampled, which ultimately lowered the teacher response rate. As detailed in this submission, in the 2015 NTPS, TLFs received from sample schools will be supplemented with vendor-purchased teacher lists and a clerical look-up operation utilizing school websites. These methods were tested in the Pilot and showed high levels of comparability to lists obtained directly from schools. This operation will help to improve the overall teacher response rate by allowing sampling teachers from schools that have not submitted a TLF.

Increased use of email and internet for collection of teacher lists and survey reminders. The 2014 NTPS pilot test demonstrated that email was an effective tool to drive participation in the NTPS teacher survey. The pilot test also showed that email addresses and teacher lists could be collected from school websites. The 2015-16 NTPS includes clerical operations to look up principal email addresses and lists of teachers as well as their email addresses during data collection. The teacher lists will be collected for schools that do not return a completed Teacher Listing Form or do not include teacher email addresses on the returned form. Depending on results of the experiment conducted during the NTPS pilot test, commercial vendor lists may also be used to identify and sample teachers at schools that did not complete a TLF. Emails will be sent to sampled teachers with links to the NTPS teacher survey. Additionally, principals of schools in the internet experiment will receive email invitations and reminders to complete the survey.
Improving response from urban schools with NTPS Flier. Past administrations of SASS demonstrated that response rates tend to be very low for urban schools. The 2015-16 NTPS includes a flier that will be sent specifically to urban schools (as determined from frame data on the CCD). This flier will be roughly the size of one half of a standard sheet of paper and is customized to include interesting statistics about city schools and why their participation is important (the flier is included in Attachment B).

4. Tests of Procedures and Methods

The 2017-18 NTPS will be built on the experience of previous rounds of SASS and NTPS. Results from the 2015-16 NTPS Schools and Principals Internet Test will impact the final 2017-18 NTPS design. These results and the final design will be described in the 2017-18 NTPS data collection clearance package expected to be submitted in early 2017.

In addition, the 2014-15 NTPS Pilot Study showed that response rates for the Internet treatment groups (for schools and principals) were lower than for the paper treatment group. Anecdotal information from the Pilot test suggested that internet response rates could potentially be increased by adjusting elements of our instruments and contact materials. The login procedure for the Internet instrument has been changed and the contact materials have been revised from those used in the Pilot. A small-scale experiment of 1,000 schools/principals will be conducted in in the 2015-16 NTPS. The sample of schools/principals will be asked to complete the instrument online rather than on paper. Additionally, because this will be a full-scale collection, these 1,000 schools will have the option to designate a school coordinator. We hypothesize that the changes to the login procedure as well as the designation of a school coordinator will increase the response rates for the Internet treatment groups to be comparable with the paper respondents. The sample size of 1,000 was developed to detect differences of 5% with a 95% alpha. The sample of 1,000 schools will not be included in the released data file and will instead be used for analysis to inform design decisions for the 2017-18 NTPS.

5. Reviewing Statisticians

The following statisticians have been contributing to the NTPS sample design:

Randall Parmer, U.S. Census Bureau, Suitland, MD

David Marker, Westat, Rockville, MD

Lou Rizzo, Westat, Bethesda, MD

Sharon Lohr, Westat, Bethesda, MD

Edward Mulrow, NORC, Bethesda, MD

Kirk Wolter, NORC, Chicago, IL

Andrew Zukerberg, NCES, Washington, DC

Marilyn Seastrom, NCES, Washington, DC

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-24