Part B NTPS 2017-18

This request is to conduct NTPS 2017-18, including all of its recruitment and data collection activities. Because of the overlap in time, this request also carries over the burden and materials for the approved preliminary activities. Section B.1 of this document describes the universe, sample design, and estimation details for NTPS 2017-18. Section B.2 describes the data collection procedures for NTPS 2017-18, including the preliminary field activities approved in an earlier submission. Section B.3 discusses methods to secure cooperation and mitigate nonresponse. In particular, it describes methods used to improve response rates in NTPS 2015-16 and how those methods will be used in NTPS 2017-18. Section B.4 describes recent developments in a long history of tests of methods and procedures to improve data quality. It also includes a description of a test to include private schools in NTPS. Section B.5 lists the names and phone numbers of those involved in the design of the study and the development of these materials.

B.1 Universe, Sample Design, and Estimation

Section B.1.1 includes information on the study universe of interest and sample design planned for NTPS 2017-18. Section B.1.2 describes the precision requirements and target sample sizes set out for the study.

B.1.1 Universe and Sample Design: Respondent Universe

B.1.1.1 Schools

The respondent universe for NTPS 2017-18 data collection consists of approximately 94,000 public schools and 25,000 private schools in the 50 U.S. states and the District of Columbia (DC) that offer instruction in any of grades 1-12 or the ungraded equivalent. To be eligible for inclusion in the sample, schools must: provide classroom instruction to students; have one or more teachers who provide instruction; serve students in at least one of grades 1-12 or the ungraded equivalent; be located in one or more buildings, and be located in the continental United States.

NCES’ 2015-16 Common Core of Data (CCD) will be used to construct the public school frame. The respondent universe for charter schools will be identified as those public charter schools that meet the NTPS definition of an eligible school found on the CCD. The universe has been adjusted to remove kindergarten-terminal schools, which are not eligible for NTPS. Table 1 presents the number of public schools on the 2012-13 CCD by urbanicity and school level. The CCD for 2015-16 is not yet available at the time of submitting this package.

Table 1. Respondent universe by school level and urbanicity for the proposed public school sample, based on the 2012-13 CCD

School level
Region	Primary	Middle	High	Combined	Total
Central City	15,308	3,699	5,407	1,727	26,141
Suburban	17,933	5,136	5,901	1,220	30,190
Town	6,138	2,340	3,481	876	12,835
Rural	12,221	3,189	6,014	3,538	24,962
Total	51,600	14,364	20,803	7,361	94,128

SOURCE: 2012-13 CCD.

The private school test frame is drawn from the 2015-16 Private School Survey (PSS) frame. Preschools and schools with kindergarten as the highest grade are excluded. Table 2 presents the number of private schools on the 2015-16 PSS by urbanicity and school level.

Details of the first-stage sample design of schools are provided in section 2.

Table 2. Respondent universe by school level and urbanicity for the proposed private school test sample, based on the 2015-16 PSS

School level
Region	Elementary	Secondary	Combined	Total
Central City	4,975	1,121	2,702	8,798
Suburban	5,005	871	2,938	8,814
Town	1,315	145	766	2,226
Rural	2,735	472	1,939	5,146
Total	14,030	2,609	8,345	24,984

SOURCE: 2015-16 PSS

B.1.1.2 Teachers

Teachers will be randomly sampled within the second design stage from roster information provided by each participating sampled school. Teachers within the sampled school are classified as ineligible for NTPS if they are a short-term substitute teacher, student teacher, or a teacher’s aide; or if they do not teach any of grades K-12 or comparable ungraded levels. The information that classifies teachers as ineligible is obtained from the Teacher Questionnaire. Details of the second-stage sample design of teachers are provided in section 2.

B.1.2 Precision Requirements and Sample Sizes

This section details the school sample sizes and precision requirements for the NTPS 2017-18 public and private school samples.

The final NTPS 2017-18 public sample will include approximately:

10,600 schools and school principals (9,100 traditional public and 1,500 public charter), with the goal of at least 6,800 interviews for each; and
47,000 teachers (42,100 traditional public and 4,900 public charter), with the goal of at least 35,000 interviews.

The final NTPS 2017-18 private school test sample will include approximately:

4,000 schools and school principals, with the goal of at least 2,300 interviews for each; and
9,000 teachers, with the goal of at least 6,000 interviews.

Sampling – Public Schools

The level of precision achieved by NTPS 2015-16 was evaluated to inform the sample design decisions for NTPS 2017-18. In particular, publishability and bias indicators (described in Section B.3.2) were reviewed in order to improve the school sample design for the 2017-18 NTPS. A key change in NTPS 2017-18 from NTPS 2015-16 is the inclusion of additional sample to enable the publication of state estimates. The 2017-18 NTPS oversampling stratification will be based preliminarily on the following domains:

Charter/Non-charter;
School Level (primary, middle, high, combined);
Urbanicity (city, suburb, town, rural);
School enrollment (four levels: schools with enrollment less than 100; schools with enrollment between 100 and 199; schools with enrollment 200 to 499; schools with enrollment 500 or more);
State Tier, state.

The NCES standards for publishability indicate that the coefficient of variation (CV) must be no larger than 50%, and if the CV is between 30% and 50%, the estimates are published with a caveat. For a population proportion of 20%, a CV of 30% corresponds to a standard error of 6%. In order to make sure that we don’t fall below the CV 30% minimum with the uncertainties about response and about exact values of design effects, we set as a target a CV of 25% as a lower bound. This corresponds to an expected standard error of 5%. This considerably reduces the chance of falling below the 30% boundary (if we set 30% itself as the target, we would be below it one-half of the time). Our target goal then for each state is to make sure that the expected standard error is no larger than 5% for a population proportion of 20% (a CV of 25%), at both the school and teacher level.

Table 3 presents a portion of the analysis for public schools by school type, grade level, urbanicity, and poverty status. Presented are the anticipated number of responding schools or principals for the NTPS design and the expected precision based on analyses using the NTPS 2015-16 final response rates and CV of 25%.

Table 3. NTPS 2017-18 school-domain expected interviews, standard errors, and design effects with state oversampling to achieve 25% CV or less

Domain	Frame Schools	Expected Sample Size (completed interviews)	Expected Standard Error	Design Effect
All	94,128	6,700	0.63%	1.680
Charter	6,530	774	1.69%	1.375
Non-charter	87,598	5,926	0.67%	1.658
Primary	51,600	3,028	0.89%	1.489
Middle	14,364	1,122	1.43%	1.431
High	20,803	1,739	1.40%	2.125
Combined	7,361	810	1.89%	1.814
City	26,141	1,941	1.17%	1.673
Suburban	30,190	1,972	1.13%	1.581
Town	12,835	1,047	1.61%	1.696
Rural	24,962	1,740	1.28%	1.775
Enrollment < 100	8,208	332	3.44%	2.464
100 <= Enrollment < 300	7,618	490	2.30%	1.621
300 <= Enrollment < 500	36,116	2,376	1.00%	1.489
500 <= Enrollment < 750	23,552	1,653	1.15%	1.377
750 <= Enrollment < 1,000	9,395	789	1.65%	1.343
1,000 <= Enrollment	9,239	1,060	1.38%	1.255
Percent FRPL < 35%	26,066	1,928	1.27%	1.947
35% <= Percent FRPL < 50%	15,561	1,194	1.46%	1.590
50% <= Percent FRPL < 75%	26,182	1,828	1.17%	1.574
75% <= Percent FRPL	24,417	1,601	1.23%	1.507
Not Participating FRPL	1,902	148	5.30%	2.603

Table 4 provides the analogous precision analysis for public school teachers. The expected standard errors were calculated based on analyses using the NTPS 2015-16 final response rates and CV of 25%.

Table 4. NTPS 2017-18 major domain expected teacher interviews, standard errors, and design effects with state oversampling to achieve 25% CV or less

Domain	Frame Full-Time Equivalent Teachers (in 1000s)	Expected Teacher Completed Interviews	Expected Standard Error	Design Effect
All	3,127.9	34,722	0.44%	4.25
Charter	144.9	3,394	1.24%	3.25
Non-charter	2,983.0	31,329	0.46%	4.14
Primary	1,473.6	13,507	0.67%	3.80
Middle	552.6	6,368	1.01%	4.09
High	924.3	11,154	0.82%	4.72
Combined	177.4	3,694	1.23%	3.47
City	920.6	10,328	0.82%	4.36
Suburban	1,202.1	11,377	0.77%	4.19
Town	368.0	5,126	1.10%	3.85
Rural	637.2	7,891	0.90%	3.99
Enrollment < 100	40.9	712	2.49%	2.75
100 <= Enrollment < 300	94.3	1,519	1.85%	3.26
300 <= Enrollment < 500	862.9	9,999	0.77%	3.73
500 <= Enrollment < 750	865.9	9,544	0.84%	4.22
750 <= Enrollment < 1,000	474.1	4,909	1.20%	4.38
1,000 <= Enrollment	789.8	8,039	0.98%	4.81
Percent FRPL < 35%	943.9	10,524	0.82%	4.46
35% <= Percent FRPL < 50%	530.3	6,253	1.05%	4.34
50% <= Percent FRPL < 75%	839.9	9,287	0.84%	4.11
75% <= Percent FRPL	755.2	7,829	0.91%	4.06
Not Participating FRPL	58.6	831	3.00%	4.68

Sampling – Private Schools

To inform the sample design for the NTPS 2017-18 Private School Pilot Study, NCES evaluated the level of precision achieved by 2011-12 SASS. The precision analysis was based on analysis variables and on proportions to address important characteristics. The following variables were evaluated:

School type (Religious – Catholic, Religious – Other, Non-Religious);
Grade Level (Elementary, Secondary, Combined); and
Region (Northeast, Midwest, South, West).

The sample design for private schools is broadly consistent with the private school design for SASS, except that the sample size is increased by roughly one-third to achieve the precision goals for the experimental study (see Section B.4). In addition, sampling rates for nonsectarian and secondary schools are set to better equalize the precisions of these domains with the other school type and grade level domains. NCES has also evaluated the sampling plan for the ability to achieve a minimum level of CV. The desired goal was to achieve a CV of less than 30 percent for a population proportion of 20% in order to meet NCES standards for reporting.

In order to better equalize precision across major schools domains for private schools from those achieved in SASS 2011-12, NCES plans to oversample as follows:

Secondary schools will be sampled at a rate proportional to 3.33 times the measure of size (as determined by number of FTE teachers);
Non-Religious schools will be sampled at a rate proportional to 1.43 times the measure of size (except for secondary non-religious schools, which are sampled at the 3.33 rate); and
Other strata will be sampled at a rate proportional to 1.0 times the measure of size.

For teachers, the expected number of completed interviews is estimated to be proportional to the product of the final school sampling factor and the number of full time equivalent (FTE) teachers over schools in the domain. The overall target number of completed interviews is 6,000. Assuming the attrition rate for the NTPS 2017-18 will be similar to the rate for the 2011-12 SASS, the sample size needs to be 9,000 in order to yield the expected number of completed teacher interviews. The teacher sample size for a sampled school should be proportional to the product of the final teacher multiplier (based on the expected attrition adjustment factors), final school oversampling factor, and measure of size for the school.

Tables 5 and 6 show expected sample sizes, standard errors, and CVs for population percentages of 20% by key domains of school type, grade level, and region.

Table 5 presents a portion of the analysis for private schools by affiliation, grade level, and region.

Table 5. School-domain expected interviews, standard errors, and design effects for the NTPS 2017-18 private school pilot study

Domain	Frame Schools	Expected Sample Size (completed interviews)	Expected Standard Error	Design Effect
All	24,861	2,266	1.08%	1.65
Catholic	6,407	742	1.83%	1.55
Other religious	11,600	774	1.80%	1.57
Nonsectarian	6,854	750	1.77%	1.46
Elementary	13,216	826	1.61%	1.34
Secondary	2,426	654	1.69%	1.17
Combined	9,219	786	1.72%	1.45
Northeast	5,787	602	2.26%	1.92
Midwest	6,105	512	2.24%	1.61
South	8,025	706	1.86%	1.53
West	4,944	446	2.38%	1.58

Table 6 provides the analogous precision analysis for private school teachers.

Table 6. Major domain expected teacher interviews, standard errors, and design for the NTPS 2017-18 private school pilot study

Domain	Frame Full-Time Equivalent Teachers	Expected Teacher Completed Interviews	Expected Standard Error	Design Effect
All	431,588	5,827	0.99%	3.58
Catholic	135,265	2,078	1.75%	3.98
Other religious	164,122	1,756	1.72%	3.24
Nonsectarian	132,201	1,993	1.65%	3.40
Elementary	163,523	1,644	1.65%	2.81
Secondary	62,614	1,933	1.66%	3.32
Combined	205,451	2,250	1.53%	3.31
Northeast	112,558	1,661	1.92%	3.84
Midwest	91,178	1,233	2.13%	3.50
South	149,772	1,848	1.74%	3.48
West	78,081	1,084	2.26%	3.47

The 2017-18 NTPS will have an implicit stratification based on the proposed systematic sampling sort order, which uses a hierarchy of the following domains:

Three-level affiliation (Catholic, non-Catholic religious, nonreligious);
Three-level school span (elementary, secondary, combined);
Four-level Census region (Northeast, South, Central, Midwest);
Four-level urbanicity (city, suburb, town, rural);
Eleven-level affiliation;
Five-level school size (enrollment <100, 100-199, 200-499, 500-749, 750+);
State;
Highest grade;
Twelve-level urbanicity (large city, medium-sized city, small city, etc.);
Zip code;
School enrollment;
PIN number.

Teachers will be sampled from roster information provided by each participating sampled school. The target teacher completed interview sample sizes are designed to be proportional to the square root of the number of full-time teachers for each school and assume an attrition rate due to nonresponse.

Sampling – Principals within All Schools

For each sampled traditional public, public charter, and private schools, the principal will be included in the survey as a result of the school being selected.

Survey Weights

Schools, principals, and teachers will be weighted by the inverse of the probability of selection. The final weight will contain adjustments for nonresponse and any other sampling or field considerations that arise after the sample has been drawn.

Response Rates

We expect the NTPS 2017-18 response rates to approximate those of NTPS 2015-16 (for public schools) and SASS 2011-12 (for private schools) or to fall lower given the long-term trend in declining response rates for federal surveys. Table 7 provides the base-weighted response rates for SASS 2011-12, as the final base-weighted response rates are not yet calculated for NTPS 2015-16.

Table 7. Base-weighted response rates for SASS 2011-12 by respondent and school type

School Type	Unit of Observation
School Type	Teacher	Principal	School
Traditional Public	77.92%	72.90%	72.68%
Charter	70.36%	69.67%	69.15%
Private	69.9%	64.7%	65.7%

B.2 Procedures for the Collection of Information

Section B.2.1 describes the operations for the already approved preliminary field activities for NTPS 2017-18, with Section B.2.1.1 describing special districts operation and Section B.2.1.2 the school pre-contact letter operation (OMB# 1850-0598 v.17). Section B.2.2 describes school-level data collection procedures for the school-level questionnaires (i.e., Teacher Listing Form, School Questionnaire, and Principal Questionnaire), with Section B.2.2.1 describing the procedures to be used with priority schools Section B.2.2.2 with non-priority schools. Section B.2.3 describes data collection procedures for the Teacher Questionnaire.

B.2.1 Preliminary Field Activities

B.2.1.1 Special Contact District Operation

Special contact districts require that a research application be submitted to and reviewed by the district before they will allow schools under their jurisdiction to participate in a study. Districts are identified as “special contact districts” prior to data collection because they were flagged as such during previous cycles of SASS, NTPS, or SSOCS, or by other NCES studies. Special contact districts are also identified during data collection when districts indicate that they will not complete the survey until a research application is submitted, reviewed, and approved.

Once a district is identified as a special contact district, basic information about the district is obtained from the NCES Common Core of Data (CCD). The basic information includes the NCES LEA ID number, district name, city, and state. The next step is to search the district’s website for a point of contact and any information available about the district’s requirements for conducting external research. Some districts identified as being a special contact district from the previous cycle may be incorrect and staff will verify whether a given district has requirements for conducting external research before proceeding.

The following are examples of the type of information that will be gathered from each district’s website in order to prepare a research application for submission to this district:

Name and contact information for the district office or department that reviews applications to conduct external research, and the name and contact information of the person in charge of that office.
Information about review schedules and submission deadlines.
Whether application fees are required, and if so, how much.
Whether a district sponsor is required.
Whether an online application is required, and if so, the link to the application if possible.
Information about research topics and/or agenda on which the district is focusing.
The web link to the main research department or office website.
Research guidelines, instructions, application forms, District Action Plans, Strategic Plan or Goals, if any.

Recruitment staff will contact districts by phone and email to obtain key information not listed on the district’s website, (e.g., requirements for the research application, research application submission deadlines, etc.).

SSOCS/NTPS staff developed a generic research application that covers the information typically requested in district research applications. Staff will customize the generic research application to each district’s specific requirements that need to be addressed or included in the research application (e.g., how the study addresses key district goals, or inclusion of a district study sponsor), or submit the generic application with minimal changes to districts that do not have specific application requirements.

Using the information obtained from the district website or phone or email exchanges, a district research request packet will be prepared. Each research application will include the following documents, where applicable:

District research application cover letter;
Research application (district-specific or generic, as required by the district);
Study summary;
FAQ document;
Special contact district approval form;
Participant informed consent form (if required by the district);
SSOCS/NTPS Project Director’s resume;
Copy of questionnaires; and
Application fee (if required by the district).

Where applicable, applications will include the draft 2017-18 NTPS questionnaires provided in Appendix B of this submission, and the 2015-16 NTPS questionnaires will be provided to districts that request them. Other information about the study may be required by the district and will be included with the application or provided upon request.

Approximately one week after the application is submitted to the district (either electronically or in hard copy, as required by the district), SSOCS/NTPS district recruitment staff will contact the district’s research office to confirm receipt of the package and to ask when the district expects to review the research application and when a decision will be made. If additional information is requested by the district (e.g., the list of sampled schools), recruitment staff will follow up on such requests and will be available to answer any questions the district may have throughout the data collection period.

To reduce burden for the special contact districts and improve operational efficiency, NCES is planning to seek research approval simultaneously for NTPS 2017-18 and SSOCS 2018. Although NCES plans to minimize overlap in the schools sampled for NTPS and SSOCS, most of the largest districts will have schools selected for both surveys. All special contact districts with schools in both surveys will receive both research applications concurrently and will be given the option to participate in NTPS only, SSOCS only, or both NTPS and SSOCS. The research request packets for the districts in both studies will contain an additional letter introducing the studies and emphasizing that SSOCS and NTPS are working together to minimize the number of schools asked to participate in both studies.

Some districts charge a fee (~$50-200) to process research application requests, which will be paid as necessary.

B.2.1.2 School Pre-Contact Letters

The school pre-contact letter is to verify school mailing addresses and to inform schools about the upcoming data collection. A letter is sent to each sampled school informing them of their selection for the study. About 4% of all school addresses get corrected by the U.S. Post Office in response to the pre-contact letter, saving time and effort during the actual data collection period.

B.2.2 School-level Data Collection Procedures

School-level data collection procedures for NTPS 2017-18 are summarized in Exhibit 1.

In July 2017, all schools will receive an advance letter addressed to the principal at the school address. The letter includes instructions for completing a brief screener interview online using the NTPS Respondent Portal. The purpose of the screener interview is to determine the school’s eligibility for the NTPS and establish a survey coordinator. The survey coordinator will be asked to facilitate the completion of NTPS questionnaires within their school, and materials will be mailed to him or her throughout data collection. Principals who do not self-screen will be contacted by telephone. A reminder email will be sent to non-responding school principals in August 2017.

After the advance letter and screener interview, schools will enter one of two data collection paths. The data collection methodology employed will be dependent on whether the school has been identified as a “priority school.” The propensity model is based on a model developed to identify priority schools for the 2015-16 NTPS data collection; the same model with updated information will be used for the 2017-18 NTPS data collection. Prior to the start of NTPS 2017-18 data collection, a propensity model will be run to identify “priority” schools. These “priority” schools have characteristics of schools from which it has been historically difficult to collect data and which have a potentially high impact on weighting. The priority flag takes into account both the response propensity and the base weight of a school to create a measure of a school’s potential effect on nonresponse weighting adjustments and final estimates. Schools with either an extremely high weight or an extremely low response propensity have a large response influence, meaning their nonresponse will disproportionately affect the nonresponse adjustment cell in which they are located. Thus, efforts are made to prioritize field operations in these school early during data collection.

Between late February and early March 2018, an additional reminder email will be sent to Principals and/or Survey Coordinators of nonresponding schools, and between late February and early June 2018, an email reminder will be sent to teachers who are not eligible for the contingency plan incentive experiment. Each of these emails will include a link to an informational NTPS video and to the relevant survey, the respondent’s User ID, and selected findings from the 2015-16 NTPS.

Exhibit 1: 2017-18 National Teacher and Principal Survey – School-Level Data Collection Operations

Shape1 Shape2

Reminder email to Principals/Schools

Late February-Early March

Reminder email to Teachers

Late February- Early June

B.2.2.1 Priority Schools

In early September 2017, principals or survey coordinators at priority schools will be mailed a letter, at the school address, informing them that their school may receive a personal visit from Census Bureau staff in the coming weeks. About ten days later, data collection will begin with a personal visit from a Census Bureau Field Representative. The expectation for the personal visit is that the Census Bureau Field Representative will complete the school’s Teacher Listing Form (TLF). In most cases, the TLF will be pre-populated with vendor or clerically-researched data and the Field Representative will only need to verify that the teacher information is complete and accurate. The Field Representative will also distribute sealed letters containing login information for the school and principal questionnaires. If the Field Representative notes that the school has shown reluctance or initially refused to participate in the study, the Regional Office of the Field Representative will send out a “letter of better understanding” to help encourage participation.

Schools for which the personal visit is unsuccessful will receive an initial package in late October 2017 addressed to the survey coordinator at the school address. If a survey coordinator was not established during the screener interview, the package will be addressed to the principal at the school address. The mailed package will contain a letter to the survey coordinator or principal and three individual sealed envelopes that contain login information for completing the TLF, Principal Questionnaire, and School Questionnaire. A few days after the initial package mailing, email will also be used to contact the survey coordinator and principal. Additionally, principals and survey coordinators will be contacted by email around the same time the initial packages are mailed to the sampled schools. The emails will include the appropriate hyperlinks and User IDs to complete the NTPS questionnaires online. A reminder email will be sent in mid-November to principals and survey coordinators.

In late-November 2017, a second package will be mailed to the survey coordinator or principal, at the school address, of nonresponding priority schools. The package will include a reminder letter, a pre-populated paper TLF and a return envelope (if applicable), and/or replacement materials for completing the principal and/or school questionnaires online. Principal and survey coordinator email addresses will be used as means of reminding nonresponding school staff to complete their questionnaires.

In early January 2018, priority schools that have not provided or verified their TLF will have their teachers sampled from the vendor or clerically-researched list of teachers. If outstanding school-level forms remain, a third package will be mailed to the survey coordinator or principal at the school address. This package will include a reminder letter, paper versions of the principal and/or school questionnaire(s), and postage-paid addressed return envelopes. Principal and survey coordinator email addresses will be used as means of reminding nonresponding schools to complete their questionnaires.

Beginning in late January 2018, priority schools that have not yet completed their school and/or principal questionnaires will be sent to a telephone reminder operation aimed at reminding the survey coordinator or school principal to complete their questionnaires. If outstanding school-level forms remain after the telephone reminder operation, one more attempt by mail, email, and telephone will be made to remind the school to complete their outstanding questionnaire(s).

B.2.2.2 Non-priority Schools

In late September 2017, all non-priority schools will receive an initial school package addressed to the survey coordinator at the school address. If a survey coordinator was not established during the screener interview, the package will be addressed to the principal at the school address. The package will contain a letter to the survey coordinator or principal, and three individual sealed envelopes that contain login information for completing the TLF, Principal Questionnaire, and School Questionnaire. Principals and survey coordinators will also be contacted by email around the same time the initial packages are mailed to the sampled schools. The emails will contain the appropriate hyperlinks and User IDs to complete the NTPS questionnaires online.

About three weeks later, a second package will be mailed to nonresponding schools. The package will include a reminder letter to the survey coordinator or principal and replacement materials for completing the outstanding questionnaires online. Principal and survey coordinator email addresses will be used as means of reminding nonresponding school staff to complete their questionnaires.

Beginning in November 2017, nonpriority schools that have not completed their TLF electronically will be sent to a telephone reminder operation aimed at reminding the survey coordinator or school principal to complete their TLF online. Non-priority schools that have completed their TLF but have not returned either the Principal Questionnaire or School Questionnaire will receive a reminder letter and email during this time.

In late November 2017, non-priority schools with outstanding school-level questionnaires will be mailed a third package. The package will include a reminder letter to the survey coordinator or principal, paper versions of the questionnaires that are still outstanding, and postage-paid return envelopes. If the TLF is one of the outstanding questionnaires, the version included in this third mailout will be pre-populated with teacher list data from the vendor or clerical research. Principal and survey coordinator email addresses will be used as means of reminding nonresponding school staff to complete their questionnaires.

In early January 2018, non-priority schools that have not yet completed their TLF will be sent to a Field operation, where sampled schools will receive an in-person visit from a Field Representative. The expectation for the personal visit is that the Census Bureau Field Representative will verify the school’s TLF, which will be pre-populated with vendor or clerically-researched data when it’s available, and distribute paper school and/or principal questionnaires as needed. After the Field operation, non-priority schools that have not provided or verified their TLF will have their teachers sampled from the vendor or clerically-researched list of teachers.

Beginning in early January 2018, principals and/or survey coordinators in non-priority schools that have completed their TLF but have not completed their school and/or principal questionnaire will be sent a reminder email and contacted by telephone.

If outstanding school and/or principal questionnaires remain after the field or telephone operation, one more attempt by mail, email, telephone, and in-person visit (if not previously visited) will be made to attempt to remind the school to complete their outstanding questionnaire(s).

B.2.3 Teacher Data Collection

Teachers will be sampled weekly from completed or verified TLFs throughout data collection. As teachers are sampled, they will be mailed an initial teacher package containing a letter that introduces the survey, provides the login information to complete their survey online. Around the same time, teachers for whom an email address is available will also be sent an email including the hyperlink and User ID to complete their teacher questionnaire online. If the school has a survey coordinator established, the individually-sealed teacher packages will be sent to the survey coordinator, at the school address, with a cover letter. If the school does not have a survey coordinator established, the teacher packages will be mailed individually to the sampled teachers at the school address in most cases. Exceptions may be made to this for late sampled teachers whose materials may be mailed directly to their school’s principal to distribute.

If the school’s teachers were sampled from a vendor or clerical list (where the school did not complete or verify a TLF), materials for the sampled teachers to complete their teacher questionnaires will be mailed directly to the teachers at their school address regardless of whether a survey coordinator was established. Exceptions may be made to this for late sampled teachers whose materials may be mailed directly to their school’s survey coordinator (when there is one established) or the principal to distribute.

Teachers with a valid email address will be sent an email containing the hyperlink to the online Teacher Questionnaire and their User ID a few days after their initial mailout.

Each sampled teacher will receive as many as three reminder packages to complete their outstanding Teacher Questionnaire. Each teacher mailing will be accompanied by an email to the teacher a few days after the mailing. The first reminder letter will contain the login information for the Teacher Questionnaire (URL and User ID) and will be sent to the survey coordinator (if applicable). The second and third reminder packages will include a letter and a paper questionnaire and will be addressed directly to the sampled teachers at the school address, regardless of whether the school has a survey coordinator established.

Beginning in late January 2018, telephone interviewers will contact survey coordinators to ask them to remind their schools’ sampled teachers to complete their questionnaires. Telephone interviewers and/or Field Representatives will contact nonresponding teachers by phone or during an in-person visit from late February through May 2018.

B.3 Methods to Secure Cooperation, Maximize Response Rates, and Deal with Nonresponse

This section describes the methods that NCES will use to secure cooperation, maximize response rates, and deal with nonresponse for NTPS 2017-18. Section B.3.1 details how NTPS plans to secure cooperation by leveraging its status as the primary source of information on K-12 schools and staffing in the United States. Section B.3.2 describes the methods that will be used to minimize nonresponse. The design is based on the results from the 2015-16 NTPS, which employed a number of different contact strategies aimed at boosting response rates.

B.3.1 Methods to Secure Cooperation and Maximize Response Rates

The entire survey process, starting with securing research cooperation from key public school groups and individual sample members and continuing throughout the distribution and collection of individual questionnaires, is designed to increase survey response rates. In addition, the following elements of the data collection plan, in particular, will contribute to overall success of the survey and will enhance the survey response rates.

Visible support from top-level Federal, State, and local education officials. Without the support of high-level officials in the U.S. Department of Education, State Education Agencies, and the sampled local school districts, surveys of public school principals and teachers cannot be successfully implemented. Obtaining endorsements from these officials is a critical factor in the success of the data collection procedures. Top-level Education Department officials will need to fully support the data collection by endorsing the survey in writing and sending advance letters and notices to sampled districts that require prior research applications and to individual survey participants (principals and teachers) to encourage participation.
Endorsements from key public school groups. The level of interest and cooperation demonstrated by key groups can often greatly influence the degree of participation of survey respondents. Endorsements are viewed as a critical factor in soliciting cooperation from state and local education officials. NCES has obtained endorsements for the 2017-18 NTPS from the following organizations or agencies:

American Association of School Administrators

Association of American Educators

Association for Supervision and Curriculum Development

American Federation of Teachers

Association for Middle Level Education

Council of the Great City Schools

National Association of Elementary School Principals

National Association of Secondary School Principals

American Association of School Librarians

American Montessori Society

National Parent Teacher Association

Stressing the importance of the survey and the respondents' participation. Official letters will be used to motivate respondents to return surveys. NTPS 2017-18 respondent letters will be sent by the U.S. Census Bureau and signed by the NCES Commissioner and/or the Associate Commissioner of the Sample Survey Division at NCES. Communications in the form of both letters and emails will be personalized for the principal and survey coordinators, which is expected to have positive effects on the survey response rates.

B.3.2 Methods to Minimize Nonresponse

A major challenge in any survey is obtaining high response rates, and this is even more important today when response rates have been falling among federal surveys in general, and in the SASS/NTPS series of studies in particular.

The main problem associated with nonresponse is the potential for nonresponse bias in the estimates produced using data collected from nonrespondents. Bias can occur when respondents are systematically different from nonrespondents. Two approaches that will be used to reduce the potential for bias are designing the data collection procedures and methods wisely to reduce nonresponse (e.g., establishing survey coordinators) and using statistical methods of sampling and weighting to reduce the effect of nonresponse on the estimates. While the statistical approaches are important in controlling biases and costs, the data collection procedures and methods are at the heart of a successful study.

Methods selected to minimize nonresponse in NTPS 2017-18 will build upon those used in NTPS 2015-16, including actions that were taken late in the data collection to boost principal and teacher response rates.

Data Collection Strategies to Minimize Non-Response

Minimize survey burden on schools. NTPS survey procedures are designed to minimize the burden on schools and sampled individuals (principals and teachers), and the survey instruments have been designed to be completed as quickly and easily as possible.

Good questionnaire design techniques have been employed to minimize item nonresponse. Questionnaires from previous rounds of SASS and NTPS were carefully analyzed to determine which items had the highest levels of item nonresponse. This information guided NCES in reviewing the clarity of item wording, definitions, and instructions. Items that were not considered to be effective or useful were removed from the survey so as to streamline the questionnaires and ease the response burden.

A key design feature of NTPS is the ability to link to other NCES collections such as EDFacts and the Civil Rights Data Collection (CRDC). Information from these sources will be incorporated into final datasets to allow researchers and policymakers to analyze those data together. This will further reduce the need to collect from schools data that have already been collected from state or district education agencies.

Recruit survey coordinators. Successive administrations of SASS and NTPS have shown that an important procedure to help maximize response rates is to establish a school-based "survey coordinator" to serve as a primary point of contact for NTPS staff. The use of a survey coordinator is expected to help keep response rates high, provide some minimal data quality checks, and simplify the follow-up process by having one point of contact.
Tailor nonresponse follow up strategies in public schools. In an effort to maximize response rates and minimize the potential for bias, NCES took a number of steps prior to the 2015-16 NTPS to identify high priority schools. These high priority schools are those to be targeted differently during data collection. The schools identified as high priority had the lowest propensity to respond (based on 2011-12 SASS data, as described below) and the highest potential impact on estimates.

As in the 2015-16 NTPS, public schools sampled for the 2017-18 NTPS will be assigned a “priority” flag based on the weighted response influence of the case. The weighted response influence takes into account both the response propensity and the base weight of a school to create a measure of a school’s potential effect on nonresponse weighting adjustments and final estimates. The weighted response influence can be calculated as:

where: is the final weighted response influence for a school,

is the baseweight for a school, and

is the estimated response propensity for a school

As the formula shows, a case with either an extremely high weight or an extremely low response propensity has a large response influence, meaning that their nonresponse will disproportionately affect the nonresponse adjustment cell in which they are located. Missing that particular school’s information may result in biased estimates (if variables in the propensity model are related to outcomes of interest), and will certainly result in increased variance in the estimates (due to more variable final weights). In order to avoid having extreme weights drive the value of weighted response influence, the formula takes the natural log of the base weight.

The weighted response propensity model for the 2015-16 NTPS was developed using data from the 2011-12 SASS. Specific categories of variables available for evaluation include geography, urbanicity, racial/ethnic makeup, enrollment, grades levels, percent free lunch recipiency, and type of school. These variables are available in the SASS 2011-2012 and NTPS 2015-16 sample files, enabling us to leverage past experience in creating the response propensity models. The NTPS 2015-16 data collection plan employed propensity modeling to identify high priority schools and modified collection strategies in order to increase response rates for those schools. Results from the NTPS 2015-16 data showed that the model and strategies used helped reduce declining response rates amongst those schools and thus the same propensity model and similar collection strategies will be used in the 2017-18 NTPS.

The priority flag was assigned at the school level in the 2015-16 NTPS, and the same will be done for the 2017-18 NTPS. During data collection, the priority flag was used to move high priority schools and schools without a survey coordinator into field follow-up operations earlier in collection in an effort to boost response rates. Schools in the high priority group generally do not respond until later in the data collection process and ultimately require field intervention.

2017-18 NTPS data collection for priority schools will begin with a personal visit from a Census Bureau Field Representative rather than beginning with a series of mailouts and telephone operations. By contacting the school staff in-person at the beginning of data collection, costs are expected to be reduced due to the omission of the mailout and telephone operations that typically precede field operations. In addition, this approach is expected to raise the probability of response by providing the field staff more time to secure the completed questionnaires. The primary focus of the operation will be obtaining a complete TLF; however, the Field Representative will also deliver the invitations to complete the school and principal questionnaires online. Throughout data collection the cases assigned to field will be reviewed by NTPS staff on a daily basis.

NTPS focuses on obtaining cooperation and improving response rates at the school level for a number of reasons. Past administrations of SASS and NTPS have shown that when cooperation is obtained at the school level, teachers and principals are more likely to respond. Additionally, evaluations of schools’ response propensities have shown that the nonresponse in past administrations was driven primarily at the school level. Results showed that schools in special contact districts are the primary driving force behind low response propensity. Special districts are those that require additional applications or documentation to collect data in their schools. Nearly 80% of the schools with high propensity for non-response reside in these special districts. For this reason, resources will continue to be allocated to focus heavily on obtaining approvals from special contact districts in order to boost response rates for this group.

Use vendor lists for teacher sampling. NTPS teacher-level response rates are calculated by multiplying response at the school level to the TLF by response at the teacher level. In the past, this has meant that if the school did not complete the TLF, teachers from that school could not be sampled, ultimately lowering the teacher response rate. The goal in NTPS 2015-16 was to improve the overall teacher response rate by allowing NTPS to sample teachers from schools that have not submitted a TLF; therefore, TLFs received from sample schools were supplemented with vendor-purchased teacher lists. When a vendor-purchased list was unavailable, a clerical operation was conducted to look up teacher information on school and/or district websites. The vendor and clerically-researched lists were evaluated in the 2015-16 NTPS and 2014 NTPS pilot test and showed high levels of comparability to lists obtained directly from schools. In the 2017-18 NTPS, TLFs received from sample schools will be supplemented with vendor-purchased teacher lists and a clerical look-up operation utilizing school websites. This operation will help to improve the overall teacher response rate by allowing teacher sampling from schools that have not submitted a TLF.
Monitor publishability and bias measures. For NTPS 2015-16, NCES monitored data collection progress throughout survey operations in order to identify and potentially minimize problems with nonresponse. The Census Bureau created weekly “publishability” reports from their data collection tracking system that showed whether key analysis cells were large enough to provide publishable estimates as of that point in time. By monitoring this publishability metric, NCES was able to identify populations of schools for which nonresponse hampered reporting. These results will be considered in designing the sample and nonresponse follow-up strategies for NTPS 2017-18. NCES also monitored R-indicators, a measure of representativeness, or lack of bias in the respondent population, on a weekly basis. The closer the R-indicator is to 1, the more balanced is the respondent population. Towards the end of data collection in 2015-16, the R-indicator for the full sample indicated that the respondent population was fairly well balanced. NCES plans to continue to monitor these two indicators in NTPS 2017-18.
Personalize principal contact materials. The National Teacher and Principal Surveys (NTPS) 2015-16 Refusal Conversion Change Request (OMB# 1850-0598 v.13) was approved in January 2016 as a result of response issues with high priority schools despite early field collection efforts, much of which were due to the data collection timing falling between Thanksgiving and Christmas and a too short length of the field operation. During the field operation, schools often communicated to field staff that they had not received any mailed materials for NTPS to date. They indicated that packages addressed generically to “School Principal/ Administrator” are often either overlooked or thrown out upon receipt at the school, as generically labelled mail is perceived as junk mail. As a result, NTPS 2015-16 requested and received clearance to send personalized letters to nonresponding principals in charter schools, schools in towns, and schools with enrollment less than 100. To be sure that all mailed 2017-18 NTPS materials intended for the school principal successfully make it to the principal, all principal contact materials will be personalized with the principal’s name. Principals’ names will be obtained from vendor-purchased school staff lists. If a principal’s name is not available from the vendor, clerical staff will research this information using school and district websites.
Use of email to target principals, survey coordinators, and teachers. The 2015-16 NTPS demonstrated that email was an effective tool to drive participation in both the NTPS teacher and principal surveys. It proved that teacher email addresses could be effectively collected on the TLF, school websites, and from vendor lists of teachers; that principal email addresses could be effectively collected from school websites and from vendor purchased school data; and that survey coordinator email addresses could be effectively collected during the screener interview. Because personalized emails carry no cost and may help boost response, throughout 2017-18 NTPS data collection, teachers, principals, and survey coordinators will be contacted via email. The emails will include login information to access the NTPS online survey instruments, in addition to text inviting and subsequently reminding these respondents to complete their survey online.
Send a third reminder email to teachers. The 2015-16 National Teacher and Principal Survey (NTPS) Teacher Reminder Email Change Request (OMB# 1850-0598 v.15) was approved in June 2016 and requested the ability to send a third reminder email (fourth email in total) to late-sample waves of teachers in NTPS 2015-16 to give them a final reminder/opportunity to complete the survey before close-out. The response rates for late-sample wave teachers in the NTPS had been leveling off and appeared to be lower than for earlier waves of teachers. This may have been a product of the timing of school testing and late-school year activities because late-sample wave teachers received an invitation to complete the survey during a period with a heavy school workload. Given that this additional reminder email carried no cost and may help response rates, a third reminder email will be sent to nonresponding teachers during 2017-18 NTPS data collection.
Send a “letter of better understanding” to principals and teachers. After the 2015-16 NTPS collection, field representatives and the regional offices recommended that in NTPS 2017-18 “letters of better understanding” be sent to principals and teachers who may be hesitant to complete the survey to help them gain a better understanding of the survey by providing them information about how the data are used and referencing some of the published data from the SASS 2011-12 First Look Reports. These letters will be sent to principals and teachers in priority schools, which tend to exhibit high non-response.
Consider new methods of minimizing nonresponse. NCES is considering a number of other methods to minimize nonresponse in NTPS 2017-18, including the use of incentives. The 2017-18 NTPS will include an experiment designed to examine the effectiveness of offering teachers a monetary incentive to boost overall teacher response. Further information about incentives are provided below in section B.4.2.

Statistical Approaches to Nonresponse

One of the methods employed to reduce the potential for nonresponse bias is adjustment of the sample weights to account for nonresponse. If schools or teachers with certain characteristics are systematically less likely than others to respond to a survey, the collected data may not accurately reflect the characteristics and experiences of the nonrespondents, which can lead to bias. To adjust for this, respondents are assigned weights that, when applied, result in them representing their own characteristics and experiences as well as those of nonrespondents with similar attributes. The school weights are also raked to sampled-based control totals in order to maintain the background characteristics of the sample. This is another method used to reduce the potential for nonresponse bias in the estimates produced from the data.

Response rates will be computed for the TLF, the School Questionnaire, the Principal Questionnaire, and the Teacher Questionnaire. Data collected through any instrument with a response rate of less than 85 percent will be evaluated for nonresponse bias. In addition to comparing the characteristics of respondents and nonrespondents using data that are available from the sampling frames (for example, school type and school locale from the school frame), we will also compare study estimates to estimates from previous rounds of NTPS and SASS. The nonresponse bias analysis will be similar to that conducted for SASS as reported in study methodology documentation (for the most recent released SASS methodology report, see http://nces.ed.gov/pubsearch/pubsinfo.asp?pubid=2010332). A methodology report covering NTPS 2017-18 will be developed and released, and will describe the methods and results of the nonresponse bias analysis.

B.4 Tests of Methods and Procedures

The SASS/NTPS series of studies has a long history of testing materials, methods, and procedures to improve the quality of its data. Section B.4.1 describes those tests that have most influenced the NTPS design, beginning with the 2014-15 NTPS Pilot Test and continuing through a new research project aimed at improving procedures for requesting research approval from special contact districts. Section B.4.2 describes a proposed experiment to include private schools, principals, and teachers in NTPS.

The following tests are planned for the 2017-18 NTPS data collection:

Incentives for teachers (and/or coordinator), and
Private School Test.

B.4.1 Tests Influencing the Design of NTPS 2017-18

2014-15 NTPS Pilot Test

Five experiments designed to optimize the design of the 2015-16 NTPS were conducted as part of the 2014-15 NTPS Pilot Test: 1) the Questionnaire Mode Experiment, 2) the TLF Email Experiment, 3) the Invitation Mode Experiment, 4) the Teacher Questionnaire Instruction Experiment, and 5) the Vendor Analysis. Each of these experiments is briefly described below, along with its results and implications for successor NTPS data collections.

Questionnaire Mode Experiment. This experiment was designed to determine whether paper questionnaires or Internet survey instruments (i.e., mail‐only versus internet sequential modes) constituted the most effective mode of collecting the TLF, School Questionnaire, and Principal Questionnaire. For all three survey instruments, the schools assigned to the paper mode had higher response rates than the schools assigned to the internet mode.

Some known issues with data collection could have impacted these response rates. First, the pilot test did not use survey coordinators, a method shown to boost response rates in SASS. Second, there were problems related to the contact materials for the internet treatment groups. As a result of this experiment, NTPS 2015-16 was primarily paper based; used improved contact materials and login procedures; and included an experimental sample of 1,000 schools, outside the main study, which were offered Internet survey at the onset of data collection and which followed standard production NTPS procedures, including the establishment of a survey coordinator.

Teacher Listing Form (TLF) Email Experiment. This experiment was designed to assess the feasibility of collecting teacher email addresses on the TLF and the quality of those collected. The pilot test design included a split-panel experiment, with half of sampled schools randomly assigned to receive a TLF that included a request for teachers’ email addresses and the other half to receive a TLF that did not request email addresses. At the end of data collection, response rates were comparable between the schools that received the TLF with the email address field and the schools that received the TLF without the email address field. As a result of this experiment and the Invitation Mode Experiment described below, NCES used the TLF with the email address field in NTPS 2015-16 and plans to continue to do so for NTPS 2017-18.
Invitation Mode Experiment. The purpose of this experiment was to identify which of three methods of inviting teachers to complete the Teacher Questionnaire yielded the best response rates. Schools were randomly assigned to the following invitation modes: 1) both email and mailed paper invitation letters to complete the internet instrument (treatment A), 2) a mailed paper invitation letter to complete the internet instrument only (treatment B), and 3) a mailed package that included a letter and paper questionnaire (treatment C). The results of the experiment indicated that a strategy using a combination of email and paper invitations (treatment A) is best for inviting teachers to complete the internet questionnaire. The response rate for treatment group A was comparable to that of treatment group C that received only mailed paper materials. As a result of this experiment, teachers sampled for NTPS 2015-16 for whom we had a valid email address were sent both email and paper invitations as the initial request to fill out the Teacher Questionnaire. Teachers without valid email addresses were sent their initial invitation as part of a mailed package that included a paper copy of the survey. For the 2017-18 NTPS, NCES plans to push for web response by both mailed and emailed correspondence, switching to a paper questionnaire at the third mailing.
Teacher Questionnaire Instruction Experiment. This experiment was designed to determine (1) whether including instructions in the NTPS questionnaire impacts response rates for questionnaire items and data quality, and (2) whether the position, format, and presence or absence of a preface in the instruction impacts response rates for questionnaire items. NCES is currently analyzing the results from this experiment and plans to incorporate these findings in a future NTPS administration.
Vendor Analysis. The purpose of this experiment was to evaluate both the feasibility of collecting teacher lists from a vendor and the reliability of the purchased information to see whether it could be used to supplement or replace school-collected TLFs. NCES purchased teacher lists from a vendor for schools sampled for the 2014-15 NTPS pilot test. The vendor teacher lists were compared with information collected from the TLFs. The results suggested that the vendor list information was comprehensive and reliable at a relatively low cost. NCES used vendor lists to sample teachers from a subset of schools that did not respond to the TLF in NTPS 2015-16 and plans to use vendor lists for the 2017-18 NTPS.

NTPS 2015-16 Full-Scale Collection

Schools and Principals Internet Test. The 2015-16 NTPS included an Internet experiment for schools and principals, which was designed to test the efficacy of offering an internet response option as the initial mode of data collection, as done previously in the Questionnaire Mode Experiment included in the 2014-15 NTPS Pilot Study, described earlier.

Key differences exist between the 2014-15 and 2015-16 NTPS internet experiments, with the most notable being that the 2015-16 experiment included the use of a survey coordinator at the school, and improved respondent contact materials and mailout packaging. In the 2015-16 NTPS, an independent sample of 1,000 public schools was selected for this experiment, which invited schools and principals to complete the NTPS school-level questionnaires using the internet at the first and second contacts by mail. A clerical operation prior to data collection obtained email addresses for sampled principals assigned to the internet treatment. Principals were sent emails as an initial mode of invitation to complete the NTPS questionnaires as well as reminder emails; the timing of these emails was a few days following the mailings.

Paper questionnaires were offered at the third and final mailout. Data collection for the internet treatment concluded after the third mailing, so the schools in the experimental treatment did not receive a fourth mailing and were not included in the telephone follow-up or field follow-up operations. When comparing the response rates for all three survey instruments at the end of the reminder telephone operation – the most reasonable time to make the comparison – and removing the cases that would have qualified for the early field operation, the response rates for schools assigned to the internet treatment are five to six percent higher than those for the paper treatment. Therefore, the initial mailout will invite respondents to complete online questionnaires during the 2017-18 NTPS data collection for all questionnaire types. Paper questionnaires will be introduced during the third mailing. Principal email addresses (purchased from the vendor) and school-based survey coordinator email addresses (collected at the time the survey coordinator is established) will be utilized during data collection. Invitations to complete the principal and school questionnaires via the Internet response option will be sent to the principal and school-based survey coordinator by email in conjunction with the various mailings.

Contact Time Tailoring Experiment. This test was designed to determine the optimal contact time for teachers. During the telephone nonresponse follow-up operation, interviewers contacted nonresponding principals and teachers to remind them to complete their questionnaire. Teachers tend to be difficult to reach during the school day due to their teaching schedules. NCES staff hypothesized that teachers may be easier to reach by phone in the late afternoon, when school had been dismissed. To test the accuracy of this theory, an experiment was embedded in the telephone nonresponse follow-up operation. A portion of the NRFU teacher workload received an experimental treatment, where they were intended to be contacted only in the afternoon between 2:00 p.m. and 5:00 p.m. (respondent time). The remainder of the NRFU teacher universe functioned as the control group. These teachers were intended to receive contacts throughout the school day, per typical telephone follow-up procedures. The research questions this test was designed to answer were as follows:

Are afternoons more productive for calling teachers?
If not afternoons, are there more productive times than others for calling teachers?
Do productive contact times for teachers hold globally, or do different types of schools have different productive call time frames?
Can we use school-level frame information (e.g. urbanicity, school size, grade level) to help tailor call times in future rounds of data collection?
If the calls are being made at “productive times,” are fewer call attempts required to successfully make contact with the teacher?
If the calls are being made at “productive times,” are fewer call attempts and total contacts required to obtain a completed interview?

Operational challenges in conducting the call time experiment were encountered. Early in the telephone nonresponse follow-up operation, telephone interviewers reported that school staff members were complaining about receiving multiple calls to reach the sampled teachers. School staff members indicated that they would prefer to know the names of the teachers the interviewer needed to reach so that they could assist the interviewer in as few phone calls as possible. As a result, the results of the experiment could not be evaluated as intended. Instead of comparing the success of reaching the sampled teachers by their treatment group, staff compared the success rates of the actual call times. Call times were categorized as ‘early’ (before 2:00 p.m.) or ‘late’ (between 2:00 p.m. and 5:00 p.m.). There was not a noticeable difference in the success rates of contacting teachers by call time. Additional analyses on the data may be conducted to help inform future administrations of NTPS.

B.4.2 Tests Included in the Design of NTPS 2017-18

To address declining response rates among teachers in the 2015-16 NTPS, NCES is proposing to test the use of incentives to increase response in the 2017-18 collection.

Testing the use of teacher incentives. The 2017-18 NTPS will include an incentive experiment designed to examine the effectiveness of offering teachers a monetary incentive to boost overall teacher response.

Teachers will be incentivized during the first 12 waves of teacher sampling, then a combination of teachers and/or school coordinators or principals will be incentivized during the remaining waves. During the first 12 waves of the teacher sampling, teachers are only sampled from returned TLFs. However, beginning in wave 13 for schools, teachers could be sampled from returned TLFs, vendor lists, or internet look-ups. This change in the teacher sampling procedure provided a natural breakpoint between the two phases of the experiment and allows us to target the most challenging cases with an additional incentive for the school coordinator or principal. Table 8 shows the breakdown of when teachers and/or school coordinators or principals will be incentivized, as well as the incentive amounts. All incentive amounts will be prepaid.

Table 8. Incentive Types and Amounts

Waves	Incentive Type	Incentive Amount
Waves	Incentive Type	Non-Priority School	Priority Schools
1-12	Teacher Incentive	$5	$5
13-20	School Coordinator/Principal Incentive Teacher Incentive	$5 $5	$10 $10

Experimental Design

Because teachers are sampled on a flow basis throughout the NTPS data collection period, the incentives experiment will occur in two phases. Phase One of the incentives experiment will occur during waves 1 through 12 of the teacher sampling period. Phase Two will be conducted during waves 13 through 20 of the teacher sampling period. The sampling design for the incentives experiment will be comprised of eight experimental groups, into which schools will be sampled at random. Table 9 shows the experimental groups and initial sample sizes.

Table 9. Experimental Groups and initial sample sizes.

Experimental Group	Phase One (Waves 1-12)	Phase Two (Waves 13-20)	Initial School Sample Sizes	Initial Teacher Sample Sizes^¹
1	Teacher Incentive	Teacher Incentive School Coordinator (SC) or Principal Incentive	1,325 Public Schools 500 Private Schools	5,784 Public School Teachers 875 Private School Teachers
2	Teacher Incentive	Teacher Incentive No SC or Principal Incentive	1,325 Public Schools 500 Private Schools	5,784 Public School Teachers 875 Private School Teachers
3	Teacher Incentive	No Teacher Incentive SC or Principal Incentive	1,325 Public Schools 500 Private Schools	5,784 Public School Teachers 875 Private School Teachers
4	Teacher Incentive	No Teacher Incentive No SC or Principal Incentive	1,325 Public Schools 500 Private Schools	5,784 Public School Teachers 875 Private School Teachers
5	No Incentive	Teacher Incentive SC or Principal Incentive	1,325 Public Schools 500 Private Schools	5,784 Public School Teachers 875 Private School Teachers
6	No Incentive	Teacher Incentive No SC or Principal Incentive	1,325 Public Schools 500 Private Schools	5,784 Public School Teachers 875 Private School Teachers
7	No Incentive	No Teacher Incentive SC or Principal Incentive	1,325 Public Schools 500 Private Schools	5,784 Public School Teachers 875 Private School Teachers
8	No Incentive	No Teacher Incentive No SC or Principal Incentive	1,325 Public Schools 500 Private Schools	5,784 Public School Teachers 875 Private School Teachers

Experimental Group

Phase One

(Waves 1-12)

Phase Two

(Waves 13-20)

Initial School Sample Sizes

Initial Teacher Sample Sizes^¹

Teacher Incentive

School Coordinator (SC) or Principal Incentive

1,325 Public Schools

500 Private Schools

5,784 Public School Teachers

875 Private School Teachers

Teacher Incentive

No SC or Principal Incentive

1,325 Public Schools

500 Private Schools

5,784 Public School Teachers

875 Private School Teachers

Teacher Incentive

No Teacher Incentive

SC or Principal Incentive

1,325 Public Schools

500 Private Schools

5,784 Public School Teachers

875 Private School Teachers

Teacher Incentive

No Teacher Incentive

No SC or Principal Incentive

1,325 Public Schools

500 Private Schools

5,784 Public School Teachers

875 Private School Teachers

No Incentive

Teacher Incentive

SC or Principal Incentive

1,325 Public Schools

500 Private Schools

5,784 Public School Teachers

875 Private School Teachers

No Incentive

Teacher Incentive

No SC or Principal Incentive

1,325 Public Schools

500 Private Schools

5,784 Public School Teachers

875 Private School Teachers

No Incentive

No Teacher Incentive

SC or Principal Incentive

1,325 Public Schools

500 Private Schools

5,784 Public School Teachers

875 Private School Teachers

No Incentive

No Teacher Incentive

No SC or Principal Incentive

1,325 Public Schools

500 Private Schools

5,784 Public School Teachers

875 Private School Teachers

The schools will be assigned into one of eight experimental groups prior to the beginning of data collection. As such, the random assignment should result in similar TLF response rates across all groups by the beginning of Phase Two of the experiment. Approximately 10,600 public schools and 4,000 private schools will be sampled for NTPS 2017-18. Because Phase One of the incentive experiment is independent of Phase Two of the data collection period, the random assignment of schools into the eight groups should result in similar TLF response rates across all groups at the start of Phase Two. Before the start of data collection, each experimental group will be assigned 1,325 public schools and 500 private schools. Therefore, approximately 5,300 public schools and 2,000 private schools will be assigned to the treatment groups that send teacher incentives during Phase One of the experiment.

Balancing treatment group assignments

To ensure a similar distribution of schools for each of the eight experimental groups, the sample will be sorted by an indicator for whether or not the school is covered on the vendor list, potentially an indicator for whether or not the teacher list was located during internet look-ups, and other selected school characteristics. The school characteristics will be selected for having significant influence on response based on past cycles of NTPS and on a model calculated to predict the likelihood of a school returning the TLF. For example, one of the school characteristics that may have a significant influence on response is the priority/non-priority school status (i.e., schools with higher potential to impact weighting and lower propensity to respond that are subject to a slightly different set of collection operations), which will be taken into account by using it as a sort variable in which the priority/non-priority status will be a stratification variable in the model.

To identify covariates that are predictive of whether and when a school will return the TLF for NTPS 2017-18, two response propensity models were analyzed, along with a time-to-event model, using NTPS 2015-16 data.

The NTPS 2015-16 frame data and response status were used to create the three models. The first response propensity model used logistic regression to predict the likelihood that a school would return a TLF by the end of data collection. This model was also used to calculate a TLF response propensity score for the NTPS 2017-18 schools, which will be used in the final sort order. The second model used logistic regression to predict the likelihood that a school would return the TLF early. In this model, the school’s return of the TLF was considered early if the school returned the TLF before the date that teachers were sampled from the vendor lists in place of the TLF. Lastly, the time-to-response model used Cox proportional hazard modeling to predict the number of days until the school would return the TLF.

Split-panel analysis was used to train and validate both the TLF response propensity model and the early vs. late TLF response propensity model. Using the sort order that was used to select the NTPS public schools into sample, data from the NTPS 2015-16 public schools were used for this analysis and were divided into two equal and representative partitions. One partition would be used to train the propensity models, while the other partition would be used for model validation.

Final Sort Order

After the three models were analyzed, the covariates from the models were compared for similarities. Covariates that appeared in two or three of the models were considered strong candidates for inclusion in the final sort order. However, given the sample size and number of experimental groups, not all of the model covariates could be included in the final sort order. Using too many covariates would result in implicit strata that were too small to be defined as effective. The number of effective implicit strata was determined by considering the number of schools per strata, and the distribution of school characteristics within the strata.

Based on the school characteristics that are significant predictors for TLF returns, and the optimal number of sorting strata, the final sort order for public schools was determined to be the following:

vendor flag,
priority flag,
Census region,
urban/rural locale code,
response propensity for each school, and
random number.

Private schools were not sampled in 2015-16, and so no predictions can be made about when private schools might return the TLF in NTPS 2017-18. The response propensity models created for public schools cannot be used to predict TLF returns for private schools with any confidence. Therefore, the final sort order for private schools will not include the response propensity score, and will simply be the vendor flag, the priority flag, the Census region, the urban/rural locale code, and a random number.

Table 10. Breakdown of Experimental Groups into Analysis Groups

Description of Two Analysis Groups

for Phase One

Description of Four Analysis Groups

for Phase Two

Experimental Groups 1, 2, 3, and 4 –

Teacher Incentive

Experimental Groups 1 and 5 – Teacher Incentive and SC/Principal Incentive

Experimental Groups 2 and 6 – Teacher Incentive and

No SC/Principal Incentive

Experimental Groups 5, 6, 7, and 8 –

No Teacher Incentive

Experimental Groups 3 and 7 – No Teacher Incentive and SC/Principal Incentive

Experimental Groups 4 and 8 – No Teacher Incentive and

No SC/Principal Incentive

For analysis purposes, the experimental groups will be collapsed based on phase of the experiment (breakdown shown in Table 10). For teachers that are sampled during Phase One, the analysis groups will collapse into treatment (1, 2, 3, and 4) and control (5, 6, 7, and 8). For Phase Two, the analysis groups will collapse into treatment one (1, 5), treatment two (2, 6), treatment three (3, 7), and control (4, 8).

The projected minimum difference in response rates that could be detected between treatment groups in Phase One is estimated to range from 4.55% to 3.82% depending on the possible TLF response rates by the end of Phase One, estimated between 41% to 60% based on previous data from the NTPS 2015-16 collection. The projected minimum difference in response rates that could be detected in Phase Two is estimated to range from 5.43% to 6.62% depending on the possible TLF response rates by the end of Phase Two, estimated between 59% to 40% based on previous data from the NTPS 2015-16 collection.

Planned Analyses of Experiment

Following data collection, analyses will be done separately for the two phases of the incentive experiment. For Phase One, the experimental groups will be collapsed into two analyses groups, while Phase Two can be collapsed into four analyses groups as described above. At the treatment level (incentive group(s) vs. control), the following analyses may be examined but not limited to:

Response rates,
R-Indicators,
Average number of contacts,
Days to response,
Correlation between school response and teacher response behaviors, and
Data collection costs.

The response rates will be calculated directly and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the incentive had on a case’s likelihood to response, given that case’s unique characteristics.

R-indicators will be used to determine the overall balance of the respondent population, as well as within each experimental group. R-indicators will be calculated for the full sample, as well as variable-level and category-level partial R-indicators to determine which characteristics specifically are contributing to imbalance within the respondent population.

The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify an incentive. If cases within an experimental group using incentives respond in a more timely fashion, which could further help increase timeliness of data releases, the use of incentives would improve the value of the data collected.

Using data collection costs associated with each mailout, the incentive itself, and estimates for interviewer costs, an average cost-per-case can be determined within each experimental group. A reduction in cost-per-case could justify the use of incentives to reduce the overall survey cost. Using incentives might lead to more initial costs; however, if cases respond in a fewer number of contacts, specifically more costly contacts such as personal visits, this could lead to a reduction in overall cost at the end of data collection.

Contingency Plan

As noted in section B.1.2, NCES has designed the NTPS for public schools to provide estimates based on the following key domains at the school level: school type (charter or non-charter); locale (city, suburban, town, rural); school level (high, middle, primary, combined); enrollment size (100-199, 200-499, 500-749, 750-999, 1,000 or more); poverty (high, low/medium); and free and reduced lunch participation (0-34%, 35-49%, 50-74%, 75% or more). Additionally, for NTPS 2017-18 public schools, individual states are also published domains. For private schools, the targeted domains at the school level are affiliation (Catholic, Other religious, nonsectarian); level (elementary, secondary, combined); and region (Northeast, Midwest, South, West). At the teacher level, NCES seeks to publish data for all of the domains mentioned above, as well as by special teacher domains including subject matter areas (Elementary, English, Math, Science, Social Science, Special Ed, Vocational, and Other), teacher race (Black, Hispanic, Other), and teacher experience (1 to 3 years, 4 to 9 years, 10 to 19 years, 20+ years). NCES statistical standards indicate that in order to publish data from a study, the coefficient of variation (CV) must not be greater than 50%. Further, CV’s of 30% or more must be flagged in tables. As indicated in B.1.2, NTPS 2017-18 has been designed to achieve a minimum CV of 25% for key domains. During the 2015-16 data collection, NCES and Census began tracking the response rate in each of these domains on a weekly basis to proactively identify groups where the CV may fall out of NCES standards.

In 2017-18, we plan to test offering an incentive to teachers if they belong to a domain that is ‘at-risk’ of not meeting NCES reporting standards towards the end of data collection. We refer to this as the ‘contingency plan’ since it will only be activated if needed during data collection. NCES monitored actual and expected response in each of the key domains on a weekly basis and identified domains that are ‘at-risk’ (at risk for meeting NCES publishability standards) as of February 12, 2018. Since this is the first year utilizing a contingency incentive, it is being implemented as an experiment with a control group that does not receive the incentive. While the plan is aimed at improving teacher response rates, because we expect teachers within a school to discuss the study, we will select schools based on meeting criteria of the domain at risk and all teachers within the school will be subject to the same treatment. We believe that if some teachers in the school were to receive an incentive and others not, it would negatively impact current and future response from that school. We expect that at the time the incentive is activated, some teachers at the school will have already responded to NTPS. We plan to provide these teachers with the contingency incentive level selected for that school as a thank you for their participation. For all other teachers in the school, the same incentive will be ‘prepaid’ and not conditional on their response. Given that schools selected for the contingency plan incentive will be based on the number of teachers in the ‘at-risk’ domain, selection for this will be independent of the main NTPS incentive experiment. Consistent with the other NTPS 2017-18 procedures, the incentive amount will vary between priority and non-priority schools. Teachers in selected non-priority schools will receive a pre-paid $10 cash incentive with their third mail-out or thank-you letter, and teachers in selected priority schools will receive a pre-paid $20 cash incentive with their third mail-out or thank-you letter. The earliest the contingency plan may go into effect will be during wave 13 (2/26/2018). The NTPS 2017-18 Contingency Plan and Email Change Request Memo (OMB# 1850-0598 v.22) provides the final details about the contingency plan.

Table 11 shows the approximate number of cases we anticipate receiving the contingency plan incentive.

Table 11. NTPS Contingency Plan Experiment Treatment Groups for Public and Private Schools

2017-18 NTPS Contingency Plan Experiment - Public Schools
		Incentive Treatment
		Control ($0)	Incentive ($20 Priority, $10 Non-priority)	Total
Priority Status	Priority	2,130	2,130	4,260
	Non-Priority	13,100	13,100	26,200
	Total	15,230	15,230	30,460
2017-18 NTPS Contingency Plan Experiment - Private Schools
		Incentive Treatment
		Control ($0)	Incentive ($20 Priority, $10 Non-priority)	Total
Priority Status	Priority	860	860	1,720
	Non-Priority	2,000	2,000	4,000
	Total	2,860	2,860	5,720

With approximately 2,130 public priority cases receiving and approximately 2,130 not receiving the contingency plan incentive, we will be able to detect a difference if the response rates between the two groups differ by at least 3.9 percentage points. With approximately 13,100 public non-priority cases receiving and approximately 13,100 not receiving the contingency plan incentive, we will be able to detect a difference if the response rates between the two groups differ by at least 1.6 percentage points.

With approximately 860 private priority cases receiving and approximately 860 not receiving the contingency plan incentive, we will be able to detect a difference if the response rates between the two groups differ by at least 6.1 percentage points. With approximately 2,000 private non-priority cases receiving and approximately 2,000 not receiving the contingency plan incentive, we will be able to detect a difference if the response rates between the two groups differ by at least 4.0 percentage points.

The initial incentives experiment and secondary contingency plan incentives experiment, as described above, will follow the same procedures with the same incentive amount (depending on priority status and treatment group assignment) for both the public and private schools in the NTPS 2017-18 sample.

B.4.3 NTPS 2017-18 Private School Test

This section describes the portion of the NTPS 2017-18 information collection for private schools in the United States as related to the private school test per se. The private school sample was designed to be both a full-scale pilot study, and to be utilized for the large-scale incentive and contact experiment. The general study design is described in Section B.1 and the testing aspects are discussed in this section. The NTPS 2017-18 collection of data from private schools and their principals and teachers is designed as a test to determine whether NCES can achieve response rates with this population that are sufficient to publish accurate and reliable estimates and that meet NCES standards for publication.

In SASS 2011-12, the response rates for private schools, particularly in specific strata, were low (see Table 12 below). Because teachers are sampled from the TLF, a form submitted by responding schools, the teacher response rate is doubly affected by low response rates at each level. In the SASS 2011-12 administration, this resulted in an overall teacher response rate of 50.0 percent, ranging from 26.9 to 64.5 percent in some strata. This meant that data did not meet NCES publishing standards. For this reason, private schools were not included in the NTPS 2015-16.

In the NTPS 2017-18, NCES will conduct an embedded test with private schools both to determine whether sufficient response can be achieved to provide reliable estimates for private schools and to evaluate specific methods for improving response rates. The private schools selected for this test will undergo data collection procedures that will be generally similar to those used with the NTPS 2017-18 public school sample (described in Section B.2.). Some procedures will be adjusted to accommodate differences specific to this sector (e.g., religious holidays and schedules).

Table 12. SASS 2011-12 Private School Response Rate, by Strata
			Base-weighted Response Rates
Strata	Schools	Principals		Teacher Listing Form (TLF)	Teachers: Questionnaire	Teachers: Overall
Total	65.7%	64.7%		71.6%	69.9%	50.0%
Affiliation Stratum
Catholic
Parochial	75.0%	74.7%		78.0%	76.8%	59.9%
Diocesan	72.7%	71.4%		74.4%	75.8%	56.4%
Private Order	66.7%	68.1%		67.3%	85.5%	57.5%
Other Religious
Baptist	66.3%	63.8%		68.8%	64.8%	44.6%
Jewish	45.5%	40.0%		49.9%	53.8%	26.9%
Lutheran	81.2%	81.1%		85.2%	72.6%	61.8%
Seventh-Day Adventist	79.1%	77.6%		82.0%	63.5%	52.0%
Other Religious	60.6%	57.8%		68.1%	64.9%	44.2%
Nonsectarian
Regular	57.5%	56.9%		66.1%	66.5%	44.0%
Special Emphasis	63.7%	66.4%		68.7%	64.1%	44.1%
Special Education	77.9%	80.7%		82.7%	77.9%	64.5%

SOURCE: Table 19. Chapter 6, Documentation for the 2011-12 Schools and Staffing Survey, NCES 2016-817 (under review)

B.4.3.1 Universe and Sample Design

The sample for the private school test is the same as for the main pilot study, so the corresponding sections in Section B.1 are also relevant here. This section discusses the particular issue of the power requirements for the experimental design, which drove the final sample size designation of 4,000 schools and 9,000 teachers (these are about 33% greater than SASS 2011-12). Section B.1 presents the precision levels for major school domains for the sample design: this section discusses the power issues.

B.4.3.2 Precision Requirements and Sample Sizes

The sample design for private schools is designed to detect a response rate difference of 4.7% for comparing school response rates between the two experimental halves for the incentives experiment (for a one-sided test of the null hypothesis of no difference with 95% confidence), and a response rate difference of 6.3% for comparing school response rates between the treatment and control halves of the priority school group for the tailored contact strategy experiment (roughly 30% of the sample in each half: the priority schools are designed to be 60% of the sample). The benchmark was a 5% response rate difference: the study design achieves a little more than this in the former case, and a little less than this in the latter case.

B.4.3.3 Private School Test Study Design

The NTPS 2017-18 private school test has been designed both to provide accurate estimates for teachers and principals in private schools in the United States as well as to examine the effects of strategies to improve response in this population.

B.4.3.3.1. Data Collection

Data collection for private schools in NTPS 2017-18 will follow the same procedures and operations as those for the public schools, except where noted. After the advance letter and screener interview, private schools will enter one of two data collection paths: priority (see Section B.2.1.1) and non-priority (see Section B.2.1.2). The data collection methodology used will depend upon each school’s identification as a “priority” school and its assignment to a treatment group (see Section B.4.3.3.3 below).

B.4.3.3.2. Methods to Enhance Response from Private Schools

Response rates have been falling among federal surveys in general, and within the sector of private education in particular (SASS/NTPS series and the Private School Survey (PSS)). As stated above, the response rates for the SASS 2011-12 were insufficient to meet NCES standards for publication.

Similar to the methods used with public schools (see Section B.3), NCES plans to use several techniques to enhance response in the collection of data from private schools. These methods include: recruitment of survey coordinators within the school; testing the use of teacher and school-level incentives; stressing the importance of the survey and of respondents’ participation; personalization of principal contact materials (when possible); use of email to target respondents; and monitoring measures of publishability and bias.

Some methods will be similar, but will be adapted to be respectful of differences relevant to private schools. These include:

Minimize survey burden on schools. NTPS survey procedures are designed to minimize burden on sampled schools and individuals (principals and teachers). Because the NTPS 2017-18 field period overlaps with the collection of data for another collection – the PSS 2017-18 – we have combined elements of the PSS into the NTPS collection in an effort to reduce burden that would come from the administration of multiple surveys in the field. Sampled private schools need to complete only one questionnaire for both surveys, thereby reducing burden on private schools and staff.
Endorsements from key private school groups. The level of interest and cooperation demonstrated by key groups can often greatly influence the degree of participation of survey respondents. NCES continues to work closely with private school associations as a key stakeholder in improving estimates on the state of private schools in the United States. Endorsements are viewed as a critical factor in soliciting cooperation from private schools. NCES has obtained endorsements for the NTPS 2017-18 from the following organizations or agencies:

Agudath Israel of America

American Association of Christian Schools

National Association of Episcopal Schools

American Association of School Administrators

American Montessori Society

National Association of Independent Schools

National Association of Private Special Education Centers

National Catholic Educational Association

Association of Christian Schools International

Association of Christian and Classical Schools

Association of Military Colleges and Schools

Association Montessori International

Association of Waldorf Schools of North America

National Christian School Association

Association of Christian Teachers and Schools

National Coalition of Girls’ Schools

National Independent Private School Association

National Society for Hebrew Day Schools

Christian Schools International

Council for American Private Education

Oral Roberts University Educational Fellowship

Evangelical Lutheran Church in America

Friends Council on Education

Solomon Schechter Day School Association

Islamic School League of America

General Conference of Seventh-Day Adventists

Southern Baptist Association of Christian Schools

Jewish Education Services of North America

Jesuit Secondary Education Association

The Association of Boarding Schools

The Jewish Education Project

Wisconsin Evangelical Lutheran Synod

Lutheran Church-Missouri Synod

Contact by private school association representatives. In addition to endorsements of the collection, private school associations have asked for resources to engage their membership more directly in efforts to encourage response. NCES will provide private school associations with a list of the schools that have been sampled and are in their membership. The private school associations will contact their member schools to encourage their participation in the survey. To ensure privacy of sample members, private school associations will be required to sign a memorandum of understanding (MOU) before receiving the list of their member schools that have been sampled.
Tailor non-response follow up strategies. Similar to the method used with the public schools, NCES will develop a propensity score model that will be used to identify and segment priority schools in the private school sample. The propensity model will use a combination of response likelihood and the risk of bias. Thus, the highest priority schools in the private school sample will be those with the lowest likelihood of response and the highest likelihood to contribute to bias. The model will be built using data from SASS 2011-12 and PSS 2015-16. The priority flag will be assigned at the school level. During data collection, the priority flag will be used to move high priority schools into field follow-up operations earlier in data collection in an effort to boost response rates. Based on prior collections, NCES knows that schools identified in the high priority group generally do not respond until later in the data collection process and usually require field intervention.

Private schools identified as “priority” schools will receive an enhanced follow-up contact strategy (see Exhibit 1 and Section B.2.1.1). Data collection for priority schools will begin with a personal visit from a Census Bureau Field Representative rather than beginning with a series of mailouts and telephone operations. When used with public schools, this has resulted in reduced costs due to the omission of the mail and telephone contacts that precede field operations and in an increase in the probability of response by providing the field staff more time to secure the completed questionnaires. As with public schools, NTPS will focus on obtaining cooperation and improving response rates at the school level, with particular emphasis on a completed TLF, because of its relation to increased response for principals and teachers.

Evaluation of vendor lists for teaching sampling. Similar to the evaluation of the use of vendor-purchased lists to enhance teacher sampling in NTPS 2015-16, NCES will evaluate the comparability of vendor lists of teachers at private schools with TLFs received from sampled schools. If there is a high accuracy rate between teacher information from submitted TLFs and purchased lists, this is an enhancement that can be added in future collections. If successful, this operation may help to improve the overall teacher response rate by allowing teacher sampling from schools that have not submitted a TLF.

B.4.3.3.3. Tests within the Private School Test

NCES plans to test two methods intended to increase response explicitly: the use of a tailored contact strategy using model-based targeting and the use of incentives (both teacher- and school-level). Although both techniques will be assigned at the school-level, the treatments will be administered, and affects assessed, at different levels (see Exhibit 2).

Tailored Contact Strategy Treatment. A propensity score model will be used to identify and segment priority schools. The highest priority schools for the collection are those with the lowest likelihood of response and the highest likelihood to contribute to bias. In order to assign schools into treatment groups, schools will be matched into pairs with similar likelihood scores and then randomly assigned to groups (“priority” early contact schedule versus “non-priority” typical contact schedule). Because the priority school data collection plan is resource intensive and is not necessary for some schools (e.g., schools with a high likelihood of response), the plan is to test this strategy with 60 percent of the sample, based on the highest priority cases as identified by the propensity models. Once they are matched into pairs, half of the schools in the test group (30 percent of schools in the starting sample) will be assigned to the treatment group (“priority”), and the other half of the schools (30 percent) will be assigned to the comparison group (“non-priority”). The remaining 40 percent of the starting sample will receive the typical contact schedule for the non-priority schools.

Incentives for Teachers and Principals/Coordinators. The private school test of the effectiveness of incentives, including recipient and amount, will follow the same procedures as outlined for public schools (see Section B.4.2). Incentive groups are assigned at the school level, but administered to teachers and to principals or coordinators. For example, teachers will be assigned to the treatment group (incentive) or comparison group (no incentive) at the school level (i.e., all teachers within a school will receive the same treatment).

Exhibit 2. Diagram of Study Design for NTPS 2017-18 Private School Test for Tailored Contact Strategy Treatment (school) and Incentives (teacher and coordinator/principal)

Table 13 below presents power calculations for the private school test for the tailored contact strategy treatment experiment and the incentives experiment. In both cases, the tests are one-sided² with a confidence level of 95%. The null hypothesis posits that the treatment and control groups will have equal response rates. For the incentives experiment, the null response rate is 65.66% (the school-level response rate for the SASS 2011-12 private schools all U.S.), and for the high-priority group the null response rate is 60.00% (the assumed school-level response rate for a high-priority stratum). The assumed design effects are the expected design effects from sampling and weighting. The power is the difference in response rates between the treatment and control groups for the two experiments (e.g., for the Incentive Experiment the alternative hypothesis is a response rate of 65.66% for the control arm and 70.66% for the treatment arm). The minimum detectable is the alternative hypothesis response rate difference that has exactly 80% power.

Table 13. Power Calculations for Incentive Experiment and Tailored Contact Strategy Experiment

Priority

Group

Half Sample Size

Null Response

Rate

Design

Effect

Power

at d=5%

Minimum Detectable d

at 80% Power

Incentive Experiment

2,000

65.66%

1.56

84.65%

4.66%

Tailored Contact Experiment

1,200

60.00%

1.62

62.70%

6.32%

From the standpoint of a 5% difference, the overall design is slightly overpowered for the incentive experiment, and slightly underpowered for the tailored contact strategy experiment (the difference in response rates has to be 6.32% to be picked up with 80% chance). Detailed information regarding the incentives experimental design and contingency plan as well as the planned operations for both public and private schools are described above in section B.4.2.

B.5 Individuals Responsible for Study Design and Performance

The following individuals are responsible for the NTPS 2017-18study design, data collection, and analysis:

Amy Ho, NCES

Deanne Swan, NCES

Andy Zukerberg, NCES

Carolyn Pickering, U.S. Census Bureau

Shawna Cox, U.S. Census Bureau

Mary Davis, U.S. Census Bureau

James Farber, U.S. Census Bureau

David Marker, Westat

Lou Rizzo, Westat

Minsun Riddles, Westat

1 The teacher samples sizes will not be exactly equal across experimental groups, as the number of teachers sampled from each school is not equal. However, each group should contain roughly the same number of teachers.

2 We are checking for response rate differences only in one direction: the treatment arm having higher response rates.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-21

Part B NTPS 2017-18