Part B NTPS 2020-21 (1)

This request is to conduct NTPS 2020-21, including all of its recruitment and data collection activities. Because of the overlap in time, this request also carries over the burden and materials for the approved preliminary activities. Section B.1 of this document describes the universe, sample design, and estimation details for NTPS 2020-21. Section B.2 describes the data collection procedures for NTPS 2020-21, including the preliminary field activities approved in an earlier submission (OMB# 1850-0598 v.26). Section B.3 discusses methods to secure cooperation and mitigate nonresponse. In particular, it describes methods used to improve response rates in NTPS 2017-18 and how those methods will be used in NTPS 2020-21. Section B.4 describes recent developments in a long history of tests of methods and procedures to improve data quality. Section B.5 lists the names of those involved in the design of the study and the development of these materials.

B.1.1 Universe and Sample Design: Respondent Universe

B.1.1.1 Schools

The respondent universe for NTPS 2020-21 data collection consists of approximately 93,000 public schools and 24,000 private schools in the 50 U.S. states and the District of Columbia (DC) that offer instruction in any of grades 1-12 or the ungraded equivalent. To be eligible for inclusion in the sample, schools must: provide classroom instruction to students; have one or more teachers who provide instruction; serve students in at least one of grades 1-12 or the ungraded equivalent; be located in one or more buildings, and be located in the continental United States.

NCES’ 2018-19 Common Core of Data (CCD)¹ will be used to construct the public school frame. The respondent universe for charter schools will be identified as those public charter schools that meet the NTPS definition of an eligible school found on the CCD. The universe has been adjusted to remove kindergarten-terminal schools, which are not eligible for NTPS. Table 1 presents the number of public schools on the 2017-18 CCD by urbanicity and school level. The CCD for 2018-19 is not yet available at the time of submitting this package. The NTPS 2020-21 school sample will be drawn in April-May 2020 and we will begin to contact sampled schools in June 2020.

Table 1. Respondent universe by school level and urbanicity for the proposed public school sample, based on the 2017-18 Common Core of Data (CCD)²

School level
Region	Primary	Middle	High	Combined	Total
Central City	15,265	3,853	5,998	948	26,064
Suburban	17,742	5,495	6,150	801	30,188
Town	5,714	2,392	3,623	507	12,236
Rural	11,794	3,493	6,655	2,824	24,766
Total	50,515	15,233	22,426	5,080	93,254

SOURCE: 2017-18 CCD.

The private school frame is drawn from the 2017-18 Private School Survey (PSS) frame. Preschools and schools with kindergarten as the highest grade are excluded. Table 2 presents the number of private schools on the 2017-18 PSS by urbanicity and school level.

Table 2. Respondent universe by school level and urbanicity for the proposed private school sample, based on the 2017-18 Private School Survey (PSS)

School level
Region	Elementary	Secondary	Combined	Total
Central City	4,621	1,070	2,555	8,246
Suburban	4,649	854	2,903	8,406
Town	1,234	148	731	2,113
Rural	2,735	447	1,882	5,064
Total	13,239	2,519	8,071	23,829

SOURCE: 2017-18 PSS.

B.1.1.2 Teachers

Teachers will be randomly sampled within the second design stage from roster information provided by each participating sampled school. Teachers within the sampled school are classified as ineligible for NTPS if they are a short-term substitute teacher, student teacher, or a teacher’s aide; or if they do not teach any of grades K-12 or comparable ungraded levels. The information that classifies teachers as ineligible is obtained from the Teacher Questionnaire.

B.1.2 Precision Requirements and Sample Sizes

This section details the school sample sizes and precision requirements for the NTPS 2020-21 public and private school samples.

The final NTPS 2020-21 public school sample will include approximately:

10,600 schools and school principals (9,300 traditional public and 1,300 public charter), with the goal of at least 6,700 interviews for each; and
53,400 teachers (46,900 traditional public and 6,500 public charter), with the goal of at least 34,700 interviews.

The final NTPS 2020-21 private school sample will include approximately:

3,000 schools and school principals, with the goal of at least 1,750 interviews for each; and
6,800 teachers, with the goal of at least 4,500 interviews.

Sampling – Public Schools

The 2020-21 NTPS oversampling stratification will be based preliminarily on the following domains:

Charter/Non-charter;
School Level (primary, middle, high, combined);
Urbanicity (city, suburb, town, rural);
State Tier (based on state).

The NCES standards for publishability indicate that the coefficient of variation (CV) must be no larger than 50%, and if the CV is between 30% and 50%, the estimates are published with a caveat. For a population proportion of 20%, a CV of 30% corresponds to a standard error of 6%. In order to make sure that we don’t exceed the 30% CV minimum with the uncertainties about response and about exact values of design effects, we set as a target a CV of 25% as a lower bound. This corresponds to an expected standard error of 5%. This considerably reduces the chance of exceeding the 30% boundary (if we set 30% itself as the target, we would be above it one-half of the time). Our target goal then for each state is to make sure that the expected standard error is no larger than 5% for a population proportion of 20% (a CV of 25%), at both the school and teacher level.

Table 3 presents a portion of the analysis for public schools by school type, grade level, urbanicity, and poverty status. Presented are the anticipated number of responding schools or principals for the NTPS design and the expected precision based on analyses using the NTPS 2017-18 final response rates and CV of 25%.

Table 3. NTPS 2020-21 public school domain expected interviews, standard errors, and design effects with state oversampling to achieve 25% CV or less

Domain	Frame Schools	Expected Sample Size (completed interviews)	Expected Standard Error	Design Effect
All	93,634	7,632	0.63%	1.68
Charter	6,819	878	1.71%	1.41
Non-charter	86,815	6,754	0.67%	1.66
Primary	51,470	3,482	0.88%	1.49
Middle	14,177	1,275	1.43%	1.42
High	20,406	1,952	1.42%	2.17
Combined	7,581	923	1.90%	1.83
City	26,085	2,212	1.17%	1.67
Suburban	30,305	2,267	1.13%	1.58
Town	12,630	1,185	1.61%	1.69
Rural	24,614	1,968	1.29%	1.79
Enrollment < 100	7,946	357	3.54%	2.45
100 <= Enrollment < 300	7,341	526	2.36%	1.60
300 <= Enrollment < 500	36,097	2,725	1.00%	1.49
500 <= Enrollment < 750	23,395	1,882	1.15%	1.38
750 <= Enrollment < 1,000	9,447	909	1.64%	1.34
1,000 <= Enrollment	9,408	1,234	1.36%	1.26
Percent FRPL < 35%	26,848	2,311	1.21%	1.86
35% <= Percent FRPL < 50%	15,724	1,389	1.44%	1.58
50% <= Percent FRPL < 75%	26,327	2,082	1.18%	1.60
75% <= Percent FRPL	23,900	1,763	1.28%	1.59
Not Participating FRPL	835	85	8.30%	3.26

Table 4 presents the analogous precision analysis for public schools by state.

Table 4. NTPS 2020-21 public school expected interviews, standard errors, and design effects by state with state oversampling to achieve 25% CV or less

State	Frame Schools	Expected Completed Interviews	Expected Standard Error	Design Effect
All	93,634	7,632	0.63%	1.68
AK	510	122	4.96%	1.65
WY	339	96	4.99%	1.30
DC	219	69	4.98%	0.95
VT	315	81	4.99%	1.11
SD	496	112	4.98%	1.52
MT	563	113	4.99%	1.54
DE	218	72	4.98%	0.98
RI	297	76	4.98%	1.04
ND	421	91	4.98%	1.24
HI	292	79	4.99%	1.08
NH	455	88	5.00%	1.20
ID	706	101	4.98%	1.38
ME	613	90	5.00%	1.23
NV	666	107	5.00%	1.47
WV	740	92	4.98%	1.26
NM	835	105	4.98%	1.42
NE	870	106	4.99%	1.45
CT	1,209	138	4.99%	1.88
UT	993	103	4.97%	1.38
OR	1,227	104	4.99%	1.42
MS	1,071	98	4.99%	1.34
IA	1,186	100	5.00%	1.37
AR	960	97	4.99%	1.32
KS	1,257	98	4.99%	1.35
KY	1,462	105	5.00%	1.44
MN	2,066	153	4.97%	2.07
SC	1,231	96	4.92%	1.28
AL	1,508	104	5.00%	1.42
AZ	2,306	165	4.99%	2.26
MD	1,421	99	4.98%	1.35
OK	1,441	105	4.99%	1.44
LA	1,349	101	4.99%	1.39
CO	1,671	117	4.78%	1.47
WA	2,257	124	4.77%	1.54
MA	1,786	120	4.66%	1.43
IN	1,881	120	4.54%	1.35
WI	1,943	124	4.52%	1.40
TN	1,787	122	4.46%	1.32
MO	2,008	142	4.27%	1.42
VA	2,072	150	4.01%	1.32
NJ	2,474	171	3.90%	1.43
GA	2,303	177	3.72%	1.34
MI	3,302	211	3.72%	1.59
NC	2,638	189	3.58%	1.33
PA	3,029	214	3.41%	1.37
OH	3,357	220	3.37%	1.37
IL	3,920	238	3.30%	1.42
FL	4,047	326	2.95%	1.56
NY	4,780	354	2.67%	1.39
TX	8,880	639	2.28%	1.82
CA	10,257	609	2.21%	1.63

Table 5 provides the analogous precision analysis for public school teachers. The expected standard errors were calculated based on analyses using the NTPS 2017-18 final response rates and CV of 25%.

Table 5. NTPS 2020-21 major domain public teacher expected teacher interviews, standard errors, and design effects with state oversampling to achieve 25% CV or less

Domain	Frame Full-Time Equivalent Teachers (in 1000s)	Expected Teacher Completed Interviews	Expected Standard Error	Design Effect
All	3,137.6	43,254	0.38%	3.08
Charter	160.4	4,328	1.07%	2.48
Non-charter	2,977.2	38,925	0.39%	3.01
Primary	1,487.2	16,999	0.57%	2.80
Middle	548.3	7,899	0.86%	2.95
High	912.8	13,725	0.70%	3.34
Combined	189.3	4,629	1.07%	2.67
City	928.7	12,939	0.69%	3.08
Suburban	1,214.0	14,301	0.65%	3.02
Town	365.0	6,326	0.96%	2.89
Rural	629.8	9,688	0.79%	3.01
Enrollment < 100	38.4	816	2.29%	2.15
100 <= Enrollment < 300	90.3	1,766	1.67%	2.46
300 <= Enrollment < 500	863.4	12,461	0.66%	2.76
500 <= Enrollment < 750	864.9	11,868	0.72%	3.09
750 <= Enrollment < 1,000	475.8	6,150	1.01%	3.18
1,000 <= Enrollment	804.7	10,194	0.81%	3.38
Percent FRPL < 35%	969.0	13,704	0.69%	3.24
35% <= Percent FRPL < 50%	541.1	7,905	0.89%	3.15
50% <= Percent FRPL < 75%	861.7	11,631	0.71%	2.97
75% <= Percent FRPL	737.8	9,487	0.79%	2.94
Not Participating FRPL	28.0	526	3.64%	3.49

Sampling – Private Schools

The NTPS private school sample will be roughly the same as the private school survey from Schools and Staffing Survey (SASS) 2011-12: a school sample size of 3,000 and about 1,750 completed school interviews. For the 2017-18 NTPS, the sample size was 4,000 (expecting 2,266 completed school interviews) to achieve sufficient power for an embedded experiment.

The sampling plan oversamples as follows:

Elementary schools are sampled at a rate proportional to the measure of size;
Secondary schools are sampled at a rate proportional to 3 times the measure of size;
Combined schools are sampled at a rate proportional to 1.2 times the measure of size;
The oversampling rates for Nonsectarian schools are increased by an additional factor of 1.25 (e.g. the oversampling rate for Nonsectarian secondary schools is 3 ×1.25 = 3.75); and
The oversampling rates for Baptist schools are increased by an additional factor of 1.1 (e.g. the oversampling rate for Baptist combined schools is 1.2 ×1.1 = 1.32).

For teachers, the expected number of completed interviews is estimated to be proportional to the product of the final school sampling factor and the number of full time equivalent (FTE) teachers over schools in the domain. The overall target number of completed interviews is 4,500. Assuming the attrition rate for the 2020-21 NTPS will be similar to the rate for 2017-18, the sample size needs to be 6,800 in order to yield the expected number of completed teacher interviews. The teacher sample size for a sampled school should be proportional to the product of the final teacher multiplier (based on the expected attrition adjustment factors), final school oversampling factor, and measure of size for the school.

Tables 6 and 7 show expected sample sizes, standard errors, and CVs for population percentages of 20% by key domains of school type, grade level, and region.

Table 6 presents a portion of the analysis for private schools by affiliation, grade level, and region.

Table 6. School-domain expected interviews, standard errors, and design effects for the NTPS 2020-21 private school sample

School domain	Frame schools	Expected completed school interviews	For 20% population percentage				Min pop % for CV <30%
School domain	Frame schools	Expected completed school interviews	Expected standard error	95% CI half-width	Design effect	CV	Min pop % for CV <30%
All	24,984	1,950	1.24%	2.44%	1.70	6.22%	1.06%
Catholic	6,428	598	2.13%	4.18%	1.53	10.66%	3.06%
Other religious	12,006	740	2.00%	3.92%	1.66	10.01%	2.71%
Nonsectarian	6,550	612	2.17%	4.25%	1.62	10.84%	3.16%
Elementary	14,030	702	1.86%	3.64%	1.36	9.29%	2.34%
Secondary	2,609	558	1.96%	3.85%	1.21	9.81%	2.61%
Combined	8,345	691	1.94%	3.80%	1.45	9.69%	2.54%
Northeast	6,018	540	2.57%	5.04%	2.00	12.86%	4.39%
Midwest	6,024	416	2.66%	5.21%	1.65	13.30%	4.68%
South	8,081	626	2.10%	4.11%	1.55	10.48%	2.96%
West	4,861	368	2.79%	5.47%	1.61	13.96%	5.13%

Table 7 provides the analogous precision analysis for private school teachers.

Table 7. Major domain expected teacher interviews for the NTPS 2020-21 private school sample

School domain	Frame teachers (FTE)	Expected completed teacher interviews	For 20% population percentage				Min pop % for CV <30%
School domain	Frame teachers (FTE)	Expected completed teacher interviews	Expected standard error	95% CI half-width	Design effect	CV	Min pop % for CV <30%
All	449,441	5,440	1.02%	2.00%	2.91	5.09%	0.71%
Catholic	136,810	1,812	1.84%	3.61%	3.17	9.20%	2.30%
Other religious	183,015	1,896	1.65%	3.23%	2.67	8.25%	1.86%
Nonsectarian	129,616	1,732	1.81%	3.54%	2.92	9.03%	2.21%
Elementary	174,278	1,433	1.74%	3.42%	2.25	8.72%	2.07%
Secondary	67,254	1,822	1.69%	3.30%	2.68	8.43%	1.93%
Combined	207,910	2,186	1.55%	3.04%	2.72	7.76%	1.64%
Northeast	128,835	1,630	1.95%	3.81%	3.19	9.73%	2.56%
Midwest	88,259	1,092	2.24%	4.39%	2.83	11.20%	3.37%
South	154,954	1,749	1.75%	3.43%	2.77	8.76%	2.09%
West	77,393	970	2.38%	4.66%	2.83	11.88%	3.77%

The 2020-21 NTPS will have an implicit stratification based on the proposed systematic sampling sort order, which uses a hierarchy of the following domains:

Three-level affiliation (Catholic, non-Catholic religious, nonreligious);
Three-level school span (elementary, secondary, combined);
Four-level Census region (Northeast, South, Midwest, West);
Four-level urbanicity (city, suburb, town, rural);
Eleven-level affiliation;
Five-level school size (enrollment <100, 100-199, 200-499, 500-749, 750+);
State;
Highest grade;
Twelve-level urbanicity (large city, medium-sized city, small city, etc.);
Zip code;
School enrollment;
PIN number.

Teachers in traditional public, public charter, and private schools will be sampled from roster information provided by each participating sampled school or from the vendor (when the school does not provide teacher information). The target teacher completed interview sample sizes are designed to be proportional to the square root of the number of full-time teachers for each school and assume an attrition rate due to nonresponse.

Sampling – Principals within All Schools

For each sampled traditional public, public charter, and private school, the principal will be included in the survey as a result of the school being selected.

Survey Weights

Schools, principals, and teachers will be weighted by the inverse of the probability of selection. The final weight will contain adjustments for nonresponse and any other sampling or field considerations that arise after the sample has been drawn.

Response Rates

We expect the NTPS 2020-21 response rates to approximate those of NTPS 2017-18 (for public and private schools) or to fall lower given the long-term trend in declining response rates for federal surveys. Table 8 provides the base-weighted response rates for NTPS 2015-16, and Table 9 provides the base-weighted response rates for NTPS 2017-18. Note that private schools were not included in NTPS 2015-16 and are therefore not included in table 8.

Table 8. Base-weighted response rates for NTPS 2015-16 by respondent and school type

School Type	Unit of Observation
School Type	Teacher	Principal	School
Traditional Public	67.9	71.8	72.5
Public Charter	66.2	71.9	73.2

Table 9. Base-weighted response rates for NTPS 2017-18 by respondent and school type

School Type	Unit of Observation
School Type	Teacher	Principal	School
Traditional Public	76.9	70.7	72.9
Public Charter	75.4	63.4	67.5
Private	75.9	62.6	64.5

B.2 Procedures for the Collection of Information

Section B.2.1 describes the operations for the already approved preliminary field activities for NTPS 2020-21, with Section B.2.1.1 describing special districts operation and Section B.2.1.2 the school pre-contact letter operation (OMB# 1850-0598 v. 26). Section B.2.2 describes school-level data collection procedures for the school-level questionnaires (i.e., Teacher Listing Form, School Questionnaire, and Principal Questionnaire), with Section B.2.2.1 describing the procedures to be used for schools with acceptable vendor data and Section B.2.2.2 for schools without acceptable vendor data. Section B.2.3 describes data collection procedures for the Teacher Questionnaire.

B.2.1 Preliminary Field Activities

B.2.1.1 Special Contact District Operation

Special contact districts require that a research application be submitted to and reviewed by the district before they will allow schools under their jurisdiction to participate in a study. Districts are identified as “special contact districts” prior to data collection because they were flagged as such during previous cycles of SASS, NTPS, or by other NCES studies. Special contact districts are also identified during data collection when districts indicate that they will not complete the survey until a research application is submitted, reviewed, and approved.

Once a district is identified as a special contact district, basic information about the district is obtained from the NCES Common Core of Data (CCD). The basic information includes the NCES LEA ID number, district name, city, and state. The next step is to search the district’s website for a point of contact and any information available about the district’s requirements for conducting external research. Some districts identified as being a special contact district from the previous cycle may be incorrect and staff will verify whether a given district has requirements for conducting external research before proceeding.

The following are examples of the type of information that will be gathered from each district’s website in order to prepare a research application for submission to this district:

Name and contact information for the district office or department that reviews applications to conduct external research, and the name and contact information of the person in charge of that office.
Information about review schedules and submission deadlines.
Whether application fees are required, and if so, how much.
Whether a district sponsor is required.
Whether an online application is required, and if so, the link to the application if possible.
Information about research topics and/or agenda on which the district is focusing.
The web link to the main research department or office website.
Research guidelines, instructions, application forms, District Action Plans, Strategic Plan or Goals, if any.

Recruitment staff will contact districts by phone and email to obtain key information not listed on the district’s website, (e.g., requirements for the research application, research application submission deadlines, etc.).

NTPS staff developed a generic research application that covers the information typically requested in district research applications. Staff will customize the generic research application to each district’s specific requirements that need to be addressed or included in the research application (e.g., how the study addresses key district goals, or inclusion of a district study sponsor), or submit the generic application with minimal changes to districts that do not have specific application requirements.

Using the information obtained from the district website or phone or email exchanges, a district research request packet will be prepared. Each research application will include the following documents, where applicable:

District research application cover letter;
Research application (district-specific or generic, as required by the district);
Study summary;
FAQ document;
Special contact district approval form;
Participant informed consent form (if required by the district);
NTPS Project Director’s resume;
Copy of questionnaires; and
Application fee (if required by the district).

Where applicable, applications will include drafts of the NTPS 2020-21 questionnaires provided in Appendix B of this submission, and the questionnaires from past cycles of NTPS will be provided to districts that request them. Other information about the study may be required by the district and will be included with the application or provided upon request.

Approximately one week after the application is submitted to the district (either electronically or in hard copy, as required by the district), NTPS district recruitment staff will contact the district’s research office to confirm receipt of the package and to ask when the district expects to review the research application and when a decision will be made. If additional information is requested by the district (e.g., the list of sampled schools), recruitment staff will follow up on such requests and will be available to answer any questions the district may have throughout the data collection period.

Some districts charge a fee (~$50-200) to process research application requests, which will be paid as necessary.

B.2.1.2 School Pre-Contact Letters

The school pre-contact letter is to verify school mailing addresses and to inform schools about the upcoming data collection. A letter is sent to each sampled school informing them of their selection for the study. About 1% of all school addresses get corrected by the U.S. Post Office in response to the pre-contact letter, saving time and effort during the actual data collection period.

B.2.2 School-level Data Collection Procedures

School-level data collection procedures for NTPS 2020-21 are summarized in Exhibit 1 (p. 15).

In July 2020, all schools will receive an advance letter addressed to the principal at the school address. The letter includes instructions for completing a brief screener interview online using the NTPS Screener internet instrument. Around the same time, principals for whom we have an email address will also be invited to complete the screener interview via email. The purpose of the screener interview is to determine the school’s eligibility for the NTPS and establish a survey coordinator. The survey coordinator will be asked to facilitate the completion of NTPS questionnaires within their school, and materials will be mailed to him or her throughout data collection. A reminder email will be sent to non-responding school principals in early August 2020. Principals who do not self-screen will be contacted by telephone in mid to late August 2020.

After the advance letter and screener interview, the data collection path – specifically, the timing and level of effort put forth to collecting a TLF from the school, which is needed to draw a sample for the Teacher Questionnaire – for each school will depend on a number of predetermined criterion. The data collection methodology employed will depend primarily on whether the school has acceptable teacher roster data available from the vendor. Schools without acceptable vendor data available do not have the vendor provided teacher roster to fall back on for the purposes of selecting a teacher sample, therefore it is important to put forth additional targeted effort and resources to obtaining a TLF from these schools.

Secondary data collection pathing will be made based upon the completion status and timing (if completed) of the school’s screener interview. Finally, data collection pathing may be made based on a school’s “priority status”. Prior to the start of NTPS 2020-21 data collection, a propensity model will be run to identify “priority” schools. The propensity model is based on a model developed for the NTPS 2015-16 and 2017-18 data collections. These “priority” schools have characteristics of schools from which it has been historically difficult to collect data and which have a potentially high impact on weighting. The priority flag takes into account both the response propensity and the base weight of a school to create a measure of a school’s potential effect on nonresponse weighting adjustments and final estimates. Schools with either an extremely high weight or an extremely low response propensity have a large response influence, meaning their nonresponse will disproportionately affect the nonresponse adjustment cell in which they are located. Thus, additional efforts are sometimes made to target data collection operations in these school early during data collection.

In September, all schools regardless of data collection path will receive an initial school package addressed to the survey coordinator at the school address. If a survey coordinator was not established during the screener interview, the package will be addressed to the principal at the school address. The package will contain a letter to the survey coordinator or principal, and three individually sealed envelopes that contain login information for completing the TLF, Principal Questionnaire, and School Questionnaire. Principals and survey coordinators will also be contacted by email around the same time the initial packages are mailed to the sampled schools. The emails will contain the appropriate hyperlinks and User IDs to complete the NTPS questionnaires online.

The timing of the mailout of this initial package will depend on whether the school has acceptable vendor data available and, for those schools without vendor data available, the timing of the school’s completion of the screener to establish a survey coordinator. For all schools WITHOUT vendor data and schools WITH vendor data who self-screened and established survey coordinator early (prior to the screener telephone operation), this initial package will be mailed in early September. These schools will be referred to as “early schools” throughout the remainder of this package. For the remaining schools WITH vendor data who did not self-screen early (late screeners or non-screeners), the initial package will be mailed approximately two weeks later. These schools will be referred to as “late schools” throughout the remainder of this package.

Following the initial mailout, all schools WITHOUT vendor data and “priority schools” WITH vendor data will receive a telephone call from an interviewer whose goal is to alert the principal or coordinator that a package has been mailed, confirm the school’s receipt of the package, and answer any questions from the school.

About three weeks after the initial mailout, a second package will be mailed to nonresponding schools. The package will include a reminder letter to the survey coordinator or principal and replacement materials for completing the outstanding questionnaires online. Principal and survey coordinator email addresses will be used as means of reminding nonresponding school staff to complete their questionnaires.

B.2.2.1 Schools without Vendor Data Available

Following the second mailout, schools without vendor data available will receive a personal visit from a Census Bureau Field Representative (FR), with the main goal of obtaining a completed TLF from the school. The FR will also distribute sealed letters containing login information for the school and principal questionnaires, as needed. If the FR notes that the school has shown reluctance or initially refused to participate in the study, the Regional Office of the FR will send out a “letter of better understanding” to help encourage participation. This operation will take place from late October through Thanksgiving.

Schools for which the personal visit is unsuccessful will receive a third reminder package in early January. This package will be mailed to the principal at the school address and will include a reminder letter, paper versions of the TLF, principal and/or school questionnaire(s), and postage-paid addressed return envelopes. Principal and survey coordinator email addresses will be used as means of reminding nonresponding schools to complete their questionnaires.

Beginning in late January, schools that have not yet completed their TLF, principal, and/or school questionnaire(s) will be sent to a telephone reminder operation aimed at reminding the survey coordinator or school principal to complete their questionnaires. Data collection for the TLF concludes following this telephone operation. If outstanding school and/or principal questionnaire(s) remain after this final mailing, one more attempt by mail (fourth and final school package), email, and telephone will be made to remind the school to complete their outstanding questionnaire(s).

B.2.2.2 Schools with Vendor Data Available

Following the second mailout, schools with vendor data available will receive a third reminder package in early November. This package will be mailed to the principal at the school address and will include a reminder letter, paper versions of the TLF and principal and/or school questionnaire(s) as needed, and postage-paid addressed return envelopes. Principal and survey coordinator email addresses will be used as means of reminding nonresponding schools to complete their questionnaires. Note that, while the timing of the initial and second school packages varies by early versus late schools down this path, the mailout schedule for the two groups converges starting with the third mailout.

Beginning in early December, schools that have not yet completed their TLF will be sent to a telephone reminder operation aimed at reminding the survey coordinator or school principal to complete their TLF and other outstanding school-level questionnaire(s). In early January, schools that have not provided or verified their TLF will have their teachers sampled from the vendor provided list of teachers. Beginning in late January, schools that have not yet completed their school and/or principal questionnaires will be sent to a telephone reminder operation aimed at reminding the survey coordinator or school principal to complete their questionnaires. If outstanding school-level forms remain after the telephone reminder operation, one more attempt by mail (fourth and final school package), email, telephone, and personal visit will be made to remind the school to complete their outstanding questionnaire(s).

Exhibit 1: 2020-21 National Teacher and Principal Survey – School-Level Data Collection Operation

B.2.3 Teacher Data Collection

Teachers will be sampled weekly from completed or verified TLFs throughout data collection. As teachers are sampled, they will be mailed an initial teacher package containing a letter that introduces the survey and provides the login information to complete their survey online. Around the same time, teachers for whom an email address is available will also be sent an email including the hyperlink and User ID to complete their teacher questionnaire online. If the school has a survey coordinator established, the individually-sealed teacher packages will be sent to the survey coordinator, at the school address, with a cover letter. If the school does not have a survey coordinator established, the teacher packages will be mailed individually to the sampled teachers at the school address in most cases. Exceptions may be made to this for late sampled teachers whose materials may be mailed directly to their school’s principal to distribute.

If the school’s teachers were sampled from a vendor or clerical list (where the school did not complete or verify a TLF), materials for the sampled teachers to complete their teacher questionnaires will be mailed directly to the teachers at their school address regardless of whether a survey coordinator was established. Exceptions may be made to this for late sampled teachers whose materials may be mailed directly to their school’s survey coordinator (when there is one established) or the principal to distribute.

Teachers with a valid email address will be sent an email containing the hyperlink to the online Teacher Questionnaire and their User ID about one week after their initial mailout.

Each sampled teacher will receive as many as three reminder packages to complete their outstanding Teacher Questionnaire. Each teacher mailing will be accompanied by an email to the teacher about a week after the mailing. The first reminder letter will contain the login information for the Teacher Questionnaire (URL and User ID) and will be sent to the survey coordinator (if applicable). The second and third reminder packages will include a letter and a paper questionnaire and will be addressed directly to the sampled teachers at the school address, regardless of whether the school has a survey coordinator established.

A contingency plan will be included in the NTPS 2020-21 and will be executed in the fourth teacher mailing as needed based on monitoring data collection status. The contingency incentive will be administered across the board for teachers in the agreed upon at-risk domains rather than experimentally, since an experiment was conducted during the NTPS 2017-18.

At the time the contingency incentive is activated, some teachers at the school will have already responded to NTPS. These teachers will be provided a letter thanking them for their participation in the NTPS, along with a contingency “thank you” incentive. The sampled non-responding teachers will receive a letter requesting their participation in the NTPS via a paper questionnaire and return envelope, along with a contingency incentive. All teachers will receive the same cash amount at the time of the contingency plan inclusion.

Both monetary (cash) and non-monetary contingency incentives will be offered to teachers. Specifically, teachers who received either the non-monetary incentive (tote) or no incentive in the first mailing will receive a monetary (cash) incentive as their contingency incentive. Teachers who received a monetary (cash) incentive in the initial teacher mailing will receive a non-monetary contingency incentive (tote).

Beginning in late January 2021, telephone interviewers will contact survey coordinators to ask them to remind their schools’ sampled teachers to complete their questionnaires. Telephone interviewers and/or Field Representatives will contact nonresponding teachers by phone or during an in-person visit from late February through May 2021.

B.3 Methods to Secure Cooperation, Maximize Response Rates, and Deal with Nonresponse

This section describes the methods that NCES will use to secure cooperation, maximize response rates, and deal with nonresponse for NTPS 2020-21. Section B.3.1 details how NTPS plans to secure cooperation by leveraging its status as the primary source of information on K-12 schools and staffing in the United States. Section B.3.2 describes the methods that will be used to minimize nonresponse. The design is based on the results from the NTPS 2017-18, which employed a number of different contact strategies aimed at boosting response rates.

B.3.1 Methods to Secure Cooperation and Maximize Response Rates

The entire survey process, starting with securing research cooperation from key public school groups and individual sample members and continuing throughout the distribution and collection of individual questionnaires, is designed to increase survey response rates. In addition, the following elements of the data collection plan, in particular, will contribute to overall success of the survey and will enhance the survey response rates.

Visible support from top-level Federal, State, and local education officials. Without the support of high-level officials in the U.S. Department of Education, State Education Agencies, and the sampled local school districts, surveys of public school principals and teachers cannot be successfully implemented. Obtaining endorsements from these officials is a critical factor in the success of the data collection procedures. Top-level Education Department officials will need to fully support the data collection by endorsing the survey in writing and sending advance letters and notices to sampled districts that require prior research applications and to individual survey participants (principals and teachers) to encourage participation.
Endorsements from key public school groups and affiliations. The level of interest and cooperation demonstrated by key groups can often greatly influence the degree of participation of survey respondents. Endorsements are viewed as a critical factor in soliciting cooperation from state and local education officials. NCES will seek endorsements for NTPS 2020-21 from the following national organizations or agencies:

American Association of School Administrators

American Counseling Association

American Association of School Librarians

American Federation of Teachers

American Montessori Society

American School Counselors Association

Association for Middle Level Education (formerly National Middle School Association)

Association for Supervision and Curriculum Development

Association of American Educators

Council of Chief State School Officers

Council of the Great City Schools

National Association of Elementary School Principals

National Association of Secondary School Principals

National Education Association

National Parent Teacher Association

The School Superintendents Association

In addition, NCES will seek endorsement for NTPS 2020-21 from the following state organizations and agencies:

Alabama

Birmingham Federation of Teachers

Alabama Education Association

Alaska

Alaska Public Employees Association

Alaska Education Association

Arizona

American Federation of Teachers - Arizona

Arizona Education Association

Arkansas

Arkansas Education Association

California

American Federation of Teachers - California

California Teachers Association

Colorado

American Federation of Teachers - Colorado

Colorado Education Association

Connecticut

American Federation of Teachers - Connecticut

Connecticut Education Association

Delaware

Delaware State Education Association

District of Columbia

Washington Teachers' Union

Florida

Florida Education Association

Georgia

American Federation of Teachers - Georgia

Georgia Association of Educators

Hawaii

Hawaii State Teachers Association

Idaho

Idaho Education Association

Illinois

American Federation of Teachers - Illinois

Illinois Education Association

Indiana

American Federation of Teachers - Indiana

Indiana State Teachers Association

Iowa

Iowa State Education Association

Kansas

American Federation of Teachers - Kansas

Kansas National Education Association

Kentucky

Kentucky Education Association

Louisiana

American Federation of Teachers - Louisiana

Louisiana Association of Educators

Maine

Maine State Employee Association

Maine Education Association

Maryland

American Federation of Teachers - Maryland

Maryland State Education Association

Massachusetts

American Federation of Teachers - Massachusetts

Massachusetts Teachers Association

Michigan

American Federation of Teachers - Michigan

Michigan Education Association

Minnesota

Education Minnesota

Mississippi

American Federation of Teachers - Mississippi

Mississippi Association of Educators

Missouri

American Federation of Teachers - Missouri

Missouri Education Association

Montana

Montana Federation of Public Employees

Nebraska

Nebraska State Education Association

Nevada

Nevada State Education Association

New Hampshire

American Federation of Teachers - New Hampshire

New Hampshire Education Association

New Jersey

American Federation of Teachers - New Jersey

New Jersey Education Association

New Mexico

American Federation of Teachers - New Mexico

New Mexico Education Association

New York

New York State United Teachers

North Carolina

North Carolina Association of Educators

North Dakota

North Dakota United

Ohio

Ohio Federation of Teachers

Ohio Education Association

Oklahoma

American Federation of Teachers - Oklahoma

Oklahoma Education Association

Oregon

American Federation of Teachers - Oregon

Oregon Education Association

Pennsylvania

American Federation of Teachers - Pennsylvania

Pennsylvania State Education Association

Rhode Island

Rhode Island Federation of Teacher and Healthcare Professionals

Rhode Island Education Association

South Carolina

South Carolina Education Association

South Dakota

South Dakota Education Association

Tennessee

Tennessee Education Association

Texas

American Federation of Teachers - Texas

Texas State Teachers Association

Utah

American Federation of Teachers - Utah

Utah Education Association

Vermont

American Federation of Teachers - Vermont

Vermont Education Association

Virginia

Fairfax County Federation of Teachers

Virginia Education Association

Washington

American Federation of Teachers - Washington

Washington Education Association

West Virginia

American Federation of Teachers - West Virginia

West Virginia Education Association

Wisconsin

American Federation of Teachers - Wisconsin

Wisconsin Education Association Council

Wyoming

Wyoming Education Association

The NTPS 2020-21 is the first cycle soliciting endorsement from state organizations and agencies. The number of state organization and agencies will be capped at two per state for efficiency of solicitation operations.

Endorsements from key private school groups. In addition to the endorsements from key public school groups, NCES will also seek endorsements for NTPS 2020-21 from the following private school organizations or agencies:

Agudath Israel of America/Lefkowitz Leadership Initiative

American Association of Christian Schools

American Association of School Librarians

American Counseling Association

American Federation of Teachers

American Montessori Society

American School Counselors Association

Association for Middle Level Education

Association for Supervision and Curriculum Development

Association Montessori International

Association of American Educators

Association of Christian Schools International

Association of Christian Teachers and Schools

Association of Classical Christian Schools

Association of Military Colleges and Schools

Association of Waldorf Schools of North America

Christian Schools International

Council for American Private Education

Council of Chief State School Officers

Council of Islamic Schools of North America

Council of the Great City Schools

Evangelical Lutheran Church in America

Friends Council on Education

General Conference of Seventh-Day Adventists

Islamic School League of America

Jesuit Schools Network

Jewish Education Services of North America

Lutheran Church-Missouri Synod

National Association of Elementary School Principals

National Association of Episcopal Schools

National Association of Independent Schools

National Association of Private Special Education Centers

National Association of Secondary School Principals

National Catholic Educational Association

National Christian School Association

National Coalition of Girls’ Schools

National Council for Private School Accreditation

National Education Association

National Independent Private School Association

National Parent Teacher Association

Office of Education, General Conference of Seventh Day Adventists

Oral Roberts University Educational Fellowship

Prizmah: Center for Jewish Day Schools

RAVSAK: Jewish Community Day School Network

Southern Baptist Association of Christian Schools

The Association of Boarding Schools

The Jewish Education Project

The School Superintendents Association

Torah Umesorah National Society for Hebrew Day Schools

United States Conference of Catholic Bishops

Wisconsin Evangelical Lutheran Synod

Stressing the importance of the survey and the respondents' participation. Official letters will be used to motivate respondents to return surveys. NTPS 2020-21 respondent letters will be sent by the U.S. Census Bureau and signed by the NCES Commissioner. Communications in the form of both letters and emails will be personalized for the principal and survey coordinators, whenever possible, which is expected to have positive effects on the survey response rates.

B.3.2 Methods to Minimize Nonresponse

A major challenge in any survey is obtaining high response rates, and this is even more important today when response rates have been falling among federal surveys, including NTPS.

The main problem associated with nonresponse is the potential for nonresponse bias in the estimates produced using data collected from nonrespondents. Bias can occur when respondents are systematically different from nonrespondents. Two approaches that will be used to reduce the potential for bias are designing the data collection procedures and methods wisely to reduce nonresponse (e.g., establishing survey coordinators) and using statistical methods of sampling and weighting to reduce the effect of nonresponse on the estimates. While the statistical approaches are important in controlling biases and costs, the data collection procedures and methods are at the heart of a successful study.

Methods selected to minimize nonresponse in NTPS 2020-21 will build upon those used in NTPS 2017-18, including actions that were taken late in the data collection to boost principal and teacher response rates.

Data Collection Strategies to Minimize Non-Response

Minimize survey burden on schools. NTPS survey procedures are designed to minimize the burden on schools and sampled individuals (principals and teachers), and the survey instruments have been designed to be completed as quickly and easily as possible.

To reduce burden on schools the TLF (both the electronic version in the NTPS Respondent Portal and the paper TLF) will be pre-populated with vendor teacher roster data, and the school will be asked to verify the teacher information rather than provide it from scratch. A small proportion of sampled schools with vendor teacher data available will be provided a blank TLF (no vendor data prepopulated) as part of an experiment aimed at assessing the quality and burden tradeoffs of offering schools a prepopulated TLF. See section B.4.2.1 for additional details.

Questionnaire design techniques have been employed to minimize item nonresponse. Questionnaires from previous rounds of SASS and NTPS were carefully analyzed to determine which items had the highest levels of item nonresponse. This information guided NCES in reviewing the clarity of item wording, definitions, and instructions. Items that were not considered to be effective or useful were removed from the survey to streamline the questionnaires and ease the response burden.

A key design feature of NTPS is the ability to link to other NCES collections such as EDFacts and the Civil Rights Data Collection (CRDC). Information from these sources will be incorporated into final datasets to allow researchers and policymakers to analyze those data together. This will further reduce the need to collect from schools data that have already been collected from state or district education agencies.

Recruit survey coordinators. Successive administrations of SASS and NTPS have shown that an important procedure to help maximize response rates is to establish a school-based "survey coordinator" to serve as a primary point of contact for NTPS staff. The use of a survey coordinator is expected to help keep response rates high, provide some minimal data quality checks, and simplify the follow-up process by having one point of contact.
Use vendor lists for teacher sampling. NTPS teacher-level response rates are calculated by multiplying response at the school level to the TLF by response at the teacher level. In the past, this has meant that if the school did not complete the TLF, teachers from that school could not be sampled, ultimately lowering the teacher response rate. One goal in NTPS 2015-16 and 2017-18 was to improve the overall teacher response rate by allowing NTPS to sample teachers from schools that have not submitted a TLF; therefore, TLFs received from sample schools were supplemented with vendor-purchased teacher lists or clerically researched teacher lists when vendor data were not available. The vendor and clerically-researched lists were evaluated in NTPS 2017-18, NTPS 2015-16, and the NTPS 2014 pilot test and showed high levels of comparability to lists obtained directly from schools.

In NTPS 2020-21 TLFs will once again be pre-populated with vendor-purchased teacher lists and those obtained through a clerical look-up operation utilizing school and district websites, and schools will be asked to verify the teacher information rather than provide it from scratch. A small proportion of sampled schools with teacher list data available will be provided a blank TLF (no vendor or clerical data prepopulated) as part of a methodological experiment; see section B.4.2.1 for additional details. However, the approach of offering respondents pre-populated TLFs is expected to help improve the overall teacher response rate and allow teacher sampling in schools that have not submitted a TLF as a last-ditch effort to collect data in such schools.

As with past cycles of NTPS, the vendor and clerically-researched teacher lists will be evaluated against school-reported data to assess the quality of the supplemental data.

Tailor nonresponse follow up strategies. As in NTPS 2017-18, schools sampled for NTPS 2020-21 will be assigned a “priority” flag based on the weighted response influence of the case. The weighted response influence takes into account both the response propensity and the base weight of a school to create a measure of a school’s potential effect on nonresponse weighting adjustments and final estimates. The priority flag was assigned at the school level in NTPS 2017-18 and the same will be done for NTPS 2020-21. During data collection, the priority flag will be used to target data collection efforts in the hopes of boosting response rates.

In addition, mail method (USPS versus FedEx) for the initial school package will be based upon school characteristic and screener outcome. As a reminder, “early schools” are those schools with no vendor data available and schools WITH vendor data available that self-screened early and established a coordinator. “Late schools” are those schools WITH vendor data that did not self-screen early (late screeners or non-screeners). All schools with no vendor data available (Early) and Late schools will be mailed via FedEx. Both of these categories of schools are the harder to reach or less likely to respond schools and, therefore, additional resources will be expended for the initial packages to stand out to school staff. All schools WITH vendor data available that self-screened and have a coordinator (Early) will be mailed via USPS; these schools are eager and likely to respond and, therefore, additional resources will not be expended to make the initial packages stand out.

NTPS focuses on obtaining cooperation and improving response rates at the school level for a number of reasons. Past administrations of SASS and NTPS have shown that when cooperation is obtained at the school level, teachers and principals are more likely to respond.

Additionally, evaluations of schools’ response propensities have shown that schools in special contact districts are the primary driving force behind low response propensity. Special districts are those that require additional applications or documentation to collect data in their schools. Nearly 80% of the schools with high propensity for non-response reside in these special districts. For this reason, resources will continue to be allocated to focus on obtaining approvals from special contact districts in order to boost response rates for this group.

Monitor publishability and bias measures. For NTPS 2017-18, NCES monitored data collection progress throughout survey operations in order to identify and potentially minimize problems with nonresponse. The Census Bureau created weekly “publishability” reports from their data collection tracking system that showed whether key analysis cells were large enough to provide publishable estimates as of that point in time. By monitoring this publishability metric, NCES was able to identify populations of schools for which nonresponse could hamper reporting. As a result, a sub-population of teachers (working in city or charter schools) with outstanding questionnaires were sent to a final follow-up operation, which ultimately lead to meeting publishability standards for those subpopulations. Please refer to “Telephone and field follow-up operations for late-sampled teachers” below for additional details.

The NTPS 2017-18 included a contingency plan with monetary boosts to combat low response from these populations of schools identified by the publishability metric. The contingency plan was executed and the results indicated that the contingency incentive significantly increased the response rate within the selected contingency incentive domains for public school teachers. Therefore, a contingency plan will also be included in the NTPS 2020-21 and will be executed in the fourth mailing as needed based on monitoring data collection status. The contingency incentive will be administered across the board for teachers in the agreed upon at-risk domains rather than experimentally, since an experiment was conducted during the NTPS 2017-18.

In addition, the results from monitoring the weekly publishability metric will be considered in designing the sample and nonresponse follow-up strategies for NTPS 2020-21.

During the NTPS 2017-18, NCES also monitored R-indicators, a measure of representativeness, or lack of bias in the respondent population, on a weekly basis. The closer the R-indicator is to 1, the more balanced is the respondent population. Towards the end of data collection in 2017-18, the R-indicator for the full sample indicated that the respondent population was fairly well balanced. NCES plans to continue to monitor these two indicators in NTPS 2020-21.

Personalize principal contact materials. As was done in NTPS 2017-18, to maximize the chances that all mailed NTPS 2020-21 materials intended for the school principal successfully make it to the principal, all principal contact materials will be personalized with the principal’s name whenever possible. Principals’ names are obtained from vendor-purchased school staff lists. If a principal’s name is not available from the vendor, clerical staff research this information using school and district websites.
Use of email to target principals, survey coordinators, and teachers. NTPS 2017-18 demonstrated that email was an effective tool to drive participation in both the NTPS teacher and principal surveys. It proved that teacher email addresses could be effectively collected on the TLF, school websites, and from vendor lists of teachers; that principal email addresses could be effectively collected from school websites and from vendor purchased school data; and that survey coordinator email addresses could be effectively collected during the screener interview. Because personalized emails carry minimal cost and may help boost response, throughout 2020-21 NTPS data collection, teachers, principals, and survey coordinators will be contacted via email. The emails will include login information to access the NTPS online survey instruments, in addition to text inviting and subsequently reminding these respondents to complete their survey online.
Use of additional reminder emails to teachers. Previous NTPS cycles showed that response rates for late-sample wave teachers in NTPS appeared to be lower than for earlier waves of teachers. This may have been a product of the timing of school testing and late-school year activities because late-sample wave teachers received an invitation to complete the survey during a period with a heavy school workload. It may also have been because the late sampled teachers were in schools that were either late responders to the TLF or TLF non-respondents (in instances where teachers were sampled from a teacher roster obtained from clerical research or the vendor data) and therefore may have had less support and encouragement from their principals and/or survey coordinators to complete their questionnaires. Given that additional reminder emails carry low cost and may help response rates, a total of 5 possible reminder emails will be sent to nonresponding teachers during NTPS 2020-21 data collection. There will be one email that coincides with each of the 4 teacher mailings and one final email after the fourth and final teacher mailing.
Send a “letter of better understanding” to principals and teachers. After the 2015-16 NTPS collection, field representatives and the regional offices recommended to send “letters of better understanding” to principals and teachers who may be hesitant to complete the survey to help them gain a better understanding of the study by providing them information about how the data are used and referencing some of the published data from NTPS First Look Reports. These letters will be sent to principals, teachers, and other staff as needed in priority schools, which tend to exhibit high non-response.
Telephone and field follow-up operations for late-sampled teachers. NTPS 2017-18 included two additional follow-up operations aimed at collecting completed questionnaires from nonresponding teachers sampled in the later data collection waves (17-20). In previous NTPS and SASS cycles, late-sampled teachers were not eligible for inclusion in telephone follow-up and/or field follow-up operations. During the phase 2 telephone follow-up operation for the NTPS 2017-18, telephone center staff made telephone calls to late-sampled teachers to remind them to complete their questionnaire and, whenever possible, collect the interview over the phone. During the phase 4 field operation, Field Representatives made personal visits to the schools to drop off the paper form(s) and schedule a time to pick up the completed forms. Additionally, both of these operations targeted domains with publishability risks (e.g. teachers in city and charter schools). The NTPS 2020-21 includes a planned Phase 2 Telephone Follow-up Operation for which most late sampled teachers will be eligible and a Phase 3 Field Follow-up Operation to ensure that all sampled teachers receive non-response follow-up by at least telephone or Field (or possibly both).
Consider new methods of minimizing nonresponse. NCES is considering a number of additional methods to minimize nonresponse in NTPS 2020-21, including the continued use of incentives. Previously, monetary (prepaid cash) incentives were the main forms of incentives used to minimize nonresponse. The results of NTPS 2017-18 indicated that the teacher incentive significantly increased the final response rates for both public and private school teacher questionnaires. Due to these favorable results, the NTPS 2020-21 will include the use of incentives, once again, with the goal of maximizing overall teacher response. Two types of incentives will be offered to teachers in an experimental manner – a monetary incentive (prepaid cash) and a non-monetary incentive. Teachers in the experimental treatment will receive a branded canvas tote bag at the first contact by mail. This treatment will be evaluated against the control group, which will receive a prepaid cash monetary incentive, as was done in NTPS 2017-18.

For NTPS 2020-21, non-monetary incentives at the school level are being include for a small subsample of schools as a tool to encourage participation and further increase response rates. Further information about incentives are provided below in section B.4.2.

Statistical Approaches to Nonresponse

One of the methods employed to reduce the potential for nonresponse bias is adjustment of the sample weights to account for nonresponse. If schools or teachers with certain characteristics are systematically less likely than others to respond to a survey, the collected data may not accurately reflect the characteristics and experiences of the nonrespondents, which can lead to bias. To adjust for this, respondents are assigned weights that, when applied, result in them representing their own characteristics and experiences as well as those of nonrespondents with similar attributes. The school weights are also raked to sampled-based control totals in order to maintain the background characteristics of the sample. This is another method used to reduce the potential for nonresponse bias in the estimates produced from the data.

Response rates will be computed for the TLF, the School Questionnaire, the Principal Questionnaire, and the Teacher Questionnaire. Data collected through any instrument with a response rate of less than 85 percent will be evaluated for nonresponse bias. In addition to comparing the characteristics of respondents and nonrespondents using data that are available from the sampling frames (for example, school type and school locale from the school frame), the NTPS 2020-21 estimates will be compared to estimates from previous rounds of NTPS. A methodology report covering NTPS 2020-21 will be developed and released, describing the methods and results of the nonresponse bias analysis.

B.4 Tests of Methods and Procedures

The SASS/NTPS series of studies has a long history of testing materials, methods, and procedures to improve the quality of its data. Section B.4.1 describes those tests that have most influenced the NTPS design, beginning with the 2014-15 NTPS Pilot Test and continuing through NTPS 2017-18. Section B.4.2 describes experiments proposed for NTPS 2020-21.

B.4.1 Tests Influencing the Design of NTPS 2020-21

2014-15 NTPS Pilot Test

Five experiments designed to optimize the design of the 2015-16 NTPS were conducted as part of the 2014-15 NTPS Pilot Test: 1) the Questionnaire Mode Experiment, 2) the TLF Email Experiment, 3) the Invitation Mode Experiment, 4) the Teacher Questionnaire Instruction Experiment, and 5) the Vendor Analysis. Each of these experiments is briefly described below, along with its results and implications for successor NTPS data collections.

Questionnaire Mode Experiment. This experiment was designed to determine whether paper questionnaires or Internet survey instruments (i.e., mail‐only versus internet sequential modes) constituted the most effective mode of collecting the TLF, School Questionnaire, and Principal Questionnaire. For all three survey instruments, the schools assigned to the paper mode had higher response rates than the schools assigned to the internet mode.

Some known issues with data collection could have impacted these response rates. First, the pilot test did not use survey coordinators, a method shown to boost response rates in SASS. Second, there were problems related to the contact materials for the internet treatment groups. As a result of this experiment, NTPS 2015-16 was primarily paper based; used improved contact materials and login procedures; and included an experimental sample of 1,000 schools, outside the main study, which were offered Internet survey at the onset of data collection and which followed standard production NTPS procedures, including the establishment of a survey coordinator.

Teacher Listing Form (TLF) Email Experiment. This experiment was designed to assess the feasibility of collecting teacher email addresses on the TLF and the quality of those collected. The pilot test design included a split-panel experiment, with half of sampled schools randomly assigned to receive a TLF that included a request for teachers’ email addresses and the other half to receive a TLF that did not request email addresses. At the end of data collection, response rates were comparable between the schools that received the TLF with the email address field and the schools that received the TLF without the email address field. As a result of this experiment and the Invitation Mode Experiment described below, NCES used the TLF with the email address field in NTPS 2015-16 and 2017-18.
Invitation Mode Experiment. The purpose of this experiment was to identify which of three methods of inviting teachers to complete the Teacher Questionnaire yielded the best response rates. Schools were randomly assigned to the following invitation modes: 1) both email and mailed paper invitation letters to complete the internet instrument (treatment A), 2) a mailed paper invitation letter to complete the internet instrument only (treatment B), and 3) a mailed package that included a letter and paper questionnaire (treatment C). The results of the experiment indicated that a strategy using a combination of email and paper invitations (treatment A) is best for inviting teachers to complete the internet questionnaire. The response rate for treatment group A was comparable to that of treatment group C that received only mailed paper materials. As a result of this experiment, teachers sampled for NTPS 2015-16 for whom we had a valid email address were sent both email and paper invitations as the initial request to fill out the Teacher Questionnaire. Teachers without valid email addresses were sent their initial invitation as part of a mailed package that included a paper copy of the survey. For the NTPS 2017-18, NCES encouraged web response with both mailed and emailed correspondence, switching to a paper questionnaire at the third mailing.
Teacher Questionnaire Instruction Experiment. This experiment was designed to determine (1) whether including instructions in the NTPS questionnaire impacts response rates for questionnaire items and data quality, and (2) whether the position, format, and presence or absence of a preface in the instruction impacts response rates for questionnaire items. Production questions and instructions, which were the product of production cognitive interviewing, were selected from the 2014-15 National Teacher and Principal Survey (NTPS). In addition, a second set questions and instructions were intentionally created to counter teachers’ natural conceptions of terms. Both sets of questions were compared to a control group with no instructions. Utilizing a factorial experiment design, we three factors varied that were predicted to alter the effectiveness of instructions: their location, format, and the presence or absence of a preface. The NTPS questions with instructions, which were the result of production cognitive interviews, increased the length of the questionnaire with no measureable improvement in data quality compared to control questions with no instructions, whereas the experimental questions with instructions meant to counter teachers’ natural conceptions of terms improved data quality by changing responses in the expected direction. Due to the lack of differences for NTPS production questions, no major changes were made to instruction position, format, or introduction in subsequent administrations of NTPS.
Vendor Analysis. The purpose of this experiment was to evaluate both the feasibility of collecting teacher lists from a vendor and the reliability of the purchased information to see whether it could be used to supplement or replace school-collected TLFs. NCES purchased teacher lists from a vendor for schools sampled for the 2014-15 NTPS pilot test. The vendor teacher lists were compared with information collected from the TLFs. The results suggested that the vendor list information was comprehensive and reliable at a relatively low cost. NCES used vendor lists to sample teachers from a subset of schools that did not respond to the TLF in NTPS 2015-16 and 2017-18.

NTPS 2015-16 Full-Scale Collection

Schools and Principals Internet Test. The 2015-16 NTPS included an Internet experiment for schools and principals, which was designed to test the efficacy of offering an internet response option as the initial mode of data collection, as done previously in the Questionnaire Mode Experiment included in the 2014-15 NTPS Pilot Study, described earlier.

Key differences exist between the 2014-15 and 2015-16 NTPS internet experiments, with the most notable being that the 2015-16 experiment included the use of a survey coordinator at the school, and improved respondent contact materials and mailout packaging. In the 2015-16 NTPS, an independent sample of 1,000 public schools was selected for this experiment, which invited schools and principals to complete the NTPS school-level questionnaires using the internet at the first and second contacts by mail. A clerical operation prior to data collection obtained email addresses for sampled principals assigned to the internet treatment. Principals were sent emails as an initial mode of invitation to complete the NTPS questionnaires as well as reminder emails; the timing of these emails was a few days following the mailings.

Paper questionnaires were offered at the third and final mailout. Data collection for the internet treatment concluded after the third mailing, so the schools in the experimental treatment did not receive a fourth mailing and were not included in the telephone follow-up or field follow-up operations. When comparing the response rates for all three survey instruments at the end of the reminder telephone operation – the most reasonable time to make the comparison – and removing the cases that would have qualified for the early field operation, the response rates for schools assigned to the internet treatment are five to six percentage points higher than those for the paper treatment. Therefore, the initial mailout invited respondents to complete online questionnaires during the NTPS 2017-18 data collection for all questionnaire types. Paper questionnaires were introduced during the third mailing. Principal email addresses (purchased from the vendor) and school-based survey coordinator email addresses (collected at the time the survey coordinator is established) were utilized during data collection. Invitations to complete the principal and school questionnaires via the Internet response option were sent to the principal and school-based survey coordinator by email in conjunction with the various mailings.

Contact Time Tailoring Experiment. This test was designed to determine the optimal contact time for teachers. During the telephone nonresponse follow-up operation, interviewers contacted nonresponding principals and teachers to remind them to complete their questionnaire. Teachers tend to be difficult to reach during the school day due to their teaching schedules. NCES staff hypothesized that teachers may be easier to reach by phone in the late afternoon, when school had been dismissed. To test the accuracy of this theory, an experiment was embedded in the telephone nonresponse follow-up operation. A portion of the NRFU teacher workload received an experimental treatment, where they were intended to be contacted only in the afternoon between 2:00 p.m. and 5:00 p.m. (respondent time). The remainder of the NRFU teacher universe functioned as the control group. These teachers were intended to receive contacts throughout the school day, per typical telephone follow-up procedures. The research questions this test was designed to answer were as follows:

Are afternoons more productive for calling teachers?
If not afternoons, are there more productive times than others for calling teachers?
Do productive contact times for teachers hold globally, or do different types of schools have different productive call time frames?
Can we use school-level frame information (e.g. urbanicity, school size, grade level) to help tailor call times in future rounds of data collection?
If the calls are being made at “productive times,” are fewer call attempts required to successfully make contact with the teacher?
If the calls are being made at “productive times,” are fewer call attempts and total contacts required to obtain a completed interview?

Operational challenges in conducting the call time experiment were encountered. Early in the telephone nonresponse follow-up operation, telephone interviewers reported that school staff members were complaining about receiving multiple calls to reach the sampled teachers. School staff members indicated that they would prefer to know the names of the teachers the interviewer needed to reach so that they could assist the interviewer in as few phone calls as possible. As a result, the results of the experiment could not be evaluated as intended. Instead of comparing the success of reaching the sampled teachers by their treatment group, staff compared the success rates of the actual call times. Call times were categorized as ‘early’ (before 2:00 p.m.) or ‘late’ (between 2:00 p.m. and 5:00 p.m.). There was not a noticeable difference in the success rates of contacting teachers by call time. Additional analyses on the data may be conducted to help inform future administrations of NTPS.

NTPS 2017-18 Full-Scale Collection

To address declining response rates among teachers in NTPS 2015-16, NCES tested the use of incentives to increase response in NTPS 2017-18. In addition, NTPS 2017-18 included a private school test that was designed to (a) provide accurate estimates for teachers and principals in private schools in the U.S. and (b) to examine the effects of strategies to improve response in this population. Each of these experiments is briefly described below, along with its results and implications for successor NTPS data collections.

Testing the use of teacher incentives. The NTPS 2017-18 included an incentive experiment designed to examine the effectiveness of offering teachers a monetary incentive to boost overall teacher response. Teachers were incentivized during the first 12 waves of teacher sampling (“phase one incentive experiment”), then a combination of teachers and/or school coordinators or principals were incentivized during the remaining waves (“phase two incentive experiment”). During the first 12 waves of the teacher sampling, teachers were only sampled from returned TLFs. However, beginning in wave 13 for schools, teachers could be sampled from returned TLFs, vendor lists, or internet look-ups. This change in the teacher sampling procedures provided a natural breakpoint between the two phases of the experiment and allowed us to target the most challenging cases with an additional incentive for the school coordinator or principal.

The results of phase one of the incentive experiment indicated that the teacher incentive led to significant increases in the response rate for both public and private school teachers. In addition, the average number of days to complete the questionnaire was significantly lower for public school teachers that received the incentive. Finally, the incentive helped increase the overall sample balance for teachers in both public and private schools.

The results of phase two of the incentive experiment indicated that the additive effect of the school coordinator incentive (in addition to the teacher incentive) was negligible for both public and private school teachers. In addition, the average number of days to complete the teacher questionnaire was significantly lower for the treatment group that received both incentives when compared to the treatment groups that did not receive a teacher incentive (with or without the school coordinator incentive) for both public and private school teachers. Given these results, teachers will be offered an incentive for the NTPS 2020-21.

Testing the use of incentives as part of a contingency plan. NTPS 2017-18 experimented with offering an incentive to teachers if they belonged to a domain that was determined to be ‘at-risk’ of not meeting NCES publishability standards towards the end of data collection (by February 12, 2018). NCES monitored actual and expected response in each of the key domains on a weekly basis. The contingency plan was to be activated in the experimental group only if needed and, based on publishability reports, it was deemed needed and was activated. The control group was not eligible to receive the contingency incentive. While the plan was aimed at improving teacher response rates, because teachers within a school were likely to discuss the study, schools were selected based on meeting criteria of the domain at risk and all teachers within the school were subject to the same treatment (experimental or control). This approach was based on the assumption that if some teachers in the school received an incentive and others did not, it would negatively impact current and future response from that school. At the time the incentive was activated, some teachers at the school have already responded to NTPS – such teachers, if assigned to the contingency incentive treatment, were provided the incentive as a “thank you” for their participation. For all other teachers in the school, the same incentive was prepaid and not conditional on their response. Given that schools selected for the contingency plan incentive were based on the number of teachers in the at-risk domain, selection for this incentive was independent of the main NTPS incentive experiment. Consistent with the other NTPS 2017-18 procedures, the incentive amount varied between priority and non-priority schools. Teachers in selected non-priority schools received $10 with their third mail-out or thank-you letter, and teachers in selected priority schools received $20 with their third mail-out or thank-you letter.

The contingency plan was executed, and the results indicated that, overall, the contingency incentive significantly increased the response rate within the selected contingency incentive domains for public school teachers. Therefore, a contingency plan will also be included in the NTPS 2020-21 and will be executed in the fourth teacher mailing as needed based on monitoring data collection status. The contingency incentive will be administered across the board for teachers in the agreed upon at-risk domains rather than experimentally, since an experiment was conducted during the NTPS 2017-18.

Private School Test. In NTPS 2017-18, NCES conducted an embedded test with private schools both to determine whether sufficient response could be achieved to provide reliable estimates for private schools and to evaluate specific methods for improving response rates. The private schools selected for this test experienced data collection procedures that were generally similar to those used with the NTPS 2017-18 public school sample. Some procedures were adjusted to accommodate differences specific to this sector (e.g., religious holidays and schedules). Results indicate that the private school data collected during NTPS 2017-18 will yield publishable estimates; therefore, private schools will be included in the NTPS 2020-21 sample.

Within the private school test was a secondary test, where a tailored contact strategy was employed for a subsample of “priority schools”. A propensity score model was used to identify and segment priority schools. The highest priority schools for the collection are those with the lowest likelihood of response and the highest likelihood to contribute to bias. In order to assign schools into treatment groups, schools were matched into pairs with similar likelihood scores and then randomly assigned to groups (“priority” early contact schedule versus “non-priority” typical contact schedule). Because the priority school data collection plan was resource intensive and was not necessary for some schools (e.g., schools with a high likelihood of response), the tailored contact strategy was tested with 60 percent of the sample, based on the highest priority cases as identified by the propensity models. Once they were matched into pairs, half of the schools in the test group (30 percent of schools in the starting sample) were assigned to the treatment group (“priority”), and the other half of the schools (30 percent) were assigned to the comparison group (“non-priority”). The remaining 40 percent of the starting sample received the typical contact schedule for the non-priority schools.

Preliminary results from the tailored contact strategy test show that the tailored contact strategy (with data collection starting with in-person visits from Census Bureau FRs) was not effective for the private priority schools.The response rate for the priority schools treated as non-priority schools for the school and principal questionnaires were not statistically different from those of the priority schools assigned to the priority school treatment.

Coordinated special district operations. NCES conducts several school-based studies within the NCES legislative mandate to report on the condition of education including, among others, NTPS, the Survey of School Crime and Safety (SSOCS), and the National Assessment of Educational Progress (NAEP). A critical step for data collection is to obtain approval from public school districts that require it before a study can be conducted with students, teachers, and/or staff. The number of such special contact districts is steadily increasing. This poses a barrier to successful data collection, because many districts and schools have complex and lengthy approval processes, reject all outside research, or only review applications for outside research once a year. This has contributed to lower response rates for non-mandatory NCES surveys. NCES continues to examine how different program areas, both within NCES and in other federal agencies, seek approval from PreK-12 public districts and schools in order to identify best practices and make recommendations for current and future operations.

To reduce burden for the special contact districts and improve operational efficiency, NCES sought research approval simultaneously for NTPS 2017-18 and SSOCS 2018. Although NCES minimized overlap in the schools sampled for NTPS and SSOCS, most of the largest districts will have at least one school selected for each of the surveys. All special contact districts with schools sampled for both NTPS and SSOCS received both research applications concurrently and were given the option to participate in NTPS only, SSOCS only, or both NTPS and SSOCS. The research request packets for the districts in both studies contained an additional letter introducing the studies and emphasizing that SSOCS and NTPS are working together to minimize the number of schools asked to participate in both studies. Some special districts found the dual application confusing, particularly districts with online application systems that do not allow for multiple applications to be linked. In addition, the samples for NTPS and SSOCS are drawn at different times, and coordinating applications delays when a list of schools sampled for both studies can be shared with a district.

As a result, during future cycles of NTPS that overlap with the SSOCS, separate application packages will be sent to special districts with schools in sample for both NTPS and SSOCS, though the staff that follow up with special districts about the status of these applications will be able to direct districts to the appropriate contact person if there are questions about other NCES studies. Note that, because the NTPS 2020-21 does not overlap with the SSOCS 2020 collection, this result is not directly relevant for the NTPS 2020-21.

B.4.2 Tests Included in the Design of NTPS 2020-21

NCES is currently considering options for tests of methods, materials, and procedures to be conducted as part of NTPS 2020-21, with the goal to increase response in the 2020-21 collection. There are two sets of tests – school-level tests and teacher-level tests. Each test described below has a two-letter code next to the top-level description; those codes can be matched with the Summary Table for Contact Materials found on pp. 30-41 of Appendix A.

B.4.2.1 Testing at the School-level

Four experiments aimed at increasing school-level response rates are planned for the 2020-21 NTPS, namely (1) testing non-monetary incentives, (2) testing new package contents, (3) testing prepopulated TLFs, and (4) testing various question layouts on the school questionnaire internet instruments. Each of these experiments is described briefly below.

Following data collection, each experiment will be evaluated using a series of metrics calculated for the control group and each treatment group of the experiment separately. These metrics will include, but not be limited to:

Response rate;
Average number of contacts;
Days to respond;
Data collection costs.

The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the experimental treatment had on a case’s likelihood to response, given that case’s unique characteristics.

The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of a particular treatment. For example, if cases within the experimental group using non-monetary incentives respond in a more timely fashion, which could reduce the number of cases included in follow-up operations, allowing finite resources, such as field, to be spread across fewer cases.

Using data collection costs associated with each mailout, any additional costs associated with a particular treatment group, and estimates for interviewer costs, an average cost-per-case can be determined within each experimental group. A reduction in cost-per-case could justify the use of a particular treatment to reduce the overall survey cost. Particular treatments may lead to more initial costs, such as the use of incentives, but if cases within a particular treatment group respond in a fewer number of contacts – specifically more costly contacts such as personal visits – this could lead to a reduction in overall cost at the end of data collection.

Testing non-monetary incentives at the school level (1S). The NTPS 2020-21 will include an experiment for a subsample of public and private schools designed to examine the effectiveness of offering a non-monetary incentive to schools to boost overall NTPS response within the school. Schools assigned to the experimental treatment will receive a non-monetary incentive with the initial mailout package. This package will be addressed to the survey coordinator at the school address. If a survey coordinator was not established during the screener interview, the package will be addressed to the principal at the school address. The package will contain the non-monetary incentive, along with a letter to the survey coordinator or principal, and three individually sealed envelopes that contain login information for completing the TLF, Principal Questionnaire, and School Questionnaire. The critical features of the non-monetary incentive item are that the item is: (1) sharable amongst school staff, (2) raise awareness about the survey within the school, leading to conversations about the NTPS among school staff, and (3) customizable, in that the item can be NTPS or education-branded and possibly include select national-level data points from previous cycles of the survey.

The details of the non-monetary incentive are still being worked out and are pending the acquisition process and the receipt of prototypes, however, the current plan is to send large (6.5 gallon), sealed metal tins of popcorn to schools in the non-monetary incentive treatment group. The popcorn tin will be NTPS or NCES branded and include national-level data points from the NTPS 2017-18. The design, printed in at least three colors, will be wrapped around the tin so that it is clearly visible to school staff (rather than printed only on the lid). NCES will submit a change request with the final details in the spring of 2020.

The minimal detectable difference in response rate between the control group and the treatment group that would be necessary to determine a statistically significant effect associated with the treatment, was calculated as follows:

where

Z_1-α = 1.96 for a 95-percent confidence-level, per NCES standards
Z_β = .845 for a standard power-level of .8
σ₁and σ₂are maximized to .5, to provide the most conservative estimate
n₁ and n₂ are the sample sizes for the control and treatment groups
D = 1.68, the design effect observed in the recent administrations of NTPS

Given that the treatment group will include 1,000 public schools and 250 private schools, a statistically significant difference will be determined if the control and treatment group’s response rates differ by 6.41% and 12.80% for public and private schools, respectively.

Testing new mailed package contents in school mailings (2S). In an effort to both increase response rates and lower mailing costs, NTPS 2020-21 will explore whether new types of mailed materials will yield higher response rates.

The NTPS 2020-21 will test two versions of letters to principals and school coordinators (included in Appendix A) to determine whether modifying contact materials to emphasize the values of the study and the benefits of participating can increase response rates compared to letters similar to those used in past NTPS administrations. There will be two versions of each letter (traditional and modified) for the screener and initial mailings, as follows:

Screener letter;
cover letter to principal and cover letter to survey coordinator (initial mailout); and
cover letter to principal/survey coordinator (second mailout).

As such, this experiment impacts the screener mailout, the initial school mailout, and the second school mailout.

4,800 public schools will receive the traditional letters and 4,800 public schools will receive the modified letter³. Given these projected sample sizes, a statistically significant difference will be determined if the traditional letter control group and the modified letter treatment group’s response rates differ by 3.71%. Similarly for private schools, 1,348 schools will receive the traditional letter and 1,348 schools will receive the modified letter. Given these projected sample sizes, a statistically significant difference will be determined if the traditional letter control group and the modified letter treatment group’s response rates differ by 7.04%.

In addition, a randomized experiment in NTPS 2020-21 will compare the effects of including a data product (a colorful data wheel that displays state-level NTPS data from the 2017-18 collection) in the Screener letters to public schools, at which time the principal is invited to complete the Screener online using the Screener internet instrument.

The Screener mail-out will include the cross between the data product experiment and the letter wording experiment, resulting in a randomized four-way panel design. Half of the traditional Screener letter mail-out will include the data product and half of the modified Screener letter mail-outs will include the data product. Each treatment group will include 2,650 schools and a statistically significant difference will be determined between any two treatments groups if the treatment groups’ response rates differ by 4.99%.

Testing prepopulated Teacher Listing Forms (TLFs) (3S). The NTPS 2020-21 will offer prepopulated TLFs to schools for verification via the NTPS Respondent Portal TLF application where vendor-provided teacher data will be loaded into the NTPS portal. The use of prepopulated TLFs via the NTPS Respondent Portal will be offered to respondents in a split-panel manner in order to assess the quality and burden tradeoffs of offering schools a prepopulated TLF. The assumption behind this TLF collection strategy is that validating a prepopulated TLF is less burdensome than completing a blank TLF, but that the data received on the blank TLF may be more accurate based on feedback from NTPS 2017-18 operations; however, this has not been validated quantitatively.

A subset of the NTPS 2020-21 schools with acceptable vendor data will be offered their prepopulated TLF via the portal, while the remaining schools with acceptable vendor data will only be offered the traditional Excel upload and manual entry options. This test will only be conducted on the schools not already designated to receive the non-monetary incentive. 15% of the public schools, or 1,395 schools, with acceptable vendor data will be offered the blank TLFs and the remaining 7,907 schools will receive the prepopulated TLFs. Given these projected sample sizes, a statistically significant difference between the group receiving the prepopulated TLFs and the group receiving the blank TLFs will be found if the groups’ response rates differ by 5.28%. Similarly, 20% of the private schools, or 481 schools, with acceptable vendor data will be offered the blank TLFs and the remaining 1,925 schools will receive the prepopulated TLFs. Given these projected sample sizes, a statistically significant difference between the two groups will be found if the groups’ response rates differ by 9.32%. If schools randomly assigned to receive a blank TLF have not completed that form after multiple contact attempts, teachers from those schools will be sampled from vendor data and given the opportunity to complete the Teacher Questionnaire.

A validation study will be conducted by staff in the Contact Centers (via coordination with Census- LCC) for a subset of schools (approximately 100 schools from each TLF submission method – prepopulated and blank) during the spring of 2021. The purpose of the call operation is to verify accuracy of TLF data and debrief schools about their experience with the TLF related task (generally) and NTPS portal instrument, as well as any discrepancy between the two teacher lists (if applicable). This will be more of an intellectual exercise aimed at confirming that our expectation that the prepopulated TLF reduces burden and improves response rate is accurate.

Testing various question layouts on the school questionnaire internet instruments (4S). The NTPS 2020-21 school questionnaires will include several different versions of items or groups of related items, with the layout of the items varying across the treatment groups. The goal of the experiment is to compare the response distributions of the different versions of the item and ultimately identify the best question layout for future cycles of NTPS.

Vacancies Item (2-4). Item 2-4 on the public and private school questionnaires asks the respondent how easy or difficult it was to fill vacancies for 12 positions in their school. The response options include the following: easy, somewhat difficult, very difficult, could not fill vacancy, no vacancy this school year, and position not offered in the school.

A filter question has been included in past administrations, first asking respondents whether their school had any teaching vacancies in any field. While removal of this filter question will allow researchers to determine whether a school did not have a vacancy in a given field because the position was not offered or because there was simply no vacancy in any field, an important distinction for estimating the percentage of schools with vacancies in a given field, it is possible that fewer vacancies would be reported without the presence of a filter question, that is, respondents may mistakenly omit vacancies when a list of teaching fields is not seen. This alternative presentation with a filter question is included in Appendix B. The presentation with the filter question is the experimental treatment.

None Boxes. In the 2017-18 NTPS, “None” boxes were included in web instruments and paper questionnaire instruments for items that asked the respondent to provide a count (e.g., number of minutes spent on various subjects/activities). The “None” boxes will be replaced in the 2020-21 NTPS School Questionnaire (SQ) internet instrument with an item-specific instruction to ”Write ‘0’ if…” for a subsample of respondents.

This split-panel experiment for the SQ instrument will assess the impact of the absence of “None” boxes on data quality. This experiment requires two instrument versions for the following survey items:

Item 2-1, Teacher counts: One version with None boxes, one without.
Item 2-2, School staffing: One version with None boxes, one without.
Item 2-4, Teaching vacancies: Once version to match paper questionnaire, one version with Yes/No filter question (as outlined above).
Item 2-5c, Newly hired teachers in their first year: One version with None box, one without.
Item 4-2b(1-4), IEP students in classroom settings: One version with None boxes, one without.
Item 4-6b(2)/c/d – NSLP, FRPL: One version with None boxes, one without
Item 4-8a/b, Title I counts: One version with None boxes, one without.

The presence of the “none” boxes is the experimental treatment.

The resulting NTPS 2020-21 school questionnaire will have two internet versions – an experimental version (with a filter question for item 2-4 AND “none” boxes) and a control version (without a filter question for item 2-4 and no “none” boxes). The two web design experiments will not be crossed for the purposes of analyses. Schools will be assigned to an instrument version treatment at the time of sampling.

Using the observed web response rates from 2017-18 as a benchmark, 4,632 public schools and 1,249 private schools are projected to respond by web during the 2020-21 data collection cycle. 20% of public schools and 25% of private schools will be assigned to receive the experimental version of the web questionnaire. Given the projected number of schools expected to respond by web and percentage of schools assigned to the experimental version of the web questionnaire, a statistically significant difference in the likelihood of responding to a particular question between the control and experimental versions of the questionnaire will be found if the two groups’ item response rates to that question differ by 6.68% for public schools and 11.95% for private schools. Each question will also be analyzed to determine if the experimental question layout results in a significantly different distribution of responses from the control question layout.

B.4.2.2 Testing at the Teacher-level

Two experiments aimed at increasing teacher-level response rates are planned for the 2020-21 NTPS, namely (1) further testing the use of teacher incentives and testing envelope packaging for teacher incentive letters, and (2) testing tailored contact materials. Each of these experiments is described briefly below.

Further testing the use of teacher incentives and testing the envelope packaging for teacher invitation letters (1T). Due to the favorable results from the use of teacher incentives for the NTPS 2017-18, the NTPS 2020-21 will include the use of incentives. Two types of incentives will be offered in an experimental manner – a prepaid cash monetary incentive and a non-monetary incentive. Teachers in the experimental treatment will receive an education-branded canvas (or similar fabric) tote bag at the first contact by mail. The treatment will be further separated into two groups – one where each of the teachers and the survey coordinator receive a tote bag, and the other where only each of the teachers receives a tote bag. The thought is that, since the survey coordinator is tasked with distributing the teacher packages, (s)he may also benefit from receiving the item, given that it is going to be apparent that there is something other than a letter in each envelope.

Finally, in order to assess the impact of receiving a non-monetary incentive over no incentive at all, a “no incentive” treatment will be included in the design. This treatment will be further separated into two groups – one where the teacher receives his or her invitation letter in a large (custom) windowed envelope and one where the teacher receives his or her invitation letter in a pressure-sealed mailer. The goal is to assess whether the use of a pressure sealed mailer (which are cheaper and more efficient for NPC assembly and QA) impacts response. The no incentive treatment using traditional envelopes is considered to be the control group for this experiment.

The resulting treatment groups are as follows:

Cash ($5) incentive treatment (teachers);
Non-monetary incentive treatment (teachers);
Non-monetary incentive combination treatment (teachers and survey coordinators);

No incentive, pressure sealer treatment; and
- No incentive, envelope treatment (CONTROL).

The treatments will be evaluated against one another and the control group.

Note that the monetary ($5 cash) incentives for teachers will be adhered to a piece of yellow cardstock using removable sticky glue to be inserted with the letter. The cardstock will be ½ sheet rather than a full sheet and will include text thanking them for their participation in the study. Using this new contact material item will (1) increase the weight of the envelope, making it feel more “substantial” and important; (2) prevent the money from free-floating inside the envelope; and (3) help ensure that the respondent notices the cash.

The final tote bag design will be included in the change request submitted in the spring of 2020.

To ensure a similar distribution of schools for each experimental group, the school sample will be sorted by an indicator for whether or not the school is covered on the vendor list and other selected school characteristics. The school characteristics will be selected for having significant influence on response based on past cycles of NTPS and on a model calculated to predict the likelihood of a school returning the TLF. For example, one of the school characteristics that may have a significant influence on response is the priority/non-priority school status (i.e., schools with higher potential to impact weighting and lower propensity to respond that are subject to a slightly different set of collection operations), which will be taken into account by using it as a sort variable in which the priority/non-priority status will be a stratification variable in the model.

The teacher treatment for each sampled school will be assigned at the time of school sampling, prior to the start of data collection. As such, the random assignment should result in a similar TLF response rate across the treatment groups. All teachers within the same school will receive the same incentive treatment; there will not be “mixed schools” where some teachers receive the prepaid cash monetary incentive while others receive the non-monetary tote bag incentive, and teachers in both schools that received the non-monetary school-level incentive and schools that did not receive this incentive will be eligible to receive a teacher incentive. Table 10 shows the experimental groups and approximate expected sample sizes.

Table 10. Experimental groups and approximate expected sample sizes

Experimental Group

Initial School Sample Sizes

Initial Teacher Sample Sizes^⁴

Monetary Incentive for teachers

5,300 Public Schools

1,200 Private Schools

26,700 Public School Teachers

2,720 Private School Teachers

Non-monetary Incentive – for teachers only

1,325 Public Schools

6,675 Public School Teachers

Non-monetary Incentive – for teachers and survey coordinators

1,325 Public Schools

900 Private Schools

6,675 Public School Teachers

2,040 Private School Teachers

No Incentive, with pressure-sealed mailers

1,325 Public Schools

6,675 Public School Teachers

No Incentive, with regular large envelope (CONTROL)

1,325 Public Schools

900 Private Schools

6,675 Public School Teachers

2,040 Private School Teachers

Following data collection, analyses will be conducted at the treatment level (incentive group vs. control). The analyses examined include:

Response rate;
R-Indicators;
Average number of contacts;
Days to respond;
Data collection costs.

The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the incentive had on a case’s likelihood to respond, given that case’s unique characteristics.

Given the project sample sizes in the table above, a statistically significant difference between the control group receiving the cash incentive and any of the other four treatment groups will be found if the response rates between the two groups differ by 3.37% for public school teachers and 7.01% for private school teachers.

R-indicators will be used to determine the overall balance of the respondent population, as well as within each experimental group. R-indicators will be calculated for the full sample, as well as variable-level and category-level partial R-indicators to determine which characteristics specifically are contributing to imbalance within the respondent population.

The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of one type of incentive over another. If cases within the experimental group using non-monetary incentives respond in a more timely fashion, this could reduce the number of cases included in follow-up operations, allowing finite resources, such as field, to be spread across fewer cases.

Using data collection costs associated with each mailout, the value of the incentive itself, and estimates for interviewer costs, an average cost-per-case can be determined within each experimental group. A reduction in cost-per-case could justify the use of incentives to reduce the overall survey cost. While using incentives leads to more initial costs, if cases respond in a fewer number of contacts – specifically more costly contacts such as personal visits – this could lead to a reduction in overall cost at the end of data collection.

Tailored Contact Materials at the teacher level (2T). Respondents sampled for NTPS receive letters and e-mails that emphasize the importance of their participation in the survey, but this information has not emphasized the ways in which NTPS data inform researchers and policymakers. In NTPS 2017-18, the statement “Public school teachers provided an average of 27 hours of instruction to students during a typical week in the 2015-16 school year. What about you?” was added to the outside of Third Reminder Teacher Letter envelopes for the final wave of sampled public school teachers.

Focus groups with teachers explored what statistics and other general revised wording is most salient to different types of respondents, and similar statements will be placed on materials sent to respondents, such as on the outside of envelopes or within enclosed letters, to determine whether targeted, persuasive messaging can increase response rates. Teachers seemed to take particular note of statistics related to finances (for example, salary and out of pocket spending on supplies) and where comparisons could be made either between statistics (for example, the amount of time spent providing instruction and worked overall) or types of teachers (for example, between teachers nationally and teachers in their own state).

NTPS 2020-21 will include an experiment in which tailored statistics will be overprinted on the exterior of the pressure-sealed mailers to non-responding teachers in the second teacher mailings. Teachers in the control group will receive their reminder letter with login information in a pressure-sealed mailer without overprinted information printed on the exterior. This experiment will be crossed with the Teacher Incentive and Packaging experiments in the first mailing, yielding a total of ten experimental treatment groups.

Given the expected number of teachers receiving the second mail-out, a statistically significant difference in response rates between the group receiving the overprinted pressure-sealer and the group receiving the pressure-sealer without the overprint will be found if the response rates differ by 2.38% for public school teachers and 6.49% for private school teachers.

Finally, the later mailings and e-mails will include tailored (with customized data/information) text in either letters or emails; however this will not be done experimentally.

The final plan for teacher-level tests included in the 2020-21 NTPS is included in Exhibit 2.

Exhibit 2: 2020-21 National Teacher and Principal Survey – Teacher-Level Data Collection Tests

B.5 Individuals Responsible for Study Design and Performance

The following individuals are responsible for the NTPS 2020-21 study design, data collection, and analysis: Maura Spiegelman and Andy Zukerberg at NCES; Shawna Cox, Walter Holmes, Teresa Thomas, Allison Zotti, and Aaron Gilary at U.S. Census Bureau; and David Marker, Lou Rizzo, and Minsun Riddles at Westat.

1 If the 2018-19 CCD is not available as of early January 2020, the most recently available CCD as of that date will be used instead.

2 The sampling estimates may be slightly modified as we conduct more analysis of our last collection. If these estimates are updated, they will be replaced in this document by January 15, 2020.

3 Schools sampled for NTPS 2020-21 that receive the school non-monetary incentive are ineligible for this experiment on package contents.

4 The teacher samples sizes will not be exactly equal across experimental groups, as the number of teachers sampled from each school is not equal. However, each group should contain roughly the same number of teachers.

File Type	application/vnd.openxmlformats-officedocument.wordprocessingml.document
File Modified	0000-00-00
File Created	2021-01-15