FOLLOW-UP SURVEYS TO THE 2020-21 NATIONAL TEACHER AND PRINCIPAL SURVEY (NTPS):
2021-22 Teacher Follow-Up Survey (TFS) and
2021-22 Principal Follow-Up Survey (PFS)
PART B
OMB# 1850-0617 v.4
National Center for Education Statistics (NCES)
U.S. Department of Education
March 2021
revised May 2021
TABLE OF CONTENTS
B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS 1
B.1 Respondent Universe and Sample Design 1
B.1.1 Background on the 2020-21 NTPS Sample 1
B.1.2 2021-22 TFS Respondent Universe and Sample Design 2
B.1.3 2021-22 PFS Respondent Universe and Sample Design 3
B.2 Procedures for the Collection of Information 4
B.2.1 2021-22 TFS Procedures for the Collection of Information 4
B.2.2 2021-22 PFS Procedures for the Collection of Information 6
B.3 Methods for Maximizing Response 7
B.3.1 Endorsements from Key Public and Private School Groups 7
B.3.2 Stressing the Importance of the Survey and the Respondents’ Participation in it 8
B.3.3 Offering a cash incentive to current and former teachers 8
B.3.4 Extensive Follow-up (by mail, email, and telephone) of Non-Respondents 8
B.4 Tests of Procedures and Methods 9
B.4.1 Testing the use of text messages in both TFS and PFS 9
B.4.2 Testing the inclusion of an infographic in TFS packages 10
This request is to conduct data collection for the two follow-up surveys to the 2020-21 National Teacher and Principal Survey (NTPS) – the 2021-22 Teacher Follow-up Survey (TFS) and the 2021-22 Principal Follow-up Survey (PFS).
Section B.1 of this document describes the universe, sample design, and estimation details for the 2021-22 TFS and PFS. Section B.2 describes the data collection procedures for both surveys. Section B.3 discusses methods to secure cooperation and mitigate nonresponse. In particular, it describes methods used to improve response rates in the 2020-21 NTPS and how those methods will be used in its two follow-up surveys. Section B.4 describes recent developments in a long history of tests of methods and procedures to improve data quality. Section B.5 lists the names of those involved in the design of the study and the development of these materials.
The 2020-21 National Teacher and Principal Survey (2020-21 NTPS) sample was a two-stage stratified sample. The schools were sampled first, and then teachers were selected within each sampled school.
The NTPS school sample is a systematic probability proportionate to size (PPS) sample, where size is defined to be the square root of the full-time equivalent number of teachers at the school. Unlike its predecessor, the School and Staffing Survey (SASS), the NTPS does not stratify schools prior to sampling. However, some schools are oversampled based on their assigned domains. The public school oversampling domains are defined by charter status (charter/not charter), grade level (elementary/middle/high/combined), urbanicity (city/suburban/town/rural), and poverty status (> 75% of enrollment eligible for free or reduced price lunch/otherwise). The private school oversampling domains are defined by affiliation (11 levels), school span (Elementary/Secondary/Combined), school size (less than 100 students, or 100+ students), and Census division (the Mid-Atlantic states vs. other regions).
Public charter schools were oversampled at a rate 3.26 times their proportional sample allocation. Proportional allocation in this instance is defined as the domain’s sum of schools’ measures of size relative to the sum of measures of size of all schools in the sampling frame. Further oversampling applied to non-charter schools by grade level, whereby combined schools were oversampled at 2.56 times the elementary allocation, middle schools were oversampled at 1.30 times the elementary allocation, and high schools were oversampled at 1.35 times the elementary allocation. In addition, once the grade level sampling differential was applied, non-charter schools had their probabilities further altered by urbanicity as follows: city schools were oversampled at a rate of 1.05 times the proportional allocation (equal to the suburban sampling rate), town schools were oversampled at a rate 1.34 times the suburban rate, and rural schools were undersampled at a rate of 1.11 times the suburban rate.
In addition to oversampling by characteristic, schools in thirty (generally smaller, based on population size) states have relatively higher sampling rates to realize lower bounds on precision for these states and to enable the calculation of reliable key state estimates.
Along with the names of employed teachers and their email addresses, sampled schools were asked to provide the following descriptive characteristics of each teacher: teaching status, where options were either part-time or full-time; and subject matter taught, where teachers were classified as special education, general elementary, math, science, English/language arts, social studies, vocational/technical, or other.
The above information for each teacher in a selected NTPS school comprised the initial teacher sampling frame. The frame was also supplemented with information from vendor-purchased teacher roster files and from a clerical look-up operation conducted as part of NTPS where teacher lists were located on school and/or district websites. Within each sampled school, teachers were stratified by subject. The strata include math, science, English/language arts, social studies, and everything else. No oversampling was done for the teacher sample, beyond the oversampling inherited from their schools.
Teacher records within a school domain and teacher stratum were sorted by the teacher subject and the teacher line number code, which is a unique number assigned to identify the teacher within the list of keyed teachers. Within each teacher stratum in each school, teachers were selected systematically with equal probability.
Table B1 below shows a summary of the expected 2020-21 NTPS allocated sample sizes.
Table B1. Summary of the expected 2020-21 NTPS sample sizes
|
School |
Teacher |
Traditional Public |
8,660 |
43,460 |
Public Charter |
1,260 |
5,790 |
Private |
3,000 |
6,300 |
Total |
12,920 |
55,550 |
The 2021-22 Teacher Follow-up Survey (2021-22 TFS) sample will be a subsample from the 2020-21 NTPS teacher sample. The 2021-22 TFS sample is a random subsample from the 2020-21 NTPS teacher sample, and, as such, a probability sample similar to 2020-21 NTPS.
A planned total of 10,273 teachers will be selected according to a stratified design similar to earlier TFS collections and based on information obtained from the NTPS schools about the status of the 2020-21 NTPS respondent teachers in the 2021-22 school year. The 2021-22 TFS sample will include former teachers (NTPS teachers who have left the teaching profession, or “Leavers”), current teachers (NTPS teachers who have remained in the teaching profession, or “Stayers”), and teachers who have left their 2020-21 school, but for whom there is no information available (the ‘unknown’ group). The current teachers group is further stratified into two groups: those who remained in the school they were teaching in 2020-21 (“Stayers”) and those who moved to different schools (“Movers”). This design provides analytic data on teachers who stay, move, or leave the teaching profession. The status for each TFS teacher is determined through a Teacher Status Form (TFS-1) mailed to schools in the fall of 2021. For the 2021-22 TFS, the Movers, Leavers, and Unknown teachers will be sampled with certainty, as will all Stayers in their first year.
Teachers will be further stratified by factors known about the teacher from the NTPS year, including sector (traditional public, public charter, and private), grade level (elementary, middle, and high)1, experience level (1 year, 2-5 years, and 6+ years of teaching), and race/ethnicity (Hispanic, Black non-Hispanic, and other). Grade level is based on the grade level the teacher is teaching, but school grade level is used as the second choice if the teacher teaches in more than one grade level.
Within the explicit strata, there will be systematic sampling with a designated sort order. The sort variables include measure of size (teacher base weight), teacher subject, Census region, urbanicity, school enrollment, and for private school teachers, type of private school.
Teachers will be selected within each stratum using a probability proportional to size sampling procedure. The measure of size will be the 2020-21 NTPS final teacher weight. This essentially ‘reverses’ the probability of selection of the teacher in NTPS, and results in a self-weighting sample within each teacher stratum. This will maximize the efficiency of the TFS sample.
The table below presents target sample sizes for important teacher categories. Note that charter school and private school teachers are oversampled relative to traditional public school teachers. This will be done to reduce the coefficient of variation (CV) for charter school estimates to approximately 20 percent. Movers, leavers, and teachers with unknown status are all selected, which makes them overrepresented in the sample relative to other teachers. This is done to increase the precision for analyses of these important subgroups. The breakdown of stayers, movers, leavers, and ‘unknown’ teachers in public, charter, and private schools is as shown in Table B2.
Table B2. Distribution of teacher types across school types
Status |
Sector |
|||
Public |
Charter |
Private |
Total |
|
Stayer |
2,810 (0.093) |
538 (0.264) |
1,026 (0.269) |
4,347 |
Mover |
2,214 |
205 |
246 |
2,665 |
Leaver |
1,434 |
107 |
278 |
1,819 |
Unknown |
1,029 |
182 |
204 |
1,415 |
Total |
7,487 |
1,032 |
1,754 |
10,273 |
Note: The number in parentheses is the estimated sampling rate for each Stayer cell.
Within these target sample sizes, Hispanic and Black Non-Hispanic stayers are oversampled by 60 percent to optimize the reliability of their comparison with other stayers (Movers, Leavers, and Unknown are already sampled with certainty). The target sample sizes at the status/sector/race level are as seen in Table B3.
Using these target sample sizes, the sample is allocated to the sampling strata, which are defined by the teacher’s sector, status, experience, grade, and race categories. This is done proportional to the cumulative teacher measure of size within each stratum. The teacher measure of size will be the 2020-21 NTPS teacher base weight.
Table B3. Distribution of teacher types across school types
Status |
Sector |
Total |
||||||||
Public |
Charter |
Private |
||||||||
Hispanic |
Black Non-Hispanic |
Other |
Hispanic |
Black Non-Hispanic |
Other |
Hispanic |
Black Non-Hispanic |
Other |
||
Stayer |
258 |
252 |
2,300 |
106 |
110 |
322 |
209 |
166 |
651 |
4,374 |
Mover |
116 |
126 |
1,972 |
22 |
20 |
163 |
14 |
20 |
212 |
2,665 |
Leaver |
44 |
67 |
1,323 |
8 |
11 |
88 |
10 |
10 |
258 |
1,819 |
Unknown |
79 |
89 |
861 |
22 |
28 |
132 |
19 |
20 |
165 |
1,415 |
Total |
497 |
534 |
6,456 |
158 |
169 |
705 |
284 |
216 |
1,286 |
10,273 |
The sampling frame for the 2021-22 PFS consists of all the traditional public and public charter school principals who completed a Principal Questionnaire and all private school principals who completed a Private School Principal Questionnaire during the 2020-21 NTPS. Any sampled NTPS principal who did not complete their questionnaire or was otherwise found to be out of scope for NTPS will not be included in the PFS frame. The 2020-21 PFS sample will include approximately 6,700 public and public charter school principals and 1,750 private school principals.
The 2016-17 PFS included a sample of 5,700 principals from public and charter schools. The large discrepancy in sample size between the 2016-17 and 2021-22 administrations of the PFS is due to the fact that the 2015-16 NTPS included fewer sampled schools than the 2020-21 NTPS, because the 2015-16 NTPS sample included public school principals only and produced estimates at the national rather than state level.
The TFS-1 will be mailed to all schools in which a teacher completed a Teacher Questionnaire during the 2020-21 administration of NTPS. A knowledgeable person at the school, such as the principal, will be asked to complete the TFS-1 by indicating the current teaching status of each teacher listed on the form. These data will be used to stratify the teachers for 2021-22 TFS sampling into groups of “stayers” (still teaching at the same school), “movers” (still teaching, but at a different school), or “leavers” (no longer teaching).
TFS was designed from its inception to sample only from respondents to SASS/NTPS in the prior year. It is possible to sample teachers for TFS from the pool of teachers initially selected for NTPS rather than only from those who responded to NTPS. By not sampling teachers from the nonresponse stratum, potential biases may be introduced into TFS. When considering unit nonresponse for SASS, there was no evidence to point to a substantial bias in teacher response rates.
The main purpose of TFS is to measure various characteristics collected on NTPS of teachers who move, leave, or stay in the teaching profession the following year. If NTPS nonrespondents were to be contacted for the TFS, there are several methodological problems that would need to be solved: 1) there would be no prior year data, so all of the NTPS teacher questionnaire items would have to be asked retrospectively of the teacher before the TFS could be administered; 2) if the teacher was determined to be out-of-scope for NTPS, then there would be no TFS interview and a sampling adjustment would be required; and 3) if the teacher was in-scope for NTPS but out-of-scope for TFS (if the teacher moves out of the country or dies), then that would also require some adjustment of the sampling weights. Logistically, trying to combine prior-year NTPS interviewing with current-year TFS interviewing is a complication that would delay the field data collection because it would likely take longer to complete the nonresponding NTPS cases. For these reasons, TFS teachers will be sampled from responding NTPS teachers only. A response rate of approximately 99 percent can be expected from the TFS-1 based on the prior TFS data collection cycle.
Using the teacher statuses provided by schools on the TFS-1, Census Bureau staff will select the sample of teachers for the 2021-22 TFS and prepare materials for data collection.
In the 2020-21 NTPS, teachers were asked to provide the following information:
Name
Street address
Cell phone number
Home phone number
Most convenient day(s) to reach her/him
Work email address
Home email address
Prior to the start of data collection, Census staff will conduct research to obtain contact information (e.g., home address, email address, and/or telephone number) for teachers who provided incomplete or no contact information in the NTPS interview. Staff will first use any contact information provided (e.g., telephone number, partial address, contact person’s information) from the NTPS questionnaire to attempt to complete the address of the sample member. If no contact information was provided, staff will use sources such as the white or yellow pages, Internet searches, and other address and phone number databases.
Data collection for the Questionnaire for Former Teachers (TFS-2) and the Questionnaire for Current Teachers (TFS-3) will begin in January 2022. The first contact with sample members will be through an initial contact letter sent by mail inviting them to participate in the survey and providing credentials for them to log on and complete the appropriate web-based survey. All teachers will also receive a $5 or $10 cash incentive in this initial mailed package. Note that the monetary ($5 or $10 cash) incentives for teachers will be adhered to a piece of yellow cardstock using removable sticky glue and will be inserted with the letter. The cardstock will be ½ sheet rather than a full sheet and will include text thanking them for their participation in the study. Adhering the case to the cardstock insert will (1) increase the weight of the envelope, making it feel more “substantial” and important; (2) prevent the money from free-floating inside the envelope; and (3) help ensure that the respondent notices the cash. Concurrently, Census will send an email invitation to participate (unless a valid email address is unavailable) to the sample member with a link to the survey and their log in credentials. If home addresses were not provided on the 2020-21 NTPS teacher questionnaire or cannot be obtained, the NTPS school address will be used as the address. Each letter and email will be customized and will provide the sampled teacher with login information to access the web-based instrument.
In late January, Census will send nonrespondents a second teacher mailout and first email reminder, and in mid-February a second reminder only by email. For a subsample of teachers, this package will also include an experimental infographic. Refer to section B.4.2 for additional information about testing the inclusion of an infographic in TFS packages.
The third mailout and third email reminder will be sent to nonrespondents in early March. This mailing will include a paper copy of the applicable TFS questionnaire, as will the fourth mailed package in mid-April and fifth (and final mailed contact) mailed package in mid-May. A fourth reminder by email only will be sent in late March, followed by a fifth and sixth reminder by email, which will be sent alongside the fourth and fifth mailed packages. Data collection will end in early July 2022. Throughout data collection, research will be conducted, as needed, to find current addresses and emails for sampled teachers whose letters and emails are returned as undeliverable.
In addition, ongoing support will be provided for sampled teachers to whom the incorrect paper questionnaire type was mailed (e.g., a mover who was incorrectly classified as a leaver and received a TFS-2 questionnaire as a result). When these cases are discovered, either because the teacher was Amish or Mennonite or in response to the late mailing(s) of the paper questionnaires, Census will send the correct form for completion.
From February through June, outbound telephone calls to remind teachers who have not yet responded to the web-based collection to complete their form. While the call is intended to encourage teachers to log on and respond to the survey themselves, interviewers will be able to administer it over the phone at the request of the respondent. Throughout the data collection period, respondents who request a paper questionnaire will be mailed one.
The 2021-22 TFS will include an experiment to test different methods of texting survey participants as a means of exploring the use of text messages as a new data collection mode. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS teacher questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number.
To explore using test messaging as a contact method for TFS, the experiment will include three treatment groups:
Replace the second mailout (web invitation letter) and third mailout (paper questionnaire and letter) with text message contacts that include the link to complete the TFS online
Include a text message contact that includes a link to complete the TFS online concurrent with the fourth mailout (second mailed paper questionnaire), and replace the fifth mailout (third mailed paper questionnaire) with a text message contact that includes a link to complete the TFS online
Include an interactive questions text message contact at the time of the fourth mailout (second mailed paper questionnaire) and replace the fifth mailout (third mailed paper questionnaire with an interactive questions text message contact
For groups one and two, the text message will invite the teacher to complete the TFS questionnaire online by providing the link to the web instrument in the text message. For group three, questions about current teaching status will be texted directly to each sampled teacher and his or her response will be collected via their text messaged response. These interactive texts will follow the same general skip pattern as the web instrument, where subsequent texts will ask follow-up questions based on the teacher’s responses. Results from this experiment will help determine whether a respondent is more likely to respond via a texted survey web link earlier in data collection versus later in data collection, as well as whether a respondent is more likely to respond via a texted survey web link or directly in an interactive text exchange later in data collection. For additional details about testing the use of text messaging in the 2021-22 TFS, refer to section B.4.1 of this document.
A response rate of approximately 75 percent (base-weighted) can be expected from the leavers after follow up, and a response rate of about 81 percent (base-weighted) can be expected from stayers and movers after follow-up, based on the prior administration of TFS. Table B4 shows weighted unit response rates in the last administration of TFS, TFS 2012-13.
Table B4. Weighted unit and overall response rates of TFS teachers, using initial base weight, by sector and teaching status: 2012–13
Sector and teaching status |
Weighted unit response rate |
Weighted overall unit response rate |
Public |
80.7 |
49.9 |
Current teacher |
81.3 |
50.3 |
Stayer |
81.6 |
50.5 |
Mover |
77.9 |
48.2 |
Former teacher |
73.8 |
45.6 |
Private |
79.3 |
39.7 |
In January 2022, Census Bureau staff will mail the PFS-1(A/B)2 to all schools to collect the current occupational status of the 2020-21 principal. Non-responding schools will be sent a reminder letter along with a second copy of the PFS-1 (A/B) approximately two weeks after the initial mailing. Those schools that are still non-responsive three weeks after Mailing 2 will receive a telephone call during which telephone center staff will follow-up by telephone to collect the occupational status of the 2020-21 principal. The telephone center staff will first ask school office staff for the occupational status of the 2020-21 principal. If the office staff is unable to provide the information, telephone center staff will ask to speak with the current school principal.
For non-responding schools or schools that were unable to report the principal’s status, Census Bureau staff will mail a modified version of the Principal Status Form (PFS-1(C/D)3) to the 2020-21 principal at his or her home address (if reported in NTPS). The modifications tailor the form to the previous years’ principal rather than the school staff. This mailing will begin in March 2022, and an email will be sent a few days following the mailing, alerting the principal to the mailing and reminding them to complete their PFS. Non-responding principals will be sent a follow-up mailing and email approximately 2 weeks after the initial mailing and email. Telephone center staff will follow-up by telephone to collect the occupational status of the non-responding principal beginning three weeks after the reminder mailing and email are sent.
The 2021-22 PFS will include an experiment to test different methods of texting survey participants as a means of exploring the use of text messages as a new data collection mode. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS principal questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number. Because the PFS does not have a web instrument for data collection, all text message contacts for the survey will be an interactive exchange with the principal.
To explore using text messaging as a contact method for PFS, the experiment will include two treatment groups:
Include an interactive questions text message contact to complete the survey concurrent with both principal-level mailouts
Replace the principal-level mailouts with an interactive questions text message contact to complete the survey
This experiment will help determine whether or not survey participants would be willing to respond to a survey via text, generally. In addition, results from this experiment will help determine whether a respondent is more likely to respond via a text exchange instead of returning a paper questionnaire or if the text serves as a reminder, prompting respondents to return their paper questionnaire. For additional details about testing the use of text messaging in the 2021-22 PFS, refer to section B.4.1 of this document.
A response rate in the approximate range of 90-95 percent is expected based on the prior administration of PFS.
This section describes the methods that NCES will use to secure cooperation, maximize response rates, and deal with nonresponse for the 2021-22 NTPS follow-up surveys. The entire survey process, starting with securing research cooperation from key groups and individual sample members and continuing throughout the distribution and collection of individual questionnaires, is designed to increase survey response rates. In addition, NCES believes that the following endorsements, stressing the survey’s importance in communication materials, and extensive follow-up with nonrespondents will further facilitate the overall success of the survey and enhance response rates.
The level of interest and cooperation demonstrated by key groups can often greatly influence the degree of participation of survey respondents. Endorsements are viewed as a critical factor in soliciting cooperation from state and local education officials.
NCES will seek endorsements for the 2021-22 TFS from the following organizations or agencies:
Agudath Israel of America/Lefkowitz Leadership Initiative
American Association of Christian Schools
American Association of School Administrators
American Federation of Teachers
American Montessori Society
Association for Middle Level Education
Association for Middle Level Education (formerly National Middle School Association)
Association for Supervision and Curriculum Development
Association Montessori International
Association of American Educators
Association of American Educators
Association of Christian Schools International
Association of Christian Teachers and Schools
Association of Classical Christian Schools
Association of Military Colleges and Schools
Association of Waldorf Schools of North America
Christian Schools International
Council for American Private Education
Council of Chief State School Officers
Council of Islamic Schools of North America
Council of the Great City Schools
Evangelical Lutheran Church in America
Friends Council on Education
General Conference of Seventh-Day Adventists
Islamic School League of America
Jesuit Schools Network
Jewish Education Services of North America
Lutheran Church-Missouri Synod
National Association of Elementary School Principals
National Association of Episcopal Schools
National Association of Independent Schools
National Association of Private Special Education Centers
National Association of Secondary School Principals
National Catholic Educational Association
National Christian School Association
National Coalition of Girls’ Schools
National Council for Private School Accreditation
National Education Association
National Independent Private School Association
National Parent Teacher Association
Office of Education, General Conference of Seventh Day Adventists
Oral Roberts University Educational Fellowship
Prizmah: Center for Jewish Day Schools
RAVSAK: Jewish Community Day School Network
Southern Baptist Association of Christian Schools
The Association of Boarding Schools
The Jewish Education Project
The School Superintendents Association
Torah Umesorah National Society for Hebrew Day Schools
United States Conference of Catholic Bishops
Wisconsin Evangelical Lutheran Synod
NCES will seek endorsements for the 2021-22 PFS from the following organizations or agencies:
American Association of School Administrators
American Federation of Teachers
National Association of Elementary School Principals
National Association of Secondary School Principals
Official letters will be used to motivate respondents to return surveys. Respondent letters for both the 2021-22 TFS and PFS will be sent by the U.S. Census Bureau and signed by the NCES Commissioner. Communications in the form of both letters and emails will be personalized with the teacher and principals’ names, which is expected to have positive effects on the survey response rates.
The 2004-05 TFS used monetary incentives to increase overall response and internet response when both mail and internet choices were offered. The results showed that the overall response of the incentive groups significantly exceeded that of the non-incentive groups.4 Additionally, the results of the use of teacher incentives for the 2017-18 NTPS indicated that the teacher incentive led to significant increases in the response rate for both public and private school teachers.5 In addition, the average number of days to complete the questionnaire was significantly lower for public school teachers that received the incentive. Finally, the incentive helped increase the overall sample balance for teachers in both public and private schools.
Due to these favorable results, the 2020-21 NTPS included the use of a cash incentives in the initial mailed packages to a subsample of early sampled teachers and as a promised incentive, contingent upon a teacher completing his or her teacher questionnaire, for teachers sampled in December and later. Mid-data collection, a switch to promised incentives was made in an attempt to encouraging teachers to respond to the survey, given the challenges posed during the 2020-21 school year by the coronavirus pandemic. As such, the 2021-22 TFS will include a cash incentive for sampled teachers. The incentives included in the TFS will not be offered in an experimental manner and will instead be offered to all sampled current and former teachers.
Data on teacher retention and attrition are in high demand from researchers and policymakers, but TFS data have not been released since 2012-13. The TFS was last fielded in 2012-13, after the 2011-12 SASS. During the 2012-13 TFS, response rates from private school teachers was problematically low, and NCES was unable to publish a TFS private school teacher data file. In 2016-17, after the 2015-16 NTPS, the overall unit response rate of the TFS 2016-17 was likely to be under 50 percent and therefore the entire collection was cancelled. In an effort to increase response rates for the 2021-22 cycle, a contingency incentive may also be used in later TFS mailings to combat low response, as identified in weekly status reports. A similar contingency incentive was included with the 2017-18 NTPS and led to a boost in response rate for teacher who received this incentive.6
At the end of week 7 of data collection (2/25/2022), we will evaluate the response rates for the TFS-2 (former teachers) and TFS-3 (current teachers) and determine whether a contingency incentive is needed. Specifically, if the TFS-2 response rate is less than 31%, those teachers will be eligible to receive a contingency incentive. If the TFS-3 response rate is less than 38%, those teachers will be eligible to receive a contingency incentive. Depending on whether response rates are struggling overall or for certain types of schools, the contingency plan may be exercised for teachers in all types of schools or only teachers sampled from schools with particular characteristics (e.g., priority schools7). The amount of the contingency incentive will be $10 or $15 based on NTPS priority status, where teachers from priority schools will receive a larger incentive. This will be a one-time, promised cash incentive to be mailed directly to the responding teacher later in the school year.
For the 2021-22 TFS, the Census Bureau will use a variety of techniques to increase response levels including:
(a) Using mixed survey modes— self-administered web-based, telephone interviewer-assisted web-based, and self-administered mail instruments, as needed, to maximize response levels.
(b) Allocating adequate time and resources to respondent tracking efforts to ensure that a high percentage of movers and leavers are successfully located and surveyed. A variety of techniques will be employed to locate survey respondents. Potential tracking sources include: (a) leads provided by school principals or their designees; (b) clerical operations using electronic databases and internet searches; and (c) post office for possible forwarding addresses.
Good questionnaire design techniques will be employed to minimize item non-response. All completed questionnaires from TFS 2012-13 were analyzed to determine which items had the highest levels of item nonresponse. This information guided NCES and the Census Bureau in reviewing the clarity of item wording, definitions, and instructions, and as such, revisions were made to TFS content for the TFS 2016-17. The TFS 2016-17 was approved by OMB (#1850-0934 v1); however, the data collection was cancelled due to lower than anticipated response rates for subdomains of 2015-16 NTPS teachers. Part C of this submission details the minor changes between the proposed TFS instruments for 2021-22 and those approved by OMB for the TFS 2016-17.
To permit sufficient time to locate sampled members who left their teaching position in the previous year and thorough follow-up of non-respondents, the TFS teacher data collection will begin in January 2022, allowing six months for all data collection activities related to teachers.
For the PFS, the Census Bureau will also utilize mixed survey modes – self-administered mail instruments and telephone, as needed, to maximize response levels. Non-respondents will receive at least two reminders. Given the brevity of the instrument, we anticipate response rates at or above 90% with this approach, as with past cycles of the survey.
NCES is currently considering options for tests of methods, materials, and procedures to be conducted as part of the 2021-22 NTPS Follow-up Surveys, with the goal to increase response in the 2020-21 collection. Each test is described in detail below.
For 2021-22 data collection cycle of the TFS and PFS, NCES will explore the use of text messaging as a new data collection mode by including an experiment to test different methods of texting survey participants. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS principal and teacher questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number.
To explore using test messaging as a contact method for TFS, the experiment will include three treatment groups:
Replace the second and third web invitation letter mailouts with text message contacts that include the link to complete the TFS online
Include a text message contact that includes a link to complete the TFS online concurrent with the fourth mailout (the first paper questionnaire mailing), and replace the fifth mailout with a text message contact that includes a link to complete the TFS online
Include an interactive questions text message contact at the time of the fourth mail-out and replace the fifth mail-out with an interactive questions text message contact
For groups one and two, the text message will invite the teacher to complete the TFS questionnaire online by providing the link to the web instrument in the text message. For group three, questions about current teaching status will be texted directly to each sampled teacher and his or her response will be collected via their text messaged response. These interactive texts will follow the same general skip pattern as the web instrument, where subsequent texts will ask follow-up questions based on the teacher’s responses. Results from this experiment will help determine whether a respondent is more likely to respond via a texted survey web link earlier in data collection versus later in data collection, as well as whether a respondent is more likely to respond via a texted survey web link or directly in an interactive text exchange later in data collection.
Only teachers who opt-in to receive text messages are eligible for inclusion in the text messaging experiment. The estimated opt-in rate for public and private school teachers is 60%. The treatment for each sampled teacher will be randomly assigned at the school-level at the time of teacher sampling, prior to the start of teacher data collection. Any school where at least one teacher consented to receiving text messages in the experiment is eligible to be included. The schools will be sorted by school size and other characteristics related to response. As such, the random assignment should result in even sample sizes and a similar distribution of school types across the treatment groups. Only schools where all teachers did not consent to receiving text messages would be considered ineligible for the experiment. If a school is selected to receive text messages, all sampled teachers who had opted-in would receive the same text message treatment, and any sampled teachers who did not consent would not receive any text messages as part of the experiment. The teachers in the sampled school who opted out of text messaging would receive the traditional mail contacts. Nonrespondent teachers may not be substituted with other sampled teachers. The experimental contact plan and expected sample sizes are displayed in Exhibit 1.
Exhibit 1: 2021-22 TFS Text Message Experimental Contact Strategy and Sample Sizes
Experimental Treatment and Sample Size |
Planned Date |
Concurrent with Text 1 |
1/10-1/12 |
1/28-1/31 |
2/16 |
3/07-3/08 |
3/31 |
4/15-4/19 |
5/20-5/23 |
6/10 |
Standard/Control
Group |
Mail Activity |
|
1st Mail (W) |
2nd Mail (W) |
|
3rd Mail (Q) |
|
4th Mail (Q) |
5th Mail (Q) |
|
Email Activity |
|
Initial |
Reminder 1 |
Reminder 2 |
Reminder 3 |
Reminder 4 |
Reminder 5 |
Reminder 6 |
|
|
Telephone Activity |
|
|
Phone Reminder |
Phone Follow-Up |
||||||
Texting Activity |
|
|
||||||||
|
||||||||||
Early
Texting - Web Link |
Mail Activity |
|
1st Mail (W) |
|
|
|
|
2nd Mail (Q) |
3rd Mail (Q) |
|
Email Activity |
|
Initial |
Reminder 1 |
Reminder 2 |
Reminder 3 |
Reminder 4 |
Reminder 5 |
Reminder 6 |
|
|
Telephone Activity |
|
|
Phone Reminder |
Phone Follow-Up |
||||||
Texting Activity |
Welcome Text |
Text 1 |
Text 2 |
|
Text 3 |
|
|
|
|
|
|
||||||||||
Late
Texting - Web Link |
Mail Activity |
|
1st Mail (W) |
2nd Mail (W) |
|
3rd Mail (Q) |
|
4th Mail (Q) |
|
|
Email Activity |
|
Initial |
Reminder 1 |
Reminder 2 |
Reminder 3 |
Reminder 4 |
Reminder 5 |
Reminder 6 |
|
|
Telephone Activity |
|
|
Phone Reminder |
Phone Follow-Up |
||||||
Texting Activity |
Welcome Text |
|
|
|
|
|
Text 1 |
Text 2 |
|
|
|
||||||||||
Late
Texting - Questions |
Mail Activity |
|
1st Mail (W) |
2nd Mail (W) |
|
3rd Mail (Q) |
|
4th Mail (Q) |
|
|
Email Activity |
|
Initial |
Reminder 1 |
Reminder 2 |
Reminder 3 |
Reminder 4 |
Reminder 5 |
Reminder 6 |
|
|
Telephone Activity |
|
|
Phone Reminder |
Phone Follow-Up |
||||||
Texting Activity |
Welcome Text |
|
|
|
|
|
Text 18 |
Text 2 |
|
Because the PFS does not have a web instrument for data collection, all text message contacts for the survey will be an interactive exchange with the principal. PFS data collection begins with contacts to the principal at the NTPS school and then shifts to contacts with the sampled principals directly using the personal contact information they provided on the NTPS if the school is not responsive.
To explore using text messaging as a contact method for PFS, the experiment will include two treatment groups:
Include an interactive questions text message contact to complete the survey concurrent with both principal-level mailouts
Replace the principal-level mailouts with an interactive questions text message contact to complete the survey
This experiment will help determine whether or not survey participants would be willing to respond to a survey via text, generally. In addition, results from this experiment will help determine whether a respondent is more likely to respond via a text exchange instead of returning a paper questionnaire or if the text serves as a reminder, prompting respondents to return their paper questionnaire.
Only principals who opt-in to receive text messages are eligible for inclusion in the text messaging experiment. The estimated opt-in rate for public and private school principals is 60%. The treatment for each principal will be randomly assigned prior to the start of PFS data collection. As such, the random assignment should result in even sample sizes and a similar distribution of school types across the treatment groups. The experimental contact plan and sample sizes are displayed in Exhibit 2.
Exhibit 2: 2021-22 PFS Text Message Experimental Contact Strategy and Sample Sizes
Experimental Treatment |
Planned Date |
School-level Contacts (PFS-1A) |
Principal-level Contacts (PFS-1C) |
|
||||||
3/04/22 |
3/18/22 |
4/11/22-4/29/22 |
Concurrent with Text 1 |
4/06/22,
4/20/22, |
5/18/22 - 5/20/22 |
5/23/22 - 6/10/22 |
|
|||
Standard/Control Group
1,322 public school principals 342 private school principals |
Mail Activity |
1st Mail |
2nd Mail |
|
|
1st Mail |
Reminder Mail |
|
|
|
Email Activity |
|
|
Reminder 1 |
Reminder 2 |
|
|
||||
Telephone Activity |
|
Telephone FU |
|
Telephone FU |
|
|||||
Texting Activity |
|
|
|
|||||||
|
||||||||||
Late Texting - Status question to determine attrition
1,322 public school principals 342 private school principals |
Mail Activity |
1st Mail |
2nd Mail |
|
|
|
||||
Email Activity |
|
|
|
|
Reminder 1 |
Reminder 2 |
|
|
||
Telephone Activity |
|
Telephone FU |
|
|
|
Telephone FU |
|
|||
Texting Activity |
|
Welcome Text |
Text 1 |
Text 2 |
|
|
||||
|
|
|
|
|
|
|
|
|
|
|
Mail & Texting - Status question to determine attrition
1,322 public school principals 342 private school principals |
Mail Activity |
1st Mail |
2nd Mail |
|
|
1st Mail |
Reminder Mail |
|
|
|
Email Activity |
|
|
Reminder 1 |
Reminder 2 |
|
|
||||
Telephone Activity |
|
Telephone FU |
|
Telephone FU |
|
|||||
Texting Activity |
|
Welcome Text |
Text 1 |
Text 2 |
|
|
Following data collection, analyses will be conducted at the treatment level (experimental group vs. control). The analyses examined include:
Response rate;
R-Indicators;
Average number of contacts; and
Days to respond.
The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the text message(s) had on a case’s likelihood to respond, given that case’s unique characteristics.
Given the project sample sizes in the table above, a statistically significant difference between the control group (no text messages) and any of the other three treatment groups will be found if the response rates between the two groups differ by:
8.63% for public school teachers (TFS)
18.48% for private school teachers (TFS)
6.26% for public school principals (PFS)
12.39% for private school principals (PFS)
R-indicators will be used to determine the overall balance of the respondent population, as well as within each experimental group. R-indicators will be calculated for the full sample, as well as variable-level and category-level partial R-indicators to determine which characteristics specifically are contributing to imbalance within the respondent population.
The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of one type of contact strategy over another. If cases within the experimental group receiving text messages respond in a more timely fashion, this could reduce the number of cases included in follow-up operations, allowing finite resources, to be spread across fewer cases.
Previous research has indicated that personalizing outreach materials can increase response rates, which has been used to inform procedures for communication materials (e.g., letters, emails) for past cycles of the NTPS, as well as the upcoming 2021-22 TFS and PFS collections.9,10,11 By and large, however, for these and other survey collections, generalized facts and figures have been included amongst materials in an effort to motivate sample participants to respond to the survey.
An experiment to include one teacher infographic, either a personalized infographic or a general infographic, in the second mailout package will be included in the 2021-22 TFS. Both types of infographics will present information with non-research audiences in mind, and the personalized infographics will provide facts from the NTPS that are more tailored to the individual sampled teacher versus the general teacher population. Since the 2021-22 TFS sample is drawn from the 2020-21 NTPS, and therefore data regarding the sampled teachers’ main teaching assignment are available prior to the beginning of the TFS, this experiment is a good fit for the TFS collection. By testing both general and the personalized infographics, we can also consider the value add of including general infographics to future NTPS mailed packages, since personalized teacher data is less available at that stage.
This experiment aims to address the following research questions:
Does the inclusion of any infographic affect response?
Does the inclusion of a personalized/targeted infographic affect response?
Does the inclusion of a general infographic affect response?
The general infographics will be developed by school classification. As a result, there will be two different variations of the general infographic:
Public school stayer/mover/leaver
Private school stayer/mover/leaver
The personalized infographics will be developed around both school classification and the 11 main teaching assignments that a teacher could select on their completed 2020-21 NTPS Teacher Questionnaire. In order for infographics to be truly personalized, there will also be different infographics for stayers/movers versus leavers. As a result, there will be 24 different variations of the personalized/targeted infographic:
Public school stayers/movers, by each of the 11 main teaching assignments
Private school stayers/movers, by each of the 11 main teaching assignments
Public school leavers
Private school leavers
The infographics will include three parts: (top) introductory text, (middle) statistics from the 2017-18 NTPS, and (bottom) call to action language.
The introductory text reminds participants of their participation in last year’s NTPS, informs them of the purpose of the TFS collection, and tells them how they can help make the data collection successful. This language will remain the same for all infographic conditions, and only minor personalization changes will be made for personalized leavers infographics (i.e., public school leavers and private school leavers) to address the fact that these are former teachers instead of current teachers.
The statistics are drawn from the 2017-18 NTPS public or private teacher data file, depending on the school type of the sampled participant. For the general infographics and the personalized stayer/mover infographics, the same items will be included. The statistics shared will be based on the targeted subgroup (i.e., for the public school and private school general infographics, or personalized for each of the two variations listed above by the eleven main teaching assignments). The personalized Leavers infographics (i.e., private school leavers and public school leavers) will contain the same items as one another, and differ slightly from the other personalized and general infographics, as a reflection of the fact that teachers who left the field and those who remained in the profession may be interested in different information.
The call to action language will remind participants how the TFS data are used and why it’s important that their experience be included in the survey results. Similarly to the introductory language, this language will remain the same for all infographic conditions, and only minor personalization changes will be made for leavers.
The treatment for each sampled teacher will be randomly assigned at the school-level at the time of teacher sampling, prior to the start of teacher data collection. The schools will be sorted by school size and other characteristics related to response. As such, the random assignment should result in even sample sizes and a similar distribution of school types across the treatment groups. The experimental treatments and expected sample sizes are displayed in Exhibit 3.
Exhibit 3: 2021-22 TFS Infographic Experiment Treatments and Sample Sizes
Conditions |
Treatment |
Sample Size |
Number of unique infographics made |
||||||||
Control |
No infographic included in the second teacher mail-out |
1,000 public school teachers 585 private school teachers |
0 infographics |
||||||||
Condition 1 |
One general infographic included in the second teacher mail-out (Appendix A, p.6) |
3,760 public school teachers 585 private school teachers |
1 infographic for public school teachers
1 infographic for private school teachers |
||||||||
Condition 2 |
One personalized/targeted infographic included in the second teacher mail-out (Appendix A, p.7-8) |
3,760 public school teachers 585 private school teachers |
24 infographics
Public School Teachers
Private School Teachers
|
Following data collection, analyses will be conducted at the treatment level (experimental group vs. control). The analyses examined include:
Response rate;
R-Indicators;
Average number of contacts; and
Days to respond.
The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the infographic had on a case’s likelihood to respond, given that case’s unique characteristics.
Given the project sample sizes in the table above, a statistically significant difference between the control group (no infographic) and one of the treatments will be found if the response rates between the two groups differ by 7.76% for public school teachers and 12.40% for private school teachers. A statistically significant difference between the two treatments will be found if the response rates differ by 5.03% for public school teachers and 10.74% for private school teachers.
R-indicators will be used to determine the overall balance of the respondent population, as well as within each experimental group. R-indicators will be calculated for the full sample, as well as variable-level and category-level partial R-indicators to determine which characteristics specifically are contributing to imbalance within the respondent population.
The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of one type of contact strategy over another. If cases within the experimental group receiving infographics respond in a more timely fashion, this could reduce the number of cases included in follow-up operations, allowing finite resources, to be spread across fewer cases.
Aaron Gilary and Alfred Meier of the U.S. Census Bureau, Jim Green of Westat, and Andy Zukerberg, Maura Spiegelman, and Julia Merlin of NCES reviewed and approved the TFS and PFS sample design and related matters for statistical quality, feasibility, and suitability to the overall objectives of the survey.
1 Elementary: teachers who taught elementary school students (any grade K-8, but at least one of the grades K-4) in the 2020-21 school year; Middle: teachers who taught middle school students (grades 5-8 exclusively) in the 2020-21 school year; High: teachers who taught high school students (any grade 7-12, but at least one of grades 9-12) in the 2020-21 school year.
2 The “A” version is for public schools; the “B” version is for private schools.
3 The “C” version is for public schools; the “D” version is for private schools.
4Tourkin, S., Parmer, R., Cox, S., & Zukerberg, A. (2005). (Inter) Net Gain? Experiments to Increase Response. Paper presented at the 60th Annual Conference of the American Association for Public Opinion Research (AAPOR), Miami Beach, FL, May.
5Zukerberg, A., Zotti, A., & Cox, S. (2019). Better Late Than Never? The Use of An Adaptive Incentive with Nonrespondents. Proceedings of the American Association for Public Opinion Research (AAPOR) Annual Conference, Toronto, Canada, May.
6 Zukerberg, A., Zotti, A., & Cox, S. (2019 Better Late Than Never? The Use of An Adaptive Incentive with Nonrespondents. Proceedings of the American Association for Public Opinion Research Annual Conference.
7 Priority Schools are schools with characteristics that correlate with a low propensity to respond and have an impact on estimates.
8 The Welcome Text will be sent at the time of Text 1.
9 Good, K. (1997). A study of factors affecting responses in electronic mail surveys. Dissertation, Western Michigan University, DAI, vol 58-10A pg. 3899, 119 pages
10 Carroll, S.H. & Zukerberg, A. (2013, November 4-6). Do Names Matter? Experiments Comparing Different Branding and Levels of Personally Identifiable Information in a Mail Questionnaire [Proceedings of a research conference]. Federal Committee on Statistical Methodology (FCSM), Washington, DC, U.S.
11 Dillman, D. A., Lesser, V., Mason, R., Carlson, J., Willits, F., Robertson, R., et al. (2007). Personalization of mail surveys for general public and populations with a group identity: Results from nine studies. Rural Sociology, 72, 632–646. doi:10.1526/003601107782638693.
File Type | application/vnd.openxmlformats-officedocument.wordprocessingml.document |
File Title | tfs omb supporting statement [Memorandum] |
Author | NCES |
File Modified | 0000-00-00 |
File Created | 2021-10-16 |