Part B - 2021-22 TFS PFS_revised

Part B - 2021-22 TFS PFS_revised.docx

FOLLOW-UP SURVEYS TO THE 2020-21 NTPS: 2021-22 Teacher Follow-Up Survey (TFS) and 2021-22 Principal Follow-Up

OMB: 1850-0617

Document [docx]
Download: docx | pdf


FOLLOW-UP SURVEYS TO THE 2020-21 NATIONAL TEACHER AND PRINCIPAL SURVEY (NTPS):

2021-22 Teacher Follow-Up Survey (TFS) and

2021-22 Principal Follow-Up Survey (PFS)




PART B



OMB# 1850-0617 v.4



National Center for Education Statistics (NCES)

U.S. Department of Education






March 2021

revised May 2021











TABLE OF CONTENTS





PART B. COLLECTION OF INFORMATION EMPLOYING STATISTICAL METHODS


This request is to conduct data collection for the two follow-up surveys to the 2020-21 National Teacher and Principal Survey (NTPS) – the 2021-22 Teacher Follow-up Survey (TFS) and the 2021-22 Principal Follow-up Survey (PFS).


Section B.1 of this document describes the universe, sample design, and estimation details for the 2021-22 TFS and PFS. Section B.2 describes the data collection procedures for both surveys. Section B.3 discusses methods to secure cooperation and mitigate nonresponse. In particular, it describes methods used to improve response rates in the 2020-21 NTPS and how those methods will be used in its two follow-up surveys. Section B.4 describes recent developments in a long history of tests of methods and procedures to improve data quality. Section B.5 lists the names of those involved in the design of the study and the development of these materials.


B.1 Respondent Universe and Sample Design

B.1.1 Background on the 2020-21 NTPS Sample

The 2020-21 National Teacher and Principal Survey (2020-21 NTPS) sample was a two-stage stratified sample. The schools were sampled first, and then teachers were selected within each sampled school.


The NTPS school sample is a systematic probability proportionate to size (PPS) sample, where size is defined to be the square root of the full-time equivalent number of teachers at the school. Unlike its predecessor, the School and Staffing Survey (SASS), the NTPS does not stratify schools prior to sampling. However, some schools are oversampled based on their assigned domains. The public school oversampling domains are defined by charter status (charter/not charter), grade level (elementary/middle/high/combined), urbanicity (city/suburban/town/rural), and poverty status (> 75% of enrollment eligible for free or reduced price lunch/otherwise). The private school oversampling domains are defined by affiliation (11 levels), school span (Elementary/Secondary/Combined), school size (less than 100 students, or 100+ students), and Census division (the Mid-Atlantic states vs. other regions).


Public charter schools were oversampled at a rate 3.26 times their proportional sample allocation. Proportional allocation in this instance is defined as the domain’s sum of schools’ measures of size relative to the sum of measures of size of all schools in the sampling frame. Further oversampling applied to non-charter schools by grade level, whereby combined schools were oversampled at 2.56 times the elementary allocation, middle schools were oversampled at 1.30 times the elementary allocation, and high schools were oversampled at 1.35 times the elementary allocation. In addition, once the grade level sampling differential was applied, non-charter schools had their probabilities further altered by urbanicity as follows: city schools were oversampled at a rate of 1.05 times the proportional allocation (equal to the suburban sampling rate), town schools were oversampled at a rate 1.34 times the suburban rate, and rural schools were undersampled at a rate of 1.11 times the suburban rate.


In addition to oversampling by characteristic, schools in thirty (generally smaller, based on population size) states have relatively higher sampling rates to realize lower bounds on precision for these states and to enable the calculation of reliable key state estimates.


Along with the names of employed teachers and their email addresses, sampled schools were asked to provide the following descriptive characteristics of each teacher: teaching status, where options were either part-time or full-time; and subject matter taught, where teachers were classified as special education, general elementary, math, science, English/language arts, social studies, vocational/technical, or other.


The above information for each teacher in a selected NTPS school comprised the initial teacher sampling frame. The frame was also supplemented with information from vendor-purchased teacher roster files and from a clerical look-up operation conducted as part of NTPS where teacher lists were located on school and/or district websites. Within each sampled school, teachers were stratified by subject. The strata include math, science, English/language arts, social studies, and everything else. No oversampling was done for the teacher sample, beyond the oversampling inherited from their schools.


Teacher records within a school domain and teacher stratum were sorted by the teacher subject and the teacher line number code, which is a unique number assigned to identify the teacher within the list of keyed teachers. Within each teacher stratum in each school, teachers were selected systematically with equal probability.


Table B1 below shows a summary of the expected 2020-21 NTPS allocated sample sizes.


Table B1. Summary of the expected 2020-21 NTPS sample sizes


School

Teacher

Traditional Public

8,660

43,460

Public Charter

1,260

5,790

Private

3,000

6,300

Total

12,920

55,550


B.1.2 2021-22 TFS Respondent Universe and Sample Design

The 2021-22 Teacher Follow-up Survey (2021-22 TFS) sample will be a subsample from the 2020-21 NTPS teacher sample. The 2021-22 TFS sample is a random subsample from the 2020-21 NTPS teacher sample, and, as such, a probability sample similar to 2020-21 NTPS.


A planned total of 10,273 teachers will be selected according to a stratified design similar to earlier TFS collections and based on information obtained from the NTPS schools about the status of the 2020-21 NTPS respondent teachers in the 2021-22 school year. The 2021-22 TFS sample will include former teachers (NTPS teachers who have left the teaching profession, or “Leavers”), current teachers (NTPS teachers who have remained in the teaching profession, or “Stayers”), and teachers who have left their 2020-21 school, but for whom there is no information available (the ‘unknown’ group). The current teachers group is further stratified into two groups: those who remained in the school they were teaching in 2020-21 (“Stayers”) and those who moved to different schools (“Movers”). This design provides analytic data on teachers who stay, move, or leave the teaching profession. The status for each TFS teacher is determined through a Teacher Status Form (TFS-1) mailed to schools in the fall of 2021. For the 2021-22 TFS, the Movers, Leavers, and Unknown teachers will be sampled with certainty, as will all Stayers in their first year.


Teachers will be further stratified by factors known about the teacher from the NTPS year, including sector (traditional public, public charter, and private), grade level (elementary, middle, and high)1, experience level (1 year, 2-5 years, and 6+ years of teaching), and race/ethnicity (Hispanic, Black non-Hispanic, and other). Grade level is based on the grade level the teacher is teaching, but school grade level is used as the second choice if the teacher teaches in more than one grade level.


Within the explicit strata, there will be systematic sampling with a designated sort order. The sort variables include measure of size (teacher base weight), teacher subject, Census region, urbanicity, school enrollment, and for private school teachers, type of private school.

Teachers will be selected within each stratum using a probability proportional to size sampling procedure. The measure of size will be the 2020-21 NTPS final teacher weight. This essentially ‘reverses’ the probability of selection of the teacher in NTPS, and results in a self-weighting sample within each teacher stratum. This will maximize the efficiency of the TFS sample.


The table below presents target sample sizes for important teacher categories. Note that charter school and private school teachers are oversampled relative to traditional public school teachers. This will be done to reduce the coefficient of variation (CV) for charter school estimates to approximately 20 percent. Movers, leavers, and teachers with unknown status are all selected, which makes them overrepresented in the sample relative to other teachers. This is done to increase the precision for analyses of these important subgroups. The breakdown of stayers, movers, leavers, and ‘unknown’ teachers in public, charter, and private schools is as shown in Table B2.


Table B2. Distribution of teacher types across school types

Status

Sector

Public

Charter

Private

Total

Stayer

2,810 (0.093)

538 (0.264)

1,026 (0.269)

4,347

Mover

2,214

205

246

2,665

Leaver

1,434

107

278

1,819

Unknown

1,029

182

204

1,415

Total

7,487

1,032

1,754

10,273

Note: The number in parentheses is the estimated sampling rate for each Stayer cell.


Within these target sample sizes, Hispanic and Black Non-Hispanic stayers are oversampled by 60 percent to optimize the reliability of their comparison with other stayers (Movers, Leavers, and Unknown are already sampled with certainty). The target sample sizes at the status/sector/race level are as seen in Table B3.


Using these target sample sizes, the sample is allocated to the sampling strata, which are defined by the teacher’s sector, status, experience, grade, and race categories. This is done proportional to the cumulative teacher measure of size within each stratum. The teacher measure of size will be the 2020-21 NTPS teacher base weight.


Table B3. Distribution of teacher types across school types

Status

Sector

Total

Public

Charter

Private

Hispanic

Black Non-Hispanic

Other

Hispanic

Black Non-Hispanic

Other

Hispanic

Black Non-Hispanic

Other

Stayer

258

252

2,300

106

110

322

209

166

651

4,374

Mover

116

126

1,972

22

20

163

14

20

212

2,665

Leaver

44

67

1,323

8

11

88

10

10

258

1,819

Unknown

79

89

861

22

28

132

19

20

165

1,415

Total

497

534

6,456

158

169

705

284

216

1,286

10,273



B.1.3 2021-22 PFS Respondent Universe and Sample Design

The sampling frame for the 2021-22 PFS consists of all the traditional public and public charter school principals who completed a Principal Questionnaire and all private school principals who completed a Private School Principal Questionnaire during the 2020-21 NTPS. Any sampled NTPS principal who did not complete their questionnaire or was otherwise found to be out of scope for NTPS will not be included in the PFS frame. The 2020-21 PFS sample will include approximately 6,700 public and public charter school principals and 1,750 private school principals.


The 2016-17 PFS included a sample of 5,700 principals from public and charter schools. The large discrepancy in sample size between the 2016-17 and 2021-22 administrations of the PFS is due to the fact that the 2015-16 NTPS included fewer sampled schools than the 2020-21 NTPS, because the 2015-16 NTPS sample included public school principals only and produced estimates at the national rather than state level.


B.2 Procedures for the Collection of Information

B.2.1 2021-22 TFS Procedures for the Collection of Information


B.2.1.1 Teacher Status Form (TFS-1)

The TFS-1 will be mailed to all schools in which a teacher completed a Teacher Questionnaire during the 2020-21 administration of NTPS. A knowledgeable person at the school, such as the principal, will be asked to complete the TFS-1 by indicating the current teaching status of each teacher listed on the form. These data will be used to stratify the teachers for 2021-22 TFS sampling into groups of “stayers” (still teaching at the same school), “movers” (still teaching, but at a different school), or “leavers” (no longer teaching).


TFS was designed from its inception to sample only from respondents to SASS/NTPS in the prior year. It is possible to sample teachers for TFS from the pool of teachers initially selected for NTPS rather than only from those who responded to NTPS. By not sampling teachers from the nonresponse stratum, potential biases may be introduced into TFS. When considering unit nonresponse for SASS, there was no evidence to point to a substantial bias in teacher response rates.


The main purpose of TFS is to measure various characteristics collected on NTPS of teachers who move, leave, or stay in the teaching profession the following year. If NTPS nonrespondents were to be contacted for the TFS, there are several methodological problems that would need to be solved: 1) there would be no prior year data, so all of the NTPS teacher questionnaire items would have to be asked retrospectively of the teacher before the TFS could be administered; 2) if the teacher was determined to be out-of-scope for NTPS, then there would be no TFS interview and a sampling adjustment would be required; and 3) if the teacher was in-scope for NTPS but out-of-scope for TFS (if the teacher moves out of the country or dies), then that would also require some adjustment of the sampling weights. Logistically, trying to combine prior-year NTPS interviewing with current-year TFS interviewing is a complication that would delay the field data collection because it would likely take longer to complete the nonresponding NTPS cases. For these reasons, TFS teachers will be sampled from responding NTPS teachers only. A response rate of approximately 99 percent can be expected from the TFS-1 based on the prior TFS data collection cycle.


B.2.1.2 Questionnaires for Former and Current Teachers (TFS-2 and TFS-3)

Using the teacher statuses provided by schools on the TFS-1, Census Bureau staff will select the sample of teachers for the 2021-22 TFS and prepare materials for data collection.


In the 2020-21 NTPS, teachers were asked to provide the following information:


Name

Street address

Cell phone number

Home phone number

Most convenient day(s) to reach her/him

Work email address

Home email address


Prior to the start of data collection, Census staff will conduct research to obtain contact information (e.g., home address, email address, and/or telephone number) for teachers who provided incomplete or no contact information in the NTPS interview. Staff will first use any contact information provided (e.g., telephone number, partial address, contact person’s information) from the NTPS questionnaire to attempt to complete the address of the sample member. If no contact information was provided, staff will use sources such as the white or yellow pages, Internet searches, and other address and phone number databases.


Data collection for the Questionnaire for Former Teachers (TFS-2) and the Questionnaire for Current Teachers (TFS-3) will begin in January 2022. The first contact with sample members will be through an initial contact letter sent by mail inviting them to participate in the survey and providing credentials for them to log on and complete the appropriate web-based survey. All teachers will also receive a $5 or $10 cash incentive in this initial mailed package. Note that the monetary ($5 or $10 cash) incentives for teachers will be adhered to a piece of yellow cardstock using removable sticky glue and will be inserted with the letter. The cardstock will be ½ sheet rather than a full sheet and will include text thanking them for their participation in the study. Adhering the case to the cardstock insert will (1) increase the weight of the envelope, making it feel more “substantial” and important; (2) prevent the money from free-floating inside the envelope; and (3) help ensure that the respondent notices the cash. Concurrently, Census will send an email invitation to participate (unless a valid email address is unavailable) to the sample member with a link to the survey and their log in credentials. If home addresses were not provided on the 2020-21 NTPS teacher questionnaire or cannot be obtained, the NTPS school address will be used as the address. Each letter and email will be customized and will provide the sampled teacher with login information to access the web-based instrument.


In late January, Census will send nonrespondents a second teacher mailout and first email reminder, and in mid-February a second reminder only by email. For a subsample of teachers, this package will also include an experimental infographic. Refer to section B.4.2 for additional information about testing the inclusion of an infographic in TFS packages.


The third mailout and third email reminder will be sent to nonrespondents in early March. This mailing will include a paper copy of the applicable TFS questionnaire, as will the fourth mailed package in mid-April and fifth (and final mailed contact) mailed package in mid-May. A fourth reminder by email only will be sent in late March, followed by a fifth and sixth reminder by email, which will be sent alongside the fourth and fifth mailed packages. Data collection will end in early July 2022. Throughout data collection, research will be conducted, as needed, to find current addresses and emails for sampled teachers whose letters and emails are returned as undeliverable.


In addition, ongoing support will be provided for sampled teachers to whom the incorrect paper questionnaire type was mailed (e.g., a mover who was incorrectly classified as a leaver and received a TFS-2 questionnaire as a result). When these cases are discovered, either because the teacher was Amish or Mennonite or in response to the late mailing(s) of the paper questionnaires, Census will send the correct form for completion.


From February through June, outbound telephone calls to remind teachers who have not yet responded to the web-based collection to complete their form. While the call is intended to encourage teachers to log on and respond to the survey themselves, interviewers will be able to administer it over the phone at the request of the respondent. Throughout the data collection period, respondents who request a paper questionnaire will be mailed one.


The 2021-22 TFS will include an experiment to test different methods of texting survey participants as a means of exploring the use of text messages as a new data collection mode. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS teacher questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number.


To explore using test messaging as a contact method for TFS, the experiment will include three treatment groups:

  1. Replace the second mailout (web invitation letter) and third mailout (paper questionnaire and letter) with text message contacts that include the link to complete the TFS online

  2. Include a text message contact that includes a link to complete the TFS online concurrent with the fourth mailout (second mailed paper questionnaire), and replace the fifth mailout (third mailed paper questionnaire) with a text message contact that includes a link to complete the TFS online

  3. Include an interactive questions text message contact at the time of the fourth mailout (second mailed paper questionnaire) and replace the fifth mailout (third mailed paper questionnaire with an interactive questions text message contact


For groups one and two, the text message will invite the teacher to complete the TFS questionnaire online by providing the link to the web instrument in the text message. For group three, questions about current teaching status will be texted directly to each sampled teacher and his or her response will be collected via their text messaged response. These interactive texts will follow the same general skip pattern as the web instrument, where subsequent texts will ask follow-up questions based on the teacher’s responses. Results from this experiment will help determine whether a respondent is more likely to respond via a texted survey web link earlier in data collection versus later in data collection, as well as whether a respondent is more likely to respond via a texted survey web link or directly in an interactive text exchange later in data collection. For additional details about testing the use of text messaging in the 2021-22 TFS, refer to section B.4.1 of this document.


A response rate of approximately 75 percent (base-weighted) can be expected from the leavers after follow up, and a response rate of about 81 percent (base-weighted) can be expected from stayers and movers after follow-up, based on the prior administration of TFS. Table B4 shows weighted unit response rates in the last administration of TFS, TFS 2012-13.


Table B4. Weighted unit and overall response rates of TFS teachers, using initial base weight, by sector and teaching status: 2012–13

Sector and teaching status

Weighted unit response rate

Weighted overall unit response rate

Public

80.7

49.9

Current teacher

81.3

50.3

Stayer

81.6

50.5

Mover

77.9

48.2

Former teacher

73.8

45.6

Private

79.3

39.7



B.2.2 2021-22 PFS Procedures for the Collection of Information

In January 2022, Census Bureau staff will mail the PFS-1(A/B)2 to all schools to collect the current occupational status of the 2020-21 principal. Non-responding schools will be sent a reminder letter along with a second copy of the PFS-1 (A/B) approximately two weeks after the initial mailing. Those schools that are still non-responsive three weeks after Mailing 2 will receive a telephone call during which telephone center staff will follow-up by telephone to collect the occupational status of the 2020-21 principal. The telephone center staff will first ask school office staff for the occupational status of the 2020-21 principal. If the office staff is unable to provide the information, telephone center staff will ask to speak with the current school principal.


For non-responding schools or schools that were unable to report the principal’s status, Census Bureau staff will mail a modified version of the Principal Status Form (PFS-1(C/D)3) to the 2020-21 principal at his or her home address (if reported in NTPS). The modifications tailor the form to the previous years’ principal rather than the school staff. This mailing will begin in March 2022, and an email will be sent a few days following the mailing, alerting the principal to the mailing and reminding them to complete their PFS. Non-responding principals will be sent a follow-up mailing and email approximately 2 weeks after the initial mailing and email. Telephone center staff will follow-up by telephone to collect the occupational status of the non-responding principal beginning three weeks after the reminder mailing and email are sent.

The 2021-22 PFS will include an experiment to test different methods of texting survey participants as a means of exploring the use of text messages as a new data collection mode. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS principal questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number. Because the PFS does not have a web instrument for data collection, all text message contacts for the survey will be an interactive exchange with the principal.


To explore using text messaging as a contact method for PFS, the experiment will include two treatment groups:

  1. Include an interactive questions text message contact to complete the survey concurrent with both principal-level mailouts

  2. Replace the principal-level mailouts with an interactive questions text message contact to complete the survey


This experiment will help determine whether or not survey participants would be willing to respond to a survey via text, generally. In addition, results from this experiment will help determine whether a respondent is more likely to respond via a text exchange instead of returning a paper questionnaire or if the text serves as a reminder, prompting respondents to return their paper questionnaire. For additional details about testing the use of text messaging in the 2021-22 PFS, refer to section B.4.1 of this document.


A response rate in the approximate range of 90-95 percent is expected based on the prior administration of PFS.


B.3 Methods for Maximizing Response

This section describes the methods that NCES will use to secure cooperation, maximize response rates, and deal with nonresponse for the 2021-22 NTPS follow-up surveys. The entire survey process, starting with securing research cooperation from key groups and individual sample members and continuing throughout the distribution and collection of individual questionnaires, is designed to increase survey response rates. In addition, NCES believes that the following endorsements, stressing the survey’s importance in communication materials, and extensive follow-up with nonrespondents will further facilitate the overall success of the survey and enhance response rates.

B.3.1 Endorsements from Key Public and Private School Groups

The level of interest and cooperation demonstrated by key groups can often greatly influence the degree of participation of survey respondents. Endorsements are viewed as a critical factor in soliciting cooperation from state and local education officials.

NCES will seek endorsements for the 2021-22 TFS from the following organizations or agencies:

Agudath Israel of America/Lefkowitz Leadership Initiative

American Association of Christian Schools

American Association of School Administrators

American Federation of Teachers

American Montessori Society

Association for Middle Level Education

Association for Middle Level Education (formerly National Middle School Association)

Association for Supervision and Curriculum Development

Association Montessori International

Association of American Educators

Association of American Educators

Association of Christian Schools International

Association of Christian Teachers and Schools

Association of Classical Christian Schools

Association of Military Colleges and Schools

Association of Waldorf Schools of North America

Christian Schools International

Council for American Private Education

Council of Chief State School Officers

Council of Islamic Schools of North America

Council of the Great City Schools

Evangelical Lutheran Church in America

Friends Council on Education

General Conference of Seventh-Day Adventists

Islamic School League of America

Jesuit Schools Network

Jewish Education Services of North America

Lutheran Church-Missouri Synod

National Association of Elementary School Principals

National Association of Episcopal Schools

National Association of Independent Schools

National Association of Private Special Education Centers

National Association of Secondary School Principals

National Catholic Educational Association

National Christian School Association

National Coalition of Girls’ Schools

National Council for Private School Accreditation

National Education Association

National Independent Private School Association

National Parent Teacher Association

Office of Education, General Conference of Seventh Day Adventists

Oral Roberts University Educational Fellowship

Prizmah: Center for Jewish Day Schools

RAVSAK: Jewish Community Day School Network

Southern Baptist Association of Christian Schools

The Association of Boarding Schools

The Jewish Education Project

The School Superintendents Association

Torah Umesorah National Society for Hebrew Day Schools

United States Conference of Catholic Bishops

Wisconsin Evangelical Lutheran Synod


NCES will seek endorsements for the 2021-22 PFS from the following organizations or agencies:

American Association of School Administrators

American Federation of Teachers

National Association of Elementary School Principals

National Association of Secondary School Principals


B.3.2 Stressing the Importance of the Survey and the Respondents’ Participation in it

Official letters will be used to motivate respondents to return surveys. Respondent letters for both the 2021-22 TFS and PFS will be sent by the U.S. Census Bureau and signed by the NCES Commissioner. Communications in the form of both letters and emails will be personalized with the teacher and principals’ names, which is expected to have positive effects on the survey response rates.


B.3.3 Offering a cash incentive to current and former teachers

The 2004-05 TFS used monetary incentives to increase overall response and internet response when both mail and internet choices were offered. The results showed that the overall response of the incentive groups significantly exceeded that of the non-incentive groups.4 Additionally, the results of the use of teacher incentives for the 2017-18 NTPS indicated that the teacher incentive led to significant increases in the response rate for both public and private school teachers.5 In addition, the average number of days to complete the questionnaire was significantly lower for public school teachers that received the incentive. Finally, the incentive helped increase the overall sample balance for teachers in both public and private schools.


Due to these favorable results, the 2020-21 NTPS included the use of a cash incentives in the initial mailed packages to a subsample of early sampled teachers and as a promised incentive, contingent upon a teacher completing his or her teacher questionnaire, for teachers sampled in December and later. Mid-data collection, a switch to promised incentives was made in an attempt to encouraging teachers to respond to the survey, given the challenges posed during the 2020-21 school year by the coronavirus pandemic. As such, the 2021-22 TFS will include a cash incentive for sampled teachers. The incentives included in the TFS will not be offered in an experimental manner and will instead be offered to all sampled current and former teachers.


Data on teacher retention and attrition are in high demand from researchers and policymakers, but TFS data have not been released since 2012-13. The TFS was last fielded in 2012-13, after the 2011-12 SASS. During the 2012-13 TFS, response rates from private school teachers was problematically low, and NCES was unable to publish a TFS private school teacher data file. In 2016-17, after the 2015-16 NTPS, the overall unit response rate of the TFS 2016-17 was likely to be under 50 percent and therefore the entire collection was cancelled. In an effort to increase response rates for the 2021-22 cycle, a contingency incentive may also be used in later TFS mailings to combat low response, as identified in weekly status reports. A similar contingency incentive was included with the 2017-18 NTPS and led to a boost in response rate for teacher who received this incentive.6


At the end of week 7 of data collection (2/25/2022), we will evaluate the response rates for the TFS-2 (former teachers) and TFS-3 (current teachers) and determine whether a contingency incentive is needed. Specifically, if the TFS-2 response rate is less than 31%, those teachers will be eligible to receive a contingency incentive.  If the TFS-3 response rate is less than 38%, those teachers will be eligible to receive a contingency incentive. Depending on whether response rates are struggling overall or for certain types of schools, the contingency plan may be exercised for teachers in all types of schools or only teachers sampled from schools with particular characteristics (e.g., priority schools7). The amount of the contingency incentive will be $10 or $15 based on NTPS priority status, where teachers from priority schools will receive a larger incentive. This will be a one-time, promised cash incentive to be mailed directly to the responding teacher later in the school year.


B.3.4 Extensive Follow-up (by mail, email, and telephone) of Non-Respondents

For the 2021-22 TFS, the Census Bureau will use a variety of techniques to increase response levels including:


(a) Using mixed survey modes— self-administered web-based, telephone interviewer-assisted web-based, and self-administered mail instruments, as needed, to maximize response levels.


(b) Allocating adequate time and resources to respondent tracking efforts to ensure that a high percentage of movers and leavers are successfully located and surveyed. A variety of techniques will be employed to locate survey respondents. Potential tracking sources include: (a) leads provided by school principals or their designees; (b) clerical operations using electronic databases and internet searches; and (c) post office for possible forwarding addresses.


Good questionnaire design techniques will be employed to minimize item non-response. All completed questionnaires from TFS 2012-13 were analyzed to determine which items had the highest levels of item nonresponse. This information guided NCES and the Census Bureau in reviewing the clarity of item wording, definitions, and instructions, and as such, revisions were made to TFS content for the TFS 2016-17. The TFS 2016-17 was approved by OMB (#1850-0934 v1); however, the data collection was cancelled due to lower than anticipated response rates for subdomains of 2015-16 NTPS teachers. Part C of this submission details the minor changes between the proposed TFS instruments for 2021-22 and those approved by OMB for the TFS 2016-17.


To permit sufficient time to locate sampled members who left their teaching position in the previous year and thorough follow-up of non-respondents, the TFS teacher data collection will begin in January 2022, allowing six months for all data collection activities related to teachers.


For the PFS, the Census Bureau will also utilize mixed survey modes – self-administered mail instruments and telephone, as needed, to maximize response levels. Non-respondents will receive at least two reminders. Given the brevity of the instrument, we anticipate response rates at or above 90% with this approach, as with past cycles of the survey.


B.4 Tests of Procedures and Methods

NCES is currently considering options for tests of methods, materials, and procedures to be conducted as part of the 2021-22 NTPS Follow-up Surveys, with the goal to increase response in the 2020-21 collection. Each test is described in detail below.


B.4.1 Testing the use of text messages in both TFS and PFS

For 2021-22 data collection cycle of the TFS and PFS, NCES will explore the use of text messaging as a new data collection mode by including an experiment to test different methods of texting survey participants. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS principal and teacher questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number.


To explore using test messaging as a contact method for TFS, the experiment will include three treatment groups:

  1. Replace the second and third web invitation letter mailouts with text message contacts that include the link to complete the TFS online

  2. Include a text message contact that includes a link to complete the TFS online concurrent with the fourth mailout (the first paper questionnaire mailing), and replace the fifth mailout with a text message contact that includes a link to complete the TFS online

  3. Include an interactive questions text message contact at the time of the fourth mail-out and replace the fifth mail-out with an interactive questions text message contact


For groups one and two, the text message will invite the teacher to complete the TFS questionnaire online by providing the link to the web instrument in the text message. For group three, questions about current teaching status will be texted directly to each sampled teacher and his or her response will be collected via their text messaged response. These interactive texts will follow the same general skip pattern as the web instrument, where subsequent texts will ask follow-up questions based on the teacher’s responses. Results from this experiment will help determine whether a respondent is more likely to respond via a texted survey web link earlier in data collection versus later in data collection, as well as whether a respondent is more likely to respond via a texted survey web link or directly in an interactive text exchange later in data collection.


Only teachers who opt-in to receive text messages are eligible for inclusion in the text messaging experiment. The estimated opt-in rate for public and private school teachers is 60%. The treatment for each sampled teacher will be randomly assigned at the school-level at the time of teacher sampling, prior to the start of teacher data collection. Any school where at least one teacher consented to receiving text messages in the experiment is eligible to be included. The schools will be sorted by school size and other characteristics related to response. As such, the random assignment should result in even sample sizes and a similar distribution of school types across the treatment groups. Only schools where all teachers did not consent to receiving text messages would be considered ineligible for the experiment. If a school is selected to receive text messages, all sampled teachers who had opted-in would receive the same text message treatment, and any sampled teachers who did not consent would not receive any text messages as part of the experiment. The teachers in the sampled school who opted out of text messaging would receive the traditional mail contacts. Nonrespondent teachers may not be substituted with other sampled teachers. The experimental contact plan and expected sample sizes are displayed in Exhibit 1.

Exhibit 1: 2021-22 TFS Text Message Experimental Contact Strategy and Sample Sizes

Experimental Treatment and Sample Size

Planned Date

Concurrent with Text 1

1/10-1/12

1/28-1/31

2/16

3/07-3/08

3/31

4/15-4/19

5/20-5/23

6/10

Standard/Control Group

1,278 public school teachers
263 private school teachers

Mail Activity

 

1st Mail (W)

2nd Mail (W)

 

3rd Mail (Q)

 

4th Mail (Q)

5th Mail (Q)

 

Email Activity

 

Initial

Reminder 1

Reminder 2

Reminder 3

Reminder 4

Reminder 5

Reminder 6

 

Telephone Activity

 

 

Phone Reminder

Phone Follow-Up

Texting Activity

 

 

 

Early Texting - Web Link

1,278 public school teachers
263 private school teachers

Mail Activity

 

1st Mail (W)

 

 

 

 

2nd Mail (Q)

3rd Mail (Q)

 

Email Activity

 

Initial

Reminder 1

Reminder 2

Reminder 3

Reminder 4

Reminder 5

Reminder 6

 

Telephone Activity

 

 

Phone Reminder

Phone Follow-Up

Texting Activity

Welcome Text

Text 1

Text 2

 

Text 3

 

 

 

 

 

Late Texting - Web Link

1,278 public school teachers
263 private school teachers

Mail Activity

 

1st Mail (W)

2nd Mail (W)

 

3rd Mail (Q)

 

4th Mail (Q)

 

 

Email Activity

 

Initial

Reminder 1

Reminder 2

Reminder 3

Reminder 4

Reminder 5

Reminder 6

 

Telephone Activity

 

 

Phone Reminder

Phone Follow-Up

Texting Activity

Welcome Text

 

 

 

 

 

Text 1

Text 2

 

 

Late Texting - Questions

1,278 public school teachers
263 private school teachers

Mail Activity

 

1st Mail (W)

2nd Mail (W)

 

3rd Mail (Q)

 

4th Mail (Q)

 

 

Email Activity

 

Initial

Reminder 1

Reminder 2

Reminder 3

Reminder 4

Reminder 5

Reminder 6

 

Telephone Activity

 

 

Phone Reminder

Phone Follow-Up

Texting Activity

Welcome Text

 

 

 

 

 

Text 18

Text 2

 



Because the PFS does not have a web instrument for data collection, all text message contacts for the survey will be an interactive exchange with the principal. PFS data collection begins with contacts to the principal at the NTPS school and then shifts to contacts with the sampled principals directly using the personal contact information they provided on the NTPS if the school is not responsive.


To explore using text messaging as a contact method for PFS, the experiment will include two treatment groups:


  1. Include an interactive questions text message contact to complete the survey concurrent with both principal-level mailouts

  2. Replace the principal-level mailouts with an interactive questions text message contact to complete the survey

This experiment will help determine whether or not survey participants would be willing to respond to a survey via text, generally. In addition, results from this experiment will help determine whether a respondent is more likely to respond via a text exchange instead of returning a paper questionnaire or if the text serves as a reminder, prompting respondents to return their paper questionnaire.


Only principals who opt-in to receive text messages are eligible for inclusion in the text messaging experiment. The estimated opt-in rate for public and private school principals is 60%. The treatment for each principal will be randomly assigned prior to the start of PFS data collection. As such, the random assignment should result in even sample sizes and a similar distribution of school types across the treatment groups. The experimental contact plan and sample sizes are displayed in Exhibit 2.



Exhibit 2: 2021-22 PFS Text Message Experimental Contact Strategy and Sample Sizes

Experimental Treatment

Planned Date

School-level Contacts (PFS-1A)

Principal-level Contacts (PFS-1C)


3/04/22

3/18/22

4/11/22-4/29/22

Concurrent with Text 1

4/06/22, 4/20/22,
5/04/22, 5/18/22

5/18/22 - 5/20/22

5/23/22 - 6/10/22


Standard/Control Group


1,322 public school principals

342 private school principals

Mail Activity

1st Mail

2nd Mail

 

 

1st Mail

Reminder Mail

 


Email Activity

 

 

Reminder 1

Reminder 2

 


Telephone Activity

 

Telephone FU

 

Telephone FU


Texting Activity

 

 


 

Late Texting - Status question to determine attrition


1,322 public school principals

342 private school principals

Mail Activity

1st Mail

2nd Mail

 

 


Email Activity

 

 

 

 

Reminder 1

Reminder 2

 


Telephone Activity

 

Telephone FU

 

 

 

Telephone FU


Texting Activity

 

Welcome Text

Text 1

Text 2

 


 


 

 

 

 

 

 

 


Mail & Texting - Status question to determine attrition


1,322 public school principals

342 private school principals

Mail Activity

1st Mail

2nd Mail

 

 

1st Mail

Reminder Mail

 


Email Activity

 

 

Reminder 1

Reminder 2

 


Telephone Activity

 

Telephone FU

 

Telephone FU


Texting Activity

 

Welcome Text

Text 1

Text 2

 




Following data collection, analyses will be conducted at the treatment level (experimental group vs. control). The analyses examined include:

  • Response rate;

  • R-Indicators;

  • Average number of contacts; and

  • Days to respond.


The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the text message(s) had on a case’s likelihood to respond, given that case’s unique characteristics.

Given the project sample sizes in the table above, a statistically significant difference between the control group (no text messages) and any of the other three treatment groups will be found if the response rates between the two groups differ by:

  • 8.63% for public school teachers (TFS)

  • 18.48% for private school teachers (TFS)

  • 6.26% for public school principals (PFS)

  • 12.39% for private school principals (PFS)

R-indicators will be used to determine the overall balance of the respondent population, as well as within each experimental group. R-indicators will be calculated for the full sample, as well as variable-level and category-level partial R-indicators to determine which characteristics specifically are contributing to imbalance within the respondent population.

The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of one type of contact strategy over another. If cases within the experimental group receiving text messages respond in a more timely fashion, this could reduce the number of cases included in follow-up operations, allowing finite resources, to be spread across fewer cases.

B.4.2 Testing the inclusion of an infographic in TFS packages

Previous research has indicated that personalizing outreach materials can increase response rates, which has been used to inform procedures for communication materials (e.g., letters, emails) for past cycles of the NTPS, as well as the upcoming 2021-22 TFS and PFS collections.9,10,11 By and large, however, for these and other survey collections, generalized facts and figures have been included amongst materials in an effort to motivate sample participants to respond to the survey.


An experiment to include one teacher infographic, either a personalized infographic or a general infographic, in the second mailout package will be included in the 2021-22 TFS. Both types of infographics will present information with non-research audiences in mind, and the personalized infographics will provide facts from the NTPS that are more tailored to the individual sampled teacher versus the general teacher population. Since the 2021-22 TFS sample is drawn from the 2020-21 NTPS, and therefore data regarding the sampled teachers’ main teaching assignment are available prior to the beginning of the TFS, this experiment is a good fit for the TFS collection. By testing both general and the personalized infographics, we can also consider the value add of including general infographics to future NTPS mailed packages, since personalized teacher data is less available at that stage.


This experiment aims to address the following research questions:

  1. Does the inclusion of any infographic affect response?

  2. Does the inclusion of a personalized/targeted infographic affect response?

  3. Does the inclusion of a general infographic affect response?


The general infographics will be developed by school classification. As a result, there will be two different variations of the general infographic:

  • Public school stayer/mover/leaver

  • Private school stayer/mover/leaver


The personalized infographics will be developed around both school classification and the 11 main teaching assignments that a teacher could select on their completed 2020-21 NTPS Teacher Questionnaire. In order for infographics to be truly personalized, there will also be different infographics for stayers/movers versus leavers. As a result, there will be 24 different variations of the personalized/targeted infographic:


  • Public school stayers/movers, by each of the 11 main teaching assignments

  • Private school stayers/movers, by each of the 11 main teaching assignments

  • Public school leavers

  • Private school leavers


The infographics will include three parts: (top) introductory text, (middle) statistics from the 2017-18 NTPS, and (bottom) call to action language.


The introductory text reminds participants of their participation in last year’s NTPS, informs them of the purpose of the TFS collection, and tells them how they can help make the data collection successful. This language will remain the same for all infographic conditions, and only minor personalization changes will be made for personalized leavers infographics (i.e., public school leavers and private school leavers) to address the fact that these are former teachers instead of current teachers.


The statistics are drawn from the 2017-18 NTPS public or private teacher data file, depending on the school type of the sampled participant. For the general infographics and the personalized stayer/mover infographics, the same items will be included. The statistics shared will be based on the targeted subgroup (i.e., for the public school and private school general infographics, or personalized for each of the two variations listed above by the eleven main teaching assignments). The personalized Leavers infographics (i.e., private school leavers and public school leavers) will contain the same items as one another, and differ slightly from the other personalized and general infographics, as a reflection of the fact that teachers who left the field and those who remained in the profession may be interested in different information.


The call to action language will remind participants how the TFS data are used and why it’s important that their experience be included in the survey results. Similarly to the introductory language, this language will remain the same for all infographic conditions, and only minor personalization changes will be made for leavers.


The treatment for each sampled teacher will be randomly assigned at the school-level at the time of teacher sampling, prior to the start of teacher data collection. The schools will be sorted by school size and other characteristics related to response. As such, the random assignment should result in even sample sizes and a similar distribution of school types across the treatment groups. The experimental treatments and expected sample sizes are displayed in Exhibit 3.


Exhibit 3: 2021-22 TFS Infographic Experiment Treatments and Sample Sizes

Conditions

Treatment

Sample Size

Number of unique infographics made

Control

No infographic included in the second teacher mail-out

1,000 public school teachers

585 private school teachers

0 infographics

Condition 1

One general infographic included in the second teacher mail-out

(Appendix A, p.6)

3,760 public school teachers

585 private school teachers

1 infographic for public school teachers


1 infographic for private school teachers

Condition 2

One personalized/targeted infographic included in the second teacher mail-out

(Appendix A, p.7-8)

3,760 public school teachers

585 private school teachers

24 infographics


Public School Teachers

Stayer/Mover

11 infographics (one for each NTPS main assignment field)


Example: “public school mathematics teacher”, “public school foreign languages teacher”

Leaver

1 infographic


Example: “Former public school teacher”

Private School Teachers

Stayer/Mover

11 infographics (one for each NTPS main assignment field)


Example: “private school mathematics teacher”, “private school foreign languages teacher”

Leaver

  1. infographic


Example: “Former private school teacher”



Following data collection, analyses will be conducted at the treatment level (experimental group vs. control). The analyses examined include:

  • Response rate;

  • R-Indicators;

  • Average number of contacts; and

  • Days to respond.


The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the infographic had on a case’s likelihood to respond, given that case’s unique characteristics.

Given the project sample sizes in the table above, a statistically significant difference between the control group (no infographic) and one of the treatments will be found if the response rates between the two groups differ by 7.76% for public school teachers and 12.40% for private school teachers. A statistically significant difference between the two treatments will be found if the response rates differ by 5.03% for public school teachers and 10.74% for private school teachers.

R-indicators will be used to determine the overall balance of the respondent population, as well as within each experimental group. R-indicators will be calculated for the full sample, as well as variable-level and category-level partial R-indicators to determine which characteristics specifically are contributing to imbalance within the respondent population.

The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of one type of contact strategy over another. If cases within the experimental group receiving infographics respond in a more timely fashion, this could reduce the number of cases included in follow-up operations, allowing finite resources, to be spread across fewer cases.


B.5 Reviewing Statisticians

Aaron Gilary and Alfred Meier of the U.S. Census Bureau, Jim Green of Westat, and Andy Zukerberg, Maura Spiegelman, and Julia Merlin of NCES reviewed and approved the TFS and PFS sample design and related matters for statistical quality, feasibility, and suitability to the overall objectives of the survey.

1 Elementary: teachers who taught elementary school students (any grade K-8, but at least one of the grades K-4) in the 2020-21 school year; Middle: teachers who taught middle school students (grades 5-8 exclusively) in the 2020-21 school year; High: teachers who taught high school students (any grade 7-12, but at least one of grades 9-12) in the 2020-21 school year.

2 The “A” version is for public schools; the “B” version is for private schools.

3 The “C” version is for public schools; the “D” version is for private schools.


4Tourkin, S., Parmer, R., Cox, S., & Zukerberg, A. (2005). (Inter) Net Gain? Experiments to Increase Response. Paper presented at the 60th Annual Conference of the American Association for Public Opinion Research (AAPOR), Miami Beach, FL, May.

5Zukerberg, A., Zotti, A., & Cox, S. (2019). Better Late Than Never? The Use of An Adaptive Incentive with Nonrespondents. Proceedings of the American Association for Public Opinion Research (AAPOR) Annual Conference, Toronto, Canada, May.


6 Zukerberg, A., Zotti, A., & Cox, S. (2019 Better Late Than Never? The Use of An Adaptive Incentive with NonrespondentsProceedings of the American Association for Public Opinion Research Annual Conference.


7 Priority Schools are schools with characteristics that correlate with a low propensity to respond and have an impact on estimates.

8 The Welcome Text will be sent at the time of Text 1.

9 Good, K. (1997). A study of factors affecting responses in electronic mail surveys. Dissertation, Western Michigan University, DAI, vol 58-10A pg. 3899, 119 pages

10 Carroll, S.H. & Zukerberg, A. (2013, November 4-6). Do Names Matter? Experiments Comparing Different Branding and Levels of Personally Identifiable Information in a Mail Questionnaire [Proceedings of a research conference]. Federal Committee on Statistical Methodology (FCSM), Washington, DC, U.S.

11 Dillman, D. A., Lesser, V., Mason, R., Carlson, J., Willits, F., Robertson, R., et al. (2007). Personalization of mail surveys for general public and populations with a group identity: Results from nine studies. Rural Sociology, 72, 632–646. doi:10.1526/003601107782638693.

File Typeapplication/vnd.openxmlformats-officedocument.wordprocessingml.document
File Titletfs omb supporting statement [Memorandum]
AuthorNCES
File Modified0000-00-00
File Created2021-10-16

© 2024 OMB.report | Privacy Policy