Web Instrument Request Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: February 24, 2022

TO: Robert Sivinski, OMB

THROUGH: Carrie Clarady, NCES

FROM: Julia Merlin, NCES

SUBJECT: 2021-22 PFS Web Instrument Change Request (OMB# 1850-0617 v.10)

The Principal Follow-Up Survey (PFS) is a follow-up survey of public and private elementary and secondary school principals who participated in the National Teacher and Principal Survey (NTPS) during the previous school year. The purpose of the survey is to determine how many principals remained at the same school, moved to another school, or left the profession. It was last fielded in 2016-17, after the 2015-16 NTPS. Historically, the PFS has also been conducted the school year following the Schools and Staffing Survey (SASS), the predecessor survey to the NTPS. Redesigned from the SASS with a focus on flexibility, timeliness, and integration with other ED data, the NTPS system allows for school, principal, and teacher characteristics to be analyzed in relation to one another. The major objectives of the PFS are to measure the attrition rate for principals; examine the characteristics of principals who stayed in the profession and those who changed professions or retired and obtain activity or occupational data for those who left the position of a K-12 principal.

The TFS/PFS main study was approved in July 2021 (OMB# 1850-0617 v.4). Two change requests to update teacher recruitment materials (OMB# 1850-0617 v.5) and add special district contact materials (OMB# 1850-0617 v.6) were approved in August 2021, a change request to add reminder emails to schools for the TFS-1 operation (OMB# 1850-0617 v.7) was approved in October 2021, a change request to correct a typo or error on the questionnaire was approved in November 2021 (OMB# 1850-0617 v.8), and a change request to update incentive procedures and recruitment material timing was approved in January 2022 ((OMB# 1850-0617 v.9). The 2020-21 NTPS (OMB# 1850-0598 v.28-30) finished data collection in late July 2021, and the TFS/PFS is a follow-up operation to the NTPS conducted the year prior.

Historically, the PFS has a very high response rate; for the 2016-17 collection, the unit response rate was over 90%. Given the high response rates and the fact that the PFS is only one item long, the survey team has historically not developed a web instrument. Unfortunately, the response rates for the 2021-22 PFS collection are considerably lower. This was unanticipated given past trends. As of February 23, 2022, we are in Week 5 of data collection for the 2021-22 PFS, and the unit response rate is 53%. At Week 5 in the 2016-17 collection, the response rate was over ten percentage points higher at 69%. To address this surprising decrease in expected response rates, NCES is proposing the addition of a PFS web instrument on Qualtrics and the addition of two new versions of a PFS recruitment emails (an initial email and a reminder email) that include the new Qualtrics link. Only a subset of PFS sampled principals will receive the Qualtrics link, replacing the ‘late texting’ experimental group in the texting experiment that is further described in Part A. The new web instrument will ideally provide another method of completion for respondents, especially those who have may prefer web to paper.

All changes in the attached materials are shown in full in the pages to follow, where new text is indicated in red. This requested change does not affect the approved total cost to the federal government for conducting this study nor the estimated respondent burden.

Changes to Part A

1. (Pages 4-5)

A.1.4b Study Design for the 2021-22 PFS

The 2021-22 PFS will be conducted during the school year following the 2020-21 NTPS. Data collection will begin with the current occupational status of the principal from the 2020-21 NTPS sampled school via the PFS-1A/1B form. Nonresponding schools will receive a second mailing and eventually a non-response follow-up telephone call to complete their PFS-1A/1B form over the phone. Principals of nonresponding schools will be contacted directly using the contact information provided on the NTPS principal questionnaire Principal Questionnaire during the prior school year. First, they will be mailed a PFS-1C/1D form at their home address, along with a reminder contact by email. An alternate version of this email containing a link to complete the PFS online via a Qualtrics web-based instrument will also be sent experimentally to a smaller group of principals. All nonresponding principals will receive a second mailing and email (either the standard or alternate Qualtrics web-instrument version, depending on their treatment group) and eventually a non-response follow-up telephone call. In addition to the mailings and emails, some principals will also be sent text messages experimentally and as means of follow-up, as well, but in an experimental manner. Further details about the use of text messages and the inclusion of the Qualtrics web instrument during the 2021-22 PFS are provided in section B.4 of Supporting Statement Part B.

Because the PFS will collect data with only one (PFS-1C/1D) or two (PFS-1A/1B) follow-up questions, an internet instrument will not be developed. PFS data will be collected entirely using paper questionnaires With the exception of the treatment group offered the Qualtrics web instrument via email, PFS data will be collected from principals and schools using paper questionnaires, as was done for the 2016-17 PFS.

2. (Page 7)

A.3b Appropriate use of Information Technology for the 2021-22 PFS

The one-page PFS questionnaire will be mailed to each 2020-21 NTPS sampled school to gather the current occupational status of the person who was the school’s principal during the NTPS.

In order to collect occupation status information about the principal if the school fails to complete their PFS questionnaire, a slightly different version of the PFS questionnaire will be mailed directly to each sampled principal at his or her home mailing address, which was collected as part of the 2020-21 NTPS Principal Questionnaire. In addition to receiving mailed PFS correspondence, sampled principals will be contacted via email throughout data collection using the email addresses collected during the 2020-21 NTPS. When necessary, additional research operations will be conducted to find email addresses for sampled principals whose email address was not collected during the 2020-21 NTPS or for whom it proves no longer valid.

In addition, some sampled principals will be sent text messages as means of follow-up, as well, but in an experimental manner. For some principals, email contacts will contain a link to complete the PFS online via a Qualtrics web-based instrument; this response mode was added in response to lower than expected response (when compared to previous data collection cycles of PFS) from schools during the 2021-22 PFS school-based collection as means of boosting response. Some other sampled principals will also be sent text messages experimentally and as means of follow-up. Further details about the use of text messages and the inclusion of the Qualtrics web instrument during the 2021-22 PFS are provided in section B.4.1 B.4.2 of Supporting Statement Part B.

Census contact center staff will implement telephone reminders and follow-up operations with sample schools and principals to encourage those that have not yet responded to complete their PFS questionnaire. When possible, interviewers will collect principal status information over the phone. These operations will take place concurrently with a reminder mailout throughout the spring of 2022.

3. Page 10

The TFS web-based data collection instrument will be housed on Census Bureau servers, and the PFS web-based instrument will be housed on the Qualtrics platform.

4. Page 14

A.16 Project Schedule

Activity	Dates
Reminder letter with a second PFS 1A/1B 1C/1D mailed to nonresponding principals	Late March 2022

Changes to Part B

5. (Pages 8-9)

B.2.2 2021-22 PFS Procedures for the Collection of Information

For non-responding schools or schools that were unable to report the principal’s status, Census Bureau staff will mail a modified version of the Principal Status Form (PFS-1(C/D)¹) to the 2020-21 principal at his or her home address (if reported in NTPS). The modifications tailor the form to the previous years’ principal rather than the school staff. This mailing will begin in March 2022, and an email will be sent a few days following the mailing, alerting the principal to the mailing and reminding them to complete their PFS. An alternate version of this email will also be sent experimentally to a smaller group of principals and will contain a link to complete the PFS online via a Qualtrics web-based instrument. All non-responding principals will be sent a follow-up mailing and email (either the standard or alternate Qualtrics version, depending on their treatment group) approximately 2 weeks after the initial mailing and email. Telephone center staff will follow-up by telephone to collect the occupational status of the non-responding principal beginning two weeks after the reminder mailing and email are sent.

The 2021-22 PFS will also include an experiment to test different methods of texting survey participants as a means of exploring the use of text messages as a new data collection mode. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS principal questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number. Because the PFS does not have a web instrument for data collection, all All text message contacts for the survey will be an interactive exchange with the principal.

To explore offering a web-based response option for PFS and to explore using text messaging as a contact method for PFS, the experiment PFS data collection will include two treatment groups:

1. Offering include an interactive questions text message contact to complete the survey concurrent with both principal-level mailouts.

2. Replace the principal-level mailouts with an interactive questions text message contact to complete the survey

2. Sending an email containing a link to complete the PFS online via a Qualtrics web-based instrument concurrent with both principal-level mailouts.

This The experiment will help determine whether or not offering principals a web-based response option helps to boost PFS response and whether survey participants would be willing to respond to a survey via text, generally. In addition, results from this experiment will help determine whether a respondent is more likely to respond via a text exchange instead of returning a paper questionnaire the text serves as a reminder, prompting respondents to return their paper questionnaire. For additional details about testing a web-based response option and testing the use of text messaging in the 2021-22 PFS, refer to section B.4.12 of this document.

A response rate in the approximate range of 90-95 percent is expected based on the prior administration of PFS.

5. (page 12)

B.3.4 Extensive Follow-up (by mail, email, and telephone) of Non-Respondents

For the PFS, the Census Bureau will also utilize mixed survey modes – self-administered mail instruments, a web-based Qualtrics instrument (for a subsample of principals), and telephone, as needed, to maximize response levels. Non-respondents will receive at least two reminders. Given the brevity of the instrument, we anticipate response rates at or above 90% with this approach, as with past cycles of the survey.

6. (pages 12-13) B.4 Tests of Procedures and Methods

B.4.1 Testing the use of text messages in both TFS and PFS

For 2021-22 data collection cycle of the TFS and PFS, NCES will explore the use of text messaging as a new data collection mode by including an experiment to test different methods of texting survey participants. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS principal and teacher questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number.

7. (pages 15-18) – the addition of the web-based instrument for the PFS complicates the text messaging experiment, so we added a new section to B.4 Tests of Procedures and Methods. Although much of the text in this section is replicated from B.4.1, the section itself is new.

B.4.2 Testing the use of a web-based instrument and text messages in PFS

For the 2021-22 data collection cycle of the PFS, NCES will explore the use of text messaging as a new data collection mode by including an experiment to test different methods of texting survey participants. In order to adhere to the necessary guidelines for texting per the Census Bureau’s Policy office, the 2020-21 NTPS principal questionnaires were updated to include a checkbox consenting contact by text message in the future alongside the respondent’s cell phone number.

Because the PFS does not have a web instrument for data collection, all text message contacts for the survey PFS will be an interactive exchange with the principal. PFS data collection begins with contacts to the principal at the NTPS school and then shifts to contacts with the sampled principals directly using the personal contact information they provided on the NTPS if the school is not responsive.

Additionally, the 2021-22 data collection of PFS will explore the option of offering principals the option to complete the PFS online via a Qualtrics web-based instrument. This response mode was added in response to lower than expected response (when compared to previous data collection cycles of PFS) from schools during the 2021-22 PFS school-based data collection as means of boosting response rates.

To explore offering a web-based response option for PFS and to explore using text messaging as a contact method for PFS, the PFS data collection will include two treatment groups:

1. Include Offering an interactive questions text message contact to complete the survey concurrent with both principal-level mailouts.

2. Replace the principal-level mailouts with an interactive questions text message contact to complete the survey

2. Sending an email containing a link to complete the PFS online via a Qualtrics web-based instrument concurrent with both principal-level mailouts.

The experiment will help determine whether offering principals a web-based response option helps to boost PFS response and whether survey participants would be willing to respond to a survey via text, generally. In addition, results from this experiment will help determine whether a respondent is more likely to respond via a text exchange instead of returning a paper questionnaire or if the text serves as a reminder, prompting respondents to return their paper questionnaire.

Following data collection, analyses will be conducted at the treatment level (experimental group vs. control). The analyses examined include:

• Response rate;

• R-Indicators;

• Average number of contacts; and

• Days to respond.

The response rates will be calculated for each treatment group and selected demographic domains and compared using significance tests for differences. To account for confounding variables, a model-based approach will also be calculated to determine what effect the text message(s) had on a case’s likelihood to respond, given that case’s unique characteristics.

Given the project sample sizes in the table above, a statistically significant difference between the control group (no text messages) and any of the other three two treatment groups will be found if the response rates between the two groups differ by:

• 6.26% for public school principals (PFS)

• 12.39% for private school principals (PFS)

R-indicators will be used to determine the overall balance of the respondent population, as well as within each experimental group. R-indicators will be calculated for the full sample, as well as variable-level and category-level partial R-indicators to determine which characteristics specifically are contributing to imbalance within the respondent population.

The average number of contacts and average days to respond across the experimental groups will be used as a proxy for timeliness of response. A reduction in the average number of contacts could be used to justify the use of one type of contact strategy over another. If cases within the experimental group receiving text messages respond in a more timely fashion, this could reduce the number of cases included in follow-up operations, allowing finite resources, to be spread across fewer cases.

B.4.2 3 Testing the inclusion of an infographic in TFS packages

Exhibit 2: 2021-22 PFS Experimental Contact Strategy and Sample Sizes

Experimental Treatment	Planned Date	School-level Contacts (PFS-1A/B)			Principal-level Contacts (PFS-1C/D)
		1/13/22	2/3/22	2/14/22-3/4/22	Concurrent with Text 1	3/16/22, 3/17/22, 3/22/22,	3/29/22, 3/31/22, 4/5/22		4/18/22 - 5/13/22
Standard/Control Group 1,322 public school principals 342 private school principals	Mail Activity	1st Mail	2nd Mail			1st Mail	Reminder Mail
	Email Activity					Reminder 1	Reminder 2
	Telephone Activity			Telephone FU					Telephone FU
	Texting Activity
Late Texting - Status question to determine attrition 1,322 public school principals 342 private school principals	Mail Activity	1st Mail	2nd Mail
	Email Activity					Reminder 1	Reminder 2
	Telephone Activity			Telephone FU				Telephone FU
	Texting Activity				Welcome Text	Text 1	Text 2
Mail & Texting Treatment - Status question to determine attrition 1,322 public school principals 342 private school principals	Mail Activity	1st Mail	2nd Mail			1st Mail	Reminder Mail
	Email Activity					Reminder 1	Reminder 2
	Telephone Activity			Telephone FU					Telephone FU
	Texting Activity				Welcome Text	Text 1	Text 2

Web Treatment 1,322 public school principals 342 private school principals	Mail Activity	1st Mail	2nd Mail			1st Mail	Reminder Mail
	Email Activity					Initial Email (with link)	Reminder Email (with link)
	Telephone Activity			Telephone FU				Telephone FU
	Texting Activity

Changes to Appendix A

8. (Page 25) updated contact materials table to reflect two new emails

Contact Materials for TFS and PFS 2021-22

Teacher and Principal Letters and E-mails Summary Table

Data Collection Activity/Operation	Correspondence Identifier(s)	Description
PFS First Principal Mailout/Initial E-mail
Initial Principal Letter	PFS-33L	This letter asks principals to complete the one-question Principal Status Form.
First Principal E-mail (Without Home Address)	PFS-33E(1)	This e-mail asks principals to provide their current occupational status by completing the one-question Principal Status Form.
First Principal E-mail (With Home Address)	PFS-33E(2)	This e-mail asks principals to provide their current occupational status by completing the one-question Principal Status Form.
First Principal E-mail (With Qualtrics Web link)	PFS-33E(3)	This e-mail asks principals to provide their current occupational status by completing the Qualtrics survey.
PFS Reminder Principal E-mail
Reminder Principal E-mail	PFS-34E(1)	This e-mail reminds principals to complete the one-question Principal Status Form.
Reminder Principal E-mail (With Qualtrics Web Link)	PFS-34E(2)	This e-mail reminds principals to complete the Qualtrics survey.

8. (page 66) First Principal E-mail (With Home Address)

PFS-33E(2)

Subject: Principal Follow-Up Survey

Body:

Dear <Principal Name>,

Thank you for making the 2020–21 National Teacher and Principal Survey (NTPS) a success!

We recently mailed a letter inviting you to participate in the Principal Follow-up Survey (PFS). One area of research that cannot be addressed with the NTPS data already collected is the attrition and retention rates of principals in public schools like you. The Principal Follow-up Survey (PFS) has only one question and will take less than five minutes to complete. To answer this question, you may participate in ONE of two ways:

1. Complete the form that was mailed to your home address and return it in the postage paid envelope.

2. Call the U.S. Census Bureau at 1-866-325-4957. Someone will be available to take your call Monday through Friday, between 8:00 a.m. and 8:00 p.m. (Eastern Time).

For more information about NTPS, and to read reports from previous surveys, please visit our website at: http://nces.ed.gov/surveys/ntps.

If you have any questions, please contact the U.S. Census Bureau, toll-free, at the number above or by e-mail at: [email protected].

Thank you in advance for your contribution to this important survey effort.

Sincerely,

Principal Follow-up Survey Team

U.S. Census Bureau, on behalf of the

National Center for Education Statistics (NCES)

1-866-325-4957| [email protected] | http://nces.ed.gov/surveys/ntps

8. (page 67) First Principal Email (with Qualtrics Web link)

PFS-33E(3)

Subject: Principal Follow-Up Survey

Body:

Dear <Principal Name>,

Thank you for making the 2020–21 National Teacher and Principal Survey (NTPS) a success!

One area of research that cannot be addressed with the NTPS data already collected is the attrition and retention rates of school principals. Therefore, we are following up to learn about any changes in your career since the last school year.

The Principal Follow-up Survey (PFS) has only two questions and will take less than five minutes to complete.

Respond now at: <PFS URL>

We encourage you to complete your PFS online. Alternatively, you may contact the U.S. Census Bureau at 1-866-325-4957. Someone will be available to take your call Monday through Friday, between 8:00 a.m. and 8:00 p.m. (Eastern Time).

For more information about NTPS, and to read reports from previous surveys, please visit our website at: http://nces.ed.gov/surveys/ntps.

If you have any questions, please contact the U.S. Census Bureau, toll-free, at the number above or by e-mail at: [email protected].

Thank you in advance for your contribution to this important survey effort.

Sincerely,

Principal Follow-up Survey Team

U.S. Census Bureau, on behalf of the

National Center for Education Statistics (NCES)

1-866-325-4957| [email protected] | http://nces.ed.gov/surveys/ntps

8. (page 69) Reminder Principal E-mail (with Qualtrics Web link)

PFS-34E(2)

Subject: Principal Follow-Up Survey Reminder

Body:

Dear <Principal Name>,

A few weeks ago, we sent you a letter (if a home address was available) and an email asking you to complete the two-question Principal Follow-up Survey (PFS) and share your current occupational status. We rely on professionals like you to help us understand the effects of school policies and practices on principals’ decisions to remain in or leave their profession.

If you have already completed the survey, thank you for your assistance, and please disregard this e-mail. If you have not had the opportunity to participate yet, we encourage you to complete the PFS online.

Respond now at: <PFS URL>

For more information about NTPS, and to read reports from previous surveys, please visit our website at: http://nces.ed.gov/surveys/ntps.

If you have any questions, please contact the U.S. Census Bureau, toll-free, at the number above or by e-mail at: [email protected]. Someone will be available to take your call or answer your e-mail Monday through Friday, between 8:00 a.m. and 8:00 p.m. (Eastern Time).

Thanks, in advance, for your participation.

Sincerely,

Principal Follow-up Survey Team

U.S. Census Bureau, on behalf of the

National Center for Education Statistics (NCES)

1-866-325-4957| [email protected] | http://nces.ed.gov/surveys/ntps

Changes to Appendix B

8. (page 77) Added mock-ups of the PFS public school principal instrument

1 The “C” version is for public schools; the “D” version is for private schools.