Revision Memo

Memorandum United States Department of Education

Institute of Education Sciences

National Center for Education Statistics

DATE: June 21, 2022

TO: Robert Sivinski, OMB

THROUGH: Carrie Clarady, NCES

FROM: Maura Spiegelman, NCES

SUBJECT: 2023-24 National Teacher and Principal Survey (NTPS) Respondent Portal Usability Testing and Focus Groups Protocol Revision Memo (OMB# 1850-0803 v.319)

The National Teacher and Principal Survey (NTPS), conducted every two to three years by the National Center for Education Statistics (NCES), is a system of related questionnaires that provides descriptive data on the context of elementary and secondary education. Redesigned from the Schools and Staffing Survey (SASS) with a focus on flexibility, timeliness, and integration with other ED data, the NTPS system allows for school, principal, and teacher characteristics to be analyzed in relation to one another. NTPS is an in-depth, nationally representative survey of first through twelfth grade public and private school teachers, principals, and schools. Kindergarten teachers in schools with at least a first grade are also surveyed. NTPS utilizes core content and a series of rotating modules to allow timely collection of important education trends as well as trend analysis. Topics covered include characteristics of teachers, principals, schools, teacher training opportunities, retention, retirement, hiring, and shortages.

The study “2023-24 National Teacher and Principal Survey (NTPS) Respondent Portal Usability Testing and Focus Groups” was approved in May 2022 (OMB# 1850-0803 v.315). This study proposed usability testing and focus groups to test and refine the online portal respondents use to provide data for the NTPS. This revision modifies the focus group protocol to:

• Collect feedback about the list of organizations that have endorsed the NTPS and are displayed in the online portal;

• Add probes about the types and circumstances of virtual classes regularly offered by schools; and

• Ask several questions in one-on-one breakout groups during the focus group window, rather than having all discussion in a large group, in order to be able to ask respondents to think aloud as they answer several survey items.

Modifications to the focus group moderator’s guide (Attachment 5) and associated recruitment materials (pp. 4-5 of Attachment 1) are shown below. Note that there are no changes in burden or in cost to the federal government associated with these revisions.

2023-24 NTPS Portal Pretesting

Attachment 1

Recruitment, Consent Forms, and Receipt Acknowledgment Materials

Email text for personal and professional contacts for focus groups

Dear [contact]:

The U.S. Census Bureau is assisting with a study conducted by the National Center for Education Statistics (NCES) of the U.S. Department of Education. Our records indicate you were the named coordinator for your school for the 2020-21 National Teacher and Principal Survey. We are interested in getting feedback on the surveys and web interface design and whether the materials provided in the survey encouraged staff to participate in the surveys. We invite you to participate in a focus group on these topics. Focus groups will be virtual through Microsoft (MS) Teams and held on multiple dates and times in [month].

Would you like participate in this voluntary ninety-minute focus group?

You will receive $75 as a thank you for your participation and you do not need to be at the school to participate.

To sign up please click the below link to answer a few questions.

[QUALTRICS Link to screener questions]

What will I be doing in the focus group?

This study is virtual. You will be shown images of the National Teacher and Principal Portal and survey screens. You will first be and asked to answer some survey questions one-on-one with a Census Bureau researcher. The rest of the time will be spent in a group discussion about other survey screens and contact materials. We will ask you your opinion on the screens and whether you used the information available in the survey to encourage participation from the staff in your school. Approximately three to five other schools will also be participating in the focus group portion.

How long is a session?

The session will last approximately 90 minutes.

When and where?

The study will be held [DAYS, DATES] virtually using MS Teams.

Interested in participating?

Please reply to this email with your contact information or call me at [ADD PHONE]. I’ll ask you some questions to make sure you are eligible for the study.

Thank you for your time and help with this important study.

Sincerely,

[Recruitment Contact Info]

Phone text for personal and professional contacts for focus groups after email

[Talking to live person]

Hi this is [Name] from the U.S. Census Bureau, is this [ person’s name]? Our records indicate that you were the survey coordinator for your school in the 2020-21 National Teacher and Principal Survey.

[If they volunteer that they don’t remember the survey:] It is ok if you don’t remember this survey.

We are calling you because we are interested in getting your feedback and thoughts on a website for the National Teacher and Principal Surveys. We are going to hold a 90-minute focus group and would like you to participate in it. The focus group will be virtually over Microsoft Teams. You will receive $75 cash for participation.

Are you interested in participating?

[Yes] Great thank you, I just have a few questions to ask to confirm your eligibility. [Go to screener].

[If need to leave a message]

Hi this is [Name] from the U.S. Census Bureau, I am trying to reach [person’s name]. We are wondering if they might be interested in participating in a virtual focus group for $75 to give feedback on an education survey they completed last year. If interested please give me a call back at xxx-xxx-xxxx or email me at [email protected].

Thank you for your time!

2023-24 NTPS Portal Focus Group

Attachment 5

Moderator’s Guide

NTPS Portal Focus Group Moderator’s Guide

Date: ____/____/__2022____ Start time: ___________ AM / PM

Moderators: _______________________________

Mode: Remote—Video/ audio

INTRODUCTION [5 MINUTES]

Welcome! Thank you for agreeing to talk with us today. My name is [MODERATOR NAME], and I am here with my Census Bureau colleagues [NAMES]. We really appreciate that you are taking this time out of your day to share your opinion and feedback for the National Teacher and Principal Survey (NTPS). The NTPS is a series of surveys from the Department of Education and U.S. Census Bureau that gather information about school, principal, and teacher experiences. Some of you were recruited because you had previously participated in the survey. It’s ok if you don’t remember this experience or you didn’t previously participate; we still want to talk with you to improve the survey experience for others.

We have a couple of other people from the Department of Education listening to our conversation today and helping us take notes so that we don’t miss anything important that you say. We’ll also be audio and video recording our conversations today so we can make sure to get all your feedback. Your name will not be associated with the recording or any of the other data collected during the session. Does anyone have any questions or concerns about that?

Today before we start the focus group we are going to go into breakout rooms to meet one on one with you and get your individual feedback on a few questions from the NTPS surveys. You will be working with me or another colleague from the US Census Bureau. We will then come back to the same meeting as a group.

Cognitive Interview [30 MINUTES]

[Send them to breakout rooms. return no later than ____ regardless of where you are in the interview]

Hello, my name is ________________, and I work for the Census Bureau.

We are working with the U.S. Department of Education to develop and refine questions for the NTPS survey about education and your school. Before surveys like the NTPS are conducted, it’s important to try out the survey questions with the help of people such as yourself to make sure the questions make sense.

In just a moment I’ll ask you to answer a few questions about your school. After you answer, I will ask some follow-up questions about your answers and what you understood the questions to mean. There are no right or wrong answers. Our purpose is not to compile information about you or your school. Instead, your responses, along with those of others, will show us how to improve these questions for the survey.

Do you have any other questions before we begin?

Now I am going to share a question on my screen and have you answer it as if you were filling out the survey for real at your school. With one major difference please say your answer out loud and let me know what you are thinking and feeling as you answer the question.

Question 1-12 to 1-14

Question 1-12

Can you tell me more about your answer to question 1-12?

(All) Can you tell me in your own words what the second option means?

(If marked the 2^nd box in 1-12 and hasn’t been mentioned yet) Can you tell me more about the online resources that are used in the classes at your school?

(If did not mark the 2^nd box in 1-12 and hasn’t been mentioned yet) What kinds of classes do you think should be included under the second option?

(All) Can you tell me in your own words what the third option means?

(If marked the 3^rd box in 1-12 and hasn’t been mentioned yet) Can you tell me more about the online classes without in-person instruction that are taught at your school?

(If did not mark the 3^rd box in 1-12 and hasn’t been mentioned yet) What kinds of classes do you think should be included under the third option?

Question 1-13 & 1-14

(If participant answers Q1-14) Now looking at 1-14, can you tell me more about your answer?

Overall

These questions used the word “typically”. Can you tell me what “typically” means to you in these questions?

Question 2-1

Let’s turn to 2-1.

Can you tell me in your own words what the first part of this question before part a means?

What does the phrase “Early Warning System” mean to you in this question? What does the phrase “Multi-Tiered System of Supports (MTSS)” mean to you in this question?

Looking at part a, tell me more about your answer to this question. Did you find this easy or difficult to answer?

Question 2-2

Let’s turn to 2-2.

Did you see the italicized instruction under 2-2?

Can you tell me what this instruction means to you in your own words?

Does your school contract out any services mentioned here that are not provided at the actual school? (If yes) Did you include them in your answers?

Question 3-2

Let’s turn to 3-2.

Were any of these questions difficult to answer? (If yes) Which ones were difficult?

Thank you very much for your feedback on these questions today. Now we will return to the bigger group discussion.

Focus Group [55 minutes total]

Intro [5 MINUTES]

[After returning to bigger group]

Thanks for your feedback so far today. Again, I’m _______. I’ll be leading the next part of our discussion today, which will focus on the NTPS Portal Website. The Portal Website is the first step of the survey when a school is selected. It is where we ask for basic information about your school, such as the grades taught in your school, and we also ask for a list of teachers. We are going to focus on what information would help encourage you and your staff to complete this survey.

RULES

Let’s go over a few ground rules quickly.

• First, we would appreciate if you can turn on your camera and keep it on. If you need to take a break and turn it off for a minute or two, that is okay.

• If you would like to speak please use the “raise your hand” function.

• I will be asking a lot of questions. There are no right or wrong answers to my questions; we just want your honest opinions. We don't need to have everyone agree. In fact, if you have a view different from others it is especially important for us to hear it because you may be the only person who is representing a different point of view here today.

• Please keep respondents’ information confidential. If you want to tell us about any experiences that you had filling out the surveys, please don’t share any names or any other information that could identify students or other staff.

• Only one person speaks at a time. Please do not speak while somebody else is talking because it’s hard to understand when too many people talk at once.

• We’d like everyone to get a turn. If you are someone who takes a little while to warm up in groups, please be sure to speak up and let us know what you are thinking. If you are the type of person who always participates, please make sure to give others a chance to talk as well.

• If you would like to use the chat function you may, but please note that it is easy to miss things in the chat when we are talking. So, if it’s really important, please raise your hand and tell us.

• Your responses are for informational purposes only. We ask that you speak freely so we can get better feedback on the National Teacher and Principal Survey.

• Due to limited time, we must move the discussion along and may sometimes have to interrupt in order to do so. Please do not be offended. If you have anything else that you’d like to share with us that you did not have time to say, please send your feedback in an email to [email protected]. [PASTE EMAIL IN CHAT]

Does anyone have any questions before we get started?

ICEBREAKER [5 MINUTES]

The first thing I want to do is get to know each of you a little better. Let's have each one of you tell us briefly:

• Your first name,

• And what your role in your school is?

I’ll go first. My name is… [BE BRIEF]

Please feel free to jump in and introduce yourself.

[SINCE THIS IS A POPCORN STYLE OF INTRODUCTIONS, CHECK THE ROSTER TO MAKE SURE EACH PARTICIPANT HAD THE OPPORTUNITY TO DO SO. MAKE SURE THE CO-MODERATOR HAS A TURN AS WELL.]

PROBES

Thank you for telling us a little more about you.

Mailing Invitation [20 MINUTES]

Now we are going to start by talking about the letter that is sent as an invitation to participate in the National Teacher and Principal Survey. You may or may not remember this, but I am going to show you an example of the letter [Show TLF welcome letter, give them time to look over and read]

1. What are your first impressions of this letter?

2. Do you think you would respond if you received this letter? Please tell me more about that.

3. What, if anything, about the letter do you dislike? Please tell me more about that.

4. What, if anything, about the letter do you like or think would encourage you to respond?

5. This letter tells you that you have been chosen by your school coordinator to upload or verify the list of teachers at your school in the portal. What do you think about that task? (If needed: What do you think this might entail? Why do you think you would be asked to do this? Does this seem easy or difficult?)

6. What would you do next if you had received this letter for real in your mailbox at school?

Portal [15 MINUTES]

Now imagine that you decided to respond to the survey. If you had logged into the portal, this is the screen you might first see if we had been able to prepopulate your teacher list using existing information:

7. 

However, if we couldn’t prepopulate your teacher list, this is what we you would see:

Now after you had finished entering or verifying your teacher information you would see this screen:

8. What, if anything, on this website might motivate your school staff to participate?

9. Assuming you were the school coordinator and you wanted staff to participate, what do you think you might do to motivate your staff to participate?

Now I am going to show you some screens about ideas to help encourage participation

10. The first screen is a sample of emails you can send to encourage participation and the second screen is a list of organizations that endorse the survey and encourage their members to participate. What do you think about this list of organizations? Do you think seeing it would encourage you to participate? If you participated in the past, did you see this list of organizations?

11. What kind of resources do you think we could add to help encourage your school and staff to participate?

12. [If needed] Please tell me more about that?

Wrapping Up [5 mins (there is 5 minutes buffer for other sections to run over)]

13. Is there anything else you’d like to tell us that you haven’t had a chance to mention yet?

1. Anyone else?

Thank you very much for your time! The feedback that you shared was very helpful. We’ve been taking careful notes so we can make recommendations. We will mail out your incentives within 5 business days. [IN CASE NOT ALL SIGNED CONSENT] Please be sure to sign the consent form- if you have not, we cannot mail out your incentive until you do.

2023-24 NTPS Portal Focus Group

Attachment 5

Moderator’s Guide

NTPS Portal Focus Group Moderator’s Guide

Date: ____/____/__2022____ Start time: ___________ AM / PM

Moderators: _______________________________

Mode: Remote—Video/ audio

INTRODUCTION [10 mins intro-to icebreaker]

Welcome! Thank you for talking with us today. My name is [MODERATOR NAME] and I’m here today with my colleague, [MODERATOR NAME]. We’ll both be guiding our conversation today. We work for the Census Bureau and we are actually talking to you all today from our homes in the greater Washington, DC area. We have a couple of other people from the Census Bureau and Department of Education listening to our conversation today and helping us take notes so that we don’t miss anything important that you say. We’ll also be recording our session today so we can make sure to get all your feedback. Does anyone have any questions or concerns about that?

We really appreciate that you are taking this time out of your day to share your experiences with the National Teacher and Principal Survey. The National Teacher and Principal Survey is a series of online surveys from the Department of Education and U.S. Census Bureau that gather information about schools, teachers, and principal’s experiences. Many of you were recruited because you had previously participated in the survey. It’s ok if you don’t remember this experience or didn’t previously participate, we still want to talk with you to improve the survey experience.

Today our focus is on the Research Resource Center in the NTPS Portal Website. The Portal Website was the first step of the survey when your school was selected. It is where we asked for basic information about your school, such as the grades taught in your school, and we asked for a list of teachers. We are going to focus on what information would help you encourage staff to complete this survey.

RULES

Let’s go over a few ground rules quickly.

• First, we would appreciate if you can turn on your camera and keep it on. If you need to take a break and turn it off for a minute or two, that is okay.

• If you would like to speak please use the “raise your hand” function.

• I will be asking a lot of questions. There are no right or wrong answers to my questions; we just want your honest opinions. We don't need to have everyone agree. In fact, if you have a view different from others it is especially important for us to hear it because you may be the only person who is representing a different point of view here today.

• Be respectful of other people’s comments. It is fine to disagree with each other’s comments. We are interested in listening to everybody's opinions and would like to hear from you if you have different ideas from what has already been expressed.

• Please keep respondents’ information confidential. If you want to tell us about any experiences that you had filling out the surveys, please don’t share any names or any other information that could identify students or other staff.

• Only one person speaks at a time. Please do not speak while somebody else is talking because it’s hard to understand when too many people talk at once.

• We’d like everyone to get a turn. If you are someone who takes a little while to warm up in groups, please be sure to speak up and let us know what you are thinking. If you are the type of person who always participates, please make sure to give others a chance to talk as well.

• Your responses are for informational purposes only. We ask that you speak freely so we can get better feedback on the National Teacher and Principal Survey.

• Due to limited time, we must move the discussion along and may sometimes have to interrupt in order to do so. Please do not be offended. If you have anything else that you’d like to share with us that you did not have time to say, please send your feedback in an email to [email protected].

Does anyone have any questions before we get started?

ICEBREAKER

The first thing I want to do is get to know each of you a little better. Let's have each one of you tell us briefly:

• Your first name,

• Where your school is located and what your role is?

• How long you’ve been working in this role?

I’ll go first. My name is… [BE BRIEF]

Please feel free to jump in and introduce yourself.

[SINCE THIS IS A POPCORN STYLE OF INTRODUCTIONS, CHECK THE ROSTER TO MAKE SURE EACH PARTICIPANT HAD THE OPPORTUNITY TO DO SO. MAKE SURE THE CO-MODERATOR HAS A TURN AS WELL.]

PROBES

Thank you for telling us a little more about you.

Research Resources

14. Now we are going to start by talking about the National Teacher and Principal Survey Portal. You may or may not remember this, but when your school was selected for the survey there was a brief survey that confirmed the grades taught in your school and some other basic information. There also was a place where you entered contact information for the teachers in your school. You may have confirmed information that was already on the website, or you may have uploaded a document or typed in new information.

15. Does anyone remember this? [If yes] please tell me more about what you remember.

16. [If no one saw it or remembers] Now it may have looked something like this (share the following screens with participants):

17. [If needed] Does anyone remember seeing something about “NTPS resources?”

    1. [If yes] Please tell me more about that? Did you click on that link? [if yes] What do you remember about it?

18. Do you think anything on this website may help you ensure your school staff participate?

19. What did you do to encourage your staff to participate?

20. What do you think would have helped you encourage the staff to participate?

    1. [IF NEEDED] Please tell me more about that.

21. Is there anything you would change about these two screens I am showing you?

22. [If needed] How do you think it could have been made clearer to you that there were resources to help encourage participation?

23. Are there any other surveys you’ve participated in that you thought had really good materials to encourage people to participate?

24. [If needed] Please tell me more about those materials?

Now let’s talk about some of the kinds of data that you imagine the Department of Education has that might be useful or interesting for you and your school.

25. Tell us a little about the kinds of national data on education, schools, or students you have used in the past?

1. [IF THEY HAVE] Where did you get that data?

26. [IF THEY HAVE] What did you learn from that data that was helpful to you?

27. [IF THEY HAVE] Was there some way that data could have been easier to access or use?

28. [IF THEY HAVEN’T] Are there certain kinds of national data on education, schools, or students you would like to use but haven’t?

29. [If they aren’t responsive to other probe] Do you ever see statistics quoted in newspapers or on television that you wonder where they come from? Please tell me more about an example of this? Do you think that kind of information might be useful for you or your school to know?

30. [IF NEEDED] What do you think about these resources?

Wrapping Up [5 mins]

31. Is there anything else you’d like to tell us that you haven’t had a chance to mention yet?

2. Anyone else?

Thank you very much for your time! The feedback that you shared was very helpful. We’ve been taking careful notes so we can make recommendations. We will mail out your incentives within 5 business days. [IN CASE NOT ALL SIGNED CONSENT] Please be sure to sign the consent not if you have not we can not mail out your incentive until you do.

15