Attachments 2-5 NTPS 2023-24 Pretesting Screener & Protocols v319

National Center for Education Statistics

Attachment 2 – Recruitment Screener for usability testing

Attachment 3 – Protocol Materials for usability testing

Attachment 4 – Recruitment Screener for focus groups

Attachment 5 – Protocol Materials for focus groups

2023-2024 National Teacher and Principal Survey (NTPS)

Respondent Portal Usability Testing and Focus Groups

OMB# 1850-0803 v.319

April 2022

revised June 2022

2023-24 NTPS Portal Usability Testing

Attachment 2

Recruitment Screener

Hello. My name is [XXXXXX] and I work for the U.S. Census Bureau. The U.S. Department of Education is conducting a study of schools to evaluate the effects of school workplace conditions, salaries, and training opportunities on the educational workforce, among other things.

We are looking for volunteers to help us improve the web survey used in this study. We want to know how easy or hard it is to complete the web survey, and we need help from people like you to do that. The information gained from the study will be used for potential revisions to the web survey. Your participation is voluntary, and you will receive no sales pitches or follow-up calls based on your participation. I’d like to ask you a few questions to see if you qualify for the study.

The next few questions are for classification purposes only.

1. What is your name? (write in)

________________________________________________

2. What is your phone number? (write in)

_________________________________________________

3. What is your email address? (write in)

_________________________________________________

4. Do you currently work at a school?

___ Yes

___ No – Thank you for your interest, but we need participants who work in a school. END CALL

5. What is your current job title or role?

_________________________________________________

6. Are you a school administrator or staff person who can provide information about the teachers at your school, including teacher’s names, email addresses, and subject matter taught?

____ Yes

____ No – Thank you for your interest, but we need participants who have access to that data. END CALL

7. Do you have access to the files containing that information at the school?

____ Yes

____ No or Not sure - Thank you for your interest, but we need participants who have access to that data. END CALL

8. What state is your school in? ___

9. What is the name of the school where you work?

________________________________________________

10. What grades does your school serve?

________________________________________________

11. How many students are enrolled in your school?

________________________________________________

12. Do you work in a public school, a charter school, or a private school?

________________________________________________

13. Have you participated in a research study or focus group in the past 6 months? (RECRUIT NONE.)

___ Yes

___ No

14. Based on your answers, we would like to invite you to participate in our study. You will receive $75. The interview is approximately 90 minutes in length. The session will be conducted virtually using Microsoft Teams where you will share your computer screen with the Census Bureau researcher. You’ll need to have access to records with the list of teachers, their emails, and classes taught at the school. Will you have access to MS Teams on your computer?

__Yes, I can use MS Teams on my computer and I will have access to my records -> Go to 15

__ I’m not sure if I can access MS Teams -> Go to 16

__No, I cannot conduct this session using my computer/I do not have access to my records-Thank you for your time today but we will need you to use the computer for this session. END CALL

15. We are conducting interviews between __________ and ___________. When are you available to participate in this research study during that time period?

	CIRCLE
Yes	1	 (CONTINUE, and collect date, time and location)

Date:

Time:

16. I can have a researcher reach out to you to conduct a tech check to see if you can access MS Teams and share your screen. Would you like that?

	CIRCLE
Yes	1	 (Be on the lookout for an email from the researcher. CONTINUE to Privacy Language)
No	2	 (CONTINUE to Privacy Language)

Date:

Time:

If they have an e-mail then say: (If not scheduled right away: Whether or not you are selected), I am going to email you some more information about how we protect the privacy of your data and your rights.

If they do not have an e-mail then read: Thanks for taking some time to answer my questions today. (If not scheduled right away: If you qualify, someone will be back in touch with you soon to schedule an interview.)

The U.S. Census Bureau is required by law to protect your information. We are conducting this voluntary survey on behalf of the National Center for Education Statistics under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151).

Your privacy is also protected by the Privacy Act, Title 5 U.S. Code. Routine uses of these data are limited to those identified in the Privacy Act System of Record Notice titled, “SORN COMMERCE/Census-7, Demographic Survey Collection (non-Census Bureau Sampling Frame).”

2023-24 NTPS Screener and Portal Usability Testing

Attachment 3

Protocol Materials

Table of Contents

Usability Testing Protocol 7

Practice Think Aloud 8

Interview 9

Satisfaction Questions 10

Debriefing Questions 12

Additional Debriefing Questions 13

Finishing up session 13

Interviewer Log Sheet 14

Comments: 15

Usability Testing Protocol

MATERIALS NEEDED FOR INTERVIEW

• Interviewer Protocol Script

• Consent Form (emailed ahead of time)

• Link to Qualtrics satisfaction survey, debriefing questions

• Link to NTPS Centurion instrument

• Snagit tool open for recording

• User IDs needed for this participant:

Test Administrator Usability Testing Script

Thank you for your time today. My name is XX and I work at the U.S. Census Bureau. I will be working with you today. My job is to evaluate how easy or difficult our web surveys are to use. What works well, we keep. When potential users, such as you, have difficulty with something, we have an opportunity to fix it before it goes live to a much larger group. If observer present: I also want to let you know that this interview is being observed by [describe observer’s role: e.g., someone who works on the survey]. They are observing the interview because they are also interested in how the web survey work for people like you.]

We are working with the U.S. Department of Education to develop and refine the web survey. Before surveys are conducted, it’s important to try out the survey with the help of people such as yourself.

Informed Consent

If they did sign before the interview, check in Qualtrics

Before we start, I just wanted to thank you for signing the document in the link I sent before the interview. Just as a quick reminder, that document explained a little bit about this interview and provided information about your rights as a participant, such as that all information you provide is protected and we won’t use your name or specific circumstances in any report. The link had a signature line to acknowledge the $75 you will receive by mail for participating. It also asked for your permission to have this session audio recorded as well as a video recording of your computer screen and image. This is so that I can concentrate on what you are saying rather than having to take extensive notes. Your name will not be associated with the recording or any of the other data collected during the session. If observer present: The additional people observing this call are also bound to keep your name and information you provide protected May I turn on the recorder? [If they say yes turn on Snagit recorder and backup audio recorder] For our purposes, we need to capture your consent orally on the recording. Please state, “I give consent to participate in this research and to be recorded.”

Participant agrees

Thank you for that.

Do you have any other questions about the document or the interview?

If they did not sign before the interview

Before we start, I noticed that you had not yet signed the document in the link we e-mailed to you; I just wanted to check did you receive that e-mail? If yes: Did you have any issues opening the link or with signing the document? If they say yes talk them through it. If unable to get signed consent, display consent form or rely on oral consent using script below. I will go over what is in that document and obtain consent orally. However, the document also contains an acknowledgement that you will receive $75 by mail for participating in this research. You can still participate right now, but you will need to sign that document before we can send the $75.

The document explains a little bit about the purpose of this study, which is to test a portion of the National Teacher and Principal Survey Questionnaire. The document provides information about your rights as a participant, such as that all information you provide is protected and we won’t use your name or specific circumstances in any report. It also asks for your permission to have this session audio recorded as well as a video of your computer screen and image so that I can concentrate on what you are saying rather than having to take extensive notes. May I turn on the recorder? If they say yes turn on Snagit recorder and backup audio recorder. For our purposes, we need to capture your consent orally on the recording. Please state, “I give consent to participate in this research and to be recorded.”

Participant agrees

Thank you for that.

Your participation in this interview is very important because it will help the Census Bureau with improving these survey questions. Your participation in this interview is voluntary, and you may decline to answer any question, or stop the interview at any time. We are scheduled to be here up to 90 minutes, and I won’t keep you any longer.

If participant is not a federal employee: As a thank you for your help, we will mail you $75 cash after the interview is completed.

Thank you.

For today’s session, you will help us evaluate the design of a survey that we are working on for the U.S. Department of Education. You will be helping us evaluate a portion of the National Teacher and Principal Survey. You will do this by completing it using your computer/this laptop.

If you run into any difficulties as you work in the web instrument, please don’t blame yourself. Any difficulties are the result of the design of the web instrument, not your skills or abilities. We are going to use your comments and experiences, as well as comments and experiences of other participants, to help improve the web instrument. I did not create it, so don’t feel like you have to hold back your thoughts to be polite to me. We appreciate your help so we can make the web instrument work well for everyone.

Think Aloud: I want you to respond to the web instrument exactly the way you would if you were alone at school, but with one major difference. I would like you to think aloud as you work within the instrument. I am interested in your answers, but I am also interested in the process you go through in your mind when you answer the questions. I would like you to tell me everything that you are thinking and feeling as you go about answering each question.

Practice Think Aloud

Let’s do a practice question before we start: Please think aloud as you answer the question, how many windows are in your home?

[Probe if they fall silent; if they just give a number, ask them to do it again but this time so that you can understand how they came up with the number. After they finish the think-aloud exercise, explain why we are asking them to do this, e.g., if the Census was interested in the number of windows, which we are not, but if we were, and someone was looking at their sliding glass door and wondered aloud that they didn’t know if they should include it or not, them saying those words would clue us in that we needed to fix our question, make it clear that sliding glass doors should be included, etc.]

Probes to use in study

• Keep talking

• Um-hum?

• What are you thinking?

Great. That is what I want you to do throughout our session. I will remind you to think aloud if you get quiet.

Interview

I’m going to paste the link for the survey into the chat window. Once I paste it there, I’ll have you click on it to open the survey website.

Now I’m going to paste your User ID to enter the survey into the chat window.

By the way, what browser are you using today?

Thank you, you can now start the survey.

Probe as needed: (If we are not able to see the teacher lists they are consulting) What are you looking at as you are adding teachers into the instrument?

[Interviewer instruction: If there is pre-loaded data, then that is the option for them; otherwise, it is either file upload or manual entry. If there is pre-loaded data and they have not edited the data, deleted or added a teacher, please ask them to perform those functions, even if it means adding a fake name, “John Doe”. Be sure to leave about 20 minutes for the rest of the session.]

Okay, we are ready to move onto the next task. We’d like you to logout of the instrument.

Satisfaction Questions

After final SUBMIT is clicked, have participant answer satisfaction questions using a Qualtrics survey.

The next task is for you to answer a short questionnaire about your experience completing the survey you just finished. I pasted the link in the chat window.

Please Note: The questionnaire will be electronic with word labels on either end of 7 radio buttons. See image below for what the final format will look like.

1. Overall experience with the survey

Unpleasant to Pleasant

2. The survey screens are visually pleasing

Never to Always

3. The survey screens look professional

Never to Always

4. Characters (words/text) on the screen

Barely legible to Very legible

5. The survey questions follow a logical sequence

Never to Always

6. Words used in the survey questions and answers

Hard to understand to Easy to understand

7. Coming up with answers to the questions

Never bothered me to Always bothered me

8. Instructions for correcting errors

Confusing to Clear

9. Entering your answers into the survey

Difficult to Easy

10. You knew what to do if you needed help answering the question

Never to Always

11. Additional Comments__________________________________________________________________

Debriefing Questions

So, now that you have worked with the web instrument, in your own words, tell me the purpose of the instrument and the important features of the instrument.

*During the session, participants will only see the screen, and the probes will be not shown to the participant. The test administrator will read the probes aloud.

If manual entry:

I noticed that you manually entered your teacher’s names. Why did you choose that way of entering teacher names? Do you have any comments on the data requested?

If file upload:

I noticed you used the file upload feature to enter your teacher list. Why did you choose that way of entering teacher names? Do you have any comments on the file structure or template?

If pre-filled data:

You worked with the data already provided for your school. Do you have any comments on that data or how you had to work with it?

Describe for me what you think would happen next after you completed the list of teachers and submitted it.

[Interviewer instruction: If they covered the task during the main session, then you can skip the vignette.]

Vignette 1

Now I’d like you to explore the other information in the website and provide feedback on that information to me. I’m especially interested in whether you would use the information provided within the instrument, and how you would use it. (send ID and PIN to participant)

[Interviewer instruction: See where they go naturally, but direct them to the resource center if they do not seem interested in going there.]

[Ask the following probes after they have found the Resources tab. If they did not find the Resources tab, direct them to the tab and ask the below probes]

Go ahead and explore this area of the survey and let me know your thoughts.

Do you think anything on this screen may help you encourage your school staff to participate?

Do you find these links useful to you in your job?

Is there anything you would change about this screen I am showing you?

Vignette 2 (if time allows)

If there is no pre-loaded data, have them do a manual entry of 5 teachers or upload a file, whatever they did not do.

We’d now like you to [manually enter your teacher list/upload a file]. Please walk us through what you would do and enter at least five teachers. Please think aloud.

Additional Debriefing Questions

Do you have any questions about the Web instrument or how you would use it?

I have slides of the web instrument available. We can go over any of these screens you would like.

Probe on any screens the participant struggled with or provided interesting feedback on during the session. Show the screen when probing.

(If participant comments on a particular screen that yielded useful feedback) I noticed you had commented on this screen when going through this screen. Can you tell me more about what you were thinking?

(If participant struggled on a particular screen) You mentioned that you could not do what you were expected to do on this screen. Can you tell me more about what you were thinking?

Finishing up session

Was there anything else you would like to mention that you feel you haven’t had a chance to yet?

If signed consent/voucher before interview

Thank you very much for your time today and we will be mailing out that $75 cash incentive in the next few days.

If did not sign consent/voucher form before interview

Just as a reminder, you need to sign the document in the link before we can send the $75 cash incentive. Would you like me to send the link again and/or talk you through any problems you may be having with it?

If send again Just to confirm, what e-mail address would you like me to send that to?

Thank you very much for your time today and we will be mailing out that $75 cash incentive as soon as you sign that document.

If applicable We would also appreciate if you would be able to forward the recruitment advertisement for this study to other principals you may know. We can email you the principal recruitment advertisement again if that is preferable.

Close out steps

Check recording is the right length and save to the participant’s folder. Fill in the interviewer’s log sheet.

Interviewer Log Sheet

Type of school: Size_______ Grades _______ Public/Private/Charter_______

P#___

TA:______________________

Date: _____________________

Location: ________________________

Browser: ________________________

ID/PIN used during testing:

ID/PIN used during debriefing:

Main data entry:

Manual OR File Upload OR Pre-populated (circle one)

Answered security questions

Notes:

Resources tab (during session):

Use spontaneously? Y or N

Notes:

Vignette 1:

Did they find the resources tab? Yes/No

Did they find the links in the resources tab useful? Yes/No

Notes:

Vignette 2:

Manual OR File Upload OR Pre-populated (circle one)

Notes:

If Pre-populated data:

Add

Edit

Delete

Comments:

2023-24 NTPS Portal Focus Group

Attachment 4

Recruitment Screener

Are you still working at FILL SCHOOL?

__Yes, I still work at [SCHOOL]

__ No,

Are you still working as a [FILL ROLE]?

Yes

No (End interview)

During the summer of 2020, did you answer part of the National Teacher and Principal Survey by getting the list of teachers into the survey?

__Yes, -- Continue

__ I’m not sure

__No, someone else interacted with that survey (Still eligible if a target pop person)

2023-24 NTPS Portal Focus Group

Attachment 5

Moderator’s Guide

NTPS Portal Focus Group Moderator’s Guide

Date: ____/____/__2022____ Start time: ___________ AM / PM

Moderators: _______________________________

Mode: Remote—Video/ audio

INTRODUCTION [5 MINUTES]

Welcome! Thank you for agreeing to talk with us today. My name is [MODERATOR NAME], and I am here with my Census Bureau colleagues [NAMES]. We really appreciate that you are taking this time out of your day to share your opinion and feedback for the National Teacher and Principal Survey (NTPS). The NTPS is a series of surveys from the Department of Education and U.S. Census Bureau that gather information about school, principal, and teacher experiences. Some of you were recruited because you had previously participated in the survey. It’s ok if you don’t remember this experience or you didn’t previously participate; we still want to talk with you to improve the survey experience for others.

We have a couple of other people from the Department of Education listening to our conversation today and helping us take notes so that we don’t miss anything important that you say. We’ll also be audio and video recording our conversations today so we can make sure to get all your feedback. Your name will not be associated with the recording or any of the other data collected during the session. Does anyone have any questions or concerns about that?

Today before we start the focus group we are going to go into breakout rooms to meet one on one with you and get your individual feedback on a few questions from the NTPS surveys. You will be working with me or another colleague from the US Census Bureau. We will then come back to the same meeting as a group.

Cognitive Interview [30 MINUTES]

[Send them to breakout rooms. return no later than ____ regardless of where you are in the interview]

Hello, my name is ________________, and I work for the Census Bureau.

We are working with the U.S. Department of Education to develop and refine questions for the NTPS survey about education and your school. Before surveys like the NTPS are conducted, it’s important to try out the survey questions with the help of people such as yourself to make sure the questions make sense.

In just a moment I’ll ask you to answer a few questions about your school. After you answer, I will ask some follow-up questions about your answers and what you understood the questions to mean. There are no right or wrong answers. Our purpose is not to compile information about you or your school. Instead, your responses, along with those of others, will show us how to improve these questions for the survey.

Do you have any other questions before we begin?

Now I am going to share a question on my screen and have you answer it as if you were filling out the survey for real at your school. With one major difference please say your answer out loud and let me know what you are thinking and feeling as you answer the question.

Question 1-12 to 1-14

Question 1-12

Can you tell me more about your answer to question 1-12?

(All) Can you tell me in your own words what the second option means?

(If marked the 2^nd box in 1-12 and hasn’t been mentioned yet) Can you tell me more about the online resources that are used in the classes at your school?

(If did not mark the 2^nd box in 1-12 and hasn’t been mentioned yet) What kinds of classes do you think should be included under the second option?

(All) Can you tell me in your own words what the third option means?

(If marked the 3^rd box in 1-12 and hasn’t been mentioned yet) Can you tell me more about the online classes without in-person instruction that are taught at your school?

(If did not mark the 3^rd box in 1-12 and hasn’t been mentioned yet) What kinds of classes do you think should be included under the third option?

Question 1-13 & 1-14

(If participant answers Q1-14) Now looking at 1-14, can you tell me more about your answer?

Overall

These questions used the word “typically”. Can you tell me what “typically” means to you in these questions?

Question 2-1

Let’s turn to 2-1.

Can you tell me in your own words what the first part of this question before part a means?

What does the phrase “Early Warning System” mean to you in this question? What does the phrase “Multi-Tiered System of Supports (MTSS)” mean to you in this question?

Looking at part a, tell me more about your answer to this question. Did you find this easy or difficult to answer?

Question 2-2

Let’s turn to 2-2.

Did you see the italicized instruction under 2-2?

Can you tell me what this instruction means to you in your own words?

Does your school contract out any services mentioned here that are not provided at the actual school? (If yes) Did you include them in your answers?

Question 3-2

Let’s turn to 3-2.

Were any of these questions difficult to answer? (If yes) Which ones were difficult?

Thank you very much for your feedback on these questions today. Now we will return to the bigger group discussion.

Focus Group [55 minutes total]

Intro [5 MINUTES]

[After returning to bigger group]

Thanks for your feedback so far today. Again, I’m _______. I’ll be leading the next part of our discussion today, which will focus on the NTPS Portal Website. The Portal Website is the first step of the survey when a school is selected. It is where we ask for basic information about your school, such as the grades taught in your school, and we also ask for a list of teachers. We are going to focus on what information would help encourage you and your staff to complete this survey.

RULES

Let’s go over a few ground rules quickly.

• First, we would appreciate if you can turn on your camera and keep it on. If you need to take a break and turn it off for a minute or two, that is okay.

• If you would like to speak please use the “raise your hand” function.

• I will be asking a lot of questions. There are no right or wrong answers to my questions; we just want your honest opinions. We don't need to have everyone agree. In fact, if you have a view different from others it is especially important for us to hear it because you may be the only person who is representing a different point of view here today.

• Please keep respondents’ information confidential. If you want to tell us about any experiences that you had filling out the surveys, please don’t share any names or any other information that could identify students or other staff.

• Only one person speaks at a time. Please do not speak while somebody else is talking because it’s hard to understand when too many people talk at once.

• We’d like everyone to get a turn. If you are someone who takes a little while to warm up in groups, please be sure to speak up and let us know what you are thinking. If you are the type of person who always participates, please make sure to give others a chance to talk as well.

• If you would like to use the chat function you may, but please note that it is easy to miss things in the chat when we are talking. So, if it’s really important, please raise your hand and tell us.

• Your responses are for informational purposes only. We ask that you speak freely so we can get better feedback on the National Teacher and Principal Survey.

• Due to limited time, we must move the discussion along and may sometimes have to interrupt in order to do so. Please do not be offended. If you have anything else that you’d like to share with us that you did not have time to say, please send your feedback in an email to [email protected]. [PASTE EMAIL IN CHAT]

Does anyone have any questions before we get started?

ICEBREAKER [5 MINUTES]

The first thing I want to do is get to know each of you a little better. Let's have each one of you tell us briefly:

• Your first name,

• And what your role in your school is?

I’ll go first. My name is… [BE BRIEF]

Please feel free to jump in and introduce yourself.

[SINCE THIS IS A POPCORN STYLE OF INTRODUCTIONS, CHECK THE ROSTER TO MAKE SURE EACH PARTICIPANT HAD THE OPPORTUNITY TO DO SO. MAKE SURE THE CO-MODERATOR HAS A TURN AS WELL.]

PROBES

Thank you for telling us a little more about you.

Mailing Invitation [20 MINUTES]

Now we are going to start by talking about the letter that is sent as an invitation to participate in the National Teacher and Principal Survey. You may or may not remember this, but I am going to show you an example of the letter [Show TLF welcome letter, give them time to look over and read]

1. What are your first impressions of this letter?

2. Do you think you would respond if you received this letter? Please tell me more about that.

3. What, if anything, about the letter do you dislike? Please tell me more about that.

4. What, if anything, about the letter do you like or think would encourage you to respond?

5. This letter tells you that you have been chosen by your school coordinator to upload or verify the list of teachers at your school in the portal. What do you think about that task? (If needed: What do you think this might entail? Why do you think you would be asked to do this? Does this seem easy or difficult?)

6. What would you do next if you had received this letter for real in your mailbox at school?

Portal [15 MINUTES]

Now imagine that you decided to respond to the survey. If you had logged into the portal, this is the screen you might first see if we had been able to prepopulate your teacher list using existing information:

7. 

However, if we couldn’t prepopulate your teacher list, this is what we you would see:

Now after you had finished entering or verifying your teacher information you would see this screen:

8. What, if anything, on this website might motivate your school staff to participate?

9. Assuming you were the school coordinator and you wanted staff to participate, what do you think you might do to motivate your staff to participate?

Now I am going to show you some screens about ideas to help encourage participation

10. The first screen is a sample of emails you can send to encourage participation and the second screen is a list of organizations that endorse the survey and encourage their members to participate. What do you think about this list of organizations? Do you think seeing it would encourage you to participate? If you participated in the past, did you see this list of organizations?

11. What kind of resources do you think we could add to help encourage your school and staff to participate?

12. [If needed] Please tell me more about that?

Wrapping Up [5 mins (there is 5 minutes buffer for other sections to run over)]

13. Is there anything else you’d like to tell us that you haven’t had a chance to mention yet?

1. Anyone else?

Thank you very much for your time! The feedback that you shared was very helpful. We’ve been taking careful notes so we can make recommendations. We will mail out your incentives within 5 business days. [IN CASE NOT ALL SIGNED CONSENT] Please be sure to sign the consent form- if you have not, we cannot mail out your incentive until you do.

26