2019-2020 National Teacher and Principal Survey (NTPS) Cognitive Interviews

National Center for Education Statistics

Attachment 5

Cognitive Interview Protocols

2019-2020 National Teacher and Principal Survey (NTPS)

Cognitive Interviews

OMB# 1850-0803 v.218

November 2017

Section 1: Introduction

Participant ID #: |___|___|___|__|__| __|___|___|

Interview Date: |__|__| / |__|__| / |__|__|__|__| (mm/dd/yyyy)

Interviewer initials: _____

Start Time: ____________ AM / PM End Time: ____________ AM / PM

Welcome participant and thank them for their time.

Read or paraphrase: Hello, my name is ________________, and I work for the Census Bureau. Thank you for agreeing to participate in our study. [IF OBSERVER PRESENT]: I also want to let you know that this interview is being observed by [DESCRIBE OBSERVER’S ROLE: e.g., someone who works on the NTPS]. They are observing the interview because they are also interested in how the questions work for people like you.

We are working with the National Center for Education Statistics, within the U.S. Department of Education, to develop and refine questions for the National Teacher and Principal Survey. Before surveys are conducted, it’s important to test the questions with the help of people such as yourself.

It is important that the questions make sense, are easy to answer, and that everyone understands the questions the same way. If you agree to take part in this study, I will hand you a questionnaire to complete. After you complete it, I will ask you some questions about the answers you gave. There are no right or wrong answers. In this study, I am less interested in your actual answers to the questions and more interested in what you thought about the questions and how you understood them. After you have gone through the questionnaire, I will ask some questions about your answers, such as what you understood the questions to mean and whether the questions reflect your actual circumstances.

In order to concentrate on what you are saying rather than having to take notes, and to make sure that I don’t miss anything you say, I would like to audio record this interview. Is that okay with you? [IF NOT OKAY WITH RECORDING]: Okay, that’s no problem.

I want you to know that your participation is voluntary and all of your responses are protected from disclosure by federal statute. I want you to know that you can speak freely. Also, you do not have to answer any question you don’t want to and can stop the interview at any time if you wish.

Your participation in this interview is very important because it will help us improve the questionnaire. We’re scheduled to be here for about [60 / 90] minutes, and I won’t keep you any longer. As a thank you for your help, we will give you [$40/$75] cash after the interview is completed.

Do you have any questions?

[Write down questions]

Section 2: Informed Consent

Before we start, I’m going to give you a written consent form. This document explains the purpose of the study, how we will protect your information, and who you can call if you have questions.

[Hand the Informed Consent document to the participant to read.]

In addition to the point I have just mentioned about confidentiality I would like to draw your attention to the fact that:

• Your participation is voluntary and you may stop the interview at any time

• You can refuse to answer any of the questions asked or take a break at any time

• All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151)

[When completed:

1. Ask participant if (s)he has any questions.

2. Ask participant if (s)he agrees to participate and agrees to be audio recorded.

3. If participant agrees, obtain the respondent’s signature to proceed – on two (2) copies.

4. Make sure respondent takes one copy with them when the interview is over.

5. IF PARTICIPANT PROVIDES CONSENT TO HAVE THE SESSION AUDIO-TAPED:] I will now turn on the audio recorder.

Section 3: Cognitive Interviewing for Paper Mode

[For paper SHOW RESPONDENT APPROPRIATE NTPS QUESTIONNAIRE:] This is the [Teacher / Principal / School] Questionnaire of the National Teacher and Principal Survey. If this were the real survey, you would receive this questionnaire from a school coordinator or in your mailbox at your school. It would have your name and school address in the address label. I am going to ask you to fill out the first section of the questionnaire just as you would if you had received it at school. Then, when you get to the section that says STOP HERE, I will give you some additional instructions.

[HAND QUESTIONNAIRE TO RESPONDENT:] Please go ahead and start filling out the questionnaire. Please remember to STOP when you get to the section that instructs you to stop.

[After filling out core sections and stopping]

Now I am going to ask you to fill out sections of this questionnaire just as you would if you had received it at school but with one difference.

I may occasionally stop the interview to ask you some additional questions about the questions you just answered.

I would also like you to think aloud as you answer the questions. While I am interested in your answers, I am even more interested in the process you go through when you answer the questions. So, as much as possible, I’d like you to think aloud for me; meaning, talk me through your thought process as you answer each question. From time to time, I may also ask you what you are thinking to remind you to do this.

I find it helps people to get the idea of thinking aloud while they are answering a question if they do a practice question first. Let’s begin with a practice question. Remember to think aloud as you answer.

How many windows are there in the house or apartment where you live?

_____________ WINDOWS

[IF NEEDED: Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

PROBES:

• How did you come up with that answer?

• Tell me more about that. Why did you say [ANSWER]?

• I noticed that you hesitated. Tell me what you were thinking.]

As you complete the questionnaire, I may stop you in a couple of places to skip you to the next set of questions to answer. After you complete the questionnaire, I am going to ask you about some specific questions that you answered.

Before we begin, do you have any questions about the process?

[Let respondent begin modules we are testing. In addition to the scripted probes that are included for each questionnaire as shown in Section 5, additional general probes that may be used as needed are presented below:]

• What did you think this question was asking?

• Can you rephrase the question in your own words?

• How did you go about answering this question?

• What were you thinking about when you answered this question?

• How did you decide on your answer?

• What was the reason for that answer?

• How easy or difficult was it to answer this question?

• What is an example?

• How did you feel about answering this question?

• What does this term or phrase mean to you?

• Can you tell me more about that?

• You seemed to hesitate / take a moment before providing your response. Can you tell me what you were thinking about?

• What do you mean by…?

• Is there any additional information that we should have provided to help you answer that question?

• Does this term / question make sense to you, or is there something else that would make more sense?

Cognitive Interviewing for Web Mode

[For web SHOW RESPONDENT APPROPRIATE NTPS login instruction letter:] This is the [Teacher / Principal / School] introductory letter with questionnaire login instructions for the National Teacher and Principal Survey. If this were the real survey, you would receive this letter from a school coordinator or in your mailbox at your school. It would have your name and school address in the address label. I am going to ask you to login and fill out the first section of the survey just as you would if you had received the letter at school. Then, when you get to section [interviewer fill based on Teacher/Principal/School] I will ask you to stop, and I will give you some additional instructions.

[HAND login instruction letter TO RESPONDENT:] Please go ahead and login and start filling out the questionnaire. Please remember I will ask you to STOP after section [interviewer fill based on Teacher/Principal/ School].

[After filling out core sections and stopping]

Now I am going to ask you to fill out sections of this questionnaire just as you would if you were filling it out at school but with one difference.

I may also occasionally stop the interview to ask you some additional questions about the questions you just answered.

I would also like you to think aloud as you answer the questions. While I am interested in your answers, I am even more interested in the process you go through when you answer the questions. So, as much as possible, I’d like you to think aloud for me; meaning, talk me through your thought process as you answer each question. From time to time, I may also ask you what you are thinking to remind you to do this.

I find it helps people to get the idea of thinking aloud while they are answering a question if they do a practice question first. Let’s begin with a practice question. Remember to think aloud as you answer.

How many windows are there in the house or apartment where you live?

_____________ WINDOWS

[IF NEEDED: Try to visualize the place where you live, and think about how many windows there are in that place. As you count up the windows, tell me what you are seeing and thinking about.

PROBES:

• How did you come up with that answer?

• Tell me more about that. Why did you say [ANSWER]?

• I noticed that you hesitated. Tell me what you were thinking.]

As you complete the questionnaire, I may stop you in a couple of places to skip you to the next set of questions to answer. After you complete the questionnaire, I am going to ask you about some specific questions that you answered.

Before we begin, do you have any questions about the process?

[Let respondent begin modules we are testing. In addition to the scripted probes that are included for each questionnaire as shown in Section 5, additional general probes that may be used as needed are presented below:]

• What did you think this question was asking?

• Can you rephrase the question in your own words?

• How did you go about answering this question?

• What were you thinking about when you answered this question?

• How did you decide on your answer?

• What was the reason for that answer?

• How easy or difficult was it to answer this question?

• What is an example?

• How did you feel about answering this question?

• What does this term or phrase mean to you?

• Can you tell me more about that?

• You seemed to hesitate / take a moment before providing your response. Can you tell me what you were thinking about?

• What do you mean by…?

• Is there any additional information that we should have provided to help you answer that question?

• Does this term / question make sense to you, or is there something else that would make more sense?

Login Letter for Web Mode

Instructions for completing the [Principal/Teacher/School] Questionnaire

Dear [School Head name/ Principal name/ Teacher name /School Staff Member],

As part of your school’s participation in the National Teacher and Principal Survey (NTPS), you have been [Interviewer fill as appropriate] [selected to fill out the NTPS Principal Questionnaire / selected to complete the NTPS Teacher Questionnaire/ identified by your school’s principal or designated survey coordinator as the staff member who is most knowledgeable about your school and who can provide the required information on the NTPS School Questionnaire]. This survey is administered by the U.S. Census Bureau on behalf of the National Center for Education Statistics (NCES), the statistical agency of the U.S. Department of Education.

The NTPS is one of the Department of Education’s primary sources of information about what is happening in K-12 schools; therefore, your response is very important.

Respond now at https://respond.census.gov/ntpspq

Log in using this User ID:

Please complete the survey within two weeks. The survey will take approximately [22/30/40] minutes to complete. Please be assured that both the U.S. Department of Education and the U.S. Census Bureau follow strict procedures to protect the confidentiality of study participants. All of the information you provide may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S.C. §9573 and 6 U.S.C. §151). Your responses to the survey questions will not be shared with your school staff or your school district.

Additional information about the NTPS is available on the [Principal/Teacher/School] Questionnaire respondent website. If you have any questions about the survey or need assistance, please contact the U.S. Census Bureau at 1–888–595–1338 between 8:00 a.m. and 8:00 p.m. (Eastern Time) Monday through Friday. The U.S. Census Bureau is also available to answer your questions via e-mail at [email protected].

Thank you in advance for your participation in this important survey.

Sincerely,

Peggy G. Carr, Ph. D.

Acting Commissioner

National Center for Education Statistics

U.S. Department of Education

Section 4: Debriefing Procedures

• Ask for any last observations / comments / concerns.

• Stop the digital audio recorder.

• Thank the participant for coming in and for helping to test the survey.

• Give the [$40 / $75] incentive and their copy of the consent form, have the respondent sign the voucher, and escort them from the interview room.

Section 5: Scripted Probes by Survey

Teacher Survey

Question 1-1

• What does an “itinerant teacher” mean to you? If needed: Does the text in parenthesis match your understanding of this term?

Question 2-1

• Please tell me what grades are taught in your school? If needed: Just to confirm, do you ever teach students of any grades besides the ones you selected?

• I’m going to give you an alternate question. Please answer as if this was part of the survey you just did.
  

• Between these two versions, which do you prefer, if either?

Question 2-7

• Please tell me more about your answer? If needed: How are students assigned to you during the day?

• What type of teachers may not have an option that applies to them in the above list?

• Can you think of a type of teacher that might need to choose more than one of the response options?

• Can you give examples of the types of teachers that would mark “You instruct several classes of different students most or all of the day in one or more subjects?” If needed: Would it apply to primary as well as secondary teachers? Please tell me why you say that?

• Can you give examples of the types of teachers that would mark “You instruct the same group of students all or most of the day in multiple subjects?” If needed: Would it apply only to primary teachers or could it apply to secondary teachers as well?

Question 2-11

• Please tell me more about the subjects you teach each day and how it varies by day?

• Now for item (a) please tell me more about your answer. If they put in an answer for (a)(1) and didn’t mention during think aloud: how easy or difficult was it to report separately for reading instruction? If they left (a)(1) blank: I noticed you didn’t put an answer for question a1. Can you tell me more about that?

• Do you have any subjects that are interdisciplinary, for example math and science? How did you come up with your answer for those subjects?

• If it appears this question format is not conducive to how the teacher’s day is structured: Now please turn to page 11. Do you think that it would be easier to describe how many minutes you spent teaching each subject with this format in Q 2-13?

Question 2-12

• For question 2-12 please tell me more about how you came up with your answer? If needed: How are classes or sections divided during your day?

Question 2-13

• Now would you please tell me more about how your day is structured?

• Were you able to find a subject-matter code in the table on page 10 that fits well for every subject you teach? Interviewer note if they used the official subject matter label provided in Table 1 or something else.

• If it appears this question format is not conducive to how the teacher’s day is structured: Now please turn back to page 9. Do you think that it would be easier to describe how many minutes you spent teaching each subject with this format in Q2-11?

Question 3-1

• For Q 3-1d (POINT) would you please tell me more about your answer? What do the phrases “College of Education, School of Education or Department of Education” mean to you in this question?

• In your own words what does the option “It was awarded by another college, school, or department, not in education”, mean?

Question 3-4

• Please tell me in your own words what “courses that focused SOLELY on teaching methods” means in this question?

• IF YES to 3-4, please give me an example of a course you took and what you learned in that course.

• Did you notice the instruction to “NOT include professional development courses, workshops, or seminars”? If they select Yes: Were you including professional development courses when you selected yes?

Q4-1

• Please tell me in your own words what an “alternative route to certification program” means?

• IF YES—please tell me about how you entered the teaching profession. What did you do before becoming a teacher?

Question 4-2

• Please tell me more about your answer for this item?

• In your owns words what does the first option “ regular or standard state certificate mean”, what about “ advanced professional certificate”?

• What is a probationary certificate?

• What about a temporary or provisional certificate, and a waiver or emergency certificate?

• What difference, if any, do you see between the last three options “probationary certificate, temporary/ provisional certificate, and waiver/emergency certificate”?

Question 5-5

• Can you please tell me in your own words what this question means by “a formal schoolwide or districtwide program for beginning teachers aimed to enhance teachers’ effectiveness”?

Question 5-6

• In Q5-6f please tell me in your own words what do you think is meant by “observation and feedback… Beyond any formal administrative observation”?

Question 5-7

• In your own words what is a “master or mentor teacher”? If needed: Does your school have this type of program what do they call it?

• IF YES: Please tell me about your master or mentor teacher? How are they assigned? At the school or district level?

Questions 6-1 to 6-3

• FOR Q 6-1 please tell me more about how you calculated your answer to 6-1? IF NEEDED: Is this information you know from memory, or would you need to look this up?

• What does “How many hours does your contract require you to work ” mean to you in this question?

• What does “typical FULL WEEK” mean to you in these questions?

• Please tell me more about how you calculated your answer for Q6-2?

• Could you tell me more about the types of activities you consider to be “delivering instruction”? If needed: Did you include time spent on discipline or administrative tasks like grading and attendance? What about time spent working one on one or with a small group of students while the class performs another task?

• For question 6-3, I see you reported X more hours than in question 6-1. Please tell me how you came up with your additional weekly hours? What kind of things did you include for before and after school? What do you do on the weekends for school?

Question 7-5

• How easy or difficult was it to answer this question?

• What does “disappointments” in item (a) mean to you? Also in this item, what is meant by the phrase “aren’t really worth it”?

• In your own words, what is item (f) asking?

• How honest would you expect other teachers to be when answering these questions?

Q7-6

• How easy or difficult was it to answer this question?

• In this question, what is meant by “remain in teaching”? If needed: Do you think the question is asking about remaining in teaching at ANY school or only at this school?

• What do you see as the difference between the second, third, and fourth response options?

• In your own words, what is meant by the fifth response option, “Until a specific life event occurs?” IF NEEDED: Other than the examples provided, what are some examples of this?

Question 7-9

• Please tell me more about your answer to Q 7-9.

• In your own words could you give me an example of what “Excellent” health is? What about “Poor”?

Question 7-10

• Please tell me how you came up with your answer to 7-10.

• In your own words what does a “24 hour period” mean in this question?

• [If needed] Did you count the time from when you laid down in bed to when you physically got up? Or did you only include the amount of time you know you were asleep.

Question 9-1

• In your own words, what is this question asking?

• What does “assigned at birth” mean to you in this question?

• What concerns, if any, do you have about providing this information on this survey?

Question 9-2

• IF NEEDED: Did you consult the definition for “gender” in order to answer this question?

• Does the definition provided for “gender” align with your understanding of this term?

• How easy or difficult was it for you to select a response to this question?

• What concerns, if any, do you have about providing this information on a survey?

Question 9-3

• How easy or difficult was it for you to select a response to this question?

• What concerns, if any, do you have about providing this information on a survey?

Principal Survey

Question 1-5

• How did you come up with your answer to this question?

• How easy or difficult was it to answer this question?

Question 1-10

• How did you come up with your answer to this question?

• How easy or difficult was it to answer this question?

• What does “regularly taught” mean to you in this question?

Question 1-12

• How did you come up with your answer to this question?

• How easy or difficult was it to answer this question?

Question 4-1

• How did you come up with your answer to this question? IF NEEDED: Is this information you know from memory, or would you need to look this up?

• What does days “you are required to work under your current contract” mean to you in this question?

• Did the instruction below the question influence your answer? [IF YES:] How did it influence your answer?

Question 4-2

• How easy or difficult was it to answer this question?

• What does “throughout the school year” mean to you in this question?

• Looking at items (a) though (d) in this grid, does the way these different tasks are grouped align with how you would group them? IF NO: How would you group them instead?

• For item (a), what does “internal administrative tasks” mean to you?

• IF “OTHER” RESPONSE: Can you tell me more about what you included for item (e) “Other”?

• [Besides what you included in “Other”,] Is there anything missing from this list that you spend time on throughout the year?

• IF NOT APPARENT: Did you check to make sure your responses totaled 100%?

Question 4-3

• IF NEEDED: How did you come up with your answer for this question? IF NEEDED: Did you include time spent on the weekends?

• What do you consider to be a “typical full week”?

• What does “school-related activities” mean to you in this question?

• For question 4-3, I see you reported X more hours than in question 4-1. Please tell me how you came up with your additional weekly hours? What kind of things did you include for before and after school? What do you do on the weekends for school?

Question 4-4

• In your own words, what is this question asking?

• How did you come up with your answer to this question?

• IF NEEDED: Did you consult the definitions of these terms to answer the question?

• Does the definition of “meet and confer agreement” align with your understanding of this term? What about the definition of “collective bargaining agreements”?

• Is it possible for principals to be represented by both types of agreements?

Question 4-5

• How easy or difficult was it to answer this question?

• What does “disappointments” in item (a) mean to you? Also in this item, what is meant by the phrase “aren’t really worth it”?

• In your own words, what is item (f) asking?

• How honest would you expect other principals to be when answering these questions?

Question 4-6

• How easy or difficult was it to answer this question?

• In this question, what is meant by “remain a principal”? If needed: Do you think the question is asking about remaining a principal at ANY school or only at this school?

• What do you see as the difference between the second, third, and fourth response options?

• In your own words, what is meant by the fifth response option, “Until a specific life event occurs?” IF NEEDED: Other than the examples provided, what are some examples of this?

Question 5-1

• In your own words, what is this question asking?

• What does “assigned at birth” mean to you in this question?

• What concerns, if any, do you have about providing this information on this survey?

Question 5-2

• IF NEEDED: Did you consult the definition for “gender” in order to answer this question?

• Does the definition provided for “gender” align with your understanding of this term?

• How easy or difficult was it for you to select a response to this question?

• What concerns, if any, do you have about providing this information on a survey?

Question 5-3

• How easy or difficult was it for you to select a response to this question?

• What concerns, if any, do you have about providing this information on a survey?

Question 5-7

• How easy or difficult was it for you to answer this question?

• What concerns, if any, do you have about providing your salary information on a survey?

School Survey

Question 1-1

• [if needed] How did you decide on your answer?

• I’m going to give you an alternate question. Please answer it as if this was a part of the survey you just did.
  

• Between these two versions, which do you prefer, if either?

Question 1-3

• Can you tell me more about your answer?

• If we had asked for the average daily attendance as the number of students instead of a percentage, how would you answer?

• Would you prefer answering as a percentage, or as a number?

Question 1-6

• Now let’s turn to 1-6. Can you tell me more about how you came to your answer?

Question 2-1

• Now let’s turn to 2-1. Can you tell me how you came up with your answer to this question?

• In your own words, can you tell me the difference between a “full time” teacher and a “part time” teacher?

• [if at elementary school/pre-kindergarten checked in 1-1] Did you include any pre-kindergarten teachers?

• Does your school have any student teachers? [if yes] Did you include them here?

• Did you include any student teachers in your answer to this question? What about short-term substitutes?

• Do you have any substitutes at this school? Are they short-term or long-term?

• What is the difference between a long-term substitute teacher, and a short-term substitute teacher?

• Were there any teachers you didn’t know how to classify?

Question 2-2

• Was this question easy or difficult to answer?

• Is this sort of demographic information kept for all of the teachers at this school?

Question 2-3

• How did you decide on your answer?

• Were there any parts of this question that you had trouble answering?

Question 2-4

• What does “support services professional staff” mean to you in this question?

Question 2-5

• In your own words, can you tell me the difference between an “instructional aide” and an “non-instructional aide”?

• Does your school use this distinction?

• Whas it difficult for you to decide where to include an aide?

Question 2-6

• What does the term “specialist” mean to you in this question?

• How did you decide on your answers?

• What does the term “coach” mean to you in this question?

Question 2-7

• How did you come to your answer for this question?

• What does “this school’s hiring authority” mean to you in this question?

• About how many teaching vacancies were there in this school?

• Can you tell me more about your answers to this question?

• What timeframe were you thinking of when you answered this question?

Question 2-8

• Can you tell me more about your answer to this question?

• If a teacher had taught in another school in your county and was transferred to your school this school year, would you report that teacher in question 2-8a?

• Can you tell me in your own words what “Record HEAD COUNTS, not FTEs” means in this question?

Question 3-1

• In your own words, what does ‘certificates of completion’ mean to you?

Question 3-2

• What does the phrase “standard diploma” mean to you in this question?

Question 3-3

• How did you come to your answer? IF NOT A HIGH SCHOOL- how certain are you of your answer?

• Would you have to look at records or speak with someone else to be able to answer this question?

Question 4-1b

• Were there any students with IEP plans that you were unsure of including?

• Does your school have any prekindergarten/postsecondary students?

Question 4-2

• Can you tell me more about how you came to your answer?

• Were there any students that you had trouble classifying for part B?

Question 5-1, page 21

• Can you tell me more about how you came to your answer?