2018 Teaching and Learning International Survey (TALIS 2018) Principals Focus Group

Appendix A

Principal Contact Materials and Facilitator’s Guide

2018 Teaching and Learning International Survey
(TALIS 2018) Principals Focus Group

OMB # 1850-0803 v. 167

August 2016

Exhibit A - 1. Principal Focus Group Recruitment Contact 1

Exhibit A - 2. Principal Focus Group Call Script 2

Exhibit A - 3. Principal Focus Group Consent Form 4

Exhibit A - 4. Principal Focus Group Facilitator’s Guide 5

Teaching and Learning International Survey (TALIS)

Principal Focus Group Recruitment Contact

(will be shared via email, letter, or phone)

Dear _____________,

My name is ______________ and I am writing on behalf of NCES, the National Center for Education Statistics. NCES supports international efforts to collect and report data on education and is currently preparing to administer a field trial for TALIS –the Teaching and Learning International Survey. In preparation for the TALIS field trial we are interested in talking with school administrators about what they think about international education studies like TALIS.

To that end, we would like to meet via WebEx between <<TIME>> on <<DATE>> with up to 9 principals to discuss TALIS. During the meeting, we will discuss your experiences with other studies you may have participated in and garner advice to guide the development of communication materials for TALIS. Based on your recommendations, we will create or adapt new materials for TALIS. We also want to know about the decision making that is involved in determining whether to participate in studies like TALIS and how we might better word our materials and discussions when approaching schools to participate.

So what’s in it for you? If you agree to take part in the WebEx meeting on <<DATE>> you will receive a $50 check to thank you for your time and participation.

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). Your participation is voluntary and your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, § 9573).

While we recognize that this is a busy time for schools, we would very much appreciate the opportunity to hear your insights in order to more effectively reach out to schools during our TALIS 2018 recruitment efforts. Please let us know at your earliest convenience if you are able to participate.

Sincerely, ….

Teaching and Learning International Survey (TALIS)

Principal Focus Group Recruitment Call Script

Hello, my name is <INTERVIEWER’S NAME> and I am calling on behalf of the National Center for Education Statistics, a division of the U.S. Department of Education. May I please speak with <principals name>?

We are recruiting potential participants for a paid focus group about how school administrators view participating in international education studies. As the U.S. begins to prepare for the TALIS field trial we are interested in talking with school administrators about what they think about international education studies like TALIS so the data collection team can make the experience for schools as good as possible

We are not selling anything. We are simply interested in hearing your thoughts about international education studies because your school is the very type of school we are interested in hearing from. We recognize your school is <insert school characteristics>

• a suburban school and we recognize that you serve a diverse student population with a focus on high achievement and several competing interests for time.

• a large urban school and we’d like to learn about unique challenges or issues with participating in international studies like TALIS.

• a rural school and we would like to learn about the unique challenges or issues that face a school like yours when considering participation in a survey like TALIS.

^{Your opinions are very important to us and to thank you for your time, we will give} you $50 as a token of our appreciation.

We would like to meet via WebEx between <<TIME>> on <<DATE>> with other principals to discuss TALIS. During the meeting, we will discuss your experiences with other studies you may have participated in and garner your advice. We also want to learn how we might better word our materials and discussions when approaching schools to participate.

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 [if requested: ESRA 2002, 20 U.S.C., § 9543]. Your participation is voluntary and your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, § 9573).

Would you be willing and available to participate in a discussion about international education studies like TALIS?

If YES, continue. If NO, thank and terminate.

^{As I mentioned, we will give} you $50 as a token of our appreciation ^{for your participation in this WebEx discussion}. The discussion will last approximately 2 hours. Prior to the WebEx you will receive specific instructions for joining the call on <DATE> at 4:00 pm EST. You will also receive a packet via Federal Express containing the draft versions of materials for TALIS. These will be discussed during the WebEx.

RESPONDENT NAME:

[VERIFY:]

SCHOOL NAME:

SCHOOL ADDRESS:

TELEPHONE/EMAIL:

Thank you. Your Fed Ex package should arrive on [DATE = +2 days from call date]

[Message in case not available : Hello, I’m calling on behalf of the National Center for Education Statistics, part of the U.S. Department of Education. We are recruiting potential participants for a paid focus group about how school administrators view participating in international education studies. ^{Your opinions are very important to us and to thank you for your time, we will give} you $50 as a token of our appreciation. If you are interested, give us a call at < phone number > or email us at < >. Thank you. Goodbye.]

TEACHING AND LEARNING INTERNATIONAL SURVEY (TALIS)

Principal Focus Group Consent Form

CONSENT FORM

PARTICIPATION IN A QUALITATIVE RESEARCH PROJECT

FOCUS GROUP PURPOSE

A focus group will be held with middle and high school principals about participation in voluntary education studies. The information obtained from this group will guide school and teacher recruitment strategies for the upcoming Teacher and Learning International Survey (TALIS) 2018 in the effort to improve participation rates and collect valid data on education contexts across the U.S.

AGENCY CONDUCTING THE STUDY

This project is being conducted by the National Center for Education Statistics, which is part of the U.S. Department of Education. Representatives from Hager Sharp, an independent communications firm, will administer the interviews and focus group.

CONFIDENTIALITY

NCES is authorized to conduct this study under the Education Sciences Reform Act of 2002 (ESRA 2002, 20 U.S.C., § 9543). Your participation is voluntary and your answers may be used only for statistical purposes and may not be disclosed, or used, in identifiable form for any other purpose except as required by law (20 U.S. Code, § 9573).

QUESTIONS

If you have any questions about the discussion group, please call Lauren Salay of Hager Sharp at 202-XXX-XXXX.

CONSENT

I, __________________________________, agree to participate in this focus group.

SIGNATURE

Teaching and Learning International Survey (TALIS) 2018

^{Facilitator’s Guide:}

^{Principals Focus Group}

Research Questions:

• What value do international surveys and comparisons have in education?

• Does the format of TALIS 2018 (online survey of teachers and principals) influence principals’ motivations to participate?

• What information do principals use when deciding whether or not to participate in a survey like TALIS? Who do they consult?

• What barriers prevent or discourage participation in TALIS?

• What would minimize these barriers in the principals’ school?

• What factors might motivate teachers to participate in a voluntary survey such as TALIS?

• What are the perceived benefits of participation for teachers and principals? How do the principals value TALIS?

• What current or potential TALIS messages or materials do principals find relevant, useful, and informative?

• What would be the most effective channels, formats, and materials to use to communicate with principals?

• What other groups (e.g., district, other administrators) would be central to the decision making process and what would be the best way to reach those influencers?

INTRODUCTION (5 MINUTES)

 My name is Maria Ivancin and I work for the Market Research Bureau, a research company, working on behalf of the National Center for Education Statistics (NCES), which is a part of the U.S. Department of Education. We do research to help clients make decisions about many different types of issues.

Today we are talking to you in a WebEx-based focus group. It allows us to talk with people in a bit more depth than when using other methods, such as surveys. We are looking to get your honest opinions. I will be asking questions, which are meant to generate discussion. There are no right or wrong answers. I encourage you to express your opinion even if it might be different from other participants. We want all types of viewpoints – positive, negative, and everything in between. We do not have to come to any consensus on what we talk about. Your individual opinion is important so I would like to hear from everyone. We will be audio recording the session so that I have a record of the discussion to write my report. Once the report is written we will destroy the recording.

Your feedback will be used only for statistical purposes and may not be used for any other purpose except as required by law. No names will appear in the summary report about what we learn through this focus group. If at any time you decide you do not want continue, that is your choice and you may stop. Before we begin, do you have any questions about what I just said?”

 Draft materials will be presented along with other information, which you should have received in the mail. I want to point out that these are not materials that I have developed, so you will not flatter me or insult me by anything that you say. My job is to ensure that we cover everything that we need to cover and that I keep the discussion going. I have no stake in the outcome of the discussion.

 Does everyone have available the packet of materials that you received? I’ll let you know at what point you should reference the packet. If you do not have your packet, you should have electronic versions of the materials. [ALTERNATE: If you don’t have your packet, that’s OK. We will walk through these materials as a group and show them on the screen.]

 Are there any questions about using the WebEx technology? [ANSWER ANY QUESTIONS ABOUT WEBEX]

 I’d like everyone to introduce themselves. Let me start (MODERATOR INTRO). Now, tell me about yourself. When you do, please be sure to tell me your name (first name is okay), a little bit about your school (size/grades), how long you’ve been in education, and anything else you’d like to tell us about yourself.

VALUE OF INTERNATIONAL ASSESSMENTS (10 MINUTES)

• Let’s start by talking broadly about national and international education studies involving samples of primary or secondary schools, staff and students. Are you aware of any assessments or surveys like that? Do any studies come to mind?

• Have you ever participated in any of these kinds of studies? Do you know anyone whose school has participated?

• What is your overall reaction to such studies? What questions or concerns do you have about these studies?

• To whom are they valuable? In what ways are they necessary or unnecessary?

• Let’s talk specifically about international studies – where we can compare aspects of the education systems across different countries. How important are international comparisons? PROBE:

o For our nation?

^{o Your state?}

^{o Your school?}

^{o District- or building-level administrators in general?}

 Let’s say you were to participate in an international study. What would be the most compelling reason in your view to do so?

BARRIERS TO TALIS (10 MINUTES)

• Let’s talk more specifically about a study called the Teaching and Learning International Survey, or TALIS.

• What, if anything, have you heard or do you know about TALIS?
  
  

• For the following discussion, I’d like you to refer to the packet of information about TALIS. Please open it up at this time. (PAUSE) If you don’t have your packet, please refer to the electronic versions we provided to you. [ALTERNATE: If you don’t have your packet, that’s OK. We will walk through these materials as a group and show them on the screen.]
  
  

• This is the information you would receive if your school was asked to participate. The first thing I’d like you to do with it is glance through it as you usually would, identify the piece you would read first, and look it over.

• [GIVE PARTICIPANTS A MOMENT TO REVIEW PACKETS, THEN GO AROUND TO EACH PRINCIPAL AND HAVE EACH ONE NAME THE PIECE THEY WOULD READ FIRST AND WHAT MADE THEM SELECT THAT PIECE.]

• At this point, what are your thoughts on TALIS?

• Is there anything that appeals to YOU about participating?

• ^{What are some potential drawbacks or issues regarding participation that you can identify?}

^PROBE:

• Perhaps the idea of selecting a sample of teachers or the questionnaire?

• Maybe the time it takes to complete it, the online nature of the study, or policies that prevent teachers from receiving incentives?

• Please take a few minutes to look through the rest of the packet. Based on the materials that you read and anything else that you may know about TALIS how would you react if your school was asked to participate in TALIS?

• What factors would you consider? Timing of the survey, time involved for you and your staff, incentives...

• What questions would you have?

• What specific information would you need or want in order to decide whether or not to participate (like confidentiality assurance, time, the number of teachers, sample procedures, research application, the questionnaires and all questions)?

• How would the decision be made to participate or not? For example, who would be involved in the decision? (District personnel, other school leadership like assistant principals and department heads…)

• What would be the best way to reach them? By phone, email, in-person visit?

• Would the support—or resistance—of any particular person or group be particularly critical in the decision to participate? Whose? (District officials? Principal colleagues? Teachers? Parents? School Boards?)

BARRIERS TO —ADMINISTERING TALIS (10 MINUTES)

• Let’s take a few minutes to talk about what it would be like to administer TALIS in your school. [BRIEFLY—use as a way to introduce details about TALIS]

TALIS is an international survey of teachers of grades 7, 8, or 9 and principals that focuses on the working conditions of teachers and teaching and learning practices in schools; TALIS is administered online, with two separate questionnaires: one for teachers about their practices, and one for principals about the school. It takes approximately 45 minutes to complete the principal questionnaire and 60 minutes to complete the teacher questionnaire.

• Participation is voluntary.

• How do you feel about TALIS, or other surveys, being administered online?

• What challenges or benefits are associated with online administration vs. paper?

• How do you feel about having a sample of teachers participate in the survey versus all teachers in the school?

• Based on the description of the survey, are there any other challenges or burden you would anticipate?

• How likely would your school be to participate in TALIS if you were selected in the sample? NOTE: You are not being asked to participate. Nothing you say in this group will commit you to participating. We want to gauge your reaction to the concept of participating in TALIS.

[FOR USE ONLY WITH PARTIPANTS WHO RESPONDED NEGATIVELY]

• Is there anything that may change your mind about participating in TALIS?

• Would something about your school need to change? What?

• Would something about the survey need to change? What? (PROBE FOR timing, access, time required, etc.)

• If one thing could change about TALIS, what would be most important to increase the likelihood that you would participate?

• Of the other studies and assessments your school participates in:

• Which is the easiest? What makes it easy?

• Which is the most difficult? What makes it difficult?

• Are there any other barriers that you see to participating in TALIS? Are there any other concerns or negative reactions that you have?

BENEFITS OF TALIS (10 MINUTES)

• Let’s talk a little more about the benefits to participating in TALIS. What do you think would be the benefits of having your school participate in TALIS? (PROBE FOR benefits to staff, students, community)

• Which of these benefits is most important to you?

REACTIONS TO TALIS 2018 MATERIALS (30 MINUTES)

• Let’s turn back to the materials. Have you had an opportunity to review the draft materials for TALIS 2018? (GIVE MORE TIME TO LOOK THROUGH IF NEEDED)

• What was your overall reaction to the packet of materials? Let’s start with <Name> and have everyone give me one statement about their reaction to the materials.

• If you were going to tell another principal about TALIS, which one or two of these materials would you share? [HAVE EACH PARTICIPANT SAY WHICH ARE THEIR TOP 2; TALLY]

• Why did you choose these two?

• Let’s talk about [PULL UP ON WEBEX SCREEN/NAME A MATERIAL MENTIONED; REPEAT SEQUENCE FOR EACH MATERIAL MENTIONED:]

ORDER: Letter from Peggy Carr, TALIS Brochure and timeline, School Leader FAQ, Teacher FAQ, Summary of Activities for School Coordinators

• What made it useful?

• What did you like about it?

• What didn’t you like?

• Was anything confusing or hard to understand?

• Was anything missing?

• How believable was this information?

• How relevant was the information to you? Did it speak to you (as opposed to someone in a different position or role)?

• What did you think about the length?

• How does this compare to the information that you receive on other types of studies?

• What would you change?

• We’ve been talking about what materials you found useful. Were there any materials in this packet that were unnecessary? Which ones? What made them unnecessary?

RECRUITMENT STRATEGIES AND SOURCES OF INFORMATION (10 MINUTES)

• This packet represents one way to communicate about TALIS with principals like you.

• Where does a packet of materials rank on the list of best ways to communicate with you?

• What would work better or be preferable to you? PROBE:

• Telephone call—from whom?

• Email?

• A personal visit?

• How helpful would it be to receive multimedia materials? PROBE:

• What about a short video that provides an overview of TALIS? How would you feel about it in addition to, or instead of, these materials?

• Or, what about a pre-designed PowerPoint presentation to use in a meeting with your teachers to help them understand their role in TALIS and what they are being asked to do?

• In what format would you prefer to receive materials? Hard copy? Electronic? Both?

If hard copy, how many sets of materials would be useful for you? What is the best way to send materials so that they don’t get misplaced or tossed? Should they be addressed to someone else in a specific role?

• Would any additional resources be helpful for encouraging participation among your teachers? (PROBE: Would sample questions be helpful?)

• Where do you get information about what’s happening in education? About what’s happening in your school system?

• What sources would you trust the most in getting information about assessments?

• Where would you go to get additional information about TALIS?

CONCLUSIONS

• Do you have any additional comments about these materials, TALIS, or assessments in general?

• Thank you.